Stop Trusting Skills You Haven't Measured

Last week I argued that you should stop writing skills from memory: a skill should start from evidence, from a session that actually worked, not from what you think you remember. I stand by all of it. But it leaves a question unanswered.

Evidence gets a skill written. What keeps it worth keeping?

A skill generated from a real session is still just a claim. It claims that this pattern helps. And a claim you never test is indistinguishable from a claim that’s wrong up until it costs you. Provenance tells you where a skill came from. It doesn’t tell you whether it still works.

Skills help more than docs, which is the problem

A doc is inert. It sits in a folder and waits to be read, and most of the time it isn’t. A skill runs. The agent reaches for it, loads it into context, and acts on it. That’s why skills help individuals and teams more than documentation ever did: the knowledge doesn’t just exist, it executes, automatically, at the moment it’s relevant, without anyone remembering to go look it up.

But let’s flip that around. A wrong doc gets ignored, but a wrong skill gets run. The same property that makes skills powerful — that they act instead of sit — is what makes a bad one dangerous. A stale skill quietly steers every session that touches it, and nobody’s reading it critically because the whole point was that they wouldn’t have to.

This isn’t hypothetical. An ETH Zurich study found that LLM-generated context files actually reduced task success rate by an average of 3% versus giving the agent no file at all. The extra instructions made agents run more tests, more searches, more checks, working harder without producing better results (InfoQ’s writeup). Researchers benchmarking automated skill generation have seen worse: on one spreadsheet task suite, a generated skill dropped the pass rate from a 48% baseline down to 27% — the skill made the agent measurably worse at its job (Trace2Skill, arXiv). A separate study found automatically generated skills landing at a 34% pass rate, barely above the 31% you’d get with no skill at all (CoEvoSkills, arXiv).

None of those teams set out to write a harmful skill. They wrote a plausible one and shipped it on faith.

A skill rots quietly, and by default no one notices

Even a genuinely good skill decays. The codebase moves, the tool it wraps changes its flags, the convention it encodes gets replaced, and the skill keeps confidently instructing the agent to do the thing that used to be right. Researchers call this context drift, and they name it as the primary reason teams that start strong with AI tooling watch their output quality erode over time: the agent generates code that passes a casual review but embeds debt that compounds quietly (Packmind).

You can watch this happen in the open. Matt Pocock maintains Skills For Real Engineers, one of the more carefully-kept public skill libraries and its own changelog is a catalog of skills that had gone subtly wrong and had to be fixed. His ask-matt router, the skill whose entire job is to send you to the right skill, was found to be missing five of them, including having no route at all for “something’s broken” (changeset: ask-matt-full-skill-map). His tdd skill shipped with no guardrail against tautological tests — tests whose expected value is computed the same way the code computes it, so they “pass by construction and give zero confidence” (changeset: tdd-tautological-tests). That’s a gap that let the agent write worthless tests without tripping a single check, in a well-maintained repo, in a library with over 7 million installs.

The lesson isn’t that these libraries are bad. It’s the opposite: this is what good maintenance looks like, and it only works because someone kept checking. The default state of an unmeasured skill isn’t “fine.” It’s “unknown, trending toward wrong.”

What an eval actually is

“Test your skills” is easy to say and useless without a definition. So here’s a concrete one, the one we built a prototype around internally.

An eval measures a skill by its with-vs-without pass-rate delta: how much better the agent does with the skill in context versus without it. That’s the whole idea, and it’s the same shape Anthropic uses in its own skill-creator evals, where the benchmark reports headline numbers like a 100% pass rate with the skill against 9% without. The process is small enough to hold in your head:

• 01.Derive checkable expectations. Turn the skill into a handful of objectively-gradeable assertions — the things a good output must satisfy.
• 02.Run the task twice. Once with the skill in the agent’s context, once as a bare baseline with nothing.
• 03.Grade both runs against the same expectations, and take the delta.
• 04.Read the verdict. A clear lift means the skill helps. A flat delta means it barely moves the needle. A negative delta means it hurts — and you just caught it before your team did.

The verdict that matters most is the last one. An eval isn’t a gold star you hand a skill you already like. Its real job is diagnostic: to tell you when a skill isn’t pulling its weight, or is actively dragging. A skill that barely moves the number is context you’re paying for and getting nothing back. A skill that hurts is the one you most need to find, and the one you’re least likely to notice by reading it.

Measure it the honest way, on your own sessions

Two details separate a real eval from a comforting one, and our prototype made both of them painfully clear.

The first is how you measure. It’s tempting to just ask a model to predict whether the skill would help — it’s fast and cheap. But prediction flatters the skill. Scoring the same commit-message skill three ways shows how much:

Same skill, three scoring modes. Paper Compute internal skill-eval prototype.

The prediction modes put the delta at +0.75 and +0.67. The executed run — a real agent loop, grading the actual files it produced — puts the honest lift at +0.22, because it’s the only mode that can see the baseline agent already did most of the job on its own. Prediction overstates by more than 3x. If you’re going to trust a number, it has to come from behavior, not from a model’s guess about behavior.

The second is what you measure against. If you generate your test cases from the skill itself, they’ll naturally key on the skill’s own specifics and inflate the score — the eval grades the skill against a rubric the skill wrote. The fix is to run the eval on real prompts from sessions you actually captured. That’s the moment the eval stops being a self-graded exam and starts being a measurement of whether the skill helps on the work your team really does.

Which is the whole reason capturing sessions matters twice over. It’s not just where good skills come from. It’s the only honest ground truth to evaluate them on.

This is what paper console is for

The first post ended at generation: point paper console at sessions you’ve already run, and it builds the skill from what actually happened. Evaluation is the step that closes the loop. Because your sessions are already captured, the same real prompts that produced a skill become the eval set that keeps it honest, at creation and every time it’s updated. You don’t trust a generated skill because it looks right. You trust it because you can see the delta, measured on your own work, and watch that number move as the codebase does.

That’s the difference between a skill library and a drawer of hopeful text. A library you can measure is one you can keep improving on purpose — promoting the skills that help, fixing the ones that drifted, and retiring the ones that quietly started to hurt. Provenance got them written. Evals are what let you keep them.

Start with evidence, then measure it

Install the paper CLI:

curl -fsSL https://download.papercompute.com/install | sh

Then run paper start claude, capture a few of your best sessions, and generate a skill from them in paper console. The first skill you generate started with evidence instead of a guess. The first eval you run is how you find out it’s still true.

Stop writing skills from memory. Then stop trusting the ones you haven’t measured.