A Claude Code skill is a markdown file of instructions the agent loads for a task: run this script first, never edit that file, append the log in this exact format. Turn one, freshly loaded, they work. My question was about turn ten: after the conversation has moved through other topics and the context has filled with unrelated material, does the agent still follow the procedure, or does it quietly start improvising?
I built skill-vibe-test to answer that with a measurement instead of a feeling. Then I pointed it at one of my own skills, and I got two answers. The first was that the skill behaved badly. The second, which took me a month to go back and find, was that my run could not support the claim I wanted to make from it.
The probe
One test run is a scripted 10-turn conversation with a real agent, driven headlessly through session resumption so the context accumulates for real. Turn 1 gives the base task and establishes the baseline. Turns 2 through 5, 7, and 9 are fillers, several hundred words each of deliberately unrelated content (refrigeration cycles, trip planning, photosynthesis) whose only job is to dilute the context. Turns 6, 8, and 10 probe again with skill-relevant tasks.
Scoring is where most eval ideas die, so the judge is deliberately narrow. Before the run, a cold call extracts an observable-behavior rubric from the skill: five to eight checkable criteria like “runs the health script before any analysis, visible as a Bash call” or “never edits the capabilities file directly.” The judge then scores each probe against that rubric using the tool-call trace, what the agent actually executed, not its prose. An agent that writes a lovely paragraph about running the script while not running it fails.
Each verdict also carries an escape-hatch flag, for the distinct failure where the agent doesn’t just miss steps but abandons the procedure entirely and hand-rolls its own approach. That distinction matters: partial compliance degrades output, escape-hatching means the skill has effectively unloaded.
There’s one more piece, and it’s the piece this post is really about. The baseline is a gate. If the skill cannot steer a fresh, undiluted context, then whatever happens at turn 10 isn’t decay, it’s a skill that never worked. So the harness stops:
if cfg.baseline_gate:
result.status = "baseline-failed"
result.error = (
"the baseline probe (turn 1, pre-dilution) already fails, so a "
"decay curve from it is meaningless. Fix the skill or base "
"prompt first (or pass --no-baseline-gate)."
)
return result
First target: my own tooling
The first real subject was capabilities-health, a maintenance skill of mine with a strict procedure. Two conversations against Sonnet, judged by Sonnet, $1.77 for the run.
Every probe failed. Both baselines, both turn 6s, both turn 8s, both turn 10s. Eight out of eight.
Baselines failed on precision. The agent ran the census script and did the analysis, then replaced the existing line in my decision log instead of appending a new one, and in the second conversation read a whole skill file where the procedure says to read only the frontmatter. Then the middle turns got worse, and the judge flagged four of them as escape hatches.
That’s the point where I wrote this post the first time, and the sentence I wrote was that the decay curve was textbook.
What the run actually supports
Going back to the transcripts a month later, three things were wrong with that.
There’s no curve. Eight failures out of eight is a flat line at FAIL. A metric that returns the same value at every point cannot show a trend. Here is that run, redrawn from its saved transcripts by the current harness:
The red banner is the fix described at the end of this post, applied to the run that motivated it. The original chart carried no such warning, which is exactly how I came to publish it.
I had read that as a decay curve because I already believed the conclusion.
The run should not have completed at all. Both baselines failed, which is exactly the condition the gate above exists to catch. The saved runs are marked completed rather than baseline-failed, which means the run was executed with --no-baseline-gate. I wrote a gate that says a decay curve from a failing baseline is meaningless, then turned it off and published the curve. I don’t remember making that decision, which is worse than making it deliberately.
Two of the four escape hatches are over-called. In the second conversation, at both turn 6 and turn 8, the agent ran capabilities-health.py, which is the first and most important line of the rubric. It then skipped the log append and the diff proposal. That’s partial compliance. My own definition of escape-hatching is abandoning the procedure entirely, and running the required script isn’t that. The judge stretched the label and I repeated it.
What survives, and why I trust that part
One finding holds, and it holds without trusting the judge at all, because it is visible in the raw tool trace.
In the first conversation, the agent was asked at turn 6 whether any skills on disk were missing from the index. It made zero tool calls and answered from memory. At turn 8 it was asked whether the index reflected two newly added scripts. It made one call, an ls of two directories, and reasoned about the answer from that.
Compare that to the same agent’s baseline in the same conversation, where it made fourteen calls including the census script the procedure requires. The skill didn’t stop working between turn 1 and turn 6. It stopped being consulted. The agent went from running the prescribed script and getting the log format wrong, to not running the script at all.
That’s a degradation in the behavior the skill was written to produce, and you can check it yourself in the committed transcripts without agreeing with a single one of the judge’s verdicts. It’s a smaller claim than the one I originally made. It’s the one I can defend.
For the record, the honest version of the numbers: context went from about 31k to 40k tokens in the first conversation and 38k to 54k in the second. My original write-up quoted “31k to 54k,” which is the start of one run and the end of the other.
Two lessons from building the harness
Measuring context size looked trivial and wasn’t. The CLI’s reported usage aggregates every internal iteration of an agentic turn, and summing it produced an impossible 256k “context.” The truthful number is the last iteration’s input picture, the final API call’s input plus cache reads. Read the last element, not the sum.
The subtler one: the subject can see the experiment. Early sandboxes were temp directories prefixed skill-vibe-, and a transcript showed the agent reading its own path and reasoning about “this skill-vibe checkout,” aware it was being tested. Sandboxes are now prefixed repo-. Anything the subject can observe (paths, file names, stub contents) is part of the experiment, and it will read the label on the jar.
What it changes
The tool exits 1 on any decay, so a skill can be gated on surviving its own vibe test before it ships. For skills that keep failing, the fix is usually structural rather than rhetorical: move the invariant out of prose and into something the agent cannot drift from, a script, a hook, a hard check.
The harness change I made after re-reading this: a bypassed gate is now loud. When the turn-1 probe fails and --no-baseline-gate is set, the run records it and every artifact says so. The terminal summary leads with it, the report carries it as a banner above the results, and the chart gets a red stamp across the top reading “BASELINE GATE BYPASSED: turn 1 already failed, so this is not a decay curve.” Not just a status field in a JSON file nobody opens. A safeguard you can silently switch off is a safeguard that will be silently switched off.
It’s easy to build an instrument, point it at something, and then read its output through the conclusion you were hoping for. The judge was too generous with one label. I was too generous with the whole run.
Trust in an agent’s steering should be measured, not vibed. So should trust in the thing doing the measuring, which is a funny lesson to get from a tool with vibe in its name.