You want to know whether the thing you just added is helping, so you run it both ways, on and off. The results come back identical. You are about to conclude it does nothing and take it out.
Wait. "No difference" always has two explanations. Either the thing genuinely does not help, or the side that was supposed to be off was never actually off. On screen those two look exactly the same.
We hit the second one ourselves. We were answering a question that sounds simple. Does this skill change the outcome compared to not having it?
The method is equally simple. Run the same task through two arms: one with the skill loaded, one with every settings source switched off. Then score both answers with the same checker.
The run came back inconclusive. Both arms did about equally well. We nearly wrote in the report that the skill was still unproven.
Then someone wondered: what if we run the command from an empty folder instead?
| Working directory | Control arm result | Verdict the tool printed |
|---|---|---|
| The project folder | passed 3 of 3 | inconclusive |
| An empty folder | passed 1 of 3 | the skill is not doing its job |
Same task, same model, same flags. Only where we stood changed, and the verdict flipped completely, on a variable nobody had counted as a variable.
Part 1A flag tells you what it strips, not what it leaves
The flag we used to strip context from the control arm reads like it removes everything that comes from a settings file. And it does exactly that.
What it does not remove is the project rules file sitting in the folder the command was launched from, which arrives through a different channel entirely. And we never moved where we stood. Every run started from the project folder.
That rules file contains a line naming the very rule we were testing. So the control arm read that rule on every run and followed it correctly, not because the model is good but because it had been handed the answer key.
The arm that was supposed to be empty had been holding the thing under test the whole time.
This pattern is not specific to measuring skills. Every time you set a flag that says "turn this off" and then trust the flag's name, you are inferring what you should be measuring. A flag is a promise, not evidence.
Part 2How to prove a control arm is clean
What works is inverting the question. Instead of asking "is the control arm clean?", find a condition the control arm must fail, then measure that it fails.
If it still succeeds, it is holding something you believed you had removed.
Here the condition was: if the control arm never read the rule, it should break that rule at least some of the time. Moved to an empty folder, it broke it in 2 runs out of 3. That is the evidence that in its old location it had been reading it.
We then turned that condition into a guard inside the runner. It walks up from the working directory all the way to the filesystem root, and stops immediately if it meets a project rules file, naming where it found it.
And we made it possible to force it red with a single variable, so we could watch it fire rather than trust a green that had never been tested.
Part 3The first check we wrote for this broke the same way
Before the file-walking guard, we tried something that looked smarter: ask the model directly whether it could see the house context, by writing into the prompt something like if you see the text of this rule name, answer LEAK.
It answered LEAK in both directions. The arm that should leak, and the arm that should not.
The reason is obvious once you see it. Our own prompt spelled out the rule name, so the model saw that token in the prompt rather than in the context, and answered honestly as instructed.
A check must never mention the token it is looking for, and a check that cannot tell the two directions apart is a check that does not exist.
So we dropped the ask-the-model approach and switched to inspecting files, because a file answers the same way every time and needs no interpretation.
The same day produced 3 more of the same family: a result that reads like an answer but does not answer the question that was asked.
- An error message counted as an answer. The guard only checked that the output file was non-empty, so a line saying the monthly spend limit had been reached sailed through and became a score for the skill.
- A checker that can only say pass or fail folds "unjudgeable" in with "fail", even though those two demand completely different next steps. It needs a third value.
- The evidence was deleted exactly when it mattered. On a failing round, the temp-folder cleanup ran before the line that saves the artifacts, so the files left on disk belonged to the previous round and were read as evidence of the round that had just failed.
That last one nearly had us conclude the control arm answered well, on a round where it produced no answer at all.
Part 4The direction of the bias tells you which old results survive
Once you know a control arm was contaminated, the first instinct is to throw out every past result. That is expensive and usually unnecessary.
Ask first: which way does this bias push?
This contamination makes the control arm stronger than the truth. Only 2 outcomes follow.
| Old result | Could the bias fabricate it? | What to do |
|---|---|---|
| Proven that the skill helps | No. A stronger control only makes passing harder. | still stands, no re-run |
| Inconclusive | Yes. An over-strong control produces exactly this. | must be re-measured |
Work out the direction first and you will find only half the results need re-running, not all of them.
Part 5Applying this to your own measurements
None of this is specific to evaluating skills. It applies to any experiment decided by comparison, from testing 2 versions of a page to comparing 2 models.
- Write down what the control arm must not have, then measure each item. Do not infer it from a flag's name.
- Find a condition the control arm must fail. If it still succeeds, it is carrying something you thought you removed.
- Make the guard forceable to red with one command, and run it that way once. A guard never seen red and a guard that is dead look identical.
- Never let a check mention the token it is hunting for, or you will get the same answer in both directions and mistake it for working.
- Work out the direction of the bias before discarding old results. Some kinds of result survive contamination by their own nature.
The hard part was never writing the measuring tools. It is the willingness to doubt the instrument before doubting the thing being measured. On the day this happened we misread results 4 times, and every one of them was caught by comparing against the real thing, not because any code complained.
For more ways a measuring tool can lie to you, read when the check says pass without having checked and the tool built to catch mistakes, making them itself. For the wider picture of measuring a whole skill library, see 60 skills were never called, so we measured before deleting. Everything else lives on the blog index.
Sources and references
Everything here comes from runs against our own tooling on 13 August 2026. No numbers are borrowed from elsewhere.
- The working-directory comparison table came from the same suite run twice, differing only in the folder the command started from.
- The guard that walks up looking for a rules file ships with a way to force it red, and was run red before it was used to judge anything.
- The account of the check that named its own search token was read from the raw output of the round where it answered identically in both directions.
- All 3 symptoms listed in Part 3 were fixed the same day; none of them is a theoretical observation.