The passing test was telling us to delete real customers

Yesterday I wrote a test suite for a course registration form. Nearly 100 checks. All passing.

Today I read them one by one and found one that said the signups table in the live system must contain zero rows. That was true when I wrote it, because nobody had signed up yet. Six people have now. The check had quietly become false and nobody knew.

The problem is not just that it is wrong. The problem is that the quickest way to make it pass again is to delete those six people. Whoever meets it on red, without knowing where it came from, is looking straight at the fastest fix.

Part 1Why 100 checks saw none of it

What found it was not writing check number 101. It was handing the work to a reviewer that had not written the code. It came back with 8 things the whole suite was blind to.

The first three are all the same shape: things a person filling in the form can do that the person writing the tests never pictured. Send an empty body and the server falls over. Send the tickbox as text rather than a true-or-false value and the system accepts it and saves the record anyway. Decline consent and your name and email stay on your machine indefinitely.

Every check in the suite sent well-formed data, because that is what its author imagined people sending. No check ever declined consent, because nobody thought of it. The same person wrote the tests and the code, so the blind spots are the same set. Adding cases means examining what you already thought of in finer detail, not examining what you did not.

This is the same reason accounting separates the person who prepares the statements from the person who reviews them, which I wrote about in professional skepticism, the review skill AI cannot replace. The preparer asks whether everything was recorded. The reviewer asks whether the numbers make sense. Different questions, and one person cannot hold both at once.

As for the check that would have you delete real records, it comes with a short rule I did not want to learn this way: never write a check that asserts the global state of a live system. The table is empty, there are no errors in the log, the queue is idle. Each becomes false the moment real people arrive. Assert only the data the test itself created.

Part 2Passing with nothing examined

That case is still the good kind, because the test did run, it just asked the wrong question. The worse kind reports passing with nothing examined at all. I found two of those the same day, auditing this blog's own tooling.

The first: a script that checks every post has been added to an index file. It reported all 86 posts present, none missing, while the post I had just finished writing was nowhere in that file. It had taken its list of posts from another file that had not been updated, so the new post was never in the list being compared. And whatever is not in the compared list cannot be reported as missing.

The number 86 looked authoritative enough that nobody counted it.

The second: the gate that runs before anything goes live calls another tool to inspect the pages. One day it started printing that it could not read that tool's result, and then concluded everything was fine. Half the gate had stopped working, with 203 files going unexamined.

The dangerous part is not that it broke. It is that it broke and still said pass. Had it stopped and said it could not check, somebody would have fixed it long ago.

Part 3An explanation that fits is not the cause

Chasing why that result could not be read, I was wrong twice.

First I saw the tool's file was only 620 bytes and concluded it was damaged or half-installed. In fact it is a small file whose job is to call the real one, working perfectly. Then I went looking for the option that tells it to print results in a machine-readable form, could not find it, and concluded the tool did not support one. I had opened the wrong file; the option lives in another one I never looked at.

Both explanations fit the symptom. Both were wrong.

What ended it was not thinking harder. It was re-running the exact call and printing the raw numbers. Seeing the result cut off at precisely 8192 bytes settled it, because 8192 does not appear by coincidence: it is the size of the pipe carrying the result between two programs. The tool wrote into the pipe and shut itself down before the contents had drained.

I am keeping the wrong turns here because they are part of the real work. Telling only the final answer would make it look like one clean diagnosis. And the lesson lands on the same place as part one: the person who built the thing already has a plausible explanation in their head, so a person or a number from outside is worth more than thinking about it harder alone.

Part 4Break it in front of yourself first

The cheapest defence is refusing to believe it passes until you have watched it fail.

Fixing that first script, what I did was create one fake post directory and run it. If it did not go red and name that directory, it was not checking anything. It went red, so I deleted the fake directory and watched it return to green. Under two minutes, and it was the first time I knew the script actually checked.

In lab work this is called a positive control: you measure a sample whose result you already know. If the instrument cannot find what you know is there, its report of finding nothing carries no information.

Three things to do today

1. Break it once, deliberately. Plant one fault. You should see it go red and point at the right place. If it stays green, it is not checking that.
2. Read the count, not just the word pass. Passing with a count of zero, or a count that does not move when you add something new, means the compared list is empty.
3. Do not stop at the summary line. Open the actual result. "No problems found" and "never ran" print the same way.

What changed how I work is no longer reading "pass" as an answer, and reading it as a question instead: did this pass because the thing is sound, or because nothing was examined? You only know once you have seen it fail.

Related: mutation testing, deliberately breaking the code to see whether the tests notice, answers the next question along: now that they are checking something, are they strong enough?