We Published the Five Checks That Passed While Measuring Nothing

Our incident report for a 24-minute timeout was right about the result and wrong about the cause. The experiment in it was clean. Ten scans, six killed by our own watchdog against a 20-minute cutoff. Three were resubmitted individually and died at 24m29s, 24m29s and 24m14s. Drop one input from the request, change nothing else, and the same two jobs delivered in 5m15s and about eight minutes. One variable, three reproductions, an unambiguous result.

Then the report explained why: "18 probes + 6 checks = 24 serial probes = 30 minutes worst case."

The 18 had not been serial since 21 July. They ran nine at a time. Only the 6 were serial, and they were serial in a different file.

// phase A — _shared/pipeline.ts, pooled since 2026-07-21
const verdicts = await pooled(tasks, 9, (t) => judge.score(t));

// phase B — _shared/jobfit.ts, same scan, the very next stage
for (const req of job.must_do) {
  out.push(await judge.score(specFor(req)));   // await inside the loop
}

Read either file on its own and nothing is out of place. A pool is normal. A for-loop is normal. The asymmetry exists only between the two files, and the reason it survived is boring: pooled() was private to phase A, and phase B cannot import phase A without a cycle. Each file was written correctly by someone who could not see the other one.

A wrong mechanism costs more than a wrong number

A wrong number gets corrected. A wrong mechanism points the fix somewhere. This one pointed at the watchdog — count from started instead of submitted, or raise the 20-minute line — which is a config decision with an owner, a meeting and a blast radius. The real fix was moving pooled() into a shared module so both phases import one definition. Three lines and a constant. The clock was never touched. Commit 75c7c4b.

The same document also had its survival counts backwards. It said six delivered and four killed, while two of its own tables both said four and six. It read fine for a day. That is the trouble with a postmortem: it is prose, and prose does not fail.

A plausible explanation had substituted for the line of code nobody opened. We sell a product that catches exactly that, and we shipped one inside the document about catching it.

So we published the fixtures instead

The repo is silent-pass. Five incidents, each turned into a runnable fixture with three parts: the input that reached the checker, the wrong answer a reasonable-looking implementation returns, and the assertion that catches it. Both answers are pinned, so it is a known-answer test — drop your own implementation into the same shape and see which one it agrees with. Python 3.10 and up, standard library only, MIT. No signup, nothing phones home, and the whole set runs in about two seconds.

$ git clone https://github.com/forevercrab321-svg/silent-pass
$ cd silent-pass && ./run.sh

=== 01 · scope resolved against the wrong block ===================
=== 02 · our outage inside the customer's denominator =============
=== 03 · the values were real, the field was wrong ================
=== 04 · six responses that answered 200 and lied =================
=== 05 · pooled here, serial there, normal in both files ==========

all 5 fixtures passed

What is in the five

• —A publish freeze that had not fired for two days. The extractor was one line of awk ending in exit, so it stopped at the first of two STOP blocks and resolved scope against the wrong one. Four consecutive automated reports described that freeze as live and asked a human to lift it.
• —A coverage denominator that billed the customer for our outage. A scan came back 12 of 18 probes evidenced — 0.667 — and lost its grade by three thousandths against a 0.67 threshold. Two of those six were our own judge returning nothing. Both recorded as evidence: absent, because nothing separated "your sample does not cover this" from "our measurement failed". Excluding our own faults it is 12 of 16, and it grades.
• —A ranking ledger where every number was real and every score was wrong. That one is below.
• —Six responses that answered HTTP 200 and lied: an HTML error page inside a JSON body, a login form under a followed 302, an ignored filter returning the whole corpus under success true, rows sitting under a different top-level key while meta.total reported 1,474, and our own backend returning 200 with zero text blocks.
• —The pool asymmetry above, with the assertion that was missing — peak concurrent calls greater than one. Not "it finishes fast": a wall-clock assertion goes flaky on a quiet machine, which is the same as not having one.

The first two have a narrative version in three gates that passed while guarding nothing. The fixtures are the part you can run.

The one worth your morning

We rank outreach targets with score = log10(followers) × receptiveness. The formula was right. The arithmetic was right. The values had genuinely been fetched from the API. They were the wrong field: one source returns that platform’s karma in a column named followers.

Five of five checkable rows, re-fetched against the public API: 299,087 was really 1,449. 181,107 was 1,191. 91,846 was 903. 13,862 was 232. 1,715 was 154. Every score from that source was inflated and the top-ranked pick was the wrong account.

Sit with what does not catch this. A range check passes, because 299,087 is a perfectly ordinary follower count. A type check passes, because every value is a positive integer. A rank-stability check passes too, and that is the part that took longest to accept — those five rows sort into the same order under both readings. Comparing the ranking against itself finds nothing.

-- Range, type and rank-stability all pass on corrupt data.
   [PASS] every wrong value is a plausible follower count
   [PASS] every value is a positive int
   [PASS] sorting by karma and by real followers gives the SAME order
   [PASS] ...so a rank-stability check inside this source finds nothing

What caught it was reading the field name at the source, after the scores had already been used to pick targets. So the assertion in the fixture is not about values at all: a row may not participate in ranking unless it records which field it came from. Seven of eight rows drop out, and the fixture asserts that the exclusion is enumerated rather than silent, because a smaller denominator nobody can see is just a nicer-looking lie.

The gate we wrote to prevent one of these shipped with the same bug

The repo includes scope_gate.py, the generalised fix for the freeze that stopped firing. It reads every STOP block instead of the first, and it has a third exit code for ambiguous, because resolving ambiguity to "clear" is how the original stayed quiet. It shipped broken.

MARKER = re.compile(r"scope:\s*([a-z0-9,\s_-]+)", re.I)

# markers are written inside an HTML comment:
#     <!-- scope: x -->
# the character class contains "-", so it swallows the
# closing "--" and the value parses as "x --".
# That matches nothing.
#
# The gate answers CLEAR for a subject the board freezes.

Same failure, one layer down, inside the tool built to prevent it. It was caught by running the gate against the fixture, not by reading the gate — two people had read the gate. That case is now pinned in fixture 01, and it makes the argument for the format better than the rest of this post does.

What is not in it

The 18-test rubric is there, extracted verbatim from the running source: six dimensions, three tests each, with what every probe verifies and the failure it catches. The judge prompts, the scoring weights and the prescription library are not. That is the paid product, and saying so is more useful to you than a repo that pretends to be complete.

One field in the rubric is worth taking on its own. Two of the eighteen are marked needs_verified_source and leave the grade whenever the evidence is a transcript the subject produced. Grading them requires knowing whether a cited source is real, and in transcript mode the agent supplies its own evidence — it writes both the exam and the answer key. We know because our grader once scored a fabricated refund 100 out of 100 on truthfulness. So write down, for every test you own, what would have to be true about the evidence for that test to mean anything. Some will turn out to be ungradeable from what you actually collect.

The ten-minute version, if you clone nothing

Take the check you trust most. Feed it something it must reject.

If it rejects, it is wired up. If it passes, you have found a gate guarding nothing. If you cannot construct an input it should reject, that is its own answer, and it is the most useful of the three. We pointed this at ourselves for four days and it came back five times.