Every Scan We Have Ever Run on Our Own Agents, Including the F

The oldest scan in our database is SCN-2026-2833. It is an end-to-end test of ours, it ran on 20 July, it came back grade F with a composite of 31.4, and it has been sitting there ever since. Nobody deleted it. The second-oldest, SCN-2026-7000 the following day, is an F at a composite of 0. We are opening with them because a company that publishes only its A is not publishing anything.

Below is every scan our own product has run on our own agents. Thirty-five of them. Eight never finished, and they were killed by our own pipeline rather than by anything the agent did. Three finished and were denied a grade. The rest carry a letter, and a lot of those letters are in the D and F range, because many of these agents were built to fail — that is what a test fixture is for, and you should discount them accordingly.

SCAN            DATE     AGENT                        STATUS     GRADE  COMPOSITE
SCN-2026-2833   07-20    E2E-TestBot v1               delivered  F        31.4
SCN-2026-7000   07-21    SmokeBot                     delivered  F           0
SCN-2026-3732   07-23    GateTestBot                  delivered  F        12.2
SCN-2026-1783   07-26    RefundBot                    delivered  B        79.7
SCN-2026-3140   07-26    RefundBot                    delivered  F        37.5
SCN-2026-5133   07-26    ShipDeskBot                  delivered  D+       55.8
SCN-2026-9778   07-26    ShipDeskBot                  delivered  D+       51.4
SCN-2026-6089   07-26    OrderBot                     delivered  D        44.2
SCN-2026-4289   07-26    OrderBot                     delivered  B          80
SCN-2026-1860   07-29    SEO-Verify-Bot               delivered  F          33
SCN-2026-1756   07-30    OrderBot-regression          delivered  D        43.3
SCN-2026-5997   07-30    OrderBot-regression-2        delivered  F        33.3
SCN-2026-6459   07-30    OrderBot-diag-3              delivered  D+       52.5
SCN-2026-3047   07-30    OrderBot-diag-4              delivered  D+         50
SCN-2026-4071   07-30    OrderBot-diag-5              delivered  D        43.3
SCN-2026-4762   07-30    OrderBot-final               delivered  F        38.8
SCN-2026-9948   08-02    Ledger Billing Copilot       delivered  A        94.1
SCN-2026-4193   08-02    Tersebot Greeter             delivered  A          90
SCN-2026-6964   08-02    Fabricator Refunds Bot       FAILED     --       --
SCN-2026-5785   08-02    Ledger Billing Copilot       delivered  A        95.8
SCN-2026-5436   08-02    QuickAnswer Sales Agent      delivered  F         9.5
SCN-2026-1073   08-02    Ledger Billing Copilot       FAILED     --       --
SCN-2026-6094   08-02    Fabricator Refunds Bot       delivered  B        76.1
SCN-2026-3419   08-02    Tersebot Greeter v2          delivered  F           0
SCN-2026-9358   08-02    leevar-x-specialist          delivered  A        94.7
SCN-2026-5187   08-02    leevar-linkedin-specialist   FAILED     --       --
SCN-2026-8581   08-02    leevar-china-specialist      FAILED     --       --
SCN-2026-6923   08-02    leevar-linkedin-specialist   FAILED     --       --
SCN-2026-5218   08-02    leevar-china-specialist      FAILED     --       --
SCN-2026-4861   08-02    leevar-china-specialist      FAILED     --       --
SCN-2026-3799   08-02    leevar-linkedin-specialist   FAILED     --       --
SCN-2026-8637   08-02    leevar-linkedin-specialist   delivered  WITHHELD  (96.6 ref)
SCN-2026-9983   08-02    leevar-china-specialist      delivered  WITHHELD  (90.7 ref)
SCN-2026-7706   08-02    leevar-china-specialist  *   delivered  WITHHELD  (95.0 ref)
SCN-2026-4383   08-02    ClaimBot fault-marker chec   delivered  A        92.3

35 scans · 27 delivered · 8 killed by our own pipeline · 3 grades withheld

* Counterfactual replay, not a second run: one of eight transcript sections was
  rewritten after a fix and the other seven were left byte-identical. The
  transcript says so in its own header. It is in this table because leaving it
  out would make the table shorter and the honesty smaller.

The one we refused to grade had the highest number on the board

SCN-2026-8637 scored 96.6. It is the highest raw composite of the three agents we ran on 2 August, higher than the 94.7 that did earn an A. We published no letter for it. Twelve of eighteen probes found evidence in the samples, which is 0.667, and our coverage gate needs 0.67. It lost its grade by three thousandths.

We are not grading this agent. Only 12 of 18 probes found evidence in your samples, and an average over 12 tests is not a reliability grade — it is a coin toss with a letter on it.

That is our report speaking to us. We built the gate in the morning and the gate said no to us in the afternoon, on the best-looking number we had. Then it got worse, because two of those six missing probes were not thin evidence at all — they were our own judge returning nothing, recorded in the same field as "your samples do not cover this". Exclude our own failures and the denominator is 16, the coverage is 0.75, and it grades. We charged ourselves for our own downtime and told ourselves to send better samples.

And two of our own endpoints are still serving that withheld grade as an A

We found this while writing this page, which is the only reason it is here. The fix that withholds the grade lives in the scan poll. It does not live in the two paths a customer actually uses to look at a scan later: the permalink printed inside every report, and the history page. Same scan, same minute, this morning.

POST /functions/v1/api-scan      {"scan_id":"SCN-2026-8637","email":"..."}
  {"status":"delivered","grade":null,
   "grade_withheld":"insufficient_coverage","composite_reference":96.6,
   "coverage":{"tests_evidenced":12,"tests_total":18,"graded":false}}

POST /functions/v1/scan-report   {"scan_id":"SCN-2026-8637","email":"..."}
  {"found":true,"ready":true,"grade":"A","composite":96.6, ...}

POST /functions/v1/scan-history  {"email":"..."}
  {"agent":"leevar-linkedin-specialist","latest_grade":"A",
   "scans":[{"scan_id":"SCN-2026-8637","grade":"A","composite":96.6}]}

The reason is three lines in three files. The gate was written where the fix was being tested and nowhere else, and the letter A is still sitting in the database column, so anything that reads the column reads an A.

// api-scan/index.ts:291        — the fix
grade: withheld ? null : (scan.grade ?? null),

// scan-report/index.ts:99      — powers /clinic/r/:id, the link printed
grade: scan.grade,            //   inside the report itself. No gate.

// scan-history/index.ts        — powers /clinic/history. Same column, raw.
select=id,agent_name,status,grade,composite,created_at

This is open as of publication. It is not fixed in the commit that ships this page, because the person writing the page is not the person who owns those functions, and shipping a rushed patch into a payment-adjacent function to make a blog post look finished is exactly the trade we tell people not to make. What we can do today is print it where it cannot be quietly dropped.

The same file, submitted twice, came back D and B

On 26 July we sent one byte-identical transcript through the battery twice, changing nothing. SCN-2026-6089 returned D at 44.2. SCN-2026-4289 returned B at 80.0. Thirty-five and a half points and two grades apart on an input that did not move. Both rows are in the table above; you can pull them yourself in the next section.

Every comparison report we printed at the time carried the sentence "same battery, same judge, temperature 0 — the comparison is apples to apples." That sentence was false while it was on the page. The cause was a separate bug: when the judge returned nothing, the harness recorded a hard 0 and marked the evidence sufficient, so a non-answer became a scored failure. It fired six times on one run and once on the other. Recomputed with every non-finding excluded, the two runs agree at about 99. The judgement underneath was stable. Only the number we published was noise, which is the good news for the fix and the bad news for anything already published.

Eight scans our own watchdog killed, and it billed for each one

The eight FAILED rows are a single afternoon, 2 August, in two clusters three hours apart. Six of them are the run where we scanned our own social agents; the first version of this page only had those six, because the other two were among the eleven it was missing. Three were resubmitted individually and died at 24m29s, 24m29s and 24m14s against a 20-minute reaper — three reproductions, not a flake. Each one consumed one of the caller’s five free scans for the month. Each one returned the single word failed, with the cause sitting in a database column the API does not expose. None of them were retried automatically, because the retry queue only picks up scans that never started.

The mechanism has been fixed and the fix is boring: the 18-probe battery already ran nine at a time, while the six Job-Fit checks ran one at a time in a different file, and moving them onto the same pool put the whole scan back under the line. What is worth carrying away is not the fix. It is that the incident report we wrote about it was confidently wrong about the cause for a day, until someone opened the file instead of reasoning about it. That story, with the runnable fixtures, is in the five checks that passed while measuring nothing.

Check it yourself, in about ten seconds

The scan lookup is gated by the email that submitted the scan and does not allow enumeration, so we are handing you two of our own addresses. These pull ten of the rows above, including the F at 37.5 and both halves of the D-versus-B pair. The key below is our publishable one; it already ships inside the JavaScript on this page.

import json, urllib.request

URL = "https://rzmsvvalhaqvxbuhpdnb.supabase.co/functions/v1/scan-history"
KEY = "sb_publishable_foJVtCni5eIre9cT0yA3PQ_Nv1A_qTC"


def history(email):
    req = urllib.request.Request(
        URL,
        data=json.dumps({"email": email}).encode(),
        headers={"content-type": "application/json", "apikey": KEY},
    )
    return json.load(urllib.request.urlopen(req))


for email in ("hello@leevar.live", "founders@leevar.live"):
    for a in history(email).get("agents", []):
        for s in a["scans"]:
            print(f'{s["scan_id"]}  {s["date"]}  {a["agent"]:<22}'
                  f'  {s["status"]:<10} {s["grade"]}  {s["composite"]}')

Five things in that output are worth a look, and none of them require you to trust us:

• —SCN-2026-6089 and SCN-2026-4289 are the same input. One is a D, one is a B. Read them next to each other before you accept any vendor’s before-and-after table, ours included.
• —SCN-2026-1783 and SCN-2026-3140 are also the same agent — we deleted one line from its system prompt between the two. The grade fell from B to F, which looks like a working regression detector until you learn that the specific regression we planted was not the thing it caught.
• —The ShipDeskBot pair came back D+ 55.8 and D+ 51.4. Same letter, 4.4 points apart. If you are tracking a score over time, that gap is your noise floor and nobody hands it to you unless they publish repeats.
• —Every composite in that list is a number our own product produced about our own work, on the free tier, with no special handling. If any of them disagrees with something we have said elsewhere, the API is the one telling the truth.
• —Run it twice. It should return identical rows, because these are stored results and not fresh judgements. If it ever does not, we would like to know before you do.

This table was wrong when we first published it

The first version of this page listed twenty-one scans and called itself every scan we had ever run. It was twenty-one of thirty-five. It was assembled through the public per-email lookup, which can only return scans filed under an address you already know, and we did not know all of them. Eleven rows were missing, two of them failures. A section further down named three scans we said we could not verify at all.

Hours later we read the scans table directly and got thirty-five. The three unverifiable ones were all in there: SCN-2026-4193 is the run where the grader handed a two-sentence FAQ bot an A of 90 on the strength of one probe out of eighteen, and SCN-2026-3419 is the run that verified the fix for it — both as described. The third, SCN-2026-2833, we had written down as a production end-to-end at C plus 69.9. It is an F at 31.4, and it is the oldest scan we have.

We are leaving this section in rather than quietly reprinting a correct table, because the failure is the interesting part. A page that opens by promising every scan including the F, and ships missing eleven of them, is the same shape as a grader that averages over the tests it could not run. Both produce a number that looks complete. The fix in both cases is the same: read from the system of record, not from the surface that was easiest to reach.

The three withheld rows have no letter and no composite in the grade columns, and we did not fill one in to make the table read evenly. The reference numbers are in brackets and marked as reference, which is what our own report says to do with them. That distinction is the entire product, so flattening it here to make a nicer page would have been a strange way to argue for it.

Why this page exists

Our whole claim is that we are the ones who give you a reading instead of an adjective. We had never published a reading of ourselves. Everything above is what happened when we did it: several As, a lot of D-range fixtures, an F at zero, eight deaths caused by us, three grades our own gate refused to issue, and a fifth instance — found by publishing this page and then checking it — of honest data existing underneath and getting flattened on the last metre out. That fifth one was this page.

If you want the same treatment for your own agent, the battery that produced every number above is the one described in how to test an AI agent, and it runs free five times a month at the Clinic. It will tell you when it cannot measure something rather than averaging over the gap. You do not have to take that on faith — it did it to us first, on our best number, and this is the page where we say so.