The reliability arc

Jump to: root cause · the four conditions · failure census · buy / harden / build · your open calls

1. Root cause — one shape, five masks

   I autopsied every failure we could reach. None were random. Each was a written promise that no mechanism enforced, and they failed in five recognisable ways.

   • The record was an afterthought — a run wrote its ledger row only at the end, so a worker that died mid-flight left nothing. Seven runs vanished this way.
   • Optional work blocked mandatory work — evidence-collection ran before the commit that saves the build, with no time limit. Three real builds finished, then were lost.
   • A green could be fake — measurement jobs reported failure while succeeding; the first fix was itself fakeable four ways; one path took custody of bytes it never checked.
   • Health checks lied — the cleanup swept living runs as dead; a courier called a slow run lost.
   • The cloud lane delivered work but not evidence — every cloud report was a 1 KB stub; the engine's real account never came home.

   Two genuine engine faults live under all of this and are named, not hidden: one sandbox model freezes mid-run, one builder dies silently on long runs. Both seats are pulled; neither can eat work anymore. Everything else was us.

   ↘ go deeper — the exact multiplier that lost three builds

   The salvage step forked the secret-scrubber once per file under scratch, and the scrubber re-walked every credential root on each fork — so cost = (files under TMP) × (bytes under engine-home). Measured: 140,931 files × a 489 MB per-run home ≈ 20s/file, a ~32-day projection, on a build whose engine had finished 43 minutes earlier. The same path did 228,899 files in 16 minutes six days before; scratch never changed — ~/.codex grew to 2.1 GB, 1.2 GB of it session history the copy inherited. A file-count cap would have missed both cases; the cure bounds wall-clock and stops copying history. Branch salvage-never-starves-custody, twin g-salvage-never-starves.sh (RED 6-of-8 against the pre-fix runner).

2. First: make the checks trustworthy

   The arc's non-negotiable order is checks first — a fix you cannot verify is a claim, not a fix, proven repeatedly this session when "done" work turned out to be a no-op. So the first landed layer makes a green mean something: 32 hollow test arms repaired, a parity gate that fails when a category drifts, and a discipline that every twin must be proven able to fail.

   ↘ go deeper — the vacuity court

   Every twin ends by running itself against sabotaged copies of its subject and requiring RED, plus a positive control proving the harness can still pass. Four tests this session passed against a broken or absent subject before this discipline caught them — a zero-byte library, a deleted verification leg, a run dir read as "history absent" because it had been reaped. Reference: g-ledger-db.sh (3-court gauntlet + control).

3. Then: the record that cannot be skipped

   The cure for the vanishing runs, and condition 3 of done. The run record moved to a real database, and all three doors now write down what they're about to do before they spend anything — so a run that dies at any instant leaves a findable trace. Proven by killing a live dispatcher outright and finding its open intent in the record.

   The four conditions of done, and where each stands:

   1 · No test passes about nothing
   MET — enforced by the merged checking layer.

   2 · Every suite runs or is retired
   MET — real exit codes, honestly classified.

   3 · No run finishes without a record
   MET — SQLite, intent-before-spend, all three doors.

   4 · No door accepts work it can't do; no failure destroys work
   IN FLIGHT — the two watchers next; the work-destroying salvage defect already cured, in review.

   5 · Survives a stranger
   NOT MET — the cold-install walk + the 50-dispatch acceptance run are the finale.

4. Then: fix what remains

   With checks and record trustworthy, the rest of the defects land as bounded, verified fixes — several found by an independent peer session running the machinery adversarially, which is the engine of this arc now: people who did not write the code try to break it before it merges.

   • Salvage never starves custody — commit first, collect evidence second, bounded in wall-clock. (In review.)
   • The measurement path anchors to a digest — it no longer takes custody of bytes without checking them against their own fingerprint, and the test that guards it can finally fail. (In review.)
   • The cloud report rides the diff — the engine's own account comes home, banked and stripped from custody, or the run loudly confesses none came back. (In review.)
   • The two watchers — a door heartbeat and a run watchdog that catch a frozen run, a door that can't pay, work nobody read. (Next.)

5. Then: prove it in the world

   The record being trustworthy turns "did we find every failure" into a number anyone can recompute. The failure census, run tonight:

   728 runs · 435 clean · 261 failures
   seven-day window, recomputable from the ledger.

   257 of 261 attributed
   to a named, understood cause.

   4 remain
   3 whose evidence predates the fix that would have kept it (permanently unknowable), 1 provider-internal crash.

   Unexplained AND explainable: zero
   the watchers keep that number visible going forward.

   The finale is the empirical answer: a cold install on a machine with no context, and a 50-dispatch acceptance run where every run ends recorded, nothing finishes unnoticed, and any non-author can recompute every verdict from the record alone.

6. The buy-over-build line — your steer

   The principle, stated plainly: the load-bearing, hard-to-get-right infrastructure should be built by a frontier provider; we keep only what is genuinely ours, and we harden that with adversaries. Where the surface is today:

   bought The run record → SQLite. A database that survived 500 kill-trials doing what our hand-appended text file provably could not. Done.

   bought Metering & tracing → Cloudflare AI Gateway. One vendor gateway replaces our hand-rolled meter and delivers the observability we were about to build. Ratified, queued.

   bought The cloud compute is already the provider's. We custody results; we don't run the runner.

   declined, for cause An off-the-shelf run-supervisor — surveyed, failed on real grounds (no exit-code API, bus-factor-one). We do check the shelf first.

   still ours The secret scrubber & the three door scripts. Where every defect tonight lived. A reviewer formally called one piece "hand-rolled parsing of a structured format — replace with a maintained parser."

   The concrete move you asked for: after the open PRs land, one buy-versus-keep proposal covering the two worst hand-built pieces — the scrubber (established secret-scanning tools exist) and the brief-format parsing (maintained parsers exist) — with real candidates, not open questions. The doors themselves get cheaper to trust as the checking layer grows teeth, so the recommendation is harden-in-place rather than rip-out mid-arc — but that call is yours, framed with receipts.

Your calls