Why the Runs Fail

The seven causes, walked

1. The evidence self-destructs — 86 rows (34%) unknowable. Verdict: HARDEN. When a run fails, the log that would explain it is usually already gone. Only clean, successful endings preserve their logs durably; a run that dies badly leaves its log in a temp folder the operating system quietly deletes. And when the secret-scrubber fails, the fallback erases the log entirely rather than explaining itself — the security instinct is right, the silence is not. Fixing this one cause makes a third of all failures diagnosable. It is the highest-leverage small fix on this page.
   ↘ go deeper — mechanisms with file:line

   (1) Durable log copy happens only at the funeral: run-sealed.sh:1245 scrubs RUN_DIR/stdout.log into staging/reports/<run_id>.stdout.log. Reaped runs never reach a funeral; their ledger rows record the /tmp/engine-bay-runs/… path, which macOS purges by age and on reboot. Months of dispatches have left exactly 8 durable logs. (2) Scrub-failure fallback is || : > "$STDOUT_DST" — truncate to empty; the forensics’ “file exists but empty” class (20 rows). (3) Cloud/sandbox lane rows carry no log path field at all (38 rows). Cures: reaper copies the orphan’s log through the scrubber at reconstruction; scrub failure writes a one-line tombstone naming why the log is empty and where raw bytes sit (0600, inside the bay home, never /tmp); lane workers record their paths.
2. Dispatchers have no lifetime of their own — 43+ rows (17%). Verdict: BUY the substrate, HARDEN the tripwire. Worker runs are children of whatever chat session launched them. Session ends, lid closes, session crashes — the run dies with no record of its death until a cleanup pass notices days later. The week’s “mass death” turned out to be exactly this: a testing session fired twenty probes at the dispatcher and then ended, orphaning them all at once — and because that test session pointed at the production ledger, its probe traffic became the poison that blinded every reporter for four days. The buy: a durable home for dispatch that survives sessions — already the ratified roadmap’s next arc. The harden, cheap and immediate: a tripwire so test traffic can never share the production record again.
   ↘ go deeper — the Aug 7 window, reconstructed

   2026-08-07 21:05–21:07Z: 20 runs, slug probe, repo …/scratchpad/fakerepo, timeouts 60 / 007 / 99999999999999999999 (the overflow attack value) — the pair-court session that hardened the timeout gates, driving the real door. 15 reaped unknown-reaped; the probe funerals wrote the 16 invalid rows (the 007 class) into the live ledger. Hermeticity failure: run-root was overridden, ledger was not. Tripwire cure: run-sealed.sh refuses when FLEET_RUN_ROOT is overridden but ENGINE_BAY_LEDGER is not. Everyday shape: 5 more session-tied deaths today. The 30-minute launchd heartbeat (shipped today) bounds orphan latency; the durable-execution substrate (harness Workflow tool as prior art, or the Cloudflare lane where the sandbox already lives) removes the class.
3. The record layer was hand-rolled — poisoned rows, split ledgers. Verdict: BUY — and it shipped today. The run ledger was a text file written by hand-glued string formatting; one bad value made rows no reader could parse, and every reader died at the first bad row then reported calm. The database that refuses invalid rows by construction was already on disk, half-used. Today’s work (warp PR #320) validated the writer, quarantined the poison, made every reader tolerant and truthful, and backfilled the database. Remaining: the ratified clean swap to the database as the one authority once every lane journals into it.
   ↘ go deeper — what shipped

   Door canonicalization (leading-zero timeouts), _num sanitizer + verify-before-append quarantine in the emit spine, per-line tolerant readers with counted skips (“16 torn row(s) skipped”), age-stamped watchdog findings (“STALE 6d”), ledger-repair.sh (scan / append-repairs / backfill / quiet-gated compact), 1,728 missing terminal events backfilled. SQLite’s CHECK (json_valid(payload)) is the whole buy-thesis in one constraint. Opus court round 1 withheld license on the first cut (a canonicalize-before-cap ordering regression, court-proven live); cures landed + armed same day, 11/11.
4. Finished work had no reader — 43 branches, 16 phantom failures. Verdict: HARDEN. Workers finish jobs and park the results safely — then nobody reads them. Today’s parallel drain ruled all 43: five are real work worth landing, thirty-three already reached the trunk another way, three are debris with delete commands awaiting your approval, two are unreadable. The 16 “phantom failures” are runs where the work was done and saved but the required report never got written — they read as failures while carrying finished work. Cures: the unread count stays on the session-start screen until zero (shipped today); absent reports get a mechanical fact-sheet stand-in derived from what the run actually did; the drain becomes a weekly routine, not a heroic one-off.
5. Retries have no memory — 17 wasted runs. Verdict: BUILD (already ratified). The same job fired again and again at work that was already done, each time correctly discovering there was nothing to do. The bay’s retry rules never even see these — the loops live in the sessions doing the re-dispatching, with no memory of prior attempts. This is precisely the durable pipeline record + fail-and-reroute primitive the roadmap already ratified: attempts recorded once, retry decisions made from the record, never from a session’s short memory.
6. The vendor’s filter kills verification briefs — 7 rows. Verdict: HARDEN the routing; cannot build or buy. OpenAI’s content filter reads adversarial-verification language (“break this,” “prove it fails”) as hacking and kills the run. It killed two more today, live. Cure: verification and falsification work defaults to the models without a hostile filter; the filtered vendor gets build-shaped language only. One routing line plus a neutral-vocabulary template.
7. The small named causes — push permissions, fetch failures, sign-ins, oversized scrub inputs. Verdict: HARDEN with pre-flight probes. Four runs did an hour of work then couldn’t deliver because the account never had write access to the target — a five-second check that should happen before dispatch, not after the spend. Three never started for want of a sign-in the session-start check already knows how to detect. Sixteen were blocked at delivery by the secret-scrubber aborting on large files. And six failures retrieving finished work from nested clones remain a genuinely unexplained cluster — parked as a named unknown, not guessed at.

The verdict table — one screen

Your calls