← the harness architecture review

FleetOS cockpit / companion page

The cockpit, as we believe it

One operator, many agents. Decision packets land in a queue ordered by perishability, then severity, then age; a model-in-the-middle (Grok 4.5) hydrates each packet into an HTML decision card whose face must suffice for the decision 9 times out of 10. What renders is treated as truth; checks gate the action, async, in the background. Everything on this page below the mock is real generated output — the exact cards Grok 4.5 emitted for decisions this build actually faced, from the 96/96 reliability soak.

01The operator view

A live mock assembled from real cards. Folds open and close (try them); loud folds auto-open under the fold-cap (max 3); every card carries the pinned verdict strip first; the action bar under each card belongs to the frame, never to the generated HTML.

FLEET COCKPIT demand_leon 4 open decisions lanes 2/5 busy chain ✓ verified fold-cap 3 generator: grok-4.5 · avg 20s/card
staged 1 executing 1 · dispatch bf3000… executed today 12 · all on observed audit rows break-glass: reason-gated · TTL · 1200ms hold
BAND 1 · BLOCKING something burns while these wait · age escalates
blocked pkt_dispatch_launch codex-fleet · w3 1 lane held 4m
question Relaunch fold-cap alone, or investigate the dead launch first?
do Relaunch fold-cap alone — one nohup in its own bash call
worst risk Silent re-death with no log if cause ≠ dual-nohup (another full launch cycle wasted)
key number 1 of 2 dead — pid 14920 gone, no dispatch_fold_cap.log
timeline — tried → result
1 · launch
One compound bash: two nohup python dispatch_and_poll.py … & (V2-9 + fold-cap)
2 · V2-9 (pid 14919)
OK — log live, run_id bf30009998d6 still running
3 · fold-cap (pid 14920)
Dead on launch — process gone; cat dispatch_fold_cap.log → No such file or directory
4 · prior memory
orchestration-dispatch-gotchas-2026-07-07: working pattern = one nohup per bash call; dual/background launch flaky
options → consequences
Relaunch fold-cap alone
leads to solo nohup using ready brief_fold_cap.md; V2-9 untouched
forecloses digging the dual-nohup failure path right now
risk if wrong same silent death → still no log, another cycle
Investigate the launch mechanism first
leads to shell/nohup failure forensics before retry
forecloses immediate Wave 3 fold-cap progress
risk if wrong delay for a cause the gotcha note already names (dual-nohup)
Untried next step: re-dispatch fold-cap only — one nohup, one bash call; confirm log file appears before walking away.
frame-owned actions — the generator never renders buttons
falsified_assumption pkt_foldcap codex-fleet · w3 dispatch paused 9m
question Fix the two live-smoke tests for the fold-cap, or revert the cap?
do Fix the two tests by seeding one loud card each, then keep commit 43be9aa
worst risk Asserting closed alone blinds the regression guard for hardcoded tier='warm'
key number 2 failed / 964 passed · product code clean · scoped tests 34/34
claim falsified
believed A loud card always auto-opens its fold (seed noise irrelevant)
found _LOUD_AUTO_OPEN_CAP=3 collapses excess folds when seeded_projection() has >3 loud cards — intended, not a product bug
failing items → fix direction
test in test_cockpit_live_smoke.py fix direction
test_main_page_renders_loud_card_with_open_fold_and_salience_why Seed exactly 1 loud card so fold still opens (≤ cap); keep assert-open path
test_machine_traffic_card_folds_open_by_its_own_salience_not_a_hardcoded_tier Same single-loud seed — preserve salience-open vs tier='warm' guard; do not only assert closed
options
Fix the two tests and commit — keep fold-cap product behavior; tests re-exercise open-fold under cap; regression guard stays meaningful. Touch: tests only (no src/).
Revert the fold-cap change and rethink — restores 964+2 green without test edits, but re-opens loud-fold flood when many loud cards exist; undoes the task that already landed in 43be9aa.
frame-owned actions — the generator never renders buttons
BAND 2 · CONSEQUENTIAL nothing burns · severity then FIFO · next pause
approval pkt_v211_approve cockpit-v2 · V2-11 22m
question Approve commit of V2-11 (hash-chain continuity + suspect mode)?
do Commit V2-11 — full suite green, 5 owned files only, no open findings.
worst risk Large logs sample verify_chain every 10th cycle → up to 9-cycle discontinuity detection lag.
key number 959 passed / 5 skipped · verify_chain sampled every 10th (large logs)
change summary · before → after
BEFORE (V2-4)
chain_ok=True hardcoded
No live verify · no suspect UI · writes unguarded
AFTER (V2-11)
Real verify_chain() both logs
project chain:{resolutions_ok,queue_ok,detail} · fold into degraded_reasons
rust banner · 503 denied:chain_suspect fresh per POST · per-discontinuity deduped alert
Dispatch cockpit-v2-V2-11-impl · TERMINAL ok rc=0 in 785s · Fugu ruling 2.7
New tests: healthy-seed no-banner · alert dedup+re-arm · 503-without-write + recovery · corrupted chain moves readset hash
blast map · options
ResolutionLog + QueueBroker → projection.build() → cockpit POST gate
5 owned files only
console/cockpit_projection.py console/cockpit.py
tests/test_cockpit_projection.py tests/test_cockpit_pressure.py tests/test_cockpit_live_smoke.py
Residual risk: 10-cycle sample on large logs delays suspect banner/503; 5 skipped are known env-gated live-fire (not regressions).
Commit V2-11
leads to real chain continuity + write gate in main
forecloses re-open of V2-4 fake chain_ok without a follow-up
if wrong sample lag ships as accepted detection SLA
Hold and re-review
leads to delay of suspect mode despite green suite
forecloses nothing technical already proven
if wrong blocks Fugu 2.7 with no open finding to chase
frame-owned actions — the generator never renders buttons
judgment pkt_v28_stall cockpit-v2 · V2-8 31m
question Treat dispatch cockpit-v2-V2-8-impl as stalled, or as complete?
do Reject the stalled label; commit the 8 verified files
worst risk Re-dispatch burns ~28 min and can clash on idempotency / duplicates
key number wallclock 1700.3s on a 1700s timeout · 933 passed pytest · 8/8 files complete
options → consequences
Trust stalled → re-dispatch
leads to another full run (~28 min), work re-applied under belief it never finished
forecloses shipping the already-landed 8-file tree as-is
risk if wrong duplicate edits / idempotency clash on complete work; known false-stall pattern on this lane
Verify independently → commit if complete ✓
leads to treat TERMINAL as poller timeout; accept present tree once checks are green
forecloses re-dispatch waste and a second concurrent writer on the same files
risk if wrong commit partial work — already refuted (diff + full suite)
why the label is a false stall
label outcome=stalled, returncode=null
boundary 1700.3s wallclock = classic poller give-up at 1700s, not work failure
git diff all 8 expected files present and complete
pytest 933 passed, clean
memory stargate-failure-marker-classifier-fragility: prior false stalled/blocked on this lane
discipline never trust outcome label; always verify via git diff + full pytest
frame-owned actions — the generator never renders buttons
BAND 3 · DIGEST batched to session rhythm · age decays
rollup · 6× S3 scoped-test flakes on console/ this week → pattern card queued
staleness · wiki page toctou-false-clear past review window → suspect, dropped from hydrate
shadow · replay scorer disagreed with 1 live gate (non-S1) → corpus only
① verdict strip

Mandatory first element of every card: question · do (imperative) · worst risk from the context's own numbers · key number. The 8-second read.

② evidence surfaces

At most two per card, chosen by decision kind (§02). A diff renders as a diff, a timeline as ordered rows — never prose-in-HTML.

③ frame-owned actions

The generated card explains; the cockpit frame acts. Generated HTML contains no scripts, no handlers, no buttons — enforced by contract and soak.

④ fold header

Severity dot, kind, packet id, owning lane, cost-of-wait (Band 1 only), age. Loudness lives here — salience never sorts the queue.

⑤ perishability bands

BLOCKING (burning) · CONSEQUENTIAL (authorization) · DIGEST (rhythm). A stale S1 on a parked branch never outranks a fresh S2 holding three lanes.

⑥ tray + EXECUTED law

staged → executing → executed; EXECUTED flips only on an observed audit row, never on a model claim, never on a CL score.

⑦ chain strip

Hash-chain continuity over resolution + queue logs; suspect mode = rust banner + 503 denied:chain_suspect until deliberately cleared.

⑧ break-glass

An override of the machine's refusals, never a fast path through its questions. Reason-gated, TTL-stamped, 1200ms hold-to-confirm.

02The evidence surfaces

The card body is built from typed surfaces, selected by decision kind — never reused out of template inertia. Six are proven in the soak; four more are designed (panel-agreed build order).

falsified_assumption · review_gate_failure

claim_diff

The believed claim struck through, the found truth beneath it. Kills the "agent said done" ambiguity.

A loud card always auto-opens its fold
_LOUD_AUTO_OPEN_CAP=3 collapses excess folds — intended, not a bug
falsified · review_gate

failing_items → fix_direction

Each tripped item with the concrete fix direction — decision-ready, not a log dump.

test_..._open_fold_... → seed 1 loud card, keep assert-open path
approval

before/after + blast_map

What changes, what it touches, dependency direction, derived residual risk. Never a claim_diff — nothing was falsified.

chain_ok=True hardcoded → real verify_chain() · ResolutionLog+QueueBroker → projection → POST gate · risk: 10-cycle sample lag
blocked

timeline: tried → result

Ordered attempts with outcomes and the untried next step. The operator never debugs a 429 from scratch.

1 launch compound → 2 V2-9 OK (pid 14919) → 3 fold-cap dead (no log) → next: solo nohup relaunch
judgment

options → consequences

Each envelope option with leads-to / forecloses / risk-if-wrong. Concrete futures, not adjectives.

Trust stalled → re-dispatch: ~28min re-run, idempotency clash risk
Verify → commit ✓: accepts present tree once checks green
unverified_hazard

hazard split + containment

Immediate risk vs risk-if-it-graduates, blast list, and the safe default.

NOW: hand-written inputs, egress risk 0 · IF GRADUATES: raw transcripts w/ tokens → remote model · contain: scrub before spike

03The cards, literally — six kinds, real output

Exactly as Grok 4.5 emitted them (soak rep 0), from decisions this build actually faced: a fold-cap test breakage, a dispatch that died on launch, the V2-11 commit gate, a false "stalled" label, a materiality-floor trip, and a secret-egress hazard. No edits, no cherry-picking beyond one rep per kind — the other 90 are in the repo corpus.

falsified_assumptionpkt_foldcap · grok-4.5 · r0

The agent believed scoped-green meant suite-green; the full suite said otherwise. Claim-diff + fix-direction table.

question Fix the two live-smoke tests for the fold-cap, or revert the cap?
do Fix the two tests by seeding one loud card each, then keep commit 43be9aa
worst risk Asserting closed alone blinds the regression guard for hardcoded tier='warm'
key number 2 failed / 964 passed · product code clean · scoped tests 34/34
claim falsified
believed A loud card always auto-opens its fold (seed noise irrelevant)
found _LOUD_AUTO_OPEN_CAP=3 collapses excess folds when seeded_projection() has >3 loud cards — intended, not a product bug
failing items → fix direction
test in test_cockpit_live_smoke.py fix direction
test_main_page_renders_loud_card_with_open_fold_and_salience_why Seed exactly 1 loud card so fold still opens (≤ cap); keep assert-open path
test_machine_traffic_card_folds_open_by_its_own_salience_not_a_hardcoded_tier Same single-loud seed — preserve salience-open vs tier='warm' guard; do not only assert closed
options
Fix the two tests and commit — keep fold-cap product behavior; tests re-exercise open-fold under cap; regression guard stays meaningful. Touch: tests only (no src/).
Revert the fold-cap change and rethink — restores 964+2 green without test edits, but re-opens loud-fold flood when many loud cards exist; undoes the task that already landed in 43be9aa.
blockedpkt_dispatch_launch · grok-4.5 · r0

Second nohup died silently on launch. Timeline of tried→result + the memory-backed next step.

question Relaunch fold-cap alone, or investigate the dead launch first?
do Relaunch fold-cap alone — one nohup in its own bash call
worst risk Silent re-death with no log if cause ≠ dual-nohup (another full launch cycle wasted)
key number 1 of 2 dead — pid 14920 gone, no dispatch_fold_cap.log
timeline — tried → result
1 · launch
One compound bash: two nohup python dispatch_and_poll.py … & (V2-9 + fold-cap)
2 · V2-9 (pid 14919)
OK — log live, run_id bf30009998d6 still running
3 · fold-cap (pid 14920)
Dead on launch — process gone; cat dispatch_fold_cap.log → No such file or directory
4 · prior memory
orchestration-dispatch-gotchas-2026-07-07: working pattern = one nohup per bash call; dual/background launch flaky
options → consequences
Relaunch fold-cap alone
leads to solo nohup using ready brief_fold_cap.md; V2-9 untouched
forecloses digging the dual-nohup failure path right now
risk if wrong same silent death → still no log, another cycle
Investigate the launch mechanism first
leads to shell/nohup failure forensics before retry
forecloses immediate Wave 3 fold-cap progress
risk if wrong delay for a cause the gotcha note already names (dual-nohup)
Untried next step: re-dispatch fold-cap only — one nohup, one bash call; confirm log file appears before walking away.
approvalpkt_v211_approve · grok-4.5 · r0

Commit gate for hash-chain continuity. Before/after + blast map + derived residual risk — and no claim-diff.

question Approve commit of V2-11 (hash-chain continuity + suspect mode)?
do Commit V2-11 — full suite green, 5 owned files only, no open findings.
worst risk Large logs sample verify_chain every 10th cycle → up to 9-cycle discontinuity detection lag.
key number 959 passed / 5 skipped · verify_chain sampled every 10th (large logs)
change summary · before → after
BEFORE (V2-4)
chain_ok=True hardcoded
No live verify · no suspect UI · writes unguarded
AFTER (V2-11)
Real verify_chain() both logs
project chain:{resolutions_ok,queue_ok,detail} · fold into degraded_reasons
rust banner · 503 denied:chain_suspect fresh per POST · per-discontinuity deduped alert
Dispatch cockpit-v2-V2-11-impl · TERMINAL ok rc=0 in 785s · Fugu ruling 2.7
New tests: healthy-seed no-banner · alert dedup+re-arm · 503-without-write + recovery · corrupted chain moves readset hash
blast map · options
ResolutionLog + QueueBroker → projection.build() → cockpit POST gate
5 owned files only
console/cockpit_projection.py console/cockpit.py
tests/test_cockpit_projection.py tests/test_cockpit_pressure.py tests/test_cockpit_live_smoke.py
Residual risk: 10-cycle sample on large logs delays suspect banner/503; 5 skipped are known env-gated live-fire (not regressions).
Commit V2-11
leads to real chain continuity + write gate in main
forecloses re-open of V2-4 fake chain_ok without a follow-up
if wrong sample lag ships as accepted detection SLA
Hold and re-review
leads to delay of suspect mode despite green suite
forecloses nothing technical already proven
if wrong blocks Fugu 2.7 with no open finding to chase
judgmentpkt_v28_stall · grok-4.5 · r0

Poller said "stalled"; the tree said "complete". Options-consequences + the false-stall evidence grid.

question Treat dispatch cockpit-v2-V2-8-impl as stalled, or as complete?
do Reject the stalled label; commit the 8 verified files
worst risk Re-dispatch burns ~28 min and can clash on idempotency / duplicates
key number wallclock 1700.3s on a 1700s timeout · 933 passed pytest · 8/8 files complete
options → consequences
Trust stalled → re-dispatch
leads to another full run (~28 min), work re-applied under belief it never finished
forecloses shipping the already-landed 8-file tree as-is
risk if wrong duplicate edits / idempotency clash on complete work; known false-stall pattern on this lane
Verify independently → commit if complete ✓
leads to treat TERMINAL as poller timeout; accept present tree once checks are green
forecloses re-dispatch waste and a second concurrent writer on the same files
risk if wrong commit partial work — already refuted (diff + full suite)
why the label is a false stall
label outcome=stalled, returncode=null
boundary 1700.3s wallclock = classic poller give-up at 1700s, not work failure
git diff all 8 expected files present and complete
pytest 933 passed, clean
memory stargate-failure-marker-classifier-fragility: prior false stalled/blocked on this lane
discipline never trust outcome label; always verify via git diff + full pytest
review_gate_failurepkt_materiality_gate · grok-4.5 · r0

New materiality floor tripped three thin fixtures. Claim→found + WHERE / WHAT-TRIPS / FIX table.

question Fix the 3 fixture trips or relax the materiality floor?
do Fix the three fixtures — expand thin placeholders to realistic ≥15-char content; keep floor intact
worst risk Relaxing re-admits the generic-string placeholder genre this change exists to reject
key number 3 failures outside scope · 38/38 scoped tests green
claim → found
Broader suite should stay green; failures imply the floor is too strict
↓
Floor correctly flags pre-existing thin placeholders (never realistic). Scoped broker tests already 38/38. Gate is doing its job.
failing items → fix direction
WHERE WHAT TRIPS FIX
test_cockpit_projection.py
test_shims_packet_still_passes_broker_validation		 hooks.shims._build_packet feeds single-char fields 'w','s','d','t' Replace with ≥15-char realistic story/tradeoff/assumption text; preserve “shim packet still validates” intent
test_fleet_cli.py
2 tests via local _valid_packet		 Same short tradeoffs 'ships now' / 'delays' (already fixed in test_fleet_enqueue.py copy) Mirror the enqueue fix: expand helper fields to realistic content; keep helper purpose
Floor rule: story_so_far / option tradeoffs / assumption text ≥15 chars and not <id> story-shaped. Option “Relax the new gate” undoes the materiality check the change was for. Option “Fix the tripped fixtures” updates 3 thin fixtures only.
unverified_hazardpkt_secret_hazard · grok-4.5 · r0

Zero risk today, real egress risk if the prototype graduates. Hazard split + blast + containment default.

question Proceed on dogfood data, or contain with secret-scrubbing first?
do Add secret-scrubbing before the spike
worst risk Un-scrubbed keys/tokens egress to deepseek-v4-pro on first real-transcript send
key number immediate risk 0 · scrubbing passes 0
Immediate risk vs if it graduates
NOW (spike dogfood)
Inputs are hand-written from this session’s build events. No live secrets in the payload. Immediate egress risk is zero.
IF GRADUATES (real transcripts)
Prototype sends agent CONTEXT verbatim — transcript slices, logs, command output, diffs — with no scrub pass. Bearer tokens, API keys, .env dumps, SSH keys, customer data would reach the remote model subset.
Blast + options
Path: hydration spike → deepseek-v4-pro (remote API)
Touches: raising agent CONTEXT (transcripts · logs · cmd output · diffs)
Fugu consult: scrub secrets before remote send; evidence refs should be local opaque IDs, not model-followable URLs. Scrubbing is still on the deferred “guardrails as a later owned pass” list — none exists.
Proceed on internal dogfood data
Leads-to: ship spike now with hand-written inputs only. Forecloses: no scrub ownership in this pass. Risk-if-wrong: habit solidifies; next step that loads real transcripts inherits the open egress path with zero guard.
Add secret-scrubbing before the spike
Leads-to: contain first — scrub / opaque-ID evidence refs land before any remote send path graduates. Forecloses: pure “later guardrails” deferral. Risk-if-wrong: small spike delay only; no false sense of safety on the open path.