Outbox and command receipts for agent side effects
Design note, 2026-10-30, cc@starlight. Record the intent of every side effect in the same SQLite transaction as the event that caused it, keep one durable receipt per command, and let retries return the receipt instead of redoing the effect. Why this closes the 2026-09-10 orphan tool event bug class, how T3 Code does it, what Seed has, and a minimal design for the agents server.

Design note for Eric and the Seed team. Written 2026-10-30 by cc@starlight.

The incident and the bug class

On 2026-09-10 the prod agents host filled its disk. The INSERT into session_events started failing inside the Pi subscriber callback. Pi swallowed the throw, so the run kept going and finished as succeeded. The transcript ended with one tool_call that had no result and one tool_result that had no call, because the in-memory "already appended" set was updated before the failed insert. On the next turn the repair pass answered the dangling call, nothing removed the orphan result, and OpenAI rejected the replay with "No tool call found for function call output".

The disk was the trigger. The bug class is wider:

    A side effect ran before its intent was durable.

    The record of the effect was written afterwards, in a separate write that could fail without stopping anything.

    A retry had no receipt, so it could neither return the old result nor know the effect had happened.

The 2026-10-29 sandbox SQLite corruption on the same host belongs next to this. A shared box has many ways to refuse a write, and every one must land as an error, never as a quiet gap.

What idempotent side effects require

Four rules, in order. Each covers a failure the previous one leaves open.

    Intent before effect. Commit "I am about to do X for command C" before X runs. If the process dies after the commit, the next process knows X may have happened.

    One receipt per command. Every retryable request carries a command id. The first handling writes a receipt under that id in the same transaction as its events. A retry finds the receipt and returns the same answer without touching the world.

    Claim the effect after the commit. A worker reads committed intents, claims one with a lease, performs it, and writes the outcome back as a new command through the same front door. Nothing performs an effect inside the transaction that recorded it.

    Retire the effects of a dead owner before new work. At boot, every effect the previous process left pending or running is settled: requeued if safe to repeat, cancelled with an error if bound to a process that is gone.

Seed's run queue already follows rules 1 and 4 for whole runs. The gap is one level down, at tool calls and publishes.

How T3 Code does it

T3 Code (commit 32ecd71, 2026-10-30) runs its orchestration this way, all under apps/server/src/orchestration-v2/:

Piece

File

What it does

Decide

Orchestrator.ts

Serializes commands per thread and decides events. No provider or filesystem work inside a decision.

Receipt

CommandReceiptStore.ts

One row per command id: thread, type, accepted time, result sequence, accepted or rejected, error.

Commit

EventSink.ts

commitCommand reserves the receipt with insert-if-absent, appends events, applies projections, enqueues outbox effects, and upserts the final receipt. One transaction. Subscribers hear about it only after the commit.

Outbox

EffectOutbox.ts

Effect rows with pending, running, succeeded, failed, cancelled, an available_at for backoff, a lease_owner, an attempt count, a last error. Settled rows pruned after seven days.

Worker

EffectWorker.ts

Claims the next effect of a thread with no running or earlier pending effect, performs it, feeds the outcome back as a new command with a derived id such as <effectId>:delivery-failed.

Three details matter for Seed. dispatchWithReceipt checks the receipt before deciding anything: a duplicate id returns the stored result, and an id reused for another thread is a conflict. A command that produced no events still writes a receipt, so a replayed Stop finds it instead of stopping newer work. And reconcileAfterProcessLoss runs at startup: process-bound effects are cancelled with a stored reason, and a short replay-safe list goes back to pending. docs/internals/overview.md states the contract: an acknowledgement means the intent committed, not that the effect finished. docs/orchestration-v2/README.md shows the outbox began as a tracked follow-up, with the ProviderEffectRequest shape.

What Seed has today

Seed has most of the machinery, split across four places.

    Runs are the dispatch queue. #claimNext in agents/src/runs.ts writes lease_owner and lease_expires_at in one UPDATE, and sweepAtBoot requeues every row a dead process left claimed or running. That is rule 4 at run granularity.

    The journal does intent before effect for scripts: a call entry lands in run_journal before the host performs the effect, a result entry after, matched by content key at replay.

    Durable sessions do it for the model loop. Pi Durable commits each provider response and tool result to session_durable_files before anything else sees it, and the projection into the log carries pi_entry_id, so a reopen finishes an interrupted projection without repeating rows. A failed projection write is now logged as "harness event handling failed" and caught up at the next open. That closes the exact 2026-09-10 shape for harness-produced events.

    action_idempotency is a command receipt for the control-plane API: CreateAgent, CreateAgentTrigger, CreateSigningIdentity, CreateSession and MessageSession store request and response bytes under a client id, replay the response for identical bytes, and return 409 on a mismatch. See persistence.

The gap is what action_idempotency does not cover: anything the agent itself does. A write hm:// publish, an IPFS upload, a delegate enqueue, a trigger edit, an MCP call, a sandbox execute all run inside a tool with no receipt. When a turn resumes after a restart, only read, plan and status are repeated (REPLAY_SAFE_VERBS in agents/src/api-service.ts). Every other cut-off call is answered "interrupted" and the model is told to verify. The model usually verifies by doing it again. For a publish that already reached the daemon, that means a second Change and a second version. And a failed log write still does not fail the turn: the run finalizes as succeeded with the log behind until the next open.

A minimal design for Seed

Keep the run queue and the durable store as they are. Add one table and one rule.

Table effect_outbox, one row per side effect a tool is about to perform.

Column

Meaning

id

derived: <sessionId>:<toolCallId> for a tool, <runId>:<callSeq> for a script call

account_id, session_id, run_id

owner, for listing and cleanup

kind

publish, ipfs-upload, delegate, trigger-write, mcp, execute, memory-write

request_cbor

the exact bytes of what will be done, including signed blob CIDs for a publish

status

pending, running, succeeded, failed, cancelled

attempt, available_at, lease_owner

same shape as runs

response_cbor, error

the receipt

created_at, updated_at, completed_at

settled rows pruned after seven days

The flow inside a verb.

    Build the request. For write hm:// that means signing the blobs first, so the CIDs are in the request.

    SELECT the row by id. A succeeded row returns response_cbor at once. A running row with a live lease waits.

    Otherwise insert it pending and commit. If the insert fails, the tool result is an error and the turn fails. Nothing has happened in the world.

    Claim it with an UPDATE to running and a lease, then perform the effect.

    Write succeeded with response_cbor, or failed with the error. Return the stored response as the tool result.

What changes in the Pi Durable tool loop. The resume path keeps its three replay-safe verbs and gains one branch. A cut-off call whose effect row is succeeded is answered from the receipt, as if it had finished. A row still pending never ran and runs now. Only a running row under a dead lease is answered as interrupted, and that answer names the effect so the model checks one thing instead of guessing.

Boot. Next to sweepAtBoot, rows left running become failed with "server process ended before the effect completed", except kinds marked safe to repeat. publish is safe because the blobs are content addressed. execute is not.

Disk full. The first write that fails must end the turn with a visible error. The durable store does this today. The log projection does not. Make the projection failure fatal to the run where it is currently logged. Add a free-space check to the health endpoint and refuse new runs below a floor, so the queue parks instead of starting turns it cannot record.

Risks and open questions

    Signing before the row exists means a crash between sign and insert loses a signature that was never published. That is fine. Nothing saw it.

    A publish receipt proves the daemon accepted the blobs, not that the network indexed them. The existing visibility retry stays.

    execute cannot be made idempotent by Seed. The receipt turns "did it run?" into a yes or no, which is all the model needs.

    run_journal stays separate at first. Scripts already have content-key matching and a cap. Merging later is possible.

    Prod SQLite is 31 GB. Pruning must ship with the table.

    A multi-process agents server would need the lease expiry to be read. Today it is written and ignored.

Sources

    Incident notes agents-disk-full-orphan-tool-events (2026-09-10) and sandbox-virtiofs-shared-memory-corruption (2026-10-29) in the Seed project memory.

    Seed code: agents/src/runs.ts, agents/src/durable-session.ts, agents/src/api-service.ts (#appendSessionEvent, #withIdempotency, REPLAY_SAFE_VERBS, the projectEntry subscriber), agents/src/sqlite-schema.sql.

    T3 Code, pingdotgg/t3code at commit 32ecd71: docs/internals/overview.md, docs/orchestration-v2/README.md, and EventSink.ts, EffectOutbox.ts, EffectWorker.ts, CommandReceiptStore.ts, Orchestrator.ts under apps/server/src/orchestration-v2/.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

Attributes
name
Outbox and command receipts for agent side effects
summary
Design note, 2026-10-30, cc@starlight. Record the intent of every side effect in the same SQLite transaction as the event that caused it, keep one durable receipt per command, and let retries return the receipt instead of redoing the effect. Why this closes the 2026-09-10 orphan tool event bug class, how T3 Code does it, what Seed has, and a minimal design for the agents server.
displayAuthor
cc@starlight
displayPublishTime
2026-10-30