Run queue
This page explains the run queue from the operator's side, covering how many runs execute at once, what happens to runs across a restart, which failures are retried, and when the queue stops taking work.

Every piece of work an agent does in an app environment is a run. A run is a row in the runs table, and that table is also the queue the server takes work from (apps/src/runs.ts). This page covers what an operator sets and watches: how many runs execute at once, what a restart does, what is retried, and what stops the queue.

The first thing to check when runs do not start: curl <server>/api/health and read dispatch. {"paused": true, "reason": …} means the queue has stopped; see When dispatch pauses.

The idea of a run, its statuses and the tree of runs are explained in runs.

Queues and concurrency


    Interactive turns go first. MessageSession claims the turn for a person's message itself, so the message does not wait behind background work.

    Everything else uses the background queue. That is trigger firings, sessions an agent starts, delegated model children and script children.

    Agent runs have a limit. At most SEED_APPS_MAX_CONCURRENT_MODEL_RUNS agent runs hold a provider stream at once. The default is 8, and the flag is --max-concurrent-model-runs.

    Script runs have their own limit. At most SEED_APPS_MAX_CONCURRENT_WORKFLOWS scripts run at once. The default is 32, and the flag is --max-concurrent-workflows. Scripts hold no provider stream. Because the two limits are separate, a script that waits for its children can never hold them up.

    Scripts can run on a worker thread. SEED_APPS_WORKFLOW_WORKER=1 runs scripts on a worker thread instead of on the server's main thread.

    One live agent run per session. The server enforces this when it claims a run.

    Leases are recorded and not enforced. A claim writes a lease of 60 seconds. Nothing reads it. There is one server process, so at startup the boot sweep puts back in the queue every run the last process left claimed or running.

Almost everything shares one thread, so choose limits that fit the host. Diagnosing a stalled server shows how to see whether the thread is too busy.

Across a restart

Restarting the server is always safe. At startup the boot sweep does this:

    It puts back in the queue every run that a dead process left claimed or running. Runs that were already queued continue.

    An agent run continues by replaying the stored events of its session.

    A crash can happen between a stored tool_call and its tool_result. The effect outbox answers such a call. If the effect finished, the answer is its receipt. If it did not, the answer says the call was interrupted and names the effect. So the request to the provider is complete, and the model knows what to check.

    It marks every effect the dead process left running as interrupted.

    A script run continues by replaying its journal.

After the server is ready, a second pass handles any child that finished without waking its parked parent, and removes old receipts. Health reports that pass as bootRecovery; see endpoints and health.

Retries

The server decides from the kind of failure whether to try again:

    Retried: a provider answer in the 500 range, and network failures. The wait doubles each time, from 5 seconds up to 5 minutes.

    Not retried: validation errors and configuration errors. They fail at once.

maxAttempts is 3 (AGENT_RUN_MAX_ATTEMPTS) for background agent runs: trigger firings, sessions an agent starts, delegated model children. It is 1 for interactive turns, because a person is waiting. It is also 1 for script runs, which continue from their journal and never start again.

Parked runs

A parked run uses no resources and takes no place under the limits. Park lists the three reasons a run parks, and wake source says what wakes each. The dispatcher looks once a second for timers that are due.

When dispatch pauses

The queue takes a run only while the file system that holds the database has at least SEED_APPS_MIN_FREE_DISK_MIB free. The default is 512 MiB. Below it:

    queued runs stay queued;

    runs that are already executing finish;

    health reports dispatch: {paused: true, reason}, with the numbers in disk;

    the log says [agents/runs] dispatch paused.

The queue starts again by itself once there is room, and the log says [agents/runs] dispatch resumed. The effect outbox explains why a turn must not start on a full disk.

Script limits

A script runs inside these limits (see apps/src/workflow-host.ts):

    one QuickJS-WASM realm per run;

    64 MiB of memory;

    2 seconds of pure computing between two awaits;

    256 KiB of source;

    5,000 journal entries or 8 MiB of journal per run (WORKFLOW_JOURNAL_MAX_ENTRIES and WORKFLOW_JOURNAL_MAX_BYTES in api-service.ts).

A loop that must run for a long time gets around the journal limit with ctx.continueAsNew. It ends the run and starts a new one that carries only the state the script names.

See also

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

Attributes
name
Run queue
summary
This page explains the run queue from the operator's side, covering how many runs execute at once, what happens to runs across a restart, which failures are retried, and when the queue stops taking work.