Part of Stem. This page defines how a daemon processes data. There is one pipeline, and every blob enters it the same way whether it came from a peer, from the local app, from the CLI or from an HM24 client. The pipeline is driven by Kind descriptors, so a new resource kind is a new document rather than new code.

The goal, restated

Today the daemon has one indexing routine per blob type (Change, Ref, Capability, Comment, Profile, Contact, DagPB), and sync and privacy each know those types by name. The goal recorded on the Seed Resources board was a unified runtime model for resources, permissions and sync, with no special cases for Seed's own data. Stem reaches it with three moves: every resource is declared by a Node blob; every rule a type needs is in its Kind descriptor; and every access question is answered by one evaluator over the authority graph.

The pipeline

flowchart TD
  T[Transfer: blobs + claimed scope + sender] --> V[Validate: hash, signature, structure]
  V --> A[Authorize: signer's level over the scope's nodes]
  A -->|not yet| S[Stash with reason]
  A --> I[Index: handler reads the Kind descriptor]
  I --> F[Facts]
  I --> L[Links]
  I --> N[Affected nodes]
  N --> R[Re-evaluate: fold state, recompute readers]
  R --> B[Materialise blob access]
  B --> W[Notify watchers]
  B --> P[Policy: keep, pin or allow collection]
  S -.->|a dependency or grant arrives| V

1. Transfer

Every batch of blobs arrives as a transfer: the blobs, the peer (or local client) that sent them, the account the sender had authenticated as if any, and the scope the sender claims they belong to. A Fetch the daemon made writes a transfer with the scope it was syncing; an accepted Offer writes one with the offered scope; a local Publish writes one with the client's scope or a derived one. The claimed scope is the context the roadmap asked uploads to carry: it says where the blobs are meant to live, so the daemon can check that the sender is allowed to put them there before it does any other work. A transfer whose scope the local policy ignores is refused in full.

2. Validate

Each blob is decoded and checked against things that need no other blob: its bytes hash to the CID the sender named; it is DAG-CBOR or a file codec; if it has a type it is one the daemon knows (the six Stem types, the five HM24 types, or a kind-defined extension); its signature verifies with the zero-sig rule from Signed blobs; its structure matches the type's schema (strict for the six Stem types; for a Snapshot, the value is validated against the schema named in the blob, which must be the kind's schema). A blob that fails is rejected with reason invalid and the whole transfer is not rolled back: unlike today, where one bad blob fails the batch, Stem rejects per blob, because a transfer is a claim by one sender and other blobs in it may still be good.

3. Authorize

Authorization asks the authority graph one question per blob: does the signer hold the level this blob requires over the node it affects? A Node blob requires write on its node, or on its parent when it has no id and so creates the node; the signer may be any key with that level, not only the owner, and the node then belongs to the owner's space. A Change or Snapshot requires that some head Node blob for a node the signer may write names it, directly or through dep and prev; a Change that nobody's Node blob has claimed yet is held until one does. A Grant requires that the signer hold its access over its subject. A Revocation requires issuer, delegate or admin standing. A Group requires nothing but a valid signature. A file requires that some authorized state embeds it.

The blob must also belong to the claimed scope: a Node blob for a node outside the scope's subtree, or a Grant whose subject is elsewhere, is rejected as out-of-scope. This is what stops a sender from using a scope it may write to as a carrier for blobs about a scope it may not.

A blob that fails authorization because of something that may still arrive is stashed, not rejected: the signer's grant has not been seen yet (PermissionDenied), a dependency is missing (FailedPrecondition), a parent node does not exist yet. The blob is stored, recorded as stashed with its reason and the identifiers it is waiting on, and re-run when a matching blob is indexed. This is today's stash mechanism, generalised: today a stashed Ref is retried when a Capability naming its signer arrives; in Stem any grant, revocation, group change, Node blob or dependency can unstash. Stashed blobs are not advertised, not served and not counted in any scope set.

4. Index

The handler is one routine. For a Node blob it reads the Kind descriptor named by kind, checks the descriptor's rules (state matches the target kind, naming matches whether a name is present, children allows the parent relationship for any existing children, target names a field that exists in the state when access is target), and emits three things.

Facts. Indexed statements about the node with provenance: signed facts read from the state (the title, the attributes the kind's schema declares as indexable, the target of a comment) and derived facts the peer computes (heads, child count, comment count, last activity). See fact.

Links. Typed edges, built in for structure (dep from Change deps and Node heads, prev from Snapshot prev and Node prev, proof from Node and Grant proof, parent from Node parent, schema from Snapshot schema and Node kind) and declared by the descriptor's link rules for content (embed, link, mention, target, file found at the JSON Pointers the rules name). See Links.

Affected nodes. The node the blob declares; for a Grant, Revocation or Group, every node whose readers or writers may change; for a Node blob that changes placement, the node and its subtree.

For a Change or Snapshot the handler emits links and facts the same way, using the schema named by the kind of whichever node claims it. For a file blob it emits nothing but its presence.

Nothing in this step mentions documents, comments or profiles by name. The core kinds are published descriptors like any other; the daemon ships with their CIDs pinned so that it can start before it has synced them.

5. Re-evaluate

Each affected node is re-folded as Resources and nodes describes, producing its head declarations, state, version and placement. Then readers are recomputed for the affected subtree as Authority and Privacy describe. Evaluation is incremental: a Change affects one node; a Grant on a subtree affects that subtree; a Node blob that moves a folder affects the folder's subtree in both old and new positions. Nothing is recomputed that the new blob does not reach.

Re-evaluation may unstash: a Node blob whose signer is now authorized, a Change whose dependency just landed, a comment whose target now exists. Unstashed blobs re-enter at step 2.

6. Materialise blob access

Every blob reachable from an affected node's state through structural links (dep, prev, file, and the Node blob itself) is mapped to that node in the blob access relation. Grants, Revocations and Groups are mapped to their subjects. From this point a request for any of these blobs is a join against the node's readers, on every surface.

7. Notify and keep

Peers with a live Watch on a scope that the affected nodes fall in are sent the CIDs that entered the scope set and that they may read. The local policy then decides retention: blobs in a followed or pinned scope are kept; blobs fetched on demand are eligible for collection when nothing retained links to them; blobs in an ignored scope were refused at step 1.

What the pipeline produces

A processed transfer results in new or changed node states, new readers, new blob access rows, new facts and links, and a transfer record with per-blob verdicts (indexed, stashed with reason, rejected with reason). The board's phrase was that a processed batch should result in new named roots; in Stem it results in re-folded nodes.

Senders

The board listed four kinds of sender and asked whether anonymous web pushes should go. In Stem:

Sender	How it enters	What it may claim
A peer answering our `Fetch`	the scope we asked for	nothing more than we asked
A peer calling `Offer`	the offered scope, with proof	only scopes our policy accepts and the peer can show authority for
The local app or CLI via `Publish`	the client's scope or a derived one	anything the signing key is authorized for
An HTTP client via a site's publish endpoint	the client's scope	anything the signing key is authorized for; the site is just a peer calling `Publish`

An anonymous push of arbitrary blobs no longer exists. A blob with no signer (a file) enters only when an authorized state embeds it, which is the ownership claim the permissions investigation asked for on raw uploads.

The migration box

HM24 blobs are accepted for as long as the transition lasts, and converted into Stem records between validation and authorization. The team's name for this step is the magic box: old data is signature-validated as it is, then passed through a conversion into what the normal indexer expects. Conversion is deterministic, so every peer derives the same nodes from the same HM24 blobs, and the original blobs are kept so signatures remain verifiable.

HM24 blob	Stem record it becomes
Ref, version shape, at path P	Node whose id is the SHA-256 CID of the earliest Ref blob for P and this genesis (that Ref is treated as the creating blob), kind document, parent the migrated id of P's parent path (or the root id), name the last segment of P, target `heads` = the Ref's heads, `prev` = the migrated Refs it supersedes by `(ts, CID)` order within the same genesis. Each HM24 generation of P becomes its own node sharing the name
Ref, tombstone shape	Node for the same id, target `tombstone`, `prev` as above
Ref, redirect shape	Node for the same id, target `redirect` to the migrated id of the target path, `republish` copied
Ref with `visibility: Private`	the node gets `access: own`; no `everyone` grant is derived for it. A public Ref under a public root inherits the root's `everyone` grant
Home document (path empty) and its Changes	The space root, whose id is derived from the owner's deterministic creating blob, kind space, target `heads` = the home Refs' heads; profile attributes merged into the root document's attributes
Profile	attributes (name, icon, description, alias) on the space root's state; `account` on an agent-signed Profile becomes the signer acting under its migrated grant
Capability, role WRITER, path P	Grant: subject the migrated node of P with subtree (or the space root when P is empty), audience key(delegate), access `write`, signer the owner
Capability, role AGENT	Grant: subject the space root with subtree, audience key(delegate), access `write`, signer the owner. The roadmap decided to discontinue path-scoped and non-recursive grants, so an AGENT becomes a whole-space writer and nothing finer
Comment with TSID T	Node whose id is the SHA-256 CID of the first Comment blob carrying T, in the author's space, kind comment, parent the root id, no name, `access: target`, target `snapshot` = a derived Snapshot whose value is the comment's target (migrated id), threadRoot, replyParent and body; `prev` links edits of the same TSID in `(ts, CID)` order; an empty body becomes a tombstone
Contact with TSID T	Node whose id is the SHA-256 CID of the first Contact blob carrying T, kind contact, parent the root id, no name, target `snapshot` with subject, name, follow and join flags derived from `subscribe.profile` and `subscribe.site`
Public space (any public Ref)	Grant: subject the space root, audience everyone, access `read`, derived once per space from the owner's first public Ref
`siteUrl` attribute on the home document	`site.url` on the root attributes; no `sync` grant is derived, so the owner must issue one before the site regains authority-peer standing

Derived Snapshots and derived Grants carry the migrated blob's CID in a source note so their origin is auditable, and are signed by nobody: they are index records that stand in for a signature the original blob already carries. The roadmap records that sub-document capabilities are expected to be discarded by the migration; the table above keeps them as write grants on the migrated node, because the id exists and the grant is harmless, but the app stops issuing them.

Changes need no conversion. Files need no conversion.

Reindex

A reindex replays every stored blob through the pipeline in storage order. Because the pipeline stashes rather than failing on missing dependencies, and because every rule is deterministic, a reindex reproduces the same state regardless of the order blobs originally arrived. The derived tables listed in Database structure are dropped and rebuilt; the two ledgers are not, since they record history rather than derive it.

Today (HM24)

Today	Stem
Seven indexers, one per blob type	One handler driven by Kind descriptors
A batch fails as a whole on one bad blob	Per-blob verdicts; transfers record them
Visibility propagated by a rules table over link types	Blob access filled from each node's structural links; readers from the authority graph
Stashed Refs retried on a matching Capability	Any blob stashed on any missing authority or dependency, retried on any matching arrival
Push of arbitrary blobs, validated after the fact	Transfers carry a claimed scope and are authorized against it first
No record of what was served or received beyond metrics	Disclosure ledger and transfer log

The shipping pipeline is documented in Signed blobs under "What happens to a blob".

Open questions

Open: whether derived index records for migrated HM24 blobs should be materialised as real signed blobs by the owner's device once it comes online, so that a Stem-only peer can hold a migrated space without the HM24 originals.

Open: how a daemon learns about a Kind descriptor it has never seen (a third-party kind). Stem's position: the descriptor is fetched like any dependency through the schema link, and until it arrives the Node blob is stashed.