Understanding syncing
The chosen tentative peer exchange, indexing pipeline and everyday syncing procedures.

Syncing moves the data a device wants and is allowed to receive, plus the evidence needed to validate it. The proposal now specifies how to do that: authenticated peer control, filtered RBSR sets, contextual block fetching, durable publication receipts and resumable change watches.

This is a tentative operational design, not a deployed HM26 protocol. It retains Seed's gRPC/libp2p and reconciliation foundations rather than replacing them with a new tree before authorization is defined.

Frontier: this peer’s sync to-do list

A frontier is the set of blocks that a peer has discovered and still needs to check or fetch to finish syncing a particular resource. It records known outstanding work, not every block the peer lacks and not every block that exists in the world.

For example, Bob learns the address of a selected version from ReconcileHeads. That address enters his frontier. If the block is missing, FetchBlock obtains it. If the block is already stored, the local walk inspects it. A locally complete subtree can be skipped.

Suppose the fetched version points to a manifest, and that manifest lists chunks A and B. Processing each parent replaces its frontier work with discovered child work. Bob cannot queue A or B from that manifest until he learns their addresses. After both chunks are verified and every required dependency is resolved, the frontier for that resource is empty; Commit checks the complete closure before Ack acknowledges it.

In the graph, “To fetch” marks discovered frontier work. Gray “Not held” outlines are observer hints and may not yet be known to Bob. The frontier table stores this work as resource, CID and admitting parent; that parent explains why the request belongs to this resource.

Four parts of one system

Part

Responsibility

Authority

Which principal may read/write/administer each exact resource

Interest

Which scopes/facets to follow, pin or open temporarily

Index

Verified facts, typed dependencies, current statements and derived views

Peer exchange

Control refresh, allowed-set differences, bytes, offers and change hints

The permissions guide states the tentative role/root assumptions. In this first policy, grants are exact-resource, admin-issued and independently revocable except permanent admins. Writers can read; admins can read/write. A hosting service has a proven reader principal because receiving plaintext gives it custody.

Opening a document on a new device

Bob opens a private resource R with a signed invitation/source bundle. His daemon verifies the expected root and hosting principal, authenticates on the peer connection, then calls GetControl for allowed grant/revocation evidence. It does not trust a root or permission just because a peer supplied a CID.

Next OpenScope captures a filtered data view and a watch cursor at the same index boundary. ReconcileBlobs finds missing CIDs; FetchBlobs obtains and verifies bytes. The daemon indexes the resource statement, its change ancestors and apply implementation, then materializes state. It prioritizes text/structure before large attachments and finally watches from that captured cursor.

A selected resource CID commits the signed version declaration. Computed state CID is a separate derived commitment. Missing dependencies or an unsupported apply implementation are reported honestly. A utility state never becomes authoritative merely because replay is inconvenient.

An already-local document can render its verified cached version immediately, followed by control refresh and watch/reconciliation repair. Offline retained display has stale authority knowledge; it is not a fresh network read grant.

Publishing an edit

Carol's daemon signs the new immutable change/resource blobs, indexes locally and commits an outbox. Offline work remains queued. Once connected, it refreshes control and verifies the host's reader/custody rights before sending private IDs or bytes.

OfferBlobs records a stable request ID/body digest and lets the host pull wanted bytes using a recipient-bound export token. The host distinguishes received, hash-verified/stored, waiting for evidence, indexed and rejected stages. A durable indexed receipt comes after the index transaction commits. After a lost reply, status lookup returns the committed receipt rather than making another publication. A relay can deliver Carol's signed statement without becoming its author.

The detailed workflows cover local/offline publication, first sync with multiple identities, reconnects and crash recovery. The peer contract includes exact candidate protobuf definitions.

Permission-safe views and change notifications

A frozen view freezes the data set, not read authority. Before each page, fingerprint, metadata item or block response, check current permission. A revocation invalidates the relevant view or prevents the next response. A recursive view's policy stamp covers every included child's authority and visible membership; checking only the root is insufficient.

The daemon commits grant/revocation changes, revision fences, selected statements, scope invalidation, durable events and receipts together. Tree repair may happen afterward, but old tree membership never bypasses current access checks. The SQLite model gives tables, queries and this transaction order.

WatchScope is a filtered projection of the local durable event log. Persist a cursor only after durably scheduling the corresponding work. Resume after disconnect; duplicate events are harmless. A compacted log or index rebuild yields reset, control refresh and RBSR repair. Local event sequence is not global authority ordering. Watch hints and fingerprints do not prove a peer knows every revocation.

Private links and shared bytes

State ancestors and required attachments belong to typed dependency closure. An ordinary mention or hyperlink does not add its private target. Even a structural reference must pass origin/publication checks before a receiver binds a previously private blob to a new audience: publisher-supplied bytes, existing authorized association, public disclosure or verified source-read context with deliberate publication. A bare CID is insufficient.

One file can have several resource associations. Deduplicate its physical bytes while keeping rights and provenance separate. Likewise, several identities can share one local byte store without sharing their query permissions. Personal revocation evidence remains obtainable after content access is removed; unrelated membership is not exposed.

What revocation and republishing do

Reader revocation stops future serving by honest peers that have learned it. It cannot recall old bytes or instantly reach offline peers. The tentative authority policy also reevaluates a revoked writer's statements, which can roll back selected state and later revive it after regrant. This consequential choice is not adopted; historical authorization needs a different mechanism if that behavior is unacceptable.

For a first live-republish procedure, use a dedicated reader-of-source/writer-of-destination automation that explicitly copies validated versions into ordinary signed destination statements. Source revocation stops future copying, while prior disclosed copies remain. This works with the proposed exchange but does not settle the native republish placeholder's wire payload.

Keep the set representation replaceable

Use existing scoped RBSR first, with tentative CID-only canonical ordering and explicitly encoded fingerprints. Test access-domain Prolly roots as an overlay for metadata/states, using the same allowed sets. Compare disclosure, bytes, CPU, round trips, overlap and revocation cost before changing the core representation. A content-addressed tree is not itself an access policy or global-completeness proof.

Read remaining decisions for the alternatives and sources for current-code evidence and limits.

Schemas

The signed shapes this page talks about, each on its own page with its schema definition:

Learn this in the sync playground

Sync playground extension · 2026-10-23. Teaching prototype; no production protocol or schema adoption.

Read the graph downward: selected version, packing lists, actual content. Compare the same positions on the two peers. The new experiment reconciles resource heads and then walks their admitted dependencies; the earlier sections describe ReconcileBlobs. These are alternative designs to compare, not interchangeable wire contracts.

Read the sync graph

Start with one resource. Read downward from its selected version to the bytes it needs.

Both peer diagrams use the same positions. A solid card means bytes are held; a dashed card is a teaching outline from the observer catalog, not an advertisement received by that peer. An amber outline means the peer has discovered work to fetch. A check means the entire containment subtree is locally complete. Green highlighting marks the current step. Content and permission evidence are separate views. Try the shared-DAG example, step through one request and response, then inspect the SQL changes.

Peer

One device with its own byte store and derived indexes.

The playground gives each peer one principal and independent SQL-shaped tables. A real device can hold multiple identities sharing a physical byte store, but queries and network disclosure still require the caller’s own authority.

Principal

A signing identity; knowing who someone is does not grant access.

Hello authenticates identity in the simulator. Resource rights come from validated grants and root trust, separately from transport identity. Private signing keys are never part of the graph or replicated SQL tables.

Manifest · packing list

A small immutable block listing the addresses of its children.

A manifest contains child addresses, not all their bytes. Read its arrows as “needs these blocks.” The playground splits supplied text into paragraph chunks and packs them in a balanced binary tree. A real format and chunking strategy remain experimental. Manifests with identical encoded children can be shared across resources; this does not share permission.

Here, manifest means a content packing list. The older RPC proposal also uses an offer manifest to bound a transfer, while the naming discussion uses a directory manifest to list named children. These are different roles; a manifest does not by itself establish authority or globally latest state.

Chunk · actual content

A leaf block holding a piece of the supplied content.

In the shared example, “Shared methods” and “Shared observations” are equal bytes in two resources. They therefore reuse chunk addresses and can reuse their manifest. A leaf with verified bytes has no more containment dependencies to fetch. The simulator chunks by paragraphs; it does not implement production file chunking.

Immutable blob

Bytes identified by their content; editing them produces a different address.

Ref, manifest, chunk and authority cards are all blobs in the teaching catalog. A local byte store keeps one physical copy of equal blobs. Authorization depends on their admitted resource context and evidence, not simply on having or naming their address.

CID · content address

An identifier used to verify that received bytes are the bytes requested.

Production CIDs describe codec/hash identity. The playground uses SHA-256 labels over its JSON serialization, not production CID encoding or canonical DAG-CBOR signatures. Human names in the UI resolve these addresses; the raw simulation still uses them as keys.

DAG · dependency graph

Boxes are immutable blocks. An arrow says “this block needs that block.”

DAG means directed acyclic graph. It is allowed to have shared children: two packing lists may point to the same chunk, so the structure is not necessarily a tree. Content and permission proof edges have different meanings and appear in separate views. Ordinary references are excluded from automatic download closure.

Containment edge

Follow this arrow to obtain content required by the selected version.

Only admitted containment edges expand the resource frontier. Ref to payload, manifest to child, and valid change to dependency are examples. A mention, navigation link or proof edge does not automatically make its target content readable.

Reference

A link to another object without an automatic right to fetch it.

An ordinary mention or placement link is not a containment edge. A bare CID cannot launder a private blob into a new audience. Origin-verified publication and current scope authority are still required.

Stored bytes

This peer has verified this block’s bytes. Its children may still be missing.

A solid box records local storage. It is deliberately weaker than a completeness check. In the partial preset Bob already has the selected head, yet must still fetch its required descendants.

Complete subtree

This block and every required containment descendant are present locally.

A check mark caches a local traversal result under fixed rules. It is not permission, a claim of globally latest state, or proof that every revocation is known. Complete subtrees can be skipped without revisiting every leaf. This experiment pins completed closures; a real garbage collector must invalidate affected ancestor markers before evicting descendants.

Not held · observer outline

An outline helps compare the two peers without moving boxes around.

The UI knows the world catalog, but a peer does not. A gray outline is not proof that this peer knows the block’s address. Only frontier rows represent discovered outstanding work. For a peer without current read access, unheld private outlines are suppressed. The exact RPC payload is the source of truth about what crosses the wire.

Active this step

The highlighted blocks participate in this request, response or local operation.

Requests do not store bytes. Responses can store blocks and extend the frontier. A local walk can inspect or skip a subtree without sending any RPC. The current event and SQL panel explain which of these happened; color alone does not imply transfer.

Possession privacy

Skipping local content can reveal that a peer already has those bytes.

Reuse complete subtrees minimizes transfer but reveals request patterns. Uniform scoped traversal fetches the offered closure even when bytes are local; physical storage still deduplicates. This experiment demonstrates one disclosure tradeoff, not a complete traffic-analysis defense.

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

People