Desktop sync failure on yacht: two daemons share one libp2p identity
Investigated 2026-10-29 15:00–15:30 CET by cc@yacht. Nothing was restarted, deleted or re-keyed; every check was read-only (one curl localhost:58001/ipfs/<cid> timed out and may have queued a bitswap want on the dev daemon).
Symptom
hm://z6MkpVa5nMUR5ZaUEyV1SE48KTNbwTuHRd83RwLgMKc4nGU3/ion/pages/seed-atlas (created 11:46Z, last change 11:51Z, version bafyreigqyriu4…) renders on hyper.media. The dev desktop app (Electron from ~/Code/Seed, data dir Seed-local, daemon on 58000/58001/58002) answered document not found for it from 12:05Z through 14:05Z and still does not have it. The production Seed.app (data dir Seed, daemon on 56000/56001/56002) has it at the web's version.
The dev daemon is missing every ion document since 01:34Z today: /ion/pages, /ion/page-inputs/seed-atlas, /ion/progress/seed-atlas-custom-page-port are all absent, although it holds a recursive subscription to that account since 2026-10-05 and is connected to hyper.media's peer 12D3KooWEDdE….
Ruled out
Space generation: account root is the same version, genesis and generation (1791197819319) on dev, prod and hyper.media.
Authorization: the change blob is signed by the account key itself (decoded from /ipfs/<version>), same as a document the dev daemon did receive.
Backend version: dev daemon built from main 1934b9fa9, prod daemon from 045ea104d; the only backend commit between them is a toolchain change.
Network: both daemons report /hypermedia/0.9.2 mainnet and CONNECTED to hyper.media; hyper.media's inbound reconcile limiter rejected nothing; its daemon has been up 18 h.
Cause
~/Library/Application Support/Seed/daemon/keys/libp2p_id_ed25519 and ~/Library/Application Support/Seed-local/daemon/keys/libp2p_id_ed25519 are byte-identical (sha1 8a3aabd6…, both dated 2026-05-25). Both daemons therefore run as peer 12D3KooWHRjLqaRQwPUSYeti9bRya8XAX7Qramyzi9SdTo2aSBu8. Seed-dev, Seed-mba and Seed-e2e have their own keys.
hyper.media's daemon (ssh hm, curl 127.0.0.1:56000/debug/p2p) shows one peer entry for that ID with two connections from this laptop's public IP (79.148.189.214, source ports 32573 and 48561, both opened 14:01:07Z) and one shared bitswap ledger.
go-libp2p's Swarm.NewStream picks bestConnToPeer: direct over relayed, then the connection with the most open streams, with no regard to which connection a request arrived on. boxo's bitswap server delivers blocks through host.NewStream(peer). So when either laptop daemon reconciles with hyper.media (that part works, it is a client-opened stream) and then asks bitswap for the missing blobs, hyper.media sends the blocks down whichever of the two connections currently wins, usually the production app's. The other daemon waits until its context expires and the discovery ends "notfound". That is also why the failure is intermittent: it depends on which connection has more streams at that moment, and on whether the other app is running.
Evidence
hyper.media tags the peer bs-engine-useful; prod tags hyper.media bs-engine-useful (it received blocks from it); dev does not. Dev's only bs-engine-useful peer is a relayed one (12D3KooWEQnz…).
Bitswap fetch outcomes since daemon start (/debug/network): dev complete 487 / idle_timeout 61 / ctx_done 2234, 2350 of 2782 fetches with ≤10 % completeness; prod complete 208 / idle_timeout 167 / ctx_done 3361, 3596 of 3736 at ≤10 %. The page's own note says ctxdone "should be near zero".
Discovery end-to-end on dev: 1.26 M connected, 141 k notfound, 30 error.
Both daemons also show a saturated sync scheduler (dev 6/6 slots, queue 277, 98 % of dispatch passes end "saturated"; prod 6/6, queue 136) over 77 recursive subscriptions each, and 0.09 % dial-to-fetch yield. That is load, not the cause, but it lengthens the window.
Both daemons reconnected to hyper.media in the same second (15:01:07/08 CET), consistent with hyper.media treating them as one peer.
Repair (done 2026-10-29 16:25–16:30 CET)
The daemon only mints a device key for a brand-new data dir; with an existing VERSION file a missing key makes it refuse to start. So the repair was to install a freshly generated key rather than delete the file: quit the dev app (TERM to Electron, which stops its daemon), move the old key to Seed-local/daemon/keys/libp2p_id_ed25519.shared-with-prod-20261029.bak, write a new ed25519 key in libp2p's marshalled format (68 bytes, same as the daemon writes), relaunch. The old device key had signed nothing in the dev database (public_keys has no row for it) and the vault does not reference it.
Result: dev daemon is now peer 12D3KooWKJawK3mfYom1UT8xqnGGmNkWkn159awSVbjsp4YnrQCx on /hypermedia/0.9.2; production keeps 12D3KooWHRjL…. A DiscoverEntity for the seed-atlas page completed in 18 s (2 peers, 5 blobs discovered, 4 downloaded) and /ion/pages, /ion/page-inputs/seed-atlas and /ion/progress/seed-atlas-custom-page-port arrived with it.
Launch gotcha found on the way: the dev app's network mode defaults to the build-time SEED_P2P_TESTNET_NAME, which direnv sets to dev from .env.vars. Launching with plain pnpm desktop under direnv puts the dev daemon on /hypermedia/0.9.2-dev (testnet) and it cannot sync with hyper.media at all. The mprocs desktop pane of ./dev up avoids that via ./dev run-desktop-mainnet, which unsets the variable.
Origin: both Seed and Seed-local data dirs were created in the same second (2026-05-25 11:29:59) with the key files written at 11:31:04, so one was cloned from the other at creation. Starlight's data dirs have distinct keys. Yacht's unused Seed-mba dir carries starlight's production key, so it must never run alongside starlight's app.
Fix as originally proposed
Give the dev data dir its own identity: quit the dev app, move Seed-local/daemon/keys/libp2p_id_ed25519 aside, start it again so the daemon generates a new key. Consequences: the dev node gets a new peer ID (peerstore entries elsewhere go stale and rebuild; hyper.media's seed-support protected tag stays with the production app's ID). This is what was done above.
Product follow-up worth an issue: the daemon could refuse to start, or at least log loudly, when its libp2p key file was copied from another data dir, and a site could flag one peer ID connecting twice from one address.
Diagnosis recipe
lsof -nP -iTCP -sTCP:LISTEN | grep seed-daem to find the daemons; grpcurl -plaintext localhost:<grpc> com.seed.daemon.v1alpha.Daemon/GetInfo for peer ID and protocol.
grpcurl … Documents/GetDocument per daemon to compare what each holds; Subscriptions/ListSubscriptions for background sync.
http://localhost:<http>/debug/network (bitswap outcomes, scheduler occupancy, sync-with-peer outcomes), /debug/p2p (connmgr tags and connections per peer), /debug/journeys (per-resource request trail), /debug/metrics.
On the site: ssh hm 'curl -s 127.0.0.1:56000/debug/p2p' and grep the laptop's peer ID to see how many connections it holds.
shasum ~/Library/Application\ Support/*/daemon/keys/libp2p_id_ed25519 to spot shared identities.
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime