beads/docs/multi-agent/federation.md
Julian Knutsen 47d5db39ff
feat(http): bd heartbeat over bd serve (issues.heartbeat), and close the re-arm orphan-lease race (#7600)
* feat(roles): issueops.Heartbeater role on the dolt, embedded and uow legs

Adds issueops.Heartbeater.Heartbeat(HeartbeatRequest{Actor, IssueID}) ->
HeartbeatResult{IssueID, Holder, HeartbeatAt, LeaseExpiresAt, Rearmed},
the lease refresh behind `bd heartbeat`, as its own role with its own
accessor.

- One body: internal/storage/issueops.ExecuteHeartbeatInTx validates
  (*HeartbeatFieldError for a blank/over-long actor or empty id), refuses
  a wisp, then runs RenewLeaseInTx, which is HeartbeatIssueInTx's body
  reporting the renewal. HeartbeatIssueInTx is now a wrapper, so the legacy
  HeartbeatIssue legs answer the same text and sentinels.
- The refresh is fenced on the holder AND on the live claim: after the
  holder-fenced UPDATE, a read in the same transaction requires the issue
  to be in_progress under the holder (the `update leases set` prefix and
  the one IF() the Postgres dialect rewrites are unchanged; a Postgres port
  should expect more serialization retries from that read). A refusal deletes the caller's orphaned lease
  row, and the legs commit that refusal.
- A refresh writes only the lease row: no row_lock, version, updated_at,
  event, journal entry, Dolt commit or hook. A re-arm (a live hand-doled
  claim with no lease row the caller holds) is recorded as a mutation:
  row_lock reminted, updated_at stamped, a lease_rearmed event, a journal
  entry, a version, a Dolt commit (LeaseRearmVersionCommit) and on_update.
  The remint makes a re-arm collide with a racing release or close instead
  of leaving an orphaned lease on an open or closed issue (a pre-existing
  race on every Dolt route). Rearmed is true only when a row was created:
  a same-second refresh (0 changed rows on embedded Dolt) or a holder
  spelled differently refreshes the existing row in place and keeps its
  granted_at/granted_node. Reported times are rounded to whole seconds,
  the stored values; the binds stay unrounded so a same-second refresh is
  never a 0-row UPDATE on Dolt.
- Typed refusals: an absent id is ErrNotFound; a foreign holder is
  *LeaseConflictError{Holder} wrapping ErrAlreadyClaimed; a wisp
  (Ephemeral) or no live claim is *LeaseConflictError wrapping the new
  ErrLeaseNotHeld, which also matches ErrNotClaimable.
- Legs: DoltStore and EmbeddedDoltStore through the issue-operation
  funnel (DOLT_ADD of the named tables, then DOLT_COMMIT -m, after the SQL
  commit); uow.NewHeartbeater through the new uow.RunTxScoped: a refresh or
  a refusal commits with the plain SQL COMMIT (no working-set scan, no Dolt
  commit, nothing another writer left uncommitted is swept), and a re-arm
  commits through the new ScopedCommitter (DOLT_ADD of exactly its tables,
  DOLT_COMMIT -m, never -Am). The legacy HeartbeatIssue doors commit a
  re-arm the same way and keep a refusal's orphan delete, and the
  proxied-server `bd heartbeat` route runs the role's body with the same
  scoped commit.
- Census: storage.Storage gains Heartbeater(); HookFiringStore wraps it
  (fires on_update for a re-arm only, RoleFiresHooks knows it), telemetry
  wraps it, httpapi.timedProvider, uow.notifyingProvider (records a re-arm)
  and externaldeps.uowProvider build it; the journal and version guards
  list the heartbeat family as entry points; the httpclient unsupported
  shell is regenerated and Heartbeater sits on its unsupported allowlist
  until the wire lands.

Conformance: RunHeartbeater* (18 cases) on dolt, embedded dolt and uow:
refresh post-state with the stored times pinned to the result; no
row_lock/current_revision/updated_at/events/journal/issue_versions/
version-control/hook change; a guarded write with the pre-heartbeat
revision still lands; foreign holder refused against an aged lease;
refused after close/release/reclaim/status move; wisp refused; hand-doled
claim re-armed as a mutation; same-second heartbeat is not a re-arm;
orphaned leases on released, closed and reassigned issues refused and
deleted; a refresh and a refusal against a dirty working set make no
commit and leave the other writer's change uncommitted, and a re-arm
commits only its own tables; absent id; request rules; no request mutation. The http leg is
waived until the wire lands.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(httpapi): POST /v0/beads/issues/{id}:heartbeat behind issues.heartbeat

Publishes issueops.Heartbeater on bd serve as heartbeatIssue, the served
lease refresh behind `bd heartbeat`.

- Spec: the operation, HeartbeatIssueRequest {actor} and
  HeartbeatIssueResponse {id, holder, heartbeat_at, lease_expires_at,
  rearmed}; no `revision` (a refresh writes nothing to the row) and no
  expected_version (the fence is the holder and the live claim).
  issues.heartbeat joins the capability vocabulary and lease_not_held
  (409) the Problem.code list; `reason: ephemeral` qualifies it for a
  wisp. apigen and the wire-shape golden are regenerated; the golden diff
  is additive only, so wire_revision stays 2.
- Server: route row on the custom-method dispatcher, handleHeartbeat in
  heartbeat.go (claim's actor rules via bodyActor, unknown member and
  query refusals), Config.Heartbeater as a required role (sourceRoles,
  roleSourceNames, the per-request uow.HeartbeaterSource accessor).
  failHeartbeat maps *LeaseConflictError typed fields: a foreign holder
  is the claim's conflict shape (already_claimed + assignee +
  issue_status); no live claim, a wisp included, is the new
  lease_not_held (+ issue_status, + reason: ephemeral for a wisp), matched
  before the generic arms so ErrLeaseNotHeld's ErrNotClaimable never
  surfaces as not_claimable. Its detail names the one state the refusing
  transaction found (closed, open and not claimed, an ephemeral wisp, ...).
- bd serve wires the role from its peeled store (serveRoleSource gains
  Heartbeater); the serve stub stores, the corpus and e2e harnesses, the
  served harness and the encode oracle bind it.
- Wire client vocabulary: OpHeartbeatIssue/issues.heartbeat and
  MethodHeartbeat are CLASSIFIED (the capability set-equality gate),
  ReasonEphemeral is pinned to the server's, and lease_not_held rebuilds
  *LeaseConflictError{Status, Ephemeral} wrapping ErrLeaseNotHeld in the
  native words (the code-table set-equality gate). No accessor yet.

Tests: heartbeat_test.go (request mapping and projection, edge refusals,
every role refusal with its code, members and state-specific detail, the
code row, the capability). The served-http Heartbeater leg
(served_heartbeater_test.go, 18 cases) runs the contract against a real
bd serve through a test-local dial over the store's own dispatch, and the
http-leg waivers are deleted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(http): bd heartbeat over http through httpclient.Heartbeater

The client leg of the served lease refresh, and `bd heartbeat` on the
role for every route.

- httpclient.Store.Heartbeater (heartbeater.go) dials POST
  /v0/beads/issues/{id}:heartbeat through Store.dispatch, so
  issues.heartbeat is preflighted: an older server is refused BEFORE the
  dial with *wire.CapabilityError naming the token (unwraps to
  *storage.ErrUnsupported{Capability: "issues.heartbeat"}); it never falls
  back to a re-claim or a metadata stamp.
- The role's request rules are restated before the dial
  (validateHeartbeatRequest, pinned to
  storeops.ValidateHeartbeatRequest): blank/over-long actor, empty id, each
  a *HeartbeatFieldError.
- Problem mapping: already_claimed (which the shared table rebuilds as the
  claim's ClaimConflictError) becomes *issueops.LeaseConflictError naming
  the holder and wrapping ErrAlreadyClaimed, in the native words ("issue
  already claimed by <holder>"); lease_not_held already arrives as
  LeaseConflictError (Ephemeral from `reason`); not_found is ErrNotFound;
  invalid_argument on `actor` is the role's field error.
- Idempotent: true, the one write this client marks replayable (a repeat
  only moves the expiry later; a replay after a reclaim fails the fence).
- Legacy Store.HeartbeatIssue rides the role. Both leave the generated
  shell and completeness_test's unsupported allowlist.
- `bd heartbeat` takes Heartbeater() from the resolved store on the direct
  routes (dolt, embedded, http) and from the provider's HeartbeaterSource
  on the proxied route (exact id resolution in a read unit of work, then
  the role); runHeartbeatProxiedServer and heartbeat_proxied_server.go are
  deleted. Success output and exit codes are unchanged; refusal text
  changes per route (see CHANGELOG).

Tests: heartbeater_test.go (stub server: preflight refusal naming the
token with no POST, request/answer projection, each refusal's typed
mapping including the wisp's reason, the 503 replay, pre-dial validation,
validator parity, the legacy door). served_heartbeater_test.go runs the
18-case contract through env.subject.Heartbeater(), with
HookedHeartbeater wired off a HookFiringStore. TestE2E_HeartbeatOverHTTP:
real bd against an in-process serve: holder ok with revision and
updated_at unchanged and a guarded write with the pre-heartbeat revision
landing; a non-holder refused against an aged lease that stays untouched;
closed, wisp and unclaimed refused, each naming its own state; an older
server's handshake refused naming issues.heartbeat with no write. Corpus
shapes gc-heartbeat, -non-holder, -closed, -wisp pending until
reference_server.json advertises the token. CHANGELOG entry for the whole
track, including the orphaned-lease race fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(bazel): auto-sync BUILD files

Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/38875641041
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.

* fix(heartbeat): roll back a no-issue refusal; restate refresh vs re-arm

Review fixes for #7600 (adopt-pr apply-fixes, iterations 1-2).

Legacy HeartbeatIssue doors commit a refusal only when it read the issue.
The dolt and embedded HeartbeatIssue callbacks committed every
LeaseRenewalRefusal, including Found=false (the issue read failed). That
refusal deletes nothing, but it can follow a fast-path UPDATE that
already extended an orphan lease row's expiry, so committing it kept the
extension. issueops.LeaseRenewalRefusalCommits now gates the commit on
Found, the rule the Heartbeater role's HeartbeatRefusalCommits applies.
New tests on both legs pin both arms: a no-issue orphan's expiry is
unchanged, and a released issue's orphan is deleted.

Refresh vs re-arm contract docs. Comments, Long help, the OpenAPI spec
and the generated apigen still said every heartbeat writes only the
lease row, makes no Dolt commit and fires no hook. A re-arm is recorded
as a mutation: it remints the revision, records an event and a version
entry, commits once (LeaseRearmVersionCommit) and fires on_update
through the wrapping hook decorator. Restated in domain, uow, issueops,
storage.go (why RoleFiresHooks answers true, so bd serve takes the
heartbeater off a peeled store), cmd/bd heartbeat and serve, httpapi,
httpclient, the spec (apigen regenerated), and the conformance and
contract-test docs.

Minor review findings:
- The 409 already_claimed and lease_not_held details now end "nothing
  on the issue was written": the refusal may have deleted and committed
  the caller's orphan lease row.
- docs/multi-agent/federation.md and the CHANGELOG entry describe the
  spelling-mismatch heartbeat: the row is refreshed in place and keeps
  granted_at and granted_node; only a re-arm stamps this node. The
  embedded spelling test pins both columns, and the lease.go comments
  say "refreshed in place" instead of "re-arms".
- The refusal's orphan delete judges the holder under actorMatches, as
  the in-place refresh does (iteration 2). It matched holder = ? on the
  actor as sent, so a dead claim's orphan held under another spelling
  of the caller (gastown.mayor, heartbeat as gastown__mayor) survived
  the refusal until TTL. The refused closure now reads the row's holder
  and deletes that stored holder only when actorMatches(holder, actor).
  Both TestHeartbeatIssueCommitsARefusalOnlyWhenItReadTheIssue twins
  gain a cross-spelling arm (row deleted) and a control: another
  identity's row (gastown.dog-3) survives.

Not changed: `bd heartbeat --json` still omits `rearmed`, since adding
it changes a CLI JSON contract (follow-up). The federation.md paragraph
post-dates the v1.2.2 docs pin, so documenting this main-only behaviour
there is a maintainer call under
engdocs/decisions/2026-07-17-docs-release-pin.md section 1.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(uow): run the scoped re-arm's Dolt commit after the SQL commit

The proxied-server heartbeat commits a re-arm with doltServerTx.CommitScoped,
which ran DOLT_ADD per table and DOLT_COMMIT('-m') INSIDE the open
transaction, before the SQL COMMIT. DOLT_ADD stages whole tables from the
session's transaction root, so that Dolt commit was built from the re-arm's
start snapshot: an issues row another session changed while the re-arm was
open (a write still only in the working set, as in batch mode or before that
writer's own Dolt commit) was staged at its old value, and HEAD disagreed
with the rows every reader sees. This is the LatentLabsSpace/NEXUS#92
ordering the DoltStore funnel (runIssueOperationTxWithMessage) already
avoids, and the review's major finding on this PR.

CommitScoped now commits the SQL transaction first, then stages exactly the
named tables (plus the versioned-history tables) and makes the Dolt commit on
the same pinned connection, the uow twin of doltAddAndCommitPostTx:

- a serialization failure of the SQL COMMIT is still returned, so
  RunTxScoped replays the unit of work;
- the Dolt commit runs detached on its own budget and replays only conflicts
  that leave nothing behind: serialization failures and Dolt's 1105
  "@autocommit transaction rolled back" (new isDoltAutocommitRollbackError,
  mirroring the DoltStore classifier of the same name);
- nothing staged, or "nothing to commit", is not an error: a concurrent
  writer's Dolt commit already took the row. A "nothing to commit" past the
  staged check is logged, as the funnel logs it, so the missing audit line
  is explicable;
- any other failure is logged, never returned. The data is committed, and
  returning it would make RunTxScoped replay a committed re-arm, which would
  then report a refresh. The session is poisoned rather than pooled, since
  its state is unknown. uow records no metrics, so that log line is the
  drop's only signal; bd.db.post_tx_commit_dropped counts only the funnel's.

Statements go through schema.DrainCall, retiring the local drainCall fork
(review nit).

Tests: five sqlmock tests pin the order (SQL COMMIT before any DOLT_ADD),
the swallow-and-poison path, the 1105 retry, the returned serialization
failure and the empty-commit tolerance; each fails against the previous
CommitScoped. TestIsDoltAutocommitRollbackError carries the DoltStore
classifier's table over to the uow copy, plus the prefix boundary; dropping
the prefix branch, the prefix's period, the 1105 check or the type check
each fails a case. TestHeartbeaterRearmKeepsAConcurrentWrite is the
real-Dolt regression: a change to another issues row made while the
re-arm's transaction is open must be at HEAD after the re-arm's Dolt
commit. Its working-set-only writer fails on the old ordering (dolt
2.1.8); the writer that Dolt-commits its own change passes on both and is
kept as coverage.

Review follow-ups folded in (comments, a file move and a test; the
absorption log line above is the only behaviour change among them):

- the freshRowLock invariant and its row_lock_guard_test.go mirror name the
  re-arm as the one deliberate remint without a lifecycle-field change,
  load-bearing for the release/close collision;
- httpclient: "a write is never retried" (wire/client.go, stall_retry.go)
  and "only a read's answer is yes" (dispatch.go) now allow for the
  Idempotent heartbeat, and httpHeartbeater and issueops.HeartbeatResult
  document that a replayed heartbeat can report Rearmed false after a
  committed re-arm, with the guarded write's revision check as the backstop;
- the proxied route moves back to cmd/bd/heartbeat_proxied_server.go, the
  file family cmd/bd/AGENTS.md prescribes;
- the ScopedCommitter and RunTxScoped docs no longer say every other
  writer's uncommitted change stays out of the commit: a staged table is
  committed whole, so such a change to a named table rides in the re-arm's
  commit, as it does in the funnel, and only other tables keep theirs;
- commitStagedTables says its drop is log-only, the absorption is logged,
  and the uow classifier gets its table test (see Tests).

Not folded in: the base-owned IssueUseCase.Heartbeat /
IssueSQLRepository.HeartbeatIssue door, dead in production since the
proxied route moved to the Heartbeater role. Its one caller is the
notifying hook-parity test (uow/notifying_test.go), and moving that case
onto the Heartbeater means re-deriving its RenewLease hook semantics, so it
is left for a follow-up.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
2026-10-16 22:38:23 +00:00

12 KiB
Raw Blame History

title description
Federation Setup Guide Configure peer-to-peer sync of beads databases across workspaces with Dolt remotes, sovereignty tiers, and topologies

Federation enables peer-to-peer synchronization of beads databases between multiple workspaces using Dolt remotes. Each workspace maintains its own database while sharing work items with configured peers.

Overview

Federation uses Dolt's distributed version control capabilities to sync issue data between independent teams or locations. Key benefits:

  • Peer-to-peer: No central server required; each town is autonomous
  • Database-native versioning: Built on Dolt's version control, not file exports
  • Flexible infrastructure: Works with DoltHub, S3, GCS, local paths, or SSH
  • Data sovereignty: Configurable tiers for compliance (GDPR, regional laws)
Just want two machines sharing one database through a bucket you own? The [Bucket Federation Quickstart](/multi-agent/bucket-federation) walks that path end to end — bucket, seed push, second replica, sync cadence — with measured timings and the failure modes.

Prerequisites

  1. Dolt backend: Federation requires the Dolt storage backend (the only supported backend)

Configuration

Enable Federation-Compatible Sync

Edit .beads/config.yaml or ~/.config/bd/config.yaml:

federation:
  remote: dolthub://myorg/beads          # Primary remote (optional)
  sovereignty: T2                        # Data sovereignty tier

Or via environment variables:

export BD_FEDERATION_REMOTE="dolthub://myorg/beads"
export BD_FEDERATION_SOVEREIGNTY="T2"

Data Sovereignty Tiers

Tier Description Use Case
T1 No restrictions Public data
T2 Organization-level Regional/company compliance
T3 Pseudonymous Identifiers removed
T4 Anonymous Maximum privacy

Adding Federation Peers

Use bd federation add-peer to register remote peers:

bd federation add-peer <name> <endpoint>

Peer Name Rules

  • Must start with a letter
  • Alphanumeric, dash, and underscore only
  • Maximum 64 characters

Supported Endpoint Formats

Format Example Description
DoltHub dolthub://org/repo DoltHub hosted repository
Google Cloud gs://bucket/path Google Cloud Storage
Amazon S3 s3://bucket/path Amazon S3
Local file:///path/to/backup Local filesystem
HTTPS https://host/path HTTPS remote
SSH ssh://host/path SSH remote
Git SSH git@host:path Git SSH shorthand

Examples

# Add a staging environment on DoltHub
bd federation add-peer staging dolthub://myorg/staging-beads

# Add a cloud backup
bd federation add-peer backup gs://mybucket/beads-backup
bd federation add-peer backup-s3 s3://mybucket/beads-backup

# Add a local backup
bd federation add-peer local file:///home/user/beads-backup

# Add a partner organization
bd federation add-peer partner-town dolthub://partner-org/beads

Credentials

Peers configured with --user (and optionally --password, otherwise prompted interactively) store SQL credentials AES-256 encrypted, locally. Stored credentials are used automatically during sync:

bd federation add-peer town-gamma 192.168.1.100:3306/beads --user sync-bot

JSON Output

For scripting, use the --json flag:

bd --json federation add-peer staging dolthub://myorg/staging-beads
# {"added":"staging","url":"dolthub://myorg/staging-beads","has_auth":false,"sovereignty":""}

Verify Configuration

List configured peers:

bd federation list-peers

Syncing with Peers

Use bd federation sync to pull from and push to peer towns, and bd federation status to check sync state without transferring data.

# Sync with all peers
bd federation sync

# Sync with a specific peer
bd federation sync --peer town-beta

# Handle conflicts
bd federation sync --strategy theirs  # or 'ours'

# Check status (ahead/behind, reachability, conflicts)
bd federation status
bd federation status --peer town-beta

Without --strategy, a sync that hits merge conflicts pauses and reports the conflicting tables for manual resolution instead of auto-resolving.

Topologies

Pattern Description Use Case
Hub-spoke Central hub, satellites sync to hub Team with central coordination
Mesh All peers sync with each other Decentralized collaboration
Hierarchical Tree of hubs Multi-team organizations

Architecture Notes

How It Works

  1. Each workspace has its own Dolt database
  2. add-peer registers a Dolt remote (similar to git remote add)
  3. bd federation sync pushes and pulls commits between peers
  4. Conflict resolution follows the configured strategy

When run against a Dolt SQL server, federation uses two ports: MySQL (3306) for multi-writer SQL access, and remotesapi (8080) for peer-to-peer push/pull:

┌─────────────────┐         ┌─────────────────┐
│  Workspace A    │◄───────►│  Workspace B    │
│  dolt sql-server│  sync   │  dolt sql-server│
│  :3306 (sql)    │         │  :3306 (sql)    │
│  :8080 (remote) │         │  :8080 (remote) │
└─────────────────┘         └─────────────────┘

Multi-Repo Support

Issues track their SourceSystem to identify which federated system created them. This enables proper attribution and trust chains across organizations.

Connectivity

Remote connectivity is validated on first push/pull operation, not when adding the peer. This allows configuring remotes before infrastructure is ready.

Leases are per-replica

A claim lease (bd ready --claim + bd heartbeat, reaped by bd reclaim) is only meaningful on the replica that granted it. The leases table is clone-local and never replicates; what crosses the bridge is the claim's visibility — status/assignee on the issue row — and that is stale on every other replica by up to one sync interval.

Two rules follow, and a federated deployment owes both:

  1. Grace window > sync interval, and lease TTL > sync interval. A TTL or bd reclaim --older-than grace shorter than the cadence at which replicas exchange state is meaningless across the bridge: the remote view is a full interval old by construction, so a reaper over there would be judging liveness from data older than the lease itself. bd reclaim defaults its grace to 2× the lease TTL; raise the TTL (or the grace) above your sync interval, never shrink the interval to fit them.
  2. Reclaim belongs to the granting replica. Each lease records the replica that granted it, and bd reclaim skips a lease granted elsewhere, naming it on stderr. Reap dead workers on the machine that hired them.

The guard is opt-in: it arms only where you name this replica.

export BEADS_NODE_ID=mini          # per-machine; or `bd config set node_id mini`

Two rules about what to name, both load-bearing:

  • node_id names the STORE, not the host. One value per beads database. Hosts that are clients of the same dolt sql-server (BEADS_DOLT_SERVER_HOST, a systemd/Docker server, Hosted Dolt, a VPS) are one replica no matter how many machines they are — give them all the same value, or leave it unset. Give them distinct ids and you rebuild the very fail-closed regression described below: a supervisor would match no worker's lease and reclaim 0 forever. Name a replica only where there is a real sync interval between it and the others.
  • node_id is per-machine, so it must never be committed. The project .beads/config.yaml is a git-tracked file. A node_id committed there propagates one machine's identity to every clone that pulls it, and then every comparison matches: the guard is fully armed and fully inert, and laptop reaps mini's leases exactly as if they were local — the precise hazard this feature exists to close, now happening while you believe you are protected. That is worse than not setting it at all. bd config set node_id therefore writes the user-global ~/.config/bd/config.yaml, alongside the other per-machine state (sync-state.json, push-state.json, redirect). Use the env var or that command; never hand-add node_id to .beads/config.yaml.

There is deliberately no hostname fallback. The hostname answers the wrong question — it names the client process's machine, not the store — and guessing gets it wrong in the topologies that most need automated reclaim: with a shared or remote dolt sql-server (BEADS_DOLT_SERVER_HOST, Hosted Dolt, a VPS) many hosts are clients of ONE store with no sync interval between them, so a per-hostname identity would stop a supervisor reaping any worker's lease at all; in a container the hostname is a per-run container ID; on macOS the transient hostname follows the network. Each of those would strand work on a deployment with no federation at all — a worse failure than the one this guard prevents.

So an unset identity degrades to the old behavior (every lease treated as local) rather than failing closed: an upgrade, and any single-store deployment, can never strand a lease the reaper could previously recover. Leases granted before this feature landed likewise carry no replica and stay reclaimable until a heartbeat re-stamps them with a configured node.

bd reclaim --any-replica disarms the guard. It is for a replica that is permanently gone (or a node that was renamed and now sees its own old leases as foreign) — not a normal setting, since only the granting machine has a first-hand view of whether the holder is alive.

A heartbeat proves the holder is alive; it does not move the lease. An ordinary heartbeat only backfills the granting replica when it is still empty — it does not overwrite a row that positively names one. So a lease normally keeps its granting replica for life, and states like these strand one where a local heartbeat is keeping it alive:

  • a replica that was renamed (mini → mini2) heartbeats its own leases, which now read foreign forever;
  • a foreign lease that arrived through the JSONL interchange, whose holder name also exists locally, gets heartbeated here but stays labelled with the remote node;
  • an import that lands on an already-expired local lease row takes the whole row from the snapshot, granting replica included — so this node's own stale lease can come back labelled with a remote one.

(No heartbeat moves an existing lease, including one whose holder is a different spelling of the lease row's holder: that row is refreshed in place and keeps its granting replica. Only a re-arm, which grants a lease to a live claim that had none, stamps this node.)

Recover a stranded lease with bd reclaim --any-replica, once you have confirmed the granting replica is not still reaping — that confirmation is the whole point of the guard, so prefer the narrow forms: bd reclaim --any-replica --id <id> for one issue, or bd unclaim --force <id>; the bare global form reverts every foreign stale lease, live peers included. bd reclaim names what it declined on stderr — one summary line per run, with bd -v expanding it to the first 20 leases individually.

Planned Features

The following operation has infrastructure support but is not yet exposed as a command:

  • bd federation push <peer> / bd federation pull <peer> - single-direction sync with one peer. bd federation sync already covers the bidirectional case.

Troubleshooting

"requires direct database access"

Federation commands require the Dolt backend with direct database access. Ensure you have the Dolt backend configured for federation operations.

"peer already exists"

A peer with that name is already configured. Use a different name or check existing peers with bd federation list-peers.

Invalid endpoint format

Ensure your endpoint matches one of the supported formats above. The scheme must be one of: dolthub://, gs://, s3://, file://, https://, ssh://, or git SSH format (git@host:path).

General health check

bd doctor --deep

Reference

  • Configuration: See Configuration for all federation settings
  • Source: cmd/bd/federation.go
  • Storage interfaces: internal/storage/versioned.go
  • Dolt implementation: internal/storage/dolt/store.go