beads/docs/multi-agent/federation.md
Steve Yegge d1e725d9f3 fix(reclaim): correct the heartbeat re-home invariant's scope in the new docs (wy-sp2l4)
Adversarial review of the previous commit refuted three of its claims:

F1 (HIGH) "a heartbeat never re-homes a foreign lease" is false on a
reachable path INSIDE the function the comment annotates: the ga-v2k49
disambiguation fallback fires when the lease row spells the holder
differently, re-arms through UpsertLeaseInTx, and its ON DUPLICATE KEY
UPDATE stamps the local node over the foreign one (reviewer confirmed it
against embedded dolt). The invariant is a property of the fast path, not
of the lease row; say so, and name the fallback.

F2 (MED) RestoreLeaseOnImportInTx takes granted_node from the snapshot
when the local row is EXPIRED, so an import can relabel this node's own
stale lease with a remote name — a third stranding state the enumeration
missed.

F3 (MED) the prose named only bare --any-replica, which reverts EVERY
foreign stale lease including live peers'. All three sites (summary line,
federation.md, reclaim --help) now name the narrow levers first:
--any-replica --id <id>, or bd unclaim --force <id>.

Also F5/F8 (LOW): -v shows the first foreignSkipDetailRows, not "the
individual leases" — the summary and both doc sites now say so.
2026-08-13 04:30:30 -07:00

326 lines
12 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
title: Federation Setup Guide
description: Configure peer-to-peer sync of beads databases across workspaces with Dolt remotes, sovereignty tiers, and topologies
---
Federation enables peer-to-peer synchronization of beads databases between
multiple workspaces using Dolt remotes. Each workspace maintains its own database
while sharing work items with configured peers.
## Overview
Federation uses Dolt's distributed version control capabilities to sync issue
data between independent teams or locations. Key benefits:
- **Peer-to-peer**: No central server required; each town is autonomous
- **Database-native versioning**: Built on Dolt's version control, not file exports
- **Flexible infrastructure**: Works with DoltHub, S3, GCS, local paths, or SSH
- **Data sovereignty**: Configurable tiers for compliance (GDPR, regional laws)
<Note>
Just want two machines sharing one database through a bucket you own? The
[Bucket Federation Quickstart](/multi-agent/bucket-federation) walks that path
end to end — bucket, seed push, second replica, sync cadence — with measured
timings and the failure modes.
</Note>
## Prerequisites
1. **Dolt backend**: Federation requires the Dolt storage backend (the only supported backend)
## Configuration
### Enable Federation-Compatible Sync
Edit `.beads/config.yaml` or `~/.config/bd/config.yaml`:
```yaml
federation:
remote: dolthub://myorg/beads # Primary remote (optional)
sovereignty: T2 # Data sovereignty tier
```
Or via environment variables:
```bash
export BD_FEDERATION_REMOTE="dolthub://myorg/beads"
export BD_FEDERATION_SOVEREIGNTY="T2"
```
### Data Sovereignty Tiers
| Tier | Description | Use Case |
|------|-------------|----------|
| T1 | No restrictions | Public data |
| T2 | Organization-level | Regional/company compliance |
| T3 | Pseudonymous | Identifiers removed |
| T4 | Anonymous | Maximum privacy |
## Adding Federation Peers
Use `bd federation add-peer` to register remote peers:
```bash
bd federation add-peer <name> <endpoint>
```
### Peer Name Rules
- Must start with a letter
- Alphanumeric, dash, and underscore only
- Maximum 64 characters
### Supported Endpoint Formats
| Format | Example | Description |
|--------|---------|-------------|
| DoltHub | `dolthub://org/repo` | DoltHub hosted repository |
| Google Cloud | `gs://bucket/path` | Google Cloud Storage |
| Amazon S3 | `s3://bucket/path` | Amazon S3 |
| Local | `file:///path/to/backup` | Local filesystem |
| HTTPS | `https://host/path` | HTTPS remote |
| SSH | `ssh://host/path` | SSH remote |
| Git SSH | `git@host:path` | Git SSH shorthand |
### Examples
```bash
# Add a staging environment on DoltHub
bd federation add-peer staging dolthub://myorg/staging-beads
# Add a cloud backup
bd federation add-peer backup gs://mybucket/beads-backup
bd federation add-peer backup-s3 s3://mybucket/beads-backup
# Add a local backup
bd federation add-peer local file:///home/user/beads-backup
# Add a partner organization
bd federation add-peer partner-town dolthub://partner-org/beads
```
### Credentials
Peers configured with `--user` (and optionally `--password`, otherwise
prompted interactively) store SQL credentials AES-256 encrypted, locally.
Stored credentials are used automatically during sync:
```bash
bd federation add-peer town-gamma 192.168.1.100:3306/beads --user sync-bot
```
### JSON Output
For scripting, use the `--json` flag:
```bash
bd --json federation add-peer staging dolthub://myorg/staging-beads
# {"added":"staging","url":"dolthub://myorg/staging-beads","has_auth":false,"sovereignty":""}
```
### Verify Configuration
List configured peers:
```bash
bd federation list-peers
```
## Syncing with Peers
Use `bd federation sync` to pull from and push to peer towns, and
`bd federation status` to check sync state without transferring data.
```bash
# Sync with all peers
bd federation sync
# Sync with a specific peer
bd federation sync --peer town-beta
# Handle conflicts
bd federation sync --strategy theirs # or 'ours'
# Check status (ahead/behind, reachability, conflicts)
bd federation status
bd federation status --peer town-beta
```
Without `--strategy`, a sync that hits merge conflicts pauses and reports the
conflicting tables for manual resolution instead of auto-resolving.
### Topologies
| Pattern | Description | Use Case |
|---------|-------------|----------|
| Hub-spoke | Central hub, satellites sync to hub | Team with central coordination |
| Mesh | All peers sync with each other | Decentralized collaboration |
| Hierarchical | Tree of hubs | Multi-team organizations |
## Architecture Notes
### How It Works
1. Each workspace has its own Dolt database
2. `add-peer` registers a Dolt remote (similar to `git remote add`)
3. `bd federation sync` pushes and pulls commits between peers
4. Conflict resolution follows the configured strategy
When run against a Dolt SQL server, federation uses two ports: MySQL (3306)
for multi-writer SQL access, and remotesapi (8080) for peer-to-peer
push/pull:
```
┌─────────────────┐ ┌─────────────────┐
│ Workspace A │◄───────►│ Workspace B │
│ dolt sql-server│ sync │ dolt sql-server│
│ :3306 (sql) │ │ :3306 (sql) │
│ :8080 (remote) │ │ :8080 (remote) │
└─────────────────┘ └─────────────────┘
```
### Multi-Repo Support
Issues track their `SourceSystem` to identify which federated system created
them. This enables proper attribution and trust chains across organizations.
### Connectivity
Remote connectivity is validated on first push/pull operation, not when adding
the peer. This allows configuring remotes before infrastructure is ready.
### Leases are per-replica
A claim lease (`bd ready --claim` + `bd heartbeat`, reaped by `bd reclaim`) is
only meaningful on the replica that granted it. The `leases` table is
clone-local and never replicates; what crosses the bridge is the claim's
*visibility* — `status`/`assignee` on the issue row — and that is stale on
every other replica by up to one sync interval.
Two rules follow, and a federated deployment owes both:
1. **Grace window > sync interval, and lease TTL > sync interval.** A TTL or
`bd reclaim --older-than` grace shorter than the cadence at which replicas
exchange state is meaningless across the bridge: the remote view is a full
interval old by construction, so a reaper over there would be judging
liveness from data older than the lease itself. `bd reclaim` defaults its
grace to 2× the lease TTL; raise the TTL (or the grace) above your sync
interval, never shrink the interval to fit them.
2. **Reclaim belongs to the granting replica.** Each lease records the replica
that granted it, and `bd reclaim` skips a lease granted elsewhere, naming
it on stderr. Reap dead workers on the machine that hired them.
The guard is **opt-in**: it arms only where you name this replica.
```bash
export BEADS_NODE_ID=mini # per-machine; or `bd config set node_id mini`
```
Two rules about what to name, both load-bearing:
- **`node_id` names the STORE, not the host.** One value per beads *database*.
Hosts that are clients of the *same* dolt sql-server (`BEADS_DOLT_SERVER_HOST`,
a systemd/Docker server, Hosted Dolt, a VPS) are **one replica** no matter how
many machines they are — give them all the same value, or leave it unset. Give
them distinct ids and you rebuild the very fail-closed regression described
below: a supervisor would match no worker's lease and reclaim 0 forever. Name a
replica only where there is a real sync interval between it and the others.
- **`node_id` is per-machine, so it must never be committed.** The project
`.beads/config.yaml` is a git-**tracked** file. A `node_id` committed there
propagates one machine's identity to every clone that pulls it, and then every
comparison matches: the guard is fully *armed* and fully *inert*, and `laptop`
reaps `mini`'s leases exactly as if they were local — the precise hazard this
feature exists to close, now happening while you believe you are protected.
That is worse than not setting it at all. `bd config set node_id` therefore
writes the **user-global** `~/.config/bd/config.yaml`, alongside the other
per-machine state (`sync-state.json`, `push-state.json`, `redirect`). Use the
env var or that command; never hand-add `node_id` to `.beads/config.yaml`.
There is deliberately no hostname fallback. The hostname answers the wrong
question — it names the client *process's* machine, not the store — and
guessing gets it wrong in the topologies that most need automated reclaim:
with a shared or remote dolt sql-server (`BEADS_DOLT_SERVER_HOST`, Hosted
Dolt, a VPS) many hosts are clients of ONE store with no sync interval
between them, so a per-hostname identity would stop a supervisor reaping any
worker's lease at all; in a container the hostname is a per-run container ID;
on macOS the transient hostname follows the network. Each of those would
strand work on a deployment with no federation at all — a worse failure than
the one this guard prevents.
So an unset identity degrades to the old behavior (every lease treated as
local) rather than failing closed: an upgrade, and any single-store
deployment, can never strand a lease the reaper could previously recover.
Leases granted before this feature landed likewise carry no replica and stay
reclaimable until a heartbeat re-stamps them with a configured node.
`bd reclaim --any-replica` disarms the guard. It is for a replica that is
permanently gone (or a node that was renamed and now sees its own old leases
as foreign) — not a normal setting, since only the granting machine has a
first-hand view of whether the holder is alive.
**A heartbeat proves the holder is alive; it does not move the lease.** An
ordinary heartbeat only *backfills* the granting replica when it is still empty
— it does not overwrite a row that positively names one. So a lease normally
keeps its granting replica for life, and states like these strand one where a
local heartbeat is keeping it alive:
- a replica that was **renamed** (`mini` → `mini2`) heartbeats its own leases,
which now read foreign forever;
- a foreign lease that arrived through the JSONL interchange, whose holder name
also exists locally, gets heartbeated here but stays labelled with the remote
node;
- an import that lands on an already-**expired** local lease row takes the whole
row from the snapshot, granting replica included — so this node's own stale
lease can come back labelled with a remote one.
(The one path that moves a lease the other way is a heartbeat whose holder is a
different *spelling* of the lease row's holder: it re-arms through the upsert,
which stamps this node.)
Recover a stranded lease with `bd reclaim --any-replica`, once you have
confirmed the granting replica is not still reaping — that confirmation is the
whole point of the guard, so prefer the narrow forms:
`bd reclaim --any-replica --id <id>` for one issue, or `bd unclaim --force
<id>`; the bare global form reverts *every* foreign stale lease, live peers
included. `bd reclaim` names what it declined on stderr — one summary line per
run, with `bd -v` expanding it to the first 20 leases individually.
## Planned Features
The following operation has infrastructure support but is not yet exposed as
a command:
- `bd federation push <peer>` / `bd federation pull <peer>` - single-direction
sync with one peer. `bd federation sync` already covers the bidirectional
case.
## Troubleshooting
### "requires direct database access"
Federation commands require the Dolt backend with direct database access. Ensure
you have the Dolt backend configured for federation operations.
### "peer already exists"
A peer with that name is already configured. Use a different name or check
existing peers with `bd federation list-peers`.
### Invalid endpoint format
Ensure your endpoint matches one of the supported formats above. The scheme
must be one of: `dolthub://`, `gs://`, `s3://`, `file://`, `https://`, `ssh://`,
or git SSH format (`git@host:path`).
### General health check
```bash
bd doctor --deep
```
## Reference
- Configuration: See [Configuration](/reference/configuration) for all federation settings
- Source: `cmd/bd/federation.go`
- Storage interfaces: `internal/storage/versioned.go`
- Dolt implementation: `internal/storage/dolt/store.go`