beads/PROPOSAL-pluggable-storage-backends.md

789 lines
57 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# PROPOSAL: Pluggable Storage Backends
**Status:** Historical — partially implemented, then superseded on 2026-07-16
**Date:** 2026-07-02
**Informed by:** the bts-rs Rust spike (a from-scratch Rust reimplementation
of the gc-contract surface that validated a backend-agnostic storage seam at 299/299 byte-parity
across two radically different backends), a five-dimension recon of this repo's storage
architecture, and a two-reviewer adversarial verification pass. All file:line citations verified
against `worktree-beads-new`.
> **Superseded scope:** This proposal is retained as the design record for the
> storage interface, shared issue core, SQLite adapter, and differential
> conformance work that grew from it. Those pieces remain. The direct
> PostgreSQL and MySQL adapters were rolled back before entering a tagged
> release because supporting additional general-purpose server databases adds
> substantial dialect, credential, schema-lifecycle, migration, CI, and
> operational complexity. Our goal is to keep Beads as simple as possible and
> consume as few resources as possible. The supported paths are now embedded
> Dolt, Dolt server, and SQLite; Dolt server's use of the MySQL wire protocol is
> unchanged. See [Storage Backends](docs/architecture/storage-backends.md) for
> the current design.
Everything below this notice is retained as historical design material, not as
current backend support or migration guidance.
---
## 1. End state
- `bd init --backend=dolt|sqlite|postgres` — the user picks the engine at init. The workspace
locator records the choice; opening consults the locator, never the environment.
- **One core command set** (the gc-contract 16 plus the portable CORE inventory, ~67 commands)
behaves identically on every backend, proven by a differential conformance harness — not by
code review.
- **Storage-specific commands are capability-gated addons.** `bd history`, `bd dolt push/pull`,
`bd vc`, `bd branch`, `bd diff`, `bd federation`, `bd backup`, `bd sql` are live only when the
opened store advertises the matching capability; otherwise they are deterministic stubs that
explain what backend provides them (Decision D2, resolved: stubs, not hidden — see §6).
Pick Dolt → you get history and remotes. Pick SQLite → you get a fast, zero-dep, pure-Go
single-writer store and those commands stub out.
- Dolt remains the default and richest backend. **Back-compat guarantee:** a workspace created
before this migration keeps opening exactly as it does today — enforced by an explicit
legacy-locator rule (§4.3), not by hope.
- Adoption path for existing workspaces: `bd export` → re-init on the new backend → `bd import`,
with documented fidelity limits (§5, Phase 4 gate). In-place cross-backend data migration is
a separate initiative.
## 2. What the Rust spike proved (the transferable payload)
Evidence: `bts-rs/docs/` (`SEAM_VALIDATION_FINDINGS.md`, `CONFORMANCE_GAPS.md`,
`RED_TEAM_FINDINGS.md`, `HISTORY_ON_YUGA_DESIGN.md`, `PERF_PG_VS_YB.md`, `REMOTE_DEPLOYMENT.md`).
### 2.1 The seam shape
Six small core traits (33 data methods) + **optional capability traits** reached via accessors
defaulting to "absent". Two backends (Postgres, file-backed memory) hit byte-parity with real bd
on 299/299 scenarios **with zero command-handler changes** — handlers never name a backend.
**Go gets the spike's hardest problem for free.** Rust's non-object-safe async traits forced a
399-line delegation enum; in Go, `storage.Storage` is already an interface and `Open()` returns
the interface value. Budget the port on semantics and conformance, not plumbing.
**Seam-size caveat (from review):** the spike's 33-method core is NOT evidence that beads' much
larger core (§4.1: ~107+24 methods after the fold) is cheap to implement per-backend. The spike's
"~1–10k LOC per adapter" anchor was measured against 33 methods. Phase 0 therefore includes a
core-shrink review: which core methods are truly primitive vs derivable-above-the-seam vs
capability material. Sizing in §8 assumes the core does NOT shrink (worst case).
### 2.2 The four seam-leak rules (each was a real bug in the spike)
1. **Construction is ON the seam.** `Open(locator, identity)` / `Init(backend, ...)` are seam
methods; otherwise every entry point names concrete backends.
2. **Identity is ON the seam** — actor identity is construction-time config in the seam package.
3. **The locator is backend-neutral and self-describing** (`{backend, workspace_id}`); backend
is read FROM the locator at open; `--backend`/env applies **only at init**.
4. **Capabilities OPTIMIZE core behavior; they never GATE it.** For every capability use inside
a core command there must be a core-interface fallback, OR the whole command is
capability-gated. (The spike's only runtime divergence violated this.)
### 2.3 Transactional-integrity rules that apply to beads' CORE (not just capabilities)
The spike's red team hit this bug class three times; in beads it lives on the core surface:
- **Same-tx side effects:** beads writes the events table inside the mutation transaction
(`issueops/create.go:601`, `labels.go:151`, `helpers.go:147`; read via core
`GetAllEventsSince`, `storage.go:84`, and `bd audit`). Any new backend must do the same —
events emitted outside the mutation tx strand or duplicate audit rows on crash.
- **Denormalized `is_blocked` is the most conformance-dangerous core semantic.** It is
maintained across 10+ issueops write paths (dependencies/close/reopen/delete/promote/…) and
read by ready queries (`sqlbuild/ready.go:96`). Delete/purge must recompute surviving
neighbours. A fresh backend reimplements this propagation on every mutation; the harness
must carry required transition fixtures (close-blocker→ready, purge-orphan re-seed).
### 2.4 The conformance oracle
- **Differential, not static:** scenarios as real processes in throwaway workspaces; volatile
normalization (`<TS>`/`<UUID>`/actor); per-step exit + stderr + JSON-aware stdout diff. A
static canonicalized corpus "pins a formatter, not a system".
- **Multiset array diffing by default**, with a per-scenario `ordered` flag valid only where
ordering is implementation-independent.
- **Property tests** for non-byte-diffable paths (minted IDs, `--graph`).
- **Concurrency proofs as capability gates:** the 24-process `ready --claim` drain applies only
to backends advertising multi-writer.
- **Epistemic honesty — what green does NOT prove (§7 risk table):** hook-firing semantics
(post-commit, rollback-suppressed — invisible to stdout/stderr diffing), equal-priority and
wall-clock-hybrid ordering (`ready`'s 48h recency rule), and anything outside the in_scope
predicate. Each gate below pins its in_scope coverage explicitly so "proven identical" is
never claimed beyond what ran.
- Operational: unique candidate-binary path per run (`cp` over an exec-mapped binary fails
silently and scores a stale binary).
### 2.5 History without Dolt (later, not now)
`bd history`'s minimal substrate is a per-issue, ordered, never-pruned field-delta changelog
written in the same tx as each mutation + `HistoryViewer{History, AsOf, Diff}` as a backward
fold. Dolt provides all three natively (`dolt_history_*`, `AS OF`, `dolt_diff`). So "pick Dolt →
get history" can later relax to "any backend with the Changelog capability" without
re-architecting.
### 2.6 Backend guidance from measured perf
Postgres bounded ops flat to 100k (list ~2ms). Remote cost = round-trips × RTT **plus ~6 RTT of
connection setup per command in the shell-out model** (~300ms pure overhead at 50ms RTT) —
mitigations: co-location, local PgBouncer ≥1.21 transaction mode, daemon endgame. Collapse write
paths to 1–2 statements; `FOR UPDATE SKIP LOCKED` IS the claim guarantee. YugabyteDB ≈ 2–3×
latency / 10× write cost — multi-region only. An embedded pure-Go store is the cheapest second
backend and the one that validates the seam.
## 3. Where beads is today (recon summary, review-corrected)
### 3.1 The good news — the seam is already 80% drawn
- `storage.DoltStorage` (`internal/storage/storage.go:200-213`) composes core `Storage` (62
methods) + 11 sub-interfaces (144 total; + `Transaction` 24). The Dolt-shaped surface is
already isolated in **5 named sub-interfaces** — VersionControl (16), HistoryViewer (3),
RemoteStore (12), SyncStore (2), FederationStore (4) = 37 methods, ~87 genuine call sites in
non-test cmd/bd. The other **6 sub-interfaces are backend-neutral** (BulkIssueStore,
DependencyQueryStore, AnnotationStore, ConfigMetadataStore, CompactionStore,
AdvancedQueryStore — ~45 methods, ~155 cmd/bd call sites serving CORE commands like
`ready --claim`, `dep`, `comment`) and fold INTO the core seam (§4.1).
- **The capability-gating pattern already ships:** 10 optional interfaces
(`storage.go:217-287`), `UnwrapStore` + type-assert at 22 cmd/bd sites, graceful errors
(`sql.go:54-57`).
- **The public API is not frozen to the god interface:** root `beads.go:20` aliases
`Storage = beads.Storage` (= `storage.Storage`); RemoteStore/SyncStore are exported as
separately assertable interfaces. The retype does not break the public type surface — but see
§4.4 for the public `Open*` constructors and the extension-compat note.
- **Real prior art for the oracle:** `tests/regression` (built for the SQLite→Dolt migration;
found 70+ bugs) contributes workspace isolation, binary management, container TestMain, and a
normalizer that already encodes cross-backend representational divergences
(`regression_test.go:587` local-vs-UTC midnight; `:647` metadata `{}` vs absent). Its
execution core does NOT transfer (§5 Phase 1).
- Integrations are portable for free (all 6 trackers bind to narrow `storage.Storage`); the
tracker engine already demonstrates capability-with-fallback (`tracker/engine.go:695-717`).
- Dialect exposure is concentrated: ~90% of SQL flows through `issueops`/`sqlbuild`/`domain/db`.
Dolt-specific SQL inside issueops sits in **7 files**: `as_of.go`, `commit_pending.go`,
`diff.go`, `federation.go`, `blocked_merge.go`, `history.go` (`dolt_history_issues`), and
`blocked_consistency.go` (`dolt_status` guard — backend-internal, moves with the backend).
The first six map 1:1 onto gateable capabilities.
### 3.2 The hazards — what actually blocks a second backend
- **H1. Two competing seams, mid-flight — and BOTH are alive.** The `DoltStorage` path covers
100% of commands. The **uow/domain layer** covers 13 commands via `*_proxied_server.go`
duals and is an ACTIVE workstream (one command per PR since 2026-05-10, latest merges
2026-06-24; the init gate at `init.go:135-140` is deliberate staging per PR #4488, and the
CI lane is dark only until the gate lifts). It is Dolt-flavored at its Tx layer
(`uow/doltserver_tx.go:28` = `CALL DOLT_COMMIT`), duplicates issueops (kept honest by
Seam-A parity tests), and is already feature-diverging from the embedded path (`--offset`
is proxied-only). **Proxied-server mode exists ONLY on the uow path** — `newDoltStore`
refuses it (`store_factory.go:48-53`). D1 must resolve the seam relationship AND proxied's
fate, with the uow workstream owner participating (§6).
- **H2. Dolt lifecycle is fused into every command's write path, with real protocol structure.**
Not just "auto-commit in PostRun": `transactHonoringAutoCommit` (`dolt_autocommit.go:28-47`)
**blanks the commit message** to suppress mid-tx commits depending on mode; a non-blank
message + success sets `commandDidExplicitDoltCommit` which suppresses PostRun auto-commit
(`main.go:1223-1227`); the tips system defers metadata writes and PostRun issues a SECOND
commit for them (`tips.go:154-183`, `main.go:1233-1257`); auto-export/backup freshness reads
`GetCurrentCommit` AFTER the commit; auto-push gates and error semantics differ from
auto-commit; the autocommit-mode default itself is chosen by the CLI from `usesSQLServer()`
(`main.go:1099-1106`). `commitPendingIfEmbedded`/`transactHonoringAutoCommit` have **21 call
sites across 18 files**. This is a protocol, not a hook — it gets a specified contract (§4.2)
and its own phase (2b).
- **H3. Global store + FIVE construction paths.** `store storage.DoltStorage`
(`main.go:45`; **159 non-test refs / 61 files**, 15 more test files). Paths: PreRun choke
point; lazy `direct_mode.go`; read-only; ~13 files opening routed/extra stores; **and the
public API** — root `beads.go:63-73` `Open/OpenFromConfig` hard-code `dolt.New*`
(+ `beads_nocgo.go` `OpenBestAvailable`), used by external Go extensions. The registry must
own all five. The `HookFiringStore` decorator EMBEDS DoltStorage
(`hook_decorator.go:28-29`) and is applied only on the main path today — the registry must
own decorator application or lazy/read-only/routed opens keep silently skipping hooks.
- **H4. The migration stream is not portable.** 53 tracked + 10 dolt-local `*.up.sql`, MySQL
dialect; migrations 0019/0028/0040/0041 execute `dolt_ignore`/`CALL DOLT_COMMIT` inline. New
backends need per-backend migration sources (the `schema` runner's two-stream design is the
extension point).
- **H5. Core-signature Dolt leaks.** `RunInTransaction(ctx, commitMsg, fn)` — where the message
is load-bearing suppression semantics per H2, NOT advisory; `GetCurrentCommit` as freshness
token (`export_auto.go:85`, `backup_auto.go:127`); `ApplyCompaction(commitHash)`; `bd edit`'s
raw `*sql.DB` keepalive (`edit.go:150`); "dolt-ignored" LocalMetadata contract.
- **H6. Command metadata in name-keyed string lists.** `noDbCommands` (27), `readOnlyCommands`
(14), dolt-subcommand patch lists (`main.go:770-808,122-137`).
- **H7. bd doctor** — 37/45 files Dolt-tinged, raw SQL on `dolt_remotes`/`dolt_status`.
- **H8. Mode-divergent semantics are unformalized.** The draft claimed this "includes a LIVE
BUG": ~~with server-mode autocommit OFF, `GetCurrentCommit` never advances, so
auto-export/backup freshness checks silently stop firing in server mode.~~
**CORRECTED AFTER VERIFICATION: REFUTED.** The claimed "live bug" does not exist. The
server-mode store (`internal/storage/dolt.DoltStore`) commits to HEAD **per write method,
unconditionally** — independent of `--dolt-auto-commit` and of `maybeAutoCommit`
(`dolt/issues.go:49` CreateIssue → `CALL DOLT_COMMIT`; UpdateIssue/CloseIssue/DeleteIssue
likewise; `RunInTransaction` → `versioncontrolops.StageAndCommit`). So in non-proxied server
mode HEAD advances on every persistent write, `DOLT_HASHOF('HEAD')` changes, and the
auto-backup freshness check DOES fire. Separately, auto-export never even reaches the
freshness check in server mode — it early-returns (`export_auto.go:38-41`, `if serverMode {
return nil }`). The `maybeAutoCommit` machinery governs the EMBEDDED working-set-persistence
flow, not the server store's per-method commits; the H8 draft was written against an
inaccurate model of the server store. The rest of H8 stands: auto-commit ON embedded / OFF
server; single-writer flock embedded-only (`store_factory.go:66-85`); server DoltStore has
real streaming iterators (`dolt/iter_issues.go:48`) while embedded ships `NewSliceIter`
stubs; `bd sql`/`bd admin` work in server mode but error in embedded (`sql.go:45`,
`admin.go:17`) — and that embedded error string is gc contract class 3.
- **H9. Auto-import is a silent data-mutation path on the core surface** (missed in the first
draft; review-added). `maybeAutoImportJSONL` runs in PreRun for every command
(`main.go:1163-1164`), takes `storage.DoltStorage`, auto-imports JSONL into empty databases,
and has a documented data-clobber history (`auto_import_upgrade_unit_test.go:74-87`). Its
"auto-importing … into empty database" strings are gc error class 4 (the write-loss guard).
A second backend that is empty-on-fresh-open is exactly the trigger condition; if its strings
differ, gc's guard goes blind.
- **H10. Env precedence can override the locator at OPEN.** `BEADS_DOLT_SERVER_MODE=1` is
precedence #1 over metadata.json (`configfile.go:232-243`); shared-server env/config
overrides pinned `dolt_mode` with a Notice (`main.go:247`); `--global` hard-requires
shared-server (`main.go:1071-1076`). Parsing is scattered (configfile, doltserver, init,
bootstrap). The spike's "locator wins at open" rule must extend to topology, with an explicit
ownership and inertness rule (§4.3).
### 3.3 The command surface (census, corrected)
~271 cobra commands (~111 top-level), all registered unconditionally **in package `init()` —
before any workspace resolution** (this kills naive hide-at-registration; see D2). Classified:
CORE ≈ 67 (incl. the gc-contract 16), STORAGE-SPECIFIC = 11 (+3 mixed: `bd gc`, `bd migrate`,
`bd restore`'s history fallback), INTEGRATION = 6 families, INFRA ≈ 27. **12 of the gc-16 have
dual (store+uow) implementations; `purge`, `count`, `version`, `stats` do not** — and `purge`
is precisely where the spike's red team found re-seeding bugs, so it is a required Phase 1
scenario, not a "covered elsewhere" case.
## 4. Target architecture — the clean-room object model, and the bridge to it
Two things are described here and it matters to keep them distinct: **§4.0 the destination**
(the clean-room object model we would build greenfield, which the migration accretes toward)
and **§4.5 the bridge** (the flat-interface waypoint the mechanical phases pass through).
The destination is what new code is written AGAINST from Phase 4 onward; the bridge is
scaffolding that de-risks the retype and is thinned afterward (Phase 7).
### 4.0 The clean-room object model (the destination)
```
┌────────────────────────────────────────────────────────────────────────┐
│ cmd/bd — commands, formatting, streams, contract strings │
│ command registry: declared metadata {needsStore, readOnly, │
│ requires: Capability} — replaces the name-keyed string lists (H6) │
└──────────────────────────────┬─────────────────────────────────────────┘
│ consumes ONLY:
┌──────────────────────────────▼─────────────────────────────────────────┐
│ package store — THE SEAM (small, frozen, documented) │
│ Driver / Registry / Locator / Identity (construction, rules 1–3) │
│ Store / Tx + ~7 domain interfaces (core; Tx-first) │
│ Capability interfaces + CapabilitySet (optional surface, rule 4)│
│ Error taxonomy + retry classes (typed, rendering-free) │
│ storetest.RunContract(t, driver) (the compliance suite) │
└──────┬───────────────┬───────────────┬───────────────┬─────────────────┘
│ │ │ │
backend/dolt backend/postgres backend/sqlite backend/fake
(topologies: (multi-writer; (pure-Go; (in-memory/file;
embedded | no history v1) single-writer; unit-test keystone)
server | fills nocgo)
proxied
INTERNAL)
```
The core, sketched (signatures indicative, not final):
```go
// construction — the spike's leak rules 1–3, verbatim
type Driver interface {
Open(ctx context.Context, loc Locator, id Identity) (Store, error)
Init(ctx context.Context, opts InitOptions) (Locator, error)
}
type Locator struct{ Backend string; Raw json.RawMessage } // self-describing; env only at Init
type Identity struct{ Actor, Email string } // on the seam; injectable for tests
type Store interface {
// Run is the PRIMARY seam op: one retryable unit of work. The seam owns
// phase-aware retry (Serialization -> bounded auto-replay of fn;
// CommitIndeterminate -> surfaced, never auto-retried; fn must be replayable,
// no external I/O while the Tx is open). Semantics lifted from uow/run_in_tx.go.
Run(ctx context.Context, op OpInfo, fn func(Tx) error) error
Begin(ctx context.Context, op OpInfo) (Tx, error) // low-level escape hatch
Capabilities() CapabilitySet // per-INSTANCE (topology-aware)
Limits() Limits // {MaxAtomicTxBytes, MaxValueBytes, MaxTxDuration,
// ConsistentScanBudget} — CLI pre-flights + chunks instead
// of discovering backend ceilings as commit failures
ChangeToken(ctx context.Context) (string, error) // equality-only freshness cursor
Close() error
}
// OpInfo + CommandScope are where the H2 commit protocol collapses to. NOTE
// (red-team blocker): commit messages are OUTCOME-derived in bd (composed after
// close/claim results — close_proxied_server.go:242-255, uow.RunInTxMsg), so the
// description travels on Commit, NOT on Begin. A CommandScope above Tx owns the
// one-version-commit-per-command rule (N txs -> ONE dolt commit), the tips
// deferred-metadata second commit, and PostWritePush ordering.
type OpInfo struct{ Command, Actor string; ReadOnly bool }
// the transaction is the first-class object — one per command/logical op
type Tx interface {
Issues() Issues // CRUD, search, ready, claims; wisps as a TIER param, not a
Deps() Dependencies // parallel ×2 method family
Labels() Labels
Comments() Comments
Events() EventReader // reads; event WRITES happen implicitly inside mutations, same tx
Config() Config
Commit(ctx context.Context, desc string) error // desc outcome-derived;
// Dolt: DOLT_COMMIT(desc); PG: COMMIT; reads: Rollback, never Commit
Rollback(ctx context.Context) error
}
// capabilities: extend core behavior, never gate it (rule 4)
type CapabilitySet struct {
History, VersionControl, RemoteSync, Backup, RawSQL bool
// MultiWriter bool was too coarse (red-team, 2 lenses independently):
Concurrency struct {
ConcurrentReaders, ConcurrentWritersSafe bool
WriterExclusion string // enforced-lock | lease | none
ContentionModel string // row | workspace-CAS
}
}
func History(s Store) (HistoryViewer, bool)
type HistoryViewer interface{ /* History, AsOf, Diff */ }
// + VersionControl, RemoteSync, Backup, RawSQL, Maintenance(GC/Flatten/Compact)
// errors: typed below the seam, rendered above it
var ErrNotFound, ErrClaimConflict error
type ErrUnsupported struct{ Op, Backend string }
type RetryClass int // Transient | Serialization | CommitIndeterminate | Permanent
```
**Every commitment traces to a scar** in the current system or the spike:
| Commitment | Scar it heals |
|---|---|
| Tx-first seam (`Begin` → domain ifaces → `Commit`) | `DoltStore`'s per-method commits = the stale-snapshot data-loss class (#3822) that forced the May pivot; command-scoped short tx is the proven concurrency unit. This is `uow.UnitOfWork` promoted to the seam. |
| ~7 small domain interfaces, no god interface | the 805-line panic wall (PR #3792); "can't easily mock DoltStorage"; 168 methods before one command works |
| `OpInfo` on `Begin` | H2: commit protocol currently lives in CLI globals (message-blanking, `commandDidExplicitDoltCommit`, tips double-commit) |
| `ChangeToken()` on core | backend-neutral freshness: `GetCurrentCommit`/`DOLT_HASHOF` is Dolt-specific, so export/backup freshness must key on a neutral opaque token any backend can supply (SQLite/Postgres have no Dolt commit hash). (NB: the earlier "H8 live bug" rationale is withdrawn — see H8; ChangeToken stands on backend-neutrality alone.) |
| per-instance `CapabilitySet`, extend-never-gate | `bd sql` works on Dolt-server, stubs on Dolt-embedded — capability varies by topology; the spike's only runtime divergence was a capability gating core behavior |
| topologies INSIDE `backend/dolt` | embedded/server/proxied are three connections to ONE engine; CLI knowledge of them produced 38 scattered mode checks |
| events written inside mutations, same tx | the spike's red team hit emit-outside-tx three times; the use-cases already do it right — make it structural + contract-tested |
| `backend/fake` + `storetest.RunContract` | nothing tests against the interface today; every suite needs a Dolt container |
| wisps as a tier param | today every use-case method is duplicated ×2 (41 where ~25 would do); the `--offset` union-paging divergence is a symptom |
| slices, not streaming iterators, in v1 | all 10 `Iter*` methods are slice stubs in the embedded backend anyway; streaming returns later as a measured need |
One-sentence summary: **the clean-room seam is Dustin's `UnitOfWork` widened to the full
domain surface, wearing bts-rs's construction/capability/error discipline.**
### 4.1 Bridge-seam arithmetic (the flat waypoint; review-corrected)
Core = today's `Storage` (62) **+ the six backend-neutral sub-interfaces (~45)** = ~107 methods,
+ `Transaction` (24), all mandatory for every backend. The ~155 cmd/bd call sites of the six
folded interfaces need **zero conversion**. The ~87 call sites of the 5 Dolt-shaped interfaces
convert to capability asserts. `DoltStorage` is deleted; the Dolt backends keep a `var _`
compile assertion for core + all capabilities. Phase 0's core-shrink review may pull some of
the ~107 above the seam (derivable) — treat any shrink as upside, not plan.
### 4.2 The WriteLifecycle contract (replaces the naive "PostWrite hook")
A specified seam protocol, designed in Phase 2b BEFORE code moves:
- `WriteSummary{didWrite, explicitCommitOccurred, readOnlyCommand, messageParts, deferredMeta}`
carried per command (absorbs today's `commandDidWrite`/`commandDidExplicitDoltCommit`
globals and the tips deferred-write protocol).
- Two seam calls bracketing the CLI's own export/backup steps: `PostWriteCommit(summary)`
(Dolt: auto-commit incl. the second tips commit; others: no-op) then — after the CLI's
auto-export/auto-backup, whose freshness check consumes a **neutral ChangeToken()** (core,
with the capability-shortcut-plus-fallback shape; motivated by backend-neutrality, since
`DOLT_HASHOF` is Dolt-specific — NOT by the withdrawn H8 "live bug", see H8) —
`PostWritePush(summary)` (Dolt: auto-push with its swallowed-error semantics; others: no-op).
- In-transaction suppression (`transactHonoringAutoCommit`'s blank-message trick) becomes an
explicit seam behavior: the backend decides mid-tx checkpointing from the summary + its own
mode; the message parameter on `RunInTransaction` stays but is re-specified as "operation
description; versioned backends MAY use it as a commit message" (D3).
- Autocommit-mode POLICY (today chosen by the CLI from `usesSQLServer()`) moves into the Dolt
backend's topology config — the CLI no longer knows modes exist.
### 4.3 Locator, legacy rule, and env precedence
- Locator in `.beads/metadata.json`: `{backend: "dolt"|"sqlite"|"postgres", ...}`. `dolt_mode`
becomes Dolt-internal topology config.
- **Legacy-value rule (review blocker):** the existing `backend` field is UNTRUSTED legacy —
workspaces in the wild may carry `backend:"sqlite"` from the pre-removal era
(removed in 87493ce91), and `GetBackend()`'s hard-return of "dolt" (`configfile.go:207-210`)
is a deliberate shim over exactly that. Rule: introduce a NEW locator key (e.g.
`storage_backend`, plus a metadata schema version); absent/legacy/unknown values + presence
of a Dolt data dir resolve to **dolt**, never to a new backend. A Phase 1 harness fixture
with a legacy `backend:"sqlite"` metadata.json pins this. This closes the trap where a legacy
value resolves to the new SQLite backend, opens an empty store, and H9 auto-import silently
re-seeds/forks the workspace.
- **Env precedence:** `BEADS_DOLT_*` (19 vars) are owned by the Dolt backend and are **inert
for non-Dolt locators**; env parsing consolidates behind the registry; the Phase 1 harness
pins the precedence table (H10). `--global`/shared-server: the CLI keeps a seam-level
"workspace redirection" concept whose only current provider is Dolt shared-server topology —
on SQLite/Postgres, `--global` errors with a capability stub (D2 pattern).
### 4.4 Errors, formatting, and the public API
- Typed errors below the seam (NotFound/ClaimConflict/Unsupported{op,backend}/retry-class);
ALL envelope/stream routing and contract strings above it — including gc error class 3
(`bd sql` embedded string), class 4 (H9 auto-import strings — now backend-neutral pinned
contract), and the claim-conflict text.
- Public API: `beads.Open*` route through the registry (H3 fifth path). **Compat note for
extension authors:** on non-Dolt workspaces, type-asserting `RemoteStore` etc. on the
returned `Storage` starts returning false — document in release notes; the assertion pattern
itself (`beads.go:27-31`) is unchanged.
- Single binary, no dynamic plugins. "Addons/packs" = capability-gated stubs inside one binary
+ build tags where deps demand (embedded Dolt is cgo-only; modernc SQLite is pure-Go → nocgo
builds gain a local backend for the first time, `beads_nocgo.go:27`).
### 4.5 Bridge vs destination — how the two relate
The mechanical phases (2a/2b/3) retype the CLI onto the FLAT core interface (§4.1) because
that keeps a 159-reference migration mechanical. That flat interface is **scaffolding, not
the destination**: from Phase 4 onward, the `uowStore` adapter's inner surface is built as
the PUBLIC clean-room seam (§4.0 `Tx` + domain interfaces) rather than as a private adapter
detail — so the destination accretes underneath the migration instead of requiring a second
one. Phase 7 then thins the flat layer opportunistically (or keeps it as sugar). Existing
components map as: `uow.UnitOfWork`/use-cases/`domain/db` → the seam's embryo;
`EmbeddedDoltStore`/`DoltStore`/`issueops` → `backend/dolt` interior; the 13 duals,
`DoltStorage`, the string lists, and the Seam-A parity tests → deleted.
### 4.6 Future-backend expandability (FDB, redb-class embedded KV, S3)
The seam must not assume SQL — two of the three named future backends are key-value and one
is object storage. Design constraints this imposes, to be validated by adversarial review:
- **FoundationDB:** interactive transactions exist but carry hard limits (~5s / ~10MB);
`bd import`-scale work cannot be one `Tx`. The seam therefore must NOT promise unbounded
transaction size — bulk operations need an explicit chunked/bulk path on the seam, and the
contract suite must pin what atomicity bulk ops actually guarantee. Ordered scans and
secondary indexes are backend-built (the bts-rs readiness-projection pattern applies).
- **redb-class embedded KV:** single-writer ACID — same shape as the SQLite slot. (redb
itself is a Rust library; in-process use from Go implies cgo/FFI or it arrives via the
bts-rs lineage — the seam question is only "can an embedded KV implement the contract",
and the spike's file-backed mem store already answered yes.)
- **S3:** no multi-key transactions. A conforming backend is manifest-CAS shaped (commit =
conditional-PUT of a new manifest; `Commit` returns a Serialization retry class on CAS
loss) — which the `Tx`+`RetryClass` model can express, but claim latency and contention
behavior must be honestly capability-flagged (likely `MultiWriter: false` or
claims-unsupported). S3 may be BETTER fitted as the target of RemoteSync/Backup/cold
history (the bts-rs BlockStore/RefStore substrate is S3-native with CAS on refs) than as
a primary Store; the seam should permit both without contortion. `ChangeToken` maps
naturally (manifest ETag).
### 4.7 Red-team revisions (Fable panel, 5 lenses, all verdicts holds-with-fixes) — NORMATIVE
These contract additions override/extend §4.0 where they conflict; fold into the sketch on
the next full edit. Convergent = demanded independently by 2+ lenses.
**Convergent (adopted):**
1. **`Store.Run(closure)` is the primary op; commands migrate ONTO it in Phase 7** (not off
it). Retry ownership lives on the seam; typed errors gain backend-supplied `RetryAdvice`
(backoff hint + budget); storetest includes an error-injecting fake and requires that a
conforming driver's advice lets 24 contending claimers converge (no livelock).
2. **CommandScope above Tx** (see §4.0 note): one version-commit per command, tips second
commit, commit→backup/export→push ordering; dolt-log divergence scenarios join the Phase
2b gate (stdout diffing alone cannot see N-commits-instead-of-one).
3. **Read-consistency contract:** statements within one Tx observe ONE snapshot (PG:
REPEATABLE READ or single-statement collapse; the two-phase ready-union read is the
motivating bug); within-Tx read-your-writes is contract-tested; read families are
classified single-snapshot-required (ready, claim-candidate, mutation-feeding reads) vs
chunk-consistent-permitted (export, audit, stats) — FDB's ~5s limit applies to READS too.
4. **Bulk/chunked port with pinned atomicity:** per-chunk atomic, idempotent upsert,
resumable journal; crash-mid-bulk + auto-import-resume (H9) fixtures in Phase 1;
`create --graph` atomicity resolved against `Limits()`.
5. **Wisp/tier respec:** tier is a DERIVED attribute (ephemeral/no_history/infra-config
routing — never caller-chosen); query methods take visibility flags, not a tier enum;
explicit cross-tier ops (PromoteFromEphemeral, AffectedByDeletion); **tiered-atomicity
contract** (Commit atomic per tier, pinned ordering, `PartialCommit` error class or
`AtomicTiers` flag); a tier×capability locality matrix (History/RemoteSync/Backup/export
MUST exclude local-tier data) with fixtures: purge-spares-no_history,
backup-excludes-wisps. All of this feeds D5.
6. **Ordering/predicate semantics become a written contract:** binary-codepoint collation
(PG DDL pins COLLATE "C"), second-precision timestamps, full tiebreak chains incl.
NULL-first, LIKE-wildcard/escape behavior, metadata `{"k": null}` counts as present;
clock ownership moves client-side (a `Clock` provider joins Identity on the construction
seam) so relative-time predicates and the 48h-hybrid ordering are deterministic +
contract-testable.
**Single-lens (adopted):**
7. **Mutation-observation contract** (repo lens, blocker): `Tx.Commit` yields a
`MutationSummary` (feedable from the same-tx event rows) consumed by a registry-owned
hook dispatcher; the decorator wraps Tx, not just Store; hook-artifact fixtures join the
Phase 1 harness; Phase 7 conversions gate on hook parity.
8. **Interleaving rule:** capability methods may not run between Begin and Commit/Rollback
on the same Store (typed error, storetest-pinned); per-topology connection budgets are
documented backend-author contract (embedded's eager second dolt_ignore connection binds
lazily or every command pays issue-#4303's dominant cost).
9. **Sync/archive TARGETS are a first-class seam concept distinct from Store** (S3 lens):
`BlockStore{Put,Get}`/`RefStore{Get,CAS}` target interfaces + per-scheme drivers
(s3/gs/file); `CapabilitySet.RemoteSync` rescoped to backend-NATIVE remotes (Dolt);
Phase 6 cold history and portable backup-to-URL route through targets — S3's best fit,
without pretending it is (or isn't) a primary Store. Backends whose commit protocol can
strand garbage (CAS-race orphans) MUST implement + declare Maintenance.GC; doctor
surfaces it.
10. **Error taxonomy additions:** `ErrBusy` (Transient; the embedded flock string is its
pinned rendering); claim errors carry structured holder/status (the gc class-2 string
is rendered ABOVE the seam from data, incl. idempotent self-reclaim success);
Open/Begin blocking-vs-fail-fast behavior specified per concurrency class; backends MAY
resolve CommitIndeterminate internally (verify-after-timeout) and return a definite
class.
11. **ChangeToken contract:** opaque, equality-only, data-changed ⇒ token-changed
(false positives permitted); Dolt = hash over non-ignored core tables (NOT HEAD);
per-tier advance semantics pinned; S3 = seq embedded in the manifest, not raw ETag;
FDB = versionstamped last-mutation key.
12. **Testing-story honesty** (repo lens): backend/fake + storetest get their own phase,
gate, and sizing row; evaluate a go-mysql-server-backed fake running the REAL SQL
(reuse, not a fifth semantics copy); Seam-A parity-suite deletion re-gated on
single-SQL-stack convergence inside backend/dolt (embedded TxProvider over the embedded
engine → delete issueops), not on dual-file collapse; §3.1's "integrations are free" is
WRONG — the tracker keys its Dolt fast path on `DB()` presence and must key on the
History capability (same audit for every RawDBAccessor consumer: SQL-speaking ≠
has-Dolt-system-tables).
13. **Streaming:** unbounded consumers (audit, sync, export) keep a cursor/`since+limit`
path (or a StreamingReads capability) before Phase 7 converts them — "slices only" is
bounded-by-workspace materialization, stated as such.
**§4.6 corrections from the future-backend lenses:** FDB limits apply to reads too (~5s
read-version expiry; 100kB value cap vs bd's unbounded LONGTEXT — chunked content or
declared `Limits`); the readiness-projection pattern transplants only with rework (chunked
catch-up outside the op Tx, lease-based projector); redb: what the spike validated is "an
embedded single-file backend behind the seam for the gc-16 surface" — the 9-sort-order
index design, ~107-method surface, and cross-process behavior remain unvalidated, and redb
has no C API (a Go binding does not exist; that backend arrives via FFI work or the bts-rs
lineage); S3 claims stay CORE (never "claims-unsupported") — the drain gate keys on
`Concurrency.ConcurrentWritersSafe` with a disclosed contention model; 412/409/503/timeout
→ RetryClass mapping table to be added.
## 5. Migration phases
Each phase lands green with the default backend byte-identical; each has a verification gate
that names its in_scope coverage.
### Phase 0 — Decisions + contract pinning (docs only)
- Resolve D1–D6 (§6) with maintainers.
- **Leak-4 audit** over capability-sourced behavior in core commands (restore's history
fallback; export/backup freshness → ChangeToken; `bd gc`'s portable decay phase; tracker
fast path). Classification defines the core command set.
- **Core-shrink review** (§2.1 caveat): triage the ~107 core methods into primitive /
derivable-above-seam / capability.
- Census additions from review: H9 auto-import; the 21 `commitPendingIfEmbedded` sites; the
seven issueops Dolt files; the WriteLifecycle protocol inventory (H2).
- Pin H8 behavior contracts (incl. the iterator and autocommit divergences) into
`BackendCapabilities` (`configfile.go:183-204`).
- **Gate:** signed-off decision doc; core-vs-capability census table; WriteLifecycle contract
spec reviewed.
### Phase 1 — The oracle before the surgery
**A new differential runner reusing `tests/regression` scaffolding** (workspace isolation,
binary management, container TestMain, the battle-tested normalizer) — NOT an extension of its
execution core: today's runner is 177 Go closures compared by end-state snapshot only, `t.Fatal`s
on any non-zero exit, and merges stdout+stderr (`regression_test.go:363-374,811-831`). New:
per-step recorder (exit code, split streams, JSON-aware diff), replayable scenario format,
multiset-default + `ordered` flag, in_scope predicate, backend/mode parameter on workspace
setup, property tests for minted IDs, the claim drain (multi-writer only), unique
candidate-binary paths.
- Scope: gc-contract 16 × their flags first (incl. **purge** — no uow dual, red-team history),
plus fixtures: legacy-locator (§4.3), auto-import class-4 trigger (H9), close-blocker→ready
and purge-orphan re-seed transitions (§2.3), the env-precedence table (H10).
- **Shake-out: Dolt-embedded vs Dolt-server.** Premise (review-corrected): these are KNOWN
DIVERGENT (H8) — the run formalizes the divergences, it does not validate on identical
engines. Allowlist discipline: entries are backend-PAIR-scoped, field-scoped, tagged
intentional-vs-known-bug, and expire with the Phase 2 scrub. **Mode waivers do not carry
into SQLite runs.** Expect the autocommit asymmetry to be the dominant waiver class. (The
earlier "expect to confirm the H8 live bug" line is withdrawn — H8's server-mode
export-freshness bug was refuted on verification; the server store commits per write and
auto-export is server-mode-disabled.)
- **Gate:** harness green on Dolt-vs-Dolt with a fully attributed allowlist; coverage report
artifact (scenarios × commands × flags) checked in.
### Phase 2a — Neutralize the core seam (the mechanical part)
- Fold the six neutral sub-interfaces into core (§4.1); retype the global `store` + decorator +
`molecules.NewLoader` + `remotecache.StoreOpener` + helpers from `DoltStorage` to the core
interface (**159 refs / 61 files + 15 test files**); convert the ~87 Dolt-shaped call sites
to UnwrapStore asserts; delete `DoltStorage`.
- Consolidate the FIVE construction paths (incl. public `beads.Open*`) into the registry;
registry owns decorator application (closing the existing lazy/read-only/routed hook gap);
new locator key + legacy rule (§4.3).
- Convert H6 string lists to declared per-command properties.
- **Gate:** full suite + Phase-1 harness Dolt-vs-Dolt: zero NEW divergences vs the Phase-1
allowlist.
### Phase 2b — The WriteLifecycle migration (the semantic part; own phase, own gate)
- Implement the §4.2 contract; migrate the 21 `commitPendingIfEmbedded`/`transact*` call
sites, PersistentPostRunE, the tips deferred-write protocol, ChangeToken for export/backup
freshness (on backend-neutrality grounds — the "fixes the silent server-mode
export-freshness bug" rationale is withdrawn; that bug was refuted on verification, so
ChangeToken is a portability requirement, not a bug fix, and needs no behavior-change release
note), retype H9 auto-import and pin its strings.
- **Gate:** Phase-1 harness re-run; the ONLY acceptable diffs are the enumerated bug fixes;
autocommit/push/export ordering pinned by new dedicated scenarios.
### Phase 3 — Capability-gate the addon surface
- `Capabilities()` is **instance-level** (computed after open, backend × topology — review
fix: `bd sql` works on Dolt-server but not Dolt-embedded, same backend). Gating happens at
**RunE-time via stubs** (D2 resolved): registration stays in `init()` (no bootstrap
restructuring; `bd --help` and the CI docs-regen stay deterministic); absent capability →
deterministic stub error naming the backend that provides it (federation_nocgo precedent;
`bd sql`'s embedded string preserved verbatim as contract).
- Surface: `bd dolt` (16 subcmds), `bd vc`, `bd branch`, `bd history`, `bd diff`,
`bd show --as-of`, `bd federation`, `bd backup`, `bd sql`, `bd flatten`, Dolt legs of
`bd compact`/`bd gc`, `db-proxy-child`, `--global` (§4.3).
- `bd doctor`: per-backend check registry (D6); Dolt checks (incl. `blocked_consistency`'s
`dolt_status` guard) move into the Dolt backend's set.
- Audit + flip the `--backend=sqlite` deprecation contract (`init.go:215-225` hard-errors
today; docs/scripts written against that string need a coordinated, release-noted flip).
- **Gate:** on Dolt, behavior + `--help` byte-identical; a test-only capability-less backend
exercises every stub; docs regen green.
### Phase 4 — Second backend: Postgres (THE PROOF backend; reordered per owner direction)
Owner-stated success criterion: **beads runs on a Postgres backend and the bd CLI simply
doesn't have the history functionality.** Postgres — not SQLite — is therefore the seam
validator and the critical-path deliverable. (The first draft's SQLite-first ordering
followed the spike's cheapest-second-backend guidance; the owner's proof criterion overrides
it.)
**Two candidate build routes — choosing is part of D1, because they hinge on the uow
relationship:**
- **Route A (recommended if D1 lands the adapter): ride the uow stack.** A Postgres
`TxProvider` (plain BEGIN/COMMIT; the commit-message param is ignored) + a dialect port of
the `domain/db` repositories (~75 methods of EXISTING, Seam-A-parity-tested SQL; the
MySQL-isms are enumerable — `ON DUPLICATE KEY`→`ON CONFLICT`, backticks, no `CALL DOLT_*`)
+ the `uowStore` adapter + the five gap units +
per-backend DDL. This REUSES the already-implemented dangerous semantics (same-tx events,
`is_blocked` propagation) instead of re-deriving them, and leans on the workstream's own
stated invariant (issue #4489): "the local-vs-proxied difference is confined to the uow
transaction layer (the mode-polymorphic db.Runner)."
- **Route B (fallback, if D1 keeps the layers separate): direct core implementation**
(~107+24 methods over a fresh schema), cribbing the bts-rs PG adapter wholesale (schema,
order/ready indexes + batched hydration, single-statement claim via `FOR UPDATE SKIP
LOCKED`, typed retry classes, optional readiness projection). Independent of the uow
roadmap; more code; a THIRD implementation of the business semantics to keep honest —
**the hard part is semantics, not SQL** (§2.3: same-tx events, `is_blocked` propagation,
delete/purge neighbour recompute, wisp/tier semantics, D5 local-tables).
**Proof-wedge milestone (demoable before Phases 2–3 fully land):** minimal locator `backend`
field + one factory arm + the PG stack + a TEMPORARY full-`DoltStorage` compat shell whose 37
Dolt-shaped methods return typed Unsupported errors (graceful errors, NOT panics — the D1
design-history lesson) + PersistentPostRun neutralized for non-Dolt. This demonstrates "bd on
Postgres, `bd history` absent" without waiting for the 159-ref retype; the shell is deleted
when Phase 2a lands. Accepted sharp edges for the demo: auto-export/backup freshness and
parts of `bd doctor` degrade on the wedge; the gc-16 harness subset is the demo's acceptance
test.
- **Gate:** Phase-1 harness PG-vs-Dolt green over the gc-16 in_scope set, THEN widened to the
CORE-67 surface before the backend is announced (coverage artifact enforces the widening —
no shipping green-on-16-only). Transition fixtures (§2.3) green. 24-process claim drain (PG
advertises multi-writer). **Remote deployment guidance shipped** — per-command
connection-setup cost measured, PgBouncer/co-location documented (§2.6).
### Phase 5 — Third backend: SQLite (the nocgo/embedded story; now optional)
Pure-Go (`modernc.org/sqlite`), documented single-writer (claim-drain gate N/A); fills the
nocgo hole (`beads_nocgo.go:27` — non-CGO builds currently have NO local backend). Under
Route A it rides the same production line (a SQLite `TxProvider` + dialect port); under
Route B, a direct implementation with the WAL lessons (`synchronous=NORMAL`,
`wal_autocheckpoint=0` + explicit checkpoint cadence, pragmas on the read pool, no unclosed
rows). Slice-iterator stubs are legal (embedded Dolt ships them; server streams — both
satisfy the interface).
- **Gate:** as Phase 4 minus the claim drain; single-writer lock UX tested. Export→import
adoption-path fidelity documented (events history, wisps, LocalMetadata do NOT obviously
round-trip — measure and state).
### Phase 6 — History as a portable capability (optional)
Changelog capability (same-tx field-delta log) + HistoryViewer-as-fold (§2.5). Dolt keeps its
native implementation. Mirror the spike-documented wire quirks (bare JSON array, PascalCase
keys, full snapshots, client-side `--limit`, volatile CommitHash normalized).
### Phase 7 — The end-game: promote the clean-room seam, thin the bridge (opportunistic)
Migrate commands off the flat conveniences onto `Begin/Tx/Commit` directly — file by file,
no flag day, each conversion deleting a `RunInTransaction` wart — until the flat layer is
thin enough to delete or small enough to keep as sugar. No deadline: the system is already
multi-backend and capability-gated before this phase starts. Exit criterion: the `store`
package's godoc IS the backend-author contract, and `storetest.RunContract` is the only
thing a new backend must pass before the differential harness.
### Explicitly deferred / out of scope
- In-place cross-backend DATA migration (`bd migrate --to-backend=X`) — the interim adoption
path is export→re-init→import with Phase-4-documented fidelity limits.
- Retiring the uow layer's 13 dual command paths (fate fixed by D1; execution follows Phase 3).
- YugabyteDB/FDB backends; dynamic plugin loading; backend-neutral federation.
## 6. Decisions required
- **D1 — Canonical seam AND proxied-server's fate** (one decision, not two). **Corrected after
git-history review: the uow/domain layer is NOT dormant** — it is an active workstream
(Dustin Brown / coffeegoddd, DoltHub), converting one command per PR since 2026-05-10 with
merges through 2026-06-24 (#4039–#4488: provider, list, update, delete, show, close, reopen,
ready, dep, config, context, query). The init gate is deliberate staging ("lifting the gate
is a separate, deliberate change", PR #4488), not abandonment. Its motivation is
**multi-process concurrency**: embedded Dolt is single-writer; a shared server + short
per-command transactions serves concurrent agents. "Freeze uow" is therefore not a viable
recommendation.
**Design-history fact that reframes this decision:**
a `storage.DoltStorage` implementation over the proxy WAS built first — `DoltServerStore`,
PR #3792, May 6–7 2026 — and died in four days as an 805-line wall of
`panic("unimplemented")` (128/132 methods), pivoted to the UOW design with only a two-line
TODO as rationale. It died because the god interface cannot be implemented incrementally
from scratch — a cause the uow workstream's own six weeks of use-case building has since
removed (the adapter delegates to them), and which Phase 2's interface shrink + capability
split removes for every future backend. "We tried a store over the proxy already" is
therefore true but not dispositive.
**Recommended reconciliation — compose, don't pick:** `storage.Storage` remains the backend
seam (what the CLI consumes; what SQLite/Postgres implement; where the capability/addon
model lives — use-cases have no answer for command gating). Build the missing **`uowStore`
adapter** (core `Storage` implemented over `UnitOfWork` use-cases; today no such adapter
exists and `store_factory.go:51/96/169` TODOs point the other way — this is the crux to
align with the uow owner). Consequences: proxied-server becomes a topology of the Dolt
backend behind the registry; the 13 `*_proxied_server.go` duals collapse; the domain
repositories over `db.Runner` (3 methods) become an ACCELERATOR for future SQL backends
(supply Runner+TxProvider+dialect, reuse the repos) rather than a rival seam; the
`Tx.Commit` = `CALL DOLT_COMMIT` (`uow/doltserver_tx.go:28`) adopts the §4.2 WriteLifecycle
contract one layer down. **Full sketch with the method-by-method mapping and gap analysis**
(headline: ~60% of core maps 1:1 to existing use-case
methods; the true remaining cost of full proxied coverage is five small gap units, not 54
more command conversions).
**The alternative (full use-case trajectory):** commands migrate to use-cases everywhere and
backends supply repositories. Coherent, but: ~54 more command conversions, dual
implementations + parity tests for the duration, non-SQL backends become impossible
(`Runner` assumes SQL), and the capability model is still needed regardless.
**Divergence governance either way:** the two paths already diverge in features (`--offset`
is proxied-only because the embedded issues+wisps union cannot page — PR #4488) — assign an
owner for keeping the paths semantically converged until they reunify, and add the
divergences to the Phase 1 allowlist as tagged, expiring entries.
**This decision requires the uow workstream owner in the room.**
- **D2 — Absent-capability UX. RESOLVED by review: stubs, not hidden.** Cobra registration
happens in package `init()` before workspace resolution; hiding requires pre-cobra locator
resolution and breaks help/docs determinism. Stubs preserve the gc error contract and the
federation_nocgo precedent. (If hiding is ever wanted, it is its own sized work item.)
- **D3 — `RunInTransaction` message param:** keep, re-specified per §4.2 (the blank-message
suppression semantics become explicit seam behavior; "advisory" was wrong).
- **D4 — MergeSlot\*/Slot\***: keep in core (bead-backed, portable); revisit when Postgres
lands (Phase 4) — real row locking may obviate them.
- **D5 — Wisps' dolt-local tables** on backends without an ignore concept: table-prefix
convention (recommended) vs separate file/schema. Feeds the Phase 4 schema design.
- **D6 — Doctor scoping:** per-backend check registry (recommended) vs neutral check
interfaces.
## 7. Risks
| Risk | Mitigation |
|---|---|
| Two-seam divergence widens while D1 is undecided (uow conversions merge weekly; `--offset`-class proxied-only features accumulate) | D1 decided WITH the uow owner in Phase 0; until then, new proxied features carry a convergence note + Phase 1 allowlist entry; the `uowStore` adapter collapses the duals once agreed |
| Legacy `backend:"sqlite"` metadata resolves to the new backend and auto-import forks data | §4.3 legacy rule (new locator key, dolt-default fallback) + Phase 1 fixture — was a review blocker, now designed-out |
| Silent drift during the Phase 2a retype | Harness EXISTS FIRST; zero-new-divergence gate; commit per file-cluster |
| WriteLifecycle migration changes semantics invisibly | Own phase (2b) + dedicated ordering scenarios; only enumerated bug-fix diffs accepted |
| Allowlist waivers rot and mask real regressions | Pair-scoped, field-scoped, tagged, expiring entries; no carryover to new-backend runs |
| Green harness over-trusted | §2.4 epistemics: hooks/timing/coverage limits stated per gate; coverage artifact required; CORE-67 widening enforced before Phase-4 announce |
| A from-scratch backend (Route B) re-opens the spike's transactional bug class | §2.3 rules are Phase 4/5 implementation requirements with mandatory transition fixtures; Route A sidesteps by reusing the parity-tested use-case semantics |
| `bd doctor`/backup partial coverage confuses users | Capability-aware stubs + doctor prints per-backend coverage |
| Public-API extensions break on non-Dolt workspaces | §4.4 compat note; assertions return false (defined behavior), release-noted |
| Schema drift between per-backend migration streams | Shared logical-schema contract suite each backend's fresh DB must pass |
## 8. Sizing (rough, sequential; review-adjusted)
| Phase | Size | Notes |
|---|---|---|
| 0 | ~1 week | decisions + leak-4 audit + core-shrink review + WriteLifecycle spec |
| 1 | 2–3 weeks | new runner on old scaffolding; gc-16 scope + fixtures (was 1–2; review: per-step core is new code) |
| 2a | 2–3 weeks | fold + retype (159 refs/61 files) + registry (5 paths) + metadata properties |
| 2b | 1–2 weeks | WriteLifecycle contract migration (semantic; own gate) |
| 3 | 1–2 weeks | instance-level capabilities + stubs + doctor registry + deprecation flip |
| 4 | Route A: 4–6 weeks / Route B: 5–8 weeks | Postgres proof: A = PG TxProvider + repo dialect port (~75 methods of existing SQL) + adapter + 5 gap units + DDL; B = ~107-method direct core + is_blocked/events semantics from scratch + schema; both + conformance to CORE-67 |
| 5 | 2–4 weeks (Route A) / 4–7 (Route B) | SQLite riding the same production line (A) or direct (B); nocgo hole |
| 7 | opportunistic | thin the flat bridge onto the §4.0 seam; no deadline |
| 6 | 2–4 weeks | optional; independent — and under Route A, the Changelog capability could later give PG history too (§2.5), making "no history on Postgres" a v1 statement, not a permanent one |
Phases 4/5 are independent of each other; 6 independent of both.
**Critical path to the owner's PROOF** ("bd on Postgres, no history"): 0 (D1 + route choice) →
1 (gc-16 harness subset) → Phase-4 proof wedge (~6–9 weeks). **Critical path to the full
end state** ("users choose a backend", shipped properly): 0 → 1 → 2a → 2b → 3 → 4
(~12–19 weeks sequential); the wedge work is on that path, not thrown away — only the compat
shell is deleted when 2a lands.
---
## Appendix: review provenance
Recon: 5 parallel read-only explorers (storage interface census; Dolt leak census; command
surface census; existing-abstraction assessment; bts-rs learnings extraction) — ~578k tokens,
311 tool calls. Adversarial pass: 2 reviewers (feasibility-vs-repo, completeness-vs-spike),
verdicts sound-with-fixes; 2 blockers, 8 majors, 5 minors, 13 missing items — all folded into
this revision. Notable review discoveries retained above: the six-neutral-interfaces blocker
(§4.1), the legacy-locator trap (§4.3), the WriteLifecycle protocol structure (§4.2/H2), the
instance-level capability requirement (Phase 3), ~~the H8 live bug (server-mode export
freshness)~~ (corrected after verification: refuted — the server store commits per write and
auto-export is server-mode-disabled; ChangeToken is retained on backend-neutrality grounds),
and the H9 auto-import hazard. Clean-room shred: 5-lens Fable panel
(object-model, repo-reality, FDB, embedded-KV, S3) — all verdicts holds-with-fixes; 2
blockers (outcome-derived commit messages vs OpInfo-at-Begin; hook-observation contract), 17
majors; all folded as §4.7.