789 lines
57 KiB
Markdown
789 lines
57 KiB
Markdown
# PROPOSAL: Pluggable Storage Backends
|
||
|
||
**Status:** Historical — partially implemented, then superseded on 2026-07-16
|
||
**Date:** 2026-07-02
|
||
**Informed by:** the bts-rs Rust spike (a from-scratch Rust reimplementation
|
||
of the gc-contract surface that validated a backend-agnostic storage seam at 299/299 byte-parity
|
||
across two radically different backends), a five-dimension recon of this repo's storage
|
||
architecture, and a two-reviewer adversarial verification pass. All file:line citations verified
|
||
against `worktree-beads-new`.
|
||
|
||
> **Superseded scope:** This proposal is retained as the design record for the
|
||
> storage interface, shared issue core, SQLite adapter, and differential
|
||
> conformance work that grew from it. Those pieces remain. The direct
|
||
> PostgreSQL and MySQL adapters were rolled back before entering a tagged
|
||
> release because supporting additional general-purpose server databases adds
|
||
> substantial dialect, credential, schema-lifecycle, migration, CI, and
|
||
> operational complexity. Our goal is to keep Beads as simple as possible and
|
||
> consume as few resources as possible. The supported paths are now embedded
|
||
> Dolt, Dolt server, and SQLite; Dolt server's use of the MySQL wire protocol is
|
||
> unchanged. See [Storage Backends](docs/architecture/storage-backends.md) for
|
||
> the current design.
|
||
|
||
Everything below this notice is retained as historical design material, not as
|
||
current backend support or migration guidance.
|
||
|
||
---
|
||
|
||
## 1. End state
|
||
|
||
- `bd init --backend=dolt|sqlite|postgres` — the user picks the engine at init. The workspace
|
||
locator records the choice; opening consults the locator, never the environment.
|
||
- **One core command set** (the gc-contract 16 plus the portable CORE inventory, ~67 commands)
|
||
behaves identically on every backend, proven by a differential conformance harness — not by
|
||
code review.
|
||
- **Storage-specific commands are capability-gated addons.** `bd history`, `bd dolt push/pull`,
|
||
`bd vc`, `bd branch`, `bd diff`, `bd federation`, `bd backup`, `bd sql` are live only when the
|
||
opened store advertises the matching capability; otherwise they are deterministic stubs that
|
||
explain what backend provides them (Decision D2, resolved: stubs, not hidden — see §6).
|
||
Pick Dolt → you get history and remotes. Pick SQLite → you get a fast, zero-dep, pure-Go
|
||
single-writer store and those commands stub out.
|
||
- Dolt remains the default and richest backend. **Back-compat guarantee:** a workspace created
|
||
before this migration keeps opening exactly as it does today — enforced by an explicit
|
||
legacy-locator rule (§4.3), not by hope.
|
||
- Adoption path for existing workspaces: `bd export` → re-init on the new backend → `bd import`,
|
||
with documented fidelity limits (§5, Phase 4 gate). In-place cross-backend data migration is
|
||
a separate initiative.
|
||
|
||
## 2. What the Rust spike proved (the transferable payload)
|
||
|
||
Evidence: `bts-rs/docs/` (`SEAM_VALIDATION_FINDINGS.md`, `CONFORMANCE_GAPS.md`,
|
||
`RED_TEAM_FINDINGS.md`, `HISTORY_ON_YUGA_DESIGN.md`, `PERF_PG_VS_YB.md`, `REMOTE_DEPLOYMENT.md`).
|
||
|
||
### 2.1 The seam shape
|
||
Six small core traits (33 data methods) + **optional capability traits** reached via accessors
|
||
defaulting to "absent". Two backends (Postgres, file-backed memory) hit byte-parity with real bd
|
||
on 299/299 scenarios **with zero command-handler changes** — handlers never name a backend.
|
||
|
||
**Go gets the spike's hardest problem for free.** Rust's non-object-safe async traits forced a
|
||
399-line delegation enum; in Go, `storage.Storage` is already an interface and `Open()` returns
|
||
the interface value. Budget the port on semantics and conformance, not plumbing.
|
||
|
||
**Seam-size caveat (from review):** the spike's 33-method core is NOT evidence that beads' much
|
||
larger core (§4.1: ~107+24 methods after the fold) is cheap to implement per-backend. The spike's
|
||
"~1–10k LOC per adapter" anchor was measured against 33 methods. Phase 0 therefore includes a
|
||
core-shrink review: which core methods are truly primitive vs derivable-above-the-seam vs
|
||
capability material. Sizing in §8 assumes the core does NOT shrink (worst case).
|
||
|
||
### 2.2 The four seam-leak rules (each was a real bug in the spike)
|
||
1. **Construction is ON the seam.** `Open(locator, identity)` / `Init(backend, ...)` are seam
|
||
methods; otherwise every entry point names concrete backends.
|
||
2. **Identity is ON the seam** — actor identity is construction-time config in the seam package.
|
||
3. **The locator is backend-neutral and self-describing** (`{backend, workspace_id}`); backend
|
||
is read FROM the locator at open; `--backend`/env applies **only at init**.
|
||
4. **Capabilities OPTIMIZE core behavior; they never GATE it.** For every capability use inside
|
||
a core command there must be a core-interface fallback, OR the whole command is
|
||
capability-gated. (The spike's only runtime divergence violated this.)
|
||
|
||
### 2.3 Transactional-integrity rules that apply to beads' CORE (not just capabilities)
|
||
The spike's red team hit this bug class three times; in beads it lives on the core surface:
|
||
- **Same-tx side effects:** beads writes the events table inside the mutation transaction
|
||
(`issueops/create.go:601`, `labels.go:151`, `helpers.go:147`; read via core
|
||
`GetAllEventsSince`, `storage.go:84`, and `bd audit`). Any new backend must do the same —
|
||
events emitted outside the mutation tx strand or duplicate audit rows on crash.
|
||
- **Denormalized `is_blocked` is the most conformance-dangerous core semantic.** It is
|
||
maintained across 10+ issueops write paths (dependencies/close/reopen/delete/promote/…) and
|
||
read by ready queries (`sqlbuild/ready.go:96`). Delete/purge must recompute surviving
|
||
neighbours. A fresh backend reimplements this propagation on every mutation; the harness
|
||
must carry required transition fixtures (close-blocker→ready, purge-orphan re-seed).
|
||
|
||
### 2.4 The conformance oracle
|
||
- **Differential, not static:** scenarios as real processes in throwaway workspaces; volatile
|
||
normalization (`<TS>`/`<UUID>`/actor); per-step exit + stderr + JSON-aware stdout diff. A
|
||
static canonicalized corpus "pins a formatter, not a system".
|
||
- **Multiset array diffing by default**, with a per-scenario `ordered` flag valid only where
|
||
ordering is implementation-independent.
|
||
- **Property tests** for non-byte-diffable paths (minted IDs, `--graph`).
|
||
- **Concurrency proofs as capability gates:** the 24-process `ready --claim` drain applies only
|
||
to backends advertising multi-writer.
|
||
- **Epistemic honesty — what green does NOT prove (§7 risk table):** hook-firing semantics
|
||
(post-commit, rollback-suppressed — invisible to stdout/stderr diffing), equal-priority and
|
||
wall-clock-hybrid ordering (`ready`'s 48h recency rule), and anything outside the in_scope
|
||
predicate. Each gate below pins its in_scope coverage explicitly so "proven identical" is
|
||
never claimed beyond what ran.
|
||
- Operational: unique candidate-binary path per run (`cp` over an exec-mapped binary fails
|
||
silently and scores a stale binary).
|
||
|
||
### 2.5 History without Dolt (later, not now)
|
||
`bd history`'s minimal substrate is a per-issue, ordered, never-pruned field-delta changelog
|
||
written in the same tx as each mutation + `HistoryViewer{History, AsOf, Diff}` as a backward
|
||
fold. Dolt provides all three natively (`dolt_history_*`, `AS OF`, `dolt_diff`). So "pick Dolt →
|
||
get history" can later relax to "any backend with the Changelog capability" without
|
||
re-architecting.
|
||
|
||
### 2.6 Backend guidance from measured perf
|
||
Postgres bounded ops flat to 100k (list ~2ms). Remote cost = round-trips × RTT **plus ~6 RTT of
|
||
connection setup per command in the shell-out model** (~300ms pure overhead at 50ms RTT) —
|
||
mitigations: co-location, local PgBouncer ≥1.21 transaction mode, daemon endgame. Collapse write
|
||
paths to 1–2 statements; `FOR UPDATE SKIP LOCKED` IS the claim guarantee. YugabyteDB ≈ 2–3×
|
||
latency / 10× write cost — multi-region only. An embedded pure-Go store is the cheapest second
|
||
backend and the one that validates the seam.
|
||
|
||
## 3. Where beads is today (recon summary, review-corrected)
|
||
|
||
### 3.1 The good news — the seam is already 80% drawn
|
||
- `storage.DoltStorage` (`internal/storage/storage.go:200-213`) composes core `Storage` (62
|
||
methods) + 11 sub-interfaces (144 total; + `Transaction` 24). The Dolt-shaped surface is
|
||
already isolated in **5 named sub-interfaces** — VersionControl (16), HistoryViewer (3),
|
||
RemoteStore (12), SyncStore (2), FederationStore (4) = 37 methods, ~87 genuine call sites in
|
||
non-test cmd/bd. The other **6 sub-interfaces are backend-neutral** (BulkIssueStore,
|
||
DependencyQueryStore, AnnotationStore, ConfigMetadataStore, CompactionStore,
|
||
AdvancedQueryStore — ~45 methods, ~155 cmd/bd call sites serving CORE commands like
|
||
`ready --claim`, `dep`, `comment`) and fold INTO the core seam (§4.1).
|
||
- **The capability-gating pattern already ships:** 10 optional interfaces
|
||
(`storage.go:217-287`), `UnwrapStore` + type-assert at 22 cmd/bd sites, graceful errors
|
||
(`sql.go:54-57`).
|
||
- **The public API is not frozen to the god interface:** root `beads.go:20` aliases
|
||
`Storage = beads.Storage` (= `storage.Storage`); RemoteStore/SyncStore are exported as
|
||
separately assertable interfaces. The retype does not break the public type surface — but see
|
||
§4.4 for the public `Open*` constructors and the extension-compat note.
|
||
- **Real prior art for the oracle:** `tests/regression` (built for the SQLite→Dolt migration;
|
||
found 70+ bugs) contributes workspace isolation, binary management, container TestMain, and a
|
||
normalizer that already encodes cross-backend representational divergences
|
||
(`regression_test.go:587` local-vs-UTC midnight; `:647` metadata `{}` vs absent). Its
|
||
execution core does NOT transfer (§5 Phase 1).
|
||
- Integrations are portable for free (all 6 trackers bind to narrow `storage.Storage`); the
|
||
tracker engine already demonstrates capability-with-fallback (`tracker/engine.go:695-717`).
|
||
- Dialect exposure is concentrated: ~90% of SQL flows through `issueops`/`sqlbuild`/`domain/db`.
|
||
Dolt-specific SQL inside issueops sits in **7 files**: `as_of.go`, `commit_pending.go`,
|
||
`diff.go`, `federation.go`, `blocked_merge.go`, `history.go` (`dolt_history_issues`), and
|
||
`blocked_consistency.go` (`dolt_status` guard — backend-internal, moves with the backend).
|
||
The first six map 1:1 onto gateable capabilities.
|
||
|
||
### 3.2 The hazards — what actually blocks a second backend
|
||
- **H1. Two competing seams, mid-flight — and BOTH are alive.** The `DoltStorage` path covers
|
||
100% of commands. The **uow/domain layer** covers 13 commands via `*_proxied_server.go`
|
||
duals and is an ACTIVE workstream (one command per PR since 2026-05-10, latest merges
|
||
2026-06-24; the init gate at `init.go:135-140` is deliberate staging per PR #4488, and the
|
||
CI lane is dark only until the gate lifts). It is Dolt-flavored at its Tx layer
|
||
(`uow/doltserver_tx.go:28` = `CALL DOLT_COMMIT`), duplicates issueops (kept honest by
|
||
Seam-A parity tests), and is already feature-diverging from the embedded path (`--offset`
|
||
is proxied-only). **Proxied-server mode exists ONLY on the uow path** — `newDoltStore`
|
||
refuses it (`store_factory.go:48-53`). D1 must resolve the seam relationship AND proxied's
|
||
fate, with the uow workstream owner participating (§6).
|
||
- **H2. Dolt lifecycle is fused into every command's write path, with real protocol structure.**
|
||
Not just "auto-commit in PostRun": `transactHonoringAutoCommit` (`dolt_autocommit.go:28-47`)
|
||
**blanks the commit message** to suppress mid-tx commits depending on mode; a non-blank
|
||
message + success sets `commandDidExplicitDoltCommit` which suppresses PostRun auto-commit
|
||
(`main.go:1223-1227`); the tips system defers metadata writes and PostRun issues a SECOND
|
||
commit for them (`tips.go:154-183`, `main.go:1233-1257`); auto-export/backup freshness reads
|
||
`GetCurrentCommit` AFTER the commit; auto-push gates and error semantics differ from
|
||
auto-commit; the autocommit-mode default itself is chosen by the CLI from `usesSQLServer()`
|
||
(`main.go:1099-1106`). `commitPendingIfEmbedded`/`transactHonoringAutoCommit` have **21 call
|
||
sites across 18 files**. This is a protocol, not a hook — it gets a specified contract (§4.2)
|
||
and its own phase (2b).
|
||
- **H3. Global store + FIVE construction paths.** `store storage.DoltStorage`
|
||
(`main.go:45`; **159 non-test refs / 61 files**, 15 more test files). Paths: PreRun choke
|
||
point; lazy `direct_mode.go`; read-only; ~13 files opening routed/extra stores; **and the
|
||
public API** — root `beads.go:63-73` `Open/OpenFromConfig` hard-code `dolt.New*`
|
||
(+ `beads_nocgo.go` `OpenBestAvailable`), used by external Go extensions. The registry must
|
||
own all five. The `HookFiringStore` decorator EMBEDS DoltStorage
|
||
(`hook_decorator.go:28-29`) and is applied only on the main path today — the registry must
|
||
own decorator application or lazy/read-only/routed opens keep silently skipping hooks.
|
||
- **H4. The migration stream is not portable.** 53 tracked + 10 dolt-local `*.up.sql`, MySQL
|
||
dialect; migrations 0019/0028/0040/0041 execute `dolt_ignore`/`CALL DOLT_COMMIT` inline. New
|
||
backends need per-backend migration sources (the `schema` runner's two-stream design is the
|
||
extension point).
|
||
- **H5. Core-signature Dolt leaks.** `RunInTransaction(ctx, commitMsg, fn)` — where the message
|
||
is load-bearing suppression semantics per H2, NOT advisory; `GetCurrentCommit` as freshness
|
||
token (`export_auto.go:85`, `backup_auto.go:127`); `ApplyCompaction(commitHash)`; `bd edit`'s
|
||
raw `*sql.DB` keepalive (`edit.go:150`); "dolt-ignored" LocalMetadata contract.
|
||
- **H6. Command metadata in name-keyed string lists.** `noDbCommands` (27), `readOnlyCommands`
|
||
(14), dolt-subcommand patch lists (`main.go:770-808,122-137`).
|
||
- **H7. bd doctor** — 37/45 files Dolt-tinged, raw SQL on `dolt_remotes`/`dolt_status`.
|
||
- **H8. Mode-divergent semantics are unformalized.** The draft claimed this "includes a LIVE
|
||
BUG": ~~with server-mode autocommit OFF, `GetCurrentCommit` never advances, so
|
||
auto-export/backup freshness checks silently stop firing in server mode.~~
|
||
**CORRECTED AFTER VERIFICATION: REFUTED.** The claimed "live bug" does not exist. The
|
||
server-mode store (`internal/storage/dolt.DoltStore`) commits to HEAD **per write method,
|
||
unconditionally** — independent of `--dolt-auto-commit` and of `maybeAutoCommit`
|
||
(`dolt/issues.go:49` CreateIssue → `CALL DOLT_COMMIT`; UpdateIssue/CloseIssue/DeleteIssue
|
||
likewise; `RunInTransaction` → `versioncontrolops.StageAndCommit`). So in non-proxied server
|
||
mode HEAD advances on every persistent write, `DOLT_HASHOF('HEAD')` changes, and the
|
||
auto-backup freshness check DOES fire. Separately, auto-export never even reaches the
|
||
freshness check in server mode — it early-returns (`export_auto.go:38-41`, `if serverMode {
|
||
return nil }`). The `maybeAutoCommit` machinery governs the EMBEDDED working-set-persistence
|
||
flow, not the server store's per-method commits; the H8 draft was written against an
|
||
inaccurate model of the server store. The rest of H8 stands: auto-commit ON embedded / OFF
|
||
server; single-writer flock embedded-only (`store_factory.go:66-85`); server DoltStore has
|
||
real streaming iterators (`dolt/iter_issues.go:48`) while embedded ships `NewSliceIter`
|
||
stubs; `bd sql`/`bd admin` work in server mode but error in embedded (`sql.go:45`,
|
||
`admin.go:17`) — and that embedded error string is gc contract class 3.
|
||
- **H9. Auto-import is a silent data-mutation path on the core surface** (missed in the first
|
||
draft; review-added). `maybeAutoImportJSONL` runs in PreRun for every command
|
||
(`main.go:1163-1164`), takes `storage.DoltStorage`, auto-imports JSONL into empty databases,
|
||
and has a documented data-clobber history (`auto_import_upgrade_unit_test.go:74-87`). Its
|
||
"auto-importing … into empty database" strings are gc error class 4 (the write-loss guard).
|
||
A second backend that is empty-on-fresh-open is exactly the trigger condition; if its strings
|
||
differ, gc's guard goes blind.
|
||
- **H10. Env precedence can override the locator at OPEN.** `BEADS_DOLT_SERVER_MODE=1` is
|
||
precedence #1 over metadata.json (`configfile.go:232-243`); shared-server env/config
|
||
overrides pinned `dolt_mode` with a Notice (`main.go:247`); `--global` hard-requires
|
||
shared-server (`main.go:1071-1076`). Parsing is scattered (configfile, doltserver, init,
|
||
bootstrap). The spike's "locator wins at open" rule must extend to topology, with an explicit
|
||
ownership and inertness rule (§4.3).
|
||
|
||
### 3.3 The command surface (census, corrected)
|
||
~271 cobra commands (~111 top-level), all registered unconditionally **in package `init()` —
|
||
before any workspace resolution** (this kills naive hide-at-registration; see D2). Classified:
|
||
CORE ≈ 67 (incl. the gc-contract 16), STORAGE-SPECIFIC = 11 (+3 mixed: `bd gc`, `bd migrate`,
|
||
`bd restore`'s history fallback), INTEGRATION = 6 families, INFRA ≈ 27. **12 of the gc-16 have
|
||
dual (store+uow) implementations; `purge`, `count`, `version`, `stats` do not** — and `purge`
|
||
is precisely where the spike's red team found re-seeding bugs, so it is a required Phase 1
|
||
scenario, not a "covered elsewhere" case.
|
||
|
||
## 4. Target architecture — the clean-room object model, and the bridge to it
|
||
|
||
Two things are described here and it matters to keep them distinct: **§4.0 the destination**
|
||
(the clean-room object model we would build greenfield, which the migration accretes toward)
|
||
and **§4.5 the bridge** (the flat-interface waypoint the mechanical phases pass through).
|
||
The destination is what new code is written AGAINST from Phase 4 onward; the bridge is
|
||
scaffolding that de-risks the retype and is thinned afterward (Phase 7).
|
||
|
||
### 4.0 The clean-room object model (the destination)
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────────────────┐
|
||
│ cmd/bd — commands, formatting, streams, contract strings │
|
||
│ command registry: declared metadata {needsStore, readOnly, │
|
||
│ requires: Capability} — replaces the name-keyed string lists (H6) │
|
||
└──────────────────────────────┬─────────────────────────────────────────┘
|
||
│ consumes ONLY:
|
||
┌──────────────────────────────▼─────────────────────────────────────────┐
|
||
│ package store — THE SEAM (small, frozen, documented) │
|
||
│ Driver / Registry / Locator / Identity (construction, rules 1–3) │
|
||
│ Store / Tx + ~7 domain interfaces (core; Tx-first) │
|
||
│ Capability interfaces + CapabilitySet (optional surface, rule 4)│
|
||
│ Error taxonomy + retry classes (typed, rendering-free) │
|
||
│ storetest.RunContract(t, driver) (the compliance suite) │
|
||
└──────┬───────────────┬───────────────┬───────────────┬─────────────────┘
|
||
│ │ │ │
|
||
backend/dolt backend/postgres backend/sqlite backend/fake
|
||
(topologies: (multi-writer; (pure-Go; (in-memory/file;
|
||
embedded | no history v1) single-writer; unit-test keystone)
|
||
server | fills nocgo)
|
||
proxied
|
||
INTERNAL)
|
||
```
|
||
|
||
The core, sketched (signatures indicative, not final):
|
||
|
||
```go
|
||
// construction — the spike's leak rules 1–3, verbatim
|
||
type Driver interface {
|
||
Open(ctx context.Context, loc Locator, id Identity) (Store, error)
|
||
Init(ctx context.Context, opts InitOptions) (Locator, error)
|
||
}
|
||
type Locator struct{ Backend string; Raw json.RawMessage } // self-describing; env only at Init
|
||
type Identity struct{ Actor, Email string } // on the seam; injectable for tests
|
||
|
||
type Store interface {
|
||
// Run is the PRIMARY seam op: one retryable unit of work. The seam owns
|
||
// phase-aware retry (Serialization -> bounded auto-replay of fn;
|
||
// CommitIndeterminate -> surfaced, never auto-retried; fn must be replayable,
|
||
// no external I/O while the Tx is open). Semantics lifted from uow/run_in_tx.go.
|
||
Run(ctx context.Context, op OpInfo, fn func(Tx) error) error
|
||
Begin(ctx context.Context, op OpInfo) (Tx, error) // low-level escape hatch
|
||
Capabilities() CapabilitySet // per-INSTANCE (topology-aware)
|
||
Limits() Limits // {MaxAtomicTxBytes, MaxValueBytes, MaxTxDuration,
|
||
// ConsistentScanBudget} — CLI pre-flights + chunks instead
|
||
// of discovering backend ceilings as commit failures
|
||
ChangeToken(ctx context.Context) (string, error) // equality-only freshness cursor
|
||
Close() error
|
||
}
|
||
|
||
// OpInfo + CommandScope are where the H2 commit protocol collapses to. NOTE
|
||
// (red-team blocker): commit messages are OUTCOME-derived in bd (composed after
|
||
// close/claim results — close_proxied_server.go:242-255, uow.RunInTxMsg), so the
|
||
// description travels on Commit, NOT on Begin. A CommandScope above Tx owns the
|
||
// one-version-commit-per-command rule (N txs -> ONE dolt commit), the tips
|
||
// deferred-metadata second commit, and PostWritePush ordering.
|
||
type OpInfo struct{ Command, Actor string; ReadOnly bool }
|
||
|
||
// the transaction is the first-class object — one per command/logical op
|
||
type Tx interface {
|
||
Issues() Issues // CRUD, search, ready, claims; wisps as a TIER param, not a
|
||
Deps() Dependencies // parallel ×2 method family
|
||
Labels() Labels
|
||
Comments() Comments
|
||
Events() EventReader // reads; event WRITES happen implicitly inside mutations, same tx
|
||
Config() Config
|
||
Commit(ctx context.Context, desc string) error // desc outcome-derived;
|
||
// Dolt: DOLT_COMMIT(desc); PG: COMMIT; reads: Rollback, never Commit
|
||
Rollback(ctx context.Context) error
|
||
}
|
||
|
||
// capabilities: extend core behavior, never gate it (rule 4)
|
||
type CapabilitySet struct {
|
||
History, VersionControl, RemoteSync, Backup, RawSQL bool
|
||
// MultiWriter bool was too coarse (red-team, 2 lenses independently):
|
||
Concurrency struct {
|
||
ConcurrentReaders, ConcurrentWritersSafe bool
|
||
WriterExclusion string // enforced-lock | lease | none
|
||
ContentionModel string // row | workspace-CAS
|
||
}
|
||
}
|
||
func History(s Store) (HistoryViewer, bool)
|
||
type HistoryViewer interface{ /* History, AsOf, Diff */ }
|
||
// + VersionControl, RemoteSync, Backup, RawSQL, Maintenance(GC/Flatten/Compact)
|
||
|
||
// errors: typed below the seam, rendered above it
|
||
var ErrNotFound, ErrClaimConflict error
|
||
type ErrUnsupported struct{ Op, Backend string }
|
||
type RetryClass int // Transient | Serialization | CommitIndeterminate | Permanent
|
||
```
|
||
|
||
**Every commitment traces to a scar** in the current system or the spike:
|
||
|
||
| Commitment | Scar it heals |
|
||
|---|---|
|
||
| Tx-first seam (`Begin` → domain ifaces → `Commit`) | `DoltStore`'s per-method commits = the stale-snapshot data-loss class (#3822) that forced the May pivot; command-scoped short tx is the proven concurrency unit. This is `uow.UnitOfWork` promoted to the seam. |
|
||
| ~7 small domain interfaces, no god interface | the 805-line panic wall (PR #3792); "can't easily mock DoltStorage"; 168 methods before one command works |
|
||
| `OpInfo` on `Begin` | H2: commit protocol currently lives in CLI globals (message-blanking, `commandDidExplicitDoltCommit`, tips double-commit) |
|
||
| `ChangeToken()` on core | backend-neutral freshness: `GetCurrentCommit`/`DOLT_HASHOF` is Dolt-specific, so export/backup freshness must key on a neutral opaque token any backend can supply (SQLite/Postgres have no Dolt commit hash). (NB: the earlier "H8 live bug" rationale is withdrawn — see H8; ChangeToken stands on backend-neutrality alone.) |
|
||
| per-instance `CapabilitySet`, extend-never-gate | `bd sql` works on Dolt-server, stubs on Dolt-embedded — capability varies by topology; the spike's only runtime divergence was a capability gating core behavior |
|
||
| topologies INSIDE `backend/dolt` | embedded/server/proxied are three connections to ONE engine; CLI knowledge of them produced 38 scattered mode checks |
|
||
| events written inside mutations, same tx | the spike's red team hit emit-outside-tx three times; the use-cases already do it right — make it structural + contract-tested |
|
||
| `backend/fake` + `storetest.RunContract` | nothing tests against the interface today; every suite needs a Dolt container |
|
||
| wisps as a tier param | today every use-case method is duplicated ×2 (41 where ~25 would do); the `--offset` union-paging divergence is a symptom |
|
||
| slices, not streaming iterators, in v1 | all 10 `Iter*` methods are slice stubs in the embedded backend anyway; streaming returns later as a measured need |
|
||
|
||
One-sentence summary: **the clean-room seam is Dustin's `UnitOfWork` widened to the full
|
||
domain surface, wearing bts-rs's construction/capability/error discipline.**
|
||
|
||
### 4.1 Bridge-seam arithmetic (the flat waypoint; review-corrected)
|
||
Core = today's `Storage` (62) **+ the six backend-neutral sub-interfaces (~45)** = ~107 methods,
|
||
+ `Transaction` (24), all mandatory for every backend. The ~155 cmd/bd call sites of the six
|
||
folded interfaces need **zero conversion**. The ~87 call sites of the 5 Dolt-shaped interfaces
|
||
convert to capability asserts. `DoltStorage` is deleted; the Dolt backends keep a `var _`
|
||
compile assertion for core + all capabilities. Phase 0's core-shrink review may pull some of
|
||
the ~107 above the seam (derivable) — treat any shrink as upside, not plan.
|
||
|
||
### 4.2 The WriteLifecycle contract (replaces the naive "PostWrite hook")
|
||
A specified seam protocol, designed in Phase 2b BEFORE code moves:
|
||
- `WriteSummary{didWrite, explicitCommitOccurred, readOnlyCommand, messageParts, deferredMeta}`
|
||
carried per command (absorbs today's `commandDidWrite`/`commandDidExplicitDoltCommit`
|
||
globals and the tips deferred-write protocol).
|
||
- Two seam calls bracketing the CLI's own export/backup steps: `PostWriteCommit(summary)`
|
||
(Dolt: auto-commit incl. the second tips commit; others: no-op) then — after the CLI's
|
||
auto-export/auto-backup, whose freshness check consumes a **neutral ChangeToken()** (core,
|
||
with the capability-shortcut-plus-fallback shape; motivated by backend-neutrality, since
|
||
`DOLT_HASHOF` is Dolt-specific — NOT by the withdrawn H8 "live bug", see H8) —
|
||
`PostWritePush(summary)` (Dolt: auto-push with its swallowed-error semantics; others: no-op).
|
||
- In-transaction suppression (`transactHonoringAutoCommit`'s blank-message trick) becomes an
|
||
explicit seam behavior: the backend decides mid-tx checkpointing from the summary + its own
|
||
mode; the message parameter on `RunInTransaction` stays but is re-specified as "operation
|
||
description; versioned backends MAY use it as a commit message" (D3).
|
||
- Autocommit-mode POLICY (today chosen by the CLI from `usesSQLServer()`) moves into the Dolt
|
||
backend's topology config — the CLI no longer knows modes exist.
|
||
|
||
### 4.3 Locator, legacy rule, and env precedence
|
||
- Locator in `.beads/metadata.json`: `{backend: "dolt"|"sqlite"|"postgres", ...}`. `dolt_mode`
|
||
becomes Dolt-internal topology config.
|
||
- **Legacy-value rule (review blocker):** the existing `backend` field is UNTRUSTED legacy —
|
||
workspaces in the wild may carry `backend:"sqlite"` from the pre-removal era
|
||
(removed in 87493ce91), and `GetBackend()`'s hard-return of "dolt" (`configfile.go:207-210`)
|
||
is a deliberate shim over exactly that. Rule: introduce a NEW locator key (e.g.
|
||
`storage_backend`, plus a metadata schema version); absent/legacy/unknown values + presence
|
||
of a Dolt data dir resolve to **dolt**, never to a new backend. A Phase 1 harness fixture
|
||
with a legacy `backend:"sqlite"` metadata.json pins this. This closes the trap where a legacy
|
||
value resolves to the new SQLite backend, opens an empty store, and H9 auto-import silently
|
||
re-seeds/forks the workspace.
|
||
- **Env precedence:** `BEADS_DOLT_*` (19 vars) are owned by the Dolt backend and are **inert
|
||
for non-Dolt locators**; env parsing consolidates behind the registry; the Phase 1 harness
|
||
pins the precedence table (H10). `--global`/shared-server: the CLI keeps a seam-level
|
||
"workspace redirection" concept whose only current provider is Dolt shared-server topology —
|
||
on SQLite/Postgres, `--global` errors with a capability stub (D2 pattern).
|
||
|
||
### 4.4 Errors, formatting, and the public API
|
||
- Typed errors below the seam (NotFound/ClaimConflict/Unsupported{op,backend}/retry-class);
|
||
ALL envelope/stream routing and contract strings above it — including gc error class 3
|
||
(`bd sql` embedded string), class 4 (H9 auto-import strings — now backend-neutral pinned
|
||
contract), and the claim-conflict text.
|
||
- Public API: `beads.Open*` route through the registry (H3 fifth path). **Compat note for
|
||
extension authors:** on non-Dolt workspaces, type-asserting `RemoteStore` etc. on the
|
||
returned `Storage` starts returning false — document in release notes; the assertion pattern
|
||
itself (`beads.go:27-31`) is unchanged.
|
||
- Single binary, no dynamic plugins. "Addons/packs" = capability-gated stubs inside one binary
|
||
+ build tags where deps demand (embedded Dolt is cgo-only; modernc SQLite is pure-Go → nocgo
|
||
builds gain a local backend for the first time, `beads_nocgo.go:27`).
|
||
|
||
### 4.5 Bridge vs destination — how the two relate
|
||
|
||
The mechanical phases (2a/2b/3) retype the CLI onto the FLAT core interface (§4.1) because
|
||
that keeps a 159-reference migration mechanical. That flat interface is **scaffolding, not
|
||
the destination**: from Phase 4 onward, the `uowStore` adapter's inner surface is built as
|
||
the PUBLIC clean-room seam (§4.0 `Tx` + domain interfaces) rather than as a private adapter
|
||
detail — so the destination accretes underneath the migration instead of requiring a second
|
||
one. Phase 7 then thins the flat layer opportunistically (or keeps it as sugar). Existing
|
||
components map as: `uow.UnitOfWork`/use-cases/`domain/db` → the seam's embryo;
|
||
`EmbeddedDoltStore`/`DoltStore`/`issueops` → `backend/dolt` interior; the 13 duals,
|
||
`DoltStorage`, the string lists, and the Seam-A parity tests → deleted.
|
||
|
||
### 4.6 Future-backend expandability (FDB, redb-class embedded KV, S3)
|
||
|
||
The seam must not assume SQL — two of the three named future backends are key-value and one
|
||
is object storage. Design constraints this imposes, to be validated by adversarial review:
|
||
- **FoundationDB:** interactive transactions exist but carry hard limits (~5s / ~10MB);
|
||
`bd import`-scale work cannot be one `Tx`. The seam therefore must NOT promise unbounded
|
||
transaction size — bulk operations need an explicit chunked/bulk path on the seam, and the
|
||
contract suite must pin what atomicity bulk ops actually guarantee. Ordered scans and
|
||
secondary indexes are backend-built (the bts-rs readiness-projection pattern applies).
|
||
- **redb-class embedded KV:** single-writer ACID — same shape as the SQLite slot. (redb
|
||
itself is a Rust library; in-process use from Go implies cgo/FFI or it arrives via the
|
||
bts-rs lineage — the seam question is only "can an embedded KV implement the contract",
|
||
and the spike's file-backed mem store already answered yes.)
|
||
- **S3:** no multi-key transactions. A conforming backend is manifest-CAS shaped (commit =
|
||
conditional-PUT of a new manifest; `Commit` returns a Serialization retry class on CAS
|
||
loss) — which the `Tx`+`RetryClass` model can express, but claim latency and contention
|
||
behavior must be honestly capability-flagged (likely `MultiWriter: false` or
|
||
claims-unsupported). S3 may be BETTER fitted as the target of RemoteSync/Backup/cold
|
||
history (the bts-rs BlockStore/RefStore substrate is S3-native with CAS on refs) than as
|
||
a primary Store; the seam should permit both without contortion. `ChangeToken` maps
|
||
naturally (manifest ETag).
|
||
|
||
### 4.7 Red-team revisions (Fable panel, 5 lenses, all verdicts holds-with-fixes) — NORMATIVE
|
||
|
||
These contract additions override/extend §4.0 where they conflict; fold into the sketch on
|
||
the next full edit. Convergent = demanded independently by 2+ lenses.
|
||
|
||
**Convergent (adopted):**
|
||
1. **`Store.Run(closure)` is the primary op; commands migrate ONTO it in Phase 7** (not off
|
||
it). Retry ownership lives on the seam; typed errors gain backend-supplied `RetryAdvice`
|
||
(backoff hint + budget); storetest includes an error-injecting fake and requires that a
|
||
conforming driver's advice lets 24 contending claimers converge (no livelock).
|
||
2. **CommandScope above Tx** (see §4.0 note): one version-commit per command, tips second
|
||
commit, commit→backup/export→push ordering; dolt-log divergence scenarios join the Phase
|
||
2b gate (stdout diffing alone cannot see N-commits-instead-of-one).
|
||
3. **Read-consistency contract:** statements within one Tx observe ONE snapshot (PG:
|
||
REPEATABLE READ or single-statement collapse; the two-phase ready-union read is the
|
||
motivating bug); within-Tx read-your-writes is contract-tested; read families are
|
||
classified single-snapshot-required (ready, claim-candidate, mutation-feeding reads) vs
|
||
chunk-consistent-permitted (export, audit, stats) — FDB's ~5s limit applies to READS too.
|
||
4. **Bulk/chunked port with pinned atomicity:** per-chunk atomic, idempotent upsert,
|
||
resumable journal; crash-mid-bulk + auto-import-resume (H9) fixtures in Phase 1;
|
||
`create --graph` atomicity resolved against `Limits()`.
|
||
5. **Wisp/tier respec:** tier is a DERIVED attribute (ephemeral/no_history/infra-config
|
||
routing — never caller-chosen); query methods take visibility flags, not a tier enum;
|
||
explicit cross-tier ops (PromoteFromEphemeral, AffectedByDeletion); **tiered-atomicity
|
||
contract** (Commit atomic per tier, pinned ordering, `PartialCommit` error class or
|
||
`AtomicTiers` flag); a tier×capability locality matrix (History/RemoteSync/Backup/export
|
||
MUST exclude local-tier data) with fixtures: purge-spares-no_history,
|
||
backup-excludes-wisps. All of this feeds D5.
|
||
6. **Ordering/predicate semantics become a written contract:** binary-codepoint collation
|
||
(PG DDL pins COLLATE "C"), second-precision timestamps, full tiebreak chains incl.
|
||
NULL-first, LIKE-wildcard/escape behavior, metadata `{"k": null}` counts as present;
|
||
clock ownership moves client-side (a `Clock` provider joins Identity on the construction
|
||
seam) so relative-time predicates and the 48h-hybrid ordering are deterministic +
|
||
contract-testable.
|
||
|
||
**Single-lens (adopted):**
|
||
7. **Mutation-observation contract** (repo lens, blocker): `Tx.Commit` yields a
|
||
`MutationSummary` (feedable from the same-tx event rows) consumed by a registry-owned
|
||
hook dispatcher; the decorator wraps Tx, not just Store; hook-artifact fixtures join the
|
||
Phase 1 harness; Phase 7 conversions gate on hook parity.
|
||
8. **Interleaving rule:** capability methods may not run between Begin and Commit/Rollback
|
||
on the same Store (typed error, storetest-pinned); per-topology connection budgets are
|
||
documented backend-author contract (embedded's eager second dolt_ignore connection binds
|
||
lazily or every command pays issue-#4303's dominant cost).
|
||
9. **Sync/archive TARGETS are a first-class seam concept distinct from Store** (S3 lens):
|
||
`BlockStore{Put,Get}`/`RefStore{Get,CAS}` target interfaces + per-scheme drivers
|
||
(s3/gs/file); `CapabilitySet.RemoteSync` rescoped to backend-NATIVE remotes (Dolt);
|
||
Phase 6 cold history and portable backup-to-URL route through targets — S3's best fit,
|
||
without pretending it is (or isn't) a primary Store. Backends whose commit protocol can
|
||
strand garbage (CAS-race orphans) MUST implement + declare Maintenance.GC; doctor
|
||
surfaces it.
|
||
10. **Error taxonomy additions:** `ErrBusy` (Transient; the embedded flock string is its
|
||
pinned rendering); claim errors carry structured holder/status (the gc class-2 string
|
||
is rendered ABOVE the seam from data, incl. idempotent self-reclaim success);
|
||
Open/Begin blocking-vs-fail-fast behavior specified per concurrency class; backends MAY
|
||
resolve CommitIndeterminate internally (verify-after-timeout) and return a definite
|
||
class.
|
||
11. **ChangeToken contract:** opaque, equality-only, data-changed ⇒ token-changed
|
||
(false positives permitted); Dolt = hash over non-ignored core tables (NOT HEAD);
|
||
per-tier advance semantics pinned; S3 = seq embedded in the manifest, not raw ETag;
|
||
FDB = versionstamped last-mutation key.
|
||
12. **Testing-story honesty** (repo lens): backend/fake + storetest get their own phase,
|
||
gate, and sizing row; evaluate a go-mysql-server-backed fake running the REAL SQL
|
||
(reuse, not a fifth semantics copy); Seam-A parity-suite deletion re-gated on
|
||
single-SQL-stack convergence inside backend/dolt (embedded TxProvider over the embedded
|
||
engine → delete issueops), not on dual-file collapse; §3.1's "integrations are free" is
|
||
WRONG — the tracker keys its Dolt fast path on `DB()` presence and must key on the
|
||
History capability (same audit for every RawDBAccessor consumer: SQL-speaking ≠
|
||
has-Dolt-system-tables).
|
||
13. **Streaming:** unbounded consumers (audit, sync, export) keep a cursor/`since+limit`
|
||
path (or a StreamingReads capability) before Phase 7 converts them — "slices only" is
|
||
bounded-by-workspace materialization, stated as such.
|
||
|
||
**§4.6 corrections from the future-backend lenses:** FDB limits apply to reads too (~5s
|
||
read-version expiry; 100kB value cap vs bd's unbounded LONGTEXT — chunked content or
|
||
declared `Limits`); the readiness-projection pattern transplants only with rework (chunked
|
||
catch-up outside the op Tx, lease-based projector); redb: what the spike validated is "an
|
||
embedded single-file backend behind the seam for the gc-16 surface" — the 9-sort-order
|
||
index design, ~107-method surface, and cross-process behavior remain unvalidated, and redb
|
||
has no C API (a Go binding does not exist; that backend arrives via FFI work or the bts-rs
|
||
lineage); S3 claims stay CORE (never "claims-unsupported") — the drain gate keys on
|
||
`Concurrency.ConcurrentWritersSafe` with a disclosed contention model; 412/409/503/timeout
|
||
→ RetryClass mapping table to be added.
|
||
|
||
## 5. Migration phases
|
||
|
||
Each phase lands green with the default backend byte-identical; each has a verification gate
|
||
that names its in_scope coverage.
|
||
|
||
### Phase 0 — Decisions + contract pinning (docs only)
|
||
- Resolve D1–D6 (§6) with maintainers.
|
||
- **Leak-4 audit** over capability-sourced behavior in core commands (restore's history
|
||
fallback; export/backup freshness → ChangeToken; `bd gc`'s portable decay phase; tracker
|
||
fast path). Classification defines the core command set.
|
||
- **Core-shrink review** (§2.1 caveat): triage the ~107 core methods into primitive /
|
||
derivable-above-seam / capability.
|
||
- Census additions from review: H9 auto-import; the 21 `commitPendingIfEmbedded` sites; the
|
||
seven issueops Dolt files; the WriteLifecycle protocol inventory (H2).
|
||
- Pin H8 behavior contracts (incl. the iterator and autocommit divergences) into
|
||
`BackendCapabilities` (`configfile.go:183-204`).
|
||
- **Gate:** signed-off decision doc; core-vs-capability census table; WriteLifecycle contract
|
||
spec reviewed.
|
||
|
||
### Phase 1 — The oracle before the surgery
|
||
**A new differential runner reusing `tests/regression` scaffolding** (workspace isolation,
|
||
binary management, container TestMain, the battle-tested normalizer) — NOT an extension of its
|
||
execution core: today's runner is 177 Go closures compared by end-state snapshot only, `t.Fatal`s
|
||
on any non-zero exit, and merges stdout+stderr (`regression_test.go:363-374,811-831`). New:
|
||
per-step recorder (exit code, split streams, JSON-aware diff), replayable scenario format,
|
||
multiset-default + `ordered` flag, in_scope predicate, backend/mode parameter on workspace
|
||
setup, property tests for minted IDs, the claim drain (multi-writer only), unique
|
||
candidate-binary paths.
|
||
- Scope: gc-contract 16 × their flags first (incl. **purge** — no uow dual, red-team history),
|
||
plus fixtures: legacy-locator (§4.3), auto-import class-4 trigger (H9), close-blocker→ready
|
||
and purge-orphan re-seed transitions (§2.3), the env-precedence table (H10).
|
||
- **Shake-out: Dolt-embedded vs Dolt-server.** Premise (review-corrected): these are KNOWN
|
||
DIVERGENT (H8) — the run formalizes the divergences, it does not validate on identical
|
||
engines. Allowlist discipline: entries are backend-PAIR-scoped, field-scoped, tagged
|
||
intentional-vs-known-bug, and expire with the Phase 2 scrub. **Mode waivers do not carry
|
||
into SQLite runs.** Expect the autocommit asymmetry to be the dominant waiver class. (The
|
||
earlier "expect to confirm the H8 live bug" line is withdrawn — H8's server-mode
|
||
export-freshness bug was refuted on verification; the server store commits per write and
|
||
auto-export is server-mode-disabled.)
|
||
- **Gate:** harness green on Dolt-vs-Dolt with a fully attributed allowlist; coverage report
|
||
artifact (scenarios × commands × flags) checked in.
|
||
|
||
### Phase 2a — Neutralize the core seam (the mechanical part)
|
||
- Fold the six neutral sub-interfaces into core (§4.1); retype the global `store` + decorator +
|
||
`molecules.NewLoader` + `remotecache.StoreOpener` + helpers from `DoltStorage` to the core
|
||
interface (**159 refs / 61 files + 15 test files**); convert the ~87 Dolt-shaped call sites
|
||
to UnwrapStore asserts; delete `DoltStorage`.
|
||
- Consolidate the FIVE construction paths (incl. public `beads.Open*`) into the registry;
|
||
registry owns decorator application (closing the existing lazy/read-only/routed hook gap);
|
||
new locator key + legacy rule (§4.3).
|
||
- Convert H6 string lists to declared per-command properties.
|
||
- **Gate:** full suite + Phase-1 harness Dolt-vs-Dolt: zero NEW divergences vs the Phase-1
|
||
allowlist.
|
||
|
||
### Phase 2b — The WriteLifecycle migration (the semantic part; own phase, own gate)
|
||
- Implement the §4.2 contract; migrate the 21 `commitPendingIfEmbedded`/`transact*` call
|
||
sites, PersistentPostRunE, the tips deferred-write protocol, ChangeToken for export/backup
|
||
freshness (on backend-neutrality grounds — the "fixes the silent server-mode
|
||
export-freshness bug" rationale is withdrawn; that bug was refuted on verification, so
|
||
ChangeToken is a portability requirement, not a bug fix, and needs no behavior-change release
|
||
note), retype H9 auto-import and pin its strings.
|
||
- **Gate:** Phase-1 harness re-run; the ONLY acceptable diffs are the enumerated bug fixes;
|
||
autocommit/push/export ordering pinned by new dedicated scenarios.
|
||
|
||
### Phase 3 — Capability-gate the addon surface
|
||
- `Capabilities()` is **instance-level** (computed after open, backend × topology — review
|
||
fix: `bd sql` works on Dolt-server but not Dolt-embedded, same backend). Gating happens at
|
||
**RunE-time via stubs** (D2 resolved): registration stays in `init()` (no bootstrap
|
||
restructuring; `bd --help` and the CI docs-regen stay deterministic); absent capability →
|
||
deterministic stub error naming the backend that provides it (federation_nocgo precedent;
|
||
`bd sql`'s embedded string preserved verbatim as contract).
|
||
- Surface: `bd dolt` (16 subcmds), `bd vc`, `bd branch`, `bd history`, `bd diff`,
|
||
`bd show --as-of`, `bd federation`, `bd backup`, `bd sql`, `bd flatten`, Dolt legs of
|
||
`bd compact`/`bd gc`, `db-proxy-child`, `--global` (§4.3).
|
||
- `bd doctor`: per-backend check registry (D6); Dolt checks (incl. `blocked_consistency`'s
|
||
`dolt_status` guard) move into the Dolt backend's set.
|
||
- Audit + flip the `--backend=sqlite` deprecation contract (`init.go:215-225` hard-errors
|
||
today; docs/scripts written against that string need a coordinated, release-noted flip).
|
||
- **Gate:** on Dolt, behavior + `--help` byte-identical; a test-only capability-less backend
|
||
exercises every stub; docs regen green.
|
||
|
||
### Phase 4 — Second backend: Postgres (THE PROOF backend; reordered per owner direction)
|
||
Owner-stated success criterion: **beads runs on a Postgres backend and the bd CLI simply
|
||
doesn't have the history functionality.** Postgres — not SQLite — is therefore the seam
|
||
validator and the critical-path deliverable. (The first draft's SQLite-first ordering
|
||
followed the spike's cheapest-second-backend guidance; the owner's proof criterion overrides
|
||
it.)
|
||
|
||
**Two candidate build routes — choosing is part of D1, because they hinge on the uow
|
||
relationship:**
|
||
- **Route A (recommended if D1 lands the adapter): ride the uow stack.** A Postgres
|
||
`TxProvider` (plain BEGIN/COMMIT; the commit-message param is ignored) + a dialect port of
|
||
the `domain/db` repositories (~75 methods of EXISTING, Seam-A-parity-tested SQL; the
|
||
MySQL-isms are enumerable — `ON DUPLICATE KEY`→`ON CONFLICT`, backticks, no `CALL DOLT_*`)
|
||
+ the `uowStore` adapter + the five gap units +
|
||
per-backend DDL. This REUSES the already-implemented dangerous semantics (same-tx events,
|
||
`is_blocked` propagation) instead of re-deriving them, and leans on the workstream's own
|
||
stated invariant (issue #4489): "the local-vs-proxied difference is confined to the uow
|
||
transaction layer (the mode-polymorphic db.Runner)."
|
||
- **Route B (fallback, if D1 keeps the layers separate): direct core implementation**
|
||
(~107+24 methods over a fresh schema), cribbing the bts-rs PG adapter wholesale (schema,
|
||
order/ready indexes + batched hydration, single-statement claim via `FOR UPDATE SKIP
|
||
LOCKED`, typed retry classes, optional readiness projection). Independent of the uow
|
||
roadmap; more code; a THIRD implementation of the business semantics to keep honest —
|
||
**the hard part is semantics, not SQL** (§2.3: same-tx events, `is_blocked` propagation,
|
||
delete/purge neighbour recompute, wisp/tier semantics, D5 local-tables).
|
||
|
||
**Proof-wedge milestone (demoable before Phases 2–3 fully land):** minimal locator `backend`
|
||
field + one factory arm + the PG stack + a TEMPORARY full-`DoltStorage` compat shell whose 37
|
||
Dolt-shaped methods return typed Unsupported errors (graceful errors, NOT panics — the D1
|
||
design-history lesson) + PersistentPostRun neutralized for non-Dolt. This demonstrates "bd on
|
||
Postgres, `bd history` absent" without waiting for the 159-ref retype; the shell is deleted
|
||
when Phase 2a lands. Accepted sharp edges for the demo: auto-export/backup freshness and
|
||
parts of `bd doctor` degrade on the wedge; the gc-16 harness subset is the demo's acceptance
|
||
test.
|
||
- **Gate:** Phase-1 harness PG-vs-Dolt green over the gc-16 in_scope set, THEN widened to the
|
||
CORE-67 surface before the backend is announced (coverage artifact enforces the widening —
|
||
no shipping green-on-16-only). Transition fixtures (§2.3) green. 24-process claim drain (PG
|
||
advertises multi-writer). **Remote deployment guidance shipped** — per-command
|
||
connection-setup cost measured, PgBouncer/co-location documented (§2.6).
|
||
|
||
### Phase 5 — Third backend: SQLite (the nocgo/embedded story; now optional)
|
||
Pure-Go (`modernc.org/sqlite`), documented single-writer (claim-drain gate N/A); fills the
|
||
nocgo hole (`beads_nocgo.go:27` — non-CGO builds currently have NO local backend). Under
|
||
Route A it rides the same production line (a SQLite `TxProvider` + dialect port); under
|
||
Route B, a direct implementation with the WAL lessons (`synchronous=NORMAL`,
|
||
`wal_autocheckpoint=0` + explicit checkpoint cadence, pragmas on the read pool, no unclosed
|
||
rows). Slice-iterator stubs are legal (embedded Dolt ships them; server streams — both
|
||
satisfy the interface).
|
||
- **Gate:** as Phase 4 minus the claim drain; single-writer lock UX tested. Export→import
|
||
adoption-path fidelity documented (events history, wisps, LocalMetadata do NOT obviously
|
||
round-trip — measure and state).
|
||
|
||
### Phase 6 — History as a portable capability (optional)
|
||
Changelog capability (same-tx field-delta log) + HistoryViewer-as-fold (§2.5). Dolt keeps its
|
||
native implementation. Mirror the spike-documented wire quirks (bare JSON array, PascalCase
|
||
keys, full snapshots, client-side `--limit`, volatile CommitHash normalized).
|
||
|
||
### Phase 7 — The end-game: promote the clean-room seam, thin the bridge (opportunistic)
|
||
Migrate commands off the flat conveniences onto `Begin/Tx/Commit` directly — file by file,
|
||
no flag day, each conversion deleting a `RunInTransaction` wart — until the flat layer is
|
||
thin enough to delete or small enough to keep as sugar. No deadline: the system is already
|
||
multi-backend and capability-gated before this phase starts. Exit criterion: the `store`
|
||
package's godoc IS the backend-author contract, and `storetest.RunContract` is the only
|
||
thing a new backend must pass before the differential harness.
|
||
|
||
### Explicitly deferred / out of scope
|
||
- In-place cross-backend DATA migration (`bd migrate --to-backend=X`) — the interim adoption
|
||
path is export→re-init→import with Phase-4-documented fidelity limits.
|
||
- Retiring the uow layer's 13 dual command paths (fate fixed by D1; execution follows Phase 3).
|
||
- YugabyteDB/FDB backends; dynamic plugin loading; backend-neutral federation.
|
||
|
||
## 6. Decisions required
|
||
|
||
- **D1 — Canonical seam AND proxied-server's fate** (one decision, not two). **Corrected after
|
||
git-history review: the uow/domain layer is NOT dormant** — it is an active workstream
|
||
(Dustin Brown / coffeegoddd, DoltHub), converting one command per PR since 2026-05-10 with
|
||
merges through 2026-06-24 (#4039–#4488: provider, list, update, delete, show, close, reopen,
|
||
ready, dep, config, context, query). The init gate is deliberate staging ("lifting the gate
|
||
is a separate, deliberate change", PR #4488), not abandonment. Its motivation is
|
||
**multi-process concurrency**: embedded Dolt is single-writer; a shared server + short
|
||
per-command transactions serves concurrent agents. "Freeze uow" is therefore not a viable
|
||
recommendation.
|
||
**Design-history fact that reframes this decision:**
|
||
a `storage.DoltStorage` implementation over the proxy WAS built first — `DoltServerStore`,
|
||
PR #3792, May 6–7 2026 — and died in four days as an 805-line wall of
|
||
`panic("unimplemented")` (128/132 methods), pivoted to the UOW design with only a two-line
|
||
TODO as rationale. It died because the god interface cannot be implemented incrementally
|
||
from scratch — a cause the uow workstream's own six weeks of use-case building has since
|
||
removed (the adapter delegates to them), and which Phase 2's interface shrink + capability
|
||
split removes for every future backend. "We tried a store over the proxy already" is
|
||
therefore true but not dispositive.
|
||
**Recommended reconciliation — compose, don't pick:** `storage.Storage` remains the backend
|
||
seam (what the CLI consumes; what SQLite/Postgres implement; where the capability/addon
|
||
model lives — use-cases have no answer for command gating). Build the missing **`uowStore`
|
||
adapter** (core `Storage` implemented over `UnitOfWork` use-cases; today no such adapter
|
||
exists and `store_factory.go:51/96/169` TODOs point the other way — this is the crux to
|
||
align with the uow owner). Consequences: proxied-server becomes a topology of the Dolt
|
||
backend behind the registry; the 13 `*_proxied_server.go` duals collapse; the domain
|
||
repositories over `db.Runner` (3 methods) become an ACCELERATOR for future SQL backends
|
||
(supply Runner+TxProvider+dialect, reuse the repos) rather than a rival seam; the
|
||
`Tx.Commit` = `CALL DOLT_COMMIT` (`uow/doltserver_tx.go:28`) adopts the §4.2 WriteLifecycle
|
||
contract one layer down. **Full sketch with the method-by-method mapping and gap analysis**
|
||
(headline: ~60% of core maps 1:1 to existing use-case
|
||
methods; the true remaining cost of full proxied coverage is five small gap units, not 54
|
||
more command conversions).
|
||
**The alternative (full use-case trajectory):** commands migrate to use-cases everywhere and
|
||
backends supply repositories. Coherent, but: ~54 more command conversions, dual
|
||
implementations + parity tests for the duration, non-SQL backends become impossible
|
||
(`Runner` assumes SQL), and the capability model is still needed regardless.
|
||
**Divergence governance either way:** the two paths already diverge in features (`--offset`
|
||
is proxied-only because the embedded issues+wisps union cannot page — PR #4488) — assign an
|
||
owner for keeping the paths semantically converged until they reunify, and add the
|
||
divergences to the Phase 1 allowlist as tagged, expiring entries.
|
||
**This decision requires the uow workstream owner in the room.**
|
||
- **D2 — Absent-capability UX. RESOLVED by review: stubs, not hidden.** Cobra registration
|
||
happens in package `init()` before workspace resolution; hiding requires pre-cobra locator
|
||
resolution and breaks help/docs determinism. Stubs preserve the gc error contract and the
|
||
federation_nocgo precedent. (If hiding is ever wanted, it is its own sized work item.)
|
||
- **D3 — `RunInTransaction` message param:** keep, re-specified per §4.2 (the blank-message
|
||
suppression semantics become explicit seam behavior; "advisory" was wrong).
|
||
- **D4 — MergeSlot\*/Slot\***: keep in core (bead-backed, portable); revisit when Postgres
|
||
lands (Phase 4) — real row locking may obviate them.
|
||
- **D5 — Wisps' dolt-local tables** on backends without an ignore concept: table-prefix
|
||
convention (recommended) vs separate file/schema. Feeds the Phase 4 schema design.
|
||
- **D6 — Doctor scoping:** per-backend check registry (recommended) vs neutral check
|
||
interfaces.
|
||
|
||
## 7. Risks
|
||
|
||
| Risk | Mitigation |
|
||
|---|---|
|
||
| Two-seam divergence widens while D1 is undecided (uow conversions merge weekly; `--offset`-class proxied-only features accumulate) | D1 decided WITH the uow owner in Phase 0; until then, new proxied features carry a convergence note + Phase 1 allowlist entry; the `uowStore` adapter collapses the duals once agreed |
|
||
| Legacy `backend:"sqlite"` metadata resolves to the new backend and auto-import forks data | §4.3 legacy rule (new locator key, dolt-default fallback) + Phase 1 fixture — was a review blocker, now designed-out |
|
||
| Silent drift during the Phase 2a retype | Harness EXISTS FIRST; zero-new-divergence gate; commit per file-cluster |
|
||
| WriteLifecycle migration changes semantics invisibly | Own phase (2b) + dedicated ordering scenarios; only enumerated bug-fix diffs accepted |
|
||
| Allowlist waivers rot and mask real regressions | Pair-scoped, field-scoped, tagged, expiring entries; no carryover to new-backend runs |
|
||
| Green harness over-trusted | §2.4 epistemics: hooks/timing/coverage limits stated per gate; coverage artifact required; CORE-67 widening enforced before Phase-4 announce |
|
||
| A from-scratch backend (Route B) re-opens the spike's transactional bug class | §2.3 rules are Phase 4/5 implementation requirements with mandatory transition fixtures; Route A sidesteps by reusing the parity-tested use-case semantics |
|
||
| `bd doctor`/backup partial coverage confuses users | Capability-aware stubs + doctor prints per-backend coverage |
|
||
| Public-API extensions break on non-Dolt workspaces | §4.4 compat note; assertions return false (defined behavior), release-noted |
|
||
| Schema drift between per-backend migration streams | Shared logical-schema contract suite each backend's fresh DB must pass |
|
||
|
||
## 8. Sizing (rough, sequential; review-adjusted)
|
||
|
||
| Phase | Size | Notes |
|
||
|---|---|---|
|
||
| 0 | ~1 week | decisions + leak-4 audit + core-shrink review + WriteLifecycle spec |
|
||
| 1 | 2–3 weeks | new runner on old scaffolding; gc-16 scope + fixtures (was 1–2; review: per-step core is new code) |
|
||
| 2a | 2–3 weeks | fold + retype (159 refs/61 files) + registry (5 paths) + metadata properties |
|
||
| 2b | 1–2 weeks | WriteLifecycle contract migration (semantic; own gate) |
|
||
| 3 | 1–2 weeks | instance-level capabilities + stubs + doctor registry + deprecation flip |
|
||
| 4 | Route A: 4–6 weeks / Route B: 5–8 weeks | Postgres proof: A = PG TxProvider + repo dialect port (~75 methods of existing SQL) + adapter + 5 gap units + DDL; B = ~107-method direct core + is_blocked/events semantics from scratch + schema; both + conformance to CORE-67 |
|
||
| 5 | 2–4 weeks (Route A) / 4–7 (Route B) | SQLite riding the same production line (A) or direct (B); nocgo hole |
|
||
| 7 | opportunistic | thin the flat bridge onto the §4.0 seam; no deadline |
|
||
| 6 | 2–4 weeks | optional; independent — and under Route A, the Changelog capability could later give PG history too (§2.5), making "no history on Postgres" a v1 statement, not a permanent one |
|
||
|
||
Phases 4/5 are independent of each other; 6 independent of both.
|
||
**Critical path to the owner's PROOF** ("bd on Postgres, no history"): 0 (D1 + route choice) →
|
||
1 (gc-16 harness subset) → Phase-4 proof wedge (~6–9 weeks). **Critical path to the full
|
||
end state** ("users choose a backend", shipped properly): 0 → 1 → 2a → 2b → 3 → 4
|
||
(~12–19 weeks sequential); the wedge work is on that path, not thrown away — only the compat
|
||
shell is deleted when 2a lands.
|
||
|
||
---
|
||
|
||
## Appendix: review provenance
|
||
|
||
Recon: 5 parallel read-only explorers (storage interface census; Dolt leak census; command
|
||
surface census; existing-abstraction assessment; bts-rs learnings extraction) — ~578k tokens,
|
||
311 tool calls. Adversarial pass: 2 reviewers (feasibility-vs-repo, completeness-vs-spike),
|
||
verdicts sound-with-fixes; 2 blockers, 8 majors, 5 minors, 13 missing items — all folded into
|
||
this revision. Notable review discoveries retained above: the six-neutral-interfaces blocker
|
||
(§4.1), the legacy-locator trap (§4.3), the WriteLifecycle protocol structure (§4.2/H2), the
|
||
instance-level capability requirement (Phase 3), ~~the H8 live bug (server-mode export
|
||
freshness)~~ (corrected after verification: refuted — the server store commits per write and
|
||
auto-export is server-mode-disabled; ChangeToken is retained on backend-neutrality grounds),
|
||
and the H9 auto-import hazard. Clean-room shred: 5-lens Fable panel
|
||
(object-model, repo-reality, FDB, embedded-KV, S3) — all verdicts holds-with-fixes; 2
|
||
blockers (outcome-derived commit messages vs OpInfo-at-Begin; hook-observation contract), 17
|
||
majors; all folded as §4.7.
|