beads/PROPOSAL-pull-config-wedge.md
Steve Yegge 387959695d
fix(dolt): pre-pull commit must include config (kv.memory.* pull-wedge) (#4412)
* fix(dolt): pre-pull commit must include config; auto-resolve convergent kv.memory.* config conflicts

bd remember stores persistent memories as kv.memory.* rows in the synced
config table. Commit() deliberately excludes config (GH#2455, to avoid
sweeping a concurrent writer's stale issue_prefix into an unrelated commit),
so the GH#2474 pre-pull auto-commit at pullFromRemote (store.go) and PullFrom
(federation.go) leaves config permanently dirty once any memory exists. The
"clean the working set before merging" step then leaves config dirty and
DOLT_MERGE refuses to start ("cannot merge with uncommitted changes"). Two
correct PRs colliding over config. Diagnosed from a live server-mode repro
(Wyvern wy-71t); see PROPOSAL-pull-config-wedge.md.

Fix (server mode): add commitBeforePull, which stages config explicitly via
DOLT_ADD rather than relying on CommitWithConfig's DOLT_COMMIT('-Am'). The
proposal author observed -Am not staging config reliably under the server-mode
stored-procedure path, and the new embedded test TestEmbeddedConfigCommitStagesConfig
confirms -Am DOES stage config in the embedded engine — so the unreliability is
server-mode-specific and explicit staging is the robust choice. Commit() and
commitBeforePull now share commitWorkingSet(includeConfig bool).

Hardening (both modes): making config a committed/synced table means two clones
editing the same memory key collide on a config merge conflict. Extend
TryAutoResolveMergeConflicts to auto-resolve config conflicts with --theirs ONLY
when every conflicted row is a kv.memory.* key (machine-convergent, same policy
as metadata); any other config key (issue_prefix above all) is left for the
operator. Without this the primary fix would trade a silent pull wedge for an
operator-visible conflict wedge on the exact bd remember workload.

Embedded pre-pull (CommitPending -> -Am) already stages config and is unchanged;
it gains the resolver hardening.

Tests: server-mode pull_config_wedge_test.go (commit-includes-config + resolver
positive/negative, run under Docker in CI) and embedded pull_config_wedge_test.go
(in-process: -Am-stages-config probe + resolver positive/negative, all green).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: add pull-config-wedge diagnosis/proposal (PR #4412)

The diagnosis-and-patch write-up behind this PR: how GH#2455 (Commit excludes
config) and GH#2474 (pre-pull auto-commit) collide over the synced config table,
leaving kv.memory.* rows permanently uncommitted so DOLT_MERGE refuses to start.
Includes the live-DB validation (Wyvern wy-71t) and the rejected alternatives.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dolt): adopt-pr maintainer fixes for pull/sync config wedge

Maintainer review fixups for PR #4412, squashed into one commit on top of
the contributor's branch.

Pre-merge config wedge (sync): bd federation sync committed with the
config-excluding Commit before its merge while this PR switched the pull
paths to commitBeforePull, so any DB that ran `bd remember` carried dirty
kv.memory.* config rows and hit the "cannot merge with uncommitted changes"
wedge. Sync now uses commitBeforePull too, with a regression test.

Pre-pull auto-commit safety (GH#2455): commitBeforePull no longer stages the
whole config table. It auto-commits only convergent memory rows
(kv.memory.*); if any other config key is dirty (issue_prefix above all) it
refuses with operator guidance instead of sweeping a stale value into the
merge basis, restoring the GH#2455 protection the broad pre-pull staging had
reopened.

Sync --strategy config commit: a config-only conflict resolution committed
through the config-excluding Commit, so it could report success without
creating the merge-resolution commit and re-wedge the next sync. Sync now
commits resolutions through a config-inclusive path (commitMergeResolution).

Embedded sync parity: EmbeddedDoltStore.Sync fetched and merged without the
pre-merge CommitPending guard already used by embedded Pull/PullRemote/PullFrom
and the new server Sync, so the same dirty kv.memory.* working set could still
wedge `bd federation sync` in embedded mode. Sync now commits pending changes
before fetch/merge (embedded Commit is DOLT_COMMIT('-Am'), which stages
config), with an embedded regression test that drives Sync with a dirty
kv.memory.* row.

KV/memory namespace collision: `bd kv set memory.*` stored to kv.memory.*,
indistinguishable from a `bd remember` memory the merge resolver auto-resolves
with --theirs. validateKVKey now reserves the memory. prefix so a generic KV
value can't be silently overridden by the remote on pull.

Shared prefix + tests: centralize the kv.memory. prefix in
internal/storage/kvkeys (used by cmd/bd and the merge resolver) with a
contract test; route the remaining cmd/bd kv. prefix literals (the doctor KV
check, the validateKVKey nested-prefix guard, and the memory listing) through
the kvkeys constants; and strengthen the server-mode non-memory config
conflict test to prove a conflict materialized before the resolver declines it.

Maintainer review fixup for PR #4412.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <ci@beads.test>
2026-06-19 21:31:33 -07:00

173 lines
8.9 KiB
Markdown

# Proposal: fix `bd dolt pull` "cannot merge with uncommitted changes" wedge
**Status:** for review (diagnosis + patch). Authored 2026-06-14 from a live repro in
the Wyvern beads DB (server mode, git-protocol remote on Bitbucket).
## Symptom
`bd dolt pull` always fails:
```
Error 1105 (HY000): cannot merge with uncommitted changes
```
`bd doctor` reports `Dolt Locks: config: modified` immediately after every command,
even right after `bd dolt commit`, and even with `--dolt-auto-commit on`. Tracked as
Wyvern wy-71t. Push is unaffected; only pull (the cross-clone *receive* path) is wedged.
## Root cause — two correct PRs collide over the `config` table
1. **Memories live in `config`.** `bd remember` stores each persistent memory as a
`kv.memory.<slug>` row in the **synced** `config` table. Confirmed via the working
set on the live DB — the only dirty rows were exactly the 5 memories:
```
SELECT to_key, diff_type FROM dolt_diff('main','WORKING','config');
kv.memory.beads-jsonl-sync-procedure added
kv.memory.bitbucket-altssh-dead added
kv.memory.local-dev-ssl-posture added
kv.memory.react-client-e2e-against-the-local-game-server added
kv.memory.react-e2e-same-character-parallel-logins added
```
2. **`Commit` deliberately excludes `config`** (GH#2455, `store.go` ~1731):
`DOLT_COMMIT('-Am', …)` once swept stale `issue_prefix` changes from concurrent
operations into commits, so `Commit` now stages every dirty table *except* `config`.
3. **The pre-pull auto-commit uses `Commit`** (GH#2474). Both pull paths try to clean
the working set before merging, but call `s.Commit`, which skips `config`:
- `internal/storage/dolt/store.go:2309` — `pullFromRemote` (default remote; what
`bd dolt pull` invokes via `st.Pull`)
- `internal/storage/dolt/federation.go:65` — `PullFrom` (named-peer pull)
**Net effect:** once any memory exists, the `kv.memory.*` rows sit permanently
uncommitted (nothing ever commits `config`), so the pre-pull "clean the working set"
step leaves `config` dirty, and `DOLT_MERGE` refuses to start — the exact case
`abortMerge`'s comment already calls out: *"a merge that REFUSED TO START on a dirty
working set."* GH#2474's intent is silently defeated by GH#2455 whenever `config` is
the dirty table, which is always after the first `bd remember`.
This is independent of the v42→v51 schema-migration fork (#4259) seen in the same DB;
that was a separate, newer problem. Migrating + force-publishing fixed the fork and
restored `push`, but `pull` stays wedged because of this config issue.
## Empirical validation (live DB)
Confirmed the mechanism end-to-end without any source change:
1. The only dirty table was `config`; the only dirty rows were the 5 `kv.memory.*`
entries (see query above).
2. Explicitly committing config — `CALL DOLT_ADD('config'); CALL DOLT_COMMIT('-m', …)` —
left the working set **clean** (`dolt_diff('HEAD','WORKING','config')` → 0 rows).
3. Running ordinary commands afterward (`bd list`, `bd show`) kept it clean (still 0).
So bd's per-command memory re-sync is **idempotent**: once the rows match HEAD,
re-writing identical values produces no diff. The "re-dirties after every command"
symptom existed *only* because the rows were never committed (Commit skips config),
so they showed as permanently uncommitted. Committing config once breaks the cycle.
This proves a clean `config` working set is the *only* thing the merge needs, and that
a pre-merge commit of config is a permanent fix, not a per-command band-aid.
**Confirmed:** after committing config, `bd dolt pull` succeeds ("Pull complete.")
repeatably, `bd dolt push` still works, and `bd doctor` no longer reports
`config: modified`. The wedge is entirely the uncommitted-config condition; the patch
just makes the pull path establish it automatically. (Without the patch the wedge
returns on the next `bd remember`, which adds a fresh uncommitted `kv.memory.*` row.)
**One wrinkle to verify in the patch:** a standalone `bd dolt commit` (which already
routes through `CommitWithConfig` → `DOLT_COMMIT('-Am', …)`) did **not** visibly clean
`config` in this server-mode DB, whereas the explicit `DOLT_ADD('config')` +
`DOLT_COMMIT('-m', …)` did. That suggests `-Am` may not be staging `config` reliably
under the SQL stored-procedure path here. The patch author should confirm `CommitWithConfig`
actually commits `config` in server mode; if not, the pre-pull step should stage config
explicitly (`CALL DOLT_ADD('config')` then commit) rather than rely on `-Am`. A regression
test should assert `dolt_status` is empty immediately after the pre-pull commit.
## Fix (primary) — pre-pull commit must include `config`
`CommitWithConfig` already exists (`store.go:1780`) for exactly this "commit everything,
including intentional config" case — `CommitPending` (`bd dolt commit`) already uses it.
Use it in the two pre-pull sites:
```diff
--- a/internal/storage/dolt/store.go
+++ b/internal/storage/dolt/store.go
@@ pullFromRemote (GH#2474 pre-pull auto-commit)
- if err := s.Commit(ctx, "auto-commit before pull"); err != nil {
+ // Include config: persistent memories live in config as kv.memory.* rows,
+ // and Commit() excludes config (GH#2455), so they never commit and leave
+ // the working set permanently dirty — DOLT_MERGE then refuses to start.
+ if err := s.CommitWithConfig(ctx, "auto-commit before pull"); err != nil {
if !isDoltNothingToCommit(err) {
return fmt.Errorf("failed to commit pending changes before pull: %w", err)
}
}
```
```diff
--- a/internal/storage/dolt/federation.go
+++ b/internal/storage/dolt/federation.go
@@ PullFrom (GH#2474 pre-pull auto-commit)
- if err := s.Commit(ctx, "auto-commit before pull"); err != nil {
+ if err := s.CommitWithConfig(ctx, "auto-commit before pull"); err != nil {
if !isDoltNothingToCommit(err) {
return nil, fmt.Errorf("failed to commit pending changes before pull: %w", err)
}
}
```
### Why this does not reintroduce GH#2455
GH#2455 guards against `-Am` sweeping a *concurrent writer's* half-applied `issue_prefix`
change into an unrelated commit. The pre-pull commit is different in kind: it is an
explicit user action (`bd dolt pull`) that **requires** a clean working set, and it is
committing *this clone's own* config state as the basis for the merge — exactly what
`CommitPending` already does for `bd dolt commit`. The race window GH#2455 worried about
(another bd process mutating `issue_prefix` at the same instant) is not meaningfully
widened by committing config here versus letting the user's next `bd dolt commit` do it.
## Fix (optional hardening) — auto-resolve convergent `config` merge conflicts
With the primary fix, `config` (incl. `kv.memory.*`) becomes a committed, synced table,
so two clones that edit the *same* memory will produce a `config` merge conflict.
`TryAutoResolveMergeConflicts` (`mergesettle.go`) currently handles only `metadata`,
`dependencies`, and `schema_migrations` — **not** `config` — so such a conflict would
stop the pull for the operator.
Recommend a follow-up that teaches the resolver to auto-resolve `config` conflicts on
machine-convergent keys (e.g. `kv.memory.*`) with `--theirs`, the same convergent policy
already used for the `metadata` table. Out of scope for the wedge fix itself; the primary
change resolves the reported bug. (Until then, divergent same-memory edits are a rare,
operator-visible case rather than a silent wedge.)
## Alternative considered — move memories to a `dolt_ignore`'d table
Rejected: memories are meant to **sync** (they round-trip through `bd export | bd import`,
and are part of cross-clone shared state). `dolt_ignore` (as used for wisps) would make
them machine-local and stop syncing them via Dolt — a behavior change, not a fix.
## Test plan
Regression test (alongside `pull_conflict_test.go` / `federation_pull_settle_test.go`):
1. Two clones of a tiny remote-backed DB.
2. On clone A: `bd remember --key t "x"` (dirties `config` with a `kv.memory.*` row,
committing nothing via the normal path).
3. On clone B: create+push an issue.
4. On clone A: `Pull` — **must succeed** (today it returns
`cannot merge with uncommitted changes`), and clone A must end with B's issue *and*
its own memory intact.
Manual repro on the Wyvern DB: a fresh `bd dolt pull` fails today purely on the 5
`kv.memory.*` rows; with the patch the pre-pull `CommitWithConfig` commits them and the
merge proceeds.
## Files
- `internal/storage/dolt/store.go:2309` (`pullFromRemote`) — primary
- `internal/storage/dolt/federation.go:65` (`PullFrom`) — primary
- `internal/storage/dolt/store.go:1780` (`CommitWithConfig`) — reused, unchanged
- `internal/storage/versioncontrolops/mergesettle.go` (`TryAutoResolveMergeConflicts`)
— optional hardening
```