* fix(schema): stop the smart gate auto-migrating a data-behind clone The smart migrate gate's first-mover verdict was computed from schema facts alone: equal content hashes against the cached remote ref, equal max version, and at-or-above the convergence floor. None of those say anything about where the clone's branch sits relative to the ref the hashes came from, so a clone that was level with the remote on SCHEMA but behind it in unpulled DATA commits satisfied every precondition and was classified a safe first-mover. It then migrated in place, minting local-only schema commits on a HEAD missing the remote's data commits. When the pending batch moves a tracked table onto the dolt_ignore plane (0062's events), every subsequent `bd dolt pull` refuses to merge (#6368) and the clone can no longer fetch the very commits it was already behind on. v1.3.0's upgrade audience is largely 1.1-era databases, which is exactly the cohort reaching this state. The decisive fact was already wired: routeSmartGate's remote-ahead branch consults adopt.IsStrictAncestor (ahead == 0 && behind >= 1) via routeAdoptFastForward, and both production call sites supply it. The equal-version path simply never read it. Read it there too, before granting the automatic migrate, and stop with a new data-behind fallback reason naming the one-command remedy. The predicate does the narrowing for free: a level clone (ahead 0, behind 0) and a clone with unpushed local commits (ahead > 0) both read false and still auto-migrate, so only a strictly-behind clone is refused. An inconclusive ancestry read degrades to the blunt #4515 block rather than assuming level, matching the file's contract that no fallback is less safe than the blunt gate. Below-floor outcomes are untouched — the check sits inside the floor arm. The stop reuses the existing blunt RemoteMigrateGateError with an empty Decision, so there is no change to the JSON or agent contract. Both shipped escape hatches still work unchanged: --force/BD_ALLOW_REMOTE_MIGRATE=1 are consulted before the smart gate, and BD_SMART_GATE=0 still yields the blunt block. Two details the reason text has to get right. The "and the migrate proceeds on its own" promise is suppressed on a shared store, where the first-mover arm stays disabled regardless (#5920) and the retry still needs consent — promising otherwise would loop the operator. And an unparseable BD_SMART_GATE value does not outrank this reason: like shared-store, the stop stands whatever the env says, and it is the only reason carrying a remedy the operator can act on now, so burying it behind an env complaint would cost them the fix. Regression test builds a genuinely data-behind clone against a real Dolt server rather than stubbing the fact the bug is about: a source regressed one migration below latest publishes C0 to a file:// remote, a real DOLT_CLONE takes C0, the source makes a pure DATA commit C1 (no schema change, so both stay at the same schema version) and pushes, and the clone runs DOLT_FETCH only. The behind-ness is then measured, not assumed. The test also proves the remedy loop self-heals: after the fast-forward the same gate call auto-migrates the clone as a true first-mover, so the fix does not over-refuse. Fixes #6575 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(schema): give the data-behind stop a working remedy, and widen it Review follow-up on the #6575 fix. The stop was right; what it told the operator to do was not reachable, and it protected the rarer half of the cohort. F1 (blocker): the refusal prescribed `bd dolt pull`, and `bd dolt pull` was refused by it. Measured on a real data-behind embedded clone (ahead=0, behind=1, clean working set) at the previous head: the pull exited 1 printing "Run `bd dolt pull` first, then retry"; `bd sync` and `bd init --reinit-local` exited 1 the same way; `bd bootstrap` reported "Database already exists … Nothing to do." and exited 0; `bd migrate schema --force` then `bd dolt push` left ahead=1 behind=1 with the push rejected non-fast-forward. The only thing that moved the clone was BD_ALLOW_REMOTE_MIGRATE=1 — performing the migration the stop exists to prevent. An embedded workspace has no external `dolt` binary either, so that was a fence with no gate. It is the #4566 deadlock one refusal over, so it takes #4566's shape of fix: a lenient open for the command whose whole purpose is to clear the refused precondition. Narrower than #4566's, though — the remote-sync open tolerates ONE gate reason (RemoteMigrateGateError.IsDataBehind) and nothing else. A pull cannot resolve a fork skew, a below-floor database, a shared-store consent decision or an opted-out blunt block, so waving it through those would buy nothing and hide them; the dirty-table and dependency-re-key refusals stay fatal on this open too. Broadening the existing LenientOpen classification instead would have granted all of that in one line. Verified end to end on both shapes, exit codes real: `bd dolt pull` 0 (0,1 -> 0,0), retrying the refused write auto-migrates as a true first-mover, `bd dolt push` 0. On a diverged clone the pull merges: 0 when it can, 1 with "merge conflicts in config require operator resolution; merge aborted and working set restored" when it cannot, and `bd dolt pull --strategy theirs` then 0 — which is why the diverged guidance names it. F2 (high): LocalIsStrictAncestorOf is ahead == 0 && behind >= 1, so a clone with any local commit of its own read "not behind" and still auto-migrated. Measured at the previous head with ahead=1, behind=1: "safe first-mover", and the migration ran. bd auto-commits every write, so behind-AND-ahead is the ORDINARY multi-machine state and the shape the fix protected was the narrower one. It reaches the identical wedge — the migration lands on a history missing the remote's commits either way — so the predicate is now behind >= 1 whatever ahead is, read from LocalAheadBehind (the counts LocalIsStrictAncestorOf was already computing and discarding). A level clone and a purely-ahead clone have nothing to pull and still auto-migrate, verified through the CLI. The remedy differs by shape even though the command does not, so the two stay separate verdicts all the way out to the message, the agent directive and the JSON: one is told its pull fast-forwards, the other that it merges and how to resolve conflicts. An adopter that wires only the older callback keeps the narrower (pre-widening) reading rather than losing the stop. F3 (medium): with data-behind outranking fallbackReasonSharedStore, a shared store's refusal rendered with neither "co-resident" nor "#5920" anywhere — Options()' sharedRisk only ever decorated the adopt option. The #5920 consequence and the `bd migrate schema` consent step the retry needs there now travel with this reason, on every surface. F4 (medium): --json emitted only fallback_reason: "data-behind" while observed/expected/options stayed the blunt default, whose option 1 is ["bd migrate --force", "bd dolt push"] — the bug followed by a push that is guaranteed rejected while the clone is still behind — and whose option 2 no-ops. The pull-first remedy lived only in prose --json never emitted. observed/expected now describe this stop, options is the single pull-first option, and data_behind_shape says which pull to expect. decision stays absent and fallback_reason unchanged, so the key agents match on does not move. The blunt body is REPLACED for this reason rather than decorated: both of its options were measured wrong here, and leaving them under a note saying "pull first" is what sent the last round back into the wedge. Error() is deliberately untouched — the headline stays the blunt #4259 line, and the body's first sentence accounts for it. Both escape hatches are unchanged and re-measured: `bd migrate --force` and BD_ALLOW_REMOTE_MIGRATE=1 are still consulted before the smart gate (exit 0), and BD_SMART_GATE=0 still yields the blunt gate unconditionally — including for `bd dolt pull`, which is what opting out means. The two corrections from the first round are preserved: the "proceeds on its own" promise is still gated on !e.Shared, and the unparseable-BD_SMART_GATE carve-out still lets data-behind outrank an env complaint (BD_SMART_GATE=maybe re-measured). Tests cover both shapes at three levels. The schema routing table pins the widened predicate, the shape split, the error degradation, the IsStrictAncestor fallback, and every remedy surface including IsDataBehind's narrowness; neutralizing the widening fails three of its subtests. The real-Dolt regression test now runs both shapes over the same two-clone file:// fixture, measuring ahead/behind rather than stubbing it and asserting that the strict-ancestor relation is false exactly where the diverged shape needs the widening. A new embedded test builds a genuinely data-behind embedded store, proves OpenForRemoteSync gets through the refusal without migrating, and proves BD_SMART_GATE=0 on the same fixture does NOT. No S0 matrix cells flip. No matrix fixture configures a Dolt remote, so routeSmartGate stays unreachable; the only `dolt pull` probes are the proxied-mode typed refusals, which are classified in historyCapabilityMatrix before any store opens and are untouched. Refs #6575 Refs #6368 Refs #4566 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(schema): retarget the shared data-behind consent verb under --global Self-review catch on the previous commit's own new text. The #5920 consent verb the shared data-behind guidance names is TARGET-SCOPED: under --global the refused open aimed at beads_global, so the project-scoped `bd migrate schema` would consent the wrong database and leave the refusal in place. The existing shared-no-remote arm already handles this (printGlobalDatabaseConsentHint and the JSON retarget), and the new arm had to as well. The JSON retarget could not simply reuse the Decision test: this stop reaches the renderer through the default empty-Decision arm, keyed on the fallback reason, so `e.Decision == "shared-no-remote"` misses it. The human body now names the --global variant beside the project-scoped one, exactly as the shared-no-remote body does. The pull itself is target-agnostic and is deliberately NOT rewritten. Refs #6575 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(schema): give the data-behind read path the pull remedy too Round 2 fixed the write path and the agent payload, but left the two lenient embedded opens rendering the blunt #4259 coordination block. Those opens do not fail on this refusal - they warn and carry on - so the warning IS the whole contract for `bd list`, `bd ready`, `bd show` (openReadOnlyCommand) and `bd dolt commit` (openWorkingSetReconcile). Both printed `gateErr` (Error(), the one-line summary) followed by the unchanged sharedGuidance bullets, so on a data-behind clone the first thing bd said was: • designated migrator (only ONE machine): bd migrate --force && bd dolt push • every other clone (another already migrated): bd bootstrap Measured on a fast-forward clone at the previous head: that migrate lands v67 on a history the clone is still behind on, and the `bd dolt push` after it exits 1, rejected non-fast-forward - the #6575 state, reached by following bd's own instruction, from the same clone whose write path says in capitals not to run it. `bd bootstrap` no-ops against an existing workspace. `bd list` is the first thing a 1.1-era upgrader runs and what agents run constantly, so this was the most-read surface of the whole stop. Both arms now branch on gateErr.IsDataBehind() and print gateErr.UserMessage(), which is where the shape-branched pull-first body already lives - the same thing server mode's warnLenientOpenRefusal has printed all along. The intent framing is kept rather than replaced: the command in hand is still succeeding against the old schema, which the body (written for the fatal refusal) does not say. Refusals that are not the data-behind stop keep the coordination bullets, which are right for them; a second test pins that so the new body cannot swallow them. Verified by running, not asserting - `bd list` and `bd dolt commit` on real data-behind clones of both shapes, then `bd dolt pull` and a re-run, which migrates. Also fixes the agent payload contradicting itself. For data-behind, AgentDirective calls `bd dolt pull` the single option, Options annotates it `when: always, for this stop` and `risk: none - a pure fast-forward`, yet human_decision_required stayed hard-coded true, so an agent keying on that field stalled for approval of a step the same payload calls unconditional and riskless - and the cost of that stall is reaching for `bd migrate --force`. It is now false for exactly that arm. It stays true when DataDiverged (the pull merges and can need conflict resolution and a --strategy choice) and when Shared (the payload's second option needs an operator who can confirm every co-resident client is upgraded, #5920); getting either backwards would be worse than the contradiction. The `when` string is deliberately unchanged for both shapes - the pull is the only way forward either way; it is the risk, not the applicability, that diverging changes. Refs #6575 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
148 lines
5.8 KiB
Go
148 lines
5.8 KiB
Go
//go:build cgo
|
|
|
|
package embeddeddolt
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"path/filepath"
|
|
"sync"
|
|
)
|
|
|
|
var (
|
|
cacheMu sync.Mutex
|
|
cache = make(map[string]*cacheEntry) // keyed by absolute dataDir
|
|
)
|
|
|
|
type cacheEntry struct {
|
|
store *EmbeddedDoltStore
|
|
refCount int // guarded by cacheMu
|
|
}
|
|
|
|
// Open returns a cached EmbeddedDoltStore for the given data directory, creating
|
|
// one via newStore if no cached instance exists. Subsequent calls with the same
|
|
// resolved dataDir return the existing store and increment a reference count.
|
|
//
|
|
// Each Open must be paired with a Close. The underlying store is only truly
|
|
// closed when the last reference calls Close.
|
|
//
|
|
// This prevents redundant engine initializations when multiple code paths open
|
|
// connectors against the same data directory in the same process.
|
|
func Open(ctx context.Context, beadsDir, database, branch string) (*EmbeddedDoltStore, error) {
|
|
return openCached(ctx, beadsDir, database, branch, openStrict)
|
|
}
|
|
|
|
// OpenForReadOnlyCommand opens like Open, except that a #4259 remote-migrate
|
|
// gate refusal skips the pending migrations with a stderr warning instead of
|
|
// failing the open (bd-578h9.5): the gate exists to stop in-place migration,
|
|
// not reads, and server mode's read-only opens already skip migration. The
|
|
// returned store is otherwise a normal writable store (read-only commands
|
|
// can still write incidentally, e.g. the post-command autocommit net).
|
|
func OpenForReadOnlyCommand(ctx context.Context, beadsDir, database, branch string) (*EmbeddedDoltStore, error) {
|
|
return openCached(ctx, beadsDir, database, branch, openReadOnlyCommand)
|
|
}
|
|
|
|
// OpenForWorkingSetReconcile opens like Open, except that a schema.
|
|
// DirtyTablesError refusal skips the pending migrations with a stderr
|
|
// warning instead of failing the open (#4566): commands whose entire purpose
|
|
// is to reconcile the Dolt working set (bd dolt commit, bd vc commit) must be
|
|
// able to open the store even when pending migrations touch dirty tables -
|
|
// otherwise the commit that would clear the dirty state and unblock the
|
|
// migration can never run. The returned store is otherwise a normal writable
|
|
// store.
|
|
func OpenForWorkingSetReconcile(ctx context.Context, beadsDir, database, branch string) (*EmbeddedDoltStore, error) {
|
|
return openCached(ctx, beadsDir, database, branch, openWorkingSetReconcile)
|
|
}
|
|
|
|
// OpenForRemoteSync opens like Open, except that the #6575 data-behind
|
|
// remote-migrate gate refusal — and only that one — skips the pending
|
|
// migrations with a stderr warning instead of failing the open: that refusal's
|
|
// entire remedy is `bd dolt pull`, which opens the store itself and so hit the
|
|
// refusal that prescribed it, leaving the refused clone with no in-band exit at
|
|
// all (the same deadlock #4566 broke for `bd dolt commit`). Every other gate
|
|
// refusal, the dirty-table guard and the dependency re-key still fail this open
|
|
// exactly as they fail a strict one. The returned store is otherwise a normal
|
|
// writable store.
|
|
func OpenForRemoteSync(ctx context.Context, beadsDir, database, branch string) (*EmbeddedDoltStore, error) {
|
|
return openCached(ctx, beadsDir, database, branch, openRemoteSync)
|
|
}
|
|
|
|
func openCached(ctx context.Context, beadsDir, database, branch string, intent openIntent) (*EmbeddedDoltStore, error) {
|
|
key, err := cacheKey(beadsDir)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
cacheMu.Lock()
|
|
if entry, ok := cache[key]; ok {
|
|
// Cache hit: the requested intent is ignored - the store keeps
|
|
// whatever intent it was opened with on the slow path below. This is
|
|
// safe today because intent is derived once per process from the
|
|
// command classification (isReadOnlyCommand /
|
|
// isWorkingSetReconcileCommand / isRemoteSyncCommand in
|
|
// cmd/bd/main.go), so a single process never opens the same data
|
|
// directory under two different intents, and autoMigrateOnVersionBump
|
|
// (cmd/bd/version_tracking.go) always opens its own openStrict store
|
|
// and closes it before the main command's open runs, so it never
|
|
// races a cache hit either. A future caller needing a genuinely
|
|
// different intent on a cache hit would have to plumb intent through
|
|
// here instead of silently reusing the first store's; it is load-
|
|
// bearing that no current caller does.
|
|
entry.refCount++
|
|
cacheMu.Unlock()
|
|
return entry.store, nil
|
|
}
|
|
cacheMu.Unlock()
|
|
|
|
// Slow path: create a new store outside the lock.
|
|
s, err := newStore(ctx, beadsDir, database, branch, intent)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
cacheMu.Lock()
|
|
// Double-check: another goroutine may have inserted while we created.
|
|
if entry, ok := cache[key]; ok {
|
|
cacheMu.Unlock()
|
|
// Discard the store we just created; use the cached one.
|
|
_ = s.Close()
|
|
entry.refCount++
|
|
return entry.store, nil
|
|
}
|
|
cache[key] = &cacheEntry{store: s, refCount: 1}
|
|
cacheMu.Unlock()
|
|
return s, nil
|
|
}
|
|
|
|
// closeCached decrements the reference count for a cached store.
|
|
// Returns true when the cache absorbed the close (refs remain, suppress real
|
|
// close). Returns false when the caller must run Close — either the
|
|
// entry was evicted (last ref) or the store was never cached.
|
|
func closeCached(s *EmbeddedDoltStore) bool {
|
|
cacheMu.Lock()
|
|
defer cacheMu.Unlock()
|
|
|
|
for key, entry := range cache {
|
|
if entry.store == s {
|
|
entry.refCount--
|
|
if entry.refCount <= 0 {
|
|
delete(cache, key)
|
|
// Actual close happens after releasing cacheMu (via caller).
|
|
return false
|
|
}
|
|
// Other references remain — suppress the real close.
|
|
return true
|
|
}
|
|
}
|
|
// Not in cache — let the caller close normally.
|
|
return false
|
|
}
|
|
|
|
// cacheKey resolves beadsDir to an absolute dataDir path for use as a cache key.
|
|
func cacheKey(beadsDir string) (string, error) {
|
|
absBeadsDir, err := filepath.Abs(beadsDir)
|
|
if err != nil {
|
|
return "", fmt.Errorf("embeddeddolt: resolving beads dir for cache key: %w", err)
|
|
}
|
|
return filepath.Join(absBeadsDir, "embeddeddolt"), nil
|
|
}
|