* perf(export): incremental auto-export via dolt_diff Every bd command that mutates state triggers maybeAutoExport, which re-serializes every non-infra, non-template issue plus its relational data to issues.jsonl. On a mature repo with 46k beads (99.5% closed), that full sweep takes ~110s per trigger and blocks the foreground bd call, making commands like `bd list` and downstream callers appear to hang for minutes. This patches the export in place instead of rewriting from scratch: - New DiffStore capability interface (internal/storage/diff_store.go) returning upserted/removed issue IDs between two Dolt commit hashes. - DoltStore.ChangedIssueIDs implementation using dolt_diff() unioned across issues/labels/dependencies/comments tables. Issue-table removals flow to Removed; everything else flows to Upserted. Commit refs are validated and inlined since dolt_diff() does not accept prepared statement bind params. - tryIncrementalExport loads the existing issues.jsonl into an insertion-ordered map, fetches fresh JSON only for upserted IDs via GetIssuesByIDs + the existing bulk loaders, drops removed IDs, and atomically renames the result over the old file. - Falls back to full exportToFile on: no prior commit, missing file, store does not implement DiffStore, dolt_diff errors, or the changed set exceeds incrementalExportThreshold (5000). - Full-export paths that run longer than 3s now print an actionable stderr warning with the three mitigation levers (bd purge --force, export.interval, export.auto). - saveExportAutoState now writes to a tmp file and renames. The previous os.WriteFile leaves a brief O_TRUNC window where a concurrent reader sees an empty file, silently parses it as a zero exportAutoState, and treats the next run as "no prior commit" - which then forces a full export on a repo that would otherwise be incremental-eligible. Safety: upserted IDs that GetIssuesByIDs does not return are NOT dropped from the export. Treating a fetch miss as a delete is destructive: any transient fetch hiccup would wipe otherwise-valid records. Real deletions arrive via changed.Removed from the issues-table diff; only explicit template/infra filter rejections take items out of the export. Measured on a 46k-bead repo: 1m48s (full) -> 228ms (incremental with ~5 changed beads). Roughly 470x on the common steady-state path. Tests (cmd/bd/export_auto_test.go, 8 cases): - orderedIssueLines insertion-order + in-place replace semantics. - loadExistingIssueLines parses issues, skips memories, handles malformed lines and missing files. - ChangedIssueIDs detects row modifications, label-only changes, and true issue-row deletions with the correct Upserted/Removed split. - tryIncrementalExport patches changed issues in place, drops deletions, drops issues flipped to template between runs, and falls back on missing file and threshold breach. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): stop dolt test-store hang on fallback per-test-DB path (be-fgd) Root cause: context.Background() in cmd/bd/test_helpers_test.go's per-test-DB fallback path had no deadline, so a slow (not stuck) full v0->v65 migration replay could block indefinitely once the shared-schema fast path fell back. Goroutine dump confirmed the block site: a per-test dolt.New() sitting in mysqlConn.readWithTimeout/net.Read. Two-part fix: - Make the shared-DB fast path succeed reliably instead of always falling back: initCmdBDSharedSchema now sets schema.SetForceAllowRemoteMigrate(true) around its dolt.New() so the v52->v65 migration applies against the ephemeral per-run test container (the #4259 remote-migrate gate exists to protect persistent remotes, which doesn't apply here). - Bound the rare/backstop fallback path itself: testStoreOpenTimeout is calibrated from a real isolated measurement (219.62s for one full migration replay, forced via a temporary diagnostic hook and reverted) rather than guessed, giving ~37% margin over that observed duration. Refs be-hka, be-fgd. * test(export): red — bulk-seed pool read timeout plumbing (refs be-uoat) TestNewTestStoreWithReadTimeout_AppliesConfiguredTimeout and the timeout-aware setupIncrementalExportTestWithReadTimeout wrapper reference newTestStoreWithPrefixAndReadTimeout, which does not exist yet (cmd/bd fails to vet/build: "undefined: newTestStoreWithPrefixAndReadTimeout"). This expresses the be-uoat round-2 gate's acceptance criterion: the bulk-seed helper used by TestTryIncrementalExport_ThresholdExceededFallsBack needs a caller-configurable Config.PoolReadTimeout so it stops flaking under host contention on the shared-branch MaxOpenConns=1 connection, without changing the pool timeout for any other test. * fix(test): green — thread configurable PoolReadTimeout into shared-branch test store helpers (refs be-uoat) Add newTestStoreSharedBranchWithReadTimeout and newTestStoreWithPrefixAndReadTimeout, siblings of the existing newTestStoreSharedBranch / newTestStoreWithPrefix that additionally thread a caller-supplied dolt.Config.PoolReadTimeout through both the shared-branch fast path and the per-test-DB fallback. cmd/bd now vets and builds clean; TestTryIncrementalExport_ThresholdExceededFallsBack and TestNewTestStoreWithReadTimeout_AppliesConfiguredTimeout both run (PASS/SKIP per Dolt test-server availability, no compile error). Existing callers are untouched — newTestStore/newTestStoreWithPrefix still default to defaultPoolReadTimeout (10s) via internal/storage/dolt. This only affects the one bulk-seed helper that opts in. * fix(test): green — assert on returned error instead of subtest propagation in PoolReadTimeout plumbing test (refs be-8wnq) TestNewTestStoreWithReadTimeout_AppliesConfiguredTimeout's 'unreasonably short timeout fails fast' case wrapped its expected-to-fail store creation in t.Run and checked the returned bool. Go's testing package unconditionally propagates a failed subtest to the parent test and process exit code, so that pattern could never report an overall PASS even when the PoolReadTimeout plumbing was working correctly. Add tryNewTestStoreWithReadTimeout, the dolt.New portion of newTestStoreWithPrefixAndReadTimeout with the error returned directly instead of calling t.Fatal. The test now asserts on that error value with no subtest wrapper for the expected-to-fail branch. * docs(release-gate): PASS — incremental auto-export via dolt_diff, round 4 (be-y1jo) Deploy gate re-evaluation for be-hka's incremental auto-export feature, round 2 of deploy (be-y1jo, superseding be-uoat's criterion-3 FAIL). Round 3 (be-8wnq) root-caused be-uoat's flake to the pool's 10s default read timeout colliding with a 5001-row bulk seed under host contention, and fixed it by threading a configurable PoolReadTimeout (5m for this one bulk-seed path) through the shared test-store helpers -- soaked 3/3 PASS. Round 4 (be-unlq, commit fc5caadcf) fixed a separate, deterministic test-logic defect (Go subtest failure unconditionally propagating to the parent) that round 3 found in the new plumbing-proof test. Independently re-verified rather than trusted: all 8 diff-owned tests green against a real Dolt container, including 3 fresh runs of the previously-flaky test (43.9-45.5s each, zero failures -- 6/6 real- container passes post-fix combined with round 3's soak). Build/vet/ gofmt clean. Policy lane's one failure independently confirmed pre-existing and diff-unrelated (.githooks/commit-msg, byte-identical to origin/main). Ancestry-scope check's single stray-commit flag independently confirmed a benign artifact of the extraction workflow (the flagged commit predates be-hka; its 4 files are a strict subset of the approved 7-file scope). refs be-y1jo, be-hka, be-uoat, be-8wnq, be-unlq * test(feat): red — Fix PR #5806 review: incremental auto-export never fires (state hash is not a commit) + memory/owner/format contract breaks (refs be-shbed) * fix: green — Fix PR #5806 review: incremental auto-export never fires (state hash is not a commit) + memory/owner/format contract breaks (refs be-shbed) * fix: green — missingJSONLIssueIDsInStore no longer blocks incremental export on proven deletions (refs be-nti0e) Round 2 fix for bee-ghosttrack's CHANGES_REQUESTED on PR #5806. TestMaybeAutoExport_SecondRunTakesIncrementalPath_ServerMode regressed because missingJSONLIssueIDsInStore treated any JSONL-only issue id as corruption -- including ids a legitimate `bd delete` removed between exports -- permanently wedging tryIncrementalExport's fast path in server mode. missingJSONLIssueIDsInStore now takes the last successful export's diff anchor (state.LastDiffAnchor) and, for whatever candidates remain after the existing ephemeral/template/infra-type filtering, asks ChangedIssueIDs (dolt_diff over issues/labels/dependencies/comments) whether each was actually removed since that anchor. Proven deletions are excluded from the missing set; anything not provably deleted still falls back to the original conservative refusal. No anchor (cold start) falls through unchanged, same as before this check existed. Also fixes cmd/bd/export_auto_missing_store_test.go's call site for the new 3-arg signature, and renames export_auto_test.go's idSet helper to idSetFromIDs to resolve a rebase collision with PR #5822's own idSet in stale_embedded_test.go. refs be-nti0e * test(export): assert the incremental path ran, not just that a diff was reached (refs be-re1zl) PR #5806 round-5 review, finding 1. Deleting e2e-b makes it an orphan-guard candidate, so the guard's own ChangedIssueIDs call satisfies a bare `changedIssueIDsCalls < 1` check even when tryIncrementalExport falls back to a full export -- the test passed while proving nothing about the path it is named for. Use both oracles the reviewer verified against a real server: an exact call count of 2 (orphan-guard deletion probe + incremental diff), and LastDirtyIDs == {e2e-a}, which maybeAutoExport explicitly nils on the full-export fallback. * fix(export): require the diff anchor to be on HEAD's history before trusting a removal (refs be-re1zl) PR #5806 round-5 review, finding 2: "removed since anchor" is not the same claim as "deleted by bd delete". dolt_diff(anchor, WORKING) reports a row as removed whenever it is absent at WORKING, which is equally true when the history moved out from under the anchor -- a CALL DOLT_RESET('--hard', ...), a branch checkout, or any other data-dir rewind on a shared server. The orphan guard honoured that as proof of deletion, so auto-export silently dropped a live record from issues.jsonl: exactly the #4988 corruption class the guard exists to prevent. Gate the deletion proof on the anchor still being reachable from HEAD (CommitExists queries dolt_log, i.e. HEAD's forward history). Fails closed -- a store that cannot answer, or an error while asking, keeps refusing, which is the conservative pre-existing behaviour. TestMaybeAutoExport_HistoryRewindDoesNotProveDeletion reproduces the reviewer's repro (export, create rw-x, export, DOLT_RESET --hard back to the baseline, export) and fails without this change. Also records the residual the reviewer noted: an unresolvable anchor (after a history squash or GC) plus a pending delete is a permanent wedge, tracked by #5896. * docs(changelog): scope incremental export and the delete guard to server mode (refs be-re1zl) PR #5806 round-5 review, finding 3 and note (b). - Add the orphan-guard change as its own user-visible entry, including how narrow the deletion proof deliberately is. - State that both the incremental path and the guard are server/DiffStore-mode only: EmbeddedDoltStore implements neither DiffStore nor StateHasher, so in embedded mode every cycle is still a full export and bd delete still wedges auto-export there -- #5896 stays open for it. - Add the anchor-never-advances caveat: with auto-commit off and nothing committing, the diff range grows until the 5000-id threshold sends every export back to a full rewrite. - Stop claiming the memory-record handling "matches full-export behavior exactly". It deliberately does not: the full path refuses to overwrite a JSONL containing memory lines, the incremental path preserves them verbatim and proceeds. Preserving is the better behaviour; say so instead. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
25 lines
979 B
Go
25 lines
979 B
Go
package storage
|
|
|
|
import "context"
|
|
|
|
// DiffStore computes the set of issue IDs whose underlying data differs
|
|
// between two Dolt commits. It's a capability interface — callers should
|
|
// type-assert after UnwrapStore and fall back to a non-incremental path
|
|
// when a store doesn't implement it.
|
|
//
|
|
// "Underlying data" includes changes to the issues table itself as well as
|
|
// its label, dependency, and comment rows. An issue ID is reported under
|
|
// Upserted if any of those rows were added or modified, and under Removed
|
|
// only if the issue row itself was deleted. Label/dependency/comment row
|
|
// deletions are reported as Upserted (the issue survived, its relational
|
|
// data shrank).
|
|
type DiffStore interface {
|
|
ChangedIssueIDs(ctx context.Context, fromCommit, toCommit string) (ChangedIssueIDs, error)
|
|
}
|
|
|
|
// ChangedIssueIDs lists the issue IDs whose stored representation differs
|
|
// between two commits.
|
|
type ChangedIssueIDs struct {
|
|
Upserted []string
|
|
Removed []string
|
|
}
|