diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a45859ed..d70d030e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -136,6 +136,70 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Incremental auto-export now actually takes the incremental path** + ([#5806](https://github.com/gastownhall/beads/pull/5806)). Change detection + compared `GetStateHash()` values — a hash of the entire database plus + working set (`DOLT_HASHOF_DB()`), not a commit — but `ChangedIssueIDs` feeds + both endpoints straight into `dolt_diff()`, which only accepts real commit + hashes or the literal `WORKING`. Every incremental attempt therefore failed + to resolve a diff and silently fell back to a full export: correct output, + but the incremental short-circuit this feature exists for never actually + ran. The "to" endpoint is now `WORKING` (dolt_diff's own literal for the + live working set) paired with the previous export's real commit hash as + "from", so the incremental path resolves and fires as designed. + + Three format/scope regressions surfaced alongside the dead code path, since + nothing had ever exercised it end-to-end: the incremental patch could leak + **memories** into the auto-export output (full export filters them; the + patch path did not), could include the configured **owner**'s own issues + where full export excludes them, and omitted the `_type` discriminator field + full export always writes. All three are now pinned by tests that diff a + patched run's output against a from-scratch full export of the same state. + The patch write is also now atomic (write-temp + rename), matching the + full-export path's existing guarantee. + + The owner and `_type` fixes match full export exactly. **Memory records + deliberately do not**, and the difference is worth stating: where the full + path *refuses* to overwrite an `issues.jsonl` that already contains memory + lines (`guardAutoExportOverwrite`), the incremental path *preserves* them + verbatim and proceeds. Preserving is the better behavior — a manually + seeded export survives patching byte-for-byte instead of wedging — so the + incremental path keeps it rather than adopting the refusal. + + **Scope: this is a server-mode (`DiffStore`) improvement only.** + `EmbeddedDoltStore` implements neither `DiffStore` nor `StateHasher`, so in + the default embedded mode the incremental path is inert (`incremental + skipped — store does not implement DiffStore`) and every cycle is still a + full export. The deletion-proof guard below is likewise server-mode only, so + [#5896](https://github.com/gastownhall/beads/issues/5896) stays open for + embedded mode, where `bd delete` still wedges auto-export. + + **Caveat: the diff anchor only advances on real commits.** In server mode + dolt auto-commit is off, so with no commits being made the anchor stays put + and the diff range grows with every cycle. That is correct but not free: + once the change set crosses the 5000-id threshold, every export falls back + to a full rewrite until something commits and the anchor moves forward. + +- **`bd delete` no longer wedges auto-export in server mode** + ([#5806](https://github.com/gastownhall/beads/pull/5806), part of + [#5896](https://github.com/gastownhall/beads/issues/5896)). Auto-export's + orphan guard refuses to overwrite an `issues.jsonl` holding issue records + absent from the local store — the #4988 protection against a JSONL that has + run ahead of the database. A normal `bd delete` produced exactly that shape, + so a single delete left `issues.jsonl` permanently stale, with a "refusing + to overwrite" warning on every later command. The guard now asks `dolt_diff` + whether each JSONL-only id was genuinely removed since the last export's + anchor and, when proven, stops treating it as corruption. + + The proof is deliberately narrow. It requires `diff_type='removed'` on the + **issues** table itself, so a label-, dependency- or comment-only change, a + delete-then-recreate, a wisp, or any diff error all still refuse. It also + requires the anchor to still be reachable from `HEAD`: after a + `DOLT_RESET --hard`, a branch checkout, or any other data-dir rewind, a row + reads as "removed" without anyone having deleted it, and honoring that + would drop a live record. Both conditions must hold, or the guard refuses as + before. + - **Disabling telemetry no longer strands the queued eventsData backlog forever** (GH#5712). `bd send-metrics` early-returned on disabled metrics *before* its prune step, and the spawn gate refused to schedule the child at diff --git a/cmd/bd/export_auto.go b/cmd/bd/export_auto.go index db2e125f4..aa94d115e 100644 --- a/cmd/bd/export_auto.go +++ b/cmd/bd/export_auto.go @@ -2,6 +2,7 @@ package main import ( "bufio" + "bytes" "context" "encoding/json" "fmt" @@ -21,12 +22,41 @@ import ( "github.com/steveyegge/beads/internal/types" ) +// incrementalExportThreshold caps the number of changed issue IDs we'll +// incrementally re-encode before falling back to a full export. At high +// change counts the per-issue SQL work (bulk loaders × changed set) stops +// being cheaper than one `SearchIssues(Limit:0)` sweep. +const incrementalExportThreshold = 5000 + +// doltWorkingRef is the literal dolt_diff() accepts as an endpoint meaning +// "the current working set", including uncommitted writes. Passing this +// instead of a state/root hash is what lets incremental auto-export see +// changes in server mode, where auto-commit is off and HEAD never moves. +const doltWorkingRef = "WORKING" + +// slowExportWarnThreshold is the duration over which an auto-export is +// considered slow enough to warn the user. Any single auto-export that +// exceeds this prints a one-line stderr tip pointing at the fix levers. +const slowExportWarnThreshold = 3 * time.Second + // exportAutoState tracks auto-export state to avoid redundant work. type exportAutoState struct { LastDoltCommit string `json:"last_dolt_commit"` Timestamp time.Time `json:"timestamp"` Issues int `json:"issues"` Memories int `json:"memories"` + + // LastDiffAnchor is the real commit hash (never a working-set/state + // hash) used as tryIncrementalExport's fromCommit on the next cycle. + // dolt_diff() rejects non-commit values, so this must never be set from + // storeStateHash — only from store.GetCurrentCommit. + LastDiffAnchor string `json:"last_diff_anchor"` + // LastDirtyIDs carries the previous cycle's raw changed.Upserted ids + // forward so a working-set value that reverts between two cycles (diff + // against the anchor sees no net change) still gets re-patched against + // the live row instead of going stale. Self-healing: a cycle that finds + // nothing new for a carried id drops it, so this never grows unbounded. + LastDirtyIDs []string `json:"last_dirty_ids,omitempty"` } const exportAutoStateFile = "export-state.json" @@ -108,7 +138,7 @@ func maybeAutoExport(ctx context.Context, allowEmptyOverwrite bool) error { } } if !allowEmptyOverwrite { - if missingIDs, err := missingJSONLIssueIDsInStore(ctx, fullPath); err != nil { + if missingIDs, err := missingJSONLIssueIDsInStore(ctx, fullPath, state.LastDiffAnchor); err != nil { fmt.Fprintf(os.Stderr, "Warning: auto-export skipped: failed to compare existing JSONL against local store: %v\n", err) return nil } else if len(missingIDs) > 0 { @@ -119,14 +149,57 @@ func maybeAutoExport(ctx context.Context, allowEmptyOverwrite bool) error { // Run the export — memories are excluded from auto-export because they // contain private agent context that must not reach git history (GH#3650). - issueCount, memoryCount, err := exportToFile(ctx, fullPath, false) + // Try an incremental re-encode first (dolt_diff over the changed-since-last- + // export range); fall back to a full export when the store doesn't support + // diffing, the commit range isn't diffable (e.g. state-hash values in server + // mode where auto-commit is off), or the change set exceeds the threshold. + // + // The diff anchor is always a real commit (never the working-set/state + // hash used for change detection above) — dolt_diff() rejects anything + // else — and the diff's "to" endpoint is always the literal "WORKING", + // which dolt_diff accepts and which captures uncommitted writes in + // server mode (where auto-commit is off and HEAD does not advance). + // Resolved after every early-return guard above so tests whose scenario + // is fully handled by those guards see no extra GetCurrentCommit call. + exportStart := time.Now() + anchorCommit, anchorErr := store.GetCurrentCommit(ctx) + if anchorErr != nil { + debug.Logf("auto-export: failed to resolve diff anchor (%v); preserving previous anchor\n", anchorErr) + anchorCommit = state.LastDiffAnchor + } + issueCount, memoryCount, dirtyIDs, didIncremental, err := tryIncrementalExport( + ctx, fullPath, state.LastDiffAnchor, doltWorkingRef, state.LastDirtyIDs, + ) if err != nil { - fmt.Fprintf(os.Stderr, "Warning: auto-export failed: %v\n", err) - return nil + debug.Logf("auto-export: incremental failed (%v); falling back to full\n", err) + } + if !didIncremental { + issueCount, memoryCount, err = exportToFile(ctx, fullPath, false) + if err != nil { + fmt.Fprintf(os.Stderr, "Warning: auto-export failed: %v\n", err) + return nil + } + dirtyIDs = nil } - debug.Logf("auto-export: wrote %d issues and %d memories to %s\n", - issueCount, memoryCount, fullPath) + if dur := time.Since(exportStart); dur > slowExportWarnThreshold && !didIncremental { + // Only warn on full exports — the whole point of incremental is to + // get us under the threshold, so a slow incremental is noise. + fmt.Fprintf(os.Stderr, + "Warning: auto-export wrote %d issues in %s (runs after every bd command that changes state).\n"+ + " Large closed-issue backlogs make this expensive. Levers:\n"+ + " bd purge --force remove closed wisps (ephemeral beads)\n"+ + " bd config set export.interval 10m reduce auto-export frequency\n"+ + " bd config set export.auto false disable auto-export entirely\n", + issueCount, dur.Round(time.Second)) + } + + mode := "full" + if didIncremental { + mode = "incremental" + } + debug.Logf("auto-export: wrote %d issues and %d memories to %s (%s, %s)\n", + issueCount, memoryCount, fullPath, mode, time.Since(exportStart).Round(time.Millisecond)) // Don't prime the throttle on an empty export (e.g. immediately after // `bd init`). Saving state here would block the first real `bd create` @@ -137,6 +210,8 @@ func maybeAutoExport(ctx context.Context, allowEmptyOverwrite bool) error { _ = os.Remove(fullPath) saveExportAutoState(beadsDir, &exportAutoState{ LastDoltCommit: currentCommit, + LastDiffAnchor: anchorCommit, + LastDirtyIDs: dirtyIDs, Timestamp: time.Now(), Issues: 0, Memories: 0, @@ -156,6 +231,8 @@ func maybeAutoExport(ctx context.Context, allowEmptyOverwrite bool) error { // Save state newState := exportAutoState{ LastDoltCommit: currentCommit, + LastDiffAnchor: anchorCommit, + LastDirtyIDs: dirtyIDs, Timestamp: time.Now(), Issues: issueCount, Memories: memoryCount, @@ -213,7 +290,7 @@ func countIssueRecordsInJSONL(path string) (int, error) { return len(ids), nil } -func missingJSONLIssueIDsInStore(ctx context.Context, path string) ([]string, error) { +func missingJSONLIssueIDsInStore(ctx context.Context, path, fromCommit string) ([]string, error) { // GH#4988: only refuse when *in-scope* JSONL issue records are absent from // the store. Ephemeral wisps, templates, and infra types are outside // auto-export scope (buildAutoExportFilter / GH#3649). Compaction can @@ -243,16 +320,95 @@ func missingJSONLIssueIDsInStore(ctx context.Context, path string) ([]string, er localIDs[issue.ID] = struct{}{} } - missing := make([]string, 0) + candidates := make([]string, 0) for _, rec := range existing { if rec.Ephemeral || rec.IsTemplate || infraTypeSet[string(rec.IssueType)] { continue } if _, ok := localIDs[rec.ID]; !ok { - missing = append(missing, rec.ID) + candidates = append(candidates, rec.ID) } } - return missing, nil + if len(candidates) == 0 { + return nil, nil + } + + // Before treating a JSONL-only id as an orphan/corruption, ask dolt_diff + // whether the deletion is real and already recorded — a normal `bd + // delete` since the last export's diff anchor, not data loss. Without an + // anchor (cold start / no successful export yet) there is nothing to + // diff against, so every candidate falls through to the conservative + // "missing" verdict below, same as before this check existed. + // + // The anchor must still be on the CURRENT HEAD's history for that proof + // to mean anything. dolt_diff(anchor, WORKING) reports a row as "removed" + // whenever it is absent at WORKING — which is equally true when the + // history moved out from under the anchor rather than when anyone deleted + // anything: a `CALL DOLT_RESET('--hard', )`, a branch checkout, + // or any other data-dir rewind on a shared server. Honoring the diff + // there would silently drop a live record from issues.jsonl, which is + // exactly #4988's corruption class this guard exists to prevent. So the + // deletion proof is accepted only when the anchor is reachable from HEAD + // (CommitExists queries dolt_log, i.e. HEAD's forward history); otherwise + // we keep refusing, which is the conservative pre-existing behavior. + // + // Residual, deliberately not fixed here (tracked by #5896): an anchor + // that has become unresolvable — after a history squash or GC — combined + // with a pending delete is a permanent wedge. The guard refuses, so the + // export skips, so state is never saved, so the anchor never refreshes. + // Recovery is `bd init --from-jsonl` or moving issues.jsonl aside. + if fromCommit != "" && diffAnchorOnCurrentHistory(ctx, fromCommit) { + if ds, ok := storage.UnwrapStore(store).(storage.DiffStore); ok { + if changed, diffErr := ds.ChangedIssueIDs(ctx, fromCommit, doltWorkingRef); diffErr == nil { + removed := make(map[string]struct{}, len(changed.Removed)) + for _, id := range changed.Removed { + removed[id] = struct{}{} + } + proven := candidates[:0] + for _, id := range candidates { + if _, ok := removed[id]; !ok { + proven = append(proven, id) + } + } + candidates = proven + } + // A diff error (e.g. anchor unreachable after history rewrite) + // leaves candidates untouched — we can't prove the deletion, so + // fall back to refusing, exactly as before this check existed. + } + } + + return candidates, nil +} + +// commitHistoryChecker is the narrow slice of storage.VersionControl that +// missingJSONLIssueIDsInStore needs. It is declared locally and asserted +// separately from storage.DiffStore so a store implementing one but not the +// other still degrades to the conservative verdict instead of panicking. +type commitHistoryChecker interface { + CommitExists(ctx context.Context, commitHash string) (bool, error) +} + +// diffAnchorOnCurrentHistory reports whether anchor is still reachable from +// the current HEAD. It is the ancestry precondition on trusting a dolt_diff +// "removed" verdict as proof of a real `bd delete` rather than a history +// rewind — see missingJSONLIssueIDsInStore. Fails closed: a store that +// cannot answer, or an error while asking, yields false so the caller keeps +// refusing to overwrite. +func diffAnchorOnCurrentHistory(ctx context.Context, anchor string) bool { + ch, ok := storage.UnwrapStore(store).(commitHistoryChecker) + if !ok { + return false + } + onHistory, err := ch.CommitExists(ctx, anchor) + if err != nil { + debug.Logf("auto-export: cannot verify diff anchor %s against HEAD history (%v); treating JSONL-only ids as unproven\n", anchor, err) + return false + } + if !onHistory { + debug.Logf("auto-export: diff anchor %s is not on HEAD's history (rewind/checkout?); treating JSONL-only ids as unproven\n", anchor) + } + return onHistory } // jsonlIssueRecord is a lightweight issue line from issues.jsonl used by @@ -644,10 +800,13 @@ func loadExportAutoState(beadsDir string) *exportAutoState { path := filepath.Join(beadsDir, exportAutoStateFile) data, err := os.ReadFile(path) //nolint:gosec if err != nil { + debug.Logf("auto-export: state-load miss (%s): %v\n", path, err) return &exportAutoState{} } var state exportAutoState if err := json.Unmarshal(data, &state); err != nil { + debug.Logf("auto-export: state-load parse error for %s (%d bytes): %v; first 200 bytes=%q\n", + path, len(data), err, string(data[:min(len(data), 200)])) return &exportAutoState{} } return &state @@ -660,6 +819,12 @@ func saveExportAutoState(beadsDir string, state *exportAutoState) { fmt.Fprintf(os.Stderr, "Warning: auto-export: failed to marshal state: %v\n", err) return } + // Write atomically — concurrent bd invocations read this file in + // maybeAutoExport, and a plain os.WriteFile leaves a brief window + // after O_TRUNC but before the data lands where readers see an empty + // file. An empty state looks like "no prior commit" to the rest of + // the pipeline, which forces a full export on a repo where the + // incremental path would otherwise fire. if err := atomicfile.WriteFile(path, data, 0o600); err != nil { fmt.Fprintf(os.Stderr, "Warning: auto-export: failed to save state: %v\n", err) } @@ -861,3 +1026,354 @@ func pathInsideDir(path, dir string) bool { sep := string(filepath.Separator) return absPath == absDir || strings.HasPrefix(absPath, absDir+sep) } + +// tryIncrementalExport attempts to update an existing export file in place +// by re-encoding only the issues that changed between fromCommit and +// toCommit (per dolt_diff), plus any ids carried over from the previous +// cycle via carryIDs (exportAutoState.LastDirtyIDs). Returns +// didIncremental=false when any precondition fails so the caller can fall +// back to a full export. +// +// dirtyIDs returns the RAW diff's upserted ids (never carryIDs) for the +// caller to persist as next cycle's carryIDs. Carrying only the +// freshly-diffed ids — not the ones merely carried-in — makes a stable +// no-op cycle self-heal: a carried id that produces no new diff simply +// drops out instead of accumulating forever. +func tryIncrementalExport(ctx context.Context, fullPath, fromCommit, toCommit string, carryIDs []string) (issueCount, memoryCount int, dirtyIDs []string, didIncremental bool, err error) { + if fromCommit == "" { + debug.Logf("auto-export: incremental skipped — no prior commit hash recorded\n") + return 0, 0, nil, false, nil + } + // Existing file is a hard requirement — without it we have nothing to + // patch, and a full export is the right answer anyway. + if _, statErr := os.Stat(fullPath); statErr != nil { + debug.Logf("auto-export: incremental skipped — existing file not found: %v\n", statErr) + return 0, 0, nil, false, nil + } + ds, ok := storage.UnwrapStore(store).(storage.DiffStore) + if !ok { + debug.Logf("auto-export: incremental skipped — store does not implement DiffStore\n") + return 0, 0, nil, false, nil + } + changed, diffErr := ds.ChangedIssueIDs(ctx, fromCommit, toCommit) + if diffErr != nil { + // Commits may be unreachable (history rewritten), in which case we + // cannot trust the diff. Fall back silently. + return 0, 0, nil, false, diffErr + } + debug.Logf("auto-export: diff %s..%s → upserted=%d removed=%d carried=%d\n", + fromCommit, toCommit, len(changed.Upserted), len(changed.Removed), len(carryIDs)) + + // carryIDs must be folded in BEFORE the total==0 fast-path: a cycle + // whose raw diff is empty can still owe a re-patch for an id carried + // from last cycle (e.g. a working-set value that reverted between two + // diff anchors nets to "no change" against the anchor, but the live row + // still needs re-fetching to correct a stale patch from last cycle). + upsertIDs := unionStrings(changed.Upserted, carryIDs) + total := len(upsertIDs) + len(changed.Removed) + if total == 0 { + // Nothing changed and nothing carried over. Still a valid + // "incremental" outcome: no issues to rewrite, just refresh the + // file in place so nothing regresses. + issueCount, memoryCount, err = rewriteExportFile(fullPath, nil, nil, nil) + if err != nil { + return 0, 0, nil, false, err + } + return issueCount, memoryCount, changed.Upserted, true, nil + } + if total > incrementalExportThreshold { + debug.Logf("auto-export: %d changes exceeds threshold %d; full export\n", + total, incrementalExportThreshold) + return 0, 0, nil, false, nil + } + + // Fetch fresh data for upserted IDs (including anything carried over + // from last cycle) and apply the same template/infra/owner filters the + // full export uses. + var records map[string][]byte + droppedByFilter := make(map[string]bool) + if len(upsertIDs) > 0 { + issues, fetchErr := store.GetIssuesByIDs(ctx, upsertIDs) + if fetchErr != nil { + return 0, 0, nil, false, fmt.Errorf("GetIssuesByIDs: %w", fetchErr) + } + infraSet := store.GetInfraTypes(ctx) + // Owner-exclusion safety net, mirroring exportToFile's: without + // this, a config-excluded owner's issue that changes would leak + // into the git-committed JSONL via the incremental path even + // though the full-export path always excludes it (be-shbed). + ownerExcludes := buildOwnerExcludeSet(ctx, storeExportSource{}, nil) + filtered := make([]*types.Issue, 0, len(issues)) + for _, iss := range issues { + // Record IDs that GetIssuesByIDs returned but we deliberately + // filtered out. Those DO need dropping from the export because + // the full-export path excludes them; leaving a stale record + // in place would diverge the two outputs. + if iss.IsTemplate { + droppedByFilter[iss.ID] = true + continue + } + if len(infraSet) > 0 && infraSet[string(iss.IssueType)] { + droppedByFilter[iss.ID] = true + continue + } + if _, excluded := ownerExcludes[iss.CreatedBy]; excluded { + droppedByFilter[iss.ID] = true + continue + } + filtered = append(filtered, iss) + } + records, err = encodeIssueRecords(ctx, filtered) + if err != nil { + return 0, 0, nil, false, err + } + // NOTE: upserted IDs absent from GetIssuesByIDs's result are + // intentionally NOT dropped. We used to flag them as "removed", + // which is destructive: any hiccup in the fetch path (partial + // failure, concurrent close, transient routing weirdness) would + // wipe otherwise-valid records. Real deletions land in + // changed.Removed from the issues-table diff; rely on that. + } + + removed := make(map[string]bool, len(changed.Removed)+len(droppedByFilter)) + for _, id := range changed.Removed { + removed[id] = true + } + for id := range droppedByFilter { + removed[id] = true + } + + issueCount, memoryCount, err = rewriteExportFile(fullPath, records, removed, upsertIDs) + if err != nil { + return 0, 0, nil, false, err + } + return issueCount, memoryCount, changed.Upserted, true, nil +} + +// unionStrings returns the deduplicated union of a and b, preserving a's +// element order followed by any of b's elements not already present in a. +func unionStrings(a, b []string) []string { + if len(b) == 0 { + return a + } + seen := make(map[string]bool, len(a)+len(b)) + out := make([]string, 0, len(a)+len(b)) + for _, s := range a { + if !seen[s] { + seen[s] = true + out = append(out, s) + } + } + for _, s := range b { + if !seen[s] { + seen[s] = true + out = append(out, s) + } + } + return out +} + +// rewriteExportFile applies a set of upserts and removals to an existing +// JSONL export and writes the result atomically. upsertOrder preserves +// append order for brand-new issue IDs; previously-present IDs keep their +// original file position even when their bodies are replaced. +func rewriteExportFile(path string, upserts map[string][]byte, removed map[string]bool, upsertOrder []string) (issueCount, memoryCount int, err error) { + lines, err := loadExistingIssueLines(path) + if err != nil { + return 0, 0, fmt.Errorf("read existing export: %w", err) + } + + // Apply upserts: replace-in-place for known IDs, append (in input + // order) for brand-new IDs. This keeps stable ordering across runs + // instead of scrambling the file on every change. + for _, id := range upsertOrder { + line, ok := upserts[id] + if !ok { + continue + } + lines.set(id, line) + } + + for id := range removed { + lines.remove(id) + } + + w, err := atomicfile.Create(path, 0o644) + if err != nil { + return 0, 0, fmt.Errorf("failed to create export file: %w", err) + } + defer func() { + if err != nil { + _ = w.Abort() + } + }() + + lines.each(func(id string, line []byte) { + if err != nil { + return + } + if _, werr := w.Write(line); werr != nil { + err = werr + return + } + if _, werr := w.Write([]byte{'\n'}); werr != nil { + err = werr + return + } + issueCount++ + }) + if err != nil { + return 0, 0, fmt.Errorf("failed to write issue line: %w", err) + } + + // Re-emit whatever memory lines were already in the file, verbatim and + // unchanged — auto-export never reads live config for memories (that + // would leak private agent context into git history, GH#3650). A file + // with no prior memory section (the auto-export-only common case) stays + // that way; a file seeded by a manual `bd export --include-memories` + // keeps its memories across every subsequent incremental patch. + for _, memLine := range lines.memoryLines { + if _, werr := w.Write(memLine); werr != nil { + return 0, 0, fmt.Errorf("failed to write memory line: %w", werr) + } + if _, werr := w.Write([]byte{'\n'}); werr != nil { + return 0, 0, fmt.Errorf("failed to write newline: %w", werr) + } + memoryCount++ + } + + if err := w.Close(); err != nil { + return 0, 0, fmt.Errorf("failed to finalize export: %w", err) + } + return issueCount, memoryCount, nil +} + +// orderedIssueLines is an insertion-ordered map of issue ID → raw JSONL +// line (without trailing newline). Removal is O(1) via the map; the order +// slice may contain IDs that have since been removed, which the iterator +// skips. +// +// memoryLines carries forward any pre-existing memory records verbatim. +// Auto-export never regenerates memories from live config (GH#3650 — that +// would leak private agent context into git history), so a file seeded by a +// manual `bd export --include-memories` must have its memory lines preserved +// byte-for-byte across every subsequent incremental patch. +type orderedIssueLines struct { + order []string + lines map[string][]byte + memoryLines [][]byte +} + +func newOrderedIssueLines() *orderedIssueLines { + return &orderedIssueLines{lines: make(map[string][]byte)} +} + +func (o *orderedIssueLines) set(id string, line []byte) { + if _, present := o.lines[id]; !present { + o.order = append(o.order, id) + } + o.lines[id] = line +} + +func (o *orderedIssueLines) remove(id string) { + delete(o.lines, id) +} + +func (o *orderedIssueLines) each(fn func(id string, line []byte)) { + for _, id := range o.order { + if line, ok := o.lines[id]; ok { + fn(id, line) + } + } +} + +// loadExistingIssueLines parses a JSONL export file and returns an +// orderedIssueLines with issue records keyed by id and any memory records +// captured verbatim into memoryLines (see the type doc — they are carried +// forward, never regenerated). Missing file returns an empty set so +// first-incremental callers behave like a fresh write. +func loadExistingIssueLines(path string) (*orderedIssueLines, error) { + data, err := os.ReadFile(path) //nolint:gosec // path is the user-configured export path + if err != nil { + if os.IsNotExist(err) { + return newOrderedIssueLines(), nil + } + return nil, err + } + out := newOrderedIssueLines() + for _, raw := range bytes.Split(data, []byte{'\n'}) { + raw = bytes.TrimSpace(raw) + if len(raw) == 0 { + continue + } + // Struct-decode _type (mirroring classifyExistingAutoExportRecord) + // rather than substring-matching: robust to key order/spacing and + // unaffected by nested "id"/"_type" keys inside comments etc., since + // Go's decoder only ever populates the outer struct's fields. + var probe struct { + Type string `json:"_type"` + ID string `json:"id"` + } + if err := json.Unmarshal(raw, &probe); err != nil { + continue + } + // bytes.Split shares the backing buffer; copy so later appends + // can't silently overwrite our captured line. + cpy := make([]byte, len(raw)) + copy(cpy, raw) + if probe.Type == "memory" { + out.memoryLines = append(out.memoryLines, cpy) + continue + } + if probe.ID == "" { + continue + } + out.set(probe.ID, cpy) + } + return out, nil +} + +// encodeIssueRecords produces the JSON wire form for a batch of issues +// using the same bulk-loader pattern the full export uses. Returns a map +// from issue ID → line bytes (no trailing newline). +func encodeIssueRecords(ctx context.Context, issues []*types.Issue) (map[string][]byte, error) { + if len(issues) == 0 { + return nil, nil + } + ids := make([]string, len(issues)) + for i, iss := range issues { + ids[i] = iss.ID + } + labelsMap, _ := store.GetLabelsForIssues(ctx, ids) + allDeps, _ := store.GetDependencyRecordsForIssues(ctx, ids) + commentsMap, _ := store.GetCommentsForIssues(ctx, ids) + commentCounts, _ := store.GetCommentCounts(ctx, ids) + depCounts, _ := store.GetDependencyCounts(ctx, ids) + + out := make(map[string][]byte, len(issues)) + for _, iss := range issues { + iss.Labels = labelsMap[iss.ID] + iss.Dependencies = allDeps[iss.ID] + iss.Comments = commentsMap[iss.ID] + counts := depCounts[iss.ID] + if counts == nil { + counts = &types.DependencyCounts{} + } + sanitizeZeroTime(iss) + rec := &exportIssueRecord{ + RecordType: "issue", + IssueWithCounts: &types.IssueWithCounts{ + Issue: iss, + DependencyCount: counts.DependencyCount, + DependentCount: counts.DependentCount, + CommentCount: commentCounts[iss.ID], + }, + } + data, err := json.Marshal(rec) + if err != nil { + return nil, fmt.Errorf("marshal %s: %w", iss.ID, err) + } + out[iss.ID] = data + } + return out, nil +} diff --git a/cmd/bd/export_auto_missing_store_test.go b/cmd/bd/export_auto_missing_store_test.go index 83c7e45f7..71a732608 100644 --- a/cmd/bd/export_auto_missing_store_test.go +++ b/cmd/bd/export_auto_missing_store_test.go @@ -85,7 +85,7 @@ func TestMissingJSONLIssueIDsInStore_IgnoresCompactedWisp(t *testing.T) { t.Fatal(err) } - missing, err := missingJSONLIssueIDsInStore(ctx, jsonlPath) + missing, err := missingJSONLIssueIDsInStore(ctx, jsonlPath, "") if err != nil { t.Fatalf("missingJSONLIssueIDsInStore: %v", err) } diff --git a/cmd/bd/export_auto_test.go b/cmd/bd/export_auto_test.go index 3f3178068..b4fa1339c 100644 --- a/cmd/bd/export_auto_test.go +++ b/cmd/bd/export_auto_test.go @@ -1,11 +1,16 @@ +//go:build cgo + package main import ( "context" + "database/sql" "encoding/json" + "fmt" "os" "os/exec" "path/filepath" + "reflect" "strings" "testing" "time" @@ -13,6 +18,7 @@ import ( "github.com/spf13/cobra" "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/testutil" "github.com/steveyegge/beads/internal/types" ) @@ -232,6 +238,7 @@ func (f *fakeStateHashStore) SearchIssues(_ context.Context, _ string, _ types.I type fakeHeadOnlyStore struct { storage.DoltStorage currentCommitCalls int + issues []*types.Issue } func (f *fakeHeadOnlyStore) GetCurrentCommit(_ context.Context) (string, error) { @@ -239,6 +246,74 @@ func (f *fakeHeadOnlyStore) GetCurrentCommit(_ context.Context) (string, error) return "head-commit-hash", nil } +func (f *fakeHeadOnlyStore) GetInfraTypes(_ context.Context) map[string]bool { return nil } + +// GetConfig lets buildOwnerExcludeSet's database fallback lookup (for +// export.exclude_owners / export.exclude_owner) run without panicking; this +// fake has no config store, so every key is unset. +func (f *fakeHeadOnlyStore) GetConfig(_ context.Context, _ string) (string, error) { + return "", nil +} + +func (f *fakeHeadOnlyStore) SearchIssues(_ context.Context, _ string, _ types.IssueFilter) ([]*types.Issue, error) { + return f.issues, nil +} + +// The four bulk-relation loaders below let exportToFile's full-export path +// (maybeAutoExport's fallback when incremental isn't available) run to +// completion against this fake: it unconditionally calls all of them +// whenever SearchIssues returns at least one issue. +func (f *fakeHeadOnlyStore) GetLabelsForIssues(_ context.Context, _ []string) (map[string][]string, error) { + return nil, nil +} + +func (f *fakeHeadOnlyStore) GetDependencyRecordsForIssues(_ context.Context, _ []string) (map[string][]*types.Dependency, error) { + return nil, nil +} + +func (f *fakeHeadOnlyStore) GetCommentsForIssues(_ context.Context, _ []string) (map[string][]*types.Comment, error) { + return nil, nil +} + +func (f *fakeHeadOnlyStore) GetCommentCounts(_ context.Context, _ []string) (map[string]int, error) { + return nil, nil +} + +func (f *fakeHeadOnlyStore) GetDependencyCounts(_ context.Context, _ []string) (map[string]*types.DependencyCounts, error) { + return nil, nil +} + +// spyDiffStore wraps a real DoltStorage and counts ChangedIssueIDs calls, +// while delegating every method — including the optional StateHasher/ +// DiffStore capability interfaces — to the wrapped store. It does NOT +// implement storage.Unwrapper, so storage.UnwrapStore returns the spy +// itself unchanged: maybeAutoExport's internal +// storage.UnwrapStore(store).(storage.DiffStore) / .(storage.StateHasher) +// type-assertions land on the spy's own overrides below, letting a test +// observe whether the real dolt_diff-backed incremental path actually ran, +// end-to-end through maybeAutoExport, against a real dolt server. +type spyDiffStore struct { + storage.DoltStorage + changedIssueIDsCalls int +} + +func (s *spyDiffStore) ChangedIssueIDs(ctx context.Context, fromCommit, toCommit string) (storage.ChangedIssueIDs, error) { + s.changedIssueIDsCalls++ + ds, ok := storage.UnwrapStore(s.DoltStorage).(storage.DiffStore) + if !ok { + return storage.ChangedIssueIDs{}, fmt.Errorf("spyDiffStore: wrapped store does not implement DiffStore") + } + return ds.ChangedIssueIDs(ctx, fromCommit, toCommit) +} + +func (s *spyDiffStore) GetStateHash(ctx context.Context) (string, error) { + sh, ok := storage.UnwrapStore(s.DoltStorage).(storage.StateHasher) + if !ok { + return s.DoltStorage.GetCurrentCommit(ctx) + } + return sh.GetStateHash(ctx) +} + func autoExportTestDir(t *testing.T) string { t.Helper() dir := t.TempDir() @@ -1284,3 +1359,1062 @@ func autoExportDataLossTestEnv(home string) []string { } return append(env, "HOME="+home, "BEADS_DOLT_AUTO_START=0", "BEADS_NO_DAEMON=1", "BD_DISABLE_METRICS=1", "BD_DISABLE_EVENT_FLUSH=1") } + +// --------------------------------------------------------------------------- +// Unit tests — no Dolt required. +// --------------------------------------------------------------------------- + +func TestOrderedIssueLines_PreservesInsertionOrderAndReplacesInPlace(t *testing.T) { + o := newOrderedIssueLines() + o.set("a", []byte(`{"id":"a","v":1}`)) + o.set("b", []byte(`{"id":"b","v":1}`)) + o.set("c", []byte(`{"id":"c","v":1}`)) + // Replace b in-place — must NOT move it to the end. + o.set("b", []byte(`{"id":"b","v":2}`)) + // Remove a. + o.remove("a") + // Add d — appended at the end. + o.set("d", []byte(`{"id":"d","v":1}`)) + + var got []string + o.each(func(id string, line []byte) { + got = append(got, string(line)) + }) + want := []string{ + `{"id":"b","v":2}`, + `{"id":"c","v":1}`, + `{"id":"d","v":1}`, + } + if len(got) != len(want) { + t.Fatalf("got %d lines, want %d: %v", len(got), len(want), got) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("line %d: got %q, want %q", i, got[i], want[i]) + } + } +} + +func TestLoadExistingIssueLines_ParsesIssuesPreservesMemories(t *testing.T) { + tmp := t.TempDir() + path := filepath.Join(tmp, "issues.jsonl") + content := strings.Join([]string{ + `{"id":"one","title":"first"}`, + `{"id":"two","title":"second","comments":[{"id":"c1","text":"hi"}]}`, + `{"_type":"memory","key":"k","value":"v"}`, + ` `, + `not valid json`, + `{"id":"","title":"empty id"}`, + `{"id":"three","title":"third"}`, + ``, + }, "\n") + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatal(err) + } + + lines, err := loadExistingIssueLines(path) + if err != nil { + t.Fatalf("loadExistingIssueLines: %v", err) + } + + var ids []string + lines.each(func(id string, _ []byte) { ids = append(ids, id) }) + wantIDs := []string{"one", "two", "three"} + if len(ids) != len(wantIDs) { + t.Fatalf("got ids %v, want %v", ids, wantIDs) + } + for i, id := range wantIDs { + if ids[i] != id { + t.Errorf("order[%d] = %q, want %q", i, ids[i], id) + } + } + + // Memories must NOT be mixed into the issue-line ordering (a memory record + // has no issue "id" to key on), but they also must not be silently + // dropped: an incremental rewrite that never regenerates memories from + // live config depends on loadExistingIssueLines carrying pre-existing + // memory lines forward verbatim via memoryLines. + lines.each(func(_ string, line []byte) { + if strings.Contains(string(line), `"_type":"memory"`) { + t.Errorf("memory record leaked into issue lines: %s", line) + } + }) + if len(lines.memoryLines) != 1 { + t.Fatalf("got %d preserved memory lines, want 1: %v", len(lines.memoryLines), lines.memoryLines) + } + if !strings.Contains(string(lines.memoryLines[0]), `"key":"k"`) { + t.Errorf("preserved memory line = %s, want the original memory record verbatim", lines.memoryLines[0]) + } + + // Comment records have an "id" field too — confirm we grabbed the + // OUTER id (which the json.Unmarshal probe does correctly because + // Go's decoder overwrites repeated keys from left to right, and the + // issue's "id" is first in the object). + var two []byte + lines.each(func(id string, line []byte) { + if id == "two" { + two = line + } + }) + if two == nil { + t.Fatal("issue 'two' not loaded") + } + if !strings.Contains(string(two), `"comments":[{"id":"c1"`) { + t.Errorf("comments not preserved verbatim: %s", two) + } +} + +func TestLoadExistingIssueLines_MissingFileReturnsEmpty(t *testing.T) { + lines, err := loadExistingIssueLines(filepath.Join(t.TempDir(), "nope.jsonl")) + if err != nil { + t.Fatalf("missing file should not error, got %v", err) + } + called := false + lines.each(func(_ string, _ []byte) { called = true }) + if called { + t.Error("empty set yielded entries") + } +} + +// --------------------------------------------------------------------------- +// Integration tests — require the shared Dolt test server. +// --------------------------------------------------------------------------- + +// setupIncrementalExportTest wires a fresh store, puts the cwd in a temp +// beads dir, and registers cleanup. Returns the store, beads dir, and ctx. +func setupIncrementalExportTest(t *testing.T) (*testHarness, context.Context) { + t.Helper() + return setupIncrementalExportTestWithReadTimeout(t, 0) +} + +// bulkSeedPoolReadTimeout is the Config.PoolReadTimeout given to tests whose +// own write volume, not server health, needs more slack than +// defaultPoolReadTimeout's 10s (internal/storage/dolt/store.go). be-uoat +// round 2: TestTryIncrementalExport_ThresholdExceededFallsBack's 5001-row +// mustCreateBatch seed writes sequentially over one held connection +// (MaxOpenConns=1 on the shared-branch test path), so any single read +// stalling past 10s under host contention trips the pool timeout mid-batch +// ("invalid connection" / "i/o timeout") — independently reproduced 2 of 3 +// runs at 269.44s/316.35s/67.80s wall-clock (release-gates/be-uoat-dolt-diff- +// export-gate.md), evidence of real contention, not a hung process. 5m +// reuses the same tier already established in this codebase for this exact +// class of host-contention stall (execWithLongTimeout's push/pull deadline, +// and testStoreOpenTimeout's 300s in test_helpers_test.go) rather than a +// guessed value — comfortably above any observed run, while still bounded +// so a genuine hang still fails. +const bulkSeedPoolReadTimeout = 5 * time.Minute + +// setupIncrementalExportTestWithReadTimeout is setupIncrementalExportTest +// with a caller-specified Config.PoolReadTimeout (0 = defaultPoolReadTimeout, +// i.e. identical to setupIncrementalExportTest). See bulkSeedPoolReadTimeout. +func setupIncrementalExportTestWithReadTimeout(t *testing.T, readTimeout time.Duration) (*testHarness, context.Context) { + t.Helper() + if testDoltServerPort == 0 { + t.Skip("Dolt test server not available") + } + if testutil.DoltContainerCrashed() { + t.Skipf("Dolt test server crashed: %v", testutil.DoltContainerCrashError()) + } + + ensureTestMode(t) + saveAndRestoreGlobals(t) + + tmpDir := t.TempDir() + beadsDir := filepath.Join(tmpDir, ".beads") + if err := os.MkdirAll(beadsDir, 0o755); err != nil { + t.Fatal(err) + } + + origWd, _ := os.Getwd() + if err := os.Chdir(tmpDir); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = os.Chdir(origWd) }) + + dbName := uniqueTestDBName(t) + testDBPath := filepath.Join(beadsDir, "dolt") + writeTestMetadata(t, testDBPath, dbName) + s := newTestStoreWithPrefixAndReadTimeout(t, testDBPath, "test", readTimeout) + store = s + storeMutex.Lock() + storeActive = true + storeMutex.Unlock() + t.Cleanup(func() { + store = nil + storeMutex.Lock() + storeActive = false + storeMutex.Unlock() + }) + + ctx := context.Background() + rootCtx = ctx + + return &testHarness{store: s, beadsDir: beadsDir}, ctx +} + +type testHarness struct { + store storage.DoltStorage + beadsDir string +} + +func (h *testHarness) mustCreate(t *testing.T, ctx context.Context, id, title string) { + t.Helper() + iss := &types.Issue{ + ID: id, + Title: title, + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeTask, + } + if err := h.store.CreateIssue(ctx, iss, "tester"); err != nil { + t.Fatalf("CreateIssue(%s): %v", id, err) + } +} + +// mustCreateBatch creates n issues (IDs "NNNNN") in a single +// transaction via CreateIssuesWithFullOptions, instead of n separate +// CreateIssue round trips over the connection. SkipPrefixValidation matches +// mustCreate/CreateIssue's single-issue behavior so callers can keep using +// IDs that don't match the store's configured issue_prefix (as the existing +// "thr-*" ids here do). +func (h *testHarness) mustCreateBatch(t *testing.T, ctx context.Context, n int, prefix string) { + t.Helper() + issues := make([]*types.Issue, n) + for i := 0; i < n; i++ { + issues[i] = &types.Issue{ + ID: fmt.Sprintf("%s%05d", prefix, i), + Title: fmt.Sprintf("t%05d", i), + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeTask, + } + } + if err := h.store.CreateIssuesWithFullOptions(ctx, issues, "tester", storage.BatchCreateOptions{SkipPrefixValidation: true}); err != nil { + t.Fatalf("CreateIssuesWithFullOptions(%d issues): %v", n, err) + } +} + +func (h *testHarness) mustCommit(t *testing.T, ctx context.Context, msg string) string { + t.Helper() + if err := h.store.Commit(ctx, msg); err != nil { + t.Fatalf("Commit(%q): %v", msg, err) + } + hash, err := h.store.GetCurrentCommit(ctx) + if err != nil { + t.Fatalf("GetCurrentCommit: %v", err) + } + return hash +} + +func TestChangedIssueIDs_DetectsUpsertsAndRemovals(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + + // Baseline: create three issues and commit. + h.mustCreate(t, ctx, "cid-a", "Alpha") + h.mustCreate(t, ctx, "cid-b", "Beta") + h.mustCreate(t, ctx, "cid-c", "Gamma") + c1 := h.mustCommit(t, ctx, "baseline") + + // Delta: + // - modify cid-a via UpdateIssue (touches issues row) + // - add a label to cid-b (touches labels row only) + // - delete cid-c (touches issues row, diff_type=removed) + if err := h.store.UpdateIssue(ctx, "cid-a", map[string]interface{}{"title": "Alpha Prime"}, "tester"); err != nil { + t.Fatalf("UpdateIssue: %v", err) + } + if err := h.store.AddLabel(ctx, "cid-b", "priority", "tester"); err != nil { + t.Fatalf("AddLabel: %v", err) + } + if err := h.store.DeleteIssue(ctx, "cid-c"); err != nil { + t.Fatalf("DeleteIssue: %v", err) + } + c2 := h.mustCommit(t, ctx, "delta") + + ds, ok := storage.UnwrapStore(h.store).(storage.DiffStore) + if !ok { + t.Fatal("DoltStore should implement DiffStore") + } + changed, err := ds.ChangedIssueIDs(ctx, c1, c2) + if err != nil { + t.Fatalf("ChangedIssueIDs: %v", err) + } + + gotUpserted := idSetFromIDs(changed.Upserted) + gotRemoved := idSetFromIDs(changed.Removed) + + for _, id := range []string{"cid-a", "cid-b"} { + if !gotUpserted[id] { + t.Errorf("%s missing from Upserted (got %v)", id, changed.Upserted) + } + if gotRemoved[id] { + t.Errorf("%s wrongly in Removed", id) + } + } + if !gotRemoved["cid-c"] { + t.Errorf("cid-c missing from Removed (got %v)", changed.Removed) + } + if gotUpserted["cid-c"] { + t.Error("cid-c wrongly in Upserted — a deleted issue must not be upserted even though cascade removes its label/dep rows") + } +} + +func TestTryIncrementalExport_PatchesChangedIssuesAndDropsRemoved(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + + // Baseline: 5 issues, full export. + h.mustCreate(t, ctx, "inc-a", "A") + h.mustCreate(t, ctx, "inc-b", "B") + h.mustCreate(t, ctx, "inc-c", "C") + h.mustCreate(t, ctx, "inc-d", "D") + h.mustCreate(t, ctx, "inc-e", "E") + c1 := h.mustCommit(t, ctx, "baseline") + + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if _, _, err := exportToFile(ctx, exportPath, true); err != nil { + t.Fatalf("exportToFile: %v", err) + } + if got := countIssueLines(t, exportPath); got != 5 { + t.Fatalf("baseline export has %d issues, want 5", got) + } + + // Mutate: rename inc-a, delete inc-b. + if err := h.store.UpdateIssue(ctx, "inc-a", map[string]interface{}{"title": "A-renamed"}, "tester"); err != nil { + t.Fatalf("UpdateIssue: %v", err) + } + if err := h.store.DeleteIssue(ctx, "inc-b"); err != nil { + t.Fatalf("DeleteIssue: %v", err) + } + c2 := h.mustCommit(t, ctx, "mutate") + + issueCount, memoryCount, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("tryIncrementalExport returned error: %v", err) + } + if !didIncremental { + t.Fatal("expected incremental path to succeed") + } + if issueCount != 4 { + t.Errorf("issueCount = %d, want 4 (5 baseline − 1 deleted)", issueCount) + } + _ = memoryCount + + // Verify file state: inc-b gone; inc-a has new title; others unchanged. + titles := loadIssueTitles(t, exportPath) + if _, ok := titles["inc-b"]; ok { + t.Error("inc-b should have been dropped from export") + } + if titles["inc-a"] != "A-renamed" { + t.Errorf("inc-a title = %q, want %q", titles["inc-a"], "A-renamed") + } + for _, id := range []string{"inc-c", "inc-d", "inc-e"} { + if _, ok := titles[id]; !ok { + t.Errorf("untouched issue %s missing from export", id) + } + } +} + +func TestTryIncrementalExport_DropsIssueWhenFlippedToTemplate(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + + h.mustCreate(t, ctx, "flip-a", "Alpha") + h.mustCreate(t, ctx, "flip-b", "Beta") + c1 := h.mustCommit(t, ctx, "baseline") + + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if _, _, err := exportToFile(ctx, exportPath, true); err != nil { + t.Fatalf("exportToFile: %v", err) + } + if got := countIssueLines(t, exportPath); got != 2 { + t.Fatalf("baseline export has %d issues, want 2", got) + } + + // Flip flip-a to a template in place. UpdateIssue doesn't toggle + // is_template directly, so go through raw SQL — that mirrors what + // bd's template-promotion flow eventually writes anyway. + doltStore, ok := h.store.(interface { + DB() *sql.DB + }) + if !ok { + t.Skip("store does not expose DB() for raw SQL; can't exercise template flip") + } + if _, err := doltStore.DB().ExecContext(ctx, `UPDATE issues SET is_template = 1 WHERE id = ?`, "flip-a"); err != nil { + t.Fatalf("UPDATE is_template: %v", err) + } + c2 := h.mustCommit(t, ctx, "promote to template") + + _, _, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("tryIncrementalExport: %v", err) + } + if !didIncremental { + t.Fatal("expected incremental path to run") + } + + titles := loadIssueTitles(t, exportPath) + if _, stillThere := titles["flip-a"]; stillThere { + t.Error("flip-a should have been dropped from export once flipped to a template") + } + if _, ok := titles["flip-b"]; !ok { + t.Error("flip-b (untouched) must remain in the export") + } +} + +func TestTryIncrementalExport_FallsBackWhenFileMissing(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + + h.mustCreate(t, ctx, "fb-a", "A") + c1 := h.mustCommit(t, ctx, "first") + h.mustCreate(t, ctx, "fb-b", "B") + c2 := h.mustCommit(t, ctx, "second") + + // No existing file → must return didIncremental=false and leave the + // disk untouched so the caller falls back to the full-export path. + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + issueCount, _, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if didIncremental { + t.Fatal("expected fallback when file is missing") + } + if issueCount != 0 { + t.Errorf("issueCount on fallback = %d, want 0", issueCount) + } + if _, err := os.Stat(exportPath); !os.IsNotExist(err) { + t.Error("fallback path must not create a file") + } +} + +func TestTryIncrementalExport_ThresholdExceededFallsBack(t *testing.T) { + h, ctx := setupIncrementalExportTestWithReadTimeout(t, bulkSeedPoolReadTimeout) + + // Seed one issue so the file exists; baseline commit. + h.mustCreate(t, ctx, "thr-0", "seed") + c1 := h.mustCommit(t, ctx, "seed") + + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if _, _, err := exportToFile(ctx, exportPath, true); err != nil { + t.Fatalf("exportToFile: %v", err) + } + sizeBefore, err := os.ReadFile(exportPath) + if err != nil { + t.Fatal(err) + } + + // Create more issues than the threshold in a single transaction/commit + // (not incrementalExportThreshold+1 separate CreateIssue round trips — + // be-fgd round-2: holding one connection open across 5001 sequential + // writes intermittently tripped a mid-stream TCP read timeout ["write + // commit result indeterminate after connection loss"], at a different + // point in the loop on each of two consecutive runs). + h.mustCreateBatch(t, ctx, incrementalExportThreshold+1, "thr-") + c2 := h.mustCommit(t, ctx, "flood") + + _, _, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if didIncremental { + t.Fatal("expected fallback when change count exceeds threshold") + } + + // File should be byte-for-byte unchanged since fallback was taken. + sizeAfter, err := os.ReadFile(exportPath) + if err != nil { + t.Fatal(err) + } + if len(sizeBefore) != len(sizeAfter) { + t.Errorf("file was touched on fallback (size %d → %d)", len(sizeBefore), len(sizeAfter)) + } +} + +// TestMaybeAutoExport_SecondRunTakesIncrementalPath_ServerMode is the be-shbed +// regression test for bee-ghosttrack's PR #5806 review finding: the root +// cause was DOLT_HASHOF_DB() (a working-set root hash) being fed straight +// into dolt_diff(), which only accepts real commits or the literal +// 'WORKING' — so dolt_diff always errored and every "incremental" export +// silently fell back to a full rewrite. This test proves the fix by driving +// maybeAutoExport itself (not tryIncrementalExport directly) end-to-end +// against the real dolt test server, and observing that the DiffStore path +// actually executes. +func TestMaybeAutoExport_SecondRunTakesIncrementalPath_ServerMode(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + initConfigForTest(t) + config.Set("export.auto", true) + config.Set("export.interval", "1ms") + + spy := &spyDiffStore{DoltStorage: h.store} + store = spy + + h.mustCreate(t, ctx, "e2e-a", "Alpha") + h.mustCreate(t, ctx, "e2e-b", "Beta") + h.mustCreate(t, ctx, "e2e-c", "Gamma") + h.mustCommit(t, ctx, "baseline") + + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("first maybeAutoExport: %v", err) + } + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if got := countIssueLines(t, exportPath); got != 3 { + t.Fatalf("after first export, %d issue lines, want 3", got) + } + + // Mutate: rename e2e-a, delete e2e-b, leave e2e-c untouched. Committed + // (not just working-set-dirty) so the state hash unambiguously moves and + // a second export is triggered. + if err := h.store.UpdateIssue(ctx, "e2e-a", map[string]interface{}{"title": "Alpha renamed"}, "tester"); err != nil { + t.Fatalf("UpdateIssue: %v", err) + } + if err := h.store.DeleteIssue(ctx, "e2e-b"); err != nil { + t.Fatalf("DeleteIssue: %v", err) + } + h.mustCommit(t, ctx, "mutate") + + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("second maybeAutoExport: %v", err) + } + + // Both ChangedIssueIDs call sites must have fired: the orphan guard's + // proof-of-deletion probe (missingJSONLIssueIDsInStore, which runs + // because deleting e2e-b leaves it JSONL-only) and tryIncrementalExport's + // own diff. A bare "called at least once" check is NOT sufficient and was + // the PR #5806 round-5 review finding: the guard's own call satisfies it + // even when tryIncrementalExport falls back to a full export, so the test + // would pass while proving nothing about the path it is named for. + if spy.changedIssueIDsCalls != 2 { + t.Errorf("ChangedIssueIDs called %d times, want 2 (orphan-guard deletion probe + incremental diff) — incremental export never actually reached the dolt_diff-backed DiffStore path (root-cause regression: WORKING-set hash fed to dolt_diff, silently falling back to full export every time)", spy.changedIssueIDsCalls) + } + + // The load-bearing oracle for this test's name: dirtyIDs reaches + // LastDirtyIDs only from a successful incremental patch — maybeAutoExport + // explicitly nils it on the full-export fallback — so an exact {e2e-a} + // here proves the export WAS incremental, not merely that a diff was + // reached. e2e-a is the only upserted id: e2e-b was removed (not + // upserted) and e2e-c was untouched. + if got, want := loadExportAutoState(h.beadsDir).LastDirtyIDs, []string{"e2e-a"}; !reflect.DeepEqual(got, want) { + t.Errorf("LastDirtyIDs = %v, want %v — a nil/empty value means the export fell back to a full rewrite instead of taking the incremental path", got, want) + } + + titles := loadIssueTitles(t, exportPath) + if _, stillThere := titles["e2e-b"]; stillThere { + t.Error("e2e-b should have been dropped from export") + } + if titles["e2e-a"] != "Alpha renamed" { + t.Errorf("e2e-a title = %q, want %q", titles["e2e-a"], "Alpha renamed") + } + if _, ok := titles["e2e-c"]; !ok { + t.Error("untouched issue e2e-c missing from export") + } + + // Content-equivalence control: the incrementally-patched file must + // describe the same issue set as a fresh full export of the same final + // state (field-for-field, not byte-for-byte — line order and formatting + // are allowed to differ). + controlPath := filepath.Join(t.TempDir(), "control.jsonl") + if _, _, err := exportToFile(ctx, controlPath, false); err != nil { + t.Fatalf("control exportToFile: %v", err) + } + got := jsonlRecordsByID(t, exportPath) + want := jsonlRecordsByID(t, controlPath) + if len(got) != len(want) { + t.Fatalf("incremental export has %d records, control full export has %d", len(got), len(want)) + } + for id, wantRec := range want { + gotRec, ok := got[id] + if !ok { + t.Errorf("record %s present in control export, missing from incremental export", id) + continue + } + if !reflect.DeepEqual(gotRec, wantRec) { + t.Errorf("record %s differs between incremental and full export:\n incremental: %v\n full: %v", id, gotRec, wantRec) + } + } +} + +// TestMaybeAutoExport_HistoryRewindDoesNotProveDeletion is the PR #5806 +// round-5 regression test for review ask 2: "removed since anchor" is not the +// same claim as "deleted by `bd delete`". dolt_diff(anchor, WORKING) reports a +// row as removed whenever it is absent at WORKING, which is equally true after +// the history is rewound out from under the anchor. Without an ancestry +// precondition the orphan guard accepted that as proof of deletion and let the +// export silently drop a live record from issues.jsonl — the exact #4988 +// corruption class the guard exists to prevent. +func TestMaybeAutoExport_HistoryRewindDoesNotProveDeletion(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + initConfigForTest(t) + config.Set("export.auto", true) + config.Set("export.interval", "1ms") + + h.mustCreate(t, ctx, "rw-a", "Alpha") + c1 := h.mustCommit(t, ctx, "baseline") + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("first maybeAutoExport: %v", err) + } + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + + // Second cycle: rw-x lands in both the store and issues.jsonl, and the + // diff anchor advances past c1 to the commit that added it. + h.mustCreate(t, ctx, "rw-x", "Rewound") + h.mustCommit(t, ctx, "add rw-x") + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("second maybeAutoExport: %v", err) + } + if titles := loadIssueTitles(t, exportPath); titles["rw-x"] != "Rewound" { + t.Fatalf("setup: rw-x must be exported before the rewind, got %v", titles) + } + + // Rewind the data dir underneath the anchor. rw-x is now absent from the + // store but still present in issues.jsonl — indistinguishable from a real + // `bd delete` if you only consult dolt_diff(anchor, WORKING). + raw, ok := storage.UnwrapStore(h.store).(storage.RawDBAccessor) + if !ok { + t.Skip("store does not expose raw DB access") + } + if _, err := raw.DB().ExecContext(ctx, "CALL DOLT_RESET('--hard', ?)", c1); err != nil { + t.Fatalf("CALL DOLT_RESET('--hard', %s): %v", c1, err) + } + + // The guard must refuse rather than honor the bogus "removed" verdict: + // the anchor is no longer reachable from HEAD, so nothing is proven. + // maybeAutoExport returns nil either way (a refusal is a warn + skip), so + // the file content is the oracle. + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("third maybeAutoExport: %v", err) + } + if _, stillThere := loadIssueTitles(t, exportPath)["rw-x"]; !stillThere { + t.Error("rw-x was silently dropped from issues.jsonl after a history rewind: the orphan guard treated a rewound-away row as a proven deletion instead of refusing to overwrite") + } +} + +// TestMaybeAutoExport_EmbeddedModeFallsBackToFullExportCleanly documents and +// locks in embedded mode's contract: EmbeddedDoltStore implements neither +// StateHasher nor DiffStore, so it can never take the incremental path — +// but per the fix (be-shbed / PR #5806 review item 3) it must still fall +// back to a full export cleanly, rather than silently producing nothing. +// This is deliberately a fake-store unit test, not a real embedded-mode +// integration test: exercising a genuine EmbeddedDoltStore end-to-end is +// out of scope (the bead explicitly excludes implementing DiffStore/ +// StateHasher on it), and fakeHeadOnlyStore already models exactly the +// capability gap that matters here — no StateHasher, no DiffStore. +func TestMaybeAutoExport_EmbeddedModeFallsBackToFullExportCleanly(t *testing.T) { + initConfigForTest(t) + config.Set("export.auto", true) + + saveAndRestoreGlobals(t) + fake := &fakeHeadOnlyStore{ + issues: []*types.Issue{ + {ID: "emb-a", Title: "Embedded Alpha", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + }, + } + store = fake + + dir := autoExportTestDir(t) + saveExportAutoState(filepath.Join(dir, ".beads"), &exportAutoState{ + LastDoltCommit: "stale-hash", + Timestamp: time.Time{}, // zero: throttle window open + }) + + if err := maybeAutoExport(context.Background(), false); err != nil { + t.Fatalf("maybeAutoExport: %v", err) + } + + exportPath := filepath.Join(dir, ".beads", "issues.jsonl") + titles := loadIssueTitles(t, exportPath) + if titles["emb-a"] != "Embedded Alpha" { + t.Errorf("emb-a title = %q, want %q (embedded mode must still produce a full export, not silently no-op)", titles["emb-a"], "Embedded Alpha") + } + + state := loadExportAutoState(filepath.Join(dir, ".beads")) + if state.LastDoltCommit != "head-commit-hash" { + t.Errorf("state LastDoltCommit = %q, want %q (anchor must advance on a successful fallback export)", state.LastDoltCommit, "head-commit-hash") + } +} + +func TestTryIncrementalExport_NeverLeaksMemoriesIntoAutoExport(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + + h.mustCreate(t, ctx, "mem-a", "Alpha") + c1 := h.mustCommit(t, ctx, "baseline") + + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + // Baseline matches real auto-export usage: includeMemories=false. + if _, memCount, err := exportToFile(ctx, exportPath, false); err != nil { + t.Fatalf("exportToFile: %v", err) + } else if memCount != 0 { + t.Fatalf("baseline memoryCount = %d, want 0", memCount) + } + + // User remembers something AFTER the baseline auto-export. + if err := h.store.SetConfig(ctx, kvPrefix+memoryPrefix+"secret-key", "private context"); err != nil { + t.Fatalf("SetConfig: %v", err) + } + + // An unrelated issue mutation triggers the next auto-export cycle. + if err := h.store.UpdateIssue(ctx, "mem-a", map[string]interface{}{"title": "Alpha updated"}, "tester"); err != nil { + t.Fatalf("UpdateIssue: %v", err) + } + c2 := h.mustCommit(t, ctx, "mutate") + + _, memCount, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("tryIncrementalExport: %v", err) + } + if !didIncremental { + t.Fatal("expected incremental path to run") + } + if memCount != 0 { + t.Errorf("incremental memoryCount = %d, want 0 (auto-export must never include memories)", memCount) + } + + got := readFile(t, exportPath) + if strings.Contains(got, "private context") { + t.Error("incremental auto-export leaked a memory record that was written after the baseline export — auto-export must never regenerate memories from live config") + } + + // Control: an explicit full export with memories included stays clean — + // i.e. this isn't a case where the memory was never written at all. + fullPath := filepath.Join(t.TempDir(), "full-control.jsonl") + if _, memCount, err := exportToFile(ctx, fullPath, true); err != nil { + t.Fatalf("exportToFile control: %v", err) + } else if memCount != 1 { + t.Fatalf("control export memoryCount = %d, want 1 (sanity: the memory really was written)", memCount) + } +} + +func TestTryIncrementalExport_PreservesPreExistingMemoryAcrossPatch(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + + h.mustCreate(t, ctx, "mem-b", "Beta") + c1 := h.mustCommit(t, ctx, "baseline") + + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if err := h.store.SetConfig(ctx, kvPrefix+memoryPrefix+"kept-key", "kept context"); err != nil { + t.Fatalf("SetConfig: %v", err) + } + // A memory already exists in the file BEFORE any incremental patching — + // e.g. from an explicit `bd export --include-memories` the user ran by + // hand. Auto-export's incremental path must not destroy it. + if _, memCount, err := exportToFile(ctx, exportPath, true); err != nil { + t.Fatalf("exportToFile: %v", err) + } else if memCount != 1 { + t.Fatalf("baseline memoryCount = %d, want 1", memCount) + } + + if err := h.store.UpdateIssue(ctx, "mem-b", map[string]interface{}{"title": "Beta updated"}, "tester"); err != nil { + t.Fatalf("UpdateIssue: %v", err) + } + c2 := h.mustCommit(t, ctx, "mutate") + + _, _, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("tryIncrementalExport: %v", err) + } + if !didIncremental { + t.Fatal("expected incremental path to run") + } + + got := readFile(t, exportPath) + if !strings.Contains(got, "kept context") { + t.Error("incremental patch destroyed a pre-existing memory record instead of preserving it") + } +} + +// TestMaybeAutoExport_WorkingSetRevertIsCorrected is the be-shbed regression +// test for PR #5806 review item 4 (LastDirtyIDs): in server mode, dolt +// auto-commit is off, so uncommitted edits live only in the working set. +// dolt_diff(anchorCommit, 'WORKING') only reports rows that differ from the +// anchor COMMIT — if an issue is dirtied in export cycle N and then reverted +// back to its committed value before cycle N+1, the working set once again +// matches the anchor commit for that row, so dolt_diff reports no change — +// even though the file on disk still shows the stale dirty value from cycle +// N. Carrying forward the previous cycle's dirty IDs and re-patching them +// unconditionally is what corrects this. +func TestMaybeAutoExport_WorkingSetRevertIsCorrected(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + initConfigForTest(t) + config.Set("export.auto", true) + config.Set("export.interval", "1ms") + + h.mustCreate(t, ctx, "revert-a", "Original title") + h.mustCommit(t, ctx, "baseline") + + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("baseline maybeAutoExport: %v", err) + } + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if got := loadIssueTitles(t, exportPath)["revert-a"]; got != "Original title" { + t.Fatalf("baseline title = %q, want %q", got, "Original title") + } + + // Dirty the issue in the working set only (no commit) — matches server + // mode with dolt auto-commit off. + if err := h.store.UpdateIssue(ctx, "revert-a", map[string]interface{}{"title": "Dirty title"}, "tester"); err != nil { + t.Fatalf("UpdateIssue (dirty): %v", err) + } + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("dirty maybeAutoExport: %v", err) + } + if got := loadIssueTitles(t, exportPath)["revert-a"]; got != "Dirty title" { + t.Fatalf("dirty title = %q, want %q", got, "Dirty title") + } + + // Revert to the committed value, again without committing. dolt_diff + // between the anchor commit and 'WORKING' now reports NO change for + // revert-a — without carrying it forward as a dirty ID from the + // previous cycle, the file would incorrectly keep showing "Dirty title". + if err := h.store.UpdateIssue(ctx, "revert-a", map[string]interface{}{"title": "Original title"}, "tester"); err != nil { + t.Fatalf("UpdateIssue (revert): %v", err) + } + if err := maybeAutoExport(ctx, false); err != nil { + t.Fatalf("revert maybeAutoExport: %v", err) + } + if got := loadIssueTitles(t, exportPath)["revert-a"]; got != "Original title" { + t.Errorf("after working-set revert, title = %q, want %q (LastDirtyIDs must carry revert-a forward so it gets re-patched even though dolt_diff(anchor, WORKING) reports no change for it)", got, "Original title") + } +} + +func TestTryIncrementalExport_ExcludesConfiguredOwnerFromPatchedIssues(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + initConfigForTest(t) + config.Set("export.exclude_owners", "bot-user") + + h.mustCreate(t, ctx, "own-a", "Kept") + if err := h.store.CreateIssue(ctx, &types.Issue{ + ID: "own-b", Title: "Original", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + CreatedBy: "bot-user", + }, "bot-user"); err != nil { + t.Fatalf("CreateIssue own-b: %v", err) + } + c1 := h.mustCommit(t, ctx, "baseline") + + // Baseline full export: exportToFile already applies owner-exclusion, so + // own-b is correctly absent from the start — this test is about the + // INCREMENTAL patch path, not the baseline. + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if _, _, err := exportToFile(ctx, exportPath, true); err != nil { + t.Fatalf("exportToFile: %v", err) + } + if _, ok := loadIssueTitles(t, exportPath)["own-b"]; ok { + t.Fatal("sanity check: baseline full export should already exclude own-b") + } + + // Mutate the excluded-owner issue. If tryIncrementalExport patches + // changed issues into the file without re-applying owner-exclusion, + // own-b would leak in here even though it never should have appeared. + if err := h.store.UpdateIssue(ctx, "own-b", map[string]interface{}{"title": "Mutated"}, "tester"); err != nil { + t.Fatalf("UpdateIssue own-b: %v", err) + } + c2 := h.mustCommit(t, ctx, "mutate") + + _, _, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("tryIncrementalExport: %v", err) + } + if !didIncremental { + t.Fatal("expected incremental path to run") + } + + titles := loadIssueTitles(t, exportPath) + if _, leaked := titles["own-b"]; leaked { + t.Error("own-b belongs to an excluded owner but was patched into the export by the incremental path") + } + if _, ok := titles["own-a"]; !ok { + t.Error("own-a (kept owner, untouched) missing from export") + } +} + +func TestTryIncrementalExport_PatchedLinesIncludeTypeField(t *testing.T) { + h, ctx := setupIncrementalExportTest(t) + + h.mustCreate(t, ctx, "typ-a", "Alpha") + c1 := h.mustCommit(t, ctx, "baseline") + + exportPath := filepath.Join(h.beadsDir, "issues.jsonl") + if _, _, err := exportToFile(ctx, exportPath, true); err != nil { + t.Fatalf("exportToFile: %v", err) + } + + if err := h.store.UpdateIssue(ctx, "typ-a", map[string]interface{}{"title": "Alpha renamed"}, "tester"); err != nil { + t.Fatalf("UpdateIssue: %v", err) + } + c2 := h.mustCommit(t, ctx, "mutate") + + _, _, _, didIncremental, err := tryIncrementalExport(ctx, exportPath, c1, c2, nil) + if err != nil { + t.Fatalf("tryIncrementalExport: %v", err) + } + if !didIncremental { + t.Fatal("expected incremental path to run") + } + + data, err := os.ReadFile(exportPath) + if err != nil { + t.Fatal(err) + } + found := false + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + var rec struct { + ID string `json:"id"` + Type string `json:"_type"` + } + if err := json.Unmarshal([]byte(line), &rec); err != nil { + t.Fatalf("unmarshal line %q: %v", line, err) + } + if rec.ID != "typ-a" { + continue + } + found = true + if rec.Type != "issue" { + t.Errorf(`patched line for typ-a has _type=%q, want "issue" — every issue record, including incrementally-patched ones, must carry a _type field so readers (and the auto-export shrink guard's own classifyExistingAutoExportRecord) can distinguish it from a memory or unknown record`, rec.Type) + } + } + if !found { + t.Fatal("typ-a not found in export after incremental patch") + } +} + +// TestNewTestStoreWithReadTimeout_AppliesConfiguredTimeout proves the +// PoolReadTimeout parameter added for be-uoat round 2 actually reaches the +// live connection, rather than just being accepted and ignored. An +// unreasonably short timeout must make store creation fail fast (the +// configured value is live); a normal one must still succeed (the plumbing +// doesn't break the default path). TestBuildServerDSN_PoolTimeouts +// (internal/storage/dolt/store_unit_test.go) already covers that +// Config.PoolReadTimeout is formatted into the DSN correctly — this test is +// at the cmd/bd harness layer instead, against the real test Dolt server, to +// prove the new newTestStoreSharedBranchWithReadTimeout/ +// newTestStoreWithPrefixAndReadTimeout plumbing actually threads the caller's +// value through to that mechanism. +func TestNewTestStoreWithReadTimeout_AppliesConfiguredTimeout(t *testing.T) { + if testDoltServerPort == 0 { + t.Skip("Dolt test server not available") + } + if testutil.DoltContainerCrashed() { + t.Skipf("Dolt test server crashed: %v", testutil.DoltContainerCrashError()) + } + ensureTestMode(t) + + tmpDir := t.TempDir() + dbPath := filepath.Join(tmpDir, ".beads", "dolt") + // 1ns can never survive a real handshake/query round trip — this is + // not a race with a slow-but-real server, it's a guaranteed trip. + s, err := tryNewTestStoreWithReadTimeout(t, dbPath, 1*time.Nanosecond) + if err == nil { + s.Close() + t.Fatal("expected store creation to fail with an unreasonably short PoolReadTimeout, but it succeeded") + } + + t.Run("normal timeout still succeeds", func(t *testing.T) { + tmpDir := t.TempDir() + dbPath := filepath.Join(tmpDir, ".beads", "dolt") + s := newTestStoreWithPrefixAndReadTimeout(t, dbPath, "test", bulkSeedPoolReadTimeout) + if s == nil { + t.Fatal("expected non-nil store") + } + }) +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +func idSetFromIDs(ids []string) map[string]bool { + out := make(map[string]bool, len(ids)) + for _, id := range ids { + out[id] = true + } + return out +} + +func countIssueLines(t *testing.T, path string) int { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + n := 0 + for _, line := range strings.Split(string(data), "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + if strings.Contains(line, `"_type":"memory"`) { + continue + } + n++ + } + return n +} + +func loadIssueTitles(t *testing.T, path string) map[string]string { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + out := make(map[string]string) + for _, line := range strings.Split(string(data), "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.Contains(line, `"_type":"memory"`) { + continue + } + var iss struct { + ID string `json:"id"` + Title string `json:"title"` + } + if err := json.Unmarshal([]byte(line), &iss); err != nil { + t.Errorf("unmarshal line %q: %v", line, err) + continue + } + if iss.ID != "" { + out[iss.ID] = iss.Title + } + } + return out +} + +func readFile(t *testing.T, path string) string { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("ReadFile(%s): %v", path, err) + } + return string(data) +} + +// jsonlRecordsByID parses every issue record in an export file (skipping +// memory records) into a generic map, keyed by "id". Used for +// content-equivalence comparisons between an incrementally-patched export +// and a fresh full export of the same state, where line order and byte +// layout are allowed to differ but the field content must match exactly. +func jsonlRecordsByID(t *testing.T, path string) map[string]map[string]interface{} { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("ReadFile(%s): %v", path, err) + } + out := make(map[string]map[string]interface{}) + for _, line := range strings.Split(string(data), "\n") { + line = strings.TrimSpace(line) + if line == "" { + continue + } + if strings.Contains(line, `"_type":"memory"`) { + continue + } + var rec map[string]interface{} + if err := json.Unmarshal([]byte(line), &rec); err != nil { + t.Fatalf("unmarshal line %q: %v", line, err) + } + id, _ := rec["id"].(string) + if id == "" { + continue + } + out[id] = rec + } + return out +} diff --git a/cmd/bd/test_dolt_server_cgo_test.go b/cmd/bd/test_dolt_server_cgo_test.go index a02bec26b..40c60bb06 100644 --- a/cmd/bd/test_dolt_server_cgo_test.go +++ b/cmd/bd/test_dolt_server_cgo_test.go @@ -9,6 +9,7 @@ import ( "os" "github.com/steveyegge/beads/internal/storage/dolt" + "github.com/steveyegge/beads/internal/storage/schema" "github.com/steveyegge/beads/internal/testutil" ) @@ -76,13 +77,38 @@ func startTestDoltServer() func() { // initCmdBDSharedSchema initializes the schema and config on the shared database // and commits to main so branches get a clean snapshot. func initCmdBDSharedSchema(port int) error { - ctx := context.Background() + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() cfg := &dolt.Config{ Path: "/tmp/cmdbd-shared-init", ServerHost: "127.0.0.1", ServerPort: port, Database: testSharedDB, + // CreateIfMissing must be true even though SetupSharedTestDB already + // ran `CREATE DATABASE IF NOT EXISTS` moments ago: on a container + // that only just reported ready, a fresh connection's SHOW DATABASES + // catalog probe can still lag that DDL (GH-1851-class propagation + // race), producing a false "database not found" here. Without this, + // dolt.New takes the immediate-fail branch with no retry at all; + // with it, dolt.New treats the resulting "database exists" (1007) on + // its own CREATE DATABASE as proof of existence and falls through to + // its already-bounded post-create catalog-registration backoff. + CreateIfMissing: true, } + + // This database lives only inside a container startTestDoltServer just + // created for this test binary's run and is torn down with it at process + // exit, so the #4259 remote-migrate gate's "don't independently fork a + // shared remote's schema" concern does not apply — there is no persistent + // remote here to fork away from. Without this override, the gate blocks + // whenever the container image's baked-in schema version lags the current + // migration set, forcing every store-opening test in the binary onto the + // slow per-test-DB fallback path (openExistingTestDB and friends in + // test_helpers_test.go), which in turn becomes unstable — confirmed via + // be-fgd round-2 triage — once two or more full-migration replays land + // back-to-back against the same long-lived container. + schema.SetForceAllowRemoteMigrate(true) + defer schema.SetForceAllowRemoteMigrate(false) store, err := dolt.New(ctx, cfg) if err != nil { return fmt.Errorf("New: %w", err) diff --git a/cmd/bd/test_helpers_test.go b/cmd/bd/test_helpers_test.go index 6ba0a348a..01e6e0d52 100644 --- a/cmd/bd/test_helpers_test.go +++ b/cmd/bd/test_helpers_test.go @@ -27,6 +27,32 @@ import ( // testDoltServerPort is the port of the shared test Dolt server (0 = not running). var testDoltServerPort int +// testStoreOpenTimeout bounds dolt.New/dolt.NewFromConfig calls in these +// helpers. The testcontainers wait strategy only confirms the TCP listener +// is up, not that Dolt's SQL engine can serve queries yet (see +// waitForDoltReady in internal/testutil/testdoltserver.go for the same class +// of race on shared-container startup). A query issued with an unbounded +// context in that narrow window can block indefinitely instead of erroring, +// because nothing ever cancels it to unstick the read — confirmed via +// goroutine dump during be-fgd round-2 triage: a per-test dolt.New() sat in +// mysqlConn.readWithTimeout / net.Read for 2+ minutes after the container +// had already logged ready. +// +// The fallback branch (per-test DB, exercised only when shared-schema init +// in test_dolt_server_cgo_test.go fails) runs a full v0->v65 migration +// replay from scratch, which is genuinely slow, not hung: a single isolated +// measurement (BEADS_TEST_FORCE_FALLBACK_DIAG diagnostic, reverted after +// use) clocked one such replay at 219.62s on this hardware. 60s (an earlier +// calibration attempt, matching contributor_routing_e2e_test.go's precedent) +// measurably undershot this and produced a clean "context deadline exceeded" +// at exactly 60.01s instead of the real result. 300s gives that observed +// value ~37% margin for run-to-run variance while still failing fast +// relative to an actually-stuck connection. In normal operation the shared +// fast path (newTestStoreSharedBranch) avoids this branch entirely and +// resolves in under a second; this bound only matters for the rare/backstop +// fallback case. +const testStoreOpenTimeout = 300 * time.Second + // writeTestMetadata writes metadata.json in the .beads directory (parent of dbPath) // so that NewFromConfig can find the correct database name and server settings when // routing reopens a store by path. @@ -74,7 +100,8 @@ func newTestStoreIsolatedDB(t *testing.T, dbPath string, prefix string) *dolt.Do t.Skipf("Dolt test server crashed: %v", testutil.DoltContainerCrashError()) } - ctx := context.Background() + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() cfg := &dolt.Config{ Path: dbPath, @@ -125,14 +152,15 @@ func newTestStoreWithPrefix(t *testing.T, dbPath string, prefix string) *dolt.Do t.Skipf("Dolt test server crashed: %v", testutil.DoltContainerCrashError()) } - ctx := context.Background() - // Fast path: use shared DB with branch-per-test isolation (bd-xmf) if testSharedDB != "" { return newTestStoreSharedBranch(t, dbPath, prefix) } // Fallback: per-test database (original slow path) + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() + cfg := &dolt.Config{ Path: dbPath, ServerHost: "127.0.0.1", @@ -172,7 +200,8 @@ func newTestStoreWithPrefix(t *testing.T, dbPath string, prefix string) *dolt.Do // the expensive CREATE DATABASE + schema init + DROP DATABASE + PURGE cycle. func newTestStoreSharedBranch(t *testing.T, dbPath string, prefix string) *dolt.DoltStore { t.Helper() - ctx := context.Background() + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() // Write metadata.json pointing to the shared database writeTestMetadata(t, dbPath, testSharedDB) @@ -209,6 +238,170 @@ func newTestStoreSharedBranch(t *testing.T, dbPath string, prefix string) *dolt. return s } +// newTestStoreSharedBranchWithReadTimeout is newTestStoreSharedBranch with a +// caller-specified Config.PoolReadTimeout. Existing shared-branch callers +// stay on the 10s default (defaultPoolReadTimeout in internal/storage/dolt); +// this is for the rare test whose own write volume — not server health — +// needs more slack, e.g. a bulk seed of thousands of rows over the single +// held connection that MaxOpenConns=1 forces on this path (be-uoat round 2). +func newTestStoreSharedBranchWithReadTimeout(t *testing.T, dbPath string, prefix string, readTimeout time.Duration) *dolt.DoltStore { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() + + // Write metadata.json pointing to the shared database + writeTestMetadata(t, dbPath, testSharedDB) + + // Open store against the shared database with MaxOpenConns=1 + // (required for DOLT_CHECKOUT session affinity) + doltNewMutex.Lock() + s, err := dolt.New(ctx, &dolt.Config{ + Path: dbPath, + ServerHost: "127.0.0.1", + ServerPort: testDoltServerPort, + Database: testSharedDB, + MaxOpenConns: 1, + PoolReadTimeout: readTimeout, + }) + doltNewMutex.Unlock() + if err != nil { + t.Fatalf("Failed to create dolt store (shared): %v", err) + } + + // Create isolated branch for this test + _, branchCleanup := testutil.StartTestBranch(t, s.DB(), testSharedDB) + + // Set prefix for this test (overrides the shared schema's default) + if err := s.SetConfig(ctx, "issue_prefix", prefix); err != nil { + branchCleanup() + s.Close() + t.Fatalf("Failed to set issue_prefix: %v", err) + } + + t.Cleanup(func() { + branchCleanup() + s.Close() + }) + return s +} + +// newTestStoreWithPrefixAndReadTimeout is newTestStoreWithPrefix with a +// caller-specified Config.PoolReadTimeout, threaded through whichever branch +// (shared-DB fast path or per-test-DB fallback) actually runs. See +// newTestStoreSharedBranchWithReadTimeout for why this knob exists. +func newTestStoreWithPrefixAndReadTimeout(t *testing.T, dbPath string, prefix string, readTimeout time.Duration) *dolt.DoltStore { + t.Helper() + + ensureTestMode(t) + + if testDoltServerPort == 0 { + t.Skip("Dolt test server not available, skipping") + } + if testutil.DoltContainerCrashed() { + t.Skipf("Dolt test server crashed: %v", testutil.DoltContainerCrashError()) + } + + // Fast path: use shared DB with branch-per-test isolation (bd-xmf) + if testSharedDB != "" { + return newTestStoreSharedBranchWithReadTimeout(t, dbPath, prefix, readTimeout) + } + + // Fallback: per-test database (original slow path) + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() + + cfg := &dolt.Config{ + Path: dbPath, + ServerHost: "127.0.0.1", + ServerPort: testDoltServerPort, + Database: uniqueTestDBName(t), + CreateIfMissing: true, + PoolReadTimeout: readTimeout, + } + writeTestMetadata(t, dbPath, cfg.Database) + + doltNewMutex.Lock() + s, err := dolt.New(ctx, cfg) + doltNewMutex.Unlock() + if err != nil { + t.Fatalf("Failed to create dolt store: %v", err) + } + + if err := s.SetConfig(ctx, "issue_prefix", prefix); err != nil { + s.Close() + t.Fatalf("Failed to set issue_prefix: %v", err) + } + if err := s.SetConfig(ctx, "types.custom", "molecule,gate,convoy,merge-request,slot,agent,role,rig,event,message"); err != nil { + s.Close() + t.Fatalf("Failed to set types.custom: %v", err) + } + + t.Cleanup(func() { + s.Close() + if cfg.Database != "" { + dropTestDatabase(cfg.Database, testDoltServerPort) + } + }) + return s +} + +// tryNewTestStoreWithReadTimeout is the dolt.New portion of +// newTestStoreWithPrefixAndReadTimeout (same branching, same Config shape), +// returning the error directly instead of calling t.Fatal on it. Use this +// when the caller expects the open itself to fail — e.g. proving a short +// PoolReadTimeout is actually honored — since a t.Run subtest's failure +// always propagates FAIL to the parent test and the process exit code in +// Go's testing package regardless of what the parent does with t.Run's bool +// return; there is no way to observe an "expected" subtest failure without +// also failing the overall run. +func tryNewTestStoreWithReadTimeout(t *testing.T, dbPath string, readTimeout time.Duration) (*dolt.DoltStore, error) { + t.Helper() + + ensureTestMode(t) + + if testDoltServerPort == 0 { + t.Skip("Dolt test server not available, skipping") + } + if testutil.DoltContainerCrashed() { + t.Skipf("Dolt test server crashed: %v", testutil.DoltContainerCrashError()) + } + + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() + + // Fast path: use shared DB with branch-per-test isolation (bd-xmf) + if testSharedDB != "" { + writeTestMetadata(t, dbPath, testSharedDB) + doltNewMutex.Lock() + s, err := dolt.New(ctx, &dolt.Config{ + Path: dbPath, + ServerHost: "127.0.0.1", + ServerPort: testDoltServerPort, + Database: testSharedDB, + MaxOpenConns: 1, + PoolReadTimeout: readTimeout, + }) + doltNewMutex.Unlock() + return s, err + } + + // Fallback: per-test database (original slow path) + cfg := &dolt.Config{ + Path: dbPath, + ServerHost: "127.0.0.1", + ServerPort: testDoltServerPort, + Database: uniqueTestDBName(t), + CreateIfMissing: true, + PoolReadTimeout: readTimeout, + } + writeTestMetadata(t, dbPath, cfg.Database) + + doltNewMutex.Lock() + s, err := dolt.New(ctx, cfg) + doltNewMutex.Unlock() + return s, err +} + // dropTestDatabase drops a test database from the shared server (best-effort cleanup). func dropTestDatabase(dbName string, port int) { dsn := doltutil.ServerDSN{Host: "127.0.0.1", Port: port, User: "root"}.String() @@ -233,7 +426,8 @@ func openExistingTestDB(t *testing.T, dbPath string) (*dolt.DoltStore, error) { // Serialize dolt.New() to avoid race in Dolt's InitStatusVariables (bd-cqjoi) doltNewMutex.Lock() defer doltNewMutex.Unlock() - ctx := context.Background() + ctx, cancel := context.WithTimeout(context.Background(), testStoreOpenTimeout) + defer cancel() // Try NewFromConfig which reads metadata.json for correct database name beadsDir := filepath.Dir(dbPath) if store, err := dolt.NewFromConfig(ctx, beadsDir); err == nil { diff --git a/internal/storage/diff_store.go b/internal/storage/diff_store.go new file mode 100644 index 000000000..ee976adee --- /dev/null +++ b/internal/storage/diff_store.go @@ -0,0 +1,25 @@ +package storage + +import "context" + +// DiffStore computes the set of issue IDs whose underlying data differs +// between two Dolt commits. It's a capability interface — callers should +// type-assert after UnwrapStore and fall back to a non-incremental path +// when a store doesn't implement it. +// +// "Underlying data" includes changes to the issues table itself as well as +// its label, dependency, and comment rows. An issue ID is reported under +// Upserted if any of those rows were added or modified, and under Removed +// only if the issue row itself was deleted. Label/dependency/comment row +// deletions are reported as Upserted (the issue survived, its relational +// data shrank). +type DiffStore interface { + ChangedIssueIDs(ctx context.Context, fromCommit, toCommit string) (ChangedIssueIDs, error) +} + +// ChangedIssueIDs lists the issue IDs whose stored representation differs +// between two commits. +type ChangedIssueIDs struct { + Upserted []string + Removed []string +} diff --git a/internal/storage/dolt/versioned.go b/internal/storage/dolt/versioned.go index 6df97416e..47859f949 100644 --- a/internal/storage/dolt/versioned.go +++ b/internal/storage/dolt/versioned.go @@ -68,6 +68,93 @@ func (s *DoltStore) PreviousExternalRef(ctx context.Context, issueID string, asO return ref, found, err } +// ChangedIssueIDs returns the set of issue IDs whose data differs between +// fromCommit and toCommit, derived from dolt_diff over the issues, labels, +// dependencies, and comments tables. An issue is reported under Removed +// only when the issues-table row itself was deleted; label/dep/comment row +// deletions for surviving issues fall under Upserted. +// +// Implements storage.DiffStore. +func (s *DoltStore) ChangedIssueIDs(ctx context.Context, fromCommit, toCommit string) (storage.ChangedIssueIDs, error) { + var out storage.ChangedIssueIDs + if fromCommit == "" || toCommit == "" { + return out, fmt.Errorf("ChangedIssueIDs: fromCommit and toCommit required") + } + // Dolt's dolt_diff() table function does not accept prepared-statement + // bind parameters — its arguments must be SQL string literals. That + // makes us inline the commit hashes into the query, so validate them + // aggressively to keep the path free of injection risk. + if !isSafeCommitRef(fromCommit) { + return out, fmt.Errorf("ChangedIssueIDs: fromCommit %q is not a valid commit ref", fromCommit) + } + if !isSafeCommitRef(toCommit) { + return out, fmt.Errorf("ChangedIssueIDs: toCommit %q is not a valid commit ref", toCommit) + } + + // dolt_diff returns from_/to_ for every column plus diff_type. + // COALESCE(to_id, from_id) extracts the row's identifying value regardless + // of whether the row was added, modified, or removed. + // + // We UNION results from the four tables that contribute to auto-export's + // per-issue JSON record. Any change to any of them means the issue's + // export representation has changed. + // + //nolint:gosec // G201: fromCommit/toCommit are constrained to [A-Za-z0-9] + // by isSafeCommitRef above; dolt_diff() does not accept bind parameters. + q := fmt.Sprintf(` + SELECT id, MAX(is_removed) AS is_removed + FROM ( + SELECT COALESCE(to_id, from_id) AS id, + CASE WHEN diff_type = 'removed' THEN 1 ELSE 0 END AS is_removed + FROM dolt_diff('%[1]s', '%[2]s', 'issues') + UNION ALL + SELECT COALESCE(to_issue_id, from_issue_id) AS id, 0 AS is_removed + FROM dolt_diff('%[1]s', '%[2]s', 'labels') + UNION ALL + SELECT COALESCE(to_issue_id, from_issue_id) AS id, 0 AS is_removed + FROM dolt_diff('%[1]s', '%[2]s', 'dependencies') + UNION ALL + SELECT COALESCE(to_issue_id, from_issue_id) AS id, 0 AS is_removed + FROM dolt_diff('%[1]s', '%[2]s', 'comments') + ) AS u + WHERE id IS NOT NULL AND id <> '' + GROUP BY id + `, fromCommit, toCommit) + rows, err := s.db.QueryContext(ctx, q) + if err != nil { + return out, fmt.Errorf("dolt_diff query: %w", err) + } + defer rows.Close() + + for rows.Next() { + var id string + var removed int + if err := rows.Scan(&id, &removed); err != nil { + return out, fmt.Errorf("scan dolt_diff row: %w", err) + } + // MAX(is_removed) resolves an id that surfaces from more than one of + // the four unioned sub-selects (e.g. its own row is untouched but a + // comment on it changed) — only the 'issues' sub-select ever + // contributes is_removed=1, so MAX just lets that value win over the + // label/dependency/comment sub-selects' hardcoded 0. It is not + // resolving a remove-then-readd within the range: dolt_diff(from, + // to, table) diffs exactly those two snapshots rather than walking + // intermediate commits, so a single call can never emit both a 0 and + // a 1 row for the same id. An issue present at toCommit — even if it + // was deleted and re-added somewhere between fromCommit and + // toCommit — always reports is_removed=0 here. + if removed == 1 { + out.Removed = append(out.Removed, id) + } else { + out.Upserted = append(out.Upserted, id) + } + } + if err := rows.Err(); err != nil { + return out, fmt.Errorf("iterate dolt_diff rows: %w", err) + } + return out, nil +} + // ListBranches returns the names of all branches. // Implements storage.VersionedStorage. func (s *DoltStore) ListBranches(ctx context.Context) ([]string, error) { @@ -154,3 +241,21 @@ func (s *DoltStore) ResolveConflictRows(ctx context.Context, table string, keys func (s *DoltStore) CommitExists(ctx context.Context, commitHash string) (bool, error) { return versioncontrolops.CommitExists(ctx, s.db, commitHash) } + +// isSafeCommitRef reports whether s may be inlined into a dolt_diff() +// SQL literal without exposing an injection surface. Dolt commit hashes +// are ASCII base32 (a-z + digits); we also allow an explicit-length cap +// to catch accidental truncation or giant inputs. +func isSafeCommitRef(s string) bool { + if len(s) == 0 || len(s) > 64 { + return false + } + for i := 0; i < len(s); i++ { + c := s[i] + if (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') { + continue + } + return false + } + return true +} diff --git a/internal/storage/dolt/versioned_test.go b/internal/storage/dolt/versioned_test.go index 1cb6ff19b..c4aab71e5 100644 --- a/internal/storage/dolt/versioned_test.go +++ b/internal/storage/dolt/versioned_test.go @@ -287,3 +287,39 @@ func TestCommitPending(t *testing.T) { } }) } + +// TestIsSafeCommitRef is a be-shbed / PR #5806 review regression test. +// dolt_diff() accepts the literal "WORKING" as an endpoint alongside real +// commit hashes, and the fix threads that literal through ChangedIssueIDs as +// the incremental path's "to" endpoint (in place of a root/working-set hash +// dolt_diff always rejected) — isSafeCommitRef's character/length check must +// keep admitting it, since no prior test in this file covered that value. +func TestIsSafeCommitRef(t *testing.T) { + valid := []string{ + "WORKING", + "a", + "0123456789abcdefABCDEF", + strings.Repeat("a", 64), + } + for _, s := range valid { + if !isSafeCommitRef(s) { + t.Errorf("isSafeCommitRef(%q) = false, want true", s) + } + } + + invalid := []string{ + "", + strings.Repeat("a", 65), + "has space", + "has-dash", + "has_underscore", + "has.dot", + "'; DROP TABLE issues; --", + "abc\ndef", + } + for _, s := range invalid { + if isSafeCommitRef(s) { + t.Errorf("isSafeCommitRef(%q) = true, want false", s) + } + } +} diff --git a/internal/testutil/testdoltserver.go b/internal/testutil/testdoltserver.go index 624c97162..fa2013506 100644 --- a/internal/testutil/testdoltserver.go +++ b/internal/testutil/testdoltserver.go @@ -4,6 +4,7 @@ package testutil import ( "context" + "database/sql" "fmt" "os" "os/exec" @@ -162,6 +163,12 @@ func startDoltContainer() error { } doltTestPort = p.Port() + + if err := waitForDoltReady(doltTestPort); err != nil { + _ = testcontainers.TerminateContainer(ctr) + return fmt.Errorf("waiting for Dolt server to be query-ready: %w", err) + } + doltSingletonSrv = &doltServer{ container: ctr, } @@ -169,6 +176,51 @@ func startDoltContainer() error { return nil } +// doltReadyProbeTimeout bounds how long waitForDoltReady polls before giving up. +const doltReadyProbeTimeout = 30 * time.Second + +// waitForDoltReady polls the server with a trivial query until it responds +// or doltReadyProbeTimeout elapses. +// +// The testcontainers wait strategy above only matches a log line ("Server +// ready. Accepting connections."), which confirms the TCP listener is up but +// not that Dolt's SQL engine can actually serve a query yet. In that narrow +// startup window, a query issued with an unbounded context (as several test +// helpers do) can block indefinitely instead of erroring, because nothing +// ever cancels it to unstick the read — confirmed via goroutine dump during +// be-fgd round-2 triage: the connection sat in mysqlConn.readWithTimeout / +// net.Read for 2+ minutes after the container had already logged ready. Each +// probe attempt here uses its own short bounded context, so a still-warming +// server fails an attempt fast and gets retried, rather than wedging. +func waitForDoltReady(port string) error { + dsn := fmt.Sprintf("root@tcp(127.0.0.1:%s)/", port) + deadline := time.Now().Add(doltReadyProbeTimeout) + var lastErr error + for { + lastErr = pingDoltOnce(dsn) + if lastErr == nil { + return nil + } + if time.Now().After(deadline) { + return fmt.Errorf("dolt server at port %s did not become query-ready within %s: %w", port, doltReadyProbeTimeout, lastErr) + } + time.Sleep(200 * time.Millisecond) + } +} + +// pingDoltOnce opens a short-lived connection and pings it with a bounded +// context so a non-responsive server fails this attempt instead of hanging. +func pingDoltOnce(dsn string) error { + db, err := sql.Open("mysql", dsn) + if err != nil { + return err + } + defer db.Close() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + return db.PingContext(ctx) +} + // terminateSharedContainer stops and removes the shared Dolt container. // Safe to call concurrently or multiple times (sync.Once). func terminateSharedContainer() { diff --git a/release-gates/be-y1jo-dolt-diff-export-gate.md b/release-gates/be-y1jo-dolt-diff-export-gate.md new file mode 100644 index 000000000..773a87d72 --- /dev/null +++ b/release-gates/be-y1jo-dolt-diff-export-gate.md @@ -0,0 +1,211 @@ +# Release gate — Incremental auto-export via `dolt_diff` (be-hka), round 4 + +- **Builder bead (CLOSED):** be-hka — incremental `bd export --auto` using + `dolt_diff()` to patch only changed issues instead of rewriting the full + JSONL file on every export. +- **Deploy bead:** be-y1jo (round 2 of deploy; supersedes be-uoat, which + FAILed on criterion 3 — see `release-gates/be-uoat-dolt-diff-export-gate.md`) +- **Review beads:** + - be-fgd — round 1, findings led to round-2 rework (dolt test-store hang) + - be-wu03 — round 2, verdict **PASS**, commit `d8f6563393a3f` (this is the + commit be-uoat evaluated and FAILed on criterion 3 only) + - be-8wnq — round 3, verdict **REQUEST CHANGES**, commit `f35368477` — fixed + be-uoat's flake at the root (see below) but introduced a genuine, + deterministic test-logic defect in the new plumbing-proof test + - be-unlq — round 4, verdict **PASS**, commit `fc5caadcf63371f6b992f32d23ef4d8c2a71e5ee` +- **Commit:** `fc5caadcf63371f6b992f32d23ef4d8c2a71e5ee` — 5 commits over + `origin/main` (`5e94e0cc5`, `561c45a03`, `9c4f2e798`, `f35368477`, + `fc5caadcf`), 7 files +- **Branch:** `builder/be-hka` (provenance only); deploy branch + `deploy/be-y1jo-gate` cut from `fc5caadcf` and pushed to `headfork` + (`quad341/beads-sec003-contrib`, confirmed GitHub fork-parent of + `gastownhall/beads`) — per be-eh6 precedent and this feature's own round-2 + gate note, `fork` (`quad341/beads`) is the pre-migration URL and is avoided + to sidestep the rename-redirect ambiguity. +- **Evaluated:** 2026-08-15 by beads/deployer + +## Scope + +Same feature as be-uoat's round-2 gate: incremental auto-export via +`dolt_diff()`, patching only changed issue lines instead of rewriting the +whole JSONL file on every export. This round makes **zero** further changes +to feature/production logic beyond what be-uoat already gated — the +production files (`cmd/bd/export_auto.go`, `internal/storage/diff_store.go`, +`internal/storage/dolt/versioned.go`) are unchanged since round 2. Rounds 3 +and 4 are both test-infrastructure-only, addressing be-uoat's criterion-3 FAIL: + +- **Round 3** (`9c4f2e798` red, `f35368477` green): root-caused be-uoat's + flake to the pool's `defaultPoolReadTimeout = 10s` colliding with the + 5001-row bulk seed in `TestTryIncrementalExport_ThresholdExceededFallsBack` + under host contention — exactly as diagnosed in be-uoat's gate. Fixed by + adding `newTestStoreSharedBranchWithReadTimeout` / + `newTestStoreWithPrefixAndReadTimeout` (siblings of the existing + unparameterized helpers) that thread a caller-supplied + `dolt.Config.PoolReadTimeout` through both the shared-branch fast path and + the per-test-DB fallback. The flaky test now opts into + `bulkSeedPoolReadTimeout = 5 * time.Minute` + (`cmd/bd/export_auto_test.go:1425`) via + `setupIncrementalExportTestWithReadTimeout` — this is the exact + evidence-calibrated-long-timeout fix be-uoat's gate recommended, arrived at + independently. Existing callers keep the 10s default; only this one + bulk-seed path opts in. Round-3 review (be-8wnq) independently soaked the + fixed test 3/3 PASS (413.24s/257.68s/316.48s, package-scoped, + `BEADS_TEST_ENV_RUN_DOLT=1`, `-timeout 40m`) but REQUEST-CHANGES'd on a + separate, genuine defect: the new plumbing-proof test + (`TestNewTestStoreWithReadTimeout_AppliesConfiguredTimeout`) wrapped its + expected-to-fail case in `t.Run` and checked the bool return — but a failed + Go subtest unconditionally propagates to the parent/package result + regardless of that bool, so the test could never report an overall PASS + even when the timeout plumbing was working correctly. Deterministic, not a + flake. +- **Round 4** (`fc5caadcf`, refs be-8wnq): fixes the round-3 defect by adding + `tryNewTestStoreWithReadTimeout`, which returns the error directly instead + of calling `t.Fatal` inside a subtest; the test now asserts on that error + value with no subtest wrapper for the expected-to-fail branch. Round-4 + review (be-unlq) confirmed this live against a real Dolt container (PASS, + 0.09s, not a self-skip) plus a full default-mode `./cmd/bd/...` run (5/5 + packages, 0 FAIL, 246.3s). + +Diff scope, confirmed directly via `git diff --name-only origin/main...fc5caadcf` +(7 files, identical set to be-uoat's already-approved scope): + +- `cmd/bd/export_auto.go` — feature logic (unchanged since round 2) +- `internal/storage/diff_store.go` — feature logic (unchanged since round 2) +- `internal/storage/dolt/versioned.go` — feature logic (unchanged since round 2) +- `cmd/bd/export_auto_test.go` — feature tests (round 3+4: PoolReadTimeout plumbing) +- `cmd/bd/test_helpers_test.go` — shared test infra (round 3+4: PoolReadTimeout plumbing) +- `cmd/bd/test_dolt_server_cgo_test.go` — shared test infra (round 2, carried forward) +- `internal/testutil/testdoltserver.go` — shared test infra (round 2, carried forward) + +## Gate criteria + +| # | Criterion | Verdict | Evidence | +|---|-----------|---------|----------| +| 1 | Review PASS present | **PASS** | be-unlq records `verdict: pass` for round 4 on commit `fc5caadcf`, full style/security/spec write-up, zero findings. | +| 2 | Acceptance criteria met | **PASS** | be-hka's 6 ACs re-checked directly: branch off clean `origin/main` containing only the feature's net diff + strictly-required infra files (confirmed via `git diff --stat`); no unrelated file touched; `c0b942088` and all other shared-branch commits absent from the 5-commit ancestry (confirmed via `git log --oneline origin/main..fc5caadcf`); `go build`/`go vet` clean (independently re-run, see below); `export_auto_test.go` suite passes (independently re-run, see below); submitted via normal handoff each round (be-fgd → be-wu03 → be-8wnq → be-unlq chain, all verified claimable). | +| 3 | Tests pass | **PASS** | All 8 diff-owned tests green on independent re-run, including 3 fresh real-container runs of the previously-flaky test with zero failures. See "Tests run" below. | +| 3b | Policy/lint lane | **PASS** (documented exception) | `make ci-pr-policy` technically FAILs, root-caused and independently confirmed to a pre-existing gap unrelated to this diff. See below. | +| 4 | No unresolved HIGH findings | **PASS** | Zero HIGH or MEDIUM findings across all 4 review rounds. Round 3's REQUEST CHANGES was a deterministic test-logic defect (not a HIGH-severity finding), fixed in round 4. One informational, non-blocking, carried-forward item from round 1/2 (`isSafeCommitRef` lacks a dedicated direct unit test). | +| 5 | Clean working tree | **PASS** | `git status --short` on `fc5caadcf` shows nothing staged/unstaged; only the 3 pre-existing, unrelated untracked scratch files already present in this worktree (`release-gates/be-hi97-no-workspace-tests-gate.md`, `release-gates/be-uoat-dolt-diff-export-gate.md`, `scripts/rebase-resolve-lib.sh`). | +| 6 | Clean divergence from `origin/main` | **PASS** | `git merge-base --is-ancestor origin/main fc5caadcf` succeeds — `fc5caadcf` is a clean fast-forward-able descendant, 5 commits ahead, 0 behind. No rebase needed. | +| 7 | Single feature theme | **PASS** | Same 7-file scope as be-uoat's already-approved round-2 gate; all files serve the one incremental-export feature or the test infra its fix strictly requires. | + +### Ancestry-scope check: documented exception on one commit + +`assert_deploy_ancestry_scope origin/main fc5caadcf be-fgd be-uoat be-8wnq` +returned rc=21, flagging exactly one commit as citing none of the accepted +bead ids: + +``` +5e94e0cc5 perf(export): incremental auto-export via dolt_diff +``` + +The `.claude/**` denylist check (check 1) passed cleanly — no output, no +refusal. Only the stray-commit check (check 2) fired, and only on this one +commit. + +This is a known, structurally-expected false positive, not a be-27c-shaped +contamination, for a specific reason: `5e94e0cc5` **predates be-hka +entirely** — it is the original feature commit (authored 2026-04-19) that +already existed, unreviewed, on the long-lived shared branch +`gc-builder-e35c0415a93c` before be-27c's scope audit (2026-08-15) discovered +it and filed be-hka specifically to extract it onto a clean branch (see +be-hka's own description). No bead existed to cite at authorship time, and no +legitimate bead id substring-matches an empty reference — passing one to +force a match would be exactly the "blanket --force flag" behavior the +function's own docstring warns against, so none was passed. + +Independently verified this is benign rather than taking the extraction +narrative on faith: `git show --stat --format='' 5e94e0cc5` shows exactly 4 +files (`cmd/bd/export_auto.go`, `cmd/bd/export_auto_test.go`, +`internal/storage/diff_store.go`, `internal/storage/dolt/versioned.go`) — a +strict subset of the already-approved 7-file cumulative scope, zero overlap +with `.claude/**`, zero unrelated files. Proceeding past this flag with that +evidence recorded, per the deployer's judgment-call authority for exactly +this kind of mechanical-check edge case. + +## Tests run on release branch (independent re-verification) + +Static checks, independently re-run on `fc5caadcf` rather than trusted from +the reviewer's report: + +| Check | Result | +|---|---| +| `go build ./...` | clean, rc=0 | +| `go vet ./...` | clean, rc=0 | +| `gofmt -l` on the 7 diff files | clean, 0 files listed | + +Diff-owned tests, run with real podman/Dolt containers (`-tags=cgo`, +`DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock`, +`TESTCONTAINERS_RYUK_DISABLED=true`), matching the methodology established +across all prior rounds of this feature: + +| Test | Result | Duration | +|---|---|---| +| `TestOrderedIssueLines_PreservesInsertionOrderAndReplacesInPlace` | PASS | 0.00s | +| `TestLoadExistingIssueLines_ParsesIssuesSkipsMemories` | PASS | 0.00s | +| `TestLoadExistingIssueLines_MissingFileReturnsEmpty` | PASS | 0.00s | +| `TestChangedIssueIDs_DetectsUpsertsAndRemovals` | PASS | 0.53s | +| `TestTryIncrementalExport_PatchesChangedIssuesAndDropsRemoved` | PASS | 0.48s | +| `TestTryIncrementalExport_DropsIssueWhenFlippedToTemplate` | PASS | 0.34s | +| `TestTryIncrementalExport_FallsBackWhenFileMissing` | PASS | 0.33s | +| `TestNewTestStoreWithReadTimeout_AppliesConfiguredTimeout` (new, round 4) | PASS | 0.19s | +| `TestTryIncrementalExport_ThresholdExceededFallsBack` — run 1/3 | PASS | 43.95s | +| `TestTryIncrementalExport_ThresholdExceededFallsBack` — run 2/3 | PASS | 45.45s | +| `TestTryIncrementalExport_ThresholdExceededFallsBack` — run 3/3 | PASS | 43.86s | + +8/8 diff-owned tests green. `TestTryIncrementalExport_ThresholdExceededFallsBack` +— the exact test that FAILed 2/3 in be-uoat's round-2 gate under the 10s +default timeout — was independently re-run 3 times against a real container +specifically because of that history. All 3 passed, tight and consistent +(43.9–45.5s), nowhere near either the old 10s collision point or the new 5m +ceiling. No failure occurred, so `vmstat` sampling was not needed. Combined +with round-3 review's independent 3/3 (413.24s/257.68s/316.48s), that is 6/6 +real-container passes post-fix with zero failures — the +`bulkSeedPoolReadTimeout=5m` fix holds under repeated, independent +real-container verification. be-uoat's criterion-3 FAIL is resolved. + +### Policy/lint lane (criterion 3b) + +`make ci-pr-policy` → FAIL (rc=2), root cause: `.githooks/commit-msg` is +missing `BEGIN/END BEADS INTEGRATION` markers, tripping the version- +consistency sub-step. Independently verified, not taken on trust: + +``` +git diff --name-only origin/main...fc5caadcf -- .githooks/commit-msg # empty — untouched by this diff +git diff origin/main -- .githooks/commit-msg # empty — byte-identical to origin/main +grep -n "BEADS INTEGRATION" .githooks/commit-msg # no hits — gap is real +``` + +`.githooks/commit-msg` is not one of the 7 diff files and is byte-identical +to `origin/main`, so this is a pre-existing gap on baseline, not something +this diff introduces or could fix within its own scope. `bd search` for +"githooks commit-msg" and "ci-pr-policy" returned no existing tracking bead. +Same disposition class as the `cmd/bd` / `internal/remotecache` failures +triaged non-blocking in be-uoat's gate: zero file overlap with the diff, +independently root-caused rather than waved through. Filing a small P3 +tracking bead for this gap separately (not blocking this deploy). + +## Findings from reviews (no action required) + +Zero HIGH or MEDIUM findings across all 4 rounds. One informational, +non-blocking, carried-forward item from round 1/2 — `isSafeCommitRef` +(`internal/storage/dolt/versioned.go:244`) is correct but still lacks a +dedicated direct unit test. Round 3's REQUEST CHANGES (test-logic defect in +the new plumbing-proof test, not a HIGH finding) is resolved in round 4 — +see "Scope" above. + +## Verdict + +**PASS** — all 7 criteria pass (3b passes with a documented, independently- +verified pre-existing-and-unrelated exception; the ancestry-scope check's +single stray-commit flag is a documented, independently-verified false +positive specific to this extraction-workflow shape). Cutting isolated +deploy branch `deploy/be-y1jo-gate` from `fc5caadcf63371f6b992f32d23ef4d8c2a71e5ee`, +pushing to `headfork`, and opening a PR against `gastownhall/beads:main`. + +**gastownhall/beads merge-authority carve-out:** this is a contributor-only +repository (`origin` push disabled; upstream is fetch-only). Per deployer +protocol, the job ends at the open PR — no merge-request routed to +mayor/mpr, no deploy-clearance status posted. Merge belongs to upstream +maintainers.