beads/cmd/bd/import_shared.go
Julian Knutsen 1abafbe670
fix(import): write large imports in bounded transactions (#4729)
* fix(import): write large imports in bounded transactions

bd import wrote the whole batch in one transaction, holding the store's
write lock for the entire import: a 26k-issue import measured ~2 minutes
of full-store outage on SQLite, with every concurrent bd command failing
or stalling past busy_timeout. Empirical re-run on a 10k-issue import
with concurrent readers/writers: 10/226 ops failed and the longest
availability gap was 46.8s (the whole import) before this change; after
it, 234/234 ops succeed with a worst gap of 1.66s.

Imports larger than importChunkSize (250, measured: ~13% total-time cost
while keeping per-transaction lock holds ~1.4s, well under the 5s
busy_timeout) now commit in bounded transactions with a 150ms pause
between commits — SQLite busy-polling has no fairness queue, so without
the pause an import loop re-takes the lock before any waiter polls
(measured: unchanged 100% failure, zero acquisitions across 20 chunk
commits). Rows land in dependency order so readiness edges commit in the
same transaction as their row: bd ready mid-import never offers blocked
work. Cross-chunk non-readiness edges are wired by a ConflictSkip
dependency pass that never rewrites rows, so a concurrent update cannot
be clobbered or stale-drop the edges. A mid-import failure leaves a
durable committed prefix and re-running the same import converges.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: auto-regenerate CLI reference

Applied from the cli-docs-freshness-patch artifact of https://github.com/gastownhall/beads/actions/runs/29147804180
(generated with CI's canonical pinned build). See
scripts/check-cli-docs-drift.sh for how drift is attributed.

* fix(import): keep chunked-import readiness edges inline (cycles + waits-for)

Maintainer review fixups on top of the chunked `bd import` contribution,
keeping every readiness-affecting relationship inline with its row so a
chunked import can never expose a blocked bead as ready.

waits-for across chunks (attempt-2 blocker): a waiter's is_blocked state
keys on whether its spawner has an active parent-child child, not on the
spawner row itself, and the per-chunk is_blocked recompute only re-evaluates
the rows in its own transaction. So a waiter whose spawner's active child
landed in a later chunk computed is_blocked=0 against a still-childless
spawner and was never re-blocked, sitting ready for the rest of the import
and after it. Order every in-batch parent-child child of a waited spawner no
later than the waiter, so the child is committed when the waiter's chunk
recomputes. Regression: waiter, spawner, three fillers, active child at
chunk size 5 asserts the waiter is not ready after import.

tolerated cycles: the cycle fallback appended stalled rows in plain file
order, which could place a valid dependent of a readiness cycle in an
earlier chunk than the cycle member it blocks on and defer that live edge.
Break each Kahn stall by emitting a row that genuinely lies on a cycle, so
only the cycle's own unsatisfiable edges are ever deferred and every valid
acyclic readiness edge still rides inline. Soften the parity comments: for a
cycle of length >=3 the specific dropped edge can differ from the
single-transaction import.

Decompose the pacer, partition, chunk-write, dependency-pass, and ordering
graph into focused helpers with no behavior change, keeping every changed
function below the complexity gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: auto-regenerate CLI reference

Applied from the cli-docs-freshness-patch artifact of https://github.com/gastownhall/beads/actions/runs/29632882720
(generated with CI's canonical pinned build). See
scripts/check-cli-docs-drift.sh for how drift is attributed.

* test(import): port chunked-import real-store tests to embedded Dolt

cmd/bd/import_chunking_test.go imported internal/storage/sqlite, which is not
present on main, so `go test -c ./cmd/bd` failed to build (PR Risk: "Build
embedded cmd test binary") and blocked the entire cmd/bd test binary. The same
file also tripped the testing.Short boundaries policy check (PR policy) via
TestImportChunkedConcurrentAvailability.

Split the suite along the store it needs:

- import_chunking_test.go keeps the mock/pure tests (chunkRecordingStore,
  ordering, transaction-boundary counting) and drops the sqlite/filepath/sync
  imports so it always compiles.
- import_chunking_embedded_test.go (//go:build cgo, gated on
  BEADS_TEST_EMBEDDED_DOLT) ports the real-store tests to embeddeddolt.Open --
  the in-process store the repo actually ships -- following the existing
  cmd/bd/*_embedded_test.go convention. Every chunking invariant (durable
  committed prefix after a crash, no ready-window at any freeze point, waits-for
  gating, concurrent-update edge survival, stale-rejection) passes unchanged
  against embedded Dolt.

Drop TestImportChunkedConcurrentAvailability: it probed SQLite two-connection
file-lock fairness, which does not exist on main (embedded Dolt serializes
writes in-process and refcount-aliases handles per data dir), so it cannot be
ported faithfully. The inter-chunk pause it validated end-to-end stays covered
by TestImportChunkedPausesBetweenChunkTransactions. Removing it also clears the
sole testing.Short() the policy check rejected.

* fix(import): qualify chunk-ordering readiness comments; add mixed regular/wisp chunked test

Maintainer review fixups (attempt-1, non-gating findings):

- Narrow the chunked-import ordering comments. They said "every valid acyclic
  readiness edge rides inline", but force-emitting a cycle member to break a
  readiness stall can emit it ahead of an acyclic blocker it owns, deferring
  that valid edge. Qualify the guarantee: it holds for non-cycle rows; only a
  force-emitted cycle member may defer a readiness edge, which is the already-
  tolerated corrupt/legacy-cycle window. No behavior change.

- Add an embedded-Dolt chunked import mixing regular issues and wisps across a
  chunk boundary. The up-front FilterCreateIssuesMixedBucketDependencies pass is
  the only one that can see a cross-bucket edge whose endpoints land in
  different chunks; the prior chunked tests seeded only regular issues. The new
  test asserts a regular->wisp edge straddling a boundary is skip-reported while
  both rows import, and a same-bucket wisp->wisp edge straddling the same
  boundary survives into wisp_dependencies.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Test User <test@test.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-19 04:48:17 -07:00

954 lines
37 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

package main
import (
"bufio"
"context"
"encoding/json"
"fmt"
"io"
"os"
"strings"
"time"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/issueops"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/internal/utils"
)
// importChunkSize bounds the number of issues written per transaction during
// import. One giant transaction holds the store's write lock for the whole
// batch (a 26k-issue import measured ~2 minutes of full-store outage on
// SQLite); bounded chunks release the lock between commits so concurrent
// readers and writers interleave.
//
// 250 was picked by measurement (8k-issue import, SQLite backend): 250- and
// 500-issue chunks both cost ~13% total time over one big transaction, but
// 250 holds the write lock ~1.4s per transaction versus ~2.8s at 500 and ~6s
// at 1000 — and the SQLite connection's busy_timeout is 5s, so per-chunk lock
// holds must stay comfortably below that for concurrent bd operations to wait
// out an import instead of failing. A var, not a const, so tests can shrink it.
var importChunkSize = 250
// importInterChunkPause is slept between import transactions. Bounding the
// transactions is not enough on its own: SQLite busy-polling has no fairness
// queue, so a loop that re-issues BEGIN IMMEDIATE microseconds after each
// COMMIT re-takes the lock before any waiter's poll fires and starves every
// concurrent bd operation for the whole import — measured as an unchanged
// 100% failure pattern with zero lock acquisitions across 20 chunk commits.
// The pause is the acquisition window that lets waiters in. 150ms costs
// ~15s on a 26k-issue import (104 chunks) against the ~2-minute write time.
var importInterChunkPause = 150 * time.Millisecond
// importPause is the sleep seam for the inter-chunk pause, swappable in tests.
var importPause = time.Sleep
// importProgress is where chunked imports report per-chunk progress.
// Swappable in tests.
var importProgress io.Writer = os.Stderr
// ImportOptions configures import behavior.
type ImportOptions struct {
DryRun bool
SkipUpdate bool
Strict bool
RenameOnImport bool
ClearDuplicateExternalRefs bool
OrphanHandling string
DeletionIDs []string
SkipPrefixValidation bool
ProtectLocalExportIDs map[string]time.Time
// ConflictSkip makes the import insert-if-new instead of UPSERT: an
// issue whose ID already exists is left untouched. Set only by the
// auto-import upgrade-recovery fallback (GH#3955); explicit `bd import`
// leaves this false and keeps UPSERT semantics.
ConflictSkip bool
// AllowStale imports rows even when their updated_at is older than the
// local issue's, overwriting newer local state. Required for the
// restore-an-older-snapshot recovery workflow, which the default stale
// guard otherwise silently no-ops per row (bd-6dnrw.9). Only settable
// via explicit `bd import --allow-stale`; auto-import paths never set it.
AllowStale bool
}
// ImportResult describes what an import operation did.
type ImportResult struct {
Created int
Updated int
Unchanged int
Skipped int
Deleted int
Collisions int
IDMapping map[string]string
CollisionIDs []string
PrefixMismatch bool
ExpectedPrefix string
MismatchPrefixes map[string]int
ImportedIDs []string
StaleSkippedIDs []string
SkippedDependencies []string
// UpdatedIssues lists existing local issues whose row the import
// rewrote (incoming strictly newer, content differs), with a
// field-level summary, so reverts of local state are visible instead
// of silent (bd-hj85c).
UpdatedIssues []ImportChange
// TieKeptLocalIDs lists incoming rows whose updated_at equals the
// local issue's but whose content differs. The upsert keeps the local
// row for these (second-granularity timestamp ties, bd-hj85c); their
// aux data still merges.
TieKeptLocalIDs []string
}
// ImportChange describes how an import row modified an existing local issue.
type ImportChange struct {
ID string `json:"id"`
Changes string `json:"changes,omitempty"`
}
// importIssuesCore imports issues into the Dolt store.
// This is a bridge function that delegates to the Dolt store's batch creation.
func importIssuesCore(ctx context.Context, _ string, store storage.DoltStorage, issues []*types.Issue, opts ImportOptions) (*ImportResult, error) {
if opts.DryRun || len(issues) == 0 {
return &ImportResult{Skipped: len(issues)}, nil
}
// The stale guard has two halves (bd-pkim8). This pre-filter reports the
// rows that are already known stale (StaleSkippedIDs) and keeps their
// labels/comments/dependencies out of the batch entirely. It is a separate
// read, though, so a local update that commits between it and the batch
// write would slip through — RejectStaleUpserts below closes that race by
// re-checking updated_at inside the upsert itself.
var staleSkippedIDs []string
var changePlan importChangePlan
if !opts.AllowStale {
filtered, skipped, plan, err := filterStaleImportIssues(ctx, store, issues)
if err != nil {
return nil, err
}
issues = filtered
staleSkippedIDs = skipped
changePlan = plan
if len(issues) == 0 {
return &ImportResult{Skipped: len(staleSkippedIDs), StaleSkippedIDs: staleSkippedIDs}, nil
}
}
var skippedDependencies []string
skippedDependencySet := make(map[string]struct{})
// In-txn half of the stale guard: rows the conditional upsert rejected
// (local update committed between the pre-filter read and the batch
// write). The transaction may retry, so dedup by ID.
staleRejectedSet := make(map[string]struct{})
actor := getActorWithGit()
batchOpts := storage.BatchCreateOptions{
OrphanHandling: storage.OrphanAllow,
SkipPrefixValidation: opts.SkipPrefixValidation,
ConflictSkip: opts.ConflictSkip,
RejectStaleUpserts: !opts.AllowStale,
SkipDependencyValidationErrors: true,
OnSkippedDependency: func(issueID, dependsOnID, reason string) {
skipped := fmt.Sprintf("%s -> %s: %s", issueID, dependsOnID, reason)
if _, ok := skippedDependencySet[skipped]; ok {
return
}
skippedDependencySet[skipped] = struct{}{}
skippedDependencies = append(skippedDependencies, skipped)
},
OnStaleRejected: func(issueID string) {
staleRejectedSet[issueID] = struct{}{}
},
}
var err error
if len(issues) <= importChunkSize {
// Small import: one transaction, dependencies inline — exactly the
// pre-chunking behavior.
err = store.CreateIssuesWithFullOptions(ctx, issues, actor, batchOpts)
} else {
err = importIssuesChunked(ctx, store, issues, actor, batchOpts)
}
if err != nil {
return nil, err
}
importedIDs := make([]string, 0, len(issues))
for _, issue := range issues {
if _, rejected := staleRejectedSet[issue.ID]; rejected {
staleSkippedIDs = append(staleSkippedIDs, issue.ID)
continue
}
importedIDs = append(importedIDs, issue.ID)
}
// Drop planned updates the in-txn guard rejected (a local update raced
// in between the pre-filter read and the batch write).
updatedIssues := make([]ImportChange, 0, len(changePlan.Updates))
updatedCount := 0
for _, change := range changePlan.Updates {
if _, rejected := staleRejectedSet[change.ID]; rejected {
continue
}
updatedIssues = append(updatedIssues, change)
updatedCount++
}
return &ImportResult{
Created: len(importedIDs),
Updated: updatedCount,
Skipped: len(staleSkippedIDs),
ImportedIDs: importedIDs,
StaleSkippedIDs: staleSkippedIDs,
SkippedDependencies: skippedDependencies,
UpdatedIssues: updatedIssues,
TieKeptLocalIDs: changePlan.TieKeptLocal,
}, nil
}
// importIssuesChunked writes a large import in bounded transactions of
// importChunkSize rows instead of one batch-wide transaction, sleeping
// importInterChunkPause between commits. One giant transaction holds the
// store's write lock for the whole import, taking every concurrent bd
// operation down with it; bounded chunks cap the per-transaction lock hold,
// and the pause between commits is the fairness window that actually lets
// waiters acquire (see importInterChunkPause).
//
// Rows are written in dependency order (orderImportIssuesForChunking): every
// readiness-affecting edge owned by a non-cycle row points at a row in the same
// or an earlier chunk — Kahn's order for the acyclic majority, plus a
// cycle-breaking fallback that still emits each cycle member before the rows it
// blocks — so that edge rides inline with its row and both commit in one
// transaction. Only a force-emitted cycle member can carry a readiness edge into
// the deferred pass, and that is the already-tolerated corrupt/legacy-cycle
// window. A concurrent reader therefore never observes a non-cycle imported bead
// without the blocking edges its import file declares —
// `bd ready` mid-import cannot offer blocked work for dispatch, and a crash
// mid-import cannot freeze a bead in a spuriously-ready state.
//
// Only edges that cannot be satisfied when their row commits are deferred to
// a final dependency pass: edges into an intra-batch dependency cycle
// (invalid for blocking types; still broken and skip-reported at wire time,
// though for a cycle of length >=3 the specific edge dropped — and thus which
// member is left spuriously ready — can differ from the single-transaction
// import, which checks the whole cycle in file order) and non-readiness edges
// (related, discovered-from) that point at a later chunk. The dependency pass submits
// row copies stripped to those deferred edges with ConflictSkip set, so an
// existing row is never rewritten: a concurrent update landing between a
// row's chunk and the dependency pass cannot stale-reject the pass and drop
// the edges (the resubmit-the-full-row alternative did exactly that,
// silently and unrecoverably). Rows whose phase-1 write was itself
// stale-rejected are excluded from the pass: a stale snapshot keeps its
// labels, comments, AND dependencies out (bd-578h9.8).
//
// Every per-row application is an idempotent upsert (conditional-update row
// write, INSERT IGNORE labels, existence-checked comments, deterministic-id
// dependency edges, created-events only for genuinely new rows), so a failure
// mid-import leaves a committed, durable prefix and re-running the same import
// converges on the full set — subject to the import's standing stale policy:
// a row a rival updated since its chunk committed is locally newer on the
// re-run, so the pre-filter stale-skips that snapshot wholesale (row and any
// still-unwired deferred edges), reported in StaleSkippedIDs. That is the
// same local-wins outcome the single-transaction import gave a rival update
// racing a crashed import, and it can only affect deferred edges (non-readiness
// edges, plus the readiness edges of force-emitted cycle members); every
// non-cycle row's readiness edges commit with their row.
func importIssuesChunked(ctx context.Context, store storage.DoltStorage, issues []*types.Issue, actor string, opts storage.BatchCreateOptions) error {
// Apply the cross-bucket dependency policy over the full set up front:
// the engine's per-batch filter only sees one chunk, so it could no
// longer detect an edge whose endpoints land in different chunks.
issues, err := issueops.FilterCreateIssuesMixedBucketDependencies(issues, opts)
if err != nil {
return err
}
ordered := orderImportIssuesForChunking(issues)
// Partitioning narrows each issue's dependency slice to its inline subset in
// place; always restore the full slices so the caller's issues stay intact
// for retries and reporting.
fullDeps := make([][]*types.Dependency, len(ordered))
for i, issue := range ordered {
fullDeps[i] = issue.Dependencies
}
defer func() {
for i, issue := range ordered {
issue.Dependencies = fullDeps[i]
}
}()
deferred := partitionChunkedImportDeps(ordered)
// Record phase-1 stale rejections locally (as well as forwarding them to
// the caller): the dependency pass must skip those rows.
phase1Stale := make(map[string]struct{})
rowOpts := opts
rowOpts.OnStaleRejected = func(issueID string) {
phase1Stale[issueID] = struct{}{}
if opts.OnStaleRejected != nil {
opts.OnStaleRejected(issueID)
}
}
pacer := &importChunkPacer{}
if err := writeImportRowChunks(ctx, store, ordered, actor, rowOpts, pacer); err != nil {
return err
}
return wireDeferredImportDeps(ctx, store, deferred, phase1Stale, len(ordered), actor, opts, pacer)
}
// importChunkPacer sleeps importInterChunkPause between import transactions. One
// shared pacer spans phase 1 and the dependency pass, so the fairness pause is
// issued before every transaction except the very first: SQLite busy-polling has
// no fairness queue, and a back-to-back BEGIN IMMEDIATE re-takes the lock before
// a waiter's poll fires (see importInterChunkPause).
type importChunkPacer struct{ transactions int }
func (p *importChunkPacer) beforeTx() {
if p.transactions > 0 {
importPause(importInterChunkPause)
}
p.transactions++
}
// deferredImportEdges carries the dependencies of one row that could not be
// wired inline with its phase-1 write and must be applied in the dependency
// pass.
type deferredImportEdges struct {
issue *types.Issue
deps []*types.Dependency
}
// partitionChunkedImportDeps splits each ordered row's dependencies into edges
// wired inline with the row (target outside the batch, or first written in the
// same or an earlier chunk) and edges deferred to the dependency pass. It
// narrows each issue's dependency slice to the inline subset in place and
// returns the deferred edges. orderImportIssuesForChunking guarantees every
// readiness edge owned by a non-cycle row points at the same or an earlier
// chunk, so only non-readiness edges (related, discovered-from) into a later
// chunk and the readiness edges of force-emitted cycle members (their own cyclic
// edges, plus any acyclic blocker they were emitted ahead of) are ever deferred
// here.
func partitionChunkedImportDeps(ordered []*types.Issue) []deferredImportEdges {
firstChunkOf := make(map[string]int, len(ordered))
for pos, issue := range ordered {
if issue.ID == "" {
continue // ID assigned by the engine at insert; nothing can reference it yet
}
if _, ok := firstChunkOf[issue.ID]; !ok {
firstChunkOf[issue.ID] = pos / importChunkSize
}
}
var deferred []deferredImportEdges
for pos, issue := range ordered {
if len(issue.Dependencies) == 0 {
continue
}
inline, later := splitDepsByChunk(issue.Dependencies, pos/importChunkSize, firstChunkOf)
if len(later) == 0 {
continue
}
issue.Dependencies = inline
deferred = append(deferred, deferredImportEdges{issue: issue, deps: later})
}
return deferred
}
// splitDepsByChunk partitions one row's dependencies (the row lands in rowChunk)
// into edges that can be wired inline and edges whose target is first written in
// a later chunk and so must be deferred.
func splitDepsByChunk(deps []*types.Dependency, rowChunk int, firstChunkOf map[string]int) (inline, later []*types.Dependency) {
for _, dep := range deps {
if targetChunk, inBatch := firstChunkOf[dep.DependsOnID]; inBatch && targetChunk > rowChunk {
later = append(later, dep)
continue
}
inline = append(inline, dep)
}
return inline, later
}
// writeImportRowChunks writes the ordered rows (dependencies already narrowed to
// their inline subset) in bounded transactions, pausing between commits.
func writeImportRowChunks(ctx context.Context, store storage.DoltStorage, ordered []*types.Issue, actor string, rowOpts storage.BatchCreateOptions, pacer *importChunkPacer) error {
total := len(ordered)
chunks := (total + importChunkSize - 1) / importChunkSize
for start, chunk := 0, 1; start < total; start, chunk = start+importChunkSize, chunk+1 {
end := min(start+importChunkSize, total)
pacer.beforeTx()
if err := store.CreateIssuesWithFullOptions(ctx, ordered[start:end], actor, rowOpts); err != nil {
return fmt.Errorf("import chunk %d/%d failed, %d issues already committed (committed rows are durable; re-run the import to resume — it converges): %w", chunk, chunks, start, err)
}
fmt.Fprintf(importProgress, "bd import: %d/%d issues committed\n", end, total)
}
return nil
}
// wireDeferredImportDeps applies the deferred edges once every target row exists,
// without rewriting the rows themselves. rowTotal is the count of phase-1 rows
// already committed, used only for the resume message.
func wireDeferredImportDeps(ctx context.Context, store storage.DoltStorage, deferred []deferredImportEdges, phase1Stale map[string]struct{}, rowTotal int, actor string, opts storage.BatchCreateOptions, pacer *importChunkPacer) error {
depRows := make([]*types.Issue, 0, len(deferred))
for _, d := range deferred {
if _, stale := phase1Stale[d.issue.ID]; stale {
continue // stale snapshot: its deps stay out too (bd-578h9.8)
}
cp := *d.issue
cp.Dependencies = d.deps
// The row landed in phase 1 and its labels/comments merged there;
// this pass carries edges only.
cp.Labels = nil
cp.Comments = nil
depRows = append(depRows, &cp)
}
if len(depRows) == 0 {
return nil
}
depOpts := opts
// Never rewrite an existing row here: the import's row write already
// happened in phase 1, and a concurrent update since then must win. With
// ConflictSkip the engine leaves the stored row untouched and still wires
// the batch's dependencies.
depOpts.ConflictSkip = true
// No row write can be stale-rejected under ConflictSkip; leaving the
// callback unset keeps a phase-2 signal from ever misreporting a row
// whose phase-1 write committed.
depOpts.OnStaleRejected = nil
depTotal := len(depRows)
depChunks := (depTotal + importChunkSize - 1) / importChunkSize
for start, chunk := 0, 1; start < depTotal; start, chunk = start+importChunkSize, chunk+1 {
end := min(start+importChunkSize, depTotal)
pacer.beforeTx()
if err := store.CreateIssuesWithFullOptions(ctx, depRows[start:end], actor, depOpts); err != nil {
return fmt.Errorf("import dependency pass chunk %d/%d failed (all %d issue rows are committed; re-run the import to resume — it converges): %w", chunk, depChunks, rowTotal, err)
}
fmt.Fprintf(importProgress, "bd import: deferred dependencies wired for %d/%d issues\n", end, depTotal)
}
return nil
}
// orderImportIssuesForChunking returns the issues reordered so that every valid
// readiness-affecting edge (blocks, parent-child, conditional-blocks, waits-for;
// the types GetReadyWork consults) points at a row in the same or an earlier
// chunk, which lets the import wire that edge in the same transaction as the
// row. It runs Kahn's algorithm over the intra-batch readiness edges, seeded in
// file order so unconstrained rows keep their relative order; duplicate IDs are
// chained in file order to preserve last-row-wins upsert semantics.
//
// A readiness cycle (invalid for blocking types; only ever present in the
// corrupted or legacy JSONL the import tolerates) cannot be fully ordered.
// Appending the stalled rows in plain file order — as an earlier version did —
// can place a valid dependent of a cycle before the cycle member it blocks on,
// deferring that live readiness edge and briefly exposing the dependent as ready
// without its blocker. Instead each stall is broken by emitting a row that
// genuinely lies on a cycle, so every readiness edge owned by a non-cycle row
// still rides inline; only a force-emitted cycle member may defer a readiness
// edge — its own unsatisfiable cycle edge, or a valid acyclic edge whose blocker
// it was emitted ahead of — and the engine skip-reports the unsatisfiable ones,
// still breaking the cycle. For a cycle of length
// >=3 the edge left to defer (and hence which member is left spuriously ready)
// can differ from the one the single-transaction import drops, which checks all
// cycle edges in file order against the already-persisted set; both break it.
func orderImportIssuesForChunking(issues []*types.Issue) []*types.Issue {
if len(issues) < 2 {
return issues
}
return buildImportOrderGraph(issues).topologicalFileOrder(issues)
}
// importOrderGraph is the intra-batch readiness-edge graph over import rows,
// indexed by position in the input slice. An edge target->dependent means the
// target (a readiness blocker, or an earlier duplicate of the same ID) must be
// emitted no later than the dependent.
type importOrderGraph struct {
dependents [][]int // dependents[t]: rows released when row t is emitted
blockers [][]int // blockers[u]: rows that must be emitted before row u
indegree []int
}
func buildImportOrderGraph(issues []*types.Issue) importOrderGraph {
n := len(issues)
indicesByID := indexImportIssuesByID(issues)
// A waits-for waiter's is_blocked state is gated on whether its spawner has
// an active parent-child child, not on the spawner row itself, so those
// child rows are readiness inputs to the waiter (see addRowReadinessEdges).
// Index them by spawner up front so the edge pass can order them.
childrenBySpawnerID := parentChildChildrenBySpawner(issues)
g := importOrderGraph{
dependents: make([][]int, n),
blockers: make([][]int, n),
indegree: make([]int, n),
}
for i, issue := range issues {
g.addRowReadinessEdges(i, issue, indicesByID, childrenBySpawnerID)
}
// Chain duplicate IDs in file order so the last row wins the upsert.
for _, indices := range indicesByID {
for k := 1; k < len(indices); k++ {
g.addEdge(indices[k-1], indices[k])
}
}
return g
}
// indexImportIssuesByID maps each import row's ID to its positions in the input
// slice. ID-less rows are skipped: their ID is assigned by the engine at insert,
// so nothing in the batch can reference them yet.
func indexImportIssuesByID(issues []*types.Issue) map[string][]int {
indicesByID := make(map[string][]int, len(issues))
for i, issue := range issues {
if issue.ID == "" {
continue
}
indicesByID[issue.ID] = append(indicesByID[issue.ID], i)
}
return indicesByID
}
// parentChildChildrenBySpawner maps each spawner ID to the positions of the
// in-batch rows that declare it as their parent-child parent. A parent-child
// dependency row is stored on the child (issue_id = child, depends_on = parent),
// so the child is the row carrying the edge.
func parentChildChildrenBySpawner(issues []*types.Issue) map[string][]int {
children := make(map[string][]int)
for i, issue := range issues {
for _, dep := range issue.Dependencies {
if dep != nil && dep.Type == types.DepParentChild && dep.DependsOnID != "" {
children[dep.DependsOnID] = append(children[dep.DependsOnID], i)
}
}
}
return children
}
// addEdge records that row target must be emitted no later than row dependent.
func (g importOrderGraph) addEdge(target, dependent int) {
g.dependents[target] = append(g.dependents[target], dependent)
g.blockers[dependent] = append(g.blockers[dependent], target)
g.indegree[dependent]++
}
// addEdgesBefore orders every target row no later than dependent, skipping any
// self-reference.
func (g importOrderGraph) addEdgesBefore(dependent int, targets []int) {
for _, target := range targets {
if target != dependent {
g.addEdge(target, dependent)
}
}
}
// addRowReadinessEdges records the intra-batch ordering edges that row i imposes:
// every readiness blocker it names must be emitted no later than i, and for a
// waits-for edge the spawner's in-batch parent-child children must be too.
//
// The waiter's is_blocked state keys on whether its spawner has an active child,
// not on the spawner row itself, and the per-chunk is_blocked recompute only
// re-evaluates the rows in its own transaction. So a waiter whose spawner's child
// lands in a later chunk would compute is_blocked=0 against a still-childless
// spawner and never be re-blocked — spuriously ready for the rest of the import
// and after it. Ordering the spawner inline is not enough; the child must be
// committed by the time the waiter's chunk recomputes.
func (g importOrderGraph) addRowReadinessEdges(i int, issue *types.Issue, indicesByID, childrenBySpawnerID map[string][]int) {
for _, dep := range issue.Dependencies {
if dep == nil || dep.DependsOnID == "" || !dep.Type.AffectsReadyWork() {
continue
}
g.addEdgesBefore(i, indicesByID[dep.DependsOnID])
if dep.Type == types.DepWaitsFor {
g.addEdgesBefore(i, childrenBySpawnerID[dep.DependsOnID])
}
}
}
// topologicalFileOrder emits rows in dependency order (Kahn's, file-order
// seeded), breaking each stall by force-emitting a row that lies on a readiness
// cycle, so every non-cycle row's readiness edges ride inline; only a
// force-emitted cycle member can leave a readiness edge (its unsatisfiable cycle
// edge, or a valid acyclic edge it was emitted ahead of) for the deferred
// dependency pass.
func (g importOrderGraph) topologicalFileOrder(issues []*types.Issue) []*types.Issue {
n := len(issues)
indegree := append([]int(nil), g.indegree...)
emitted := make([]bool, n)
queue := make([]int, 0, n)
for i := range n {
if indegree[i] == 0 {
queue = append(queue, i)
}
}
ordered := make([]*types.Issue, 0, n)
for len(ordered) < n {
if len(queue) == 0 {
queue = append(queue, g.nextCycleParticipant(emitted))
}
i := queue[0]
queue = queue[1:]
if emitted[i] {
continue
}
emitted[i] = true
ordered = append(ordered, issues[i])
queue = g.release(i, indegree, emitted, queue)
}
return ordered
}
// release decrements the indegree of every row that row i blocks and enqueues
// any that becomes free and has not already been emitted.
func (g importOrderGraph) release(i int, indegree []int, emitted []bool, queue []int) []int {
for _, j := range g.dependents[i] {
indegree[j]--
if indegree[j] == 0 && !emitted[j] {
queue = append(queue, j)
}
}
return queue
}
// nextCycleParticipant returns an un-emitted row that lies on a readiness cycle,
// used to break a Kahn stall.
func (g importOrderGraph) nextCycleParticipant(emitted []bool) int {
start := 0
for start < len(emitted) && emitted[start] {
start++
}
return g.cycleParticipant(start, emitted)
}
// cycleParticipant walks blocker edges from an un-emitted row until it revisits
// a row, which therefore lies on a cycle. At a stall every un-emitted row still
// has an un-emitted blocker, so the walk always closes a cycle.
func (g importOrderGraph) cycleParticipant(start int, emitted []bool) int {
seen := make(map[int]bool)
v := start
for !seen[v] {
seen[v] = true
next := -1
for _, b := range g.blockers[v] {
if !emitted[b] {
next = b
break
}
}
if next == -1 {
return v // defensive: no un-emitted blocker (unexpected at a stall)
}
v = next
}
return v
}
// importChangePlan reports how the import batch relates to existing local
// issues, so the import can surface what it changed instead of doing it
// silently (bd-hj85c).
type importChangePlan struct {
// Updates lists existing issues the batch will rewrite: incoming row
// strictly newer and row content differs.
Updates []ImportChange
// TieKeptLocal lists incoming rows with the same updated_at as the
// local issue but different row content. The stale-guarded upsert keeps
// every stored column for these (second-granularity timestamp tie),
// while their aux data still merges.
TieKeptLocal []string
}
func filterStaleImportIssues(ctx context.Context, store storage.DoltStorage, issues []*types.Issue) ([]*types.Issue, []string, importChangePlan, error) {
var plan importChangePlan
ids := make([]string, 0, len(issues))
seen := make(map[string]struct{}, len(issues))
for _, issue := range issues {
if issue == nil || issue.ID == "" {
continue
}
if _, ok := seen[issue.ID]; ok {
continue
}
seen[issue.ID] = struct{}{}
ids = append(ids, issue.ID)
}
if len(ids) == 0 {
return issues, nil, plan, nil
}
localIssues, err := store.GetIssuesByIDs(ctx, ids)
if err != nil {
return nil, nil, plan, fmt.Errorf("check existing issues before import: %w", err)
}
localByID := make(map[string]*types.Issue, len(localIssues))
for _, issue := range localIssues {
if issue != nil && issue.ID != "" && !issue.UpdatedAt.IsZero() {
localByID[issue.ID] = issue
}
}
if len(localByID) == 0 {
return issues, nil, plan, nil
}
filtered := make([]*types.Issue, 0, len(issues))
skippedIDs := make([]string, 0)
for _, issue := range issues {
if issue == nil || issue.ID == "" || issue.UpdatedAt.IsZero() {
filtered = append(filtered, issue)
continue
}
local, ok := localByID[issue.ID]
if !ok {
filtered = append(filtered, issue)
continue
}
// Compare at second granularity: updated_at is DATETIME(0) in the
// store, so a sub-second component on the JSONL side must not turn
// a tie into a spurious "newer" classification.
incomingAt := issue.UpdatedAt.UTC().Truncate(time.Second)
localAt := local.UpdatedAt.UTC().Truncate(time.Second)
if incomingAt.Before(localAt) {
skippedIDs = append(skippedIDs, issue.ID)
continue
}
if summary := importRowChangeSummary(local, issue); summary != "" {
if incomingAt.Equal(localAt) {
plan.TieKeptLocal = append(plan.TieKeptLocal, issue.ID)
} else {
plan.Updates = append(plan.Updates, ImportChange{ID: issue.ID, Changes: summary})
}
}
filtered = append(filtered, issue)
}
return filtered, skippedIDs, plan, nil
}
// importRowChangeSummary summarizes the differences between the local issue
// row and the incoming import row, restricted to the columns the import
// upsert rewrites. Returns "" when none of those fields differ. Status,
// priority, and type transitions show old → new; long-form fields are listed
// by name only.
func importRowChangeSummary(local, incoming *types.Issue) string {
var parts []string
if local.Status != incoming.Status {
parts = append(parts, fmt.Sprintf("status %s → %s", local.Status, incoming.Status))
}
if local.Priority != incoming.Priority {
parts = append(parts, fmt.Sprintf("priority %d → %d", local.Priority, incoming.Priority))
}
if local.IssueType != incoming.IssueType {
parts = append(parts, fmt.Sprintf("type %s → %s", local.IssueType, incoming.IssueType))
}
if local.Assignee != incoming.Assignee {
parts = append(parts, "assignee")
}
if local.Title != incoming.Title {
parts = append(parts, "title")
}
if local.Description != incoming.Description {
parts = append(parts, "description")
}
if local.Design != incoming.Design {
parts = append(parts, "design")
}
if local.AcceptanceCriteria != incoming.AcceptanceCriteria {
parts = append(parts, "acceptance_criteria")
}
if local.Notes != incoming.Notes {
if incoming.Notes == "" {
parts = append(parts, "notes cleared")
} else {
parts = append(parts, "notes")
}
}
if local.CloseReason != incoming.CloseReason {
parts = append(parts, "close_reason")
}
if !stringPtrEqual(local.ExternalRef, incoming.ExternalRef) {
parts = append(parts, "external_ref")
}
if !intPtrEqual(local.EstimatedMinutes, incoming.EstimatedMinutes) {
parts = append(parts, "estimate")
}
if string(local.Metadata) != string(incoming.Metadata) {
parts = append(parts, "metadata")
}
return strings.Join(parts, ", ")
}
func stringPtrEqual(a, b *string) bool {
if a == nil || b == nil {
return a == nil && b == nil
}
return *a == *b
}
func intPtrEqual(a, b *int) bool {
if a == nil || b == nil {
return a == nil && b == nil
}
return *a == *b
}
// importLocalResult holds counts from a local JSONL import.
type importLocalResult struct {
Issues int
Memories int
}
// memoryRecord represents a memory entry in the JSONL export.
type memoryRecord struct {
Type string `json:"_type"`
Key string `json:"key"`
Value string `json:"value"`
}
// importFromLocalJSONL imports issues (and memories) from a local JSONL file on disk
// into the Dolt store. Returns the number of issues imported and any error.
// This is a convenience wrapper around importFromLocalJSONLFull.
func importFromLocalJSONL(ctx context.Context, store storage.DoltStorage, localPath string) (int, error) {
result, err := importFromLocalJSONLFull(ctx, store, localPath)
if err != nil {
return 0, err
}
return result.Issues, nil
}
// parseJSONLFile reads a JSONL file and returns parsed issues and config
// entries (memories). Pure function — no store I/O.
func parseJSONLFile(path string) ([]*types.Issue, map[string]string, error) {
//nolint:gosec // G304: path from user-provided CLI argument
data, err := os.ReadFile(path)
if err != nil {
return nil, nil, fmt.Errorf("failed to read JSONL file %s: %w", path, err)
}
scanner := bufio.NewScanner(strings.NewReader(string(data)))
// Allow up to 64MB per line for large descriptions
scanner.Buffer(make([]byte, 0, 1024*1024), 64*1024*1024)
var issues []*types.Issue
configEntries := make(map[string]string)
for scanner.Scan() {
line := scanner.Text()
if line == "" {
continue
}
// Peek at the record to check for _type field
var peek map[string]json.RawMessage
if err := json.Unmarshal([]byte(line), &peek); err != nil {
return nil, nil, fmt.Errorf("failed to parse JSONL line: %w", err)
}
// Skip the optional beads-jsonl metadata/header record.
// Canonical exports produced by the stable-ordering /
// git-merge convention prepend a schema+provenance line, e.g.
// {"_schema":"beads-jsonl/1","_dolt_branch":"main",
// "_dolt_commit":"...","_sort":"stable-v1"}. It carries no
// _type and no issue fields; without this guard it falls
// through to the issue path, unmarshals into an empty Issue,
// and aborts the whole import with "validation failed for
// issue : title is required". Identified by the _schema
// sentinel, which real issue/memory records never carry.
if _, isHeader := peek["_schema"]; isHeader {
continue
}
// Check if this is a memory record
if rawType, ok := peek["_type"]; ok {
var typeStr string
if err := json.Unmarshal(rawType, &typeStr); err == nil && typeStr == "memory" {
var mem memoryRecord
if err := json.Unmarshal([]byte(line), &mem); err != nil {
return nil, nil, fmt.Errorf("failed to parse memory record: %w", err)
}
if mem.Key != "" && mem.Value != "" {
configEntries[kvPrefix+memoryPrefix+mem.Key] = mem.Value
}
continue
}
}
// Regular issue record
var issue types.Issue
if err := json.Unmarshal([]byte(line), &issue); err != nil {
return nil, nil, fmt.Errorf("failed to parse issue from JSONL: %w", err)
}
// Skip tombstone entries: these are deleted issues exported by older
// versions (pre-v0.50) with status "tombstone" and deleted_at set.
// They are not valid for re-import since "tombstone" is not a real status.
if issue.Status == "tombstone" {
continue
}
// v0.35–v0.37 exported "wisp" (bool), renamed to "ephemeral" in v0.38+.
// map old field name so the flag is preserved on import.
if _, hasWisp := peek["wisp"]; hasWisp && !issue.Ephemeral {
var wisp bool
if err := json.Unmarshal(peek["wisp"], &wisp); err == nil && wisp {
issue.Ephemeral = true
}
}
issue.SetDefaults()
issues = append(issues, &issue)
}
if err := scanner.Err(); err != nil {
return nil, nil, fmt.Errorf("failed to scan JSONL: %w", err)
}
return issues, configEntries, nil
}
// importFromLocalJSONLFull imports issues and memories from a local JSONL file
// using UPSERT semantics (an existing issue row is overwritten). Used by the
// explicit recovery paths: `bd bootstrap` and `bd init --from-jsonl`.
func importFromLocalJSONLFull(ctx context.Context, store storage.DoltStorage, localPath string) (*importLocalResult, error) {
return importFromLocalJSONLWithOpts(ctx, store, localPath, false)
}
// importFromLocalJSONLConflictSkip is the auto-import upgrade-recovery
// fallback (GH#3955; the fallbackImporter seam in auto_import_upgrade.go).
// It is identical to importFromLocalJSONLFull except that an issue whose ID
// already exists is left untouched instead of being overwritten, so a
// regressed emptiness guard can never clobber live rows — worst case is a
// no-op.
func importFromLocalJSONLConflictSkip(ctx context.Context, store storage.DoltStorage, localPath string) (*importLocalResult, error) {
return importFromLocalJSONLWithOpts(ctx, store, localPath, true)
}
// importFromLocalJSONLWithOpts is the shared implementation. It detects
// memory records (lines with "_type":"memory") and imports them via
// SetConfig, while routing regular issue records through the normal path.
// conflictSkip selects insert-if-new (true) vs UPSERT (false) for issue rows.
func importFromLocalJSONLWithOpts(ctx context.Context, store storage.DoltStorage, localPath string, conflictSkip bool) (*importLocalResult, error) {
issues, configEntries, err := parseJSONLFile(localPath)
if err != nil {
return nil, err
}
result := &importLocalResult{}
// Import memories
for key, value := range configEntries {
if err := store.SetConfig(ctx, key, value); err != nil {
return nil, fmt.Errorf("failed to import config %q: %w", key, err)
}
result.Memories++
}
// Import issues
if len(issues) > 0 {
// Auto-detect prefix from first issue if not already configured
configuredPrefix, err := store.GetConfig(ctx, "issue_prefix")
if err == nil && strings.TrimSpace(configuredPrefix) == "" {
firstPrefix := utils.ExtractIssuePrefix(issues[0].ID)
if firstPrefix != "" {
if err := store.SetConfig(ctx, "issue_prefix", firstPrefix); err != nil {
return nil, fmt.Errorf("failed to set issue_prefix from imported issues: %w", err)
}
}
}
opts := ImportOptions{
SkipPrefixValidation: true,
ConflictSkip: conflictSkip,
}
importResult, err := importIssuesCore(ctx, "", store, issues, opts)
if err != nil {
return nil, err
}
result.Issues = importResult.Created
}
return result, nil
}