beads/cmd/bd/import.go
cyb3rd0g01 48b9e904dc
fix(import): seed issue_prefix from config.yaml before the batch guard needs it (#6624)
* fix(import): seed issue_prefix from config.yaml before the batch guard needs it

NewBatchContext (via ReadConfigPrefix) unconditionally requires an
issue_prefix row in the database's own config table before creating
any issue. bd init writes that row via SetConfig, but a database an
external tool provisioned directly (e.g. an orchestrator that created
the Dolt database via SQL and pointed config.yaml at it without ever
calling bd init) never gets one. Every bd import then failed with
"issue_prefix config is missing", even though config.yaml carried the
prefix and bd where/bd context resolved it correctly.

bd import already had logic to sync config.yaml's prefix into that
table, but only after a successful import — useless when the import
itself can't start without the row. This moves an equivalent, guarded
sync (only fires when the table is empty, never overwrites an existing
prefix) before the import attempt, reading config.yaml directly via
GetStringFromDir rather than the global config singleton, since that
singleton isn't guaranteed to be initialized this early in the import
path.

Added a regression test that reproduces the exact failure with stock
bd (bd init --server, then delete the config-table row and write
config.yaml by hand — no orchestrator involved) and confirms it now
imports successfully.

* fix(import): address review — proxied-mode parity, --global exclusion, GetString

Three issues from @bee-ghosttrack's review of ffb32fb:

1. Proxied-server mode had the identical bug: importer.go's ImportBatch ran
   CreateIssuesInTxWithResult (-> NewBatchContext -> ReadConfigPrefix) before
   its own SyncIssuePrefix reconciliation, in the same transaction. Seeds the
   prefix before create there too, when the table is empty.

2. --global was not excluded from either sync (import.go's existing
   post-import sync and this fix's new pre-import seed, plus the proxied
   path's SyncIssuePrefix source). config.yaml is per-project; the shared
   global store's own prefix must win in --global mode (selectCreateIDPrefix
   already established this for `bd create`), or a project's local prefix
   silently overwrites the shared database with two dolt commits.

3. GetStringFromDir's rationale didn't hold: config.Initialize() has already
   run by main.go's init() and prepareSelectedCommandContext, and
   GetStringFromDir skips the config.local.yaml overlay GetString honors —
   disagreeing with the sync 40 lines below it. Switched to GetString,
   matching import.go:415 and import_proxied_server.go's existing usage.

Smaller fixes: TrimSpace + validatePrefix on the seeded value (a failed
validation now warns instead of writing an invalid prefix); SetConfig/Commit
failures now warn to stderr instead of being silently swallowed.

Test changes:
- Added TestImporterUOW/SeedsPrefixBeforeCreateNeedsIt, exercising the
  proxied-mode fix directly at the unit-of-work layer (no CLI/container
  overhead) by clearing config-table issue_prefix and confirming
  ImportBatch still creates issues. Revert-mutant confirmed: fails with
  the identical "issue_prefix config is missing" without the fix.
- Rewrote the E2E test with this repo's server-mode conventions:
  skipIfNoDolt, uniqueTestDBName + dropTestDatabase, explicit --server-port
  — no longer leaks a database on the shared test server.
- Added TestCLI_Import_GlobalModeIgnoresProjectConfigYAMLPrefix (separate
  file, cgo-only build tag, using the shared-server harness from
  global_identity_integration_test.go): confirms a --global import does not
  adopt a project's config.yaml prefix. Revert-mutant confirmed: without the
  --global guards, the shared global store's prefix is overwritten by the
  local project's ("global" -> "localproj").

go build ./... clean; go vet ./cmd/bd/... (both with and without -tags
"cgo integration") clean; gofmt -l clean on all touched files.

* fix(import): commit the seeded prefix on an empty batch; one resolved prefix

Maintainer review fixup for PR #6624 (iterations 1-2). Addresses the two major
findings and the nearby minors/nits.

MAJOR — the proxied seed was discarded on a batch that landed no rows.
The seed wrote issue_prefix but left ImportBatchResult.PrefixSynced false, so
importBatchCommitMessage produced "" for a batch with Created == 0 and no
memories, RunTxResultWithin skipped uw.Commit on the empty message, and the
deferred closeAttempt ROLLED BACK the seeded row. `bd import` reported success,
the prefix never landed, and a re-run dedup'd again — the state never
converged. Reachable against a missing-prefix database whenever every input row
is dropped by the in-transaction stale guard or the CLI's title dedup. The seed
now reports itself through PrefixSynced, restoring the "the prefix sync runs
even when the batch is otherwise empty" contract (be-llaf) that
import_proxied_server.go's comment promises; the field's doc says so too.

MAJOR — that arm had no coverage in any lane that runs.
TestImporterUOW/SeedsPrefixEvenWhenEveryRowIsRejected pins it: empty stored
prefix + SyncIssuePrefix set + every row stale-rejected => the prefix is
committed, under the sync message, with history moving exactly one entry. It
fails on the pre-fixup tree on all four assertions. The existing
SeedsPrefixBeforeCreateNeedsIt cannot catch this (Created == 1 carries the
commit) and PrefixSyncAloneCommitsUnderTheSyncMessage runs with a non-empty
stored prefix, so its seed never fires. Both are tag-free and container-free,
so unlike the classic-path e2e they execute in ordinary lanes.

Minors and nits:

- Seed errors are returned, not warned. A SetConfig failure here is a
  precondition failure for the very next statement: swallowing it surfaced as
  the "issue_prefix config is missing" error this seed exists to prevent, and
  hid serialization failures from RunTxResult's retry classifier. Drops the
  os import.
- Classic path resolves ONE prefix for both its writers (importSyncPrefix;
  iteration 2 below extends the same resolver to the proxied path). The
  seed used a TrimSpace'd + validatePrefix'd value while the post-import sync
  used the raw one, so (a) a prefix the seed refused still reached the database
  through the sync, bypassing the guard this PR adds, and (b) a padded
  `issue-prefix: " bd"` made the two disagree and the sync overwrote the
  trimmed value. TestImportSyncPrefix pins the resolver with no store and no
  container; each arm mutation-proved independently. Side effect: the flagged
  worst_changed function drops from cyclo 25/cognitive 43 to 22/34.
- Warnings name the whole effect on both paths: an invalid issue-prefix
  disables the sync as well as the seed, which base did not do.
- The --global isolation test anchors what its "before" value IS, so the
  after == before comparison cannot go vacuous. The anchor is
  doltserver.GlobalIssuePrefix, NOT the --prefix passed to init: --prefix names
  the project's prefix and `bd init --global` always writes the constant
  (init.go), which an unanchored equality had been hiding.
- Comments record two boundaries the diff left unstated: the classic seed
  commits before an import that may fail (deferring it is not an option — the
  post-import CommitWithConfig is gated on dbPrefix != syncPrefix, which a seed
  makes false, so the row would stay uncommitted because store.Commit excludes
  config (GH#2455), and a dirty internal config key blocks the next pull); and
  a --global import against a global database missing its own
  config.issue_prefix stays unseedable by design.

Iteration 2 — three non-gating findings (both majors above stayed fixed; the
fresh synthesis and quality scorecard both say approve):

- The classic seed now runs BEFORE the memory writes. CommitWithConfig is
  DOLT_COMMIT -Am, so it stages everything dirty at that instant: with the seed
  below the memory loop, a `bd import` carrying memories against a
  missing-prefix database whose issue phase then failed left those kv.memory.*
  rows PERMANENTLY COMMITTED, under a message asserting the commit is the
  prefix seed — where at base the failure left the whole working set for the
  operator to discard. Seeding first restores "the only thing this commit can
  carry is the prefix". Pure statement reorder: the seed reads only issues,
  syncPrefix and the config table, and the ordering is now recorded as
  load-bearing at the block so a later edit does not undo it.
- The proxied path CALLS importSyncPrefix() instead of restating its four rules
  inline. The two copies were identical rule for rule — same globalFlag guard,
  same TrimSpace, same validatePrefix, byte-identical warning — but only the
  classic copy was pinned, so the underscore-key fallback ReadConfigPrefix
  honors, or hyphen normalization, could have reintroduced ACROSS import modes
  exactly the divergence the bullet above closed within one mode. One
  implementation now, so TestImportSyncPrefix's six rows govern both routes.
  Drops the os, strings and config imports from import_proxied_server.go.
- The uow subtests no longer lean on suite order.
  SeedsPrefixEvenWhenEveryRowIsRejected snapshots the sync-message history
  count before the batch and asserts +1: the old `n != 0` form could not fail
  in a full-suite run, because PrefixSyncAloneCommitsUnderTheSyncMessage has
  already committed under that message by then. Mutation-proved — routing an
  all-rejected batch to a "0 issues from ..." message reds only the new
  assertion ("moved 1 -> 1"), and the 1 in that output is precisely the count
  that would have satisfied the old form. SeedsPrefixBeforeCreateNeedsIt also
  restores issue_prefix the way its two neighbours do, instead of leaving the
  fixture to be restored by the code under test.

The PR description's stale "GetStringFromDir / config.GetString returned empty
in this exact spot" claim and its "single-file production change" scope claim
were corrected on GitHub, and the CI-lane reality for each added test is now
recorded there.

Validation (iteration 1): gofmt clean; go build ./...; go vet on cmd/bd,
internal/storage/uow and issueops; full internal/storage/uow package green
(570s, under a private temp root); focused cmd/bd import tests,
TestImportSyncPrefix, TestEmbeddedImport/prefix_sync{,_no_config_key} and the
--global isolation e2e all green.

Validation (iteration 2): gofmt and `git diff --check` clean; go build
./cmd/... ./internal/storage/uow/...; go vet on cmd/bd and
internal/storage/uow; full TestImporterUOW green in subtest order (9/9) plus
the mutation A/B above; TestImportSyncPrefix green (6/6). Two reds in the wider
`go test -run Import ./cmd/bd` are environmental on this box and reproduce
identically with the fix set stashed: TestCLI_Import_GlobalModeIgnoresProjectConfigYAMLPrefix
dies inside `bd init` because the cloned remote is one schema version behind
this build (v66 -> v67), and TestImportBlockedDuringMigrationFreeze cannot
start its server (port 3307 busy).

---------

Co-authored-by: ilasic <ilasic@reversinglabs.com>
Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
2026-09-26 15:01:04 -07:00

620 lines
25 KiB
Go

package main
import (
"bufio"
"context"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/beads"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/metrics"
"github.com/steveyegge/beads/internal/storage/uow"
"github.com/steveyegge/beads/internal/types"
)
var importCmd = &cobra.Command{
Use: "import [file|-]",
Short: "Import issues from a JSONL file or stdin into the database",
Long: `Import issues from a JSONL file (newline-delimited JSON) into the database.
If no file is specified, imports from the configured import.path under .beads/
(default: issues.jsonl). Use "-" to read from stdin; redirecting stdin without
"-" or a file argument is an error, so a typo'd 'bd import < file' cannot
silently import the default file instead. This is the incremental counterpart to
'bd export': new issues are created and existing issues are updated (upsert
semantics).
Memory records (lines with "_type":"memory") are automatically detected and
imported as persistent memories (equivalent to 'bd remember'). This makes
'bd export | bd import' a full round-trip for both issues and memories.
Each JSONL line should map to an issue. The importer accepts every field
'bd export' emits — see 'bd export' output for the canonical schema. Only
"title" is required; everything else is optional.
Common fields:
title Required. Short summary.
description Long-form body.
design, notes, Additional content sections.
acceptance_criteria
issue_type bug | feature | task | epic | chore | ...
priority 0-4 (0 = critical). 0 is preserved (no omitempty).
status open | in_progress | blocked | closed | ...
(rows with status "tombstone" are skipped)
assignee, owner, Ownership metadata.
created_by
labels Array of strings.
dependencies Array of {issue_id, depends_on_id, type, ...}.
comments Array of comment objects.
external_ref, Cross-system identifiers (e.g. "gh-9").
source_system
due_at, defer_until RFC3339 timestamps for scheduling.
metadata Arbitrary JSON object preserved verbatim.
Timestamps (created_at, updated_at, started_at, closed_at) are preserved
when present in the JSONL and otherwise filled in by the importer. The
legacy "wisp" boolean is accepted as an alias for "ephemeral".
By default a row only rewrites an existing local issue when its
updated_at is strictly newer. Older rows are skipped (reported as
stale_skipped_ids) and rows with the same updated_at keep every local
column — updated_at has second granularity, so a timestamp tie can be
two distinct same-second updates, and the local row wins the tie
(reported as tie_kept_local_ids; the row's labels/comments/dependencies
still merge). The guard is also enforced inside the upsert itself, so a
local update that lands while the import is running is preserved rather
than overwritten. Existing issues that the import did rewrite are listed
with a field-level summary (updated_issues), so local state changed by
an import is visible. To deliberately restore an older snapshot, pass
--allow-stale, which imports every row even when it overwrites newer
local state.
Large imports are written in bounded transactions (a few hundred issues
each, with a short pause between commits) with progress on stderr, so
concurrent bd commands keep working while the import runs instead of
stalling on one batch-wide write lock. Rows land in dependency order
with their blocking edges in the same transaction, so a half-finished
import never shows a blocked issue as ready. If an import fails partway,
the already-committed chunks are durable and the command exits nonzero;
re-running the same import is safe and converges (rows upsert,
labels/comments/dependencies deduplicate).
EXAMPLES:
bd import # Import from configured import.path
bd import backup.jsonl # Import from a specific file
bd import -i backup.jsonl # Legacy alias for a specific file
bd import - # Read JSONL from stdin
cat issues.jsonl | bd import - # Pipe JSONL from another tool
bd import --dry-run # Show what would be imported
bd import --dedup # Skip issues with duplicate titles
bd import --allow-stale old.jsonl # Restore an older snapshot (overwrites newer local rows)
bd import --json # Structured output with created and skipped IDs`,
GroupID: "sync",
SilenceUsage: true,
SilenceErrors: true,
RunE: runImport,
}
var (
importDryRun bool
importDedup bool
importAllowStale bool
importInput string
)
func init() {
importCmd.Flags().StringVarP(&importInput, "input", "i", "", "Read JSONL from a specific file")
importCmd.Flags().BoolVar(&importDryRun, "dry-run", false, "Show what would be imported without importing")
importCmd.Flags().BoolVar(&importDedup, "dedup", false, "Skip lines whose title matches an existing open issue")
importCmd.Flags().BoolVar(&importAllowStale, "allow-stale", false, "Import rows even when older than the local issue (required to restore an older snapshot)")
rootCmd.AddCommand(importCmd)
}
// importPoolReadTimeout is the per-I/O read deadline a `bd import` gets on
// its shared-pool connections when neither the caller, BEADS_DOLT_POOL_READ_TIMEOUT
// nor dolt.pool-read-timeout set one. The pool default (10s) is a fast-fail
// tuned for interactive commands; an import's chunk COMMIT of 250 rows with
// their aux tables legitimately outlives it whenever the server pauses — a
// stock dolt sql-server's auto_gc took 16.5s mid-import in the wy-9we0jf
// rollback drill, and every such pause surfaced as "i/o timeout" followed by
// "write commit result indeterminate" (wy-sbgucn). 5m matches the repo's
// other known-long operations (execWithLongTimeout, withReadTxLongTimeout).
const importPoolReadTimeout = 5 * time.Minute
// bulkLoadPoolReadTimeout returns the pool read-timeout fallback for cmd: the
// bulk-load deadline for `bd import`, zero (keep the pool default) otherwise.
// It is a FALLBACK, not an override — an operator's explicit setting still wins
// (see dolt.Config.PoolReadTimeoutFallback).
func bulkLoadPoolReadTimeout(cmd *cobra.Command) time.Duration {
if cmd != nil && cmd.Name() == "import" {
return importPoolReadTimeout
}
return 0
}
func runImport(cmd *cobra.Command, args []string) error {
// Explicit call, not inherited from CheckReadonly: runImport doesn't call
// CheckReadonly at all (a separate, pre-existing gap — readonlyMode
// doesn't gate bd import either), so it can't pick up the freeze check
// folded into CheckReadonly the way create/update/close/remember do.
// The path that makes this call load-bearing rather than redundant is
// `bd import --dry-run`: a preview sets useReadOnly, so it skips the early
// gate in PersistentPreRunE, and runImport is the only chokepoint left.
// Plain `bd import` is already stopped by that early gate.
if err := migrationFreezeGateFor(cmd, "import"); err != nil {
return err
}
evt := metrics.NewCommandEvent("import")
defer func() {
if c := metrics.Global(); c != nil {
c.CloseEventAndAdd(evt)
}
}()
if err := runImportInner(args); err != nil {
if _, isExit := err.(*exitError); isExit {
return err
}
return HandleErrorRespectJSON("%v", err)
}
return nil
}
func runImportInner(args []string) error {
ctx := rootCtx
if importInput != "" && len(args) > 0 {
return fmt.Errorf("use either --input or a positional file, not both")
}
fromStdin := importInput == "-" || (len(args) > 0 && args[0] == "-")
if fromStdin {
return runImportFromReader(ctx, os.Stdin, "stdin")
}
// Determine source file
var jsonlPath string
if importInput != "" {
jsonlPath = importInput
} else if len(args) > 0 {
jsonlPath = args[0]
} else {
// bd-axluy: `bd import < file` (or `... | bd import`) without "-"
// used to silently ignore stdin and import the default JSONL — a
// mutating command diverging from what the user piped. Demand an
// explicit source instead. /dev/null (the stdin subprocesses get by
// default) is a character device, so scripted bare `bd import` with
// no redirection still works.
if fi, statErr := os.Stdin.Stat(); statErr == nil && fi.Mode()&os.ModeCharDevice == 0 {
return fmt.Errorf("stdin is redirected, but without \"-\" bd import ignores it and imports the default JSONL instead; use 'bd import -' to import what you piped, or name a file explicitly")
}
beadsDir := beads.FindBeadsDir()
if beadsDir == "" {
return fmt.Errorf("%s — %s", activeWorkspaceNotFoundError(), diagHint())
}
if globalFlag {
jsonlPath = filepath.Join(beadsDir, "global-issues.jsonl")
} else {
jsonlPath = configuredImportJSONLPath(beadsDir)
}
}
info, err := os.Stat(jsonlPath)
if err != nil {
return fmt.Errorf("cannot read %s: %w", jsonlPath, err)
}
if info.Size() == 0 {
if jsonOutput {
return outputJSON(importResultJSON{Source: jsonlPath})
}
fmt.Fprintf(os.Stderr, "Empty file: %s\n", jsonlPath)
return nil
}
f, err := os.Open(jsonlPath) //nolint:gosec // G304: CLI argument
if err != nil {
return fmt.Errorf("cannot open %s: %w", jsonlPath, err)
}
defer f.Close()
return runImportFromReader(ctx, f, jsonlPath)
}
type importResultJSON struct {
Source string `json:"source"`
Created int `json:"created"`
Updated int `json:"updated,omitempty"`
Unchanged int `json:"unchanged,omitempty"`
Skipped int `json:"skipped"`
DedupHits int `json:"dedup_skipped,omitempty"`
Memories int `json:"memories,omitempty"`
IDs []string `json:"ids,omitempty"`
UpdatedIssues []ImportChange `json:"updated_issues,omitempty"`
TieKeptLocalIDs []string `json:"tie_kept_local_ids,omitempty"`
StaleSkippedIDs []string `json:"stale_skipped_ids,omitempty"`
SkippedDependencies []string `json:"skipped_dependencies,omitempty"`
DryRun bool `json:"dry_run,omitempty"`
}
func runImportFromReader(ctx context.Context, r io.Reader, source string) error {
issues, memories, err := parseImportRecords(r)
if err != nil {
return err
}
if usesProxiedServer() {
return runImportRecordsProxied(ctx, issues, memories, source)
}
if store == nil {
return fmt.Errorf("no database — run 'bd init' or 'bd bootstrap' first")
}
return runImportRecordsClassic(ctx, issues, memories, source)
}
// parseImportRecords scans one JSONL stream into issue rows and memory
// records — the `bd import` / `bd import -` parse loop, shared by the classic
// and proxied modes. Same record vocabulary as parseJSONLFile (the bootstrap
// reader): the optional _schema header and tombstones are skipped, and the
// "wisp_plane" boolean is honored as the explicit wisps-plane marker (and
// the legacy "wisp" alias for "ephemeral") via applyImportWispPlane.
func parseImportRecords(r io.Reader) ([]*types.Issue, []memoryRecord, error) {
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 0, 1024*1024), 64*1024*1024)
var issues []*types.Issue
var memories []memoryRecord
for scanner.Scan() {
line := scanner.Text()
if line == "" {
continue
}
var peek map[string]json.RawMessage
if err := json.Unmarshal([]byte(line), &peek); err != nil {
return nil, nil, fmt.Errorf("failed to parse JSONL line: %w", err)
}
// Skip the optional beads-jsonl header record (§J1.3). A canonical
// export may prepend a provenance line, e.g.
// {"_schema":"beads-jsonl/1","_dolt_branch":"main","_sort":"stable-v1"}.
// It carries no _type and no issue fields; without this guard it falls
// through to the issue path, unmarshals into an empty Issue, and aborts
// the whole import with "title is required". parseJSONLFile (the
// bootstrap reader) has always skipped it; this loop — the one `bd
// import` and `bd import -` run through — did not.
if _, isHeader := peek["_schema"]; isHeader {
continue
}
if rawType, ok := peek["_type"]; ok {
var typeStr string
if err := json.Unmarshal(rawType, &typeStr); err == nil && typeStr == "memory" {
var mem memoryRecord
if err := json.Unmarshal([]byte(line), &mem); err != nil {
return nil, nil, fmt.Errorf("failed to parse memory record: %w", err)
}
if mem.Key != "" && mem.Value != "" {
memories = append(memories, mem)
}
continue
}
}
var issue types.Issue
if err := json.Unmarshal([]byte(line), &issue); err != nil {
return nil, nil, fmt.Errorf("failed to parse issue from JSONL: %w", err)
}
if issue.Status == "tombstone" {
continue
}
applyImportWispPlane(peek, &issue)
issue.SetDefaults()
issues = append(issues, &issue)
}
if err := scanner.Err(); err != nil {
return nil, nil, fmt.Errorf("failed to scan JSONL: %w", err)
}
return issues, memories, nil
}
// runImportRecordsClassic is the classic (embedded/direct store) import
// pipeline over the parsed records: dedup, dry-run classification, memory
// writes, the batch issue import, the final commit and the issue_prefix
// reconciliation.
func runImportRecordsClassic(ctx context.Context, issues []*types.Issue, memories []memoryRecord, source string) error {
// Dedup: skip issues whose title matches an existing open issue
dedupHits := 0
if importDedup && len(issues) > 0 {
issues, dedupHits = filterDuplicatesByTitle(ctx, store, issues)
}
result := importResultJSON{
Source: source,
DedupHits: dedupHits,
DryRun: importDryRun,
}
if importDryRun {
result.Memories = len(memories)
result.Skipped = dedupHits
classification, err := classifyDryRunImport(ctx, store, issues, importAllowStale)
if err != nil {
return fmt.Errorf("dry-run: %w", err)
}
applyImportDryRunClassification(&result, classification)
return renderImportDryRun(result, len(memories), source, dedupHits)
}
// ONE value for both prefix writers on this path — the seed just below and
// the post-import sync at the bottom. Resolved once so they cannot disagree
// on trimming or on validation; see importSyncPrefix.
syncPrefix := importSyncPrefix()
// Seed issue_prefix from config.yaml before the config table has one, so
// NewBatchContext's ReadConfigPrefix (below, via importIssuesCore) does
// not reject an externally-provisioned database that config.yaml already
// names a prefix for.
//
// It runs BEFORE the memory writes below, deliberately: CommitWithConfig
// is DOLT_COMMIT -Am, so it stages everything dirty at that instant.
// Seeding first is what keeps the prefix the only thing this commit can
// carry — otherwise this run's kv.memory.* rows land in a commit named for
// the prefix seed, and stay permanently committed when the import below
// then fails, instead of being left in the working set for the operator to
// discard.
//
// The commit lands HERE, before an import that may fail, which means a
// failed `bd import` can leave the seeded prefix behind. Accepted: the
// value is config.yaml's own and is the converged one either way, so the
// residue is the state the next successful import would reach. Deferring
// it to the post-import CommitWithConfig below is NOT an alternative —
// that block only fires when dbPrefix != syncPrefix, which a seed makes
// false, so the row would stay uncommitted (store.Commit excludes config,
// GH#2455) and a dirty internal config key then blocks the next pull.
if len(issues) > 0 && syncPrefix != "" {
if dbPrefix, _ := store.GetConfig(ctx, "issue_prefix"); dbPrefix == "" {
if setErr := store.SetConfig(ctx, "issue_prefix", syncPrefix); setErr != nil {
fmt.Fprintf(os.Stderr, "warning: failed to seed issue_prefix from config.yaml: %v\n", setErr)
} else if commitErr := store.CommitWithConfig(ctx, "bd import: seed issue_prefix from config.yaml"); commitErr != nil {
fmt.Fprintf(os.Stderr, "warning: failed to commit seeded issue_prefix: %v\n", commitErr)
}
}
}
// Import memories
for _, mem := range memories {
storageKey := kvPrefix + memoryPrefix + mem.Key
if err := store.SetConfig(ctx, storageKey, mem.Value); err != nil {
return fmt.Errorf("failed to import memory %q: %w", mem.Key, err)
}
result.Memories++
}
// Import issues
if len(issues) > 0 {
opts := ImportOptions{SkipPrefixValidation: true, AllowStale: importAllowStale}
importResult, err := importIssuesCore(ctx, "", store, issues, opts)
if err != nil {
return fmt.Errorf("import failed: %w", err)
}
applyImportOutcome(&result, importResult)
}
if result.Created > 0 || result.Memories > 0 {
commitMsg := fmt.Sprintf("bd import: %d issues", result.Created)
if result.Memories > 0 {
commitMsg += fmt.Sprintf(", %d memories", result.Memories)
}
commitMsg += fmt.Sprintf(" from %s", filepath.Base(source))
if err := store.Commit(ctx, commitMsg); err != nil {
// An import can be a working-set no-op: re-importing an
// identical snapshot, or equal-timestamp rows whose guarded
// upsert kept every local column (bd-hj85c).
if !strings.Contains(err.Error(), "nothing to commit") {
return fmt.Errorf("commit: %w", err)
}
}
}
// Sync issue_prefix from config.yaml to the database if stale (be-llaf).
// store.Commit skips the config table (GH#2455), so we use CommitWithConfig
// for this intentional config update after the issues commit completes.
// config.yaml is authoritative here and existing issue IDs are intentionally
// left unchanged: this deliberately bypasses the `bd config set issue_prefix`
// guard for the import/migration flow and is not a rename. Reconciles
// against the SAME resolved value the seed used, so a padded config.yaml
// prefix cannot make this test true forever and overwrite the trimmed
// value the seed stored on every import.
if syncPrefix != "" {
if dbPrefix, _ := store.GetConfig(ctx, "issue_prefix"); dbPrefix != syncPrefix {
if setErr := store.SetConfig(ctx, "issue_prefix", syncPrefix); setErr == nil {
_ = store.CommitWithConfig(ctx, "bd import: sync issue_prefix from config.yaml")
}
}
}
return renderImportOutcome(result, source, dedupHits)
}
// importSyncPrefix resolves the ONE config.yaml issue_prefix value EVERY
// import writer uses: the classic path's seed (before the batch needs it) and
// its post-import sync (after the batch lands), and the SyncIssuePrefix the
// proxied path hands ImportBatch (import_proxied_server.go), which seeds and
// syncs inside its one transaction. "" means "reconcile nothing", which is
// what --global, an absent value, and an invalid value all resolve to.
//
// Resolving once is what keeps those writers honest: the seed stored the
// TRIMMED value while the sync used to compare the raw one, so a padded
// `issue-prefix: " bd"` made them disagree and the sync overwrote the seed's
// value; and the sync used to skip validatePrefix entirely, so a prefix the
// seed refused still reached the database through it.
func importSyncPrefix() string {
// --global: config.yaml is a per-project file, and the shared global
// store's own prefix must win (selectCreateIDPrefix), not whatever project
// happened to import into it.
if globalFlag {
return ""
}
yamlPrefix := strings.TrimSpace(config.GetString("issue-prefix"))
if yamlPrefix == "" {
return ""
}
if err := validatePrefix(yamlPrefix); err != nil {
// Naming both effects: an invalid value disables the sync as well as
// the seed, and base did sync an invalid prefix through.
fmt.Fprintf(os.Stderr, "warning: ignoring invalid issue-prefix %q from config.yaml (not seeding or syncing issue_prefix): %v\n", yamlPrefix, err)
return ""
}
return yamlPrefix
}
// applyImportDryRunClassification folds a dry-run classification into the
// command's JSON result, identically in both modes.
func applyImportDryRunClassification(result *importResultJSON, classification *ImportResult) {
result.Created = classification.Created
result.Updated = classification.Updated
result.Unchanged = classification.Unchanged
result.Skipped += classification.Skipped
result.IDs = append(result.IDs, classification.ImportedIDs...)
result.StaleSkippedIDs = classification.StaleSkippedIDs
result.UpdatedIssues = classification.UpdatedIssues
result.TieKeptLocalIDs = classification.TieKeptLocalIDs
}
// applyImportOutcome folds a real import's outcome into the command's JSON
// result, identically in both modes.
func applyImportOutcome(result *importResultJSON, importResult *ImportResult) {
result.Created = importResult.Created
result.Updated = importResult.Updated
result.Unchanged = importResult.Unchanged
result.Skipped += importResult.Skipped
result.SkippedDependencies = append(result.SkippedDependencies, importResult.SkippedDependencies...)
result.IDs = append(result.IDs, importResult.ImportedIDs...)
result.UpdatedIssues = append(result.UpdatedIssues, importResult.UpdatedIssues...)
result.TieKeptLocalIDs = append(result.TieKeptLocalIDs, importResult.TieKeptLocalIDs...)
result.StaleSkippedIDs = append(result.StaleSkippedIDs, importResult.StaleSkippedIDs...)
}
// renderImportDryRun reports a dry run (JSON or stderr), identically in both
// modes.
func renderImportDryRun(result importResultJSON, memoriesCount int, source string, dedupHits int) error {
if jsonOutput {
return outputJSON(result)
}
// The leading count is the sum of the breakdown that follows it
// (not len(issues)), which can be larger when rows were stale
// skipped — those are reported separately below instead of being
// folded into a total the breakdown then wouldn't add up to.
considered := result.Created + result.Updated + result.Unchanged
//nolint:gosec // G705: stderr, not a browser context
fmt.Fprintf(os.Stderr, "Would import %d issues (%d new, %d updated, %d unchanged) and %d memories from %s",
considered, result.Created, result.Updated, result.Unchanged, memoriesCount, source)
if dedupHits > 0 {
fmt.Fprintf(os.Stderr, " (%d duplicates skipped)", dedupHits) //nolint:gosec // G705: stderr, not a browser context
}
if len(result.StaleSkippedIDs) > 0 {
fmt.Fprintf(os.Stderr, " (%d stale skipped)", len(result.StaleSkippedIDs))
}
fmt.Fprintln(os.Stderr)
return nil
}
// renderImportOutcome reports a completed import (JSON or stderr),
// identically in both modes.
func renderImportOutcome(result importResultJSON, source string, dedupHits int) error {
if jsonOutput {
return outputJSON(result)
}
fmt.Fprintf(os.Stderr, "Imported %d issues", result.Created)
if result.Memories > 0 {
fmt.Fprintf(os.Stderr, " and %d memories", result.Memories)
}
fmt.Fprintf(os.Stderr, " from %s", source)
if dedupHits > 0 {
fmt.Fprintf(os.Stderr, " (%d duplicates skipped)", dedupHits) //nolint:gosec // G705: stderr, not a browser context
}
if staleSkipped := result.Skipped - dedupHits; staleSkipped > 0 {
fmt.Fprintf(os.Stderr, " (%d stale skipped; use --allow-stale to restore older rows)", staleSkipped) //nolint:gosec // G705: stderr, not a browser context
}
if result.Unchanged > 0 {
fmt.Fprintf(os.Stderr, " (%d already present, unchanged)", result.Unchanged) //nolint:gosec // G705: stderr, not a browser context
}
fmt.Fprintln(os.Stderr)
if len(result.UpdatedIssues) > 0 {
fmt.Fprintf(os.Stderr, "Updated %d existing issue(s):\n", len(result.UpdatedIssues))
for _, change := range result.UpdatedIssues {
fmt.Fprintf(os.Stderr, " %s: %s\n", change.ID, change.Changes)
}
}
if len(result.TieKeptLocalIDs) > 0 {
fmt.Fprintf(os.Stderr, "Kept local state for %d issue(s) with the same updated_at but different content (use --allow-stale to overwrite): %s\n",
len(result.TieKeptLocalIDs), strings.Join(result.TieKeptLocalIDs, ", "))
}
for _, skipped := range result.SkippedDependencies {
fmt.Fprintf(os.Stderr, "Skipped dependency: %s\n", skipped)
}
return nil
}
// importTitleSearcher is the read seam the --dedup filter needs. It lives in
// THIS file because naming types.IssueFilter is denied by default under
// cmd/bd and import.go is the named exception for the bulk-movement family
// (.golangci.yml, forbidigo): the classic storage.DoltStorage satisfies it
// directly, and uowImportTitleSearcher adapts the proxied unit of work.
type importTitleSearcher interface {
SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error)
}
// uowImportTitleSearcher adapts a unit of work's issue use case to the
// classic []*types.Issue search shape filterDuplicatesByTitle consumes. Both
// stacks run the same issueops search underneath (issues merged with wisps),
// so --dedup sees the same title universe in both modes.
type uowImportTitleSearcher struct {
uw uow.UnitOfWork
}
func (s uowImportTitleSearcher) SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) {
page, err := s.uw.IssueUseCase().SearchIssues(ctx, query, filter)
if err != nil {
return nil, err
}
return page.Items, nil
}
// filterDuplicatesByTitle removes issues whose title matches an existing open issue.
func filterDuplicatesByTitle(ctx context.Context, st importTitleSearcher, issues []*types.Issue) ([]*types.Issue, int) {
existing, err := st.SearchIssues(ctx, "", types.IssueFilter{})
if err != nil {
return issues, 0
}
titleSet := make(map[string]bool, len(existing))
for _, issue := range existing {
if issue.Status != types.StatusClosed {
titleSet[strings.ToLower(issue.Title)] = true
}
}
var kept []*types.Issue
skipped := 0
for _, issue := range issues {
if titleSet[strings.ToLower(issue.Title)] {
skipped++
continue
}
kept = append(kept, issue)
}
return kept, skipped
}