beads/cmd/bd/import.go
Isis Anisoptera 45a116d1a7
fix(update): fence unforced --notes overwrites behind --force across all surfaces (#5946)
* fix(update): fence unforced --notes overwrites behind --force across all surfaces

bd update --notes silently replaced existing notes, destroying handoff
history. The contract now refuses an unforced overwrite of non-empty
notes with ErrNotesOverwrite (409 notes_overwrite_refused over HTTP,
force_notes_overwrite to override), enforced inside the mutation
transaction on both the dolt and uow backends, for single updates and
batch apply alike. CAS guards keep precedence: a stale
--if-assignee/--if-status still surfaces as the exit-13 mismatch, on
both backends, before the fence speaks.

CLI: --force and --if-assignee are no longer mutually exclusive (that
exclusion made overwriting under a guard impossible); both update
routes translate the fence's sentinel into the same actionable advice
(--force to overwrite, --append-notes to preserve history), and the
post---force warning is worded as an audit statement. The shipped
plugin skill docs and bd note help now prescribe --append-notes for
handoffs instead of the newly fenced bare --notes.

The fence is the single enforcement point: no client-side pre-read
copies of the rule. Conformance contracts, unit tests for
AuthorizeNotesOverwrite, and embedded/proxied/httpapi regression
coverage included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MAFpn9pUw8WKMfymteZHbh

* fix(update): exempt explicit clears from the notes fence; share its predicate

CI surfaced three gaps in the fence's reach:

- Clearing notes (--notes "" / patch.notes: "") was refused, diverging
  from every sibling text field's clear and from the backend behavior
  contracts. The fence now targets only its actual footgun — replacing
  non-empty notes with different non-empty content — via a single shared
  predicate, issueops.NotesReplacement, used by both the contract's
  AuthorizeNotesOverwrite and the CLI's warning path, so the two can no
  longer drift. Prose updated to match everywhere the rule is stated
  (ErrNotesOverwrite, ForceNotesOverwrite, OpenAPI spec + regenerated
  types).
- The revision-CAS integration tests' incidental concurrent writes now
  patch design instead of notes, so they no longer depend on notes-fence
  semantics at all.
- The historical-upgrade harness deliberately replaces fixture notes and
  asserts the new value; its two bd update --notes calls now pass
  --force.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MAFpn9pUw8WKMfymteZHbh

* fix(update): address review — changelog BREAKING entry, problem.go doc block, close-policy-under-guard test

Review follow-ups on #5946 (bee-ghosttrack, MERGE-AFTER-FIXES):

- CHANGELOG: BREAKING entry under [Unreleased] ### Changed for the notes
  fence on both wires (CLI exit 1, HTTP 409 notes_overwrite_refused), with
  the --force / --append-notes migration line. It also names the two
  deliberate edges: the explicit-clear exemption, and --force widening to
  the close-policy half under --if-assignee now that the flag pair is legal.
- problem.go: CodeNotesOverwrite's comment had landed inside
  CodeDependencyCycle's doc block; each const now sits under its own docs.
- assign_fence_embedded_test.go: pin the previously untested close-policy
  half of --force under --if-assignee (refused unforced, authorized forced).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED

* test(update): align TestWarnNotesReplacement with the post-fence warning copy

The test landed on main after this branch was cut and pins the pre-fence
wording, where the warning fired on any unforced overwrite. Since the fence,
an unforced overwrite is refused outright and warnNotesReplacement fires only
after a forced one — the audit trail for a habitual --force — so the copy it
pins is now '--force replaced existing notes'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED

* docs(import): record bd import's notes-fence exemption as by-design (GH#6190)

Import consults none of the per-field live-state fences — notes overwrite,
live-claim reassign, close policy — and that is the intended shape, not an
omission: its contract is wholesale row replacement, guarded by its own
staleness check (only strictly-newer rows rewrite local state, --allow-stale
is the single deliberate-overwrite opt-in) and reported field-by-field in
updated_issues. Named in AuthorizeNotesOverwrite's enforcement-boundary
comment and in bd import's help; the published CLI docs pick the help text
up at the next release pin bump.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED

* fix(update): refuse --notes "" and add --clear-notes; stop --notes help reading as additive

GH#6021: an empty --notes wiped the whole notes field at exit 0 behind the
same ✓ Updated receipt as a successful write — and the empty string is
exactly what a dead command substitution collapses to. The empty value is
now refused on both CLI routes (the accident and the deliberate clear are
the same bytes at the flag layer), and the deliberate clear gets its own
verb, --clear-notes, mutually exclusive with --notes/--append-notes. A verb
rather than the proposed --allow-empty-notes opt-in: what needs authorizing
is an intent, not an input path — --allow-empty-description unblocks
stdin/file plumbing, which notes does not have — and an opt-in passed
habitually would silently disarm the guard while a habitual --clear-notes
fails loudly. --force does not bypass the refusal; the fence's clear
exemption stands, so the verb alone authorizes the erase. The HTTP API is
unchanged: patch.notes "" still clears.

GH#6272: bd update --help called --notes "Additional notes" and the prime
text said "Add supplementary notes" — both read as appending, so agents
following the injected guidance faithfully clobbered prior context. The
update help now names the operation with the --append-notes contrast at the
point of use, the shared/create help is field-named like --description and
--design, and the prime line points at bd update --append-notes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED

* fix(bazel): enumerate the PR-new files in cmd/bd and issueops srcs

The base added cmd/bd/BUILD.bazel and internal/storage/issueops/BUILD.bazel
with EXPLICIT srcs lists after this branch was cut, so the three files this
PR adds were never enumerated:

  cmd/bd/update_notes_guard.go        (go_library bd_lib)
  cmd/bd/update_notes_guard_test.go   (go_test bd_test)
  internal/storage/issueops/aggregate_authorize_notes_test.go (go_test)

The go_library omission is a hard build break, not a silent skip:
update.go:215,220 and update_input.go:106,111 are enumerated and call
validateNotesUpdate / clearNotesRequested, so //cmd/bd:bd_lib compiles the
callers without the callee. bazel.yml arms unconditionally for fork PRs
(head.repo.fork == true) and builds //cmd/bd:bd + //cmd/bd:bd_test, so this
reds the lane. make bazel-sync-check cannot see it -- go_srcs.py only
validates the glob-based go_srcs filegroup, never the rule srcs lists.

Generated by make bazel-sync (gazelle), not hand-edited, so the workflow's
own BUILD-files-in-sync drift step agrees byte-for-byte.

* fix(notes): name --clear-notes in bd note's rm guidance, correct the fence boundary comment, pin both request-shape rules

Takes all three non-gating minors from the review. No behavior change: two
prose fixes and two test cases.

1. cmd/bd/note.go's rm|remove|delete branch advised
   `bd update <id> --notes "text" --force` under "To change or clear notes",
   but clearing through --notes is exactly what this PR made impossible --
   validateNotesUpdate refuses an empty value unconditionally and --force does
   not reach it. The branch now names both verbs separately: --notes --force to
   replace, --clear-notes to clear. The edit and update branches speak only of
   replacing and were already correct, so they are untouched.

2. aggregate.go's enforcement-boundary comment claimed the UpdateIssueInTx
   funnel's "one notes-replacing caller is bd edit". Under this PR's own
   NotesReplacement predicate there are at least four more (bd note's
   read-combine-write, bd defer --reason, bd rename and delete-tombstoning's
   mechanical ID rewrites, bd compact's clears). The exemption is still sound --
   every funnel writer derives its new value from the current notes -- so the
   comment now names those CLASSES instead of a single caller, which is the
   claim that is actually true and that a future fence auditor can check.

3. Both branches this PR adds to ValidateUpdateRequest were uncovered: the
   batch apply leg reaches them here and nowhere else, while the single-issue
   HTTP handler's own pre-check returns before storage. Added a sibling table
   (the existing one varies only the patch; these two rules are properties of
   the REQUEST) with a case each. Mutation-checked, disjoint: neutralizing the
   force-without-notes branch reds only the first case, neutralizing the
   notes+append branch reds only the second, and the pre-existing
   canonical-fields table stays green under both.

Not an amend of 5ffcc9c16c: that commit's message states it was generated by
make bazel-sync and not hand-edited, so folding hand-written Go into it would
falsify the claim its own BUILD-files-in-sync argument rests on. No new files,
so the srcs lists it repaired stay correct.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
2026-10-01 05:02:40 -07:00

627 lines
26 KiB
Go

package main
import (
"bufio"
"context"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/beads"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/metrics"
"github.com/steveyegge/beads/internal/storage/uow"
"github.com/steveyegge/beads/internal/types"
)
var importCmd = &cobra.Command{
Use: "import [file|-]",
Short: "Import issues from a JSONL file or stdin into the database",
Long: `Import issues from a JSONL file (newline-delimited JSON) into the database.
If no file is specified, imports from the configured import.path under .beads/
(default: issues.jsonl). Use "-" to read from stdin; redirecting stdin without
"-" or a file argument is an error, so a typo'd 'bd import < file' cannot
silently import the default file instead. This is the incremental counterpart to
'bd export': new issues are created and existing issues are updated (upsert
semantics).
Memory records (lines with "_type":"memory") are automatically detected and
imported as persistent memories (equivalent to 'bd remember'). This makes
'bd export | bd import' a full round-trip for both issues and memories.
Each JSONL line should map to an issue. The importer accepts every field
'bd export' emits — see 'bd export' output for the canonical schema. Only
"title" is required; everything else is optional.
Common fields:
title Required. Short summary.
description Long-form body.
design, notes, Additional content sections.
acceptance_criteria
issue_type bug | feature | task | epic | chore | ...
priority 0-4 (0 = critical). 0 is preserved (no omitempty).
status open | in_progress | blocked | closed | ...
(rows with status "tombstone" are skipped)
assignee, owner, Ownership metadata.
created_by
labels Array of strings.
dependencies Array of {issue_id, depends_on_id, type, ...}.
comments Array of comment objects.
external_ref, Cross-system identifiers (e.g. "gh-9").
source_system
due_at, defer_until RFC3339 timestamps for scheduling.
metadata Arbitrary JSON object preserved verbatim.
Timestamps (created_at, updated_at, started_at, closed_at) are preserved
when present in the JSONL and otherwise filled in by the importer. The
legacy "wisp" boolean is accepted as an alias for "ephemeral".
By default a row only rewrites an existing local issue when its
updated_at is strictly newer. Older rows are skipped (reported as
stale_skipped_ids) and rows with the same updated_at keep every local
column — updated_at has second granularity, so a timestamp tie can be
two distinct same-second updates, and the local row wins the tie
(reported as tie_kept_local_ids; the row's labels/comments/dependencies
still merge). The guard is also enforced inside the upsert itself, so a
local update that lands while the import is running is preserved rather
than overwritten. Existing issues that the import did rewrite are listed
with a field-level summary (updated_issues), so local state changed by
an import is visible. To deliberately restore an older snapshot, pass
--allow-stale, which imports every row even when it overwrites newer
local state.
The staleness guard is import's ONLY overwrite guard: the per-field
fences that bd update enforces (notes overwrite, live-claim reassign,
close policy) do not apply to imported rows. Import replaces rows
wholesale by design — a restore must not fail field-by-field — so
--allow-stale is the single deliberate-overwrite opt-in, and
updated_issues is where to look for what an import rewrote.
Large imports are written in bounded transactions (a few hundred issues
each, with a short pause between commits) with progress on stderr, so
concurrent bd commands keep working while the import runs instead of
stalling on one batch-wide write lock. Rows land in dependency order
with their blocking edges in the same transaction, so a half-finished
import never shows a blocked issue as ready. If an import fails partway,
the already-committed chunks are durable and the command exits nonzero;
re-running the same import is safe and converges (rows upsert,
labels/comments/dependencies deduplicate).
EXAMPLES:
bd import # Import from configured import.path
bd import backup.jsonl # Import from a specific file
bd import -i backup.jsonl # Legacy alias for a specific file
bd import - # Read JSONL from stdin
cat issues.jsonl | bd import - # Pipe JSONL from another tool
bd import --dry-run # Show what would be imported
bd import --dedup # Skip issues with duplicate titles
bd import --allow-stale old.jsonl # Restore an older snapshot (overwrites newer local rows)
bd import --json # Structured output with created and skipped IDs`,
GroupID: "sync",
SilenceUsage: true,
SilenceErrors: true,
RunE: runImport,
}
var (
importDryRun bool
importDedup bool
importAllowStale bool
importInput string
)
func init() {
importCmd.Flags().StringVarP(&importInput, "input", "i", "", "Read JSONL from a specific file")
importCmd.Flags().BoolVar(&importDryRun, "dry-run", false, "Show what would be imported without importing")
importCmd.Flags().BoolVar(&importDedup, "dedup", false, "Skip lines whose title matches an existing open issue")
importCmd.Flags().BoolVar(&importAllowStale, "allow-stale", false, "Import rows even when older than the local issue (required to restore an older snapshot)")
rootCmd.AddCommand(importCmd)
}
// importPoolReadTimeout is the per-I/O read deadline a `bd import` gets on
// its shared-pool connections when neither the caller, BEADS_DOLT_POOL_READ_TIMEOUT
// nor dolt.pool-read-timeout set one. The pool default (10s) is a fast-fail
// tuned for interactive commands; an import's chunk COMMIT of 250 rows with
// their aux tables legitimately outlives it whenever the server pauses — a
// stock dolt sql-server's auto_gc took 16.5s mid-import in the wy-9we0jf
// rollback drill, and every such pause surfaced as "i/o timeout" followed by
// "write commit result indeterminate" (wy-sbgucn). 5m matches the repo's
// other known-long operations (execWithLongTimeout, withReadTxLongTimeout).
const importPoolReadTimeout = 5 * time.Minute
// bulkLoadPoolReadTimeout returns the pool read-timeout fallback for cmd: the
// bulk-load deadline for `bd import`, zero (keep the pool default) otherwise.
// It is a FALLBACK, not an override — an operator's explicit setting still wins
// (see dolt.Config.PoolReadTimeoutFallback).
func bulkLoadPoolReadTimeout(cmd *cobra.Command) time.Duration {
if cmd != nil && cmd.Name() == "import" {
return importPoolReadTimeout
}
return 0
}
func runImport(cmd *cobra.Command, args []string) error {
// Explicit call, not inherited from CheckReadonly: runImport doesn't call
// CheckReadonly at all (a separate, pre-existing gap — readonlyMode
// doesn't gate bd import either), so it can't pick up the freeze check
// folded into CheckReadonly the way create/update/close/remember do.
// The path that makes this call load-bearing rather than redundant is
// `bd import --dry-run`: a preview sets useReadOnly, so it skips the early
// gate in PersistentPreRunE, and runImport is the only chokepoint left.
// Plain `bd import` is already stopped by that early gate.
if err := migrationFreezeGateFor(cmd, "import"); err != nil {
return err
}
evt := metrics.NewCommandEvent("import")
defer func() {
if c := metrics.Global(); c != nil {
c.CloseEventAndAdd(evt)
}
}()
if err := runImportInner(args); err != nil {
if _, isExit := err.(*exitError); isExit {
return err
}
return HandleErrorRespectJSON("%v", err)
}
return nil
}
func runImportInner(args []string) error {
ctx := rootCtx
if importInput != "" && len(args) > 0 {
return fmt.Errorf("use either --input or a positional file, not both")
}
fromStdin := importInput == "-" || (len(args) > 0 && args[0] == "-")
if fromStdin {
return runImportFromReader(ctx, os.Stdin, "stdin")
}
// Determine source file
var jsonlPath string
if importInput != "" {
jsonlPath = importInput
} else if len(args) > 0 {
jsonlPath = args[0]
} else {
// bd-axluy: `bd import < file` (or `... | bd import`) without "-"
// used to silently ignore stdin and import the default JSONL — a
// mutating command diverging from what the user piped. Demand an
// explicit source instead. /dev/null (the stdin subprocesses get by
// default) is a character device, so scripted bare `bd import` with
// no redirection still works.
if fi, statErr := os.Stdin.Stat(); statErr == nil && fi.Mode()&os.ModeCharDevice == 0 {
return fmt.Errorf("stdin is redirected, but without \"-\" bd import ignores it and imports the default JSONL instead; use 'bd import -' to import what you piped, or name a file explicitly")
}
beadsDir := beads.FindBeadsDir()
if beadsDir == "" {
return fmt.Errorf("%s — %s", activeWorkspaceNotFoundError(), diagHint())
}
if globalFlag {
jsonlPath = filepath.Join(beadsDir, "global-issues.jsonl")
} else {
jsonlPath = configuredImportJSONLPath(beadsDir)
}
}
info, err := os.Stat(jsonlPath)
if err != nil {
return fmt.Errorf("cannot read %s: %w", jsonlPath, err)
}
if info.Size() == 0 {
if jsonOutput {
return outputJSON(importResultJSON{Source: jsonlPath})
}
fmt.Fprintf(os.Stderr, "Empty file: %s\n", jsonlPath)
return nil
}
f, err := os.Open(jsonlPath) //nolint:gosec // G304: CLI argument
if err != nil {
return fmt.Errorf("cannot open %s: %w", jsonlPath, err)
}
defer f.Close()
return runImportFromReader(ctx, f, jsonlPath)
}
type importResultJSON struct {
Source string `json:"source"`
Created int `json:"created"`
Updated int `json:"updated,omitempty"`
Unchanged int `json:"unchanged,omitempty"`
Skipped int `json:"skipped"`
DedupHits int `json:"dedup_skipped,omitempty"`
Memories int `json:"memories,omitempty"`
IDs []string `json:"ids,omitempty"`
UpdatedIssues []ImportChange `json:"updated_issues,omitempty"`
TieKeptLocalIDs []string `json:"tie_kept_local_ids,omitempty"`
StaleSkippedIDs []string `json:"stale_skipped_ids,omitempty"`
SkippedDependencies []string `json:"skipped_dependencies,omitempty"`
DryRun bool `json:"dry_run,omitempty"`
}
func runImportFromReader(ctx context.Context, r io.Reader, source string) error {
issues, memories, err := parseImportRecords(r)
if err != nil {
return err
}
if usesProxiedServer() {
return runImportRecordsProxied(ctx, issues, memories, source)
}
if store == nil {
return fmt.Errorf("no database — run 'bd init' or 'bd bootstrap' first")
}
return runImportRecordsClassic(ctx, issues, memories, source)
}
// parseImportRecords scans one JSONL stream into issue rows and memory
// records — the `bd import` / `bd import -` parse loop, shared by the classic
// and proxied modes. Same record vocabulary as parseJSONLFile (the bootstrap
// reader): the optional _schema header and tombstones are skipped, and the
// "wisp_plane" boolean is honored as the explicit wisps-plane marker (and
// the legacy "wisp" alias for "ephemeral") via applyImportWispPlane.
func parseImportRecords(r io.Reader) ([]*types.Issue, []memoryRecord, error) {
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 0, 1024*1024), 64*1024*1024)
var issues []*types.Issue
var memories []memoryRecord
for scanner.Scan() {
line := scanner.Text()
if line == "" {
continue
}
var peek map[string]json.RawMessage
if err := json.Unmarshal([]byte(line), &peek); err != nil {
return nil, nil, fmt.Errorf("failed to parse JSONL line: %w", err)
}
// Skip the optional beads-jsonl header record (§J1.3). A canonical
// export may prepend a provenance line, e.g.
// {"_schema":"beads-jsonl/1","_dolt_branch":"main","_sort":"stable-v1"}.
// It carries no _type and no issue fields; without this guard it falls
// through to the issue path, unmarshals into an empty Issue, and aborts
// the whole import with "title is required". parseJSONLFile (the
// bootstrap reader) has always skipped it; this loop — the one `bd
// import` and `bd import -` run through — did not.
if _, isHeader := peek["_schema"]; isHeader {
continue
}
if rawType, ok := peek["_type"]; ok {
var typeStr string
if err := json.Unmarshal(rawType, &typeStr); err == nil && typeStr == "memory" {
var mem memoryRecord
if err := json.Unmarshal([]byte(line), &mem); err != nil {
return nil, nil, fmt.Errorf("failed to parse memory record: %w", err)
}
if mem.Key != "" && mem.Value != "" {
memories = append(memories, mem)
}
continue
}
}
var issue types.Issue
if err := json.Unmarshal([]byte(line), &issue); err != nil {
return nil, nil, fmt.Errorf("failed to parse issue from JSONL: %w", err)
}
if issue.Status == "tombstone" {
continue
}
applyImportWispPlane(peek, &issue)
issue.SetDefaults()
issues = append(issues, &issue)
}
if err := scanner.Err(); err != nil {
return nil, nil, fmt.Errorf("failed to scan JSONL: %w", err)
}
return issues, memories, nil
}
// runImportRecordsClassic is the classic (embedded/direct store) import
// pipeline over the parsed records: dedup, dry-run classification, memory
// writes, the batch issue import, the final commit and the issue_prefix
// reconciliation.
func runImportRecordsClassic(ctx context.Context, issues []*types.Issue, memories []memoryRecord, source string) error {
// Dedup: skip issues whose title matches an existing open issue
dedupHits := 0
if importDedup && len(issues) > 0 {
issues, dedupHits = filterDuplicatesByTitle(ctx, store, issues)
}
result := importResultJSON{
Source: source,
DedupHits: dedupHits,
DryRun: importDryRun,
}
if importDryRun {
result.Memories = len(memories)
result.Skipped = dedupHits
classification, err := classifyDryRunImport(ctx, store, issues, importAllowStale)
if err != nil {
return fmt.Errorf("dry-run: %w", err)
}
applyImportDryRunClassification(&result, classification)
return renderImportDryRun(result, len(memories), source, dedupHits)
}
// ONE value for both prefix writers on this path — the seed just below and
// the post-import sync at the bottom. Resolved once so they cannot disagree
// on trimming or on validation; see importSyncPrefix.
syncPrefix := importSyncPrefix()
// Seed issue_prefix from config.yaml before the config table has one, so
// NewBatchContext's ReadConfigPrefix (below, via importIssuesCore) does
// not reject an externally-provisioned database that config.yaml already
// names a prefix for.
//
// It runs BEFORE the memory writes below, deliberately: CommitWithConfig
// is DOLT_COMMIT -Am, so it stages everything dirty at that instant.
// Seeding first is what keeps the prefix the only thing this commit can
// carry — otherwise this run's kv.memory.* rows land in a commit named for
// the prefix seed, and stay permanently committed when the import below
// then fails, instead of being left in the working set for the operator to
// discard.
//
// The commit lands HERE, before an import that may fail, which means a
// failed `bd import` can leave the seeded prefix behind. Accepted: the
// value is config.yaml's own and is the converged one either way, so the
// residue is the state the next successful import would reach. Deferring
// it to the post-import CommitWithConfig below is NOT an alternative —
// that block only fires when dbPrefix != syncPrefix, which a seed makes
// false, so the row would stay uncommitted (store.Commit excludes config,
// GH#2455) and a dirty internal config key then blocks the next pull.
if len(issues) > 0 && syncPrefix != "" {
if dbPrefix, _ := store.GetConfig(ctx, "issue_prefix"); dbPrefix == "" {
if setErr := store.SetConfig(ctx, "issue_prefix", syncPrefix); setErr != nil {
fmt.Fprintf(os.Stderr, "warning: failed to seed issue_prefix from config.yaml: %v\n", setErr)
} else if commitErr := store.CommitWithConfig(ctx, "bd import: seed issue_prefix from config.yaml"); commitErr != nil {
fmt.Fprintf(os.Stderr, "warning: failed to commit seeded issue_prefix: %v\n", commitErr)
}
}
}
// Import memories
for _, mem := range memories {
storageKey := kvPrefix + memoryPrefix + mem.Key
if err := store.SetConfig(ctx, storageKey, mem.Value); err != nil {
return fmt.Errorf("failed to import memory %q: %w", mem.Key, err)
}
result.Memories++
}
// Import issues
if len(issues) > 0 {
opts := ImportOptions{SkipPrefixValidation: true, AllowStale: importAllowStale}
importResult, err := importIssuesCore(ctx, "", store, issues, opts)
if err != nil {
return fmt.Errorf("import failed: %w", err)
}
applyImportOutcome(&result, importResult)
}
if result.Created > 0 || result.Memories > 0 {
commitMsg := fmt.Sprintf("bd import: %d issues", result.Created)
if result.Memories > 0 {
commitMsg += fmt.Sprintf(", %d memories", result.Memories)
}
commitMsg += fmt.Sprintf(" from %s", filepath.Base(source))
if err := store.Commit(ctx, commitMsg); err != nil {
// An import can be a working-set no-op: re-importing an
// identical snapshot, or equal-timestamp rows whose guarded
// upsert kept every local column (bd-hj85c).
if !strings.Contains(err.Error(), "nothing to commit") {
return fmt.Errorf("commit: %w", err)
}
}
}
// Sync issue_prefix from config.yaml to the database if stale (be-llaf).
// store.Commit skips the config table (GH#2455), so we use CommitWithConfig
// for this intentional config update after the issues commit completes.
// config.yaml is authoritative here and existing issue IDs are intentionally
// left unchanged: this deliberately bypasses the `bd config set issue_prefix`
// guard for the import/migration flow and is not a rename. Reconciles
// against the SAME resolved value the seed used, so a padded config.yaml
// prefix cannot make this test true forever and overwrite the trimmed
// value the seed stored on every import.
if syncPrefix != "" {
if dbPrefix, _ := store.GetConfig(ctx, "issue_prefix"); dbPrefix != syncPrefix {
if setErr := store.SetConfig(ctx, "issue_prefix", syncPrefix); setErr == nil {
_ = store.CommitWithConfig(ctx, "bd import: sync issue_prefix from config.yaml")
}
}
}
return renderImportOutcome(result, source, dedupHits)
}
// importSyncPrefix resolves the ONE config.yaml issue_prefix value EVERY
// import writer uses: the classic path's seed (before the batch needs it) and
// its post-import sync (after the batch lands), and the SyncIssuePrefix the
// proxied path hands ImportBatch (import_proxied_server.go), which seeds and
// syncs inside its one transaction. "" means "reconcile nothing", which is
// what --global, an absent value, and an invalid value all resolve to.
//
// Resolving once is what keeps those writers honest: the seed stored the
// TRIMMED value while the sync used to compare the raw one, so a padded
// `issue-prefix: " bd"` made them disagree and the sync overwrote the seed's
// value; and the sync used to skip validatePrefix entirely, so a prefix the
// seed refused still reached the database through it.
func importSyncPrefix() string {
// --global: config.yaml is a per-project file, and the shared global
// store's own prefix must win (selectCreateIDPrefix), not whatever project
// happened to import into it.
if globalFlag {
return ""
}
yamlPrefix := strings.TrimSpace(config.GetString("issue-prefix"))
if yamlPrefix == "" {
return ""
}
if err := validatePrefix(yamlPrefix); err != nil {
// Naming both effects: an invalid value disables the sync as well as
// the seed, and base did sync an invalid prefix through.
fmt.Fprintf(os.Stderr, "warning: ignoring invalid issue-prefix %q from config.yaml (not seeding or syncing issue_prefix): %v\n", yamlPrefix, err)
return ""
}
return yamlPrefix
}
// applyImportDryRunClassification folds a dry-run classification into the
// command's JSON result, identically in both modes.
func applyImportDryRunClassification(result *importResultJSON, classification *ImportResult) {
result.Created = classification.Created
result.Updated = classification.Updated
result.Unchanged = classification.Unchanged
result.Skipped += classification.Skipped
result.IDs = append(result.IDs, classification.ImportedIDs...)
result.StaleSkippedIDs = classification.StaleSkippedIDs
result.UpdatedIssues = classification.UpdatedIssues
result.TieKeptLocalIDs = classification.TieKeptLocalIDs
}
// applyImportOutcome folds a real import's outcome into the command's JSON
// result, identically in both modes.
func applyImportOutcome(result *importResultJSON, importResult *ImportResult) {
result.Created = importResult.Created
result.Updated = importResult.Updated
result.Unchanged = importResult.Unchanged
result.Skipped += importResult.Skipped
result.SkippedDependencies = append(result.SkippedDependencies, importResult.SkippedDependencies...)
result.IDs = append(result.IDs, importResult.ImportedIDs...)
result.UpdatedIssues = append(result.UpdatedIssues, importResult.UpdatedIssues...)
result.TieKeptLocalIDs = append(result.TieKeptLocalIDs, importResult.TieKeptLocalIDs...)
result.StaleSkippedIDs = append(result.StaleSkippedIDs, importResult.StaleSkippedIDs...)
}
// renderImportDryRun reports a dry run (JSON or stderr), identically in both
// modes.
func renderImportDryRun(result importResultJSON, memoriesCount int, source string, dedupHits int) error {
if jsonOutput {
return outputJSON(result)
}
// The leading count is the sum of the breakdown that follows it
// (not len(issues)), which can be larger when rows were stale
// skipped — those are reported separately below instead of being
// folded into a total the breakdown then wouldn't add up to.
considered := result.Created + result.Updated + result.Unchanged
//nolint:gosec // G705: stderr, not a browser context
fmt.Fprintf(os.Stderr, "Would import %d issues (%d new, %d updated, %d unchanged) and %d memories from %s",
considered, result.Created, result.Updated, result.Unchanged, memoriesCount, source)
if dedupHits > 0 {
fmt.Fprintf(os.Stderr, " (%d duplicates skipped)", dedupHits) //nolint:gosec // G705: stderr, not a browser context
}
if len(result.StaleSkippedIDs) > 0 {
fmt.Fprintf(os.Stderr, " (%d stale skipped)", len(result.StaleSkippedIDs))
}
fmt.Fprintln(os.Stderr)
return nil
}
// renderImportOutcome reports a completed import (JSON or stderr),
// identically in both modes.
func renderImportOutcome(result importResultJSON, source string, dedupHits int) error {
if jsonOutput {
return outputJSON(result)
}
fmt.Fprintf(os.Stderr, "Imported %d issues", result.Created)
if result.Memories > 0 {
fmt.Fprintf(os.Stderr, " and %d memories", result.Memories)
}
fmt.Fprintf(os.Stderr, " from %s", source)
if dedupHits > 0 {
fmt.Fprintf(os.Stderr, " (%d duplicates skipped)", dedupHits) //nolint:gosec // G705: stderr, not a browser context
}
if staleSkipped := result.Skipped - dedupHits; staleSkipped > 0 {
fmt.Fprintf(os.Stderr, " (%d stale skipped; use --allow-stale to restore older rows)", staleSkipped) //nolint:gosec // G705: stderr, not a browser context
}
if result.Unchanged > 0 {
fmt.Fprintf(os.Stderr, " (%d already present, unchanged)", result.Unchanged) //nolint:gosec // G705: stderr, not a browser context
}
fmt.Fprintln(os.Stderr)
if len(result.UpdatedIssues) > 0 {
fmt.Fprintf(os.Stderr, "Updated %d existing issue(s):\n", len(result.UpdatedIssues))
for _, change := range result.UpdatedIssues {
fmt.Fprintf(os.Stderr, " %s: %s\n", change.ID, change.Changes)
}
}
if len(result.TieKeptLocalIDs) > 0 {
fmt.Fprintf(os.Stderr, "Kept local state for %d issue(s) with the same updated_at but different content (use --allow-stale to overwrite): %s\n",
len(result.TieKeptLocalIDs), strings.Join(result.TieKeptLocalIDs, ", "))
}
for _, skipped := range result.SkippedDependencies {
fmt.Fprintf(os.Stderr, "Skipped dependency: %s\n", skipped)
}
return nil
}
// importTitleSearcher is the read seam the --dedup filter needs. It lives in
// THIS file because naming types.IssueFilter is denied by default under
// cmd/bd and import.go is the named exception for the bulk-movement family
// (.golangci.yml, forbidigo): the classic storage.DoltStorage satisfies it
// directly, and uowImportTitleSearcher adapts the proxied unit of work.
type importTitleSearcher interface {
SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error)
}
// uowImportTitleSearcher adapts a unit of work's issue use case to the
// classic []*types.Issue search shape filterDuplicatesByTitle consumes. Both
// stacks run the same issueops search underneath (issues merged with wisps),
// so --dedup sees the same title universe in both modes.
type uowImportTitleSearcher struct {
uw uow.UnitOfWork
}
func (s uowImportTitleSearcher) SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) {
page, err := s.uw.IssueUseCase().SearchIssues(ctx, query, filter)
if err != nil {
return nil, err
}
return page.Items, nil
}
// filterDuplicatesByTitle removes issues whose title matches an existing open issue.
func filterDuplicatesByTitle(ctx context.Context, st importTitleSearcher, issues []*types.Issue) ([]*types.Issue, int) {
existing, err := st.SearchIssues(ctx, "", types.IssueFilter{})
if err != nil {
return issues, 0
}
titleSet := make(map[string]bool, len(existing))
for _, issue := range existing {
if issue.Status != types.StatusClosed {
titleSet[strings.ToLower(issue.Title)] = true
}
}
var kept []*types.Issue
skipped := 0
for _, issue := range issues {
if titleSet[strings.ToLower(issue.Title)] {
skipped++
continue
}
kept = append(kept, issue)
}
return kept, skipped
}