* fix(update): fence unforced --notes overwrites behind --force across all surfaces bd update --notes silently replaced existing notes, destroying handoff history. The contract now refuses an unforced overwrite of non-empty notes with ErrNotesOverwrite (409 notes_overwrite_refused over HTTP, force_notes_overwrite to override), enforced inside the mutation transaction on both the dolt and uow backends, for single updates and batch apply alike. CAS guards keep precedence: a stale --if-assignee/--if-status still surfaces as the exit-13 mismatch, on both backends, before the fence speaks. CLI: --force and --if-assignee are no longer mutually exclusive (that exclusion made overwriting under a guard impossible); both update routes translate the fence's sentinel into the same actionable advice (--force to overwrite, --append-notes to preserve history), and the post---force warning is worded as an audit statement. The shipped plugin skill docs and bd note help now prescribe --append-notes for handoffs instead of the newly fenced bare --notes. The fence is the single enforcement point: no client-side pre-read copies of the rule. Conformance contracts, unit tests for AuthorizeNotesOverwrite, and embedded/proxied/httpapi regression coverage included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MAFpn9pUw8WKMfymteZHbh * fix(update): exempt explicit clears from the notes fence; share its predicate CI surfaced three gaps in the fence's reach: - Clearing notes (--notes "" / patch.notes: "") was refused, diverging from every sibling text field's clear and from the backend behavior contracts. The fence now targets only its actual footgun — replacing non-empty notes with different non-empty content — via a single shared predicate, issueops.NotesReplacement, used by both the contract's AuthorizeNotesOverwrite and the CLI's warning path, so the two can no longer drift. Prose updated to match everywhere the rule is stated (ErrNotesOverwrite, ForceNotesOverwrite, OpenAPI spec + regenerated types). - The revision-CAS integration tests' incidental concurrent writes now patch design instead of notes, so they no longer depend on notes-fence semantics at all. - The historical-upgrade harness deliberately replaces fixture notes and asserts the new value; its two bd update --notes calls now pass --force. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MAFpn9pUw8WKMfymteZHbh * fix(update): address review — changelog BREAKING entry, problem.go doc block, close-policy-under-guard test Review follow-ups on #5946 (bee-ghosttrack, MERGE-AFTER-FIXES): - CHANGELOG: BREAKING entry under [Unreleased] ### Changed for the notes fence on both wires (CLI exit 1, HTTP 409 notes_overwrite_refused), with the --force / --append-notes migration line. It also names the two deliberate edges: the explicit-clear exemption, and --force widening to the close-policy half under --if-assignee now that the flag pair is legal. - problem.go: CodeNotesOverwrite's comment had landed inside CodeDependencyCycle's doc block; each const now sits under its own docs. - assign_fence_embedded_test.go: pin the previously untested close-policy half of --force under --if-assignee (refused unforced, authorized forced). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED * test(update): align TestWarnNotesReplacement with the post-fence warning copy The test landed on main after this branch was cut and pins the pre-fence wording, where the warning fired on any unforced overwrite. Since the fence, an unforced overwrite is refused outright and warnNotesReplacement fires only after a forced one — the audit trail for a habitual --force — so the copy it pins is now '--force replaced existing notes'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED * docs(import): record bd import's notes-fence exemption as by-design (GH#6190) Import consults none of the per-field live-state fences — notes overwrite, live-claim reassign, close policy — and that is the intended shape, not an omission: its contract is wholesale row replacement, guarded by its own staleness check (only strictly-newer rows rewrite local state, --allow-stale is the single deliberate-overwrite opt-in) and reported field-by-field in updated_issues. Named in AuthorizeNotesOverwrite's enforcement-boundary comment and in bd import's help; the published CLI docs pick the help text up at the next release pin bump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED * fix(update): refuse --notes "" and add --clear-notes; stop --notes help reading as additive GH#6021: an empty --notes wiped the whole notes field at exit 0 behind the same ✓ Updated receipt as a successful write — and the empty string is exactly what a dead command substitution collapses to. The empty value is now refused on both CLI routes (the accident and the deliberate clear are the same bytes at the flag layer), and the deliberate clear gets its own verb, --clear-notes, mutually exclusive with --notes/--append-notes. A verb rather than the proposed --allow-empty-notes opt-in: what needs authorizing is an intent, not an input path — --allow-empty-description unblocks stdin/file plumbing, which notes does not have — and an opt-in passed habitually would silently disarm the guard while a habitual --clear-notes fails loudly. --force does not bypass the refusal; the fence's clear exemption stands, so the verb alone authorizes the erase. The HTTP API is unchanged: patch.notes "" still clears. GH#6272: bd update --help called --notes "Additional notes" and the prime text said "Add supplementary notes" — both read as appending, so agents following the injected guidance faithfully clobbered prior context. The update help now names the operation with the --append-notes contrast at the point of use, the shared/create help is field-named like --description and --design, and the prime line points at bd update --append-notes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JyuQ6Tjv3D7tpNqeb2WDED * fix(bazel): enumerate the PR-new files in cmd/bd and issueops srcs The base added cmd/bd/BUILD.bazel and internal/storage/issueops/BUILD.bazel with EXPLICIT srcs lists after this branch was cut, so the three files this PR adds were never enumerated: cmd/bd/update_notes_guard.go (go_library bd_lib) cmd/bd/update_notes_guard_test.go (go_test bd_test) internal/storage/issueops/aggregate_authorize_notes_test.go (go_test) The go_library omission is a hard build break, not a silent skip: update.go:215,220 and update_input.go:106,111 are enumerated and call validateNotesUpdate / clearNotesRequested, so //cmd/bd:bd_lib compiles the callers without the callee. bazel.yml arms unconditionally for fork PRs (head.repo.fork == true) and builds //cmd/bd:bd + //cmd/bd:bd_test, so this reds the lane. make bazel-sync-check cannot see it -- go_srcs.py only validates the glob-based go_srcs filegroup, never the rule srcs lists. Generated by make bazel-sync (gazelle), not hand-edited, so the workflow's own BUILD-files-in-sync drift step agrees byte-for-byte. * fix(notes): name --clear-notes in bd note's rm guidance, correct the fence boundary comment, pin both request-shape rules Takes all three non-gating minors from the review. No behavior change: two prose fixes and two test cases. 1. cmd/bd/note.go's rm|remove|delete branch advised `bd update <id> --notes "text" --force` under "To change or clear notes", but clearing through --notes is exactly what this PR made impossible -- validateNotesUpdate refuses an empty value unconditionally and --force does not reach it. The branch now names both verbs separately: --notes --force to replace, --clear-notes to clear. The edit and update branches speak only of replacing and were already correct, so they are untouched. 2. aggregate.go's enforcement-boundary comment claimed the UpdateIssueInTx funnel's "one notes-replacing caller is bd edit". Under this PR's own NotesReplacement predicate there are at least four more (bd note's read-combine-write, bd defer --reason, bd rename and delete-tombstoning's mechanical ID rewrites, bd compact's clears). The exemption is still sound -- every funnel writer derives its new value from the current notes -- so the comment now names those CLASSES instead of a single caller, which is the claim that is actually true and that a future fence auditor can check. 3. Both branches this PR adds to ValidateUpdateRequest were uncovered: the batch apply leg reaches them here and nowhere else, while the single-issue HTTP handler's own pre-check returns before storage. Added a sibling table (the existing one varies only the patch; these two rules are properties of the REQUEST) with a case each. Mutation-checked, disjoint: neutralizing the force-without-notes branch reds only the first case, neutralizing the notes+append branch reds only the second, and the pre-existing canonical-fields table stays green under both. Not an amend of 5ffcc9c16c: that commit's message states it was generated by make bazel-sync and not hand-edited, so folding hand-written Go into it would falsify the claim its own BUILD-files-in-sync argument rests on. No new files, so the srcs lists it repaired stay correct. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
627 lines
26 KiB
Go
627 lines
26 KiB
Go
package main
|
|
|
|
import (
|
|
"bufio"
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"io"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/spf13/cobra"
|
|
"github.com/steveyegge/beads/internal/beads"
|
|
"github.com/steveyegge/beads/internal/config"
|
|
"github.com/steveyegge/beads/internal/metrics"
|
|
"github.com/steveyegge/beads/internal/storage/uow"
|
|
"github.com/steveyegge/beads/internal/types"
|
|
)
|
|
|
|
var importCmd = &cobra.Command{
|
|
Use: "import [file|-]",
|
|
Short: "Import issues from a JSONL file or stdin into the database",
|
|
Long: `Import issues from a JSONL file (newline-delimited JSON) into the database.
|
|
|
|
If no file is specified, imports from the configured import.path under .beads/
|
|
(default: issues.jsonl). Use "-" to read from stdin; redirecting stdin without
|
|
"-" or a file argument is an error, so a typo'd 'bd import < file' cannot
|
|
silently import the default file instead. This is the incremental counterpart to
|
|
'bd export': new issues are created and existing issues are updated (upsert
|
|
semantics).
|
|
|
|
Memory records (lines with "_type":"memory") are automatically detected and
|
|
imported as persistent memories (equivalent to 'bd remember'). This makes
|
|
'bd export | bd import' a full round-trip for both issues and memories.
|
|
|
|
Each JSONL line should map to an issue. The importer accepts every field
|
|
'bd export' emits — see 'bd export' output for the canonical schema. Only
|
|
"title" is required; everything else is optional.
|
|
|
|
Common fields:
|
|
title Required. Short summary.
|
|
description Long-form body.
|
|
design, notes, Additional content sections.
|
|
acceptance_criteria
|
|
issue_type bug | feature | task | epic | chore | ...
|
|
priority 0-4 (0 = critical). 0 is preserved (no omitempty).
|
|
status open | in_progress | blocked | closed | ...
|
|
(rows with status "tombstone" are skipped)
|
|
assignee, owner, Ownership metadata.
|
|
created_by
|
|
labels Array of strings.
|
|
dependencies Array of {issue_id, depends_on_id, type, ...}.
|
|
comments Array of comment objects.
|
|
external_ref, Cross-system identifiers (e.g. "gh-9").
|
|
source_system
|
|
due_at, defer_until RFC3339 timestamps for scheduling.
|
|
metadata Arbitrary JSON object preserved verbatim.
|
|
|
|
Timestamps (created_at, updated_at, started_at, closed_at) are preserved
|
|
when present in the JSONL and otherwise filled in by the importer. The
|
|
legacy "wisp" boolean is accepted as an alias for "ephemeral".
|
|
|
|
By default a row only rewrites an existing local issue when its
|
|
updated_at is strictly newer. Older rows are skipped (reported as
|
|
stale_skipped_ids) and rows with the same updated_at keep every local
|
|
column — updated_at has second granularity, so a timestamp tie can be
|
|
two distinct same-second updates, and the local row wins the tie
|
|
(reported as tie_kept_local_ids; the row's labels/comments/dependencies
|
|
still merge). The guard is also enforced inside the upsert itself, so a
|
|
local update that lands while the import is running is preserved rather
|
|
than overwritten. Existing issues that the import did rewrite are listed
|
|
with a field-level summary (updated_issues), so local state changed by
|
|
an import is visible. To deliberately restore an older snapshot, pass
|
|
--allow-stale, which imports every row even when it overwrites newer
|
|
local state.
|
|
|
|
The staleness guard is import's ONLY overwrite guard: the per-field
|
|
fences that bd update enforces (notes overwrite, live-claim reassign,
|
|
close policy) do not apply to imported rows. Import replaces rows
|
|
wholesale by design — a restore must not fail field-by-field — so
|
|
--allow-stale is the single deliberate-overwrite opt-in, and
|
|
updated_issues is where to look for what an import rewrote.
|
|
|
|
Large imports are written in bounded transactions (a few hundred issues
|
|
each, with a short pause between commits) with progress on stderr, so
|
|
concurrent bd commands keep working while the import runs instead of
|
|
stalling on one batch-wide write lock. Rows land in dependency order
|
|
with their blocking edges in the same transaction, so a half-finished
|
|
import never shows a blocked issue as ready. If an import fails partway,
|
|
the already-committed chunks are durable and the command exits nonzero;
|
|
re-running the same import is safe and converges (rows upsert,
|
|
labels/comments/dependencies deduplicate).
|
|
|
|
EXAMPLES:
|
|
bd import # Import from configured import.path
|
|
bd import backup.jsonl # Import from a specific file
|
|
bd import -i backup.jsonl # Legacy alias for a specific file
|
|
bd import - # Read JSONL from stdin
|
|
cat issues.jsonl | bd import - # Pipe JSONL from another tool
|
|
bd import --dry-run # Show what would be imported
|
|
bd import --dedup # Skip issues with duplicate titles
|
|
bd import --allow-stale old.jsonl # Restore an older snapshot (overwrites newer local rows)
|
|
bd import --json # Structured output with created and skipped IDs`,
|
|
GroupID: "sync",
|
|
SilenceUsage: true,
|
|
SilenceErrors: true,
|
|
RunE: runImport,
|
|
}
|
|
|
|
var (
|
|
importDryRun bool
|
|
importDedup bool
|
|
importAllowStale bool
|
|
importInput string
|
|
)
|
|
|
|
func init() {
|
|
importCmd.Flags().StringVarP(&importInput, "input", "i", "", "Read JSONL from a specific file")
|
|
importCmd.Flags().BoolVar(&importDryRun, "dry-run", false, "Show what would be imported without importing")
|
|
importCmd.Flags().BoolVar(&importDedup, "dedup", false, "Skip lines whose title matches an existing open issue")
|
|
importCmd.Flags().BoolVar(&importAllowStale, "allow-stale", false, "Import rows even when older than the local issue (required to restore an older snapshot)")
|
|
rootCmd.AddCommand(importCmd)
|
|
}
|
|
|
|
// importPoolReadTimeout is the per-I/O read deadline a `bd import` gets on
|
|
// its shared-pool connections when neither the caller, BEADS_DOLT_POOL_READ_TIMEOUT
|
|
// nor dolt.pool-read-timeout set one. The pool default (10s) is a fast-fail
|
|
// tuned for interactive commands; an import's chunk COMMIT of 250 rows with
|
|
// their aux tables legitimately outlives it whenever the server pauses — a
|
|
// stock dolt sql-server's auto_gc took 16.5s mid-import in the wy-9we0jf
|
|
// rollback drill, and every such pause surfaced as "i/o timeout" followed by
|
|
// "write commit result indeterminate" (wy-sbgucn). 5m matches the repo's
|
|
// other known-long operations (execWithLongTimeout, withReadTxLongTimeout).
|
|
const importPoolReadTimeout = 5 * time.Minute
|
|
|
|
// bulkLoadPoolReadTimeout returns the pool read-timeout fallback for cmd: the
|
|
// bulk-load deadline for `bd import`, zero (keep the pool default) otherwise.
|
|
// It is a FALLBACK, not an override — an operator's explicit setting still wins
|
|
// (see dolt.Config.PoolReadTimeoutFallback).
|
|
func bulkLoadPoolReadTimeout(cmd *cobra.Command) time.Duration {
|
|
if cmd != nil && cmd.Name() == "import" {
|
|
return importPoolReadTimeout
|
|
}
|
|
return 0
|
|
}
|
|
|
|
func runImport(cmd *cobra.Command, args []string) error {
|
|
// Explicit call, not inherited from CheckReadonly: runImport doesn't call
|
|
// CheckReadonly at all (a separate, pre-existing gap — readonlyMode
|
|
// doesn't gate bd import either), so it can't pick up the freeze check
|
|
// folded into CheckReadonly the way create/update/close/remember do.
|
|
// The path that makes this call load-bearing rather than redundant is
|
|
// `bd import --dry-run`: a preview sets useReadOnly, so it skips the early
|
|
// gate in PersistentPreRunE, and runImport is the only chokepoint left.
|
|
// Plain `bd import` is already stopped by that early gate.
|
|
if err := migrationFreezeGateFor(cmd, "import"); err != nil {
|
|
return err
|
|
}
|
|
|
|
evt := metrics.NewCommandEvent("import")
|
|
defer func() {
|
|
if c := metrics.Global(); c != nil {
|
|
c.CloseEventAndAdd(evt)
|
|
}
|
|
}()
|
|
|
|
if err := runImportInner(args); err != nil {
|
|
if _, isExit := err.(*exitError); isExit {
|
|
return err
|
|
}
|
|
return HandleErrorRespectJSON("%v", err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func runImportInner(args []string) error {
|
|
ctx := rootCtx
|
|
if importInput != "" && len(args) > 0 {
|
|
return fmt.Errorf("use either --input or a positional file, not both")
|
|
}
|
|
|
|
fromStdin := importInput == "-" || (len(args) > 0 && args[0] == "-")
|
|
|
|
if fromStdin {
|
|
return runImportFromReader(ctx, os.Stdin, "stdin")
|
|
}
|
|
|
|
// Determine source file
|
|
var jsonlPath string
|
|
if importInput != "" {
|
|
jsonlPath = importInput
|
|
} else if len(args) > 0 {
|
|
jsonlPath = args[0]
|
|
} else {
|
|
// bd-axluy: `bd import < file` (or `... | bd import`) without "-"
|
|
// used to silently ignore stdin and import the default JSONL — a
|
|
// mutating command diverging from what the user piped. Demand an
|
|
// explicit source instead. /dev/null (the stdin subprocesses get by
|
|
// default) is a character device, so scripted bare `bd import` with
|
|
// no redirection still works.
|
|
if fi, statErr := os.Stdin.Stat(); statErr == nil && fi.Mode()&os.ModeCharDevice == 0 {
|
|
return fmt.Errorf("stdin is redirected, but without \"-\" bd import ignores it and imports the default JSONL instead; use 'bd import -' to import what you piped, or name a file explicitly")
|
|
}
|
|
beadsDir := beads.FindBeadsDir()
|
|
if beadsDir == "" {
|
|
return fmt.Errorf("%s — %s", activeWorkspaceNotFoundError(), diagHint())
|
|
}
|
|
if globalFlag {
|
|
jsonlPath = filepath.Join(beadsDir, "global-issues.jsonl")
|
|
} else {
|
|
jsonlPath = configuredImportJSONLPath(beadsDir)
|
|
}
|
|
}
|
|
|
|
info, err := os.Stat(jsonlPath)
|
|
if err != nil {
|
|
return fmt.Errorf("cannot read %s: %w", jsonlPath, err)
|
|
}
|
|
if info.Size() == 0 {
|
|
if jsonOutput {
|
|
return outputJSON(importResultJSON{Source: jsonlPath})
|
|
}
|
|
fmt.Fprintf(os.Stderr, "Empty file: %s\n", jsonlPath)
|
|
return nil
|
|
}
|
|
|
|
f, err := os.Open(jsonlPath) //nolint:gosec // G304: CLI argument
|
|
if err != nil {
|
|
return fmt.Errorf("cannot open %s: %w", jsonlPath, err)
|
|
}
|
|
defer f.Close()
|
|
|
|
return runImportFromReader(ctx, f, jsonlPath)
|
|
}
|
|
|
|
type importResultJSON struct {
|
|
Source string `json:"source"`
|
|
Created int `json:"created"`
|
|
Updated int `json:"updated,omitempty"`
|
|
Unchanged int `json:"unchanged,omitempty"`
|
|
Skipped int `json:"skipped"`
|
|
DedupHits int `json:"dedup_skipped,omitempty"`
|
|
Memories int `json:"memories,omitempty"`
|
|
IDs []string `json:"ids,omitempty"`
|
|
UpdatedIssues []ImportChange `json:"updated_issues,omitempty"`
|
|
TieKeptLocalIDs []string `json:"tie_kept_local_ids,omitempty"`
|
|
StaleSkippedIDs []string `json:"stale_skipped_ids,omitempty"`
|
|
SkippedDependencies []string `json:"skipped_dependencies,omitempty"`
|
|
DryRun bool `json:"dry_run,omitempty"`
|
|
}
|
|
|
|
func runImportFromReader(ctx context.Context, r io.Reader, source string) error {
|
|
issues, memories, err := parseImportRecords(r)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
if usesProxiedServer() {
|
|
return runImportRecordsProxied(ctx, issues, memories, source)
|
|
}
|
|
|
|
if store == nil {
|
|
return fmt.Errorf("no database — run 'bd init' or 'bd bootstrap' first")
|
|
}
|
|
return runImportRecordsClassic(ctx, issues, memories, source)
|
|
}
|
|
|
|
// parseImportRecords scans one JSONL stream into issue rows and memory
|
|
// records — the `bd import` / `bd import -` parse loop, shared by the classic
|
|
// and proxied modes. Same record vocabulary as parseJSONLFile (the bootstrap
|
|
// reader): the optional _schema header and tombstones are skipped, and the
|
|
// "wisp_plane" boolean is honored as the explicit wisps-plane marker (and
|
|
// the legacy "wisp" alias for "ephemeral") via applyImportWispPlane.
|
|
func parseImportRecords(r io.Reader) ([]*types.Issue, []memoryRecord, error) {
|
|
scanner := bufio.NewScanner(r)
|
|
scanner.Buffer(make([]byte, 0, 1024*1024), 64*1024*1024)
|
|
|
|
var issues []*types.Issue
|
|
var memories []memoryRecord
|
|
|
|
for scanner.Scan() {
|
|
line := scanner.Text()
|
|
if line == "" {
|
|
continue
|
|
}
|
|
|
|
var peek map[string]json.RawMessage
|
|
if err := json.Unmarshal([]byte(line), &peek); err != nil {
|
|
return nil, nil, fmt.Errorf("failed to parse JSONL line: %w", err)
|
|
}
|
|
|
|
// Skip the optional beads-jsonl header record (§J1.3). A canonical
|
|
// export may prepend a provenance line, e.g.
|
|
// {"_schema":"beads-jsonl/1","_dolt_branch":"main","_sort":"stable-v1"}.
|
|
// It carries no _type and no issue fields; without this guard it falls
|
|
// through to the issue path, unmarshals into an empty Issue, and aborts
|
|
// the whole import with "title is required". parseJSONLFile (the
|
|
// bootstrap reader) has always skipped it; this loop — the one `bd
|
|
// import` and `bd import -` run through — did not.
|
|
if _, isHeader := peek["_schema"]; isHeader {
|
|
continue
|
|
}
|
|
|
|
if rawType, ok := peek["_type"]; ok {
|
|
var typeStr string
|
|
if err := json.Unmarshal(rawType, &typeStr); err == nil && typeStr == "memory" {
|
|
var mem memoryRecord
|
|
if err := json.Unmarshal([]byte(line), &mem); err != nil {
|
|
return nil, nil, fmt.Errorf("failed to parse memory record: %w", err)
|
|
}
|
|
if mem.Key != "" && mem.Value != "" {
|
|
memories = append(memories, mem)
|
|
}
|
|
continue
|
|
}
|
|
}
|
|
|
|
var issue types.Issue
|
|
if err := json.Unmarshal([]byte(line), &issue); err != nil {
|
|
return nil, nil, fmt.Errorf("failed to parse issue from JSONL: %w", err)
|
|
}
|
|
if issue.Status == "tombstone" {
|
|
continue
|
|
}
|
|
applyImportWispPlane(peek, &issue)
|
|
issue.SetDefaults()
|
|
issues = append(issues, &issue)
|
|
}
|
|
if err := scanner.Err(); err != nil {
|
|
return nil, nil, fmt.Errorf("failed to scan JSONL: %w", err)
|
|
}
|
|
return issues, memories, nil
|
|
}
|
|
|
|
// runImportRecordsClassic is the classic (embedded/direct store) import
|
|
// pipeline over the parsed records: dedup, dry-run classification, memory
|
|
// writes, the batch issue import, the final commit and the issue_prefix
|
|
// reconciliation.
|
|
func runImportRecordsClassic(ctx context.Context, issues []*types.Issue, memories []memoryRecord, source string) error {
|
|
// Dedup: skip issues whose title matches an existing open issue
|
|
dedupHits := 0
|
|
if importDedup && len(issues) > 0 {
|
|
issues, dedupHits = filterDuplicatesByTitle(ctx, store, issues)
|
|
}
|
|
|
|
result := importResultJSON{
|
|
Source: source,
|
|
DedupHits: dedupHits,
|
|
DryRun: importDryRun,
|
|
}
|
|
|
|
if importDryRun {
|
|
result.Memories = len(memories)
|
|
result.Skipped = dedupHits
|
|
|
|
classification, err := classifyDryRunImport(ctx, store, issues, importAllowStale)
|
|
if err != nil {
|
|
return fmt.Errorf("dry-run: %w", err)
|
|
}
|
|
applyImportDryRunClassification(&result, classification)
|
|
return renderImportDryRun(result, len(memories), source, dedupHits)
|
|
}
|
|
|
|
// ONE value for both prefix writers on this path — the seed just below and
|
|
// the post-import sync at the bottom. Resolved once so they cannot disagree
|
|
// on trimming or on validation; see importSyncPrefix.
|
|
syncPrefix := importSyncPrefix()
|
|
|
|
// Seed issue_prefix from config.yaml before the config table has one, so
|
|
// NewBatchContext's ReadConfigPrefix (below, via importIssuesCore) does
|
|
// not reject an externally-provisioned database that config.yaml already
|
|
// names a prefix for.
|
|
//
|
|
// It runs BEFORE the memory writes below, deliberately: CommitWithConfig
|
|
// is DOLT_COMMIT -Am, so it stages everything dirty at that instant.
|
|
// Seeding first is what keeps the prefix the only thing this commit can
|
|
// carry — otherwise this run's kv.memory.* rows land in a commit named for
|
|
// the prefix seed, and stay permanently committed when the import below
|
|
// then fails, instead of being left in the working set for the operator to
|
|
// discard.
|
|
//
|
|
// The commit lands HERE, before an import that may fail, which means a
|
|
// failed `bd import` can leave the seeded prefix behind. Accepted: the
|
|
// value is config.yaml's own and is the converged one either way, so the
|
|
// residue is the state the next successful import would reach. Deferring
|
|
// it to the post-import CommitWithConfig below is NOT an alternative —
|
|
// that block only fires when dbPrefix != syncPrefix, which a seed makes
|
|
// false, so the row would stay uncommitted (store.Commit excludes config,
|
|
// GH#2455) and a dirty internal config key then blocks the next pull.
|
|
if len(issues) > 0 && syncPrefix != "" {
|
|
if dbPrefix, _ := store.GetConfig(ctx, "issue_prefix"); dbPrefix == "" {
|
|
if setErr := store.SetConfig(ctx, "issue_prefix", syncPrefix); setErr != nil {
|
|
fmt.Fprintf(os.Stderr, "warning: failed to seed issue_prefix from config.yaml: %v\n", setErr)
|
|
} else if commitErr := store.CommitWithConfig(ctx, "bd import: seed issue_prefix from config.yaml"); commitErr != nil {
|
|
fmt.Fprintf(os.Stderr, "warning: failed to commit seeded issue_prefix: %v\n", commitErr)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Import memories
|
|
for _, mem := range memories {
|
|
storageKey := kvPrefix + memoryPrefix + mem.Key
|
|
if err := store.SetConfig(ctx, storageKey, mem.Value); err != nil {
|
|
return fmt.Errorf("failed to import memory %q: %w", mem.Key, err)
|
|
}
|
|
result.Memories++
|
|
}
|
|
|
|
// Import issues
|
|
if len(issues) > 0 {
|
|
opts := ImportOptions{SkipPrefixValidation: true, AllowStale: importAllowStale}
|
|
importResult, err := importIssuesCore(ctx, "", store, issues, opts)
|
|
if err != nil {
|
|
return fmt.Errorf("import failed: %w", err)
|
|
}
|
|
applyImportOutcome(&result, importResult)
|
|
}
|
|
|
|
if result.Created > 0 || result.Memories > 0 {
|
|
commitMsg := fmt.Sprintf("bd import: %d issues", result.Created)
|
|
if result.Memories > 0 {
|
|
commitMsg += fmt.Sprintf(", %d memories", result.Memories)
|
|
}
|
|
commitMsg += fmt.Sprintf(" from %s", filepath.Base(source))
|
|
if err := store.Commit(ctx, commitMsg); err != nil {
|
|
// An import can be a working-set no-op: re-importing an
|
|
// identical snapshot, or equal-timestamp rows whose guarded
|
|
// upsert kept every local column (bd-hj85c).
|
|
if !strings.Contains(err.Error(), "nothing to commit") {
|
|
return fmt.Errorf("commit: %w", err)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Sync issue_prefix from config.yaml to the database if stale (be-llaf).
|
|
// store.Commit skips the config table (GH#2455), so we use CommitWithConfig
|
|
// for this intentional config update after the issues commit completes.
|
|
// config.yaml is authoritative here and existing issue IDs are intentionally
|
|
// left unchanged: this deliberately bypasses the `bd config set issue_prefix`
|
|
// guard for the import/migration flow and is not a rename. Reconciles
|
|
// against the SAME resolved value the seed used, so a padded config.yaml
|
|
// prefix cannot make this test true forever and overwrite the trimmed
|
|
// value the seed stored on every import.
|
|
if syncPrefix != "" {
|
|
if dbPrefix, _ := store.GetConfig(ctx, "issue_prefix"); dbPrefix != syncPrefix {
|
|
if setErr := store.SetConfig(ctx, "issue_prefix", syncPrefix); setErr == nil {
|
|
_ = store.CommitWithConfig(ctx, "bd import: sync issue_prefix from config.yaml")
|
|
}
|
|
}
|
|
}
|
|
|
|
return renderImportOutcome(result, source, dedupHits)
|
|
}
|
|
|
|
// importSyncPrefix resolves the ONE config.yaml issue_prefix value EVERY
|
|
// import writer uses: the classic path's seed (before the batch needs it) and
|
|
// its post-import sync (after the batch lands), and the SyncIssuePrefix the
|
|
// proxied path hands ImportBatch (import_proxied_server.go), which seeds and
|
|
// syncs inside its one transaction. "" means "reconcile nothing", which is
|
|
// what --global, an absent value, and an invalid value all resolve to.
|
|
//
|
|
// Resolving once is what keeps those writers honest: the seed stored the
|
|
// TRIMMED value while the sync used to compare the raw one, so a padded
|
|
// `issue-prefix: " bd"` made them disagree and the sync overwrote the seed's
|
|
// value; and the sync used to skip validatePrefix entirely, so a prefix the
|
|
// seed refused still reached the database through it.
|
|
func importSyncPrefix() string {
|
|
// --global: config.yaml is a per-project file, and the shared global
|
|
// store's own prefix must win (selectCreateIDPrefix), not whatever project
|
|
// happened to import into it.
|
|
if globalFlag {
|
|
return ""
|
|
}
|
|
yamlPrefix := strings.TrimSpace(config.GetString("issue-prefix"))
|
|
if yamlPrefix == "" {
|
|
return ""
|
|
}
|
|
if err := validatePrefix(yamlPrefix); err != nil {
|
|
// Naming both effects: an invalid value disables the sync as well as
|
|
// the seed, and base did sync an invalid prefix through.
|
|
fmt.Fprintf(os.Stderr, "warning: ignoring invalid issue-prefix %q from config.yaml (not seeding or syncing issue_prefix): %v\n", yamlPrefix, err)
|
|
return ""
|
|
}
|
|
return yamlPrefix
|
|
}
|
|
|
|
// applyImportDryRunClassification folds a dry-run classification into the
|
|
// command's JSON result, identically in both modes.
|
|
func applyImportDryRunClassification(result *importResultJSON, classification *ImportResult) {
|
|
result.Created = classification.Created
|
|
result.Updated = classification.Updated
|
|
result.Unchanged = classification.Unchanged
|
|
result.Skipped += classification.Skipped
|
|
result.IDs = append(result.IDs, classification.ImportedIDs...)
|
|
result.StaleSkippedIDs = classification.StaleSkippedIDs
|
|
result.UpdatedIssues = classification.UpdatedIssues
|
|
result.TieKeptLocalIDs = classification.TieKeptLocalIDs
|
|
}
|
|
|
|
// applyImportOutcome folds a real import's outcome into the command's JSON
|
|
// result, identically in both modes.
|
|
func applyImportOutcome(result *importResultJSON, importResult *ImportResult) {
|
|
result.Created = importResult.Created
|
|
result.Updated = importResult.Updated
|
|
result.Unchanged = importResult.Unchanged
|
|
result.Skipped += importResult.Skipped
|
|
result.SkippedDependencies = append(result.SkippedDependencies, importResult.SkippedDependencies...)
|
|
result.IDs = append(result.IDs, importResult.ImportedIDs...)
|
|
result.UpdatedIssues = append(result.UpdatedIssues, importResult.UpdatedIssues...)
|
|
result.TieKeptLocalIDs = append(result.TieKeptLocalIDs, importResult.TieKeptLocalIDs...)
|
|
result.StaleSkippedIDs = append(result.StaleSkippedIDs, importResult.StaleSkippedIDs...)
|
|
}
|
|
|
|
// renderImportDryRun reports a dry run (JSON or stderr), identically in both
|
|
// modes.
|
|
func renderImportDryRun(result importResultJSON, memoriesCount int, source string, dedupHits int) error {
|
|
if jsonOutput {
|
|
return outputJSON(result)
|
|
}
|
|
// The leading count is the sum of the breakdown that follows it
|
|
// (not len(issues)), which can be larger when rows were stale
|
|
// skipped — those are reported separately below instead of being
|
|
// folded into a total the breakdown then wouldn't add up to.
|
|
considered := result.Created + result.Updated + result.Unchanged
|
|
//nolint:gosec // G705: stderr, not a browser context
|
|
fmt.Fprintf(os.Stderr, "Would import %d issues (%d new, %d updated, %d unchanged) and %d memories from %s",
|
|
considered, result.Created, result.Updated, result.Unchanged, memoriesCount, source)
|
|
if dedupHits > 0 {
|
|
fmt.Fprintf(os.Stderr, " (%d duplicates skipped)", dedupHits) //nolint:gosec // G705: stderr, not a browser context
|
|
}
|
|
if len(result.StaleSkippedIDs) > 0 {
|
|
fmt.Fprintf(os.Stderr, " (%d stale skipped)", len(result.StaleSkippedIDs))
|
|
}
|
|
fmt.Fprintln(os.Stderr)
|
|
return nil
|
|
}
|
|
|
|
// renderImportOutcome reports a completed import (JSON or stderr),
|
|
// identically in both modes.
|
|
func renderImportOutcome(result importResultJSON, source string, dedupHits int) error {
|
|
if jsonOutput {
|
|
return outputJSON(result)
|
|
}
|
|
|
|
fmt.Fprintf(os.Stderr, "Imported %d issues", result.Created)
|
|
if result.Memories > 0 {
|
|
fmt.Fprintf(os.Stderr, " and %d memories", result.Memories)
|
|
}
|
|
fmt.Fprintf(os.Stderr, " from %s", source)
|
|
if dedupHits > 0 {
|
|
fmt.Fprintf(os.Stderr, " (%d duplicates skipped)", dedupHits) //nolint:gosec // G705: stderr, not a browser context
|
|
}
|
|
if staleSkipped := result.Skipped - dedupHits; staleSkipped > 0 {
|
|
fmt.Fprintf(os.Stderr, " (%d stale skipped; use --allow-stale to restore older rows)", staleSkipped) //nolint:gosec // G705: stderr, not a browser context
|
|
}
|
|
if result.Unchanged > 0 {
|
|
fmt.Fprintf(os.Stderr, " (%d already present, unchanged)", result.Unchanged) //nolint:gosec // G705: stderr, not a browser context
|
|
}
|
|
fmt.Fprintln(os.Stderr)
|
|
if len(result.UpdatedIssues) > 0 {
|
|
fmt.Fprintf(os.Stderr, "Updated %d existing issue(s):\n", len(result.UpdatedIssues))
|
|
for _, change := range result.UpdatedIssues {
|
|
fmt.Fprintf(os.Stderr, " %s: %s\n", change.ID, change.Changes)
|
|
}
|
|
}
|
|
if len(result.TieKeptLocalIDs) > 0 {
|
|
fmt.Fprintf(os.Stderr, "Kept local state for %d issue(s) with the same updated_at but different content (use --allow-stale to overwrite): %s\n",
|
|
len(result.TieKeptLocalIDs), strings.Join(result.TieKeptLocalIDs, ", "))
|
|
}
|
|
for _, skipped := range result.SkippedDependencies {
|
|
fmt.Fprintf(os.Stderr, "Skipped dependency: %s\n", skipped)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// importTitleSearcher is the read seam the --dedup filter needs. It lives in
|
|
// THIS file because naming types.IssueFilter is denied by default under
|
|
// cmd/bd and import.go is the named exception for the bulk-movement family
|
|
// (.golangci.yml, forbidigo): the classic storage.DoltStorage satisfies it
|
|
// directly, and uowImportTitleSearcher adapts the proxied unit of work.
|
|
type importTitleSearcher interface {
|
|
SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error)
|
|
}
|
|
|
|
// uowImportTitleSearcher adapts a unit of work's issue use case to the
|
|
// classic []*types.Issue search shape filterDuplicatesByTitle consumes. Both
|
|
// stacks run the same issueops search underneath (issues merged with wisps),
|
|
// so --dedup sees the same title universe in both modes.
|
|
type uowImportTitleSearcher struct {
|
|
uw uow.UnitOfWork
|
|
}
|
|
|
|
func (s uowImportTitleSearcher) SearchIssues(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) {
|
|
page, err := s.uw.IssueUseCase().SearchIssues(ctx, query, filter)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return page.Items, nil
|
|
}
|
|
|
|
// filterDuplicatesByTitle removes issues whose title matches an existing open issue.
|
|
func filterDuplicatesByTitle(ctx context.Context, st importTitleSearcher, issues []*types.Issue) ([]*types.Issue, int) {
|
|
existing, err := st.SearchIssues(ctx, "", types.IssueFilter{})
|
|
if err != nil {
|
|
return issues, 0
|
|
}
|
|
|
|
titleSet := make(map[string]bool, len(existing))
|
|
for _, issue := range existing {
|
|
if issue.Status != types.StatusClosed {
|
|
titleSet[strings.ToLower(issue.Title)] = true
|
|
}
|
|
}
|
|
|
|
var kept []*types.Issue
|
|
skipped := 0
|
|
for _, issue := range issues {
|
|
if titleSet[strings.ToLower(issue.Title)] {
|
|
skipped++
|
|
continue
|
|
}
|
|
kept = append(kept, issue)
|
|
}
|
|
return kept, skipped
|
|
}
|