beads/cmd/bd/import_proxied_server.go
cyb3rd0g01 48b9e904dc
fix(import): seed issue_prefix from config.yaml before the batch guard needs it (#6624)
* fix(import): seed issue_prefix from config.yaml before the batch guard needs it

NewBatchContext (via ReadConfigPrefix) unconditionally requires an
issue_prefix row in the database's own config table before creating
any issue. bd init writes that row via SetConfig, but a database an
external tool provisioned directly (e.g. an orchestrator that created
the Dolt database via SQL and pointed config.yaml at it without ever
calling bd init) never gets one. Every bd import then failed with
"issue_prefix config is missing", even though config.yaml carried the
prefix and bd where/bd context resolved it correctly.

bd import already had logic to sync config.yaml's prefix into that
table, but only after a successful import — useless when the import
itself can't start without the row. This moves an equivalent, guarded
sync (only fires when the table is empty, never overwrites an existing
prefix) before the import attempt, reading config.yaml directly via
GetStringFromDir rather than the global config singleton, since that
singleton isn't guaranteed to be initialized this early in the import
path.

Added a regression test that reproduces the exact failure with stock
bd (bd init --server, then delete the config-table row and write
config.yaml by hand — no orchestrator involved) and confirms it now
imports successfully.

* fix(import): address review — proxied-mode parity, --global exclusion, GetString

Three issues from @bee-ghosttrack's review of ffb32fb:

1. Proxied-server mode had the identical bug: importer.go's ImportBatch ran
   CreateIssuesInTxWithResult (-> NewBatchContext -> ReadConfigPrefix) before
   its own SyncIssuePrefix reconciliation, in the same transaction. Seeds the
   prefix before create there too, when the table is empty.

2. --global was not excluded from either sync (import.go's existing
   post-import sync and this fix's new pre-import seed, plus the proxied
   path's SyncIssuePrefix source). config.yaml is per-project; the shared
   global store's own prefix must win in --global mode (selectCreateIDPrefix
   already established this for `bd create`), or a project's local prefix
   silently overwrites the shared database with two dolt commits.

3. GetStringFromDir's rationale didn't hold: config.Initialize() has already
   run by main.go's init() and prepareSelectedCommandContext, and
   GetStringFromDir skips the config.local.yaml overlay GetString honors —
   disagreeing with the sync 40 lines below it. Switched to GetString,
   matching import.go:415 and import_proxied_server.go's existing usage.

Smaller fixes: TrimSpace + validatePrefix on the seeded value (a failed
validation now warns instead of writing an invalid prefix); SetConfig/Commit
failures now warn to stderr instead of being silently swallowed.

Test changes:
- Added TestImporterUOW/SeedsPrefixBeforeCreateNeedsIt, exercising the
  proxied-mode fix directly at the unit-of-work layer (no CLI/container
  overhead) by clearing config-table issue_prefix and confirming
  ImportBatch still creates issues. Revert-mutant confirmed: fails with
  the identical "issue_prefix config is missing" without the fix.
- Rewrote the E2E test with this repo's server-mode conventions:
  skipIfNoDolt, uniqueTestDBName + dropTestDatabase, explicit --server-port
  — no longer leaks a database on the shared test server.
- Added TestCLI_Import_GlobalModeIgnoresProjectConfigYAMLPrefix (separate
  file, cgo-only build tag, using the shared-server harness from
  global_identity_integration_test.go): confirms a --global import does not
  adopt a project's config.yaml prefix. Revert-mutant confirmed: without the
  --global guards, the shared global store's prefix is overwritten by the
  local project's ("global" -> "localproj").

go build ./... clean; go vet ./cmd/bd/... (both with and without -tags
"cgo integration") clean; gofmt -l clean on all touched files.

* fix(import): commit the seeded prefix on an empty batch; one resolved prefix

Maintainer review fixup for PR #6624 (iterations 1-2). Addresses the two major
findings and the nearby minors/nits.

MAJOR — the proxied seed was discarded on a batch that landed no rows.
The seed wrote issue_prefix but left ImportBatchResult.PrefixSynced false, so
importBatchCommitMessage produced "" for a batch with Created == 0 and no
memories, RunTxResultWithin skipped uw.Commit on the empty message, and the
deferred closeAttempt ROLLED BACK the seeded row. `bd import` reported success,
the prefix never landed, and a re-run dedup'd again — the state never
converged. Reachable against a missing-prefix database whenever every input row
is dropped by the in-transaction stale guard or the CLI's title dedup. The seed
now reports itself through PrefixSynced, restoring the "the prefix sync runs
even when the batch is otherwise empty" contract (be-llaf) that
import_proxied_server.go's comment promises; the field's doc says so too.

MAJOR — that arm had no coverage in any lane that runs.
TestImporterUOW/SeedsPrefixEvenWhenEveryRowIsRejected pins it: empty stored
prefix + SyncIssuePrefix set + every row stale-rejected => the prefix is
committed, under the sync message, with history moving exactly one entry. It
fails on the pre-fixup tree on all four assertions. The existing
SeedsPrefixBeforeCreateNeedsIt cannot catch this (Created == 1 carries the
commit) and PrefixSyncAloneCommitsUnderTheSyncMessage runs with a non-empty
stored prefix, so its seed never fires. Both are tag-free and container-free,
so unlike the classic-path e2e they execute in ordinary lanes.

Minors and nits:

- Seed errors are returned, not warned. A SetConfig failure here is a
  precondition failure for the very next statement: swallowing it surfaced as
  the "issue_prefix config is missing" error this seed exists to prevent, and
  hid serialization failures from RunTxResult's retry classifier. Drops the
  os import.
- Classic path resolves ONE prefix for both its writers (importSyncPrefix;
  iteration 2 below extends the same resolver to the proxied path). The
  seed used a TrimSpace'd + validatePrefix'd value while the post-import sync
  used the raw one, so (a) a prefix the seed refused still reached the database
  through the sync, bypassing the guard this PR adds, and (b) a padded
  `issue-prefix: " bd"` made the two disagree and the sync overwrote the
  trimmed value. TestImportSyncPrefix pins the resolver with no store and no
  container; each arm mutation-proved independently. Side effect: the flagged
  worst_changed function drops from cyclo 25/cognitive 43 to 22/34.
- Warnings name the whole effect on both paths: an invalid issue-prefix
  disables the sync as well as the seed, which base did not do.
- The --global isolation test anchors what its "before" value IS, so the
  after == before comparison cannot go vacuous. The anchor is
  doltserver.GlobalIssuePrefix, NOT the --prefix passed to init: --prefix names
  the project's prefix and `bd init --global` always writes the constant
  (init.go), which an unanchored equality had been hiding.
- Comments record two boundaries the diff left unstated: the classic seed
  commits before an import that may fail (deferring it is not an option — the
  post-import CommitWithConfig is gated on dbPrefix != syncPrefix, which a seed
  makes false, so the row would stay uncommitted because store.Commit excludes
  config (GH#2455), and a dirty internal config key blocks the next pull); and
  a --global import against a global database missing its own
  config.issue_prefix stays unseedable by design.

Iteration 2 — three non-gating findings (both majors above stayed fixed; the
fresh synthesis and quality scorecard both say approve):

- The classic seed now runs BEFORE the memory writes. CommitWithConfig is
  DOLT_COMMIT -Am, so it stages everything dirty at that instant: with the seed
  below the memory loop, a `bd import` carrying memories against a
  missing-prefix database whose issue phase then failed left those kv.memory.*
  rows PERMANENTLY COMMITTED, under a message asserting the commit is the
  prefix seed — where at base the failure left the whole working set for the
  operator to discard. Seeding first restores "the only thing this commit can
  carry is the prefix". Pure statement reorder: the seed reads only issues,
  syncPrefix and the config table, and the ordering is now recorded as
  load-bearing at the block so a later edit does not undo it.
- The proxied path CALLS importSyncPrefix() instead of restating its four rules
  inline. The two copies were identical rule for rule — same globalFlag guard,
  same TrimSpace, same validatePrefix, byte-identical warning — but only the
  classic copy was pinned, so the underscore-key fallback ReadConfigPrefix
  honors, or hyphen normalization, could have reintroduced ACROSS import modes
  exactly the divergence the bullet above closed within one mode. One
  implementation now, so TestImportSyncPrefix's six rows govern both routes.
  Drops the os, strings and config imports from import_proxied_server.go.
- The uow subtests no longer lean on suite order.
  SeedsPrefixEvenWhenEveryRowIsRejected snapshots the sync-message history
  count before the batch and asserts +1: the old `n != 0` form could not fail
  in a full-suite run, because PrefixSyncAloneCommitsUnderTheSyncMessage has
  already committed under that message by then. Mutation-proved — routing an
  all-rejected batch to a "0 issues from ..." message reds only the new
  assertion ("moved 1 -> 1"), and the 1 in that output is precisely the count
  that would have satisfied the old form. SeedsPrefixBeforeCreateNeedsIt also
  restores issue_prefix the way its two neighbours do, instead of leaving the
  fixture to be restored by the code under test.

The PR description's stale "GetStringFromDir / config.GetString returned empty
in this exact spot" claim and its "single-file production change" scope claim
were corrected on GitHub, and the CI-lane reality for each added test is now
recorded there.

Validation (iteration 1): gofmt clean; go build ./...; go vet on cmd/bd,
internal/storage/uow and issueops; full internal/storage/uow package green
(570s, under a private temp root); focused cmd/bd import tests,
TestImportSyncPrefix, TestEmbeddedImport/prefix_sync{,_no_config_key} and the
--global isolation e2e all green.

Validation (iteration 2): gofmt and `git diff --check` clean; go build
./cmd/... ./internal/storage/uow/...; go vet on cmd/bd and
internal/storage/uow; full TestImporterUOW green in subtest order (9/9) plus
the mutation A/B above; TestImportSyncPrefix green (6/6). Two reds in the wider
`go test -run Import ./cmd/bd` are environmental on this box and reproduce
identically with the fix set stashed: TestCLI_Import_GlobalModeIgnoresProjectConfigYAMLPrefix
dies inside `bd init` because the cloned remote is one schema version behind
this build (v66 -> v67), and TestImportBlockedDuringMigrationFreeze cannot
start its server (port 3307 busy).

---------

Co-authored-by: ilasic <ilasic@reversinglabs.com>
Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
2026-09-26 15:01:04 -07:00

181 lines
7 KiB
Go

package main
import (
"context"
"errors"
"fmt"
"path/filepath"
"github.com/steveyegge/beads/internal/storage/uow"
"github.com/steveyegge/beads/internal/types"
publicops "github.com/steveyegge/beads/issueops"
)
// proxiedImporter hands back the guarded bulk-import surface for the
// proxied-server provider, through the provider's OWN capability accessor —
// the same two-step proxiedIssueReader and proxiedBatchCloser perform, and
// for the same reason: the accessor is where each layer is added, so a
// command that reached for the constructor would get an unlayered importer.
func proxiedImporter() (publicops.Importer, error) {
if uowProvider == nil {
return nil, errors.New("proxied-server UOW provider not initialized")
}
src, ok := uowProvider.(uow.ImporterSource)
if !ok {
return nil, fmt.Errorf("proxied-server provider %T does not offer the import surface", uowProvider)
}
return src.Importer()
}
// runImportRecordsProxied is the proxied-server import pipeline over the
// parsed records. It mirrors runImportRecordsClassic stage for stage — dedup,
// dry-run classification, stale pre-filter, batch write, issue_prefix
// reconciliation — through the SAME classification and reporting code, with
// two deliberate structural differences:
//
// - ONE COMMIT PER INVOCATION. The classic path chunks a large import into
// bounded transactions (a SQLite write-lock fairness measure) and commits
// the issue_prefix sync separately; the proxied path has no PostRun
// auto-commit and the whole import — rows, memories, prefix sync — lands
// in ONE unit of work with ONE history entry, the Importer capability's
// contract.
//
// - The stale guard keeps its classic two-half shape: the pre-filter runs in
// its own read (reporting StaleSkippedIDs and keeping stale rows' aux data
// out of the batch), and RejectStaleUpserts re-checks updated_at inside
// the write transaction, closing the same race the classic path closes.
func runImportRecordsProxied(ctx context.Context, issues []*types.Issue, memories []memoryRecord, source string) error {
if uowProvider == nil {
return fmt.Errorf("proxied-server UOW provider not initialized")
}
// Dedup: skip issues whose title matches an existing open issue.
dedupHits := 0
if importDedup && len(issues) > 0 {
type dedupOutcome struct {
kept []*types.Issue
hits int
}
out, err := uow.RunTxRead(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (dedupOutcome, error) {
kept, hits := filterDuplicatesByTitle(ctx, uowImportTitleSearcher{uw: uw}, issues)
return dedupOutcome{kept: kept, hits: hits}, nil
})
if err != nil {
return err
}
issues = out.kept
dedupHits = out.hits
}
result := importResultJSON{
Source: source,
DedupHits: dedupHits,
DryRun: importDryRun,
}
if importDryRun {
result.Memories = len(memories)
result.Skipped = dedupHits
classification, err := uow.RunTxRead(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (*ImportResult, error) {
return classifyDryRunImport(ctx, uw.IssueUseCase(), issues, importAllowStale)
})
if err != nil {
return fmt.Errorf("dry-run: %w", err)
}
applyImportDryRunClassification(&result, classification)
return renderImportDryRun(result, len(memories), source, dedupHits)
}
// Pre-filter half of the stale guard (bd-pkim8): report rows already
// known stale and keep their labels/comments/dependencies out of the
// batch entirely. A local update racing between this read and the batch
// write is caught by RejectStaleUpserts inside the write transaction.
var staleSkippedIDs []string
var changePlan importChangePlan
if !importAllowStale && len(issues) > 0 {
type staleOutcome struct {
filtered []*types.Issue
skipped []string
plan importChangePlan
}
out, err := uow.RunTxRead(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (staleOutcome, error) {
filtered, skipped, plan, err := filterStaleImportIssues(ctx, uw.IssueUseCase(), issues)
if err != nil {
return staleOutcome{}, err
}
return staleOutcome{filtered: filtered, skipped: skipped, plan: plan}, nil
})
if err != nil {
return err
}
issues = out.filtered
staleSkippedIDs = out.skipped
changePlan = out.plan
}
memoryEntries := make([]publicops.ImportMemory, 0, len(memories))
for _, mem := range memories {
memoryEntries = append(memoryEntries, publicops.ImportMemory{
Key: kvPrefix + memoryPrefix + mem.Key,
Value: mem.Value,
})
}
importer, err := proxiedImporter()
if err != nil {
return err
}
// The ONE resolver both import paths share for config.yaml's issue_prefix
// (importSyncPrefix, import.go): the --global exclusion, the single trim,
// validatePrefix, the warning that names BOTH effects, and "" meaning
// "reconcile nothing". Calling it instead of restating its rules here is
// what keeps the two modes from drifting — a second hand-maintained copy
// of this policy is the very divergence one resolver exists to prevent,
// and the resolver's table test then covers this route too.
//
// Accepted boundary: a --global import against an externally-provisioned
// GLOBAL database that is itself missing config.issue_prefix stays
// unseedable and still fails with "issue_prefix config is missing" — that
// row is `bd init --global`'s to write, and config.yaml cannot speak for
// the shared store.
syncPrefix := importSyncPrefix()
// THE BATCH: rows, memories and the issue_prefix reconciliation in one
// transaction, one history entry. The prefix sync runs even when the
// batch is otherwise empty, exactly as the classic path's post-commit
// sync does (be-llaf; config.yaml is authoritative, not a rename) — and
// so does the seed, which reports itself through the same PrefixSynced
// result so an otherwise-empty batch still commits it.
batch, err := importer.ImportBatch(ctx, publicops.ImportBatchRequest{
Actor: getActorWithGit(),
Issues: issues,
Memories: memoryEntries,
AllowStale: importAllowStale,
SkipPrefixValidation: true,
SyncIssuePrefix: syncPrefix,
Source: filepath.Base(source),
})
if err != nil {
return fmt.Errorf("import failed: %w", err)
}
result.Memories = batch.MemoriesImported
if len(issues) > 0 {
staleRejectedSet := make(map[string]struct{}, len(batch.StaleRejectedIDs))
for _, id := range batch.StaleRejectedIDs {
staleRejectedSet[id] = struct{}{}
}
skippedDependencies := make([]string, 0, len(batch.SkippedDependencies))
for _, dep := range batch.SkippedDependencies {
skippedDependencies = append(skippedDependencies, fmt.Sprintf("%s -> %s: %s", dep.IssueID, dep.DependsOnID, dep.Reason))
}
applyImportOutcome(&result, assembleImportResult(issues, staleSkippedIDs, changePlan, staleRejectedSet, skippedDependencies))
} else {
result.Skipped += len(staleSkippedIDs)
result.StaleSkippedIDs = append(result.StaleSkippedIDs, staleSkippedIDs...)
}
return renderImportOutcome(result, source, dedupHits)
}