beads/cmd/bd/export.go
Jim Wordelman a9a48d6e36
test(cli,storage): BEADS_MAX_ROWS behavioral + opt-out gate tests (be-x42v) (#4160)
* feat(storage): be-x42v.1 BEADS_MAX_ROWS defensive cap foundation

Add IssueFilter.MaxRows + IssueFilter.MaxRowsSource and a typed
issueops.ErrTooManyRows error. searchTableInTx routes its LIMIT through
EffectiveSearchLimit (LIMIT MaxRows+1 when MaxRows>0, with Limit
preserved otherwise); SearchIssuesInTx checks EnforceMaxRowsCap after
the issues+wisps merge so the cap reflects the count returned to the
caller. Both helpers are exported so future PG-backend support can
reuse the same logic.

Default MaxRows=0 disables the cap and is byte-identical to today at
every existing call site. Limit semantics are unchanged: Limit=0 still
means unlimited at the contract level.

PG backend is not on origin/main yet; PG cap support is a follow-up
when the postgres backend merges. Validator's per-backend bead exercises
the issueops path via Dolt/embedded fixtures.

Refs: be-x42v (designer §3/§6.2), be-jp5s (architecture D1/R-03).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): be-x42v.2 wire --max-rows / BEADS_MAX_ROWS on CLI surface

Adds cmd/bd/max_rows.go with three helpers: addMaxRowsFlag registers
--max-rows int with the designer's spec help text, resolveMaxRows
picks flag > env > 0 (negative flag → exit 1, bad env → warn-and-ignore),
and handleMaxRowsError pattern-matches *issueops.ErrTooManyRows to
emit the two-line stderr message and exit 2 with stdout empty.

The flag is wired on bd list, bd ready, bd dep tree, bd find-duplicates,
and bd graph (designer §4). The env-only resolveMaxRowsEnvOnly variant
is wired on bd lint and the SearchIssues paths inside bd doctor
(conventions + pollution checks). bd cleanup, bd gc, bd export,
bd export --auto, bd migrate-issues, and bd jira explicitly set
MaxRows=0 / MaxRowsSource="" in code so a misconfigured env can't
abort a data-integrity sweep partway through.

bd ready needed storage-layer support too: WorkFilter gains MaxRows +
MaxRowsSource and GetReadyWorkInTx routes its LIMIT through
EffectiveSearchLimit and enforces the cap via EnforceMaxRowsCap on
the row count returned to the caller. bd dep tree applies the cap at
the CLI layer on the TreeNode count since the tree walk doesn't go
through IssueFilter.

bd config show gains a standalone-env collector so BEADS_MAX_ROWS
shows up alongside BEADS_DIR etc. when set. The Viper-managed env
detection skips it because BEADS_MAX_ROWS isn't bound to a config key.

Adds a CONTRIBUTING.md section documenting the opt-out rule for new
IssueFilter sites that aren't on designer §4's wired-up list, with a
pointer to the validator's opt-out test gates (be-x42v.4).

Smoke-tested against the rig database (298 issues):
- bd list --max-rows 5 → exit 2, "--max-rows=5"
- BEADS_MAX_ROWS=5 bd list → exit 2, "BEADS_MAX_ROWS=5"
- BEADS_MAX_ROWS=banana bd list --limit 3 → warn + exit 0
- BEADS_MAX_ROWS=5 bd list --max-rows 0 --limit 3 → exit 0
- bd list --max-rows -1 → exit 1, "must be non-negative"
- bd ready --max-rows 5 → exit 2 (storage layer fires)
- bd dep tree be-jp5s --max-rows 1 → exit 2 (CLI-layer count)
- bd find-duplicates --max-rows 5 → exit 2
- bd graph --all --max-rows 5 → exit 2
- BEADS_MAX_ROWS=5 bd lint → exit 2 (env-only)
- BEADS_MAX_ROWS=5 bd export → exit 0, 299 lines (opt-out)
- BEADS_MAX_ROWS=10 bd config show → lists BEADS_MAX_ROWS

The 10 CLI behavioral test scenarios for bd list are owned by
be-x42v.3 (validator). Coverage for the non-list paths, the env-only
doctor family, and config show is filed as be-u8z9 (needs-tests,
routed to validator).

Refs: be-x42v (designer §2-§5), be-x42v.1 (foundation).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: release gate PASS for be-x42v.1 + be-x42v.2 (BEADS_MAX_ROWS)

Documents PASS evaluation across the six gate criteria for the
storage foundation (be-x42v.1, 3900c1bdc) and the CLI wiring
(be-x42v.2, dea537c56) shipping together. Review verdicts: be-xf0o
(foundation) and be-2p12 (CLI), both PASS from Opus 4.7 reviewer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: regenerate CLI reference for --max-rows flag

* docs: revert federation to pure-go stub in CLI_REFERENCE.md

CI's check-doc-flags freshness check runs generate-cli-docs.sh with
CGO_ENABLED=0. The previous regeneration used a CGO binary, expanding
the federation section with subcommands. Revert to the requires-CGO
stub that the pure-go binary produces.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs: revert website federation.md pages to pure-go stub (be-x42v)

Three website cli-reference/federation.md files still had CGO-enabled
content after CLI_REFERENCE.md was reverted. This commit brings them
into sync with the pure-go stub, resolving the Check doc flags freshness
CI failure on PR #3908.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(cli): be-u8z9 BEADS_MAX_ROWS non-list command paths + config show

Locks down the manually-verified behavior shipped in be-x42v.2 across the
CLI surface beyond `bd list`.

cmd/bd/max_rows_test.go (new):
  - ReadyMaxRows_FlagOverCap_Exits2  - bd ready --max-rows N → exit 2
  - ReadyMaxRows_EnvOverCap_Exits2   - BEADS_MAX_ROWS=N bd ready → exit 2
  - DepTreeMaxRows_TreeNodes_Exits2  - tree-node count cap in dep tree
  - FindDuplicatesMaxRows_Exits2     - find-duplicates --max-rows N
  - GraphAllMaxRows_Exits2           - graph --all --max-rows N
  - LintMaxRows_EnvOnly_Exits2       - bd lint with BEADS_MAX_ROWS=N
  - LintMaxRows_NoFlagAccepted       - cobra rejects --max-rows on lint
  - ConfigShow_ListsBeadsMaxRows     - bd config show surfaces the env var
  - ConventionsMaxRows_EnvOnly       - skipped (bd doctor needs server mode,
                                       follow-up filed as be-pc8c)
  - PollutionMaxRows_EnvOnly         - skipped (same reason)

cmd/bd/ready_max_rows_test.go (new):
  - TestGetReadyWork_MaxRowsSuite — exercises WorkFilter.MaxRows in
    GetReadyWork (under cap / over cap / zero disabled / with Limit
    matrix), via newTestStore. Companion to the SearchIssues parity
    work in be-x42v.3.

Refs: be-u8z9 (this bead), be-x42v.2 (builder), be-pc8c (follow-up).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(cli,storage): be-x42v.3 BEADS_MAX_ROWS bd list scenarios + backend parity

Covers the 10 CLI scenarios from designer §6.1 against `bd list` and the
5 storage scenarios from designer §6.2 against both Dolt-server and
embedded backends.

cmd/bd/max_rows_test.go (extended):
  TestEmbeddedMaxRowsList — 10 subtests on a shared 21-issue rig:
    - Disabled_NoEnv             (baseline, no cap)
    - Flag_UnderCap              (--max-rows 100 over 21 rows → success)
    - Flag_OverCap               (--max-rows 5 → exit 2, source attribution, empty stdout)
    - Env_OverCap                (BEADS_MAX_ROWS=5 → exit 2, env source)
    - Flag_OverridesEnv          (env=5 flag=100 → success)
    - Flag_Zero_OverridesEnv     (env=5 --max-rows 0 → success, explicit disable)
    - BadEnv_LogsAndIgnores      (env=banana → warning + normal output, exit 0)
    - Negative_FlagRejected      (--max-rows -1 → "must be non-negative", exit 1)
    - LimitSet_CapTighter        (--limit 100 --max-rows 5 → exit 2)
    - LimitSet_CapLooser         (--limit 5 --max-rows 100 → 5 rows, exit 0)

internal/storage/embeddeddolt/search_max_rows_test.go (new):
  TestSearchIssues_MaxRows_{NotExceeded,Exceeded,Zero_NoCap,WithLimit}
  Exercises SearchIssues end-to-end against the embedded backend on a
  6-row fixture, including the four-quadrant (Limit, MaxRows) matrix.

internal/storage/dolt/search_max_rows_test.go (new):
  Identical scenarios against the dolt-server backend. Per-backend
  parity is implicit: both SearchIssues entry points wrap
  issueops.SearchIssuesInTx, so cap behavior is shared by construction.
  Same fixture + same assertions ensures divergence cannot ship silently.

PG backend coverage is deferred until the postgres-backend branch lands
(per PR #3908 description and parent designer notes).

Refs: be-x42v.3 (this bead), be-x42v.2 (builder), be-u8z9 (sibling validator for non-list paths).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(cli): be-x42v.4 opt-out gate tests for data-integrity commands

Add filterCapturingStore spy and four MaxRows=0 gate tests asserting
that bd export, bd export --auto, bd jira sync, and bd migrate-issues
always construct IssueFilter with MaxRows=0 / MaxRowsSource="" per
designer §4.1 opt-out rule. A BEADS_MAX_ROWS env var on these paths
would corrupt backups, sync state, or migration runs mid-flight.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(cli): be-x42v.4 add TestExport_BypassesBeadsMaxRows behavioral gate

End-to-end belt-and-suspenders test (designer §6.4): sets BEADS_MAX_ROWS=1,
seeds 5 task issues, runs runExport, and asserts 5 JSONL lines in the output.
Completes the be-x42v.4 acceptance criteria for the bypass behavioral test
alongside the filter-spy gate already committed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(cli): be-pc8c server-mode BEADS_MAX_ROWS gate for bd doctor --check={conventions,pollution}

TestDoctorConventionsMaxRows_EnvOnly_Exits2 and TestDoctorPollutionMaxRows_EnvOnly_Exits2
cover the two subtests that were skipped in max_rows_test.go because bd doctor refuses
to run in embedded mode (doctor.go:188). Both tests seed 6 open issues against the
Dolt test server with BEADS_MAX_ROWS=3, assert exit 2, and confirm BEADS_MAX_ROWS=3
in the combined output.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(search): enforce MaxRows cap when wisps table is empty

SearchIssuesInTx skipped EnforceMaxRowsCap on the common empty-wisps
path: when filter.Ephemeral is nil/false and the wisps table is empty,
the early `return results, nil` returned before the post-merge cap check.
The issues-only result set (LIMIT cap+1) could exceed MaxRows yet never
trip ErrTooManyRows. Enforce the cap before the early return.

Fixes TestSearchIssues_MaxRows_Exceeded_ReturnsErrTooManyRows and
TestSearchIssues_MaxRows_WithLimit (internal/storage/embeddeddolt).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(rebase): re-home BEADS_MAX_ROWS enforcement onto post-rebase seams

Fixes gaps exposed by rebasing PR #4160 onto current upstream/main,
found by actually running the PR's own gate tests rather than just
resolving textual conflicts:

- internal/storage/issueops/search.go: the Q2 perf-escape-hatch
  SkipWisps early return (added by main's search perf work, landed
  after this PR's original design) skipped EnforceMaxRowsCap entirely.
  bd list sets SkipWisps=true by default, so `bd list --max-rows N`
  silently ignored the cap. Enforce the cap on that return path too,
  mirroring the existing empty-wisps-table fix alongside it.

- internal/storage/issueops/search_counts.go: SearchIssuesWithCountsInTx
  (the `--json` query path, already present at the PR's merge-base but
  never wired for MaxRows by the original be-x42v.1/.2 commits) neither
  sized its SQL LIMIT off the cap nor enforced it on the merged result.
  Thread filter.MaxRows through EffectiveSearchLimit for the query LIMIT
  and check EnforceMaxRowsCap in the shared finishSearchIssuesWithCounts
  terminal helper (now returns an error), which every exit path already
  routed through. Fixes be-x42v.3's TestEmbeddedMaxRowsList/Flag_OverCap
  (`bd list --json --max-rows 5`), which failed after the rebase.

- cmd/bd/jira_max_rows_gate_test.go,
  cmd/bd/migrate_issues_max_rows_gate_test.go: seed INSERTs predate
  migration 0049 (LONGTEXT NOT NULL on issues.description/design/
  acceptance_criteria/notes, no default) that landed on main after
  this branch's merge-base. Provide empty-string values for the
  now-required columns.

Not fixed (separate, pre-existing gap; no gate test in this PR
exercises it): GetReadyWorkWithCountsInTx (`bd ready --json` /
`bd list --ready --json`) also doesn't thread MaxRows. Left alone to
keep this rebase to conflict resolution rather than expanding scope;
worth a follow-up bead.

Agent-Signature: claude-sonnet-medium on behalf of maphew

* fix(review): address adjudicated cross-vendor review follow-up (be-x42v.4)

Works the MUST-FIX / SHOULD-FIX list from the adjudicated claude-opus +
codex-gpt-5.6-sol review on the rebase (25e35cc3e), landing coverage gaps
the vendors found in the feature's own promise rather than the rebase
mechanics.

MUST-FIX:

1. internal/storage/issueops/claim.go: ClaimReadyIssueInTx forwarded the
   caller's MaxRows/MaxRowsSource into its internal unbounded
   GetReadyWorkInTx scan, so a rig-wide cap (e.g. BEADS_MAX_ROWS=100 with
   150 ready issues) hard-failed a claim that only ever consumes one row.
   Bound the internal scan via Limit instead, then clear the cap fields.

2. cmd/bd/ready.go, internal/storage/issueops/ready_work_counts.go:
   GetReadyWorkWithCountsInTx (ready --json / list --ready --json) never
   enforced MaxRows at all — a pre-existing gap in the original PR (the
   function predates this branch's merge-base), not something the rebase
   introduced. Threaded EffectiveSearchLimit into its query LIMIT and added
   EnforceMaxRowsCap via a new finishReadyWorkWithCounts terminal helper,
   mirroring finishSearchIssuesWithCounts. Wired handleMaxRowsError on the
   bd ready --json error branch.

3. cmd/bd/max_rows.go + list/ready/graph/dep/find-duplicates: proxied-server
   mode silently ignored the cap (the proxied repository doesn't thread
   MaxRows through the UOW pipeline). Added rejectMaxRowsUnderProxiedServer,
   called on every command that registers --max-rows before it diverts to
   its *ProxiedServer path — an explicit --max-rows or BEADS_MAX_ROWS now
   errors out under --proxied-server instead of no-oping.

4. cmd/bd/list.go: `--limit N --max-rows N` errored instead of truncating.
   withFetchOneExtra unconditionally bumped Limit to N+1 for its own
   >N-matches truncation-hint probe, which then crossed EffectiveSearchLimit's
   `limit > maxRows` branch and tripped the cap on the probe row alone, even
   though the delivered set is always trimmed back to N. withFetchOneExtra
   now skips the bump when MaxRows is active and no looser than Limit;
   >N-matches semantics for limit<maxRows are unaffected.

5. cmd/bd/list.go: `bd list --watch` swallowed the initial-query cap error
   (logged to stderr, returned void, exited 0). watchIssues now returns the
   initial-query error instead of handling it itself; runListCore routes it
   through handleMaxRowsError for exit-code-2 semantics. Mid-poll refresh
   failures still just log and continue (interactive command, not a reason
   to kill the watch loop).

6. cmd/bd/max_rows.go + every call site: handleMaxRowsError and the
   --max-rows usage-error path in resolveMaxRows called os.Exit directly
   from deep inside RunE, bypassing both the command's own metrics defer
   and main()'s post-ExecuteC metrics.CloseAndFlush(). Converted both to
   return errors: resolveMaxRows now returns (int, string, error) and
   handleMaxRowsError returns error, both following the exact &exitError{Code:
   N}-via-RunE precedent already used elsewhere in this package (e.g.
   list_input.go's skip-labels-conflict check, HandleErrorRespectJSON).
   doctor_conventions.go's runConventionsLint (called from a "()
   []doctorCheck" advisory-checks aggregator with no error channel) needed
   its own (([]doctorCheck, error) signature to propagate the one non-advisory
   failure mode; every other call site was already RunE-adjacent.

SHOULD-FIX (both mechanical, done):

7. cmd/bd/graph.go: `bd graph <issue>` (no --all) never resolved the cap.
   loadGraphSubgraph is a BFS over per-ID GetDependents/GetDependencies
   lookups with no IssueFilter to thread MaxRows through, so — mirroring
   dep tree — the cap is now checked post-hoc against the final
   connected-component node count. Documented both graph's per-status
   (--all) and post-hoc (single-issue) cap semantics, and dep tree's
   post-hoc semantics, in their flag help text.

8. internal/storage/issueops/ready_work.go: getReadyWispsInTx's unbounded
   (Limit<=0) branch queried and hydrated the *entire* wisps table before
   the post-merge cap check in GetReadyWorkInTx ever ran, defeating the
   "circuit breaker against pathological queries" purpose for
   --include-ephemeral. readyWorkWispIssueFilter now propagates
   MaxRows/MaxRowsSource so EffectiveSearchLimit bounds that query to
   cap+1 up front.

Tests added: TestClaimReadyIssue_MaxRows_LargePoolUnderLowCap (1),
ReadyMaxRowsJSON_FlagOverCap_Exits2 + ListReadyMaxRowsJSON_FlagOverCap_Exits2
(2), reject_max_rows_flag + reject_max_rows_env + allow_max_rows_zero under
TestProxiedServerList (3), LimitEqualsCap_TruncatesNotErrors +
LimitEqualsCap_JSON_TruncatesNotErrors (4), Watch_FlagOverCap_Exits2 (5),
GraphSingleIssueMaxRows_NodeCount_Exits2 + _UnderCap_NoError (7),
ReadyIncludeEphemeralMaxRows_FlagOverCap_Exits2 (8).

Verified: go build/vet clean; full MaxRows-named + FilterHasZeroMaxRows +
BypassesBeadsMaxRows sweep passes in both dolt-server and
BEADS_TEST_EMBEDDED_DOLT=1 modes (one dolt-testcontainer flake observed and
confirmed non-reproducing on retry, consistent with prior sessions);
TestProxiedServerList passes with BEADS_TEST_PROXIED_SERVER=1.

Agent-Signature: claude-sonnet-medium on behalf of maphew

* fix(review): address round-2 delta review regressions (be-x42v.4 round 3)

opus verified all eight round-2 items as implemented, but codex found three
regressions those fixes introduced, confirmed on adjudication. Common
thread: enforce the cap on the DELIVERED result, not on fetched/merged
intermediates, and don't let the safety rejection block one-row operations.

1. cmd/bd/ready.go: proxied `bd ready --claim` was rejected by
   rejectMaxRowsUnderProxiedServer before it could dispatch to
   runReadyProxiedClaim, even though --claim always delivers exactly one
   row and the direct-path fix (claim.go, round 2) already exempts claim
   from cap enforcement for that exact reason. Moved the claimReady flag
   parse before the proxied branch and exempt --claim from the rejection;
   bulk (non-claim) proxied `bd ready` still rejects an active cap. Added
   claim_succeeds_under_env_max_rows_cap plus
   reject_bulk_ready_under_max_rows_{flag,env} to
   ready_proxied_integration_test.go.

2. internal/storage/issueops/ready_work_counts.go: with --include-ephemeral,
   GetReadyWorkWithCountsInTx's issues and wisps queries are each
   independently bounded by EffectiveSearchLimit(filter.Limit,
   filter.MaxRows), so the merged pre-trim slice can hold up to ~2x that
   per-table bound even when filter.Limit <= filter.MaxRows (e.g. limit=2,
   cap=3, 2 rows/table -> merged 4 -> false "too many rows" even though the
   delivered page, trimmed to 2, is well within the cap). Reordered
   finishReadyWorkWithCounts to trim to Limit before checking the cap,
   mirroring GetReadyWorkInTx's non-counts path (mergeReadyWisps already
   trims before its caller's EnforceMaxRowsCap runs). Verified this
   reproduces "4 found, --max-rows=3 exceeded" on the pre-fix code and is
   fixed after. Added ReadyIncludeEphemeralMaxRowsJSON_MergedUnderLimit_NoError.

3. cmd/bd/list.go: round-2's fix for `--limit N --max-rows N` (skip the
   withFetchOneExtra probe-row bump entirely when Limit>=MaxRows) stopped
   the false cap error but also stopped fetching the extra row, so
   len(results) could never exceed effectiveLimit and the GH#3212
   truncation notice silently stopped firing — a partial result started
   presenting as complete. Restored the probe-row fetch (Limit still bumps
   to N+1) and instead bump MaxRows to N+1 in lockstep, but only in the
   exact Limit==MaxRows case: this keeps the probe row inside the
   (temporarily N+1) cap so EnforceMaxRowsCap doesn't trip on it, while the
   CLI's own len(results)>effectiveLimit check still sees the extra row and
   correctly reports truncated=true. Tighter-cap (Limit>MaxRows) and
   looser-cap (Limit<MaxRows) cases are unaffected — MaxRows is left
   unbumped there, so a genuine violation still reports the true cap value.
   printTruncationHint's text is gated on ui.IsStderrTerminal() and
   unconditionally suppressed for this subprocess test harness's piped
   stderr (matches the existing limit_truncation_hint subtest's documented
   constraint), so the fix is verified via 4 new unit tests asserting
   withFetchOneExtra's exact bump values directly
   (TestWithFetchOneExtra_LimitEqualsCap_BumpsBothForTruncationProbe et al.)
   rather than the unobservable hint text; extended
   LimitEqualsCap_TruncatesNotErrors's comment to document this and point
   to the unit tests.

Opus nits:

4. internal/storage/issueops/claim.go: bounding the claim scan at
   Limit=MaxRows (round 2) meant a BEADS_MAX_ROWS=1 rig would only attempt
   the single top-of-queue ready row and return "nothing to claim" if it
   was transiently unclaimable (e.g. a race with another agent), even with
   plenty of other ready work available. Restored the unbounded scan
   (Limit=0); the breaker's job is to bound delivered payloads, and a
   claim's payload is always exactly one row regardless of scan size.

5. cmd/bd/find_duplicates.go: replaced the inlined proxied-server rejection
   check with a call to rejectMaxRowsUnderProxiedServer so the message
   can't drift from the other 4 commands that use it.

Verified: go build/vet clean; full MaxRows-named + FilterHasZeroMaxRows +
BypassesBeadsMaxRows + WithFetchOneExtra sweep passes in both dolt-server
and BEADS_TEST_EMBEDDED_DOLT=1 modes; TestProxiedServerList +
TestProxiedServerReady (full suites, ~200s) pass with
BEADS_TEST_PROXIED_SERVER=1; broader TestList|TestWatch|TestGraph|
TestDepTree|TestReady|TestClaim|TestFindDuplicates sweep clean except the
same pre-existing unrelated ICU-header build failure noted in the round-2
commit (confirmed via stash-based control test to reproduce identically
without this round's changes).

Agent-Signature: claude-sonnet-medium on behalf of maphew

* fix(review): address round-3 delta review items (be-x42v.4 round 4)

Converging: four small items adjudicated from round-3 delta reviews.

1. internal/storage/issueops/search.go, search_counts.go: round 3's
   trim-before-cap fix landed in finishReadyWorkWithCounts but not on the
   sibling bd list search paths. `bd list --include-infra --limit 2
   --max-rows 5` (3 durable + 3 wisps) exited 2 in both output modes even
   though the delivered page is 2 rows, since --include-infra disables the
   default SkipWisps escape hatch so the wisps merge actually runs.
   Reproduced live: "6 found, --max-rows=5 exceeded" on the pre-fix code.
   Mirrored the ready remedy: trim to filter.Limit (guarded by Limit>0)
   before EnforceMaxRowsCap in searchInTx's four exit points (added a
   trimToSearchLimit helper) and in finishSearchIssuesWithCounts (swapped
   the cap/trim order). Fixed search_counts.go's doc comment, which still
   asserted the old cap-before-trim rationale and directly contradicted
   ready_work_counts.go's. Single-source results are unaffected (a lone
   query never exceeds Limit when Limit>0, so the trim there is a no-op);
   Limit=0 and genuine Limit>MaxRows overage still error. Added
   IncludeInfraMergedUnderLimit_NoError (both bd list output modes) to
   TestEmbeddedMaxRowsList.

2. cmd/bd/ready.go: the round-3 fix that exempts proxied `ready --claim`
   from rejectMaxRowsUnderProxiedServer also skipped resolveMaxRows
   entirely for that branch, so `--max-rows -1` was silently accepted
   under proxied claim (every other command, direct or proxied, rejects
   it with exit 1). Still call resolveMaxRows in the claim-exempt branch
   for its validation side effect, ignoring the resolved cap for the
   claim itself. Reproduced live (claim succeeded instead of failing) on
   the pre-fix code. Added claim_rejects_invalid_max_rows_flag to
   ready_proxied_integration_test.go.

3. cmd/bd/find_duplicates.go, cmd/bd/max_rows.go: find-duplicates resolved
   --max-rows twice under proxied mode (once to build its filter, again
   inside rejectMaxRowsUnderProxiedServer), doubling the malformed-env
   stderr warning. Split rejectMaxRowsUnderProxiedServer into itself plus
   a new rejectResolvedMaxRowsUnderProxiedServer(maxRows int) that takes
   an already-resolved value; find_duplicates.go now calls the latter with
   its existing resolved maxRows instead of re-resolving.

4. internal/storage/issueops/claim.go: added a comment sentence noting the
   unbounded-scan trade-off per the adjudication note — parity with
   pre-PR main and correctness-first (preserves claim's existing
   fairness/ordering across the ready set); a bounded, paged scan is a
   deliberate follow-up, not folded in here.

Verified: go build/vet clean; full MaxRows-named + FilterHasZeroMaxRows +
BypassesBeadsMaxRows + WithFetchOneExtra sweep passes in both dolt-server
and BEADS_TEST_EMBEDDED_DOLT=1 modes (including the two new regression
tests); full TestProxiedServerReady suite (~211s) passes with
BEADS_TEST_PROXIED_SERVER=1; internal/storage/issueops unit tests pass;
broader TestList|TestWatch|TestGraph|TestDepTree|TestReady|TestClaim|
TestFindDuplicates sweep clean except the same pre-existing unrelated
ICU-header build failure noted in prior rounds. Both new regression tests
(item 1 and item 2) reproduce the reported failure on the pre-fix code via
git-stash and pass after.

Agent-Signature: claude-sonnet-medium on behalf of maphew

* fix(review): sort merged issues+wisps before trim in searchInTx (be-x42v.4 round 5)

codex P1, confirmed live: searchInTx's merge branch concatenates two
independently ORDER BY'd legs (`results = append(filtered,
wispResults...)`), and round-4's trimToSearchLimit then sliced that
UNSORTED concatenation. With Limit=N and durable rows alone filling N, a
higher-ranked wisp (per filter.SortBy/SortDesc) was silently dropped from
the page — before round 4 the CLI received the full merged set and its own
sort-then-trim produced correct page membership; round 4 moved a naive
prefix-trim into storage ahead of any global re-sort.

Fix: added a `less func(a, b T, sortBy string, sortDesc bool) bool` field
to searchProjection[T], populated for issueProjection via sqlbuild.Less
(the existing Go-side mirror of the SQL ORDER BY, already used correctly
by search_counts.go's sortSearchIssuesWithCounts). searchInTx's merge
branch now re-sorts the concatenation via a new sortMergedResults helper
before trimToSearchLimit runs, mirroring how finishSearchIssuesWithCounts
already orders before its own trim. idProjection (bare-ID search, no
sortable payload) leaves less nil — sortMergedResults no-ops in that case;
documented why this is safe today (no current caller combines Limit>0 with
a merged, non-SkipWisps ID-only search — see id_parser.go, which always
passes Limit=0) and where to look if that changes. The other three
trimToSearchLimit call sites in searchInTx are single-source (already
SQL-ordered), so they don't need this.

Checked the two round-3/round-4-cited "mirror" precedents per the
adjudication note, since this bug's shape (unsorted-concat-then-trim) could
plausibly recur there: GetReadyWorkInTx's mergeReadyWisps already calls
sortReadyIssues(kept, filter.SortPolicy) before its own Limit trim, and
GetReadyWorkWithCountsInTx / SearchIssuesWithCountsInTx both already sort
(sortIssuesWithCountsByPolicy / sortSearchIssuesWithCounts) before their
finish* trim-then-cap helpers. All three were already globally ordered
before trimming; only searchInTx had the bug.

Added IncludeInfraMerged_WispOutranksDurable_InDeliveredPage to
TestEmbeddedMaxRowsList: 3 durable P2 issues + 1 wisp at P0 (outranks under
the default priority-ASC sort), --limit 1 in both bd list output modes —
asserts the P0 wisp is the delivered row, not an arbitrary P2 durable row.
Reproduces exactly against 77227aa67 (the durable row is returned instead
of the wisp) and passes after the fix.

Verified: go build/vet clean; full MaxRows-named + FilterHasZeroMaxRows +
BypassesBeadsMaxRows + WithFetchOneExtra + Search sweep passes in both
dolt-server and BEADS_TEST_EMBEDDED_DOLT=1 modes (including the new
regression test); internal/storage/issueops unit tests pass.

Agent-Signature: claude-sonnet-medium on behalf of matt wilkie

* docs: remove --max-rows lines from pinned-generator CLI docs (be-x42v.4 CI fix)

CI's "Check doc flags freshness" job compares docs/CLI_REFERENCE.md and
docs/cli-reference/*.md against bd built from the pin in
docs/cli-docs.pin (v1.1.0), which predates --max-rows/BEADS_MAX_ROWS —
the same pinned-doc-generator trap that hit PRs 3920 and 4022 in this
sweep. Any doc lines this PR added for the new flag are flagged stale
against that older pinned build, and the failure cascades into "PR
Policy (wrapper timing)" via make ci-pr-policy.

Applied CI's own generated fix verbatim (downloaded the
cli-docs-freshness-patch artifact from the failing job
https://github.com/gastownhall/beads/actions/runs/30067827283/job/89403539322,
git apply --check passed cleanly, applied as-is — no manual edits).
Removes the six --max-rows generated-doc lines (bd list, bd ready, bd dep
tree, bd find-duplicates, bd graph — CLI_REFERENCE.md and the matching
cli-reference/*.md) plus the incidental column-width realignment in the
bd graph flag tables that removing the widest flag name triggers.

The flag help strings themselves (cmd/bd/max_rows.go's addMaxRowsFlag)
are untouched — those are the source of truth for docs generation and
will regenerate correctly once docs/cli-docs.pin advances past this
PR's merge. This is a pinned-generator artifact of the CI pipeline, not
a statement that the flag is undocumented.

Agent-Signature: claude-sonnet-medium on behalf of maphew

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-24 01:40:36 -07:00

377 lines
12 KiB
Go

package main
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"sort"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/atomicfile"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/metrics"
"github.com/steveyegge/beads/internal/storage/domain"
"github.com/steveyegge/beads/internal/types"
)
var exportCmd = &cobra.Command{
Use: "export",
Short: "Export issues to JSONL format",
Long: `Export all issues to JSONL (newline-delimited JSON) format.
Each line is a complete JSON object representing one issue, including its
labels, dependencies, and comments.
This command is for issue export, migration, and interoperability. It exports
records from the issues table; it is not a full database backup and does not
capture Dolt branches, commit history, working-set state, or non-issue tables.
For supported full backup/restore flows, use 'bd backup init', 'bd backup sync',
and 'bd backup restore'.
By default, exports only regular issues (excluding infrastructure beads
like agents, roles, and messages). Use --all to include everything.
Memories (from 'bd remember') are excluded by default because they may
contain sensitive agent context. Use --include-memories or --all to
include them.
EXAMPLES:
bd export # Export issues to stdout
bd export -o issues.jsonl # Export issues to file
bd export --include-memories # Export issues + memories
bd export --all -o full.jsonl # Include infra + templates + gates + memories
bd export --scrub -o clean.jsonl # Exclude test/pollution records`,
GroupID: "sync",
SilenceUsage: true,
SilenceErrors: true,
RunE: runExport,
}
var (
exportOutput string
exportAll bool
exportIncludeInfra bool
exportScrub bool
exportNoMemories bool
exportIncludeMemories bool
exportExcludeOwners []string
exportVerbose bool
)
func init() {
exportCmd.Flags().StringVarP(&exportOutput, "output", "o", "", "Output file path (default: stdout)")
exportCmd.Flags().BoolVar(&exportAll, "all", false, "Include all records (infra, templates, gates, memories)")
exportCmd.Flags().BoolVar(&exportIncludeInfra, "include-infra", false, "Include infrastructure beads (agents, roles, messages)")
exportCmd.Flags().BoolVar(&exportScrub, "scrub", false, "Exclude test/pollution records")
exportCmd.Flags().BoolVar(&exportIncludeMemories, "include-memories", false, "Include persistent memories (from 'bd remember') in the export")
exportCmd.Flags().BoolVar(&exportNoMemories, "no-memories", false, "Exclude persistent memories (deprecated: now the default)")
_ = exportCmd.Flags().MarkHidden("no-memories")
exportCmd.Flags().StringArrayVar(&exportExcludeOwners, "exclude-owner", nil, "Exclude issues created by this identity (repeatable; also reads export.exclude_owners config)")
exportCmd.Flags().BoolVar(&exportVerbose, "verbose", false, "Print filtered issue count when owners are excluded")
rootCmd.AddCommand(exportCmd)
}
func runExport(cmd *cobra.Command, args []string) error {
if usesProxiedServer() {
return HandleErrorRespectJSON("export is not supported in proxied-server mode")
}
evt := metrics.NewCommandEvent("export")
defer func() {
if c := metrics.Global(); c != nil {
c.CloseEventAndAdd(evt)
}
}()
ctx := rootCtx
// Determine output destination. File output uses atomic writes
// (temp file + rename) so concurrent exports and crashes never
// leave a truncated or interleaved JSONL file.
var w io.Writer
var aw *atomicfile.Writer
if exportOutput != "" {
var err error
aw, err = atomicfile.Create(exportOutput, 0o644)
if err != nil {
return HandleErrorRespectJSON("failed to create output file: %v", err)
}
defer func() {
// Abort is a no-op if Close was already called.
_ = aw.Abort()
}()
w = aw
} else {
w = os.Stdout
}
// Build filter for issues table. Export all statuses by default.
// Opt out of BEADS_MAX_ROWS (designer §4.1) — export is a data-integrity
// path and must never abort partway through an export run.
filter := types.IssueFilter{
Limit: 0,
MaxRows: 0,
MaxRowsSource: "",
}
// Exclude infra types by default (agents, roles, messages).
if !exportAll && !exportIncludeInfra {
var infraTypes []string
if store != nil {
infraSet := store.GetInfraTypes(ctx)
if len(infraSet) > 0 {
for t := range infraSet {
infraTypes = append(infraTypes, t)
}
}
}
if len(infraTypes) == 0 {
infraTypes = domain.DefaultInfraTypes()
}
for _, t := range infraTypes {
filter.ExcludeTypes = append(filter.ExcludeTypes, types.IssueType(t))
}
}
// Exclude templates by default
if !exportAll {
isTemplate := false
filter.IsTemplate = &isTemplate
}
// Exclude ephemeral wisps by default — they are private/transient and
// must not reach git history or external integrations (GH#3649).
// --all overrides to include everything.
if !exportAll {
persistentOnly := false
filter.Ephemeral = &persistentOnly
}
issues, err := store.SearchIssues(ctx, "", filter)
if err != nil {
return HandleErrorRespectJSON("failed to search issues: %v", err)
}
// Scrub test/pollution records if requested
if exportScrub {
issues = filterOutPollution(issues)
}
// Owner-keyed filtering: exclude issues by created_by identity.
// Merges --exclude-owner flag values with export.exclude_owners config.
ownerExcludes := buildOwnerExcludeSet(ctx, exportExcludeOwners)
filteredOwnerCount := 0
if len(ownerExcludes) > 0 {
before := len(issues)
issues = filterOutOwners(issues, ownerExcludes)
filteredOwnerCount = before - len(issues)
}
if len(issues) == 0 && exportNoMemories {
if exportOutput != "" {
fmt.Fprintln(os.Stderr, "No issues to export.")
}
return nil
}
// Bulk-load relational data
issueIDs := make([]string, len(issues))
for i, issue := range issues {
issueIDs[i] = issue.ID
}
labelsMap, _ := store.GetLabelsForIssues(ctx, issueIDs)
allDeps, _ := store.GetDependencyRecordsForIssues(ctx, issueIDs)
commentsMap, _ := store.GetCommentsForIssues(ctx, issueIDs)
commentCounts, _ := store.GetCommentCounts(ctx, issueIDs)
depCounts, _ := store.GetDependencyCounts(ctx, issueIDs)
// Populate relational data on each issue
for _, issue := range issues {
issue.Labels = labelsMap[issue.ID]
issue.Dependencies = allDeps[issue.ID]
issue.Comments = commentsMap[issue.ID]
}
// Write JSONL: one JSON object per line
count := 0
for _, issue := range issues {
counts := depCounts[issue.ID]
if counts == nil {
counts = &types.DependencyCounts{}
}
// Sanitize zero-value timestamps that can't be marshaled to JSON.
// NULL datetime columns scanned as time.Time{} (year 0001) cause
// MarshalJSON to fail with "year outside of range [0,9999]". (GH#2488)
sanitizeZeroTime(issue)
record := &exportIssueRecord{
RecordType: "issue",
IssueWithCounts: &types.IssueWithCounts{
Issue: issue,
DependencyCount: counts.DependencyCount,
DependentCount: counts.DependentCount,
CommentCount: commentCounts[issue.ID],
},
}
data, err := json.Marshal(record)
if err != nil {
return HandleErrorRespectJSON("failed to marshal issue %s: %v", issue.ID, err)
}
if _, err := w.Write(data); err != nil {
return HandleErrorRespectJSON("failed to write: %v", err)
}
if _, err := w.Write([]byte{'\n'}); err != nil {
return HandleErrorRespectJSON("failed to write newline: %v", err)
}
count++
}
// Export memories only when explicitly requested (GH#3650).
// Memories may contain sensitive agent context and are excluded by default.
memoryCount := 0
if (exportIncludeMemories || exportAll) && !exportNoMemories {
allConfig, err := store.GetAllConfig(ctx)
if err != nil {
return HandleErrorRespectJSON("failed to read config for memories: %v", err)
}
fullPrefix := kvPrefix + memoryPrefix
// Sort keys for deterministic output order (GH#3474).
var memKeys []string
for k := range allConfig {
if strings.HasPrefix(k, fullPrefix) {
memKeys = append(memKeys, k)
}
}
sort.Strings(memKeys)
for _, k := range memKeys {
v := allConfig[k]
userKey := strings.TrimPrefix(k, fullPrefix)
record := map[string]string{
"_type": "memory",
"key": userKey,
"value": v,
}
data, err := json.Marshal(record)
if err != nil {
return HandleErrorRespectJSON("failed to marshal memory %s: %v", userKey, err)
}
if _, err := w.Write(data); err != nil {
return HandleErrorRespectJSON("failed to write: %v", err)
}
if _, err := w.Write([]byte{'\n'}); err != nil {
return HandleErrorRespectJSON("failed to write newline: %v", err)
}
memoryCount++
}
}
// Finalize atomic write if writing to file (fsync + rename).
if aw != nil {
if err := aw.Close(); err != nil {
return HandleErrorRespectJSON("failed to finalize export file: %v", err)
}
}
// Print summary to stderr (not stdout, to avoid mixing with JSONL)
if exportOutput != "" {
if memoryCount > 0 {
fmt.Fprintf(os.Stderr, "Exported %d issues and %d memories to %s\n", count, memoryCount, exportOutput)
} else {
fmt.Fprintf(os.Stderr, "Exported %d issues to %s\n", count, exportOutput)
}
if exportVerbose && filteredOwnerCount > 0 {
fmt.Fprintf(os.Stderr, " (%d filtered as personal by owner exclusion)\n", filteredOwnerCount)
}
}
return nil
}
// exportIssueRecord wraps IssueWithCounts with a _type discriminator so that
// every line in the JSONL export is self-describing. Memory lines already
// carry "_type":"memory"; this gives issue lines "_type":"issue". (GH#3271)
type exportIssueRecord struct {
RecordType string `json:"_type"`
*types.IssueWithCounts
}
// sanitizeZeroTime replaces Go zero-value time.Time fields with Unix epoch.
// NULL datetime columns in Dolt scan as time.Time{} (year 0001-01-01), which
// causes json.Marshal to fail with "year outside of range [0,9999]". (GH#2488)
func sanitizeZeroTime(issue *types.Issue) {
epoch := time.Unix(0, 0).UTC()
if issue.CreatedAt.IsZero() {
issue.CreatedAt = epoch
}
if issue.UpdatedAt.IsZero() {
issue.UpdatedAt = epoch
}
}
// filterOutPollution removes issues that look like test/pollution records.
func filterOutPollution(issues []*types.Issue) []*types.Issue {
var clean []*types.Issue
for _, issue := range issues {
if !isTestIssue(issue.Title) {
clean = append(clean, issue)
}
}
return clean
}
// buildOwnerExcludeSet merges --exclude-owner flag values with the
// export.exclude_owners (and legacy export.exclude_owner) config entries.
// Returns the combined set as a map for O(1) lookup.
func buildOwnerExcludeSet(ctx context.Context, flagOwners []string) map[string]struct{} {
set := make(map[string]struct{})
for _, o := range flagOwners {
if o != "" {
set[o] = struct{}{}
}
}
// export.* keys are YAML-only (config.IsYamlOnlyKey returns true for the
// "export." prefix), so bd config set stores them in config.yaml rather than
// the database. Read from YAML first, then fall back to the database for any
// instance that was written directly to the store.
addOwners := func(val string) {
for _, o := range strings.Split(val, ",") {
if o = strings.TrimSpace(o); o != "" {
set[o] = struct{}{}
}
}
}
if val := config.GetYamlConfig("export.exclude_owners"); val != "" {
addOwners(val)
}
if val := config.GetYamlConfig("export.exclude_owner"); val != "" {
set[strings.TrimSpace(val)] = struct{}{}
}
if store == nil {
return set
}
// Also read from database for any value stored there directly.
if val, err := store.GetConfig(ctx, "export.exclude_owners"); err == nil && val != "" {
addOwners(val)
}
if val, err := store.GetConfig(ctx, "export.exclude_owner"); err == nil && val != "" {
set[strings.TrimSpace(val)] = struct{}{}
}
return set
}
// filterOutOwners removes issues whose created_by identity is in the exclude set.
func filterOutOwners(issues []*types.Issue, exclude map[string]struct{}) []*types.Issue {
var keep []*types.Issue
for _, issue := range issues {
if _, excluded := exclude[issue.CreatedBy]; !excluded {
keep = append(keep, issue)
}
}
return keep
}