beads/cmd/bd/max_rows_test.go
Julian Knutsen ef988df84b
fix(ci,tests): un-hide TestEmbedded* tests, fence cmd/bd tests from ancestor .beads, cut #7051 race long poles, fix main lint (#7128)
* fix(lint): drop unparam findings in primeGitCmd and expandStep

primeGitCmd's error return was always nil (GH#4927 made every path
infallible but left the signature); drop it and update both call sites.

expandStep's depth parameter was always called with 0 from every caller;
recursion depth is tracked internally by expandStepWith via its own
depth+1 calls on children, so the parameter was dead. Hardcode the 0
start inside expandStep and drop the parameter.

Fixes golangci-lint unparam findings:
  cmd/bd/prime.go:306:67: primeGitCmd - result 1 (error) is always nil
  internal/formula/expand.go:197:49: expandStep - depth always receives 0

make ci-pr-lint is clean (native + windows/amd64 + darwin/arm64 non-CGO).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(dolt): give each TestBatchApplyContract subtest its own context

TestBatchApplyContract's subtests all shared one 90s context built once
in newDoltBatchApplyFixture. BoundsTheItemCount legitimately needs most
of that budget against the server-Dolt tier (ApplyBatchInTx's create
path calls ExecuteCreate once per item, ~5-8 round trips each, for all
1000 items at the MaxApplyBatchItems bound raised by #7051). Once
earlier subtests had spent part of the shared budget, BoundsTheItemCount
would blow the deadline and every subtest that ran after it in the same
parent would fail too with "context deadline exceeded", masking the
real failure behind unrelated red subtests (nightly run 37089445362,
shard 2/16: BoundsTheItemCount at 78.9s, deadline exceeded).

Give each t.Run its own fresh testContext(t) instead of sharing one,
mirroring openStoreWithOwnBudget's established rationale for the same
bug class (be-gvnsq). The store and its copy-on-write branch stay
shared across subtests, and subtests still run in sequence; only the
per-subtest deadline is decoupled. No production code changed: the
underlying per-item round-trip cost is genuine, documented, deliberate
behavior (see applyCreate's doc comment), and batching it safely would
be a larger, riskier change than this CI flake warrants.

Verified via bazel --nohome_rc test --config=doltserver-integration
//internal/storage/dolt:dolt_server_full_test --test_filter=TestBatchApplyContract
-t- --test_output=all: shard 2/16 (same shard as the original failure)
now passes, TestBatchApplyContract in 77.68s with BoundsTheItemCount at
66.08s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(embeddeddolt): skip redundant 1000-item wall-clock case under -race

TestLargeBatchApplyWallClock_Embedded's 1000-item shape was taking
597-651s under --config=embedded (-race), inflated by race-instrumenting
the in-process Dolt engine's own internal goroutine/lock machinery on
every statement issueops.ApplyBatchInTx issues. The Dolt server (TCP)
tier doesn't pay this cost: its sql-server runs in a separate,
non-raced subprocess (dolt_race_off), but the embedded backend runs
Dolt entirely inside this test binary, so -race instruments all of it.

This pushed the embedded-storage CI lane's shard 5 over its 19-minute
per-shard test timeout (panic: test timed out after 19m0s).

The 1000-item case carries no pass/fail assertion on duration or count
(the test is explicitly a logging-only measurement, not a regression
gate), so skipping it under race removes no coverage. Real 1000-item
(issueops.MaxApplyBatchItems) correctness coverage is unaffected: it
still runs and is asserted by TestBatchApplyContract/BoundsTheItemCount
in this same package, under the same race lane.

Adds the established raceEnabled build-tag pair (see
internal/workapi and cmd/bd for the sibling idiom) to this package so
the skip can be conditioned on -race.

Verified via Bazel (--config=embedded, --nohome_rc):
  TestLargeBatchApplyWallClock_Embedded: 651.23s -> 86.79s
  (1000 now SKIPped; 356 still runs and passes)
  Shard 5 of embeddeddolt_embedded_test: FAIL -> PASS

Not fixed in this commit: TestBatchApplyContract/BoundsTheItemCount's
embedded instantiation still takes ~903-945s under -race and leaves
shard 4 of embeddeddolt_embedded_test over its 19-minute budget. Unlike
the wall-clock test, that subtest carries real correctness assertions
via the shared backend/conformance.RunBatchApplyBoundsTheItemCount, and
shrinking it would require a cross-backend conformance-test design
decision (e.g. a per-fixture item-count override) that trades away real
MaxApplyBatchItems boundary coverage for one backend - judged out of
scope for an isolated test-only fix here. Flagging for a deliberate
follow-up rather than weakening the assertion to make it fast.

TestLargeBatchApplyStatementCounts_Embedded and the dolt-tier
TestLargeBatchApplyStatementCounts_Dolt were checked and left
untouched: neither includes the 1000-item shape, neither showed any
slowness or flakiness, and both pin exact statement-count regression
baselines that should not be disturbed without cause.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(cmd/bd): fence test-binary beads/git discovery against ancestor .beads

cmd/bd's test binary ambiently discovers its beads workspace the same way
production bd does: via internal/beads.FindBeadsDir's worktree-fallback and
ancestor-walk logic. When this package's checkout is itself a linked git
worktree (e.g. a sibling checkout like beads-bazel-y next to beads), the
first cached git-context call made anywhere in the test binary - before any
individual test has had a chance to chdir into its own isolated fixture -
permanently answers isWorktree=true/a real --git-common-dir for the rest of
the process. beads.FindBeadsDir then treats that as license to read (and
tests that don't isolate properly could write) the main checkout's shared
.beads database.

Confirmed via direct reproduction in a disposable scratch main+worktree pair
under /data/tmp (never /data/projects/beads): running a correctly-built `bd
--version` from the worktree checkout's cmd/bd directory resolves and warns
about the fake main checkout's .beads permissions, demonstrating the
fallback is reachable from plain ambient discovery, not just from a
contrived unit test.

internal/git.PinNoRepositoryForTesting seeds the process-wide git context
cache with a "not a git repository" sentinel as the first statement of
testMainInner, making the worktree-fallback vector inert by default. Tests
that need real git behavior for a fixture they control already call
git.ResetCaches/beads.ResetCaches after chdir'ing there (the existing
cmd/bd/git_test_helpers.go idiom), which overrides the pin for their own
scope - this is unchanged.

Separately, testMainInner now appends (never overwrites) a
BEADS_CEILING_DIRECTORIES entry for the package's own repo root, bounding
any literal ancestor-directory walk that starts at or under this checkout -
closing a second vector (an ancestor walk escaping past a nested worktree
root into a directory-with-.beads further up) that the pin alone doesn't
address. The existing pollution guard is extended to also watch the
worktree-fallback .beads directory (resolved via a direct, cache-independent
git shell-out) as a defense-in-depth detection backstop.

Verified via two full before/after `go test -tags "cgo integration" -v .`
runs of cmd/bd in a disposable linked-worktree scratch clone: identical
pass/fail sets before and after (no regressions), and zero filesystem
writes observed (inotify-monitored) in the scratch main checkout's .beads
in either run - the existing subprocess test helpers already discipline
themselves to set an explicit cmd.Dir/BEADS_DIR, so no currently-live test
was caught actively polluting the fallback directory in this repro. This
change is defense-in-depth against the mechanism itself (and against
regressions reintroducing it), rather than a fix for an empirically
isolated failing test in the current suite.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): make embedded-dolt and go:embed tests discoverable by their CI shards

embedded-test-shard.sh discovers cmd/bd's embedded-dolt tier by grepping
`^func TestEmbedded` out of cmd/bd/*_embedded_test.go — a filename-glob
match, not a package-wide one. 11 TestEmbedded* functions lived in files
that didn't match that glob (auto_import_upgrade_test.go,
dep_list_cross_db_test.go, and four files mixing TestEmbedded* with
unrelated TestProxiedServer*/TestServerMode*/TestWithFetchOneExtra_*
siblings: max_rows_test.go, store_factory_test.go,
purge_wisps_plane_integration_test.go, types_create_agreement_test.go).
They compiled and would run under a plain `go test`, but were invisible
to every CI lane that drives this tier off that discovery (the 20-shard
embedded cmd matrix), so they never ran in CI at all.

Fixed by relocating them into *_embedded_test.go files: a straight
`git mv` for the two files that were purely TestEmbedded* content, and a
function-level split for the four mixed files (moving just the
TestEmbedded* function itself; shared helpers/types stay put since
Go doesn't care which file in a package defines a symbol). The newly
discoverable tests aren't in embedded-cmd-test-shards.txt — consistent
with the ~55 pre-existing discovered-but-unlisted tests already in this
suite, they fall back to the manifest's documented hash(name)%shards
assignment. Verified via `BEADS_TEST_SHARD_LIST_ONLY=1` across all 20
shards: all 212 tests (201 + 11) are now listed exactly once, no shard
comes up empty. Also verified for real: built the cmd/bd test binary
and ran all 11 against a prebuilt bd binary (BEADS_TEST_BD_BINARY) from
the repo root, matching how CI invokes the prebuilt CMD_BINARY — all 11
pass.

Separately, 5 functions had a `TestEmbedded` prefix for an unrelated
reason — go:embed'd content (agents_test.go's AGENTS.md template tests,
schema_test.go's migration-source duplicate-version check) — and so were
being silently caught and skipped by the repo-wide `-skip '^TestEmbedded'`
pattern PR Core/main CI lanes use to exclude the Dolt-embedded tier from
./... runs. Renamed to a TestGoEmbed* prefix so they run where every
other unit test in their packages does; no other references to the old
names exist in the repo. Confirmed with `-skip '^TestEmbedded'` locally:
all 5 now run (previously would have been silently skipped).

Repo-wide grep for stray `^func TestEmbedded` outside cmd/bd,
internal/storage/embeddeddolt and internal/storage/dolt now returns only
internal/tracker's TestEmbeddedDoltExternalRefChangedAfterUsesHistoryFastPath,
which is genuinely about the Dolt-embedded backend and correctly named.

cmd/bd/BUILD.bazel regenerated via `bazel run //:gazelle` to reflect the
renames/new files; it also picked up internal/ceiling as a go_test dep,
a sync gap left over from test_repo_beads_guard_test.go's import in the
prior commit on this branch.

Verified: go build/vet (default, -tags cgo) clean; gofmt clean; go test
./scripts/...; bazel --nohome_rc test //scripts:scripts_test; bazel
--nohome_rc build //cmd/bd:bd_embedded_test; bazel --nohome_rc run
//:gazelle (cmd/bd/BUILD.bazel diff only); python3 tools/bazel/go_srcs.py
--check (make bazel-sync-check); make ci-pr-policy; make ci-pr-lint
(gofmt + golangci-lint native/windows/darwin, 0 issues). cmd/bd test
binary runs were built from cmd/bd (not the module root) and executed
with BEADS_TEST_BD_BINARY set to a binary built the same way, to avoid
buildEmbeddedBD's `go build -o X .` resolving "." against the wrong
package when invoked outside `go test`'s own working directory — not a
bug in this change, reproduced identically against an untouched,
already-correctly-named test (TestEmbeddedVersion). /data/projects/beads/.beads
checked before and after: no change beyond the pre-existing 166 untracked
.beads/formulas/*.toml files already present at session start.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(embeddeddolt): shrink BoundsTheItemCount's real apply under -race

TestBatchApplyContract/BoundsTheItemCount's embedded instantiation
measured 903-945s under -race (shard 4's single biggest long pole),
because the in-process cgo Dolt engine's own goroutine/lock machinery
runs inside the race-instrumented binary for a full 1000-item apply.

Every backend's ApplyBatch validates via the same storage.PlanApplyBatch
before opening any transaction, so the refusal half of this subtest is
already O(1) regardless of count, and the exact 1000/1001 boundary is
already pinned, for real and sub-millisecond, by
TestPlanApplyBatchAcceptsTheBoundary in internal/storage/batch_apply_test.go.
That leaves only "does this backend's own write body durably land N
accepted rows" as the expensive, backend-specific thing this subtest's
"at bound" half needs N real items for - and N does not need to be 1000
to prove that.

Split RunBatchApplyBoundsTheItemCount into a thin wrapper (unchanged,
still used by internal/storage/dolt and internal/storage/uow at the
full MaxApplyBatchItems) plus a new RunBatchApplyBoundsTheItemCountAtScale
taking an explicit atBoundItems. Embedded's subtest now applies 150 items
under -race (65.33s locally, down from 749.22s unmodified on the same
host) and the full 1000 otherwise, via the existing raceEnabled idiom.

internal/storage/dolt's non-race BoundsTheItemCount still covers the
shared inner write body at the full 1000-item count every PR. No CI lane
covers embedded's own post-commit wrapper at 1000 items anymore after
this change (the only non-race embeddeddolt build is compile-only and
never executed) - a known, reported, deliberate gap, documented in both
the updated large_batch_apply_measure_test.go comment and the task's
impl-report.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(git): make test-repository pin survive ResetCaches

PinNoRepositoryForTesting wrote its "no repository" sentinel into the
same gitCtx variable ResetCaches() unconditionally clears, so any test
fixture that calls ResetCaches() after the pin is set (runInDir's
cleanup, resetRepoCachesForTest, 25+ call sites) silently un-pins the
whole test binary for every later test. A reviewer's probe test proved
the bypass: GetRepoRoot() returned the real checkout path after a single
runInDir round-trip.

Replace it with PinNoRepositoryUnderForTesting(root), a separate,
directory-scoped pin checked live against os.Getwd() on every
getGitContext() call, before the gitCtxOnce/gitCtx one-shot cache is
touched at all. The pin survives ResetCaches() for any working directory
at-or-under the pinned root, while fixtures that chdir OUTSIDE that root
(the normal t.TempDir() pattern) still get real git detection.

Add TestPinNoRepositoryUnderForTestingSurvivesResetCaches, reproducing
the reviewer's exact repro and proving the fix: pins a real repo root,
confirms unpinned after chdir+ResetCaches outside the root, and confirms
STILL pinned after chdir+ResetCaches back inside the root (mimicking
runInDir's cleanup).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(cmd/bd): widen guard fallback coverage, document -C/doctor gap

Two review follow-ups on the test-repo .beads guard:

1. Honest scope documentation: FindBeadsDirFrom/ResolveBeadsDirForRepo's
   -C/doctor paths shell out to git directly with the ceiling env var
   stripped, so PinNoRepositoryUnderForTesting (see previous commit) does
   not fence them. Document this residual gap explicitly rather than
   implying full coverage; covering it would need those call sites to
   consult the pin too, which is out of scope here.

2. The guard's fallback-.beads watch list only watched metadata.json and
   skipped watching entirely when the main .beads directory was absent —
   exactly the layout where the worktree fallback is actually in use.
   Restructure to check the main and fallback directories independently,
   add config.yaml/routes.jsonl to the flat watch list, and add a
   recursive snapshot of dolt/ (snapshotDoltTree) so a real pollution
   write anywhere under the fallback's Dolt-backed state is caught.
   diffSnapshots is generalized to also report newly-appeared keys, since
   a recursive tree snapshot can legitimately gain entries between
   before/after in ways the old fixed-name flat list never could.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): pin MaxRows embedded tests off their shared race shard

TestEmbeddedMaxRowsNonListPaths and TestEmbeddedMaxRowsList both hashed
to embedded cmd shard 16, which measured 746s wall under -race
(605s + 214s) against a 19-20 minute budget on a 192-core host with
effectively unbounded test.parallel; CI's 4-way parallelism would be
worse. Neither test had ever run under race in CI before this branch
surfaced them to their own shards.

Both tests spawn many independent bd subprocesses per subtest
(bdInit + exec.Command), so their cost is dominated by subtest/subprocess
count rather than a single large-N parameter; buildEmbeddedBD also never
passes -race to the spawned bd binary, so there is no raceEnabled-style
"shrink N" lever analogous to BoundsTheItemCount's that would apply here
without just deleting coverage. Pin them to shards 11 and 13 instead,
the two lightest shards by current manifest+hash-fallback test count
(verified via BEADS_TEST_SHARD_LIST_ONLY=1, which lists shard
composition without executing any tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(embeddeddolt): correct coverage comments, range-check, document build tags

- large_batch_apply_measure_test.go's doc comment above
  TestLargeBatchApplyWallClock_Embedded had three factual errors: the
  150-item sibling subtest does not always run at 150 "race or not" (it's
  1000 without -race); the 356-item shape is not "non-race only" (it runs
  under race too, as the same comment's next line said); and the
  server-Dolt full suite does not run on "every PR" (full_embedded in
  .github/scripts/ci-embedded-tier.sh is unconditional on merge_group/push
  but conditional on PRs, per pr-risk.yml's gating). Rewrote the comment to
  be self-contained (no out-of-repo citation) and accurate, and reworded
  the related t.Skip message. Fixed the same "every PR" overstatement in
  batch_apply_contract_test.go's BoundsTheItemCount race-branch comment.
- Added a range-check to
  conformance.RunBatchApplyBoundsTheItemCountAtScale: atBoundItems must be
  1..publicops.MaxApplyBatchItems, since that's the production bound this
  contract pins.
- Documented why race_detector_on_test.go/race_detector_off_test.go use
  `cgo && race`/`cgo && !race` instead of the plain `race`/`!race` pair
  used by internal/workapi and cmd/bd: every consumer of raceEnabled here
  is itself cgo-gated, so the matching cgo tag keeps this file out of
  non-cgo builds too, where it would otherwise compile with no consumer
  and trip an unused-lint finding.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(dolt): give BoundsTheItemCount 3x timeout budget

The server-Dolt tier's BoundsTheItemCount subtest measured 66s against
the package's shared 90s testTimeout — a thin margin on a loaded CI
runner. Give it a dedicated 3x (270s) context.WithTimeout instead of
raising the shared testTimeout constant, since that constant also scales
every other subtest's 5*testTimeout cleanup bound package-wide; a local
override keeps the blast radius to the one expensive subtest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(ci): run embedded-Dolt batch-apply suite non-race in nightly

The nightly full-test job's main suite (go test -race ... ./...) never
sets BEADS_TEST_EMBEDDED_DOLT, so every skipUnlessEmbeddedDolt-gated test
in internal/storage/embeddeddolt — including TestBatchApplyContract and
the 1000-item case of TestLargeBatchApplyWallClock_Embedded — is skipped
on every CI lane, not just this one. Add a dedicated non-race step so a
real 1000-item apply through the embedded backend's own
version-commit-published-after-the-tx wrapper gets exercised somewhere:
non-race because the embedded engine's own goroutine/lock machinery runs
inside this test binary and the race detector dramatically inflates its
wall-clock (see race_detector_on_test.go), which would defeat the wall-
clock test's purpose and reintroduce the kind of shard-timeout pressure
the MaxRows shard-pin commit just worked around elsewhere.

Verified locally from a fresh /tmp clone (outside any .beads ancestor,
per this task's safety rules):
BEADS_TEST_EMBEDDED_DOLT=1 go test -tags gms_pure_go -run '^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded)$' ./internal/storage/embeddeddolt
PASS, 178.3s total (BoundsTheItemCount 81.06s, WallClock/1000 63.14s),
comfortably inside the job's 45-minute timeout.

Add TestNightlyFullTestRunsEmbeddedDoltBatchApplySuite to
scripts/ci_workflow_test.go, pinning the step's exact run command, its
BEADS_TEST_EMBEDDED_DOLT=1 env, that it runs after the main suite step,
and that it stays non-race.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 04:18:14 -07:00

64 lines
2.6 KiB
Go

//go:build cgo
// be-u8z9: unit tests for workapi.WithFetchOneExtra, the probe-row bump
// helper used by the CLI-layer --max-rows / BEADS_MAX_ROWS enforcement
// exercised end-to-end in max_rows_embedded_test.go.
package main
import (
"testing"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/internal/workapi"
)
func TestWithFetchOneExtra_LimitEqualsCap_BumpsBothForTruncationProbe(t *testing.T) {
got := workapi.WithFetchOneExtra(types.IssueFilter{Limit: 5, MaxRows: 5, MaxRowsSource: "--max-rows"})
if got.Limit != 6 {
t.Errorf("Limit == MaxRows: Limit = %d, want 6 (bumped so the query still fetches the truncation-detection probe row)", got.Limit)
}
if got.MaxRows != 6 {
t.Errorf("Limit == MaxRows: MaxRows = %d, want 6 (bumped in lockstep so the probe row alone doesn't trip EnforceMaxRowsCap)", got.MaxRows)
}
if got.MaxRowsSource != "--max-rows" {
t.Errorf("MaxRowsSource must be preserved unchanged, got %q", got.MaxRowsSource)
}
}
// TestWithFetchOneExtra_LimitOverCap_OnlyBumpsLimit covers the tighter-cap
// case (--limit 100 --max-rows 5, LimitSet_CapTighter): MaxRows must NOT
// bump, or a genuine cap violation would report the wrong Cap value (N+1
// instead of the user's true --max-rows=N) in the error message.
func TestWithFetchOneExtra_LimitOverCap_OnlyBumpsLimit(t *testing.T) {
got := workapi.WithFetchOneExtra(types.IssueFilter{Limit: 100, MaxRows: 5})
if got.Limit != 101 {
t.Errorf("Limit = %d, want 101", got.Limit)
}
if got.MaxRows != 5 {
t.Errorf("MaxRows must stay unbumped so a real cap violation reports the true cap; got %d, want 5", got.MaxRows)
}
}
// TestWithFetchOneExtra_LimitUnderCap_OnlyBumpsLimit covers the
// looser-cap case (--limit 5 --max-rows 100, LimitSet_CapLooser): the
// probe-row bump alone never crosses EffectiveSearchLimit's `limit >
// maxRows` branch here, so no MaxRows adjustment is needed.
func TestWithFetchOneExtra_LimitUnderCap_OnlyBumpsLimit(t *testing.T) {
got := workapi.WithFetchOneExtra(types.IssueFilter{Limit: 5, MaxRows: 100})
if got.Limit != 6 {
t.Errorf("Limit = %d, want 6", got.Limit)
}
if got.MaxRows != 100 {
t.Errorf("MaxRows must stay unbumped, got %d, want 100", got.MaxRows)
}
}
// TestWithFetchOneExtra_NoLimit_Unaffected covers the unlimited case
// (Limit == 0): workapi.WithFetchOneExtra is a no-op regardless of MaxRows.
func TestWithFetchOneExtra_NoLimit_Unaffected(t *testing.T) {
got := workapi.WithFetchOneExtra(types.IssueFilter{Limit: 0, MaxRows: 5})
if got.Limit != 0 || got.MaxRows != 5 {
t.Errorf("unlimited Limit must pass through unchanged, got Limit=%d MaxRows=%d", got.Limit, got.MaxRows)
}
}