beads/scripts/embedded_shard_count_test.go
Julian Knutsen 3e7705a2cd
fix(bazel): remove host-go dep from compact GC test; drop frozen shard blocks (#7478)
* fix(bazel): remove host-go dep from compact GC test; drop frozen shard blocks

Replace the ungated `go build` in compact_gc_target_test.go's dolt-stand-in
fixture with a Bazel-built go_binary (cmd/bd:compact_gc_fixture) wired via
rlocationpath env + data, mirroring the existing preflight_go_fixture
pattern. Falls back to `go build` only when not running under Bazel
(bazeltest.IsBazel()). Also gate test_repo_beads_guard_test.go's `go env
GOCACHE`/`GOMODCACHE` probes on !bazeltest.IsBazel(), since they previously
ran unconditionally in every cmd/bd TestMain.

Separately, delete the legacy 20-shard embedded-cmd and 15-shard
proxied-cmd "frozen" manifest blocks in .github/scripts/*-test-shards.txt:
their only consumers were pr-risk.yml fork/push jobs that have since
migrated to bazel.yml. Update gen_embedded_cmd_shard_manifest.py and
gen_proxied_shard_manifest.py to drop the frozen-block rendering/guard
logic and default to the live Bazel-only shard counts (100 and 34),
and refresh stale comments/docs (cmd/bd/BUILD.bazel,
scripts/shard_manifest_test.go, engdocs/TESTING.md,
engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md) that still described the deleted
blocks. Verified no other BUILD/.github/scripts references read them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* build(bazel): auto-sync BUILD files

Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/38069936930
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.

* fix(bazel): gate remaining host-go calls; fix shard-manifest --weights footgun

Address PR #7478 review findings (bee-ghosttrack):

- scripts/ci/lib/test-env.sh: skip the GOCACHE/GOMODCACHE `go env` probes
  when TEST_SRCDIR is set, so scripts/test.sh's beads_test_env_enter never
  shells out to a bare `go` when reached from a Bazel-run go_test (e.g.
  //scripts:shell_scripts_test's TestTestScriptPrebuiltTestBinaryContract),
  where a farm masks host go/dolt to exit 127. Nothing on that path needs a
  cache directory.
- scripts/ci/fmt-check.sh: describe_gofmt() no longer calls a PATH `go`
  under Bazel. It now resolves BEADS_TEST_GO (the hermetic SDK go already
  wired into scripts/prlintmake:prlintmake_test's env) via TEST_SRCDIR, and
  skips the version-reporting arm entirely when that isn't set (e.g.
  scripts/repochecks:fmt_test), rather than ever touching a possibly-masked
  PATH `go`.
- Verified with a PATH stub (go/dolt scripts that log and exit 127) via
  `bazel --nohome_rc test --config=prcore //scripts/... //cmd/bd:bd_test
  --test_env=PATH=<stubdir>:/usr/bin:/bin:/usr/local/bin`: 26/26 targets
  pass, zero stub invocations logged.
- gen_proxied_shard_manifest.py / gen_embedded_cmd_shard_manifest.py:
  default --weights to duration now that the manifests' only remaining
  blocks are duration-packed (the legacy inits-packed blocks were deleted
  alongside the PR Risk fork/push jobs that read them). Previously the bare
  `--write` default (inits) silently clobbered the live block's duration
  header and repacked with the wrong cost model; --check couldn't see it.
  Added TestProxiedShardManifestBareWriteIsIdempotent and
  TestEmbeddedCmdShardManifestBareWriteIsIdempotent to pin a bare --write
  as a no-op against the committed files.
- cmd/bd/compact_gc_target_test.go: resolve the non-Bazel fallback fixture
  path from this file's own directory (runtime.Caller) instead of the test
  binary's working directory.
- Refresh stale comments in .github/workflows/bazel.yml,
  scripts/shard_manifest_test.go, scripts/pr_risk_bazel_coverage_test.go,
  and scripts/embedded_shard_count_test.go that still described the deleted
  legacy 20/15-shard blocks and their retired fork/push jobs in the present
  tense.
- Confirmed cmd/bd/BUILD.bazel's compact_gc_fixture go_binary (implicit
  pure/race defaults) still passes TestRunCompactDoltTargetsOnlyAuthorizedActiveDatabase
  under `bazel test //cmd/bd:bd_test`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-10 19:24:21 +00:00

159 lines
6.4 KiB
Go

package scripts_test
// The Bazel shard counts the CI policy tests (//scripts:scripts_test) and the
// shard-manifest tests (//scripts:go_test_sources_test) check against, read
// from each sharded target's own BUILD.bazel rule. Helpers only: both targets
// compile this file, so a test here would run twice.
import (
"fmt"
"regexp"
"strconv"
"testing"
)
// shardCountPattern matches a rule block's ` shard_count = N,` line, the
// same pattern bazelProxiedShardCount (below) inlines
// for its own single use; shared here since both embedded accessors below
// need it.
var shardCountPattern = regexp.MustCompile(`(?m)^ shard_count = (\d+),$`)
// embeddedCmdTargets are the cmd/bd rules that split the bazel-embedded
// lane's cmd/bd manifest block between them (more shards than Bazel's
// 50-per-rule cap), in shard order.
var embeddedCmdTargets = []string{"bd_embedded_test", "bd_embedded_part2_test"}
// embeddedCmdPart is one of embeddedCmdTargets: its Bazel shard_count and the
// range of the manifest block it runs (go_test_manifest_shard.sh's
// --shard-offset/--shard-total args).
type embeddedCmdPart struct {
label string
shards, offset, total int
}
// spec is the part's check_shard_coverage.py --suite SHARDS argument.
func (p embeddedCmdPart) spec() string {
return fmt.Sprintf("%d@%d/%d", p.shards, p.offset, p.total)
}
var (
shardOffsetArg = regexp.MustCompile(`(?m)^ "--shard-offset=(\d+)",$`)
shardTotalArg = regexp.MustCompile(`(?m)^ "--shard-total=(\d+)",$`)
)
// bazelEmbeddedCmdParts reads embeddedCmdTargets' shard_count and shard
// range from cmd/bd/BUILD.bazel and fails unless the ranges tile one block
// exactly once, in order.
func bazelEmbeddedCmdParts(t *testing.T) []embeddedCmdPart {
t.Helper()
build := readPolicyFile(t, sourceRepoRoot(t), "cmd/bd/BUILD.bazel")
var parts []embeddedCmdPart
next := 0
for _, name := range embeddedCmdTargets {
rule := bazelRuleBlock(build, name)
p := embeddedCmdPart{label: "//cmd/bd:" + name}
for _, f := range []struct {
re *regexp.Regexp
dst *int
what string
}{{shardCountPattern, &p.shards, "shard_count = N,"}, {shardOffsetArg, &p.offset, `"--shard-offset=N",`}, {shardTotalArg, &p.total, `"--shard-total=N",`}} {
m := f.re.FindStringSubmatch(rule)
if m == nil {
t.Fatalf("cmd/bd:%s has no %s in cmd/bd/BUILD.bazel:\n%s", name, f.what, rule)
}
n, err := strconv.Atoi(m[1])
if err != nil {
t.Fatalf("cmd/bd:%s %s: %v", name, f.what, err)
}
*f.dst = n
}
if p.offset != next || (len(parts) > 0 && p.total != parts[0].total) {
t.Fatalf("cmd/bd:%s runs shards %d..%d of %d; want it to start at shard %d of the same block as %v", name, p.offset+1, p.offset+p.shards, p.total, next+1, parts)
}
next += p.shards
parts = append(parts, p)
}
if next != parts[0].total {
t.Fatalf("%v run %d shards of a %d-shard block; their shard_counts must sum to --shard-total", parts, next, parts[0].total)
}
return parts
}
// bazelEmbeddedCmdShardCount returns the shard total of the bazel-embedded
// lane's cmd/bd manifest block, which embeddedCmdTargets split between them
// (bazelEmbeddedCmdParts): the single source of truth for the Bazel-only
// lane's cmd/bd shard split (slice F1). It no longer has to equal PR
// Risk's/main.yml's legacy "Test (Embedded Dolt Cmd N/20)" fork/push jobs'
// matrix size — those jobs, and the frozen 20-shard block they read, have
// since been deleted (see engdocs/TESTING.md) — mirrors bazelProxiedShardCount
// (F2; see that function's doc comment below for the shared rationale, not
// repeated here).
func bazelEmbeddedCmdShardCount(t *testing.T) int {
t.Helper()
return bazelEmbeddedCmdParts(t)[0].total
}
// bazelEmbeddedStorageShardCount returns
// embeddeddolt:embeddeddolt_embedded_test's own shard_count from
// internal/storage/embeddeddolt/BUILD.bazel: the single source of truth for
// the Bazel-only bazel-embedded lane's storage shard split (slice F1), which
// no longer has to equal PR Risk's/main.yml's legacy "Test (Embedded Dolt
// Storage N/5)" fork/push jobs' matrix size. Mirrors
// bazelEmbeddedCmdShardCount above; see its doc comment.
func bazelEmbeddedStorageShardCount(t *testing.T) int {
t.Helper()
root := sourceRepoRoot(t)
rule := bazelRuleBlock(readPolicyFile(t, root, "internal/storage/embeddeddolt/BUILD.bazel"), "embeddeddolt_embedded_test")
m := shardCountPattern.FindStringSubmatch(rule)
if m == nil {
t.Fatalf("embeddeddolt:embeddeddolt_embedded_test has no `shard_count = N,` in internal/storage/embeddeddolt/BUILD.bazel:\n%s", rule)
}
n, err := strconv.Atoi(m[1])
if err != nil {
t.Fatalf("embeddeddolt:embeddeddolt_embedded_test shard_count: %v", err)
}
return n
}
// bazelServerFullShardCount returns dolt:dolt_server_full_test's own
// shard_count from internal/storage/dolt/BUILD.bazel: the single source of
// truth for the bazel-server-storage lane's full-suite shard split, now that
// PR Risk's legacy "Test (Server Dolt Full Suite N/16)" job, whose matrix it
// used to mirror, is retired (ga-96smfk.22). Mirrors
// bazelEmbeddedCmdShardCount above.
func bazelServerFullShardCount(t *testing.T) int {
t.Helper()
root := sourceRepoRoot(t)
rule := bazelRuleBlock(readPolicyFile(t, root, "internal/storage/dolt/BUILD.bazel"), "dolt_server_full_test")
m := shardCountPattern.FindStringSubmatch(rule)
if m == nil {
t.Fatalf("dolt:dolt_server_full_test has no `shard_count = N,` in internal/storage/dolt/BUILD.bazel:\n%s", rule)
}
n, err := strconv.Atoi(m[1])
if err != nil {
t.Fatalf("dolt:dolt_server_full_test shard_count: %v", err)
}
return n
}
// The default shard manifest of a PR Risk shard script.
var shardManifestDefault = regexp.MustCompile(`\$\{BEADS_TEST_SHARD_MANIFEST:-([^}]+)\}`)
// bazelProxiedShardCount returns cmd/bd:bd_proxied_test's own shard_count
// from cmd/bd/BUILD.bazel: the single source of truth for the Bazel-only
// bazel-proxied lane's shard split, which no longer has to equal PR
// Risk's/main.yml's legacy test-proxied-cmd jobs' matrix size (F2).
func bazelProxiedShardCount(t *testing.T) int {
t.Helper()
root := sourceRepoRoot(t)
rule := bazelRuleBlock(readPolicyFile(t, root, "cmd/bd/BUILD.bazel"), "bd_proxied_test")
m := regexp.MustCompile(`(?m)^ shard_count = (\d+),$`).FindStringSubmatch(rule)
if m == nil {
t.Fatalf("cmd/bd:bd_proxied_test has no `shard_count = N,` in cmd/bd/BUILD.bazel:\n%s", rule)
}
n, err := strconv.Atoi(m[1])
if err != nil {
t.Fatalf("cmd/bd:bd_proxied_test shard_count: %v", err)
}
return n
}