beads/scripts/shard_manifest_test.go
Julian Knutsen 3e7705a2cd
fix(bazel): remove host-go dep from compact GC test; drop frozen shard blocks (#7478)
* fix(bazel): remove host-go dep from compact GC test; drop frozen shard blocks

Replace the ungated `go build` in compact_gc_target_test.go's dolt-stand-in
fixture with a Bazel-built go_binary (cmd/bd:compact_gc_fixture) wired via
rlocationpath env + data, mirroring the existing preflight_go_fixture
pattern. Falls back to `go build` only when not running under Bazel
(bazeltest.IsBazel()). Also gate test_repo_beads_guard_test.go's `go env
GOCACHE`/`GOMODCACHE` probes on !bazeltest.IsBazel(), since they previously
ran unconditionally in every cmd/bd TestMain.

Separately, delete the legacy 20-shard embedded-cmd and 15-shard
proxied-cmd "frozen" manifest blocks in .github/scripts/*-test-shards.txt:
their only consumers were pr-risk.yml fork/push jobs that have since
migrated to bazel.yml. Update gen_embedded_cmd_shard_manifest.py and
gen_proxied_shard_manifest.py to drop the frozen-block rendering/guard
logic and default to the live Bazel-only shard counts (100 and 34),
and refresh stale comments/docs (cmd/bd/BUILD.bazel,
scripts/shard_manifest_test.go, engdocs/TESTING.md,
engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md) that still described the deleted
blocks. Verified no other BUILD/.github/scripts references read them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* build(bazel): auto-sync BUILD files

Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/38069936930
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.

* fix(bazel): gate remaining host-go calls; fix shard-manifest --weights footgun

Address PR #7478 review findings (bee-ghosttrack):

- scripts/ci/lib/test-env.sh: skip the GOCACHE/GOMODCACHE `go env` probes
  when TEST_SRCDIR is set, so scripts/test.sh's beads_test_env_enter never
  shells out to a bare `go` when reached from a Bazel-run go_test (e.g.
  //scripts:shell_scripts_test's TestTestScriptPrebuiltTestBinaryContract),
  where a farm masks host go/dolt to exit 127. Nothing on that path needs a
  cache directory.
- scripts/ci/fmt-check.sh: describe_gofmt() no longer calls a PATH `go`
  under Bazel. It now resolves BEADS_TEST_GO (the hermetic SDK go already
  wired into scripts/prlintmake:prlintmake_test's env) via TEST_SRCDIR, and
  skips the version-reporting arm entirely when that isn't set (e.g.
  scripts/repochecks:fmt_test), rather than ever touching a possibly-masked
  PATH `go`.
- Verified with a PATH stub (go/dolt scripts that log and exit 127) via
  `bazel --nohome_rc test --config=prcore //scripts/... //cmd/bd:bd_test
  --test_env=PATH=<stubdir>:/usr/bin:/bin:/usr/local/bin`: 26/26 targets
  pass, zero stub invocations logged.
- gen_proxied_shard_manifest.py / gen_embedded_cmd_shard_manifest.py:
  default --weights to duration now that the manifests' only remaining
  blocks are duration-packed (the legacy inits-packed blocks were deleted
  alongside the PR Risk fork/push jobs that read them). Previously the bare
  `--write` default (inits) silently clobbered the live block's duration
  header and repacked with the wrong cost model; --check couldn't see it.
  Added TestProxiedShardManifestBareWriteIsIdempotent and
  TestEmbeddedCmdShardManifestBareWriteIsIdempotent to pin a bare --write
  as a no-op against the committed files.
- cmd/bd/compact_gc_target_test.go: resolve the non-Bazel fallback fixture
  path from this file's own directory (runtime.Caller) instead of the test
  binary's working directory.
- Refresh stale comments in .github/workflows/bazel.yml,
  scripts/shard_manifest_test.go, scripts/pr_risk_bazel_coverage_test.go,
  and scripts/embedded_shard_count_test.go that still described the deleted
  legacy 20/15-shard blocks and their retired fork/push jobs in the present
  tense.
- Confirmed cmd/bd/BUILD.bazel's compact_gc_fixture go_binary (implicit
  pure/race defaults) still passes TestRunCompactDoltTargetsOnlyAuthorizedActiveDatabase
  under `bazel test //cmd/bd:bd_test`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-10 19:24:21 +00:00

335 lines
16 KiB
Go

package scripts_test
// The shard manifests and shard scripts against the Go tests they shard:
// //scripts:go_test_sources_test, which reads the _test.go files of the
// sharded packages (and no non-test Go source), so an edit to a workflow file,
// a doc or a non-test Go file leaves it cached.
import (
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"testing"
)
// S3 (F1, mirroring F2's TestProxiedShardManifestGeneratorNotStale below —
// see that test's doc comment for the full --check rationale, not repeated
// here): the Bazel-only 100-shard cmd block and 40-shard storage block.
// gen_embedded_{cmd,storage}_shard_manifest.py --check verifies only that
// the committed block names every discovered test exactly once, failing with
// the exact command to fix it when a name is missing, stale, or duplicated —
// run here so a drifted block fails `go test ./scripts/...`
// (//scripts:go_test_sources_test under Bazel) instead of only surfacing as
// a test silently never running in any shard. The cmd file's legacy,
// hand-assigned 20-shard block (frozen for PR Risk's and main.yml's retired
// fork/push jobs) has been deleted; those jobs no longer exist (see
// engdocs/TESTING.md). The storage file's legacy 5-shard block is still
// frozen and deliberately excluded here: its header documents that (see
// .github/scripts/embedded-storage-test-shards.txt and engdocs/TESTING.md),
// so a --check against it is expected to report "missing" entries by design
// (see that generator's module docstring) and is not what this test runs.
func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) {
python := requireHostTool(t, "python3")
root := sourceRepoRoot(t)
shards := strconv.Itoa(bazelEmbeddedCmdShardCount(t))
cmd := exec.Command(python, "scripts/ci/gen_embedded_cmd_shard_manifest.py", shards, "--weights=duration", "--check")
cmd.Dir = root
if out, err := cmd.CombinedOutput(); err != nil {
t.Errorf("gen_embedded_cmd_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out)
}
}
// TestPackConcurrentModelsSerialAndParallelTests pins
// _embedded_shard_manifest_lib.py's pack_concurrent: a serial test (no
// top-level t.Parallel) holds its shard alone for its whole duration, while
// parallel tests share -test.parallel slots, so four 30s parallel tests fit
// beside a 90s one in the same wall time. Plain LPT would put the serial
// test with a parallel one and split the rest evenly by summed duration.
func TestPackConcurrentModelsSerialAndParallelTests(t *testing.T) {
python := requireHostTool(t, "python3")
root := sourceRepoRoot(t)
const script = `
import sys
sys.path.insert(0, 'scripts/ci')
from _embedded_shard_manifest_lib import pack_concurrent
costs = {'serial': 100, 'a': 90, 'b': 80, 'c': 30, 'd': 30, 'e': 30, 'f': 30}
shards, walls = pack_concurrent(costs, 2, {'serial'}, 4)
print(sorted(map(sorted, shards)), walls)
`
cmd := exec.Command(python, "-c", script)
cmd.Dir = root
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("pack_concurrent: %v\n%s", err, out)
}
const want = "[['a', 'b', 'c', 'd', 'e', 'f'], ['serial']] [100.0, 90]\n"
if string(out) != want {
t.Errorf("pack_concurrent = %q, want %q", out, want)
}
}
// TestStorageEmbeddedShardManifestGeneratorNotStale mirrors
// TestCmdEmbeddedShardManifestGeneratorNotStale above for the storage tier's
// Bazel-only 20-shard block; see that test's doc comment.
func TestStorageEmbeddedShardManifestGeneratorNotStale(t *testing.T) {
python := requireHostTool(t, "python3")
root := sourceRepoRoot(t)
shards := strconv.Itoa(bazelEmbeddedStorageShardCount(t))
cmd := exec.Command(python, "scripts/ci/gen_embedded_storage_shard_manifest.py", shards, "--weights=duration", "--check")
cmd.Dir = root
if out, err := cmd.CombinedOutput(); err != nil {
t.Errorf("gen_embedded_storage_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out)
}
}
// Review G1: the checker's input is the real shard scripts' list-only
// output (every retired lane's: the embedded, proxied and server suites). Every name they list, for every shard, must be a test go test
// runs (declared `func Name(t *testing.T)` in the package's _test.go files),
// or one check_shard_coverage.py drops as NOT_TESTS; and each NOT_TESTS name
// must really not be a test (TestMain takes *testing.M). Otherwise the
// checker reports a listed test that "did not run" on every real run.
func TestShardScriptsListOnlyRealTests(t *testing.T) {
// The CI shard scripts use bash 4 associative arrays (Linux runners only).
requireAutofixBash(t)
root := sourceRepoRoot(t)
m := regexp.MustCompile(`(?m)^NOT_TESTS = frozenset\(\{([^}]*)\}\)`).FindStringSubmatch(readPolicyFile(t, root, "tools/bazel/check_shard_coverage.py"))
if m == nil {
t.Fatal("tools/bazel/check_shard_coverage.py has no NOT_TESTS = frozenset({...})")
}
notTests := map[string]bool{}
for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(m[1], -1) {
notTests[q[1]] = true
}
for _, c := range []struct{ job, script, pkg string }{
{"test-embedded-cmd", ".github/scripts/embedded-test-shard.sh", "cmd/bd"},
{"test-embedded-storage", ".github/scripts/embedded-storage-test-shard.sh", "internal/storage/embeddeddolt"},
// D2 step 2: the proxied and server lanes' suites.
{"test-proxied-cmd", ".github/scripts/proxied-test-shard.sh", "cmd/bd"},
{"test-server-storage-full", ".github/scripts/server-storage-test-shard.sh", "internal/storage/dolt"},
} {
var src strings.Builder
files, err := filepath.Glob(filepath.Join(root, c.pkg, "*_test.go"))
if err != nil || len(files) == 0 {
t.Fatalf("%s: no _test.go files (%v)", c.pkg, err)
}
for _, f := range files {
data, err := os.ReadFile(f)
if err != nil {
t.Fatal(err)
}
src.Write(data)
src.WriteString("\n")
}
declared := map[string]bool{}
for _, d := range regexp.MustCompile(`(?m)^func (Test\w*)\(\w+ \*testing\.T\) \{`).FindAllStringSubmatch(src.String(), -1) {
declared[d[1]] = true
}
// B1: validate every total this script's committed manifest holds a
// block for, plus this job's own PR Risk matrix size and (for
// test-proxied-cmd) the Bazel lane's own shard_count — not just
// whichever total happens to equal this PR Risk job's matrix. Before
// F2 those always coincided; now the Bazel-only bazel-proxied lane
// reads a manifest block (30) that no PR-Risk-matrix-only check ever
// exercises, so a fork PR (which never runs bazel-proxied) could
// corrupt that block and still merge green. Looping over every
// distinct total in the manifest closes that gap for this script and
// any other script that later grows a second block the same way.
mm := shardManifestDefault.FindStringSubmatch(readPolicyFile(t, root, c.script))
if mm == nil {
t.Fatalf("%s has no ${BEADS_TEST_SHARD_MANIFEST:-...} default manifest", c.script)
}
totalsSet := map[int]bool{}
for _, line := range strings.Split(readPolicyFile(t, root, mm[1]), "\n") {
line, _, _ = strings.Cut(line, "#")
fields := strings.Fields(line)
if len(fields) == 0 {
continue
}
if n, err := strconv.Atoi(fields[0]); err == nil {
totalsSet[n] = true
}
}
switch c.script {
case ".github/scripts/proxied-test-shard.sh":
totalsSet[bazelProxiedShardCount(t)] = true
case ".github/scripts/embedded-test-shard.sh":
// F1: the Bazel-only bazel-embedded lane reads a 100-shard cmd
// block (split over embeddedCmdTargets); the legacy, frozen
// 20-shard block PR Risk's/main.yml's retired fork/push jobs
// used to read has been deleted (those jobs no longer exist; see
// engdocs/TESTING.md), so there is no longer a second check that
// would catch this total drifting on its own. Without this,
// "BUILD.bazel's shard_count and bazel.yml's
// check_shard_coverage.py arg both drift to a new total with no
// manifest block" passes every policy test (see review S3) --
// the lane then silently goes 100% hash fallback and loses its
// duration balancing.
totalsSet[bazelEmbeddedCmdShardCount(t)] = true
case ".github/scripts/embedded-storage-test-shard.sh":
// F1: mirrors the cmd case above for the 40-shard storage block.
totalsSet[bazelEmbeddedStorageShardCount(t)] = true
case ".github/scripts/server-storage-test-shard.sh":
totalsSet[bazelServerFullShardCount(t)] = true
}
totals := make([]int, 0, len(totalsSet))
for n := range totalsSet {
totals = append(totals, n)
}
sort.Ints(totals)
for _, shards := range totals {
// The scripts' hash fallback forks per test (seconds per shard
// for the server suite): list the shards concurrently.
outs, errs := make([][]byte, shards+1), make([]error, shards+1)
var wg sync.WaitGroup
for k := 1; k <= shards; k++ {
wg.Add(1)
go func(k int) {
defer wg.Done()
cmd := exec.Command("bash", c.script, strconv.Itoa(k), strconv.Itoa(shards))
cmd.Dir = root
cmd.Env = append(os.Environ(), "BEADS_TEST_SHARD_LIST_ONLY=1")
outs[k], errs[k] = cmd.Output()
}(k)
}
wg.Wait()
listed, manifestSum := 0, 0
for k := 1; k <= shards; k++ {
if errs[k] != nil {
// The script itself exits 1 on a duplicate manifest entry
// or a manifest entry that names no discovered test
// (rename/typo/stale-after-delete), for the requested
// total only: this is what catches a corrupted block
// that a PR-Risk-matrix-only check at a different total
// would never see.
t.Fatalf("%s %d %d: %v\n%s", c.script, k, shards, errs[k], outs[k])
}
for _, line := range strings.Split(string(outs[k]), "\n") {
if mc := regexp.MustCompile(`^ manifest: (\d+), fallback: \d+$`).FindStringSubmatch(line); mc != nil {
n, _ := strconv.Atoi(mc[1])
manifestSum += n
continue
}
name, ok := strings.CutPrefix(line, " ")
if !ok || !strings.HasPrefix(name, "Test") || strings.ContainsAny(name, " :") {
continue
}
listed++
isTest := declared[name]
switch {
case notTests[name] && isTest:
t.Errorf("%s shard %d/%d lists %s, which check_shard_coverage.py drops, but it is a real test", c.script, k, shards, name)
case !notTests[name] && !isTest:
t.Errorf("%s shard %d/%d lists %s, which is not a `func %s(t *testing.T)` test in %s: check_shard_coverage.py would report it missing on every run (add it to NOT_TESTS only if go test never runs it)",
c.script, k, shards, name, name, c.pkg)
}
}
}
if listed < 50 {
t.Errorf("%s at %d shards listed only %d tests; did the list-only output format change?", c.script, shards, listed)
}
// S1: a total that is supposed to have a committed manifest
// block (every total this loop considers does: it is either a
// live job's matrix size or a total this script's own manifest
// already names) must not have silently gone 100% hash fallback,
// which is what "the whole block was deleted" looks like from
// here: check_shard_coverage.py would still pass (it rebuilds
// its expectation from this same script), so nothing else
// catches it.
if manifestSum == 0 {
t.Errorf("%s at %d shards: manifest entries for this total sum to 0 across all shards (100%% hash fallback); its committed block in %s may have been deleted", c.script, shards, mm[1])
}
}
}
for name := range notTests {
if name != "TestMain" {
t.Errorf("NOT_TESTS has %s; only TestMain is never a test", name)
}
}
}
// S3: the manifest's Bazel-only 34-shard block, the file's only remaining
// block now that its legacy, frozen 15-shard block (read by PR Risk's and
// main.yml's retired fork/push jobs) has been deleted — those jobs no
// longer exist (see engdocs/TESTING.md). TestShardScriptsListOnlyRealTests's
// B1 fix catches outright corruption, but not a committed block that has
// drifted from the currently discovered TestProxiedServer*/TestServerMode*
// test set, e.g. a test added, renamed, or removed without anyone running
// --write. gen_proxied_shard_manifest.py --check verifies only that the
// committed block names every discovered test exactly once -- not that its
// shard *assignments* match a fresh LPT pack -- and fails with the exact
// command to fix it when a name is missing, stale, or duplicated. It
// deliberately does NOT fail merely because proxied_test_durations.json's
// weights changed and the existing packing is now suboptimal: two PRs each
// adding one proxied test would otherwise force a full repack and conflict
// on unrelated shard lines (see --repack below for the explicit opt-in to
// that). Run --check here so a block with missing/stale/duplicate names
// fails go test ./scripts/... (//scripts:go_test_sources_test under Bazel)
// instead of only surfacing as a test silently never running in any shard.
func TestProxiedShardManifestGeneratorNotStale(t *testing.T) {
python := requireHostTool(t, "python3")
root := sourceRepoRoot(t)
shards := strconv.Itoa(bazelProxiedShardCount(t))
cmd := exec.Command(python, "scripts/ci/gen_proxied_shard_manifest.py", shards, "--weights=duration", "--check")
cmd.Dir = root
if out, err := cmd.CombinedOutput(); err != nil {
t.Errorf("gen_proxied_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out)
}
}
// bareWriteReproducesManifest runs `generator --write --manifest COPY` with
// no other flags (in particular no --weights) against a scratch copy of
// manifest, and asserts the result is byte-identical to the committed file.
//
// This pins the --weights default against the footgun a reviewer caught on
// this PR: the manifest's only remaining block is duration-packed (the
// legacy, frozen inits-packed blocks were deleted alongside the PR Risk
// fork/push jobs that read them — see engdocs/TESTING.md), so a --weights
// default of "inits" made a bare `--write` silently rewrite the live
// block's header (and, since costs then came from the wrong model, its
// incremental placement of any newly-discovered test) to the inits form
// without --check ever catching it. Defaulting --weights to "duration"
// makes a bare `--write` idempotent on an up-to-date, duration-packed file.
func bareWriteReproducesManifest(t *testing.T, root, generator, manifest string) {
t.Helper()
python := requireHostTool(t, "python3")
want, err := os.ReadFile(filepath.Join(root, manifest))
if err != nil {
t.Fatalf("read %s: %v", manifest, err)
}
scratch := filepath.Join(t.TempDir(), filepath.Base(manifest))
if err := os.WriteFile(scratch, want, 0o644); err != nil {
t.Fatalf("seed scratch copy of %s: %v", manifest, err)
}
cmd := exec.Command(python, generator, "--write", "--manifest", scratch)
cmd.Dir = root
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("%s --write --manifest %s (no --weights): %v\n%s", generator, scratch, err, out)
}
got, err := os.ReadFile(scratch)
if err != nil {
t.Fatalf("read %s after --write: %v", scratch, err)
}
if string(got) != string(want) {
t.Errorf("%s --write with no --weights changed %s; a bare --write should reproduce the "+
"committed, duration-packed file exactly (--weights defaulting to anything but "+
"\"duration\" clobbers its header and repacks with the wrong cost model):\ngot:\n%s\nwant:\n%s",
generator, manifest, got, want)
}
}
func TestProxiedShardManifestBareWriteIsIdempotent(t *testing.T) {
bareWriteReproducesManifest(t, sourceRepoRoot(t),
"scripts/ci/gen_proxied_shard_manifest.py", ".github/scripts/proxied-cmd-test-shards.txt")
}
func TestEmbeddedCmdShardManifestBareWriteIsIdempotent(t *testing.T) {
bareWriteReproducesManifest(t, sourceRepoRoot(t),
"scripts/ci/gen_embedded_cmd_shard_manifest.py", ".github/scripts/embedded-cmd-test-shards.txt")
}