* fix(bazel): remove host-go dep from compact GC test; drop frozen shard blocks Replace the ungated `go build` in compact_gc_target_test.go's dolt-stand-in fixture with a Bazel-built go_binary (cmd/bd:compact_gc_fixture) wired via rlocationpath env + data, mirroring the existing preflight_go_fixture pattern. Falls back to `go build` only when not running under Bazel (bazeltest.IsBazel()). Also gate test_repo_beads_guard_test.go's `go env GOCACHE`/`GOMODCACHE` probes on !bazeltest.IsBazel(), since they previously ran unconditionally in every cmd/bd TestMain. Separately, delete the legacy 20-shard embedded-cmd and 15-shard proxied-cmd "frozen" manifest blocks in .github/scripts/*-test-shards.txt: their only consumers were pr-risk.yml fork/push jobs that have since migrated to bazel.yml. Update gen_embedded_cmd_shard_manifest.py and gen_proxied_shard_manifest.py to drop the frozen-block rendering/guard logic and default to the live Bazel-only shard counts (100 and 34), and refresh stale comments/docs (cmd/bd/BUILD.bazel, scripts/shard_manifest_test.go, engdocs/TESTING.md, engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md) that still described the deleted blocks. Verified no other BUILD/.github/scripts references read them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * build(bazel): auto-sync BUILD files Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/38069936930 (`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy). Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see scripts/bazel-autofix-push.sh. * fix(bazel): gate remaining host-go calls; fix shard-manifest --weights footgun Address PR #7478 review findings (bee-ghosttrack): - scripts/ci/lib/test-env.sh: skip the GOCACHE/GOMODCACHE `go env` probes when TEST_SRCDIR is set, so scripts/test.sh's beads_test_env_enter never shells out to a bare `go` when reached from a Bazel-run go_test (e.g. //scripts:shell_scripts_test's TestTestScriptPrebuiltTestBinaryContract), where a farm masks host go/dolt to exit 127. Nothing on that path needs a cache directory. - scripts/ci/fmt-check.sh: describe_gofmt() no longer calls a PATH `go` under Bazel. It now resolves BEADS_TEST_GO (the hermetic SDK go already wired into scripts/prlintmake:prlintmake_test's env) via TEST_SRCDIR, and skips the version-reporting arm entirely when that isn't set (e.g. scripts/repochecks:fmt_test), rather than ever touching a possibly-masked PATH `go`. - Verified with a PATH stub (go/dolt scripts that log and exit 127) via `bazel --nohome_rc test --config=prcore //scripts/... //cmd/bd:bd_test --test_env=PATH=<stubdir>:/usr/bin:/bin:/usr/local/bin`: 26/26 targets pass, zero stub invocations logged. - gen_proxied_shard_manifest.py / gen_embedded_cmd_shard_manifest.py: default --weights to duration now that the manifests' only remaining blocks are duration-packed (the legacy inits-packed blocks were deleted alongside the PR Risk fork/push jobs that read them). Previously the bare `--write` default (inits) silently clobbered the live block's duration header and repacked with the wrong cost model; --check couldn't see it. Added TestProxiedShardManifestBareWriteIsIdempotent and TestEmbeddedCmdShardManifestBareWriteIsIdempotent to pin a bare --write as a no-op against the committed files. - cmd/bd/compact_gc_target_test.go: resolve the non-Bazel fallback fixture path from this file's own directory (runtime.Caller) instead of the test binary's working directory. - Refresh stale comments in .github/workflows/bazel.yml, scripts/shard_manifest_test.go, scripts/pr_risk_bazel_coverage_test.go, and scripts/embedded_shard_count_test.go that still described the deleted legacy 20/15-shard blocks and their retired fork/push jobs in the present tense. - Confirmed cmd/bd/BUILD.bazel's compact_gc_fixture go_binary (implicit pure/race defaults) still passes TestRunCompactDoltTargetsOnlyAuthorizedActiveDatabase under `bazel test //cmd/bd:bd_test`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
335 lines
16 KiB
Go
335 lines
16 KiB
Go
package scripts_test
|
|
|
|
// The shard manifests and shard scripts against the Go tests they shard:
|
|
// //scripts:go_test_sources_test, which reads the _test.go files of the
|
|
// sharded packages (and no non-test Go source), so an edit to a workflow file,
|
|
// a doc or a non-test Go file leaves it cached.
|
|
|
|
import (
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"regexp"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
)
|
|
|
|
// S3 (F1, mirroring F2's TestProxiedShardManifestGeneratorNotStale below —
|
|
// see that test's doc comment for the full --check rationale, not repeated
|
|
// here): the Bazel-only 100-shard cmd block and 40-shard storage block.
|
|
// gen_embedded_{cmd,storage}_shard_manifest.py --check verifies only that
|
|
// the committed block names every discovered test exactly once, failing with
|
|
// the exact command to fix it when a name is missing, stale, or duplicated —
|
|
// run here so a drifted block fails `go test ./scripts/...`
|
|
// (//scripts:go_test_sources_test under Bazel) instead of only surfacing as
|
|
// a test silently never running in any shard. The cmd file's legacy,
|
|
// hand-assigned 20-shard block (frozen for PR Risk's and main.yml's retired
|
|
// fork/push jobs) has been deleted; those jobs no longer exist (see
|
|
// engdocs/TESTING.md). The storage file's legacy 5-shard block is still
|
|
// frozen and deliberately excluded here: its header documents that (see
|
|
// .github/scripts/embedded-storage-test-shards.txt and engdocs/TESTING.md),
|
|
// so a --check against it is expected to report "missing" entries by design
|
|
// (see that generator's module docstring) and is not what this test runs.
|
|
func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) {
|
|
python := requireHostTool(t, "python3")
|
|
root := sourceRepoRoot(t)
|
|
shards := strconv.Itoa(bazelEmbeddedCmdShardCount(t))
|
|
cmd := exec.Command(python, "scripts/ci/gen_embedded_cmd_shard_manifest.py", shards, "--weights=duration", "--check")
|
|
cmd.Dir = root
|
|
if out, err := cmd.CombinedOutput(); err != nil {
|
|
t.Errorf("gen_embedded_cmd_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out)
|
|
}
|
|
}
|
|
|
|
// TestPackConcurrentModelsSerialAndParallelTests pins
|
|
// _embedded_shard_manifest_lib.py's pack_concurrent: a serial test (no
|
|
// top-level t.Parallel) holds its shard alone for its whole duration, while
|
|
// parallel tests share -test.parallel slots, so four 30s parallel tests fit
|
|
// beside a 90s one in the same wall time. Plain LPT would put the serial
|
|
// test with a parallel one and split the rest evenly by summed duration.
|
|
func TestPackConcurrentModelsSerialAndParallelTests(t *testing.T) {
|
|
python := requireHostTool(t, "python3")
|
|
root := sourceRepoRoot(t)
|
|
const script = `
|
|
import sys
|
|
sys.path.insert(0, 'scripts/ci')
|
|
from _embedded_shard_manifest_lib import pack_concurrent
|
|
costs = {'serial': 100, 'a': 90, 'b': 80, 'c': 30, 'd': 30, 'e': 30, 'f': 30}
|
|
shards, walls = pack_concurrent(costs, 2, {'serial'}, 4)
|
|
print(sorted(map(sorted, shards)), walls)
|
|
`
|
|
cmd := exec.Command(python, "-c", script)
|
|
cmd.Dir = root
|
|
out, err := cmd.CombinedOutput()
|
|
if err != nil {
|
|
t.Fatalf("pack_concurrent: %v\n%s", err, out)
|
|
}
|
|
const want = "[['a', 'b', 'c', 'd', 'e', 'f'], ['serial']] [100.0, 90]\n"
|
|
if string(out) != want {
|
|
t.Errorf("pack_concurrent = %q, want %q", out, want)
|
|
}
|
|
}
|
|
|
|
// TestStorageEmbeddedShardManifestGeneratorNotStale mirrors
|
|
// TestCmdEmbeddedShardManifestGeneratorNotStale above for the storage tier's
|
|
// Bazel-only 20-shard block; see that test's doc comment.
|
|
func TestStorageEmbeddedShardManifestGeneratorNotStale(t *testing.T) {
|
|
python := requireHostTool(t, "python3")
|
|
root := sourceRepoRoot(t)
|
|
shards := strconv.Itoa(bazelEmbeddedStorageShardCount(t))
|
|
cmd := exec.Command(python, "scripts/ci/gen_embedded_storage_shard_manifest.py", shards, "--weights=duration", "--check")
|
|
cmd.Dir = root
|
|
if out, err := cmd.CombinedOutput(); err != nil {
|
|
t.Errorf("gen_embedded_storage_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out)
|
|
}
|
|
}
|
|
|
|
// Review G1: the checker's input is the real shard scripts' list-only
|
|
// output (every retired lane's: the embedded, proxied and server suites). Every name they list, for every shard, must be a test go test
|
|
// runs (declared `func Name(t *testing.T)` in the package's _test.go files),
|
|
// or one check_shard_coverage.py drops as NOT_TESTS; and each NOT_TESTS name
|
|
// must really not be a test (TestMain takes *testing.M). Otherwise the
|
|
// checker reports a listed test that "did not run" on every real run.
|
|
func TestShardScriptsListOnlyRealTests(t *testing.T) {
|
|
// The CI shard scripts use bash 4 associative arrays (Linux runners only).
|
|
requireAutofixBash(t)
|
|
root := sourceRepoRoot(t)
|
|
m := regexp.MustCompile(`(?m)^NOT_TESTS = frozenset\(\{([^}]*)\}\)`).FindStringSubmatch(readPolicyFile(t, root, "tools/bazel/check_shard_coverage.py"))
|
|
if m == nil {
|
|
t.Fatal("tools/bazel/check_shard_coverage.py has no NOT_TESTS = frozenset({...})")
|
|
}
|
|
notTests := map[string]bool{}
|
|
for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(m[1], -1) {
|
|
notTests[q[1]] = true
|
|
}
|
|
for _, c := range []struct{ job, script, pkg string }{
|
|
{"test-embedded-cmd", ".github/scripts/embedded-test-shard.sh", "cmd/bd"},
|
|
{"test-embedded-storage", ".github/scripts/embedded-storage-test-shard.sh", "internal/storage/embeddeddolt"},
|
|
// D2 step 2: the proxied and server lanes' suites.
|
|
{"test-proxied-cmd", ".github/scripts/proxied-test-shard.sh", "cmd/bd"},
|
|
{"test-server-storage-full", ".github/scripts/server-storage-test-shard.sh", "internal/storage/dolt"},
|
|
} {
|
|
var src strings.Builder
|
|
files, err := filepath.Glob(filepath.Join(root, c.pkg, "*_test.go"))
|
|
if err != nil || len(files) == 0 {
|
|
t.Fatalf("%s: no _test.go files (%v)", c.pkg, err)
|
|
}
|
|
for _, f := range files {
|
|
data, err := os.ReadFile(f)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
src.Write(data)
|
|
src.WriteString("\n")
|
|
}
|
|
declared := map[string]bool{}
|
|
for _, d := range regexp.MustCompile(`(?m)^func (Test\w*)\(\w+ \*testing\.T\) \{`).FindAllStringSubmatch(src.String(), -1) {
|
|
declared[d[1]] = true
|
|
}
|
|
|
|
// B1: validate every total this script's committed manifest holds a
|
|
// block for, plus this job's own PR Risk matrix size and (for
|
|
// test-proxied-cmd) the Bazel lane's own shard_count — not just
|
|
// whichever total happens to equal this PR Risk job's matrix. Before
|
|
// F2 those always coincided; now the Bazel-only bazel-proxied lane
|
|
// reads a manifest block (30) that no PR-Risk-matrix-only check ever
|
|
// exercises, so a fork PR (which never runs bazel-proxied) could
|
|
// corrupt that block and still merge green. Looping over every
|
|
// distinct total in the manifest closes that gap for this script and
|
|
// any other script that later grows a second block the same way.
|
|
mm := shardManifestDefault.FindStringSubmatch(readPolicyFile(t, root, c.script))
|
|
if mm == nil {
|
|
t.Fatalf("%s has no ${BEADS_TEST_SHARD_MANIFEST:-...} default manifest", c.script)
|
|
}
|
|
totalsSet := map[int]bool{}
|
|
for _, line := range strings.Split(readPolicyFile(t, root, mm[1]), "\n") {
|
|
line, _, _ = strings.Cut(line, "#")
|
|
fields := strings.Fields(line)
|
|
if len(fields) == 0 {
|
|
continue
|
|
}
|
|
if n, err := strconv.Atoi(fields[0]); err == nil {
|
|
totalsSet[n] = true
|
|
}
|
|
}
|
|
switch c.script {
|
|
case ".github/scripts/proxied-test-shard.sh":
|
|
totalsSet[bazelProxiedShardCount(t)] = true
|
|
case ".github/scripts/embedded-test-shard.sh":
|
|
// F1: the Bazel-only bazel-embedded lane reads a 100-shard cmd
|
|
// block (split over embeddedCmdTargets); the legacy, frozen
|
|
// 20-shard block PR Risk's/main.yml's retired fork/push jobs
|
|
// used to read has been deleted (those jobs no longer exist; see
|
|
// engdocs/TESTING.md), so there is no longer a second check that
|
|
// would catch this total drifting on its own. Without this,
|
|
// "BUILD.bazel's shard_count and bazel.yml's
|
|
// check_shard_coverage.py arg both drift to a new total with no
|
|
// manifest block" passes every policy test (see review S3) --
|
|
// the lane then silently goes 100% hash fallback and loses its
|
|
// duration balancing.
|
|
totalsSet[bazelEmbeddedCmdShardCount(t)] = true
|
|
case ".github/scripts/embedded-storage-test-shard.sh":
|
|
// F1: mirrors the cmd case above for the 40-shard storage block.
|
|
totalsSet[bazelEmbeddedStorageShardCount(t)] = true
|
|
case ".github/scripts/server-storage-test-shard.sh":
|
|
totalsSet[bazelServerFullShardCount(t)] = true
|
|
}
|
|
totals := make([]int, 0, len(totalsSet))
|
|
for n := range totalsSet {
|
|
totals = append(totals, n)
|
|
}
|
|
sort.Ints(totals)
|
|
|
|
for _, shards := range totals {
|
|
// The scripts' hash fallback forks per test (seconds per shard
|
|
// for the server suite): list the shards concurrently.
|
|
outs, errs := make([][]byte, shards+1), make([]error, shards+1)
|
|
var wg sync.WaitGroup
|
|
for k := 1; k <= shards; k++ {
|
|
wg.Add(1)
|
|
go func(k int) {
|
|
defer wg.Done()
|
|
cmd := exec.Command("bash", c.script, strconv.Itoa(k), strconv.Itoa(shards))
|
|
cmd.Dir = root
|
|
cmd.Env = append(os.Environ(), "BEADS_TEST_SHARD_LIST_ONLY=1")
|
|
outs[k], errs[k] = cmd.Output()
|
|
}(k)
|
|
}
|
|
wg.Wait()
|
|
listed, manifestSum := 0, 0
|
|
for k := 1; k <= shards; k++ {
|
|
if errs[k] != nil {
|
|
// The script itself exits 1 on a duplicate manifest entry
|
|
// or a manifest entry that names no discovered test
|
|
// (rename/typo/stale-after-delete), for the requested
|
|
// total only: this is what catches a corrupted block
|
|
// that a PR-Risk-matrix-only check at a different total
|
|
// would never see.
|
|
t.Fatalf("%s %d %d: %v\n%s", c.script, k, shards, errs[k], outs[k])
|
|
}
|
|
for _, line := range strings.Split(string(outs[k]), "\n") {
|
|
if mc := regexp.MustCompile(`^ manifest: (\d+), fallback: \d+$`).FindStringSubmatch(line); mc != nil {
|
|
n, _ := strconv.Atoi(mc[1])
|
|
manifestSum += n
|
|
continue
|
|
}
|
|
name, ok := strings.CutPrefix(line, " ")
|
|
if !ok || !strings.HasPrefix(name, "Test") || strings.ContainsAny(name, " :") {
|
|
continue
|
|
}
|
|
listed++
|
|
isTest := declared[name]
|
|
switch {
|
|
case notTests[name] && isTest:
|
|
t.Errorf("%s shard %d/%d lists %s, which check_shard_coverage.py drops, but it is a real test", c.script, k, shards, name)
|
|
case !notTests[name] && !isTest:
|
|
t.Errorf("%s shard %d/%d lists %s, which is not a `func %s(t *testing.T)` test in %s: check_shard_coverage.py would report it missing on every run (add it to NOT_TESTS only if go test never runs it)",
|
|
c.script, k, shards, name, name, c.pkg)
|
|
}
|
|
}
|
|
}
|
|
if listed < 50 {
|
|
t.Errorf("%s at %d shards listed only %d tests; did the list-only output format change?", c.script, shards, listed)
|
|
}
|
|
// S1: a total that is supposed to have a committed manifest
|
|
// block (every total this loop considers does: it is either a
|
|
// live job's matrix size or a total this script's own manifest
|
|
// already names) must not have silently gone 100% hash fallback,
|
|
// which is what "the whole block was deleted" looks like from
|
|
// here: check_shard_coverage.py would still pass (it rebuilds
|
|
// its expectation from this same script), so nothing else
|
|
// catches it.
|
|
if manifestSum == 0 {
|
|
t.Errorf("%s at %d shards: manifest entries for this total sum to 0 across all shards (100%% hash fallback); its committed block in %s may have been deleted", c.script, shards, mm[1])
|
|
}
|
|
}
|
|
}
|
|
for name := range notTests {
|
|
if name != "TestMain" {
|
|
t.Errorf("NOT_TESTS has %s; only TestMain is never a test", name)
|
|
}
|
|
}
|
|
}
|
|
|
|
// S3: the manifest's Bazel-only 34-shard block, the file's only remaining
|
|
// block now that its legacy, frozen 15-shard block (read by PR Risk's and
|
|
// main.yml's retired fork/push jobs) has been deleted — those jobs no
|
|
// longer exist (see engdocs/TESTING.md). TestShardScriptsListOnlyRealTests's
|
|
// B1 fix catches outright corruption, but not a committed block that has
|
|
// drifted from the currently discovered TestProxiedServer*/TestServerMode*
|
|
// test set, e.g. a test added, renamed, or removed without anyone running
|
|
// --write. gen_proxied_shard_manifest.py --check verifies only that the
|
|
// committed block names every discovered test exactly once -- not that its
|
|
// shard *assignments* match a fresh LPT pack -- and fails with the exact
|
|
// command to fix it when a name is missing, stale, or duplicated. It
|
|
// deliberately does NOT fail merely because proxied_test_durations.json's
|
|
// weights changed and the existing packing is now suboptimal: two PRs each
|
|
// adding one proxied test would otherwise force a full repack and conflict
|
|
// on unrelated shard lines (see --repack below for the explicit opt-in to
|
|
// that). Run --check here so a block with missing/stale/duplicate names
|
|
// fails go test ./scripts/... (//scripts:go_test_sources_test under Bazel)
|
|
// instead of only surfacing as a test silently never running in any shard.
|
|
func TestProxiedShardManifestGeneratorNotStale(t *testing.T) {
|
|
python := requireHostTool(t, "python3")
|
|
root := sourceRepoRoot(t)
|
|
shards := strconv.Itoa(bazelProxiedShardCount(t))
|
|
cmd := exec.Command(python, "scripts/ci/gen_proxied_shard_manifest.py", shards, "--weights=duration", "--check")
|
|
cmd.Dir = root
|
|
if out, err := cmd.CombinedOutput(); err != nil {
|
|
t.Errorf("gen_proxied_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out)
|
|
}
|
|
}
|
|
|
|
// bareWriteReproducesManifest runs `generator --write --manifest COPY` with
|
|
// no other flags (in particular no --weights) against a scratch copy of
|
|
// manifest, and asserts the result is byte-identical to the committed file.
|
|
//
|
|
// This pins the --weights default against the footgun a reviewer caught on
|
|
// this PR: the manifest's only remaining block is duration-packed (the
|
|
// legacy, frozen inits-packed blocks were deleted alongside the PR Risk
|
|
// fork/push jobs that read them — see engdocs/TESTING.md), so a --weights
|
|
// default of "inits" made a bare `--write` silently rewrite the live
|
|
// block's header (and, since costs then came from the wrong model, its
|
|
// incremental placement of any newly-discovered test) to the inits form
|
|
// without --check ever catching it. Defaulting --weights to "duration"
|
|
// makes a bare `--write` idempotent on an up-to-date, duration-packed file.
|
|
func bareWriteReproducesManifest(t *testing.T, root, generator, manifest string) {
|
|
t.Helper()
|
|
python := requireHostTool(t, "python3")
|
|
want, err := os.ReadFile(filepath.Join(root, manifest))
|
|
if err != nil {
|
|
t.Fatalf("read %s: %v", manifest, err)
|
|
}
|
|
scratch := filepath.Join(t.TempDir(), filepath.Base(manifest))
|
|
if err := os.WriteFile(scratch, want, 0o644); err != nil {
|
|
t.Fatalf("seed scratch copy of %s: %v", manifest, err)
|
|
}
|
|
cmd := exec.Command(python, generator, "--write", "--manifest", scratch)
|
|
cmd.Dir = root
|
|
if out, err := cmd.CombinedOutput(); err != nil {
|
|
t.Fatalf("%s --write --manifest %s (no --weights): %v\n%s", generator, scratch, err, out)
|
|
}
|
|
got, err := os.ReadFile(scratch)
|
|
if err != nil {
|
|
t.Fatalf("read %s after --write: %v", scratch, err)
|
|
}
|
|
if string(got) != string(want) {
|
|
t.Errorf("%s --write with no --weights changed %s; a bare --write should reproduce the "+
|
|
"committed, duration-packed file exactly (--weights defaulting to anything but "+
|
|
"\"duration\" clobbers its header and repacks with the wrong cost model):\ngot:\n%s\nwant:\n%s",
|
|
generator, manifest, got, want)
|
|
}
|
|
}
|
|
|
|
func TestProxiedShardManifestBareWriteIsIdempotent(t *testing.T) {
|
|
bareWriteReproducesManifest(t, sourceRepoRoot(t),
|
|
"scripts/ci/gen_proxied_shard_manifest.py", ".github/scripts/proxied-cmd-test-shards.txt")
|
|
}
|
|
|
|
func TestEmbeddedCmdShardManifestBareWriteIsIdempotent(t *testing.T) {
|
|
bareWriteReproducesManifest(t, sourceRepoRoot(t),
|
|
"scripts/ci/gen_embedded_cmd_shard_manifest.py", ".github/scripts/embedded-cmd-test-shards.txt")
|
|
}
|