beads/tools/bazel/equivalence.py
Julian Knutsen a4509deb2d
ci(bazel): nogo replaces go vet and golangci-lint (ga-96smfk.18) (#7341)
* ci(bazel): nogo replaces go vet and golangci-lint (ga-96smfk.18)

Lint and vet now run as nogo inside the Bazel build, so they gate wherever
the Bazel lanes execute (rbe-west), and the golangci-lint and go vet jobs
leave pr.yml and main.yml.

- //tools/nogo (own go.mod, joined to beads' for go_deps only through
  gazelle.go.work): go test's vet checks (cmd/go's defaultVetFlags, the set
  scripts/ci/go-test-vet.sh ran) plus the eight linters .golangci.yml
  enables, each a wrapper over the library version golangci-lint v2.10.1
  uses (depguard, errcheck, forbidigo, gosec, misspell, sloglint, unconvert,
  unparam).
- .golangci.yml stays the single lint policy: Bazel embeds it and
  tools/nogo/internal/golangci applies it as golangci-lint does (enabled
  set, settings, run.tests: false as library-unit-only analysis, lax
  generated-file detection, path/path-except/text exclusion rules), with
  strict decoding so an unimplemented key fails the build.
- gascity's rules_go //nolint-range patch and x/tools ctrlflow patch,
  byte-identical.
- Windows/darwin: cross-configured, not a golangci leg. The pure lane's
  release cross-compile (//tools/bazel:release_cross, every library and
  binary for all eight release platforms, cgo off) runs nogo on every
  compile, a superset of golangci-lint's windows and darwin legs.
  scripts/pr-lint (make ci-pr-lint, bd preflight) runs `bazel build
  --config=nogo //...` plus release_cross for windows_amd64,darwin_arm64.
  release_cross skips tools/nogo, a separate Go module `go build ./...`
  never compiled.
- pr.yml: scripts-go-checks (vet) and pr-lint-wrapper (x3) are gone, with
  their ci-gate entries; main.yml: pr-lint-wrapper (x6) and go-vet-cache.
  scripts/ci/go-test-vet.sh and install-golangci-lint.sh are deleted.
- pre-commit hook and .pre-commit-config.yaml run `make lint-changed
  LINT_CHANGED_SCOPE=staged` (nogo over the staged packages); Makefile adds
  lint, vet and lint-changed. pr-lint.sh no longer runs gofmt
  (//scripts/repochecks:fmt_test owns it; ga-96smfk.41).
- internal/testutil/integration: five gosec findings in integration-tagged
  library files golangci-lint never built, now linted by the integration
  lanes, carry reasoned //nolint directives.

Parity: with the exclusion rules removed, golangci-lint and nogo report the
same 3,916 findings natively and 3,859 each for windows and darwin (file,
line, linter, text); with .golangci.yml as committed both report none.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): equivalence.py leaves nested Go modules' tests out of go test's set

tools/nogo is its own Go module, so `go test ./...` in the root never runs
its tests; Bazel's unit lane does. Its packages are neither expected nor
extra.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 05:17:22 +00:00

470 lines
22 KiB
Python

#!/usr/bin/env python3
"""Check that `bazel test --config=ci` ran the same Go tests `go test` would.
The Bazel lane is only a safe replacement for the `go test` lanes in pr.yml
(PR Core runs `go test -race -short -skip '^TestEmbedded' ./...`) if it runs
the same tests. A tag filter, a stray `manual` tag, a package gazelle never
picked up, or a sharding bug that makes a shard run zero tests all shrink the
Bazel set silently: every target Bazel did run is still green. This script
makes that shrinkage a failure.
Two checks always run, a third when `go test -json` output is supplied:
1. Package set. Every package `go list` reports with test files must have a
go_test target (`bazel query 'kind("go_test rule", //...)'`).
2. Test-name set. For every such package, each top-level `func TestXxx(t
*testing.T)` in the files `go list` selects (the same -race and
gms_pure_go build context as PR Core, minus the `^TestEmbedded` skip) must
appear as a <testcase> in the test.xml of a go_test target that this Bazel
invocation tested (read from --build_event_json_file). Bazel testcases
that `go test` would not run are reported too. This check only sees names:
a test that runs and passes under `go test` but calls t.Skip under Bazel
(a missing env var, no git checkout, a file that is not declared as data)
still counts as present here.
3. Skip parity (--go-test-json, nightly). Given the `go test -json` output of
the PR Core command on the same commit, every top-level test that passed
under `go test` and was skipped under Bazel is a divergence. The Bazel lane
alone cannot see this (a skip looks the same whether or not `go test` also
skips), so the nightly workflow runs PR Core's `go test -json` and passes
it in. Other status differences (pass/fail either way, skipped under go
test but run under Bazel) are reported, not failed: a failing test already
fails its own lane.
Job mode (--job, with --go-test-json): the go test JSON is the whole expected
set instead of `go list ./...`. Use it to prove that a Bazel selection (one or
more --bep runs, e.g. --config=ci plus --config=doltserver) replaces one specific
CI job: every top-level test the job's command ran must have run under Bazel,
Bazel must run no other test in those packages (TestEmbedded* included: job
mode has no implicit ^TestEmbedded skip, so it can check the embedded tier), and no test the job passed may
be skipped under Bazel. A test seen in several targets (a go_test and its
go_test_variant.sh sh_test) keeps its best status here, since each variant is
a different way of running it and the job needs one run that does what it
does; the default mode keeps the worst. A failure in any target still fails a
job comparison (listed per target), and so does a missing test.xml of any
target in the job's packages, variant or not.
Packages of nested Go modules (a go.mod below the root: tools/nogo, the nogo
analyzers' own module) are outside `go test ./...`, so their Bazel tests are
neither expected nor reported as extra.
Divergences listed in the allowlist (tools/bazel/equivalence_allowlist.txt)
are expected; anything else fails. The allowlist format is one entry per line:
<package dir> <TestName or glob> # why Bazel does not run it
<package dir> <TestName or glob> skip # why Bazel skips a test go test runs
The first form (kind "run") covers the name checks; `*` as the test name
matches the whole package (including the package-set check). The `skip` form
covers only check 3. Every entry needs a justification after `#`.
Requires: go (with the module cache the packages need), bazel for the query
unless --no-query, and the BEP JSON plus bazel-testlogs of a `--config=ci`
run (which sets GO_TEST_WRAP_TESTV=1 and downloads test.xml).
"""
import argparse
import fnmatch
import json
import os
import re
import subprocess
import sys
import xml.etree.ElementTree as ET
GO_LIST_ARGS = ["list", "-e", "-race", "-tags", "gms_pure_go", "-json", "./..."]
# pr.yml's go test lanes pass -skip '^TestEmbedded' (as does test:prcore).
SKIP_RE = re.compile(r"^TestEmbedded")
TEST_FUNC_RE = re.compile(r"^func\s+(Test\w*)\s*\(\s*\w+\s+\*testing\.T\s*\)", re.M)
STATUS_RANK = {"skipped": 0, "passed": 1, "failed": 2}
# Job mode: the variant that actually ran a test wins (a failure still fails
# the Bazel run that produced it).
BEST_RANK = {"failed": 0, "skipped": 1, "passed": 2}
GO_TEST_STATUS = {"pass": "passed", "fail": "failed", "skip": "skipped"}
ALLOW_KINDS = ("run", "skip")
DEFAULT_ALLOWLIST = os.path.join(os.path.dirname(os.path.abspath(__file__)), "equivalence_allowlist.txt")
def is_go_test_name(name):
# cmd/go: "TestXxx" where Xxx does not start with a lower-case letter.
return name == "Test" or not name[4].islower()
def go_expected(root, go_json_path=None):
"""Return ({pkg_dir: set(test names)} for packages with test files,
{import path: pkg_dir} for every package)."""
if go_json_path:
with open(go_json_path, encoding="utf-8") as f:
raw = f.read()
else:
raw = subprocess.run(["go", *GO_LIST_ARGS], cwd=root, check=True, capture_output=True, text=True).stdout
decoder = json.JSONDecoder()
pkgs, idx = [], 0
while idx < len(raw):
while idx < len(raw) and raw[idx].isspace():
idx += 1
if idx >= len(raw):
break
obj, idx = decoder.raw_decode(raw, idx)
pkgs.append(obj)
expected, dirs = {}, {}
for p in pkgs:
rel = os.path.relpath(p["Dir"], root)
rel = "" if rel == "." else rel.replace(os.sep, "/")
if p.get("ImportPath"):
dirs[p["ImportPath"]] = rel
files = (p.get("TestGoFiles") or []) + (p.get("XTestGoFiles") or [])
if not files:
continue
names = set()
for name in files:
with open(os.path.join(p["Dir"], name), encoding="utf-8", errors="replace") as f:
for m in TEST_FUNC_RE.finditer(f.read()):
t = m.group(1)
if is_go_test_name(t) and not SKIP_RE.search(t):
names.add(t)
expected[rel] = names
return expected, dirs
def in_nested_module(root, pkg):
"""Whether pkg (a directory relative to root) belongs to a Go module
other than root's: a go.mod at or above it, below root."""
parts = pkg.split("/") if pkg else []
return any(os.path.exists(os.path.join(root, *parts[:i], "go.mod")) for i in range(1, len(parts) + 1))
def go_test_statuses(path, dirs):
"""Return {pkg_dir: {test: status}} for top-level tests in `go test -json`
output. Packages go list does not know are skipped."""
out = {}
with open(path, encoding="utf-8", errors="replace") as f:
for line in f:
if not line.startswith("{"):
continue
try:
ev = json.loads(line)
except ValueError:
continue
test, action = ev.get("Test"), ev.get("Action")
if not test or "/" in test or action not in GO_TEST_STATUS:
continue
pkg = dirs.get(ev.get("Package", ""))
if pkg is None:
continue
out.setdefault(pkg, {})[test] = GO_TEST_STATUS[action]
return out
def label_pkg(label):
# "//cmd/bd:bd_test" -> "cmd/bd", "@@//:x" -> ""
label = label.lstrip("@")
return label[2:].split(":", 1)[0] if label.startswith("//") else label.split(":", 1)[0]
def query_go_test_pkgs(root, bazel):
out = subprocess.run(
[bazel, "query", 'kind("go_test rule", //...)', "--output=label"],
cwd=root, check=True, capture_output=True, text=True,
).stdout
return {label_pkg(l) for l in out.split()}
# go_test_variant.sh variants are sh_tests that run a go_test binary, which
# writes the same test.xml.
TEST_KINDS = ("go_test rule", "sh_test rule")
def read_bep(path):
"""Return ({label: shard_count} for tested go_test targets and sh_test
variants, set of go_test labels configured, this invocation's testlogs dir
or None)."""
go_tests, tests, shards = set(), set(), {}
exec_root, testlogs_rel = None, None
with open(path, encoding="utf-8") as f:
for line in f:
if not line.strip():
continue
ev = json.loads(line)
eid = ev.get("id", {})
if "workspace" in eid:
exec_root = ev.get("workspaceInfo", {}).get("localExecRoot")
elif "convenienceSymlinksIdentified" in eid:
for link in ev.get("convenienceSymlinksIdentified", {}).get("convenienceSymlinks", []):
if link.get("path", "").endswith("testlogs") and link.get("target"):
testlogs_rel = link["target"]
elif "targetConfigured" in eid:
kind = ev.get("configured", {}).get("targetKind")
if kind in TEST_KINDS:
tests.add(eid["targetConfigured"]["label"])
if kind == "go_test rule":
go_tests.add(eid["targetConfigured"]["label"])
elif "testResult" in eid:
tr = eid["testResult"]
shards[tr["label"]] = max(shards.get(tr["label"], 0), int(tr.get("shard", 1) or 1))
tested = {l: n for l, n in shards.items() if l in tests}
testlogs = None
if exec_root and testlogs_rel:
# The symlink target is relative to the output base, two levels above
# the exec root (<output_base>/execroot/<workspace>).
testlogs = os.path.join(os.path.dirname(os.path.dirname(exec_root)), testlogs_rel)
return tested, go_tests, testlogs
def testlog_xmls(testlogs, label, shard_count):
pkg = label_pkg(label)
name = label.split(":", 1)[1]
base = os.path.join(testlogs, pkg, name)
if shard_count <= 1:
return [os.path.join(base, "test.xml")]
return [os.path.join(base, f"shard_{i}_of_{shard_count}", "test.xml") for i in range(1, shard_count + 1)]
def bazel_observed(tested, testlogs, go_tests, best=False, observed=None, failures=None, no_xml=None):
"""Return ({pkg: {test: status}}, [problems]), merged into observed.
Job mode (best) also collects (pkg, "Test (label)") for every failed test
in failures and (pkg, label) for every missing test.xml in no_xml."""
observed, problems = ({} if observed is None else observed), []
for label, n in sorted(tested.items()):
pkg = label_pkg(label)
per = observed.setdefault(pkg, {})
for xml_path in testlog_xmls(testlogs, label, n):
if not os.path.exists(xml_path):
if best and no_xml is not None:
# Bazel writes a test.xml for every test, so a variant's
# is missing only if it was never downloaded.
no_xml.append((pkg, f"{label}: {os.path.relpath(xml_path, testlogs)}"))
continue
if label not in go_tests:
continue # an sh_test that is not a variant may write none
problems.append(f"{label}: {xml_path} missing (was test.xml downloaded? run with --config=ci)")
continue
try:
tree = ET.parse(xml_path)
except ET.ParseError as e:
problems.append(f"{label}: cannot parse {xml_path}: {e}")
continue
cases = tree.getroot().iter("testcase")
count = 0
for tc in cases:
name = tc.get("name", "")
if "/" in name or not name.startswith("Test"):
continue
count += 1
if tc.find("failure") is not None or tc.find("error") is not None:
status = "failed"
elif tc.find("skipped") is not None:
status = "skipped"
else:
status = "passed"
if status == "failed" and failures is not None:
failures.append((pkg, f"{name} ({label})"))
# A test seen in several targets keeps its worst status (best in
# job mode).
rank = BEST_RANK if best else STATUS_RANK
if name not in per or rank[status] > rank[per[name]]:
per[name] = status
if count == 0 and label in go_tests and os.path.getsize(xml_path) < 64:
# An empty <testsuites/> means the binary ran no tests at all
# (or GO_TEST_WRAP_TESTV was off). The test-name diff reports
# which tests are missing; note it here for the log.
problems.append(f"{label}: {os.path.relpath(xml_path, testlogs)} lists no testcases")
return observed, problems
def load_allowlist(path):
entries, errors = [], []
if not path or not os.path.exists(path):
return entries, errors
with open(path, encoding="utf-8") as f:
for n, line in enumerate(f, 1):
body, _, why = line.partition("#")
body = body.strip()
if not body:
continue
fields = body.split()
kind = fields[2] if len(fields) == 3 else "run"
if len(fields) not in (2, 3) or kind not in ALLOW_KINDS or not why.strip():
errors.append(
f"{path}:{n}: want '<package dir> <test glob> [skip] # justification', got {line.strip()!r}"
)
continue
entries.append({"pkg": fields[0], "test": fields[1], "kind": kind, "why": why.strip(), "line": n, "used": False})
return entries, errors
def allowed(entries, pkg, test, kind="run"):
for e in entries:
if e["kind"] == kind and fnmatch.fnmatchcase(pkg, e["pkg"]) and fnmatch.fnmatchcase(test, e["test"]):
e["used"] = True
return True
return False
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument(
"--bep", required=True, action="append",
help="--build_event_json_file of a bazel test run (repeatable: results are merged)",
)
ap.add_argument("--root", default=os.getcwd(), help="workspace root (default: cwd)")
ap.add_argument("--testlogs", default=None, help="default: from the BEP, else <root>/bazel-testlogs")
ap.add_argument("--allowlist", default=DEFAULT_ALLOWLIST)
ap.add_argument("--go-list-json", default=None, help="precomputed `go list -json` output (tests)")
ap.add_argument(
"--go-test-json", default=None,
help="`go test -json` output of the PR Core command on the same commit; enables the skip-parity check",
)
ap.add_argument("--bazel", default=os.environ.get("BAZEL", "bazel"))
ap.add_argument("--no-query", action="store_true", help="skip the bazel query package-set check")
ap.add_argument("--details", action="store_true", help="list every divergence (nightly)")
ap.add_argument("--summary", default=None, help="append Markdown here (default: $GITHUB_STEP_SUMMARY)")
ap.add_argument(
"--job", action="store_true",
help="compare with --go-test-json as one CI job's complete test set (see Job mode above)",
)
args = ap.parse_args(argv)
if args.job and not args.go_test_json:
ap.error("--job needs --go-test-json")
root = os.path.abspath(args.root)
entries, allow_errors = load_allowlist(args.allowlist)
expected, dirs = go_expected(root, args.go_list_json)
observed, problems, configured, tested = {}, [], set(), {}
failures, no_xml = [], []
for bep in args.bep:
bep_tested, bep_configured, bep_testlogs = read_bep(bep)
configured |= bep_configured
tested.update(bep_tested)
# Prefer the BEP's own testlogs dir: the bazel-testlogs symlink follows
# whatever bazel command ran last, possibly in another output base.
testlogs = args.testlogs or bep_testlogs or os.path.join(root, "bazel-testlogs")
_, bep_problems = bazel_observed(
bep_tested, testlogs, bep_configured, args.job, observed,
failures if args.job else None, no_xml if args.job else None)
problems += bep_problems
observed = {pkg: seen for pkg, seen in observed.items() if not in_nested_module(root, pkg)}
target_pkgs = None if (args.no_query or args.job) else query_go_test_pkgs(root, args.bazel)
go_status = go_test_statuses(args.go_test_json, dirs) if args.go_test_json else None
if args.job:
# The job's own results are the expected set; other packages Bazel
# ran are out of scope.
expected = {pkg: set(ts) for pkg, ts in go_status.items()}
observed = {pkg: seen for pkg, seen in observed.items() if pkg in expected}
failures = sorted(set(f for f in failures if f[0] in expected))
no_xml = sorted(set(m for m in no_xml if m[0] in expected))
no_target, missing, extra, allowlisted = [], [], [], []
if target_pkgs is not None:
for pkg in sorted(set(expected) - target_pkgs):
(allowlisted if allowed(entries, pkg, "*") else no_target).append((pkg, "*"))
for pkg, names in sorted(expected.items()):
seen = observed.get(pkg, {})
for t in sorted(names - set(seen)):
(allowlisted if allowed(entries, pkg, t) else missing).append((pkg, t))
for pkg, seen in sorted(observed.items()):
for t in sorted(set(seen) - expected.get(pkg, set())):
# The --config=ci lane skips ^TestEmbedded; a job comparison has
# no implicit skip (the embedded tier's jobs run exactly those).
if not args.job and SKIP_RE.search(t):
continue
(allowlisted if allowed(entries, pkg, t) else extra).append((pkg, t))
# Skip parity: passed under go test, skipped under Bazel.
skip_div, skip_allowed, parity_notes = [], [], []
if go_status is not None:
for pkg, seen in sorted(observed.items()):
gs = go_status.get(pkg, {})
for t, bs in sorted(seen.items()):
g = gs.get(t)
if g is None or g == bs:
continue
if g == "passed" and bs == "skipped":
(skip_allowed if allowed(entries, pkg, t, "skip") else skip_div).append((pkg, t))
else:
parity_notes.append((pkg, f"{t} (go test {g}, bazel {bs})"))
allowlisted += skip_allowed
# Dedupe allowlisted pairs (a package-level entry also covers its tests).
allowlisted = sorted(set(allowlisted))
# Skip entries can only be exercised when go test statuses are known.
# (The allowlist describes the --config=ci lane; a job comparison uses only
# the entries that apply.)
unused = [] if args.job else [e for e in entries if not e["used"] and (e["kind"] == "run" or go_status is not None)]
statuses = {"passed": 0, "skipped": 0, "failed": 0}
for seen in observed.values():
for s in seen.values():
statuses[s] += 1
n_expected = sum(len(v) for v in expected.values())
ok = not (no_target or missing or extra or skip_div or failures or no_xml or allow_errors)
limit = None if args.details else 40
def listing(title, pairs):
out = [f"{title} ({len(pairs)}):"]
shown = pairs if limit is None else pairs[:limit]
out += [f" {p or '.'} {t}" for p, t in shown]
if limit is not None and len(pairs) > limit:
out.append(f" ... {len(pairs) - limit} more (run with --details)")
return out
lines = [
f"go test: {len(expected)} packages with tests, {n_expected} top-level tests"
+ ("" if args.job else " (excluding ^TestEmbedded)"),
f"bazel: {len(configured)} go_test targets configured, {len(tested)} tested; "
f"{sum(len(v) for v in observed.values())} top-level tests seen "
f"({statuses['passed']} passed, {statuses['skipped']} skipped, {statuses['failed']} failed)",
]
if target_pkgs is not None:
lines.append(f"query: {len(target_pkgs)} packages with a go_test target")
if go_status is None:
lines.append("skips: not compared (no --go-test-json; the nightly run checks skip parity)")
else:
n_go = sum(len(v) for v in go_status.values())
lines.append(
f"skips: compared with {n_go} go test results: {len(skip_div) + len(skip_allowed)} passed under "
f"go test but skipped under Bazel ({len(skip_allowed)} allowlisted), {len(parity_notes)} other differences"
)
lines.append(f"allowlisted divergences: {len(allowlisted)}")
for title, pairs in (
("packages with Go tests but no go_test target", no_target),
("tests go test runs that Bazel did not", missing),
("tests Bazel ran that go test would not", extra),
("tests go test passed that Bazel skipped", skip_div),
("tests that failed in a Bazel target", failures),
("Bazel targets without test.xml", no_xml),
):
if pairs:
lines += listing(title, pairs)
if parity_notes:
lines += listing("other status differences (reported, not failed)", parity_notes)
if args.details and allowlisted:
lines += listing("allowlisted", allowlisted)
for e in unused:
lines.append(f"note: allowlist line {e['line']} ({e['pkg']} {e['test']} {e['kind']}) matched nothing; remove it")
for p in problems:
lines.append(f"note: {p}")
lines += allow_errors
verdict = "EQUIVALENCE: ok" if ok else "EQUIVALENCE: FAIL (unexplained divergence; fix the BUILD tags/targets or add a justified allowlist entry)"
lines.append(verdict)
print("\n".join(lines))
summary = args.summary or os.environ.get("GITHUB_STEP_SUMMARY")
if summary:
head = (
f"**Equivalence**: {'ok' if ok else 'FAIL'}; go test {n_expected} tests in {len(expected)} packages, "
f"Bazel saw {sum(len(v) for v in observed.values())} "
f"(missing {len(missing)}, extra {len(extra)}, packages without target {len(no_target)}, "
f"skipped only under Bazel {len(skip_div) if go_status is not None else 'not compared'}, "
f"allowlisted {len(allowlisted)})\n\n"
)
with open(summary, "a", encoding="utf-8") as f:
f.write(head)
f.write("<details><summary>Equivalence detail</summary>\n\n```\n" + "\n".join(lines) + "\n```\n</details>\n\n")
return 0 if ok else 1
if __name__ == "__main__":
sys.exit(main())