beads/tools/bazel/critpath.py
Julian Knutsen 6da2a43477
ci(bazel): advisory Bazel workflow + go test/Bazel equivalence check (#6881)
* ci(bazel): add advisory Bazel workflow and setup-bazel action

Add .github/workflows/bazel.yml, a side-by-side lane that runs
`bazel test //... --config=ci` (PR Core parity) on pull_request,
merge_group, push to main and workflow_dispatch. It is advisory: not in
ci-gate's CI_GATE_REQUIRED.

.github/actions/setup-bazel installs a sha256-pinned Bazelisk (Bazel from
.bazelversion), restores the runner cache (--disk_cache and
--repository_cache only), and writes a CI bazelrc outside the workspace
and outside the cached directory. Remote execution is enabled only when
BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY are all set (base64
PEM cert/key; optional RBE_TLS_CA, RBE_INSTANCE); none set (fork PRs, or
workflow_dispatch rbe=off) runs locally; a partial set fails. The endpoint
and its bare host are masked. The rc logic lives in write-bazelrc.sh so it
is testable offline.

The runner cache is saved only by push-to-main runs under a run-unique key
and restored by prefix, so it never freezes at its first save and PRs do
not churn the quota. The job also reports the tier (T1/T2/local) and
wall time in the step summary, runs tools/bazel/critpath.py against the
profile, and checks BUILD drift with make bazel-sync-check plus
make bazel-sync and a clean git status.

tools/bazel/critpath.py reports the critical path as the sum of the
chained "critical path component" events rather than the longest one (a
cold race build: 395s over 75 components, where the single-component
version printed 69s and a false scheduling gap), lists the longest
members and Bazel's processes: cache line, and checks a report-only
budget (warning annotation, exit 0) unless --enforce is given.

scripts/ci_workflow_test.go covers the new lane: not wired into any gate,
every action SHA-pinned to the repo's released SHAs, no monolithic
actions/cache, no unbounded apt-get, the cache topology, secrets only
reaching setup-bazel, --profile after `test`, and the rc writer's
none/partial/full/invalid-PEM/secret-dir-in-workspace behaviour.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): check go test / Bazel test equivalence in the Bazel lane

tools/bazel/equivalence.py compares the Go tests `bazel test //...
--config=ci` ran with the ones pr.yml's go test lanes run, so a tag
filter, a stray manual tag, a package without a go_test target, or a
shard that runs zero tests cannot shrink the Bazel lane while it stays
green:

- package set: every package `go list -race -tags gms_pure_go` reports
  with test files has a go_test target (bazel query);
- test-name set: every top-level TestXxx in the files go list selects
  (minus ^TestEmbedded, as PR Core skips) appears as a testcase in the
  test.xml of a go_test target this invocation tested (from the BEP);
  runtime skips count as run. Tests Bazel ran that go test would not
  are divergences too.

Expected differences go in tools/bazel/equivalence_allowlist.txt, one
justified line each (empty today); unused entries are reported.
Results go to the step summary; the full listing with --details.

.bazelrc test:ci now sets GO_TEST_WRAP_TESTV=1 (rules_go lists passing
tests in test.xml only under -test.v) and downloads test.xml despite
--remote_download_minimal.

The Bazel workflow runs the check after the critical-path report, and
nightly.yml calls the workflow (suite "bazel", and on schedule) with the
full listing.

Verified locally against a remote --config=ci run: 103 packages and
8843 tests on both sides. Tagging //internal/gitignore:gitignore_test
manual keeps `bazel test` green but fails the check, listing its two
tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(fs): run RepoContextAndRole from its own git repo

The test relied on its working directory being inside a git checkout to
make BEADS_DIR external to the CWD repo. Under Bazel the working
directory is the runfiles tree, not a git repository, so RepoContext
failed with "not a git repository" and the test skipped itself.
Create a second temporary repo and chdir into it instead; go test keeps
passing and Bazel now runs the assertions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(bazel): fail equivalence on tests Bazel skips but go test passes

The equivalence docstring claimed runtime skips count as run because go
test skips them the same way. It does not: against PR Core's go test
-json on the same commit, 11 tests passed under go test and skipped
under Bazel while the check printed ok with an empty allowlist.

- equivalence.py --go-test-json compares per-test status with the
  Bazel test.xml: passed under go test and skipped under Bazel is a
  divergence; other status differences are listed, not failed. The
  in-job (PR) run cannot see skip divergences and says so.
- The allowlist gains a `skip` kind (`<pkg> <test> skip  # why`, the
  justification still mandatory) and lists the 10 intentional skips:
  cmd/bd TestGOMODCACHENotUnderTestHome and TestZZStdioNotLeaked, and
  the scripts policy tests that walk the source checkout. The eleventh,
  fs TestContextRepository_RepoContextAndRole, is fixed in the test.
- scripts/ci/pr-core.sh writes `go test -json` to
  BEADS_PR_CORE_GO_TEST_JSON when set (same flags otherwise).
- nightly.yml runs that in a new job, uploads the JSON, and calls
  bazel.yml with go-test-json-artifact, which downloads it and passes
  --go-test-json to the equivalence step.

Verified on a remote `bazel test //... --config=ci` run: ok with the
allowlist; a TEST_SRCDIR-only t.Skip injected into
internal/gitignore TestAppendLines fails the check (and passes the
name-only in-job check, as documented).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): freeze the runner cache per lock, mask key material, tighten policy

Runner cache: every push to main saved ~2.3 GB (repository cache, disk
cache, Bazelisk) under a run-unique key, on a repo already over its
Actions cache quota. The cache now holds only the repository cache and
Bazelisk, whose content is fixed by .bazelversion and MODULE.bazel.lock:
exact key bazel-repo-v2-<os>-<hash>, restore falls back to the previous
lock's entry, and only a push to main that missed the key saves. No
--disk_cache in either mode (remote runs have the remote cache; local
runs never used it). About 2 GB per lock change.

Credentials:
- write-bazelrc.sh masks every non-armor line of each decoded PEM
  (cert, key, CA). GitHub only masks the base64 value as stored. The CA
  used to be decoded inside the rc redirect, so its mask commands would
  have gone into the rc; it is decoded first now.
- The CI rc sets build:remote-exec --noremote_upload_local_results, so
  CI never writes locally executed (PR-code) results into the shared
  action cache. Developer rcs keep Bazel's default.
- RBE_INSTANCE stays optional; an empty one is a ::warning.
- bazel.yml's header states the accepted risk: same-repo PR code runs
  with the RBE client certificate (as in gascity); forks never get it;
  what is in place and what is pending a human decision.

Policy tests (each proved with a scratch mutation that now fails):
triggers must be exactly pull_request, merge_group, push,
workflow_dispatch and workflow_call (no pull_request_target); secrets
may appear only in the setup-bazel step's env, not in workflow or job
env, run, with or if; no continue-on-error anywhere in bazel.yml or
setup-bazel; the cache-save if is matched exactly; the critical-path
and equivalence steps must end with the bare script call and the test
step with exit "$rc", with no exit 0, || true, || :, ; true, set +e or
trap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(ci): record the same-repo RBE decision in bazel.yml

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(ci): bazel wrapper version check; keep go-test prose out of the build-tag scan

The wrapper prepends --bazelrc, so 'bazel --version' reached the server
as an unknown startup option; ask bazelisk directly. check-build-tags
matched 'go test' in workflow prose and step names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(ci): run the RBE lane on Blacksmith runners; mask the executor's IPs

GitHub-hosted runners are refused by the remote-execution edge; the
Blacksmith pool (as in gascity) is admitted and does not count against
GitHub's concurrent-job cap. Forks and rbe=off keep ubuntu-latest.
Bazel prints the resolved executor address in connection errors, so mask
every address the endpoint host resolves to.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): target rbe-west (instance oss) behind RBE_WEST_WORKERS

The original edge does not admit CI runners. Mirror gascity: remote
execution uses secrets.RBE_WEST_EXECUTOR and instance oss when the repo
variable RBE_WEST_WORKERS is true; same-repo runs are skipped until then
instead of failing, while forks and rbe=off still run locally.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 23:05:21 -07:00

175 lines
6.9 KiB
Python

#!/usr/bin/env python3
"""Critical-path report and CI budget check for a `bazel --profile` JSON.
gap = elapsed - critical_path
The critical path is the SUM of the chained "critical path component" events
Bazel records, not the single longest one: a cold race build is a long chain
of compile/link actions, and reporting only its longest member understates the
path and turns the difference into a phantom "scheduling gap".
Decision rule:
gap < 15% of elapsed the chain itself is the bottleneck. Attack its
longest member (shard/speed up a test, split or slim
a build action, warm the CAS if input upload
dominates).
gap >= 15% of elapsed actions waited for capacity: raise --jobs, grow the
executor pool, restore the runner disk cache.
Measurement tiers (caching makes single numbers lie):
T1 steady: warm remote cache + restored runner disk cache (the PR loop).
T2 cold client: fresh runner disk cache, warm remote cache.
T3 cold farm: after worker recycle / CAS eviction. Tracked, never gated.
local no remote execution (fork PRs). Reported, never gated.
Usage:
critpath.py PROFILE [--tier T2] [--budget SECONDS] [--enforce]
[--log BAZEL_OUTPUT_LOG] [--top N] [--summary FILE]
Without --enforce an over-budget run prints a GitHub warning annotation and
exits 0 (report-only phase); with --enforce it exits 1.
"""
import argparse
import json
import os
import re
import sys
CP_CAT = "critical path component"
DECISION_GAP_PCT = 15.0
def load_events(path):
with open(path, encoding="utf-8") as f:
prof = json.load(f)
if isinstance(prof, list):
return prof
return prof.get("traceEvents", [])
def short_name(name, width=90):
# "action 'GoLink cmd/bd/bd_test_/bd_test'" -> "GoLink cmd/bd/bd_test_/bd_test"
m = re.match(r"action '(.*)'$", name)
if m:
name = m.group(1)
return name if len(name) <= width else name[: width - 1] + "…"
def processes_line(log_path):
"""Return Bazel's final "N processes: ..." line from its console log."""
if not log_path or not os.path.exists(log_path):
return None
found = None
with open(log_path, encoding="utf-8", errors="replace") as f:
for line in f:
m = re.search(r"(\d[\d,]* processes?: .*)$", line.strip())
if m:
found = m.group(1)
return found
def analyze(events):
elapsed = max((e.get("dur", 0) for e in events if e.get("name") == "buildTargets"), default=0) / 1e6
stamps = [e["ts"] for e in events if e.get("ts")]
wall = (max(stamps) - min(stamps)) / 1e6 if stamps else 0.0
chain = [e for e in events if e.get("cat") == CP_CAT]
cp = sum(e.get("dur", 0) for e in chain) / 1e6
longest = sorted(chain, key=lambda e: e.get("dur", 0), reverse=True)
return {
"elapsed": elapsed,
"wall": wall,
"cp": cp,
"chain_len": len(chain),
"longest": longest,
}
def next_step(res):
elapsed, cp = res["elapsed"], res["cp"]
gap = max(elapsed - cp, 0.0)
pct = gap / elapsed * 100 if elapsed else 0.0
if not res["longest"]:
return pct, "NEXT: no critical path recorded (nothing executed, or a fully cached run)"
if pct < DECISION_GAP_PCT:
top = res["longest"][0]
name = short_name(top["name"], 200)
secs = top.get("dur", 0) / 1e6
m = re.search(r"Testing (\S+)", name)
if m:
return pct, f"NEXT: shard or speed up {m.group(1)} ({secs:.0f}s)"
if "upload" in name.lower():
return pct, "NEXT: input upload dominates; warm the remote CAS or slim the declared inputs"
return pct, f"NEXT: attack {short_name(name, 70)} ({secs:.0f}s)"
return pct, "NEXT: scheduling gap; raise --jobs, grow the executor pool, or restore the runner disk cache"
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("profile")
ap.add_argument("--tier", default="T2")
ap.add_argument("--budget", type=float, default=None, help="seconds; compared with max(elapsed, wall)")
ap.add_argument("--enforce", action="store_true", help="exit 1 when over budget")
ap.add_argument("--log", help="bazel console log, for the 'processes:' cache line")
ap.add_argument("--top", type=int, default=8, help="longest chain members to list")
ap.add_argument("--summary", default=None, help="append Markdown here (default: $GITHUB_STEP_SUMMARY)")
args = ap.parse_args(argv)
if not os.path.exists(args.profile):
print(f"critpath: profile {args.profile} not found (bazel did not start?)", file=sys.stderr)
return 1 if args.enforce else 0
res = analyze(load_events(args.profile))
pct, advice = next_step(res)
gap = max(res["elapsed"] - res["cp"], 0.0)
procs = processes_line(args.log)
tier = args.tier
lines = [
f"[{tier}] wall clock {res['wall']:7.1f}s (server start to last event)",
f"[{tier}] elapsed {res['elapsed']:7.1f}s (build graph)",
f"[{tier}] critical path {res['cp']:7.1f}s (sum of {res['chain_len']} chained components)",
f"[{tier}] gap {gap:7.1f}s ({pct:.0f}% of elapsed)",
]
if procs:
lines.append(f"[{tier}] processes {procs}")
top = [e for e in res["longest"][: args.top] if e.get("dur", 0) > 0]
if top:
lines.append("")
lines.append(f"longest critical-path members (top {len(top)}):")
for e in top:
lines.append(f" {e['dur'] / 1e6:7.1f}s {short_name(e['name'])}")
lines.append("")
lines.append(advice)
verdict = None
over = False
if args.budget is not None:
gated = max(res["elapsed"], res["wall"])
over = gated > args.budget
mode = "enforcing" if args.enforce else "report-only"
if over:
verdict = f"BUDGET ({mode}): over, {gated:.1f}s > {args.budget:.0f}s ({tier})"
else:
verdict = f"BUDGET ({mode}): within, {gated:.1f}s <= {args.budget:.0f}s ({tier})"
lines.append("")
lines.append(verdict)
print("\n".join(lines))
if over and not args.enforce and os.environ.get("GITHUB_ACTIONS") == "true":
print(f"::warning title=Bazel critical-path budget::{verdict}")
summary = args.summary or os.environ.get("GITHUB_STEP_SUMMARY")
if summary:
with open(summary, "a", encoding="utf-8") as f:
f.write(
f"**Tier {tier}**: elapsed {res['elapsed']:.0f}s, wall {res['wall']:.0f}s, "
f"critical path {res['cp']:.0f}s ({res['chain_len']} components), gap {pct:.0f}%"
)
f.write(f"; {verdict}\n\n" if verdict else "\n\n")
f.write("<details><summary>Critical path</summary>\n\n```\n" + "\n".join(lines) + "\n```\n</details>\n\n")
return 1 if (over and args.enforce) else 0
if __name__ == "__main__":
sys.exit(main())