ci(bazel): advisory Bazel workflow + go test/Bazel equivalence check (#6881)

* ci(bazel): add advisory Bazel workflow and setup-bazel action

Add .github/workflows/bazel.yml, a side-by-side lane that runs
`bazel test //... --config=ci` (PR Core parity) on pull_request,
merge_group, push to main and workflow_dispatch. It is advisory: not in
ci-gate's CI_GATE_REQUIRED.

.github/actions/setup-bazel installs a sha256-pinned Bazelisk (Bazel from
.bazelversion), restores the runner cache (--disk_cache and
--repository_cache only), and writes a CI bazelrc outside the workspace
and outside the cached directory. Remote execution is enabled only when
BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY are all set (base64
PEM cert/key; optional RBE_TLS_CA, RBE_INSTANCE); none set (fork PRs, or
workflow_dispatch rbe=off) runs locally; a partial set fails. The endpoint
and its bare host are masked. The rc logic lives in write-bazelrc.sh so it
is testable offline.

The runner cache is saved only by push-to-main runs under a run-unique key
and restored by prefix, so it never freezes at its first save and PRs do
not churn the quota. The job also reports the tier (T1/T2/local) and
wall time in the step summary, runs tools/bazel/critpath.py against the
profile, and checks BUILD drift with make bazel-sync-check plus
make bazel-sync and a clean git status.

tools/bazel/critpath.py reports the critical path as the sum of the
chained "critical path component" events rather than the longest one (a
cold race build: 395s over 75 components, where the single-component
version printed 69s and a false scheduling gap), lists the longest
members and Bazel's processes: cache line, and checks a report-only
budget (warning annotation, exit 0) unless --enforce is given.

scripts/ci_workflow_test.go covers the new lane: not wired into any gate,
every action SHA-pinned to the repo's released SHAs, no monolithic
actions/cache, no unbounded apt-get, the cache topology, secrets only
reaching setup-bazel, --profile after `test`, and the rc writer's
none/partial/full/invalid-PEM/secret-dir-in-workspace behaviour.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): check go test / Bazel test equivalence in the Bazel lane

tools/bazel/equivalence.py compares the Go tests `bazel test //...
--config=ci` ran with the ones pr.yml's go test lanes run, so a tag
filter, a stray manual tag, a package without a go_test target, or a
shard that runs zero tests cannot shrink the Bazel lane while it stays
green:

- package set: every package `go list -race -tags gms_pure_go` reports
  with test files has a go_test target (bazel query);
- test-name set: every top-level TestXxx in the files go list selects
  (minus ^TestEmbedded, as PR Core skips) appears as a testcase in the
  test.xml of a go_test target this invocation tested (from the BEP);
  runtime skips count as run. Tests Bazel ran that go test would not
  are divergences too.

Expected differences go in tools/bazel/equivalence_allowlist.txt, one
justified line each (empty today); unused entries are reported.
Results go to the step summary; the full listing with --details.

.bazelrc test:ci now sets GO_TEST_WRAP_TESTV=1 (rules_go lists passing
tests in test.xml only under -test.v) and downloads test.xml despite
--remote_download_minimal.

The Bazel workflow runs the check after the critical-path report, and
nightly.yml calls the workflow (suite "bazel", and on schedule) with the
full listing.

Verified locally against a remote --config=ci run: 103 packages and
8843 tests on both sides. Tagging //internal/gitignore:gitignore_test
manual keeps `bazel test` green but fails the check, listing its two
tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(fs): run RepoContextAndRole from its own git repo

The test relied on its working directory being inside a git checkout to
make BEADS_DIR external to the CWD repo. Under Bazel the working
directory is the runfiles tree, not a git repository, so RepoContext
failed with "not a git repository" and the test skipped itself.
Create a second temporary repo and chdir into it instead; go test keeps
passing and Bazel now runs the assertions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(bazel): fail equivalence on tests Bazel skips but go test passes

The equivalence docstring claimed runtime skips count as run because go
test skips them the same way. It does not: against PR Core's go test
-json on the same commit, 11 tests passed under go test and skipped
under Bazel while the check printed ok with an empty allowlist.

- equivalence.py --go-test-json compares per-test status with the
  Bazel test.xml: passed under go test and skipped under Bazel is a
  divergence; other status differences are listed, not failed. The
  in-job (PR) run cannot see skip divergences and says so.
- The allowlist gains a `skip` kind (`<pkg> <test> skip  # why`, the
  justification still mandatory) and lists the 10 intentional skips:
  cmd/bd TestGOMODCACHENotUnderTestHome and TestZZStdioNotLeaked, and
  the scripts policy tests that walk the source checkout. The eleventh,
  fs TestContextRepository_RepoContextAndRole, is fixed in the test.
- scripts/ci/pr-core.sh writes `go test -json` to
  BEADS_PR_CORE_GO_TEST_JSON when set (same flags otherwise).
- nightly.yml runs that in a new job, uploads the JSON, and calls
  bazel.yml with go-test-json-artifact, which downloads it and passes
  --go-test-json to the equivalence step.

Verified on a remote `bazel test //... --config=ci` run: ok with the
allowlist; a TEST_SRCDIR-only t.Skip injected into
internal/gitignore TestAppendLines fails the check (and passes the
name-only in-job check, as documented).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): freeze the runner cache per lock, mask key material, tighten policy

Runner cache: every push to main saved ~2.3 GB (repository cache, disk
cache, Bazelisk) under a run-unique key, on a repo already over its
Actions cache quota. The cache now holds only the repository cache and
Bazelisk, whose content is fixed by .bazelversion and MODULE.bazel.lock:
exact key bazel-repo-v2-<os>-<hash>, restore falls back to the previous
lock's entry, and only a push to main that missed the key saves. No
--disk_cache in either mode (remote runs have the remote cache; local
runs never used it). About 2 GB per lock change.

Credentials:
- write-bazelrc.sh masks every non-armor line of each decoded PEM
  (cert, key, CA). GitHub only masks the base64 value as stored. The CA
  used to be decoded inside the rc redirect, so its mask commands would
  have gone into the rc; it is decoded first now.
- The CI rc sets build:remote-exec --noremote_upload_local_results, so
  CI never writes locally executed (PR-code) results into the shared
  action cache. Developer rcs keep Bazel's default.
- RBE_INSTANCE stays optional; an empty one is a ::warning.
- bazel.yml's header states the accepted risk: same-repo PR code runs
  with the RBE client certificate (as in gascity); forks never get it;
  what is in place and what is pending a human decision.

Policy tests (each proved with a scratch mutation that now fails):
triggers must be exactly pull_request, merge_group, push,
workflow_dispatch and workflow_call (no pull_request_target); secrets
may appear only in the setup-bazel step's env, not in workflow or job
env, run, with or if; no continue-on-error anywhere in bazel.yml or
setup-bazel; the cache-save if is matched exactly; the critical-path
and equivalence steps must end with the bare script call and the test
step with exit "$rc", with no exit 0, || true, || :, ; true, set +e or
trap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(ci): record the same-repo RBE decision in bazel.yml

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(ci): bazel wrapper version check; keep go-test prose out of the build-tag scan

The wrapper prepends --bazelrc, so 'bazel --version' reached the server
as an unknown startup option; ask bazelisk directly. check-build-tags
matched 'go test' in workflow prose and step names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(ci): run the RBE lane on Blacksmith runners; mask the executor's IPs

GitHub-hosted runners are refused by the remote-execution edge; the
Blacksmith pool (as in gascity) is admitted and does not count against
GitHub's concurrent-job cap. Forks and rbe=off keep ubuntu-latest.
Bazel prints the resolved executor address in connection errors, so mask
every address the endpoint host resolves to.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): target rbe-west (instance oss) behind RBE_WEST_WORKERS

The original edge does not admit CI runners. Mirror gascity: remote
execution uses secrets.RBE_WEST_EXECUTOR and instance oss when the repo
variable RBE_WEST_WORKERS is true; same-repo runs are skipped until then
instead of failing, while forks and rbe=off still run locally.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-09-27 23:05:21 -07:00 • committed by GitHub
commit 6da2a43477
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 1678 additions and 2 deletions

View file

@ -32,6 +32,12 @@ test:prcore --test_tag_filters=-requires-docker,-embedded,-manual
test:ci --config=prcore
test:ci --keep_going --test_summary=terse
# tools/bazel/equivalence.py compares the Go tests Bazel ran with the ones
# `go test` would run, from each target's test.xml. rules_go only lists passing
# tests in test.xml under -test.v, and --remote_download_minimal would leave
# test.xml on the remote side, so ask for both.
test:ci --test_env=GO_TEST_WRAP_TESTV=1
test:ci --remote_download_regex=.*/test\.xml$
# Docker lane: only the requires-docker targets (pr.yml's container-backed
# jobs), on a host with a docker daemon. Race like those jobs, but no -short

70
.github/actions/setup-bazel/action.yml vendored Normal file
View file

@ -0,0 +1,70 @@
name: setup-bazel
description: >
Install a sha256-pinned Bazelisk (Bazel itself comes from .bazelversion),
restore the runner Bazel cache (--repository_cache and Bazelisk only),
and write a CI bazelrc outside the workspace. When the caller passes all of
BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY in the step env, that rc
enables --config=remote-exec for every bazel command; with none of them the
job runs locally (fork PRs); a partial set fails. `bazel` on PATH is a
wrapper that always passes the generated rc.
outputs:
remote:
description: '"true" when remote execution is configured'
value: ${{ steps.rc.outputs.remote }}
cache-restored:
description: '"true" when a runner Bazel cache was restored (any prefix match)'
value: ${{ steps.restore.outputs.cache-matched-key != '' }}
cache-hit:
description: '"true" when the cache for this .bazelversion and MODULE.bazel.lock was restored exactly'
value: ${{ steps.restore.outputs.cache-hit }}
runs:
using: composite
steps:
# Keyed on what determines its content; saved only by a push to main that
# missed the exact key (see the caller's "Save Bazel runner cache" step).
# A lock change restores the previous lock's entry, which still holds most
# of the downloads.
- name: Restore Bazel runner cache
id: restore
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ runner.temp }}/bazel-ci-cache
key: bazel-repo-v2-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }}
restore-keys: |
bazel-repo-v2-${{ runner.os }}-
- name: Install Bazelisk
shell: bash
run: |
set -euo pipefail
version=v1.29.0
case "$(uname -m)" in
x86_64) arch=amd64; sha=5a408715e932c0250d28bd84555f12edbf70117de42f9181691c736eacc4a992 ;;
aarch64 | arm64) arch=arm64; sha=e20e8b0f4f240091b7a55bf17b9398bd4f40ee70ae0208dff95dd4c445fb4010 ;;
*) echo "setup-bazel: unsupported arch $(uname -m)" >&2; exit 1 ;;
esac
curl -fsSL --retry 3 --max-time 120 -o "$RUNNER_TEMP/bazelisk" \
"https://github.com/bazelbuild/bazelisk/releases/download/${version}/bazelisk-linux-${arch}"
echo "${sha} $RUNNER_TEMP/bazelisk" | sha256sum -c -
sudo install -m 755 "$RUNNER_TEMP/bazelisk" /usr/local/bin/bazelisk
rm -f "$RUNNER_TEMP/bazelisk"
echo "BAZELISK_HOME=$RUNNER_TEMP/bazel-ci-cache/bazelisk" >> "$GITHUB_ENV"
- name: Write CI bazelrc
id: rc
shell: bash
env:
BAZEL_CI_CACHE_DIR: ${{ runner.temp }}/bazel-ci-cache
BAZEL_CI_SECRET_DIR: ${{ runner.temp }}/bazel-ci-secret
run: bash "$GITHUB_ACTION_PATH/write-bazelrc.sh"
- name: Install bazel wrapper
shell: bash
env:
RC: ${{ steps.rc.outputs.rc }}
run: |
set -euo pipefail
printf '%s\n' '#!/usr/bin/env bash' "exec /usr/local/bin/bazelisk --bazelrc=\"$RC\" \"\$@\"" > "$RUNNER_TEMP/bazel-wrapper"
sudo install -m 755 "$RUNNER_TEMP/bazel-wrapper" /usr/local/bin/bazel
rm -f "$RUNNER_TEMP/bazel-wrapper"
/usr/local/bin/bazelisk --version

View file

@ -0,0 +1,152 @@
#!/usr/bin/env bash
# Write the CI bazelrc for .github/actions/setup-bazel.
#
# Always: --repository_cache under BAZEL_CI_CACHE_DIR (the directory the
# runner cache saves and restores). Never --disk_cache: remote runs have the
# remote cache, and a local run's disk cache (about 7 GB) would not fit next to
# its output base on a hosted runner.
#
# Remote execution only when all three of BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT
# and RBE_TLS_KEY are set (repo secrets; empty on fork PRs, so those runs stay
# local). Exactly one or two of them is a misconfiguration and fails. Optional:
# RBE_TLS_CA (base64 PEM CA bundle) and RBE_INSTANCE (instance name; a warning
# when empty, because the default instance may share its action cache with
# other repos on the same farm). Cert and key are base64-encoded PEM (raw PEM
# is accepted too). Every line of the decoded PEM is masked in the job log.
#
# CI uploads only remotely executed results (--noremote_upload_local_results):
# a job's locally executed actions (PR code included) never enter the shared
# action cache from here. Developer rcs keep Bazel's default.
#
# The rc and the key material go to BAZEL_CI_SECRET_DIR, which must be outside
# the workspace (tests that walk the checkout must never see an endpoint or a
# cert path) and outside BAZEL_CI_CACHE_DIR (the runner cache must never store
# credentials). The endpoint is masked in the job log.
#
# Outputs (to $GITHUB_OUTPUT when set, else stdout): rc=<path>, remote=true|false.
set -euo pipefail
cache_dir="${BAZEL_CI_CACHE_DIR:?BAZEL_CI_CACHE_DIR is required}"
secret_dir="${BAZEL_CI_SECRET_DIR:?BAZEL_CI_SECRET_DIR is required}"
workspace="${GITHUB_WORKSPACE:-$PWD}"
abs() { mkdir -p "$1" && (cd "$1" && pwd -P); }
cache_dir="$(abs "$cache_dir")"
secret_dir="$(abs "$secret_dir")"
workspace="$(cd "$workspace" && pwd -P)"
case "$secret_dir/" in
"$workspace"/*)
echo "setup-bazel: BAZEL_CI_SECRET_DIR ($secret_dir) must be outside the workspace" >&2
exit 1
;;
"$cache_dir"/*)
echo "setup-bazel: BAZEL_CI_SECRET_DIR must not be inside the cached BAZEL_CI_CACHE_DIR" >&2
exit 1
;;
esac
chmod 700 "$secret_dir"
mkdir -p "$cache_dir/repo"
mask() {
local v="${1:-}"
[[ -z "$v" ]] && return 0
printf '::add-mask::%s\n' "$v"
}
# Mask the endpoint and its bare host[:port]: Bazel prints bytestream:// URIs
# that carry the host without the scheme.
executor="${BAZEL_REMOTE_EXECUTOR:-}"
if [[ -n "$executor" ]]; then
mask "$executor"
hostport="${executor#*://}"
hostport="${hostport%%/*}"
mask "$hostport"
mask "${hostport%:*}"
# Bazel's connection errors print the resolved address ("host/1.2.3.4:443").
if command -v getent >/dev/null 2>&1; then
while read -r addr _; do
mask "$addr"
done < <(getent ahosts "${hostport%:*}" 2>/dev/null | sort -u -k1,1)
fi
fi
cert="${RBE_TLS_CERT:-}"
key="${RBE_TLS_KEY:-}"
ca="${RBE_TLS_CA:-}"
instance="${RBE_INSTANCE:-}"
rc="$secret_dir/ci.bazelrc"
umask 077
{
echo "# Generated by .github/actions/setup-bazel; not a workspace file."
echo "common --repository_cache=$cache_dir/repo"
} >"$rc"
# write_pem DEST VALUE: decode base64 (or accept raw PEM) and check it is PEM.
write_pem() {
local dest="$1" body="$2"
body="$(printf '%s' "$body" | tr -d '\r')"
if [[ "$body" != *"-----BEGIN "* ]]; then
if ! body="$(printf '%s' "$body" | tr -d '\n ' | base64 -d 2>/dev/null)"; then
echo "setup-bazel: $(basename "$dest") secret is neither PEM nor base64 PEM" >&2
return 1
fi
fi
if [[ "$body" != *"-----BEGIN "* ]]; then
echo "setup-bazel: $(basename "$dest") secret does not decode to PEM" >&2
return 1
fi
# GitHub masks the secret as stored (base64), not what it decodes to.
local line
while IFS= read -r line; do
[[ -z "$line" || "$line" == -----* ]] || mask "$line"
done <<<"$body"
printf '%s\n' "$body" >"$dest"
chmod 600 "$dest"
}
set_fields=0
for v in "$executor" "$cert" "$key"; do
[[ -n "$v" ]] && set_fields=$((set_fields + 1))
done
remote=false
if [[ "$set_fields" -eq 3 ]]; then
write_pem "$secret_dir/client.crt" "$cert"
write_pem "$secret_dir/client.key" "$key"
# Outside the rc block below: write_pem's mask commands go to the log.
if [[ -n "$ca" ]]; then
write_pem "$secret_dir/ca.crt" "$ca"
fi
{
echo "build:remote-exec --remote_executor=$executor"
echo "build:remote-exec --tls_client_certificate=$secret_dir/client.crt"
echo "build:remote-exec --tls_client_key=$secret_dir/client.key"
if [[ -n "$ca" ]]; then
echo "build:remote-exec --tls_certificate=$secret_dir/ca.crt"
fi
if [[ -n "$instance" ]]; then
echo "build:remote-exec --remote_instance_name=$instance"
fi
echo "build:remote-exec --noremote_upload_local_results"
# Every command in this job (test, run //:gazelle, query) uses the
# executor; the committed .bazelrc holds the generic remote-exec tuning.
echo "build --config=remote-exec"
} >>"$rc"
if [[ -z "$instance" ]]; then
echo "::warning title=RBE_INSTANCE not set::remote execution uses the executor's default instance, whose action cache other repos on the farm may share; set RBE_INSTANCE to a beads-only instance"
fi
remote=true
echo "setup-bazel: remote execution enabled"
elif [[ "$set_fields" -ne 0 ]]; then
echo "setup-bazel: remote execution is partially configured; BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY must be set together (or none, for local execution)" >&2
exit 1
else
echo "setup-bazel: no remote execution secrets; running locally"
fi
out="${GITHUB_OUTPUT:-/dev/stdout}"
{
echo "rc=$rc"
echo "remote=$remote"
} >>"$out"

250
.github/workflows/bazel.yml vendored Normal file
View file

@ -0,0 +1,250 @@
# Bazel side-by-side lane (advisory). Runs `bazel test //... --config=ci`, the
# PR Core-parity selection (race, -short, dolt skip, -requires-docker), on the
# same commit as pr.yml's go test lanes. It is NOT in ci-gate's
# CI_GATE_REQUIRED; promotion needs the equivalence record described in
# tools/bazel/equivalence.py and a human decision.
#
# Execution mode, passed to .github/actions/setup-bazel:
# repo variable RBE_WEST_WORKERS=true plus secrets RBE_WEST_EXECUTOR (grpcs://
# endpoint), RBE_TLS_CERT, RBE_TLS_KEY (base64; RBE_TLS_CA optional):
# remote execution on rbe-west, instance "oss" (shared with gascity).
# fork PRs and rbe=off dispatches: local execution, slower but the same
# results. Same-repo runs are skipped while RBE_WEST_WORKERS is unset.
# workflow_dispatch with rbe=off simulates a fork run.
#
# Accepted risk (same model as gascity's bazel-test.yml): on push, merge_group
# and same-repo pull_request runs, code from the PR head runs with the RBE
# client certificate. setup-bazel is itself PR code, the key file is readable
# by every later step and by locally executed Bazel actions, and a .bazelrc
# edit can point the BEP or the executor anywhere. Anyone who can push a
# branch here can therefore use or copy the certificate and write to the
# farm's cache. Fork PRs never get it: pull_request from a fork receives no
# secrets, this workflow has no pull_request_target trigger, and forks cannot
# write the runner cache. In place: secrets are referenced by the setup step
# alone and never by a workflow- or job-level env (policy-tested), decoded key
# material is masked in logs, and CI uploads only remotely executed results
# (--noremote_upload_local_results in the generated rc), and the certificate
# is issued to beads CI alone, so it can be revoked without affecting other
# clients. Maintainers decided (2026-09-28) that same-repo PRs keep remote
# execution under this model. Still open: a beads-only RBE_INSTANCE and
# farm-side denial of client action-cache writes. The lane is advisory and
# nothing downstream consumes artifacts built here.
#
# Runner cache: one entry per (.bazelversion, MODULE.bazel.lock) pair, about
# 2 GB (Bazel's --repository_cache plus Bazelisk's Bazel download), written
# only by a push to main that missed it. No --disk_cache: remote runs use the
# remote cache, and local runs would need about 7 GB more.
#
# Workflow-file changes suppress pull_request runs under the org approval
# policy; validate edits to this file with workflow_dispatch on the branch.
name: Bazel
on:
pull_request:
branches: [ main, 'release/**' ]
merge_group:
push:
branches: [ main ]
workflow_dispatch:
inputs:
rbe:
description: "Remote execution: on (use secrets) or off (fork simulation)"
required: true
type: choice
default: "on"
options: [ "on", "off" ]
details:
description: "List every equivalence divergence"
required: false
type: boolean
default: false
workflow_call:
inputs:
rbe:
type: string
required: false
default: "on"
details:
type: boolean
required: false
default: true
go-test-json-artifact:
description: >
Artifact (from the calling run) holding go-test.json, PR Core's
`go-test -json` on this commit; enables the equivalence skip-parity
check. nightly.yml sets it.
type: string
required: false
default: ""
# Main runs always finish (they may save the runner cache); PR runs supersede.
concurrency:
group: bazel-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
env:
BD_DISABLE_METRICS: "1"
BD_DISABLE_EVENT_FLUSH: "1"
# Phase B report-only budget for T1/T2 (tools/bazel/critpath.py).
BAZEL_BUDGET_SECONDS: "420"
jobs:
bazel-test:
name: Bazel / test (advisory)
# Remote-execution runs use the org's Blacksmith pool (the farm's edge
# admits it; GitHub-hosted runners are refused) and do not count against
# GitHub's concurrent-job cap. Fork PRs and rbe=off have no remote
# executor and build everything locally, so they get the larger
# GitHub-hosted runner.
runs-on: ${{ (inputs.rbe || 'on') != 'off' && github.event.pull_request.head.repo.fork != true && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
# Remote execution goes to rbe-west (instance "oss", the Blacksmith-backed
# worker pool shared with gascity), switched on by the repo variable
# RBE_WEST_WORKERS. Until it is set, only fork PRs and rbe=off dispatches
# run (locally); same-repo runs are skipped rather than failing on an
# unreachable executor.
if: ${{ vars.RBE_WEST_WORKERS == 'true' || inputs.rbe == 'off' || github.event.pull_request.head.repo.fork == true }}
timeout-minutes: 100
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
# Go is for tools/bazel/equivalence.py's `go list` and for make; Bazel
# uses its own hermetic SDK.
- name: Set up Go
id: setup-go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: 'go.mod'
cache: false
# Read-only: main.yml's build-artifacts job owns this cache.
- name: Restore Go module cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/go/pkg/mod
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
- name: Set up Bazel
id: bazel
uses: ./.github/actions/setup-bazel
env:
BAZEL_REMOTE_EXECUTOR: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_WEST_EXECUTOR || '' }}
RBE_TLS_CERT: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_TLS_CERT || '' }}
RBE_TLS_KEY: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_TLS_KEY || '' }}
RBE_TLS_CA: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && 'oss' || '' }}
# T1: remote + restored repository cache; T2: remote, cold runner; local:
# no remote execution (reported, never budgeted).
- name: Measurement tier
id: tier
env:
REMOTE: ${{ steps.bazel.outputs.remote }}
RESTORED: ${{ steps.bazel.outputs.cache-restored }}
run: |
if [ "$REMOTE" = "true" ] && [ "$RESTORED" = "true" ]; then tier=T1
elif [ "$REMOTE" = "true" ]; then tier=T2
else tier=local; fi
echo "tier=$tier" >> "$GITHUB_OUTPUT"
{
echo "### Bazel / test (advisory)"
echo
echo "Tier **$tier**: remote execution $REMOTE, runner cache restored $RESTORED."
echo
} >> "$GITHUB_STEP_SUMMARY"
# --profile and --build_event_json_file are command options, so they go
# after `test` (a startup position fails).
- name: bazel test //... --config=ci
id: test
timeout-minutes: 90
run: |
set -o pipefail
start=$(date +%s)
rc=0
bazel test //... --config=ci \
--profile="$RUNNER_TEMP/bazel-profile.json" \
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
echo "bazel test: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
echo >> "$GITHUB_STEP_SUMMARY"
exit "$rc"
- name: Critical-path report
if: ${{ always() && steps.test.outcome != 'skipped' }}
env:
TIER: ${{ steps.tier.outputs.tier }}
run: |
budget=()
if [ "$TIER" != "local" ]; then budget=(--budget "$BAZEL_BUDGET_SECONDS"); fi
python3 tools/bazel/critpath.py "$RUNNER_TEMP/bazel-profile.json" \
--tier "$TIER" --log "$RUNNER_TEMP/bazel-test.log" "${budget[@]}"
# nightly.yml's PR Core `go test -json` on the same commit.
- name: Download go-test results
if: ${{ always() && inputs.go-test-json-artifact != '' && steps.test.outcome != 'skipped' }}
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: ${{ inputs.go-test-json-artifact }}
path: ${{ runner.temp }}/go-test-json
# Fails on any Go test that `go test` runs and Bazel did not (tag filter,
# manual, missing target, empty shard) unless it is allowlisted. With
# go test results (nightly) it also fails on tests that passed under go
# test and were skipped under Bazel; a PR run cannot see those.
- name: Go test equivalence
if: ${{ always() && steps.test.outcome != 'skipped' }}
timeout-minutes: 15
env:
DETAILS: ${{ inputs.details && '--details' || '' }}
GO_TEST_JSON: ${{ inputs.go-test-json-artifact != '' && format('{0}/go-test-json/go-test.json', runner.temp) || '' }}
run: |
args=()
if [ -n "$DETAILS" ]; then args+=("$DETAILS"); fi
if [ -n "$GO_TEST_JSON" ]; then args+=(--go-test-json "$GO_TEST_JSON"); fi
python3 tools/bazel/equivalence.py --bep "$RUNNER_TEMP/bazel-bep.json" "${args[@]}"
# Runs last: it rewrites BUILD files in the checkout. main moves BUILD
# inputs (imports, go.mod) without touching Bazel, so drift shows up here
# first.
- name: BUILD files in sync (gazelle, go_srcs, MODULE.bazel)
if: ${{ always() && steps.bazel.outcome == 'success' }}
timeout-minutes: 15
run: |
start=$(date +%s)
make bazel-sync-check
make bazel-sync
drift="$(git status --porcelain --untracked-files=all)"
echo "BUILD sync check: $(( $(date +%s) - start ))s" >> "$GITHUB_STEP_SUMMARY"
if [ -n "$drift" ]; then
echo "::error title=Bazel BUILD files out of sync::run 'make bazel-sync' and commit the result"
echo "$drift"
git diff | head -200
echo "BUILD files **out of sync**; run \`make bazel-sync\` and commit the result." >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
- name: Upload test logs
if: ${{ failure() && steps.test.outcome == 'failure' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: bazel-testlogs
path: |
bazel-testlogs/**/test.log
bazel-testlogs/**/test.xml
if-no-files-found: ignore
retention-days: 7
# Only a push to main that missed the exact key writes the runner cache.
# Its content (--repository_cache and Bazelisk's download, never the rc
# or credentials) is fixed by .bazelversion and MODULE.bazel.lock, so one
# entry per lock is enough and later pushes do not churn the quota.
- name: Save Bazel runner cache
if: ${{ always() && github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.bazel.outcome == 'success' && steps.bazel.outputs.cache-hit != 'true' }}
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ runner.temp }}/bazel-ci-cache
key: bazel-repo-v2-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }}

View file

@ -26,6 +26,7 @@ on:
- nix
- mcp-package
- npm-package
- bazel
smoke_version:
description: "Optional previous release for cross-version-smoke"
required: false
@ -38,12 +39,75 @@ env:
jobs:
measure:
name: Measure selected CI suite
if: ${{ github.event_name == 'workflow_dispatch' && inputs.suite != 'full-test' }}
if: ${{ github.event_name == 'workflow_dispatch' && inputs.suite != 'full-test' && inputs.suite != 'bazel' }}
uses: ./.github/workflows/ci-measurements.yml
with:
suite: ${{ inputs.suite }}
smoke_version: ${{ inputs.smoke_version }}
# PR Core's go test (scripts/ci/pr-core.sh) with -json output, so the Bazel
# job below can compare per-test status: a test that passes here and skips
# under Bazel fails the equivalence check unless allowlisted.
bazel-go-test-json:
name: Bazel equivalence (go-test side)
if: ${{ github.event_name == 'schedule' || inputs.suite == 'bazel' }}
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
- name: Set up Go
id: setup-go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: 'go.mod'
cache: false
- name: Restore Go module cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/go/pkg/mod
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
- name: Install Dolt
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
git config --global user.name "CI Bot"
git config --global user.email "ci@beads.test"
dolt config --global --add user.name "CI Bot"
dolt config --global --add user.email "ci@beads.test"
- name: Run PR core go-test (JSON)
env:
BEADS_PR_CORE_GO_TEST_JSON: ${{ runner.temp }}/go-test.json
run: make ci-pr-core
# Uploaded even when tests fail: the status comparison still applies.
- name: Upload go-test results
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: bazel-equivalence-go-test-json
path: ${{ runner.temp }}/go-test.json
if-no-files-found: error
retention-days: 7
# Bazel lane with the full go test equivalence listing and the skip-parity
# check (advisory, like the PR run). Remote execution when the repo secrets
# are present.
bazel:
name: Bazel equivalence (full detail)
needs: bazel-go-test-json
if: ${{ !cancelled() && (github.event_name == 'schedule' || inputs.suite == 'bazel') }}
uses: ./.github/workflows/bazel.yml
secrets: inherit
with:
details: true
go-test-json-artifact: bazel-equivalence-go-test-json
full-test:
name: Full Test Suite
if: ${{ github.event_name == 'schedule' || inputs.suite == 'full-test' }}

View file

@ -258,6 +258,15 @@ func TestContextRepository_RepoContextAndRole(t *testing.T) {
require.NoError(t, os.MkdirAll(beadsDir, 0o750))
require.NoError(t, os.WriteFile(filepath.Join(beadsDir, "metadata.json"), []byte("{}"), 0o600))
// Run from a second, unrelated git repo so BEADS_DIR is external to the
// CWD repo. Relying on the test's own CWD being inside a checkout breaks
// under Bazel, whose runfiles tree is not a git repository.
cwdRepo := t.TempDir()
cmd = exec.Command("git", "init")
cmd.Dir = cwdRepo
require.NoError(t, cmd.Run())
t.Chdir(cwdRepo)
t.Setenv("BEADS_DIR", beadsDir)
beads.ResetCaches()
git.ResetCaches()

View file

@ -30,5 +30,15 @@ GO_TEST_PARALLEL="${GO_TEST_PARALLEL:-4}"
# main.yml's integration package and cmd/bd shards, ci-measurements.yml,
# nightly.yml. Splitting or de-slowing cmd/bd is the real fix and belongs on main
# — this only stops the clock from being the thing that fails.
# BEADS_PR_CORE_GO_TEST_JSON=<file>: same run, `go test -json` output to <file>
# (nightly.yml feeds it to tools/bazel/equivalence.py --go-test-json).
go_test() {
if [[ -n "${BEADS_PR_CORE_GO_TEST_JSON:-}" ]]; then
go test -json "$@" >"$BEADS_PR_CORE_GO_TEST_JSON"
else
go test "$@"
fi
}
ci_time "pr-core go test" -- \
go test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -race -short -timeout=30m -skip '^TestEmbedded' ./...
go_test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -race -short -timeout=30m -skip '^TestEmbedded' ./...

View file

@ -1,11 +1,14 @@
package scripts_test
import (
"encoding/base64"
"fmt"
"os"
"os/exec"
"path/filepath"
"reflect"
"regexp"
"sort"
"strings"
"testing"
@ -1414,3 +1417,521 @@ func captureOne(t *testing.T, pattern, body, source string) string {
}
return matches[1]
}
// --- Bazel lane (.github/workflows/bazel.yml, advisory) ----------------------
const (
bazelWorkflowName = "bazel.yml"
bazelJobName = "bazel-test"
setupBazelActionDir = ".github/actions/setup-bazel"
uploadArtifactSHA = "043fb46d1a93c77aae656e7c1c64a875d1fc6a0a"
downloadArtifactSHA = "3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c"
checkoutSHA = "3d3c42e5aac5ba805825da76410c181273ba90b1"
bazelCacheKeyPrefix = "bazel-repo-v2-${{ runner.os }}-"
bazelCacheKey = bazelCacheKeyPrefix + "${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }}"
bazelCachePath = "${{ runner.temp }}/bazel-ci-cache"
// Save only from a push to main that missed the exact key: the content is
// fixed by the key, so re-saving every push only churns the quota.
bazelCacheSaveIf = "${{ always() && github.event_name == 'push' && github.ref == 'refs/heads/main' && " +
"steps.bazel.outcome == 'success' && steps.bazel.outputs.cache-hit != 'true' }}"
)
// The only triggers bazel.yml may have. pull_request_target (and
// workflow_run) would run with secrets in the context of fork PRs.
var bazelWorkflowTriggers = []string{"merge_group", "pull_request", "push", "workflow_call", "workflow_dispatch"}
type ciCompositeAction struct {
Runs struct {
Using string `yaml:"using"`
Steps []ciWorkflowStep `yaml:"steps"`
} `yaml:"runs"`
}
func readSetupBazelAction(t *testing.T) ciCompositeAction {
t.Helper()
path := filepath.Join(sourceRepoRoot(t), setupBazelActionDir, "action.yml")
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var action ciCompositeAction
if err := yaml.Unmarshal(data, &action); err != nil {
t.Fatalf("parse %s: %v", path, err)
}
if action.Runs.Using != "composite" {
t.Fatalf("%s runs.using = %q, want composite", path, action.Runs.Using)
}
return action
}
// The Bazel lane is advisory until the equivalence record and a human promote
// it: no ci-gate may need it or list it in CI_GATE_REQUIRED.
func TestBazelWorkflowIsAdvisory(t *testing.T) {
workflow := readCIWorkflow(t, bazelWorkflowName)
if len(workflow.Jobs) != 1 {
t.Errorf("%s jobs = %d, want only %q", bazelWorkflowName, len(workflow.Jobs), bazelJobName)
}
job := workflow.job(t, bazelJobName)
// The farm admits the Blacksmith pool only; forks (no secrets) and rbe=off
// build locally on the GitHub-hosted runner.
if want := "${{ (inputs.rbe || 'on') != 'off' && github.event.pull_request.head.repo.fork != true && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}"; job.RunsOn != want {
t.Errorf("%s runs-on = %q, want %q", bazelJobName, job.RunsOn, want)
}
if job.ContinueOnError {
t.Errorf("%s continue-on-error hides failures; the lane is advisory by staying out of ci-gate", bazelJobName)
}
for _, name := range []string{"pr.yml", "pr-risk.yml"} {
for jobName, j := range readCIWorkflow(t, name).Jobs {
for _, need := range j.Needs {
if strings.Contains(strings.ToLower(need), "bazel") {
t.Errorf("%s job %q needs %q; the Bazel lane is advisory", name, jobName, need)
}
}
for _, step := range j.Steps {
for key, value := range step.Env {
if strings.Contains(strings.ToUpper(key), "BAZEL") ||
(key == "CI_GATE_REQUIRED" && strings.Contains(strings.ToUpper(value), "BAZEL")) {
t.Errorf("%s job %q step %q env %s=%q wires the Bazel lane into a gate", name, jobName, step.Name, key, value)
}
}
}
}
}
}
// Every action in the Bazel lane is pinned to a full commit SHA, with the same
// released SHAs the Go lanes use, and the monolithic actions/cache is banned
// here too (restore everywhere, save only where the topology says).
func TestBazelWorkflowActionsArePinned(t *testing.T) {
steps := map[string][]ciWorkflowStep{
bazelWorkflowName: readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName).Steps,
setupBazelActionDir + "/action.yml": readSetupBazelAction(t).Runs.Steps,
}
want := map[string]string{
"actions/checkout": checkoutSHA,
setupGoActionFamily: setupGoSHA,
cacheRestoreActionFamily: cacheSHA,
cacheSaveActionFamily: cacheSHA,
"actions/upload-artifact": uploadArtifactSHA,
"actions/download-artifact": downloadArtifactSHA,
}
for file, list := range steps {
for _, step := range list {
if step.Uses == "" || strings.HasPrefix(step.Uses, "./") {
continue
}
family, sha, found := strings.Cut(step.Uses, "@")
if !found || !actionPin.MatchString(sha) {
t.Errorf("%s step %q action %q is not pinned to a 40-hex commit SHA", file, step.Name, step.Uses)
continue
}
if isForbiddenGoCacheActionFamily(family) {
t.Errorf("%s step %q uses forbidden monolithic %q; use actions/cache/restore and /save", file, step.Name, family)
continue
}
wantSHA, known := want[family]
if !known {
t.Errorf("%s step %q uses %q, which this policy does not know; add its released SHA here", file, step.Name, family)
} else if sha != wantSHA {
t.Errorf("%s step %q action %q has SHA %q, want %q", file, step.Name, family, sha, wantSHA)
}
if family == setupGoActionFamily && step.With["cache"] != "false" {
t.Errorf("%s setup-go cache = %q, want false", file, step.With["cache"])
}
if family == cacheSaveActionFamily && file != bazelWorkflowName {
t.Errorf("%s step %q saves a cache; only %s's push-to-main step may", file, step.Name, bazelWorkflowName)
}
}
for _, step := range list {
if strings.Contains(step.Run, "apt-get") {
t.Errorf("%s step %q runs apt-get; bound it (see ci_workflow_apt_get_timeout_test.go) or drop it", file, step.Name)
}
}
}
}
// The runner cache holds only content fixed by .bazelversion and
// MODULE.bazel.lock (repository cache, Bazelisk), so it is keyed on exactly
// that and written only by a push to main that missed the key; it never holds
// the generated rc or credentials.
func TestBazelWorkflowCacheTopology(t *testing.T) {
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName)
var saves []ciWorkflowStep
for _, step := range job.Steps {
if actionFamily(step.Uses) == cacheSaveActionFamily {
saves = append(saves, step)
}
}
if len(saves) != 1 {
t.Fatalf("%s has %d cache save steps, want 1", bazelWorkflowName, len(saves))
}
save := saves[0]
if save.If != bazelCacheSaveIf {
t.Errorf("cache save if = %q, want exactly %q", save.If, bazelCacheSaveIf)
}
if save.With["key"] != bazelCacheKey || save.With["path"] != bazelCachePath {
t.Errorf("cache save key/path = %q / %q, want %q / %q", save.With["key"], save.With["path"], bazelCacheKey, bazelCachePath)
}
var restore, writer ciWorkflowStep
for _, step := range readSetupBazelAction(t).Runs.Steps {
switch {
case actionFamily(step.Uses) == cacheRestoreActionFamily:
restore = step
case step.Env["BAZEL_CI_SECRET_DIR"] != "":
writer = step
}
}
if restore.With["path"] != bazelCachePath || restore.With["key"] != bazelCacheKey ||
strings.TrimSpace(restore.With["restore-keys"]) != bazelCacheKeyPrefix {
t.Errorf("setup-bazel restore path/key/restore-keys = %q / %q / %q; want the save step's path and key with a %q prefix fallback",
restore.With["path"], restore.With["key"], restore.With["restore-keys"], bazelCacheKeyPrefix)
}
if writer.Env["BAZEL_CI_CACHE_DIR"] != bazelCachePath {
t.Errorf("rc writer BAZEL_CI_CACHE_DIR = %q, want %q", writer.Env["BAZEL_CI_CACHE_DIR"], bazelCachePath)
}
if secret := writer.Env["BAZEL_CI_SECRET_DIR"]; !strings.HasPrefix(secret, "${{ runner.temp }}/") || strings.HasPrefix(secret, bazelCachePath) {
t.Errorf("rc writer BAZEL_CI_SECRET_DIR = %q; want a runner.temp directory outside the cached %q", secret, bazelCachePath)
}
}
// The lane runs the committed ci config over //..., reports the critical path
// and the go test equivalence even when tests fail, catches BUILD drift, and
// hands secrets to nothing but setup-bazel.
func TestBazelWorkflowRunsCIConfigWithReports(t *testing.T) {
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName)
if job.TimeoutMinutes == 0 {
t.Errorf("%s has no timeout-minutes", bazelJobName)
}
testStep := job.step(t, "bazel test //... --config=ci")
for _, required := range []string{"bazel test //... --config=ci", "--profile=", "--build_event_json_file=", "set -o pipefail"} {
if !strings.Contains(testStep.Run, required) {
t.Errorf("test step does not contain %q:\n%s", required, testStep.Run)
}
}
if strings.Contains(testStep.Run, "--config=remote-exec") {
t.Errorf("test step selects remote-exec itself; setup-bazel's rc does that only when secrets are present")
}
if i, j := strings.Index(testStep.Run, "bazel test"), strings.Index(testStep.Run, "--profile="); j < i {
t.Errorf("--profile is a command option and must follow `bazel test`")
}
for name, script := range map[string]string{
"Critical-path report": "tools/bazel/critpath.py",
"Go test equivalence": "tools/bazel/equivalence.py",
} {
step := job.step(t, name)
if !strings.Contains(step.Run, script) || !strings.Contains(step.If, "always()") {
t.Errorf("step %q must run %s under always(); run=%q if=%q", name, script, step.Run, step.If)
}
assertReportStepKeepsExitStatus(t, step, "python3 "+script)
}
assertTestStepKeepsExitStatus(t, testStep)
sync := job.step(t, "BUILD files in sync (gazelle, go_srcs, MODULE.bazel)")
for _, required := range []string{"make bazel-sync-check", "make bazel-sync\n", "git status --porcelain --untracked-files=all", "exit 1"} {
if !strings.Contains(sync.Run, required) {
t.Errorf("sync step does not contain %q:\n%s", required, sync.Run)
}
}
}
// swallowedExit matches shell that hides a failing command's status.
var swallowedExit = regexp.MustCompile(`\bexit\b|\|\|\s*(true|:)|;\s*(true|:)\s*$|\bset\s+\+e\b|\btrap\b`)
// shellCommands joins backslash continuations and drops blank and comment
// lines.
func shellCommands(run string) []string {
var cmds []string
cur := ""
for _, line := range strings.Split(run, "\n") {
trimmed := strings.TrimSpace(line)
if cur == "" && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
continue
}
if strings.HasSuffix(trimmed, "\\") {
cur += strings.TrimSuffix(trimmed, "\\") + " "
continue
}
cmds = append(cmds, strings.Join(strings.Fields(cur+trimmed), " "))
cur = ""
}
if cur != "" {
cmds = append(cmds, strings.Join(strings.Fields(cur), " "))
}
return cmds
}
// A report step's status is its script's: the script is the last command,
// nothing after it on that line, and nothing in the step can mask a failure.
func assertReportStepKeepsExitStatus(t *testing.T, step ciWorkflowStep, invocation string) {
t.Helper()
if step.Shell != "" {
t.Errorf("step %q overrides shell %q; the default bash -e is part of the contract", step.Name, step.Shell)
}
cmds := shellCommands(step.Run)
if len(cmds) == 0 {
t.Errorf("step %q runs nothing", step.Name)
return
}
last := cmds[len(cmds)-1]
if !strings.HasPrefix(last, invocation+" ") || strings.ContainsAny(last, ";&|") {
t.Errorf("step %q must end with a bare %q so its exit status is the step's; last command %q", step.Name, invocation, last)
}
for _, cmd := range cmds {
if swallowedExit.MatchString(cmd) {
t.Errorf("step %q command %q can hide the script's exit status", step.Name, cmd)
}
}
}
// The bazel test step captures bazel's status through tee and must exit
// with it.
func assertTestStepKeepsExitStatus(t *testing.T, step ciWorkflowStep) {
t.Helper()
if step.Shell != "" {
t.Errorf("step %q overrides shell %q", step.Name, step.Shell)
}
cmds := shellCommands(step.Run)
if len(cmds) == 0 || cmds[len(cmds)-1] != `exit "$rc"` {
t.Errorf("step %q must end with exit \"$rc\"; commands %q", step.Name, cmds)
}
exits := 0
for _, cmd := range cmds {
if regexp.MustCompile(`(^|[;&|]\s*)exit\b`).MatchString(cmd) {
exits++
}
if regexp.MustCompile(`\|\|\s*(true|:)|;\s*(true|:)\s*$|\bset\s+\+e\b|\btrap\b`).MatchString(cmd) {
t.Errorf("step %q command %q can hide bazel's exit status", step.Name, cmd)
}
}
if exits != 1 {
t.Errorf("step %q has %d exit commands, want only the final exit \"$rc\"", step.Name, exits)
}
}
func readYAMLNode(t *testing.T, rel string) *yaml.Node {
t.Helper()
path := filepath.Join(sourceRepoRoot(t), rel)
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var doc yaml.Node
if err := yaml.Unmarshal(data, &doc); err != nil {
t.Fatalf("parse %s: %v", path, err)
}
if doc.Kind != yaml.DocumentNode || len(doc.Content) != 1 || doc.Content[0].Kind != yaml.MappingNode {
t.Fatalf("%s: want a YAML mapping document", path)
}
return doc.Content[0]
}
// walkYAML calls fn with the dotted path of every mapping key and scalar.
func walkYAML(node *yaml.Node, path string, fn func(path string, key bool, value string)) {
switch node.Kind {
case yaml.MappingNode:
for i := 0; i+1 < len(node.Content); i += 2 {
k := node.Content[i].Value
fn(path+"."+k, true, k)
walkYAML(node.Content[i+1], path+"."+k, fn)
}
case yaml.SequenceNode:
for i, item := range node.Content {
walkYAML(item, fmt.Sprintf("%s[%d]", path, i), fn)
}
case yaml.ScalarNode:
fn(path, false, node.Value)
case yaml.AliasNode:
if node.Alias != nil {
walkYAML(node.Alias, path, fn)
}
}
}
func yamlMapKeys(node *yaml.Node, key string) []string {
for i := 0; i+1 < len(node.Content); i += 2 {
if node.Content[i].Value != key {
continue
}
v := node.Content[i+1]
switch v.Kind {
case yaml.MappingNode:
var keys []string
for j := 0; j < len(v.Content); j += 2 {
keys = append(keys, v.Content[j].Value)
}
return keys
case yaml.SequenceNode:
var keys []string
for _, item := range v.Content {
keys = append(keys, item.Value)
}
return keys
default:
return []string{v.Value}
}
}
return nil
}
// bazel.yml hands the RBE credentials to PR-head code on same-repo PRs (an
// accepted, documented risk). Keep that exposure from growing: only the
// listed triggers (never pull_request_target), secrets referenced from the
// setup-bazel step's env and nowhere else (not workflow/job env, run, with or
// if), and no continue-on-error anywhere, so a red lane stays red.
func TestBazelWorkflowSecretsAndFailureSurface(t *testing.T) {
root := readYAMLNode(t, filepath.Join(".github", "workflows", bazelWorkflowName))
triggers := yamlMapKeys(root, "on")
sort.Strings(triggers)
if !reflect.DeepEqual(triggers, bazelWorkflowTriggers) {
t.Errorf("%s triggers = %v, want exactly %v", bazelWorkflowName, triggers, bazelWorkflowTriggers)
}
setupSteps := map[string]bool{}
for i, step := range readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName).Steps {
if step.Uses == "./"+setupBazelActionDir {
setupSteps[fmt.Sprintf(".jobs.%s.steps[%d].env.", bazelJobName, i)] = true
}
}
if len(setupSteps) != 1 {
t.Errorf("%s has %d setup-bazel steps, want 1", bazelWorkflowName, len(setupSteps))
}
secretRef := regexp.MustCompile(`\bsecrets\s*(\.|\[)`)
walkYAML(root, "", func(path string, key bool, value string) {
if key && value == "continue-on-error" {
t.Errorf("%s: %s hides failures; the lane is advisory by staying out of ci-gate", bazelWorkflowName, path)
}
if key && (value == "secrets" && strings.HasPrefix(path, ".jobs.")) {
t.Errorf("%s: %s passes secrets to a called workflow or container", bazelWorkflowName, path)
}
if key || !secretRef.MatchString(value) {
return
}
prefix := path[:strings.LastIndex(path, ".")+1]
if !setupSteps[prefix] {
t.Errorf("%s: %s reads secrets (%q); only the setup-bazel step's env may", bazelWorkflowName, path, value)
}
})
action := readYAMLNode(t, filepath.Join(setupBazelActionDir, "action.yml"))
walkYAML(action, "", func(path string, key bool, value string) {
if key && value == "continue-on-error" {
t.Errorf("setup-bazel: %s hides failures", path)
}
if !key && secretRef.MatchString(value) {
t.Errorf("setup-bazel: %s reads secrets directly; the caller passes them in the step env", path)
}
})
}
// write-bazelrc.sh: no secrets means a local-only rc (fork PRs), a partial set
// fails, and the full set enables remote-exec with the key material and rc
// outside the workspace and the runner cache.
func TestSetupBazelRCWriter(t *testing.T) {
bash := requireHostTool(t, "bash")
script := filepath.Join(sourceRepoRoot(t), setupBazelActionDir, "write-bazelrc.sh")
pem := func(kind string) string {
body := "-----BEGIN " + kind + "-----\nMIIBfake" + strings.ReplaceAll(kind, " ", "") + "\nline2\n-----END " + kind + "-----\n"
return base64.StdEncoding.EncodeToString([]byte(body))
}
run := func(t *testing.T, extra ...string) (string, string, string, error) {
t.Helper()
dir := t.TempDir()
workspace := filepath.Join(dir, "ws")
if err := os.MkdirAll(workspace, 0o755); err != nil {
t.Fatal(err)
}
out := filepath.Join(dir, "github_output")
secret := filepath.Join(dir, "secret")
cmd := exec.Command(bash, script)
cmd.Dir = workspace
cmd.Env = append([]string{
"PATH=" + os.Getenv("PATH"),
"GITHUB_WORKSPACE=" + workspace,
"GITHUB_OUTPUT=" + out,
"BAZEL_CI_CACHE_DIR=" + filepath.Join(dir, "cache"),
"BAZEL_CI_SECRET_DIR=" + secret,
}, extra...)
logs, err := cmd.CombinedOutput()
outputs, _ := os.ReadFile(out)
rc, _ := os.ReadFile(filepath.Join(secret, "ci.bazelrc"))
return string(outputs), string(rc), string(logs), err
}
executor := "BAZEL_REMOTE_EXECUTOR=grpcs://" + "farm.invalid:443"
t.Run("no secrets stays local", func(t *testing.T) {
outputs, rc, logs, err := run(t)
if err != nil {
t.Fatalf("err=%v\n%s", err, logs)
}
if !strings.Contains(outputs, "remote=false") || strings.Contains(rc, "remote") || strings.Contains(rc, "tls") {
t.Errorf("outputs=%q rc=%q; want remote=false and no remote flags", outputs, rc)
}
if !strings.Contains(rc, "--repository_cache=") || strings.Contains(rc, "--disk_cache=") {
t.Errorf("local rc = %q; want --repository_cache and no --disk_cache (local runs never save it)", rc)
}
})
for name, env := range map[string][]string{
"executor only": {executor},
"cert and key": {"RBE_TLS_CERT=" + pem("CERTIFICATE"), "RBE_TLS_KEY=" + pem("PRIVATE KEY")},
"executor no key": {executor, "RBE_TLS_CERT=" + pem("CERTIFICATE")},
"cert not pem b64": {executor, "RBE_TLS_CERT=bm90IGEgcGVt", "RBE_TLS_KEY=" + pem("PRIVATE KEY")},
} {
t.Run("rejects "+name, func(t *testing.T) {
if _, _, logs, err := run(t, env...); err == nil {
t.Errorf("want failure, got success:\n%s", logs)
}
})
}
t.Run("full set enables remote-exec", func(t *testing.T) {
outputs, rc, logs, err := run(t, executor, "RBE_TLS_CERT="+pem("CERTIFICATE"), "RBE_TLS_KEY="+pem("PRIVATE KEY"))
if err != nil {
t.Fatalf("err=%v\n%s", err, logs)
}
for _, want := range []string{
"build:remote-exec --remote_executor=grpcs://farm.invalid:443", "--tls_client_certificate=", "--tls_client_key=",
"build:remote-exec --noremote_upload_local_results", "build --config=remote-exec",
} {
if !strings.Contains(rc, want) {
t.Errorf("rc lacks %q:\n%s", want, rc)
}
}
if strings.Contains(rc, "--disk_cache") || strings.Contains(rc, "--remote_instance_name") {
t.Errorf("rc sets --disk_cache or an instance nobody asked for:\n%s", rc)
}
if !strings.Contains(outputs, "remote=true") {
t.Errorf("outputs = %q, want remote=true", outputs)
}
for _, want := range []string{"::add-mask::farm.invalid\n", "::add-mask::MIIBfakeCERTIFICATE\n", "::add-mask::MIIBfakePRIVATEKEY\n", "::add-mask::line2\n", "::warning title=RBE_INSTANCE not set::"} {
if !strings.Contains(logs, want) {
t.Errorf("log lacks %q:\n%s", want, logs)
}
}
if strings.Contains(logs, "::add-mask::-----") {
t.Errorf("PEM armor lines are masked (they would blank every PEM header in the log):\n%s", logs)
}
})
t.Run("CA and instance", func(t *testing.T) {
_, rc, logs, err := run(t, executor, "RBE_TLS_CERT="+pem("CERTIFICATE"), "RBE_TLS_KEY="+pem("PRIVATE KEY"),
"RBE_TLS_CA="+pem("CA CERT"), "RBE_INSTANCE=beads")
if err != nil {
t.Fatalf("err=%v\n%s", err, logs)
}
for _, want := range []string{"build:remote-exec --tls_certificate=", "build:remote-exec --remote_instance_name=beads"} {
if !strings.Contains(rc, want) {
t.Errorf("rc lacks %q:\n%s", want, rc)
}
}
if !strings.Contains(logs, "::add-mask::MIIBfakeCACERT\n") || strings.Contains(logs, "RBE_INSTANCE not set") {
t.Errorf("want the CA masked and no instance warning:\n%s", logs)
}
})
t.Run("secret dir inside workspace fails", func(t *testing.T) {
dir := t.TempDir()
cmd := exec.Command(bash, script)
cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "GITHUB_WORKSPACE=" + dir,
"BAZEL_CI_CACHE_DIR=" + filepath.Join(t.TempDir(), "c"), "BAZEL_CI_SECRET_DIR=" + filepath.Join(dir, "s")}
if out, err := cmd.CombinedOutput(); err == nil {
t.Errorf("want failure for a secret dir inside the workspace:\n%s", out)
}
})
}

175
tools/bazel/critpath.py Normal file
View file

@ -0,0 +1,175 @@
#!/usr/bin/env python3
"""Critical-path report and CI budget check for a `bazel --profile` JSON.
gap = elapsed - critical_path
The critical path is the SUM of the chained "critical path component" events
Bazel records, not the single longest one: a cold race build is a long chain
of compile/link actions, and reporting only its longest member understates the
path and turns the difference into a phantom "scheduling gap".
Decision rule:
gap < 15% of elapsed the chain itself is the bottleneck. Attack its
longest member (shard/speed up a test, split or slim
a build action, warm the CAS if input upload
dominates).
gap >= 15% of elapsed actions waited for capacity: raise --jobs, grow the
executor pool, restore the runner disk cache.
Measurement tiers (caching makes single numbers lie):
T1 steady: warm remote cache + restored runner disk cache (the PR loop).
T2 cold client: fresh runner disk cache, warm remote cache.
T3 cold farm: after worker recycle / CAS eviction. Tracked, never gated.
local no remote execution (fork PRs). Reported, never gated.
Usage:
critpath.py PROFILE [--tier T2] [--budget SECONDS] [--enforce]
[--log BAZEL_OUTPUT_LOG] [--top N] [--summary FILE]
Without --enforce an over-budget run prints a GitHub warning annotation and
exits 0 (report-only phase); with --enforce it exits 1.
"""
import argparse
import json
import os
import re
import sys
CP_CAT = "critical path component"
DECISION_GAP_PCT = 15.0
def load_events(path):
with open(path, encoding="utf-8") as f:
prof = json.load(f)
if isinstance(prof, list):
return prof
return prof.get("traceEvents", [])
def short_name(name, width=90):
# "action 'GoLink cmd/bd/bd_test_/bd_test'" -> "GoLink cmd/bd/bd_test_/bd_test"
m = re.match(r"action '(.*)'$", name)
if m:
name = m.group(1)
return name if len(name) <= width else name[: width - 1] + "…"
def processes_line(log_path):
"""Return Bazel's final "N processes: ..." line from its console log."""
if not log_path or not os.path.exists(log_path):
return None
found = None
with open(log_path, encoding="utf-8", errors="replace") as f:
for line in f:
m = re.search(r"(\d[\d,]* processes?: .*)$", line.strip())
if m:
found = m.group(1)
return found
def analyze(events):
elapsed = max((e.get("dur", 0) for e in events if e.get("name") == "buildTargets"), default=0) / 1e6
stamps = [e["ts"] for e in events if e.get("ts")]
wall = (max(stamps) - min(stamps)) / 1e6 if stamps else 0.0
chain = [e for e in events if e.get("cat") == CP_CAT]
cp = sum(e.get("dur", 0) for e in chain) / 1e6
longest = sorted(chain, key=lambda e: e.get("dur", 0), reverse=True)
return {
"elapsed": elapsed,
"wall": wall,
"cp": cp,
"chain_len": len(chain),
"longest": longest,
}
def next_step(res):
elapsed, cp = res["elapsed"], res["cp"]
gap = max(elapsed - cp, 0.0)
pct = gap / elapsed * 100 if elapsed else 0.0
if not res["longest"]:
return pct, "NEXT: no critical path recorded (nothing executed, or a fully cached run)"
if pct < DECISION_GAP_PCT:
top = res["longest"][0]
name = short_name(top["name"], 200)
secs = top.get("dur", 0) / 1e6
m = re.search(r"Testing (\S+)", name)
if m:
return pct, f"NEXT: shard or speed up {m.group(1)} ({secs:.0f}s)"
if "upload" in name.lower():
return pct, "NEXT: input upload dominates; warm the remote CAS or slim the declared inputs"
return pct, f"NEXT: attack {short_name(name, 70)} ({secs:.0f}s)"
return pct, "NEXT: scheduling gap; raise --jobs, grow the executor pool, or restore the runner disk cache"
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("profile")
ap.add_argument("--tier", default="T2")
ap.add_argument("--budget", type=float, default=None, help="seconds; compared with max(elapsed, wall)")
ap.add_argument("--enforce", action="store_true", help="exit 1 when over budget")
ap.add_argument("--log", help="bazel console log, for the 'processes:' cache line")
ap.add_argument("--top", type=int, default=8, help="longest chain members to list")
ap.add_argument("--summary", default=None, help="append Markdown here (default: $GITHUB_STEP_SUMMARY)")
args = ap.parse_args(argv)
if not os.path.exists(args.profile):
print(f"critpath: profile {args.profile} not found (bazel did not start?)", file=sys.stderr)
return 1 if args.enforce else 0
res = analyze(load_events(args.profile))
pct, advice = next_step(res)
gap = max(res["elapsed"] - res["cp"], 0.0)
procs = processes_line(args.log)
tier = args.tier
lines = [
f"[{tier}] wall clock {res['wall']:7.1f}s (server start to last event)",
f"[{tier}] elapsed {res['elapsed']:7.1f}s (build graph)",
f"[{tier}] critical path {res['cp']:7.1f}s (sum of {res['chain_len']} chained components)",
f"[{tier}] gap {gap:7.1f}s ({pct:.0f}% of elapsed)",
]
if procs:
lines.append(f"[{tier}] processes {procs}")
top = [e for e in res["longest"][: args.top] if e.get("dur", 0) > 0]
if top:
lines.append("")
lines.append(f"longest critical-path members (top {len(top)}):")
for e in top:
lines.append(f" {e['dur'] / 1e6:7.1f}s {short_name(e['name'])}")
lines.append("")
lines.append(advice)
verdict = None
over = False
if args.budget is not None:
gated = max(res["elapsed"], res["wall"])
over = gated > args.budget
mode = "enforcing" if args.enforce else "report-only"
if over:
verdict = f"BUDGET ({mode}): over, {gated:.1f}s > {args.budget:.0f}s ({tier})"
else:
verdict = f"BUDGET ({mode}): within, {gated:.1f}s <= {args.budget:.0f}s ({tier})"
lines.append("")
lines.append(verdict)
print("\n".join(lines))
if over and not args.enforce and os.environ.get("GITHUB_ACTIONS") == "true":
print(f"::warning title=Bazel critical-path budget::{verdict}")
summary = args.summary or os.environ.get("GITHUB_STEP_SUMMARY")
if summary:
with open(summary, "a", encoding="utf-8") as f:
f.write(
f"**Tier {tier}**: elapsed {res['elapsed']:.0f}s, wall {res['wall']:.0f}s, "
f"critical path {res['cp']:.0f}s ({res['chain_len']} components), gap {pct:.0f}%"
)
f.write(f"; {verdict}\n\n" if verdict else "\n\n")
f.write("<details><summary>Critical path</summary>\n\n```\n" + "\n".join(lines) + "\n```\n</details>\n\n")
return 1 if (over and args.enforce) else 0
if __name__ == "__main__":
sys.exit(main())

388
tools/bazel/equivalence.py Normal file
View file

@ -0,0 +1,388 @@
#!/usr/bin/env python3
"""Check that `bazel test --config=ci` ran the same Go tests `go test` would.
The Bazel lane is only a safe replacement for the `go test` lanes in pr.yml
(PR Core runs `go test -race -short -skip '^TestEmbedded' ./...`) if it runs
the same tests. A tag filter, a stray `manual` tag, a package gazelle never
picked up, or a sharding bug that makes a shard run zero tests all shrink the
Bazel set silently: every target Bazel did run is still green. This script
makes that shrinkage a failure.
Two checks always run, a third when `go test -json` output is supplied:
1. Package set. Every package `go list` reports with test files must have a
go_test target (`bazel query 'kind("go_test rule", //...)'`).
2. Test-name set. For every such package, each top-level `func TestXxx(t
*testing.T)` in the files `go list` selects (the same -race and
gms_pure_go build context as PR Core, minus the `^TestEmbedded` skip) must
appear as a <testcase> in the test.xml of a go_test target that this Bazel
invocation tested (read from --build_event_json_file). Bazel testcases
that `go test` would not run are reported too. This check only sees names:
a test that runs and passes under `go test` but calls t.Skip under Bazel
(a missing env var, no git checkout, a file that is not declared as data)
still counts as present here.
3. Skip parity (--go-test-json, nightly). Given the `go test -json` output of
the PR Core command on the same commit, every top-level test that passed
under `go test` and was skipped under Bazel is a divergence. The Bazel lane
alone cannot see this (a skip looks the same whether or not `go test` also
skips), so the nightly workflow runs PR Core's `go test -json` and passes
it in. Other status differences (pass/fail either way, skipped under go
test but run under Bazel) are reported, not failed: a failing test already
fails its own lane.
Divergences listed in the allowlist (tools/bazel/equivalence_allowlist.txt)
are expected; anything else fails. The allowlist format is one entry per line:
<package dir> <TestName or glob> # why Bazel does not run it
<package dir> <TestName or glob> skip # why Bazel skips a test go test runs
The first form (kind "run") covers the name checks; `*` as the test name
matches the whole package (including the package-set check). The `skip` form
covers only check 3. Every entry needs a justification after `#`.
Requires: go (with the module cache the packages need), bazel for the query
unless --no-query, and the BEP JSON plus bazel-testlogs of a `--config=ci`
run (which sets GO_TEST_WRAP_TESTV=1 and downloads test.xml).
"""
import argparse
import fnmatch
import json
import os
import re
import subprocess
import sys
import xml.etree.ElementTree as ET
GO_LIST_ARGS = ["list", "-e", "-race", "-tags", "gms_pure_go", "-json", "./..."]
# pr.yml's go test lanes pass -skip '^TestEmbedded' (as does test:prcore).
SKIP_RE = re.compile(r"^TestEmbedded")
TEST_FUNC_RE = re.compile(r"^func\s+(Test\w*)\s*\(\s*\w+\s+\*testing\.T\s*\)", re.M)
STATUS_RANK = {"skipped": 0, "passed": 1, "failed": 2}
GO_TEST_STATUS = {"pass": "passed", "fail": "failed", "skip": "skipped"}
ALLOW_KINDS = ("run", "skip")
DEFAULT_ALLOWLIST = os.path.join(os.path.dirname(os.path.abspath(__file__)), "equivalence_allowlist.txt")
def is_go_test_name(name):
# cmd/go: "TestXxx" where Xxx does not start with a lower-case letter.
return name == "Test" or not name[4].islower()
def go_expected(root, go_json_path=None):
"""Return ({pkg_dir: set(test names)} for packages with test files,
{import path: pkg_dir} for every package)."""
if go_json_path:
with open(go_json_path, encoding="utf-8") as f:
raw = f.read()
else:
raw = subprocess.run(["go", *GO_LIST_ARGS], cwd=root, check=True, capture_output=True, text=True).stdout
decoder = json.JSONDecoder()
pkgs, idx = [], 0
while idx < len(raw):
while idx < len(raw) and raw[idx].isspace():
idx += 1
if idx >= len(raw):
break
obj, idx = decoder.raw_decode(raw, idx)
pkgs.append(obj)
expected, dirs = {}, {}
for p in pkgs:
rel = os.path.relpath(p["Dir"], root)
rel = "" if rel == "." else rel.replace(os.sep, "/")
if p.get("ImportPath"):
dirs[p["ImportPath"]] = rel
files = (p.get("TestGoFiles") or []) + (p.get("XTestGoFiles") or [])
if not files:
continue
names = set()
for name in files:
with open(os.path.join(p["Dir"], name), encoding="utf-8", errors="replace") as f:
for m in TEST_FUNC_RE.finditer(f.read()):
t = m.group(1)
if is_go_test_name(t) and not SKIP_RE.search(t):
names.add(t)
expected[rel] = names
return expected, dirs
def go_test_statuses(path, dirs):
"""Return {pkg_dir: {test: status}} for top-level tests in `go test -json`
output. Packages go list does not know are skipped."""
out = {}
with open(path, encoding="utf-8", errors="replace") as f:
for line in f:
if not line.startswith("{"):
continue
try:
ev = json.loads(line)
except ValueError:
continue
test, action = ev.get("Test"), ev.get("Action")
if not test or "/" in test or action not in GO_TEST_STATUS:
continue
pkg = dirs.get(ev.get("Package", ""))
if pkg is None:
continue
out.setdefault(pkg, {})[test] = GO_TEST_STATUS[action]
return out
def label_pkg(label):
# "//cmd/bd:bd_test" -> "cmd/bd", "@@//:x" -> ""
label = label.lstrip("@")
return label[2:].split(":", 1)[0] if label.startswith("//") else label.split(":", 1)[0]
def query_go_test_pkgs(root, bazel):
out = subprocess.run(
[bazel, "query", 'kind("go_test rule", //...)', "--output=label"],
cwd=root, check=True, capture_output=True, text=True,
).stdout
return {label_pkg(l) for l in out.split()}
def read_bep(path):
"""Return ({label: shard_count} for tested go_test targets, set of go_test
labels configured, this invocation's testlogs dir or None)."""
go_tests, shards = set(), {}
exec_root, testlogs_rel = None, None
with open(path, encoding="utf-8") as f:
for line in f:
if not line.strip():
continue
ev = json.loads(line)
eid = ev.get("id", {})
if "workspace" in eid:
exec_root = ev.get("workspaceInfo", {}).get("localExecRoot")
elif "convenienceSymlinksIdentified" in eid:
for link in ev.get("convenienceSymlinksIdentified", {}).get("convenienceSymlinks", []):
if link.get("path", "").endswith("testlogs") and link.get("target"):
testlogs_rel = link["target"]
elif "targetConfigured" in eid:
if ev.get("configured", {}).get("targetKind") == "go_test rule":
go_tests.add(eid["targetConfigured"]["label"])
elif "testResult" in eid:
tr = eid["testResult"]
shards[tr["label"]] = max(shards.get(tr["label"], 0), int(tr.get("shard", 1) or 1))
tested = {l: n for l, n in shards.items() if l in go_tests}
testlogs = None
if exec_root and testlogs_rel:
# The symlink target is relative to the output base, two levels above
# the exec root (<output_base>/execroot/<workspace>).
testlogs = os.path.join(os.path.dirname(os.path.dirname(exec_root)), testlogs_rel)
return tested, go_tests, testlogs
def testlog_xmls(testlogs, label, shard_count):
pkg = label_pkg(label)
name = label.split(":", 1)[1]
base = os.path.join(testlogs, pkg, name)
if shard_count <= 1:
return [os.path.join(base, "test.xml")]
return [os.path.join(base, f"shard_{i}_of_{shard_count}", "test.xml") for i in range(1, shard_count + 1)]
def bazel_observed(tested, testlogs):
"""Return ({pkg: {test: status}}, [problems])."""
observed, problems = {}, []
for label, n in sorted(tested.items()):
pkg = label_pkg(label)
per = observed.setdefault(pkg, {})
for xml_path in testlog_xmls(testlogs, label, n):
if not os.path.exists(xml_path):
problems.append(f"{label}: {xml_path} missing (was test.xml downloaded? run with --config=ci)")
continue
try:
tree = ET.parse(xml_path)
except ET.ParseError as e:
problems.append(f"{label}: cannot parse {xml_path}: {e}")
continue
cases = tree.getroot().iter("testcase")
count = 0
for tc in cases:
name = tc.get("name", "")
if "/" in name or not name.startswith("Test"):
continue
count += 1
if tc.find("failure") is not None or tc.find("error") is not None:
status = "failed"
elif tc.find("skipped") is not None:
status = "skipped"
else:
status = "passed"
# A test seen in several targets keeps its worst status.
if name not in per or STATUS_RANK[status] > STATUS_RANK[per[name]]:
per[name] = status
if count == 0 and os.path.getsize(xml_path) < 64:
# An empty <testsuites/> means the binary ran no tests at all
# (or GO_TEST_WRAP_TESTV was off). The test-name diff reports
# which tests are missing; note it here for the log.
problems.append(f"{label}: {os.path.relpath(xml_path, testlogs)} lists no testcases")
return observed, problems
def load_allowlist(path):
entries, errors = [], []
if not path or not os.path.exists(path):
return entries, errors
with open(path, encoding="utf-8") as f:
for n, line in enumerate(f, 1):
body, _, why = line.partition("#")
body = body.strip()
if not body:
continue
fields = body.split()
kind = fields[2] if len(fields) == 3 else "run"
if len(fields) not in (2, 3) or kind not in ALLOW_KINDS or not why.strip():
errors.append(
f"{path}:{n}: want '<package dir> <test glob> [skip] # justification', got {line.strip()!r}"
)
continue
entries.append({"pkg": fields[0], "test": fields[1], "kind": kind, "why": why.strip(), "line": n, "used": False})
return entries, errors
def allowed(entries, pkg, test, kind="run"):
for e in entries:
if e["kind"] == kind and fnmatch.fnmatchcase(pkg, e["pkg"]) and fnmatch.fnmatchcase(test, e["test"]):
e["used"] = True
return True
return False
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--bep", required=True, help="--build_event_json_file of the bazel test run")
ap.add_argument("--root", default=os.getcwd(), help="workspace root (default: cwd)")
ap.add_argument("--testlogs", default=None, help="default: from the BEP, else <root>/bazel-testlogs")
ap.add_argument("--allowlist", default=DEFAULT_ALLOWLIST)
ap.add_argument("--go-list-json", default=None, help="precomputed `go list -json` output (tests)")
ap.add_argument(
"--go-test-json", default=None,
help="`go test -json` output of the PR Core command on the same commit; enables the skip-parity check",
)
ap.add_argument("--bazel", default=os.environ.get("BAZEL", "bazel"))
ap.add_argument("--no-query", action="store_true", help="skip the bazel query package-set check")
ap.add_argument("--details", action="store_true", help="list every divergence (nightly)")
ap.add_argument("--summary", default=None, help="append Markdown here (default: $GITHUB_STEP_SUMMARY)")
args = ap.parse_args(argv)
root = os.path.abspath(args.root)
entries, allow_errors = load_allowlist(args.allowlist)
expected, dirs = go_expected(root, args.go_list_json)
tested, configured, bep_testlogs = read_bep(args.bep)
# Prefer the BEP's own testlogs dir: the bazel-testlogs symlink follows
# whatever bazel command ran last, possibly in another output base.
testlogs = args.testlogs or bep_testlogs or os.path.join(root, "bazel-testlogs")
observed, problems = bazel_observed(tested, testlogs)
target_pkgs = None if args.no_query else query_go_test_pkgs(root, args.bazel)
no_target, missing, extra, allowlisted = [], [], [], []
if target_pkgs is not None:
for pkg in sorted(set(expected) - target_pkgs):
(allowlisted if allowed(entries, pkg, "*") else no_target).append((pkg, "*"))
for pkg, names in sorted(expected.items()):
seen = observed.get(pkg, {})
for t in sorted(names - set(seen)):
(allowlisted if allowed(entries, pkg, t) else missing).append((pkg, t))
for pkg, seen in sorted(observed.items()):
for t in sorted(set(seen) - expected.get(pkg, set())):
if SKIP_RE.search(t):
continue
(allowlisted if allowed(entries, pkg, t) else extra).append((pkg, t))
# Skip parity: passed under go test, skipped under Bazel.
skip_div, skip_allowed, parity_notes = [], [], []
go_status = go_test_statuses(args.go_test_json, dirs) if args.go_test_json else None
if go_status is not None:
for pkg, seen in sorted(observed.items()):
gs = go_status.get(pkg, {})
for t, bs in sorted(seen.items()):
g = gs.get(t)
if g is None or g == bs:
continue
if g == "passed" and bs == "skipped":
(skip_allowed if allowed(entries, pkg, t, "skip") else skip_div).append((pkg, t))
else:
parity_notes.append((pkg, f"{t} (go test {g}, bazel {bs})"))
allowlisted += skip_allowed
# Dedupe allowlisted pairs (a package-level entry also covers its tests).
allowlisted = sorted(set(allowlisted))
# Skip entries can only be exercised when go test statuses are known.
unused = [e for e in entries if not e["used"] and (e["kind"] == "run" or go_status is not None)]
statuses = {"passed": 0, "skipped": 0, "failed": 0}
for seen in observed.values():
for s in seen.values():
statuses[s] += 1
n_expected = sum(len(v) for v in expected.values())
ok = not (no_target or missing or extra or skip_div or allow_errors)
limit = None if args.details else 40
def listing(title, pairs):
out = [f"{title} ({len(pairs)}):"]
shown = pairs if limit is None else pairs[:limit]
out += [f" {p or '.'} {t}" for p, t in shown]
if limit is not None and len(pairs) > limit:
out.append(f" ... {len(pairs) - limit} more (run with --details)")
return out
lines = [
f"go test: {len(expected)} packages with tests, {n_expected} top-level tests (excluding ^TestEmbedded)",
f"bazel: {len(configured)} go_test targets configured, {len(tested)} tested; "
f"{sum(len(v) for v in observed.values())} top-level tests seen "
f"({statuses['passed']} passed, {statuses['skipped']} skipped, {statuses['failed']} failed)",
]
if target_pkgs is not None:
lines.append(f"query: {len(target_pkgs)} packages with a go_test target")
if go_status is None:
lines.append("skips: not compared (no --go-test-json; the nightly run checks skip parity)")
else:
n_go = sum(len(v) for v in go_status.values())
lines.append(
f"skips: compared with {n_go} go test results: {len(skip_div) + len(skip_allowed)} passed under "
f"go test but skipped under Bazel ({len(skip_allowed)} allowlisted), {len(parity_notes)} other differences"
)
lines.append(f"allowlisted divergences: {len(allowlisted)}")
for title, pairs in (
("packages with Go tests but no go_test target", no_target),
("tests go test runs that Bazel did not", missing),
("tests Bazel ran that go test would not", extra),
("tests go test passed that Bazel skipped", skip_div),
):
if pairs:
lines += listing(title, pairs)
if parity_notes:
lines += listing("other status differences (reported, not failed)", parity_notes)
if args.details and allowlisted:
lines += listing("allowlisted", allowlisted)
for e in unused:
lines.append(f"note: allowlist line {e['line']} ({e['pkg']} {e['test']} {e['kind']}) matched nothing; remove it")
for p in problems:
lines.append(f"note: {p}")
lines += allow_errors
verdict = "EQUIVALENCE: ok" if ok else "EQUIVALENCE: FAIL (unexplained divergence; fix the BUILD tags/targets or add a justified allowlist entry)"
lines.append(verdict)
print("\n".join(lines))
summary = args.summary or os.environ.get("GITHUB_STEP_SUMMARY")
if summary:
head = (
f"**Equivalence**: {'ok' if ok else 'FAIL'}; go test {n_expected} tests in {len(expected)} packages, "
f"Bazel saw {sum(len(v) for v in observed.values())} "
f"(missing {len(missing)}, extra {len(extra)}, packages without target {len(no_target)}, "
f"skipped only under Bazel {len(skip_div) if go_status is not None else 'not compared'}, "
f"allowlisted {len(allowlisted)})\n\n"
)
with open(summary, "a", encoding="utf-8") as f:
f.write(head)
f.write("<details><summary>Equivalence detail</summary>\n\n```\n" + "\n".join(lines) + "\n```\n</details>\n\n")
return 0 if ok else 1
if __name__ == "__main__":
sys.exit(main())

View file

@ -0,0 +1,31 @@
# Expected divergences between the Go tests `go test` runs (pr.yml) and the
# ones `bazel test //... --config=ci` runs, for tools/bazel/equivalence.py.
#
# One entry per line, each with a justification:
#
# <package dir> <TestName or glob> # why Bazel does not run it
# <package dir> <TestName or glob> skip # why Bazel skips a test go test runs
#
# `*` as the test name covers the whole package, including a package with no
# go_test target. `skip` entries are checked only when go test statuses are
# supplied (--go-test-json, the nightly run). Keep this list short: every entry
# is a test the Bazel lane does not exercise. The checker reports entries that
# no longer match anything.
# The in-test `go build` of bd that this test guards does not happen under
# Bazel (bd comes in through BEADS_TEST_BD_BINARY); go test still runs it.
cmd/bd TestGOMODCACHENotUnderTestHome skip # no in-test go build under Bazel; guarded by go test
# Needs TestAAAStdioBaseline in the same process; sharding splits them.
# TestMain's checkStdioAfterRun still checks every shard.
cmd/bd TestZZStdioNotLeaked skip # baseline test lands in another shard; checkStdioAfterRun covers each shard
# Repository-policy tests that walk the whole source checkout (git ls-files,
# every BUILD file, .beads/ formulas). Bazel's runfiles hold only declared
# data, so they skip under TEST_SRCDIR; PR Core's go test runs them.
scripts TestBazelGoSrcsBlocksCurrent skip # walks the source checkout; runs under go test
scripts TestBazelNoPackagesUnderDocsTrees skip # walks the source checkout; runs under go test
scripts TestBazelNoRemoteEndpointsInTrackedFiles skip # scans git-tracked files; runs under go test
scripts TestBazelTestTagsFollowTaxonomy skip # reads every BUILD file in the checkout; runs under go test
scripts TestBazelTreeGoSrcsListsEveryPackage skip # walks the source checkout; runs under go test
scripts TestCheckTestingShortPassesOnCleanRepoTree skip # walks the whole checkout; runs under go test
scripts TestReleaseFormula* skip # .beads/ is in .bazelignore, so the formula cannot be data; runs under go test