ci(bazel): advisory Bazel workflow + go test/Bazel equivalence check (#6881)
* ci(bazel): add advisory Bazel workflow and setup-bazel action Add .github/workflows/bazel.yml, a side-by-side lane that runs `bazel test //... --config=ci` (PR Core parity) on pull_request, merge_group, push to main and workflow_dispatch. It is advisory: not in ci-gate's CI_GATE_REQUIRED. .github/actions/setup-bazel installs a sha256-pinned Bazelisk (Bazel from .bazelversion), restores the runner cache (--disk_cache and --repository_cache only), and writes a CI bazelrc outside the workspace and outside the cached directory. Remote execution is enabled only when BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY are all set (base64 PEM cert/key; optional RBE_TLS_CA, RBE_INSTANCE); none set (fork PRs, or workflow_dispatch rbe=off) runs locally; a partial set fails. The endpoint and its bare host are masked. The rc logic lives in write-bazelrc.sh so it is testable offline. The runner cache is saved only by push-to-main runs under a run-unique key and restored by prefix, so it never freezes at its first save and PRs do not churn the quota. The job also reports the tier (T1/T2/local) and wall time in the step summary, runs tools/bazel/critpath.py against the profile, and checks BUILD drift with make bazel-sync-check plus make bazel-sync and a clean git status. tools/bazel/critpath.py reports the critical path as the sum of the chained "critical path component" events rather than the longest one (a cold race build: 395s over 75 components, where the single-component version printed 69s and a false scheduling gap), lists the longest members and Bazel's processes: cache line, and checks a report-only budget (warning annotation, exit 0) unless --enforce is given. scripts/ci_workflow_test.go covers the new lane: not wired into any gate, every action SHA-pinned to the repo's released SHAs, no monolithic actions/cache, no unbounded apt-get, the cache topology, secrets only reaching setup-bazel, --profile after `test`, and the rc writer's none/partial/full/invalid-PEM/secret-dir-in-workspace behaviour. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(bazel): check go test / Bazel test equivalence in the Bazel lane tools/bazel/equivalence.py compares the Go tests `bazel test //... --config=ci` ran with the ones pr.yml's go test lanes run, so a tag filter, a stray manual tag, a package without a go_test target, or a shard that runs zero tests cannot shrink the Bazel lane while it stays green: - package set: every package `go list -race -tags gms_pure_go` reports with test files has a go_test target (bazel query); - test-name set: every top-level TestXxx in the files go list selects (minus ^TestEmbedded, as PR Core skips) appears as a testcase in the test.xml of a go_test target this invocation tested (from the BEP); runtime skips count as run. Tests Bazel ran that go test would not are divergences too. Expected differences go in tools/bazel/equivalence_allowlist.txt, one justified line each (empty today); unused entries are reported. Results go to the step summary; the full listing with --details. .bazelrc test:ci now sets GO_TEST_WRAP_TESTV=1 (rules_go lists passing tests in test.xml only under -test.v) and downloads test.xml despite --remote_download_minimal. The Bazel workflow runs the check after the critical-path report, and nightly.yml calls the workflow (suite "bazel", and on schedule) with the full listing. Verified locally against a remote --config=ci run: 103 packages and 8843 tests on both sides. Tagging //internal/gitignore:gitignore_test manual keeps `bazel test` green but fails the check, listing its two tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(fs): run RepoContextAndRole from its own git repo The test relied on its working directory being inside a git checkout to make BEADS_DIR external to the CWD repo. Under Bazel the working directory is the runfiles tree, not a git repository, so RepoContext failed with "not a git repository" and the test skipped itself. Create a second temporary repo and chdir into it instead; go test keeps passing and Bazel now runs the assertions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(bazel): fail equivalence on tests Bazel skips but go test passes The equivalence docstring claimed runtime skips count as run because go test skips them the same way. It does not: against PR Core's go test -json on the same commit, 11 tests passed under go test and skipped under Bazel while the check printed ok with an empty allowlist. - equivalence.py --go-test-json compares per-test status with the Bazel test.xml: passed under go test and skipped under Bazel is a divergence; other status differences are listed, not failed. The in-job (PR) run cannot see skip divergences and says so. - The allowlist gains a `skip` kind (`<pkg> <test> skip # why`, the justification still mandatory) and lists the 10 intentional skips: cmd/bd TestGOMODCACHENotUnderTestHome and TestZZStdioNotLeaked, and the scripts policy tests that walk the source checkout. The eleventh, fs TestContextRepository_RepoContextAndRole, is fixed in the test. - scripts/ci/pr-core.sh writes `go test -json` to BEADS_PR_CORE_GO_TEST_JSON when set (same flags otherwise). - nightly.yml runs that in a new job, uploads the JSON, and calls bazel.yml with go-test-json-artifact, which downloads it and passes --go-test-json to the equivalence step. Verified on a remote `bazel test //... --config=ci` run: ok with the allowlist; a TEST_SRCDIR-only t.Skip injected into internal/gitignore TestAppendLines fails the check (and passes the name-only in-job check, as documented). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(bazel): freeze the runner cache per lock, mask key material, tighten policy Runner cache: every push to main saved ~2.3 GB (repository cache, disk cache, Bazelisk) under a run-unique key, on a repo already over its Actions cache quota. The cache now holds only the repository cache and Bazelisk, whose content is fixed by .bazelversion and MODULE.bazel.lock: exact key bazel-repo-v2-<os>-<hash>, restore falls back to the previous lock's entry, and only a push to main that missed the key saves. No --disk_cache in either mode (remote runs have the remote cache; local runs never used it). About 2 GB per lock change. Credentials: - write-bazelrc.sh masks every non-armor line of each decoded PEM (cert, key, CA). GitHub only masks the base64 value as stored. The CA used to be decoded inside the rc redirect, so its mask commands would have gone into the rc; it is decoded first now. - The CI rc sets build:remote-exec --noremote_upload_local_results, so CI never writes locally executed (PR-code) results into the shared action cache. Developer rcs keep Bazel's default. - RBE_INSTANCE stays optional; an empty one is a ::warning. - bazel.yml's header states the accepted risk: same-repo PR code runs with the RBE client certificate (as in gascity); forks never get it; what is in place and what is pending a human decision. Policy tests (each proved with a scratch mutation that now fails): triggers must be exactly pull_request, merge_group, push, workflow_dispatch and workflow_call (no pull_request_target); secrets may appear only in the setup-bazel step's env, not in workflow or job env, run, with or if; no continue-on-error anywhere in bazel.yml or setup-bazel; the cache-save if is matched exactly; the critical-path and equivalence steps must end with the bare script call and the test step with exit "$rc", with no exit 0, || true, || :, ; true, set +e or trap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(ci): record the same-repo RBE decision in bazel.yml Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(ci): bazel wrapper version check; keep go-test prose out of the build-tag scan The wrapper prepends --bazelrc, so 'bazel --version' reached the server as an unknown startup option; ask bazelisk directly. check-build-tags matched 'go test' in workflow prose and step names. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(ci): run the RBE lane on Blacksmith runners; mask the executor's IPs GitHub-hosted runners are refused by the remote-execution edge; the Blacksmith pool (as in gascity) is admitted and does not count against GitHub's concurrent-job cap. Forks and rbe=off keep ubuntu-latest. Bazel prints the resolved executor address in connection errors, so mask every address the endpoint host resolves to. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(bazel): target rbe-west (instance oss) behind RBE_WEST_WORKERS The original edge does not admit CI runners. Mirror gascity: remote execution uses secrets.RBE_WEST_EXECUTOR and instance oss when the repo variable RBE_WEST_WORKERS is true; same-repo runs are skipped until then instead of failing, while forks and rbe=off still run locally. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
ae1a493e01
commit
6da2a43477
11 changed files with 1678 additions and 2 deletions
6
.bazelrc
6
.bazelrc
|
|
@ -32,6 +32,12 @@ test:prcore --test_tag_filters=-requires-docker,-embedded,-manual
|
|||
|
||||
test:ci --config=prcore
|
||||
test:ci --keep_going --test_summary=terse
|
||||
# tools/bazel/equivalence.py compares the Go tests Bazel ran with the ones
|
||||
# `go test` would run, from each target's test.xml. rules_go only lists passing
|
||||
# tests in test.xml under -test.v, and --remote_download_minimal would leave
|
||||
# test.xml on the remote side, so ask for both.
|
||||
test:ci --test_env=GO_TEST_WRAP_TESTV=1
|
||||
test:ci --remote_download_regex=.*/test\.xml$
|
||||
|
||||
# Docker lane: only the requires-docker targets (pr.yml's container-backed
|
||||
# jobs), on a host with a docker daemon. Race like those jobs, but no -short
|
||||
|
|
|
|||
70
.github/actions/setup-bazel/action.yml
vendored
Normal file
70
.github/actions/setup-bazel/action.yml
vendored
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
name: setup-bazel
|
||||
description: >
|
||||
Install a sha256-pinned Bazelisk (Bazel itself comes from .bazelversion),
|
||||
restore the runner Bazel cache (--repository_cache and Bazelisk only),
|
||||
and write a CI bazelrc outside the workspace. When the caller passes all of
|
||||
BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY in the step env, that rc
|
||||
enables --config=remote-exec for every bazel command; with none of them the
|
||||
job runs locally (fork PRs); a partial set fails. `bazel` on PATH is a
|
||||
wrapper that always passes the generated rc.
|
||||
outputs:
|
||||
remote:
|
||||
description: '"true" when remote execution is configured'
|
||||
value: ${{ steps.rc.outputs.remote }}
|
||||
cache-restored:
|
||||
description: '"true" when a runner Bazel cache was restored (any prefix match)'
|
||||
value: ${{ steps.restore.outputs.cache-matched-key != '' }}
|
||||
cache-hit:
|
||||
description: '"true" when the cache for this .bazelversion and MODULE.bazel.lock was restored exactly'
|
||||
value: ${{ steps.restore.outputs.cache-hit }}
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
# Keyed on what determines its content; saved only by a push to main that
|
||||
# missed the exact key (see the caller's "Save Bazel runner cache" step).
|
||||
# A lock change restores the previous lock's entry, which still holds most
|
||||
# of the downloads.
|
||||
- name: Restore Bazel runner cache
|
||||
id: restore
|
||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: ${{ runner.temp }}/bazel-ci-cache
|
||||
key: bazel-repo-v2-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }}
|
||||
restore-keys: |
|
||||
bazel-repo-v2-${{ runner.os }}-
|
||||
|
||||
- name: Install Bazelisk
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
version=v1.29.0
|
||||
case "$(uname -m)" in
|
||||
x86_64) arch=amd64; sha=5a408715e932c0250d28bd84555f12edbf70117de42f9181691c736eacc4a992 ;;
|
||||
aarch64 | arm64) arch=arm64; sha=e20e8b0f4f240091b7a55bf17b9398bd4f40ee70ae0208dff95dd4c445fb4010 ;;
|
||||
*) echo "setup-bazel: unsupported arch $(uname -m)" >&2; exit 1 ;;
|
||||
esac
|
||||
curl -fsSL --retry 3 --max-time 120 -o "$RUNNER_TEMP/bazelisk" \
|
||||
"https://github.com/bazelbuild/bazelisk/releases/download/${version}/bazelisk-linux-${arch}"
|
||||
echo "${sha} $RUNNER_TEMP/bazelisk" | sha256sum -c -
|
||||
sudo install -m 755 "$RUNNER_TEMP/bazelisk" /usr/local/bin/bazelisk
|
||||
rm -f "$RUNNER_TEMP/bazelisk"
|
||||
echo "BAZELISK_HOME=$RUNNER_TEMP/bazel-ci-cache/bazelisk" >> "$GITHUB_ENV"
|
||||
|
||||
- name: Write CI bazelrc
|
||||
id: rc
|
||||
shell: bash
|
||||
env:
|
||||
BAZEL_CI_CACHE_DIR: ${{ runner.temp }}/bazel-ci-cache
|
||||
BAZEL_CI_SECRET_DIR: ${{ runner.temp }}/bazel-ci-secret
|
||||
run: bash "$GITHUB_ACTION_PATH/write-bazelrc.sh"
|
||||
|
||||
- name: Install bazel wrapper
|
||||
shell: bash
|
||||
env:
|
||||
RC: ${{ steps.rc.outputs.rc }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
printf '%s\n' '#!/usr/bin/env bash' "exec /usr/local/bin/bazelisk --bazelrc=\"$RC\" \"\$@\"" > "$RUNNER_TEMP/bazel-wrapper"
|
||||
sudo install -m 755 "$RUNNER_TEMP/bazel-wrapper" /usr/local/bin/bazel
|
||||
rm -f "$RUNNER_TEMP/bazel-wrapper"
|
||||
/usr/local/bin/bazelisk --version
|
||||
152
.github/actions/setup-bazel/write-bazelrc.sh
vendored
Normal file
152
.github/actions/setup-bazel/write-bazelrc.sh
vendored
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
#!/usr/bin/env bash
|
||||
# Write the CI bazelrc for .github/actions/setup-bazel.
|
||||
#
|
||||
# Always: --repository_cache under BAZEL_CI_CACHE_DIR (the directory the
|
||||
# runner cache saves and restores). Never --disk_cache: remote runs have the
|
||||
# remote cache, and a local run's disk cache (about 7 GB) would not fit next to
|
||||
# its output base on a hosted runner.
|
||||
#
|
||||
# Remote execution only when all three of BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT
|
||||
# and RBE_TLS_KEY are set (repo secrets; empty on fork PRs, so those runs stay
|
||||
# local). Exactly one or two of them is a misconfiguration and fails. Optional:
|
||||
# RBE_TLS_CA (base64 PEM CA bundle) and RBE_INSTANCE (instance name; a warning
|
||||
# when empty, because the default instance may share its action cache with
|
||||
# other repos on the same farm). Cert and key are base64-encoded PEM (raw PEM
|
||||
# is accepted too). Every line of the decoded PEM is masked in the job log.
|
||||
#
|
||||
# CI uploads only remotely executed results (--noremote_upload_local_results):
|
||||
# a job's locally executed actions (PR code included) never enter the shared
|
||||
# action cache from here. Developer rcs keep Bazel's default.
|
||||
#
|
||||
# The rc and the key material go to BAZEL_CI_SECRET_DIR, which must be outside
|
||||
# the workspace (tests that walk the checkout must never see an endpoint or a
|
||||
# cert path) and outside BAZEL_CI_CACHE_DIR (the runner cache must never store
|
||||
# credentials). The endpoint is masked in the job log.
|
||||
#
|
||||
# Outputs (to $GITHUB_OUTPUT when set, else stdout): rc=<path>, remote=true|false.
|
||||
set -euo pipefail
|
||||
|
||||
cache_dir="${BAZEL_CI_CACHE_DIR:?BAZEL_CI_CACHE_DIR is required}"
|
||||
secret_dir="${BAZEL_CI_SECRET_DIR:?BAZEL_CI_SECRET_DIR is required}"
|
||||
workspace="${GITHUB_WORKSPACE:-$PWD}"
|
||||
|
||||
abs() { mkdir -p "$1" && (cd "$1" && pwd -P); }
|
||||
cache_dir="$(abs "$cache_dir")"
|
||||
secret_dir="$(abs "$secret_dir")"
|
||||
workspace="$(cd "$workspace" && pwd -P)"
|
||||
case "$secret_dir/" in
|
||||
"$workspace"/*)
|
||||
echo "setup-bazel: BAZEL_CI_SECRET_DIR ($secret_dir) must be outside the workspace" >&2
|
||||
exit 1
|
||||
;;
|
||||
"$cache_dir"/*)
|
||||
echo "setup-bazel: BAZEL_CI_SECRET_DIR must not be inside the cached BAZEL_CI_CACHE_DIR" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
chmod 700 "$secret_dir"
|
||||
mkdir -p "$cache_dir/repo"
|
||||
|
||||
mask() {
|
||||
local v="${1:-}"
|
||||
[[ -z "$v" ]] && return 0
|
||||
printf '::add-mask::%s\n' "$v"
|
||||
}
|
||||
|
||||
# Mask the endpoint and its bare host[:port]: Bazel prints bytestream:// URIs
|
||||
# that carry the host without the scheme.
|
||||
executor="${BAZEL_REMOTE_EXECUTOR:-}"
|
||||
if [[ -n "$executor" ]]; then
|
||||
mask "$executor"
|
||||
hostport="${executor#*://}"
|
||||
hostport="${hostport%%/*}"
|
||||
mask "$hostport"
|
||||
mask "${hostport%:*}"
|
||||
# Bazel's connection errors print the resolved address ("host/1.2.3.4:443").
|
||||
if command -v getent >/dev/null 2>&1; then
|
||||
while read -r addr _; do
|
||||
mask "$addr"
|
||||
done < <(getent ahosts "${hostport%:*}" 2>/dev/null | sort -u -k1,1)
|
||||
fi
|
||||
fi
|
||||
|
||||
cert="${RBE_TLS_CERT:-}"
|
||||
key="${RBE_TLS_KEY:-}"
|
||||
ca="${RBE_TLS_CA:-}"
|
||||
instance="${RBE_INSTANCE:-}"
|
||||
|
||||
rc="$secret_dir/ci.bazelrc"
|
||||
umask 077
|
||||
{
|
||||
echo "# Generated by .github/actions/setup-bazel; not a workspace file."
|
||||
echo "common --repository_cache=$cache_dir/repo"
|
||||
} >"$rc"
|
||||
|
||||
# write_pem DEST VALUE: decode base64 (or accept raw PEM) and check it is PEM.
|
||||
write_pem() {
|
||||
local dest="$1" body="$2"
|
||||
body="$(printf '%s' "$body" | tr -d '\r')"
|
||||
if [[ "$body" != *"-----BEGIN "* ]]; then
|
||||
if ! body="$(printf '%s' "$body" | tr -d '\n ' | base64 -d 2>/dev/null)"; then
|
||||
echo "setup-bazel: $(basename "$dest") secret is neither PEM nor base64 PEM" >&2
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
if [[ "$body" != *"-----BEGIN "* ]]; then
|
||||
echo "setup-bazel: $(basename "$dest") secret does not decode to PEM" >&2
|
||||
return 1
|
||||
fi
|
||||
# GitHub masks the secret as stored (base64), not what it decodes to.
|
||||
local line
|
||||
while IFS= read -r line; do
|
||||
[[ -z "$line" || "$line" == -----* ]] || mask "$line"
|
||||
done <<<"$body"
|
||||
printf '%s\n' "$body" >"$dest"
|
||||
chmod 600 "$dest"
|
||||
}
|
||||
|
||||
set_fields=0
|
||||
for v in "$executor" "$cert" "$key"; do
|
||||
[[ -n "$v" ]] && set_fields=$((set_fields + 1))
|
||||
done
|
||||
|
||||
remote=false
|
||||
if [[ "$set_fields" -eq 3 ]]; then
|
||||
write_pem "$secret_dir/client.crt" "$cert"
|
||||
write_pem "$secret_dir/client.key" "$key"
|
||||
# Outside the rc block below: write_pem's mask commands go to the log.
|
||||
if [[ -n "$ca" ]]; then
|
||||
write_pem "$secret_dir/ca.crt" "$ca"
|
||||
fi
|
||||
{
|
||||
echo "build:remote-exec --remote_executor=$executor"
|
||||
echo "build:remote-exec --tls_client_certificate=$secret_dir/client.crt"
|
||||
echo "build:remote-exec --tls_client_key=$secret_dir/client.key"
|
||||
if [[ -n "$ca" ]]; then
|
||||
echo "build:remote-exec --tls_certificate=$secret_dir/ca.crt"
|
||||
fi
|
||||
if [[ -n "$instance" ]]; then
|
||||
echo "build:remote-exec --remote_instance_name=$instance"
|
||||
fi
|
||||
echo "build:remote-exec --noremote_upload_local_results"
|
||||
# Every command in this job (test, run //:gazelle, query) uses the
|
||||
# executor; the committed .bazelrc holds the generic remote-exec tuning.
|
||||
echo "build --config=remote-exec"
|
||||
} >>"$rc"
|
||||
if [[ -z "$instance" ]]; then
|
||||
echo "::warning title=RBE_INSTANCE not set::remote execution uses the executor's default instance, whose action cache other repos on the farm may share; set RBE_INSTANCE to a beads-only instance"
|
||||
fi
|
||||
remote=true
|
||||
echo "setup-bazel: remote execution enabled"
|
||||
elif [[ "$set_fields" -ne 0 ]]; then
|
||||
echo "setup-bazel: remote execution is partially configured; BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY must be set together (or none, for local execution)" >&2
|
||||
exit 1
|
||||
else
|
||||
echo "setup-bazel: no remote execution secrets; running locally"
|
||||
fi
|
||||
|
||||
out="${GITHUB_OUTPUT:-/dev/stdout}"
|
||||
{
|
||||
echo "rc=$rc"
|
||||
echo "remote=$remote"
|
||||
} >>"$out"
|
||||
250
.github/workflows/bazel.yml
vendored
Normal file
250
.github/workflows/bazel.yml
vendored
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# Bazel side-by-side lane (advisory). Runs `bazel test //... --config=ci`, the
|
||||
# PR Core-parity selection (race, -short, dolt skip, -requires-docker), on the
|
||||
# same commit as pr.yml's go test lanes. It is NOT in ci-gate's
|
||||
# CI_GATE_REQUIRED; promotion needs the equivalence record described in
|
||||
# tools/bazel/equivalence.py and a human decision.
|
||||
#
|
||||
# Execution mode, passed to .github/actions/setup-bazel:
|
||||
# repo variable RBE_WEST_WORKERS=true plus secrets RBE_WEST_EXECUTOR (grpcs://
|
||||
# endpoint), RBE_TLS_CERT, RBE_TLS_KEY (base64; RBE_TLS_CA optional):
|
||||
# remote execution on rbe-west, instance "oss" (shared with gascity).
|
||||
# fork PRs and rbe=off dispatches: local execution, slower but the same
|
||||
# results. Same-repo runs are skipped while RBE_WEST_WORKERS is unset.
|
||||
# workflow_dispatch with rbe=off simulates a fork run.
|
||||
#
|
||||
# Accepted risk (same model as gascity's bazel-test.yml): on push, merge_group
|
||||
# and same-repo pull_request runs, code from the PR head runs with the RBE
|
||||
# client certificate. setup-bazel is itself PR code, the key file is readable
|
||||
# by every later step and by locally executed Bazel actions, and a .bazelrc
|
||||
# edit can point the BEP or the executor anywhere. Anyone who can push a
|
||||
# branch here can therefore use or copy the certificate and write to the
|
||||
# farm's cache. Fork PRs never get it: pull_request from a fork receives no
|
||||
# secrets, this workflow has no pull_request_target trigger, and forks cannot
|
||||
# write the runner cache. In place: secrets are referenced by the setup step
|
||||
# alone and never by a workflow- or job-level env (policy-tested), decoded key
|
||||
# material is masked in logs, and CI uploads only remotely executed results
|
||||
# (--noremote_upload_local_results in the generated rc), and the certificate
|
||||
# is issued to beads CI alone, so it can be revoked without affecting other
|
||||
# clients. Maintainers decided (2026-09-28) that same-repo PRs keep remote
|
||||
# execution under this model. Still open: a beads-only RBE_INSTANCE and
|
||||
# farm-side denial of client action-cache writes. The lane is advisory and
|
||||
# nothing downstream consumes artifacts built here.
|
||||
#
|
||||
# Runner cache: one entry per (.bazelversion, MODULE.bazel.lock) pair, about
|
||||
# 2 GB (Bazel's --repository_cache plus Bazelisk's Bazel download), written
|
||||
# only by a push to main that missed it. No --disk_cache: remote runs use the
|
||||
# remote cache, and local runs would need about 7 GB more.
|
||||
#
|
||||
# Workflow-file changes suppress pull_request runs under the org approval
|
||||
# policy; validate edits to this file with workflow_dispatch on the branch.
|
||||
name: Bazel
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [ main, 'release/**' ]
|
||||
merge_group:
|
||||
push:
|
||||
branches: [ main ]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
rbe:
|
||||
description: "Remote execution: on (use secrets) or off (fork simulation)"
|
||||
required: true
|
||||
type: choice
|
||||
default: "on"
|
||||
options: [ "on", "off" ]
|
||||
details:
|
||||
description: "List every equivalence divergence"
|
||||
required: false
|
||||
type: boolean
|
||||
default: false
|
||||
workflow_call:
|
||||
inputs:
|
||||
rbe:
|
||||
type: string
|
||||
required: false
|
||||
default: "on"
|
||||
details:
|
||||
type: boolean
|
||||
required: false
|
||||
default: true
|
||||
go-test-json-artifact:
|
||||
description: >
|
||||
Artifact (from the calling run) holding go-test.json, PR Core's
|
||||
`go-test -json` on this commit; enables the equivalence skip-parity
|
||||
check. nightly.yml sets it.
|
||||
type: string
|
||||
required: false
|
||||
default: ""
|
||||
|
||||
# Main runs always finish (they may save the runner cache); PR runs supersede.
|
||||
concurrency:
|
||||
group: bazel-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
env:
|
||||
BD_DISABLE_METRICS: "1"
|
||||
BD_DISABLE_EVENT_FLUSH: "1"
|
||||
# Phase B report-only budget for T1/T2 (tools/bazel/critpath.py).
|
||||
BAZEL_BUDGET_SECONDS: "420"
|
||||
|
||||
jobs:
|
||||
bazel-test:
|
||||
name: Bazel / test (advisory)
|
||||
# Remote-execution runs use the org's Blacksmith pool (the farm's edge
|
||||
# admits it; GitHub-hosted runners are refused) and do not count against
|
||||
# GitHub's concurrent-job cap. Fork PRs and rbe=off have no remote
|
||||
# executor and build everything locally, so they get the larger
|
||||
# GitHub-hosted runner.
|
||||
runs-on: ${{ (inputs.rbe || 'on') != 'off' && github.event.pull_request.head.repo.fork != true && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
||||
# Remote execution goes to rbe-west (instance "oss", the Blacksmith-backed
|
||||
# worker pool shared with gascity), switched on by the repo variable
|
||||
# RBE_WEST_WORKERS. Until it is set, only fork PRs and rbe=off dispatches
|
||||
# run (locally); same-repo runs are skipped rather than failing on an
|
||||
# unreachable executor.
|
||||
if: ${{ vars.RBE_WEST_WORKERS == 'true' || inputs.rbe == 'off' || github.event.pull_request.head.repo.fork == true }}
|
||||
timeout-minutes: 100
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
|
||||
|
||||
# Go is for tools/bazel/equivalence.py's `go list` and for make; Bazel
|
||||
# uses its own hermetic SDK.
|
||||
- name: Set up Go
|
||||
id: setup-go
|
||||
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
|
||||
with:
|
||||
go-version-file: 'go.mod'
|
||||
cache: false
|
||||
|
||||
# Read-only: main.yml's build-artifacts job owns this cache.
|
||||
- name: Restore Go module cache
|
||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: ~/go/pkg/mod
|
||||
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
|
||||
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
|
||||
|
||||
- name: Set up Bazel
|
||||
id: bazel
|
||||
uses: ./.github/actions/setup-bazel
|
||||
env:
|
||||
BAZEL_REMOTE_EXECUTOR: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_WEST_EXECUTOR || '' }}
|
||||
RBE_TLS_CERT: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_TLS_CERT || '' }}
|
||||
RBE_TLS_KEY: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_TLS_KEY || '' }}
|
||||
RBE_TLS_CA: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && secrets.RBE_TLS_CA || '' }}
|
||||
RBE_INSTANCE: ${{ (inputs.rbe || 'on') != 'off' && vars.RBE_WEST_WORKERS == 'true' && 'oss' || '' }}
|
||||
|
||||
# T1: remote + restored repository cache; T2: remote, cold runner; local:
|
||||
# no remote execution (reported, never budgeted).
|
||||
- name: Measurement tier
|
||||
id: tier
|
||||
env:
|
||||
REMOTE: ${{ steps.bazel.outputs.remote }}
|
||||
RESTORED: ${{ steps.bazel.outputs.cache-restored }}
|
||||
run: |
|
||||
if [ "$REMOTE" = "true" ] && [ "$RESTORED" = "true" ]; then tier=T1
|
||||
elif [ "$REMOTE" = "true" ]; then tier=T2
|
||||
else tier=local; fi
|
||||
echo "tier=$tier" >> "$GITHUB_OUTPUT"
|
||||
{
|
||||
echo "### Bazel / test (advisory)"
|
||||
echo
|
||||
echo "Tier **$tier**: remote execution $REMOTE, runner cache restored $RESTORED."
|
||||
echo
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
# --profile and --build_event_json_file are command options, so they go
|
||||
# after `test` (a startup position fails).
|
||||
- name: bazel test //... --config=ci
|
||||
id: test
|
||||
timeout-minutes: 90
|
||||
run: |
|
||||
set -o pipefail
|
||||
start=$(date +%s)
|
||||
rc=0
|
||||
bazel test //... --config=ci \
|
||||
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
||||
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
||||
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
||||
echo "bazel test: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
||||
echo >> "$GITHUB_STEP_SUMMARY"
|
||||
exit "$rc"
|
||||
|
||||
- name: Critical-path report
|
||||
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
||||
env:
|
||||
TIER: ${{ steps.tier.outputs.tier }}
|
||||
run: |
|
||||
budget=()
|
||||
if [ "$TIER" != "local" ]; then budget=(--budget "$BAZEL_BUDGET_SECONDS"); fi
|
||||
python3 tools/bazel/critpath.py "$RUNNER_TEMP/bazel-profile.json" \
|
||||
--tier "$TIER" --log "$RUNNER_TEMP/bazel-test.log" "${budget[@]}"
|
||||
|
||||
# nightly.yml's PR Core `go test -json` on the same commit.
|
||||
- name: Download go-test results
|
||||
if: ${{ always() && inputs.go-test-json-artifact != '' && steps.test.outcome != 'skipped' }}
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
|
||||
with:
|
||||
name: ${{ inputs.go-test-json-artifact }}
|
||||
path: ${{ runner.temp }}/go-test-json
|
||||
|
||||
# Fails on any Go test that `go test` runs and Bazel did not (tag filter,
|
||||
# manual, missing target, empty shard) unless it is allowlisted. With
|
||||
# go test results (nightly) it also fails on tests that passed under go
|
||||
# test and were skipped under Bazel; a PR run cannot see those.
|
||||
- name: Go test equivalence
|
||||
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
||||
timeout-minutes: 15
|
||||
env:
|
||||
DETAILS: ${{ inputs.details && '--details' || '' }}
|
||||
GO_TEST_JSON: ${{ inputs.go-test-json-artifact != '' && format('{0}/go-test-json/go-test.json', runner.temp) || '' }}
|
||||
run: |
|
||||
args=()
|
||||
if [ -n "$DETAILS" ]; then args+=("$DETAILS"); fi
|
||||
if [ -n "$GO_TEST_JSON" ]; then args+=(--go-test-json "$GO_TEST_JSON"); fi
|
||||
python3 tools/bazel/equivalence.py --bep "$RUNNER_TEMP/bazel-bep.json" "${args[@]}"
|
||||
|
||||
# Runs last: it rewrites BUILD files in the checkout. main moves BUILD
|
||||
# inputs (imports, go.mod) without touching Bazel, so drift shows up here
|
||||
# first.
|
||||
- name: BUILD files in sync (gazelle, go_srcs, MODULE.bazel)
|
||||
if: ${{ always() && steps.bazel.outcome == 'success' }}
|
||||
timeout-minutes: 15
|
||||
run: |
|
||||
start=$(date +%s)
|
||||
make bazel-sync-check
|
||||
make bazel-sync
|
||||
drift="$(git status --porcelain --untracked-files=all)"
|
||||
echo "BUILD sync check: $(( $(date +%s) - start ))s" >> "$GITHUB_STEP_SUMMARY"
|
||||
if [ -n "$drift" ]; then
|
||||
echo "::error title=Bazel BUILD files out of sync::run 'make bazel-sync' and commit the result"
|
||||
echo "$drift"
|
||||
git diff | head -200
|
||||
echo "BUILD files **out of sync**; run \`make bazel-sync\` and commit the result." >> "$GITHUB_STEP_SUMMARY"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Upload test logs
|
||||
if: ${{ failure() && steps.test.outcome == 'failure' }}
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: bazel-testlogs
|
||||
path: |
|
||||
bazel-testlogs/**/test.log
|
||||
bazel-testlogs/**/test.xml
|
||||
if-no-files-found: ignore
|
||||
retention-days: 7
|
||||
|
||||
# Only a push to main that missed the exact key writes the runner cache.
|
||||
# Its content (--repository_cache and Bazelisk's download, never the rc
|
||||
# or credentials) is fixed by .bazelversion and MODULE.bazel.lock, so one
|
||||
# entry per lock is enough and later pushes do not churn the quota.
|
||||
- name: Save Bazel runner cache
|
||||
if: ${{ always() && github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.bazel.outcome == 'success' && steps.bazel.outputs.cache-hit != 'true' }}
|
||||
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: ${{ runner.temp }}/bazel-ci-cache
|
||||
key: bazel-repo-v2-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }}
|
||||
66
.github/workflows/nightly.yml
vendored
66
.github/workflows/nightly.yml
vendored
|
|
@ -26,6 +26,7 @@ on:
|
|||
- nix
|
||||
- mcp-package
|
||||
- npm-package
|
||||
- bazel
|
||||
smoke_version:
|
||||
description: "Optional previous release for cross-version-smoke"
|
||||
required: false
|
||||
|
|
@ -38,12 +39,75 @@ env:
|
|||
jobs:
|
||||
measure:
|
||||
name: Measure selected CI suite
|
||||
if: ${{ github.event_name == 'workflow_dispatch' && inputs.suite != 'full-test' }}
|
||||
if: ${{ github.event_name == 'workflow_dispatch' && inputs.suite != 'full-test' && inputs.suite != 'bazel' }}
|
||||
uses: ./.github/workflows/ci-measurements.yml
|
||||
with:
|
||||
suite: ${{ inputs.suite }}
|
||||
smoke_version: ${{ inputs.smoke_version }}
|
||||
|
||||
# PR Core's go test (scripts/ci/pr-core.sh) with -json output, so the Bazel
|
||||
# job below can compare per-test status: a test that passes here and skips
|
||||
# under Bazel fails the equivalence check unless allowlisted.
|
||||
bazel-go-test-json:
|
||||
name: Bazel equivalence (go-test side)
|
||||
if: ${{ github.event_name == 'schedule' || inputs.suite == 'bazel' }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
steps:
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
|
||||
|
||||
- name: Set up Go
|
||||
id: setup-go
|
||||
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
|
||||
with:
|
||||
go-version-file: 'go.mod'
|
||||
cache: false
|
||||
|
||||
- name: Restore Go module cache
|
||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: ~/go/pkg/mod
|
||||
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
|
||||
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
|
||||
|
||||
- name: Install Dolt
|
||||
run: ./scripts/ci/install-dolt.sh
|
||||
|
||||
- name: Configure Git and Dolt identity
|
||||
run: |
|
||||
git config --global user.name "CI Bot"
|
||||
git config --global user.email "ci@beads.test"
|
||||
dolt config --global --add user.name "CI Bot"
|
||||
dolt config --global --add user.email "ci@beads.test"
|
||||
|
||||
- name: Run PR core go-test (JSON)
|
||||
env:
|
||||
BEADS_PR_CORE_GO_TEST_JSON: ${{ runner.temp }}/go-test.json
|
||||
run: make ci-pr-core
|
||||
|
||||
# Uploaded even when tests fail: the status comparison still applies.
|
||||
- name: Upload go-test results
|
||||
if: ${{ always() }}
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
||||
with:
|
||||
name: bazel-equivalence-go-test-json
|
||||
path: ${{ runner.temp }}/go-test.json
|
||||
if-no-files-found: error
|
||||
retention-days: 7
|
||||
|
||||
# Bazel lane with the full go test equivalence listing and the skip-parity
|
||||
# check (advisory, like the PR run). Remote execution when the repo secrets
|
||||
# are present.
|
||||
bazel:
|
||||
name: Bazel equivalence (full detail)
|
||||
needs: bazel-go-test-json
|
||||
if: ${{ !cancelled() && (github.event_name == 'schedule' || inputs.suite == 'bazel') }}
|
||||
uses: ./.github/workflows/bazel.yml
|
||||
secrets: inherit
|
||||
with:
|
||||
details: true
|
||||
go-test-json-artifact: bazel-equivalence-go-test-json
|
||||
|
||||
full-test:
|
||||
name: Full Test Suite
|
||||
if: ${{ github.event_name == 'schedule' || inputs.suite == 'full-test' }}
|
||||
|
|
|
|||
|
|
@ -258,6 +258,15 @@ func TestContextRepository_RepoContextAndRole(t *testing.T) {
|
|||
require.NoError(t, os.MkdirAll(beadsDir, 0o750))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(beadsDir, "metadata.json"), []byte("{}"), 0o600))
|
||||
|
||||
// Run from a second, unrelated git repo so BEADS_DIR is external to the
|
||||
// CWD repo. Relying on the test's own CWD being inside a checkout breaks
|
||||
// under Bazel, whose runfiles tree is not a git repository.
|
||||
cwdRepo := t.TempDir()
|
||||
cmd = exec.Command("git", "init")
|
||||
cmd.Dir = cwdRepo
|
||||
require.NoError(t, cmd.Run())
|
||||
t.Chdir(cwdRepo)
|
||||
|
||||
t.Setenv("BEADS_DIR", beadsDir)
|
||||
beads.ResetCaches()
|
||||
git.ResetCaches()
|
||||
|
|
|
|||
|
|
@ -30,5 +30,15 @@ GO_TEST_PARALLEL="${GO_TEST_PARALLEL:-4}"
|
|||
# main.yml's integration package and cmd/bd shards, ci-measurements.yml,
|
||||
# nightly.yml. Splitting or de-slowing cmd/bd is the real fix and belongs on main
|
||||
# — this only stops the clock from being the thing that fails.
|
||||
# BEADS_PR_CORE_GO_TEST_JSON=<file>: same run, `go test -json` output to <file>
|
||||
# (nightly.yml feeds it to tools/bazel/equivalence.py --go-test-json).
|
||||
go_test() {
|
||||
if [[ -n "${BEADS_PR_CORE_GO_TEST_JSON:-}" ]]; then
|
||||
go test -json "$@" >"$BEADS_PR_CORE_GO_TEST_JSON"
|
||||
else
|
||||
go test "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
ci_time "pr-core go test" -- \
|
||||
go test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -race -short -timeout=30m -skip '^TestEmbedded' ./...
|
||||
go_test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -race -short -timeout=30m -skip '^TestEmbedded' ./...
|
||||
|
|
|
|||
|
|
@ -1,11 +1,14 @@
|
|||
package scripts_test
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
|
|
@ -1414,3 +1417,521 @@ func captureOne(t *testing.T, pattern, body, source string) string {
|
|||
}
|
||||
return matches[1]
|
||||
}
|
||||
|
||||
// --- Bazel lane (.github/workflows/bazel.yml, advisory) ----------------------
|
||||
|
||||
const (
|
||||
bazelWorkflowName = "bazel.yml"
|
||||
bazelJobName = "bazel-test"
|
||||
setupBazelActionDir = ".github/actions/setup-bazel"
|
||||
uploadArtifactSHA = "043fb46d1a93c77aae656e7c1c64a875d1fc6a0a"
|
||||
downloadArtifactSHA = "3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c"
|
||||
checkoutSHA = "3d3c42e5aac5ba805825da76410c181273ba90b1"
|
||||
bazelCacheKeyPrefix = "bazel-repo-v2-${{ runner.os }}-"
|
||||
bazelCacheKey = bazelCacheKeyPrefix + "${{ hashFiles('.bazelversion', 'MODULE.bazel.lock') }}"
|
||||
bazelCachePath = "${{ runner.temp }}/bazel-ci-cache"
|
||||
// Save only from a push to main that missed the exact key: the content is
|
||||
// fixed by the key, so re-saving every push only churns the quota.
|
||||
bazelCacheSaveIf = "${{ always() && github.event_name == 'push' && github.ref == 'refs/heads/main' && " +
|
||||
"steps.bazel.outcome == 'success' && steps.bazel.outputs.cache-hit != 'true' }}"
|
||||
)
|
||||
|
||||
// The only triggers bazel.yml may have. pull_request_target (and
|
||||
// workflow_run) would run with secrets in the context of fork PRs.
|
||||
var bazelWorkflowTriggers = []string{"merge_group", "pull_request", "push", "workflow_call", "workflow_dispatch"}
|
||||
|
||||
type ciCompositeAction struct {
|
||||
Runs struct {
|
||||
Using string `yaml:"using"`
|
||||
Steps []ciWorkflowStep `yaml:"steps"`
|
||||
} `yaml:"runs"`
|
||||
}
|
||||
|
||||
func readSetupBazelAction(t *testing.T) ciCompositeAction {
|
||||
t.Helper()
|
||||
path := filepath.Join(sourceRepoRoot(t), setupBazelActionDir, "action.yml")
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var action ciCompositeAction
|
||||
if err := yaml.Unmarshal(data, &action); err != nil {
|
||||
t.Fatalf("parse %s: %v", path, err)
|
||||
}
|
||||
if action.Runs.Using != "composite" {
|
||||
t.Fatalf("%s runs.using = %q, want composite", path, action.Runs.Using)
|
||||
}
|
||||
return action
|
||||
}
|
||||
|
||||
// The Bazel lane is advisory until the equivalence record and a human promote
|
||||
// it: no ci-gate may need it or list it in CI_GATE_REQUIRED.
|
||||
func TestBazelWorkflowIsAdvisory(t *testing.T) {
|
||||
workflow := readCIWorkflow(t, bazelWorkflowName)
|
||||
if len(workflow.Jobs) != 1 {
|
||||
t.Errorf("%s jobs = %d, want only %q", bazelWorkflowName, len(workflow.Jobs), bazelJobName)
|
||||
}
|
||||
job := workflow.job(t, bazelJobName)
|
||||
// The farm admits the Blacksmith pool only; forks (no secrets) and rbe=off
|
||||
// build locally on the GitHub-hosted runner.
|
||||
if want := "${{ (inputs.rbe || 'on') != 'off' && github.event.pull_request.head.repo.fork != true && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}"; job.RunsOn != want {
|
||||
t.Errorf("%s runs-on = %q, want %q", bazelJobName, job.RunsOn, want)
|
||||
}
|
||||
if job.ContinueOnError {
|
||||
t.Errorf("%s continue-on-error hides failures; the lane is advisory by staying out of ci-gate", bazelJobName)
|
||||
}
|
||||
for _, name := range []string{"pr.yml", "pr-risk.yml"} {
|
||||
for jobName, j := range readCIWorkflow(t, name).Jobs {
|
||||
for _, need := range j.Needs {
|
||||
if strings.Contains(strings.ToLower(need), "bazel") {
|
||||
t.Errorf("%s job %q needs %q; the Bazel lane is advisory", name, jobName, need)
|
||||
}
|
||||
}
|
||||
for _, step := range j.Steps {
|
||||
for key, value := range step.Env {
|
||||
if strings.Contains(strings.ToUpper(key), "BAZEL") ||
|
||||
(key == "CI_GATE_REQUIRED" && strings.Contains(strings.ToUpper(value), "BAZEL")) {
|
||||
t.Errorf("%s job %q step %q env %s=%q wires the Bazel lane into a gate", name, jobName, step.Name, key, value)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Every action in the Bazel lane is pinned to a full commit SHA, with the same
|
||||
// released SHAs the Go lanes use, and the monolithic actions/cache is banned
|
||||
// here too (restore everywhere, save only where the topology says).
|
||||
func TestBazelWorkflowActionsArePinned(t *testing.T) {
|
||||
steps := map[string][]ciWorkflowStep{
|
||||
bazelWorkflowName: readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName).Steps,
|
||||
setupBazelActionDir + "/action.yml": readSetupBazelAction(t).Runs.Steps,
|
||||
}
|
||||
want := map[string]string{
|
||||
"actions/checkout": checkoutSHA,
|
||||
setupGoActionFamily: setupGoSHA,
|
||||
cacheRestoreActionFamily: cacheSHA,
|
||||
cacheSaveActionFamily: cacheSHA,
|
||||
"actions/upload-artifact": uploadArtifactSHA,
|
||||
"actions/download-artifact": downloadArtifactSHA,
|
||||
}
|
||||
for file, list := range steps {
|
||||
for _, step := range list {
|
||||
if step.Uses == "" || strings.HasPrefix(step.Uses, "./") {
|
||||
continue
|
||||
}
|
||||
family, sha, found := strings.Cut(step.Uses, "@")
|
||||
if !found || !actionPin.MatchString(sha) {
|
||||
t.Errorf("%s step %q action %q is not pinned to a 40-hex commit SHA", file, step.Name, step.Uses)
|
||||
continue
|
||||
}
|
||||
if isForbiddenGoCacheActionFamily(family) {
|
||||
t.Errorf("%s step %q uses forbidden monolithic %q; use actions/cache/restore and /save", file, step.Name, family)
|
||||
continue
|
||||
}
|
||||
wantSHA, known := want[family]
|
||||
if !known {
|
||||
t.Errorf("%s step %q uses %q, which this policy does not know; add its released SHA here", file, step.Name, family)
|
||||
} else if sha != wantSHA {
|
||||
t.Errorf("%s step %q action %q has SHA %q, want %q", file, step.Name, family, sha, wantSHA)
|
||||
}
|
||||
if family == setupGoActionFamily && step.With["cache"] != "false" {
|
||||
t.Errorf("%s setup-go cache = %q, want false", file, step.With["cache"])
|
||||
}
|
||||
if family == cacheSaveActionFamily && file != bazelWorkflowName {
|
||||
t.Errorf("%s step %q saves a cache; only %s's push-to-main step may", file, step.Name, bazelWorkflowName)
|
||||
}
|
||||
}
|
||||
for _, step := range list {
|
||||
if strings.Contains(step.Run, "apt-get") {
|
||||
t.Errorf("%s step %q runs apt-get; bound it (see ci_workflow_apt_get_timeout_test.go) or drop it", file, step.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The runner cache holds only content fixed by .bazelversion and
|
||||
// MODULE.bazel.lock (repository cache, Bazelisk), so it is keyed on exactly
|
||||
// that and written only by a push to main that missed the key; it never holds
|
||||
// the generated rc or credentials.
|
||||
func TestBazelWorkflowCacheTopology(t *testing.T) {
|
||||
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName)
|
||||
var saves []ciWorkflowStep
|
||||
for _, step := range job.Steps {
|
||||
if actionFamily(step.Uses) == cacheSaveActionFamily {
|
||||
saves = append(saves, step)
|
||||
}
|
||||
}
|
||||
if len(saves) != 1 {
|
||||
t.Fatalf("%s has %d cache save steps, want 1", bazelWorkflowName, len(saves))
|
||||
}
|
||||
save := saves[0]
|
||||
if save.If != bazelCacheSaveIf {
|
||||
t.Errorf("cache save if = %q, want exactly %q", save.If, bazelCacheSaveIf)
|
||||
}
|
||||
if save.With["key"] != bazelCacheKey || save.With["path"] != bazelCachePath {
|
||||
t.Errorf("cache save key/path = %q / %q, want %q / %q", save.With["key"], save.With["path"], bazelCacheKey, bazelCachePath)
|
||||
}
|
||||
|
||||
var restore, writer ciWorkflowStep
|
||||
for _, step := range readSetupBazelAction(t).Runs.Steps {
|
||||
switch {
|
||||
case actionFamily(step.Uses) == cacheRestoreActionFamily:
|
||||
restore = step
|
||||
case step.Env["BAZEL_CI_SECRET_DIR"] != "":
|
||||
writer = step
|
||||
}
|
||||
}
|
||||
if restore.With["path"] != bazelCachePath || restore.With["key"] != bazelCacheKey ||
|
||||
strings.TrimSpace(restore.With["restore-keys"]) != bazelCacheKeyPrefix {
|
||||
t.Errorf("setup-bazel restore path/key/restore-keys = %q / %q / %q; want the save step's path and key with a %q prefix fallback",
|
||||
restore.With["path"], restore.With["key"], restore.With["restore-keys"], bazelCacheKeyPrefix)
|
||||
}
|
||||
if writer.Env["BAZEL_CI_CACHE_DIR"] != bazelCachePath {
|
||||
t.Errorf("rc writer BAZEL_CI_CACHE_DIR = %q, want %q", writer.Env["BAZEL_CI_CACHE_DIR"], bazelCachePath)
|
||||
}
|
||||
if secret := writer.Env["BAZEL_CI_SECRET_DIR"]; !strings.HasPrefix(secret, "${{ runner.temp }}/") || strings.HasPrefix(secret, bazelCachePath) {
|
||||
t.Errorf("rc writer BAZEL_CI_SECRET_DIR = %q; want a runner.temp directory outside the cached %q", secret, bazelCachePath)
|
||||
}
|
||||
}
|
||||
|
||||
// The lane runs the committed ci config over //..., reports the critical path
|
||||
// and the go test equivalence even when tests fail, catches BUILD drift, and
|
||||
// hands secrets to nothing but setup-bazel.
|
||||
func TestBazelWorkflowRunsCIConfigWithReports(t *testing.T) {
|
||||
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName)
|
||||
if job.TimeoutMinutes == 0 {
|
||||
t.Errorf("%s has no timeout-minutes", bazelJobName)
|
||||
}
|
||||
testStep := job.step(t, "bazel test //... --config=ci")
|
||||
for _, required := range []string{"bazel test //... --config=ci", "--profile=", "--build_event_json_file=", "set -o pipefail"} {
|
||||
if !strings.Contains(testStep.Run, required) {
|
||||
t.Errorf("test step does not contain %q:\n%s", required, testStep.Run)
|
||||
}
|
||||
}
|
||||
if strings.Contains(testStep.Run, "--config=remote-exec") {
|
||||
t.Errorf("test step selects remote-exec itself; setup-bazel's rc does that only when secrets are present")
|
||||
}
|
||||
if i, j := strings.Index(testStep.Run, "bazel test"), strings.Index(testStep.Run, "--profile="); j < i {
|
||||
t.Errorf("--profile is a command option and must follow `bazel test`")
|
||||
}
|
||||
for name, script := range map[string]string{
|
||||
"Critical-path report": "tools/bazel/critpath.py",
|
||||
"Go test equivalence": "tools/bazel/equivalence.py",
|
||||
} {
|
||||
step := job.step(t, name)
|
||||
if !strings.Contains(step.Run, script) || !strings.Contains(step.If, "always()") {
|
||||
t.Errorf("step %q must run %s under always(); run=%q if=%q", name, script, step.Run, step.If)
|
||||
}
|
||||
assertReportStepKeepsExitStatus(t, step, "python3 "+script)
|
||||
}
|
||||
assertTestStepKeepsExitStatus(t, testStep)
|
||||
sync := job.step(t, "BUILD files in sync (gazelle, go_srcs, MODULE.bazel)")
|
||||
for _, required := range []string{"make bazel-sync-check", "make bazel-sync\n", "git status --porcelain --untracked-files=all", "exit 1"} {
|
||||
if !strings.Contains(sync.Run, required) {
|
||||
t.Errorf("sync step does not contain %q:\n%s", required, sync.Run)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// swallowedExit matches shell that hides a failing command's status.
|
||||
var swallowedExit = regexp.MustCompile(`\bexit\b|\|\|\s*(true|:)|;\s*(true|:)\s*$|\bset\s+\+e\b|\btrap\b`)
|
||||
|
||||
// shellCommands joins backslash continuations and drops blank and comment
|
||||
// lines.
|
||||
func shellCommands(run string) []string {
|
||||
var cmds []string
|
||||
cur := ""
|
||||
for _, line := range strings.Split(run, "\n") {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
if cur == "" && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
|
||||
continue
|
||||
}
|
||||
if strings.HasSuffix(trimmed, "\\") {
|
||||
cur += strings.TrimSuffix(trimmed, "\\") + " "
|
||||
continue
|
||||
}
|
||||
cmds = append(cmds, strings.Join(strings.Fields(cur+trimmed), " "))
|
||||
cur = ""
|
||||
}
|
||||
if cur != "" {
|
||||
cmds = append(cmds, strings.Join(strings.Fields(cur), " "))
|
||||
}
|
||||
return cmds
|
||||
}
|
||||
|
||||
// A report step's status is its script's: the script is the last command,
|
||||
// nothing after it on that line, and nothing in the step can mask a failure.
|
||||
func assertReportStepKeepsExitStatus(t *testing.T, step ciWorkflowStep, invocation string) {
|
||||
t.Helper()
|
||||
if step.Shell != "" {
|
||||
t.Errorf("step %q overrides shell %q; the default bash -e is part of the contract", step.Name, step.Shell)
|
||||
}
|
||||
cmds := shellCommands(step.Run)
|
||||
if len(cmds) == 0 {
|
||||
t.Errorf("step %q runs nothing", step.Name)
|
||||
return
|
||||
}
|
||||
last := cmds[len(cmds)-1]
|
||||
if !strings.HasPrefix(last, invocation+" ") || strings.ContainsAny(last, ";&|") {
|
||||
t.Errorf("step %q must end with a bare %q so its exit status is the step's; last command %q", step.Name, invocation, last)
|
||||
}
|
||||
for _, cmd := range cmds {
|
||||
if swallowedExit.MatchString(cmd) {
|
||||
t.Errorf("step %q command %q can hide the script's exit status", step.Name, cmd)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The bazel test step captures bazel's status through tee and must exit
|
||||
// with it.
|
||||
func assertTestStepKeepsExitStatus(t *testing.T, step ciWorkflowStep) {
|
||||
t.Helper()
|
||||
if step.Shell != "" {
|
||||
t.Errorf("step %q overrides shell %q", step.Name, step.Shell)
|
||||
}
|
||||
cmds := shellCommands(step.Run)
|
||||
if len(cmds) == 0 || cmds[len(cmds)-1] != `exit "$rc"` {
|
||||
t.Errorf("step %q must end with exit \"$rc\"; commands %q", step.Name, cmds)
|
||||
}
|
||||
exits := 0
|
||||
for _, cmd := range cmds {
|
||||
if regexp.MustCompile(`(^|[;&|]\s*)exit\b`).MatchString(cmd) {
|
||||
exits++
|
||||
}
|
||||
if regexp.MustCompile(`\|\|\s*(true|:)|;\s*(true|:)\s*$|\bset\s+\+e\b|\btrap\b`).MatchString(cmd) {
|
||||
t.Errorf("step %q command %q can hide bazel's exit status", step.Name, cmd)
|
||||
}
|
||||
}
|
||||
if exits != 1 {
|
||||
t.Errorf("step %q has %d exit commands, want only the final exit \"$rc\"", step.Name, exits)
|
||||
}
|
||||
}
|
||||
|
||||
func readYAMLNode(t *testing.T, rel string) *yaml.Node {
|
||||
t.Helper()
|
||||
path := filepath.Join(sourceRepoRoot(t), rel)
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var doc yaml.Node
|
||||
if err := yaml.Unmarshal(data, &doc); err != nil {
|
||||
t.Fatalf("parse %s: %v", path, err)
|
||||
}
|
||||
if doc.Kind != yaml.DocumentNode || len(doc.Content) != 1 || doc.Content[0].Kind != yaml.MappingNode {
|
||||
t.Fatalf("%s: want a YAML mapping document", path)
|
||||
}
|
||||
return doc.Content[0]
|
||||
}
|
||||
|
||||
// walkYAML calls fn with the dotted path of every mapping key and scalar.
|
||||
func walkYAML(node *yaml.Node, path string, fn func(path string, key bool, value string)) {
|
||||
switch node.Kind {
|
||||
case yaml.MappingNode:
|
||||
for i := 0; i+1 < len(node.Content); i += 2 {
|
||||
k := node.Content[i].Value
|
||||
fn(path+"."+k, true, k)
|
||||
walkYAML(node.Content[i+1], path+"."+k, fn)
|
||||
}
|
||||
case yaml.SequenceNode:
|
||||
for i, item := range node.Content {
|
||||
walkYAML(item, fmt.Sprintf("%s[%d]", path, i), fn)
|
||||
}
|
||||
case yaml.ScalarNode:
|
||||
fn(path, false, node.Value)
|
||||
case yaml.AliasNode:
|
||||
if node.Alias != nil {
|
||||
walkYAML(node.Alias, path, fn)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func yamlMapKeys(node *yaml.Node, key string) []string {
|
||||
for i := 0; i+1 < len(node.Content); i += 2 {
|
||||
if node.Content[i].Value != key {
|
||||
continue
|
||||
}
|
||||
v := node.Content[i+1]
|
||||
switch v.Kind {
|
||||
case yaml.MappingNode:
|
||||
var keys []string
|
||||
for j := 0; j < len(v.Content); j += 2 {
|
||||
keys = append(keys, v.Content[j].Value)
|
||||
}
|
||||
return keys
|
||||
case yaml.SequenceNode:
|
||||
var keys []string
|
||||
for _, item := range v.Content {
|
||||
keys = append(keys, item.Value)
|
||||
}
|
||||
return keys
|
||||
default:
|
||||
return []string{v.Value}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// bazel.yml hands the RBE credentials to PR-head code on same-repo PRs (an
|
||||
// accepted, documented risk). Keep that exposure from growing: only the
|
||||
// listed triggers (never pull_request_target), secrets referenced from the
|
||||
// setup-bazel step's env and nowhere else (not workflow/job env, run, with or
|
||||
// if), and no continue-on-error anywhere, so a red lane stays red.
|
||||
func TestBazelWorkflowSecretsAndFailureSurface(t *testing.T) {
|
||||
root := readYAMLNode(t, filepath.Join(".github", "workflows", bazelWorkflowName))
|
||||
triggers := yamlMapKeys(root, "on")
|
||||
sort.Strings(triggers)
|
||||
if !reflect.DeepEqual(triggers, bazelWorkflowTriggers) {
|
||||
t.Errorf("%s triggers = %v, want exactly %v", bazelWorkflowName, triggers, bazelWorkflowTriggers)
|
||||
}
|
||||
|
||||
setupSteps := map[string]bool{}
|
||||
for i, step := range readCIWorkflow(t, bazelWorkflowName).job(t, bazelJobName).Steps {
|
||||
if step.Uses == "./"+setupBazelActionDir {
|
||||
setupSteps[fmt.Sprintf(".jobs.%s.steps[%d].env.", bazelJobName, i)] = true
|
||||
}
|
||||
}
|
||||
if len(setupSteps) != 1 {
|
||||
t.Errorf("%s has %d setup-bazel steps, want 1", bazelWorkflowName, len(setupSteps))
|
||||
}
|
||||
secretRef := regexp.MustCompile(`\bsecrets\s*(\.|\[)`)
|
||||
walkYAML(root, "", func(path string, key bool, value string) {
|
||||
if key && value == "continue-on-error" {
|
||||
t.Errorf("%s: %s hides failures; the lane is advisory by staying out of ci-gate", bazelWorkflowName, path)
|
||||
}
|
||||
if key && (value == "secrets" && strings.HasPrefix(path, ".jobs.")) {
|
||||
t.Errorf("%s: %s passes secrets to a called workflow or container", bazelWorkflowName, path)
|
||||
}
|
||||
if key || !secretRef.MatchString(value) {
|
||||
return
|
||||
}
|
||||
prefix := path[:strings.LastIndex(path, ".")+1]
|
||||
if !setupSteps[prefix] {
|
||||
t.Errorf("%s: %s reads secrets (%q); only the setup-bazel step's env may", bazelWorkflowName, path, value)
|
||||
}
|
||||
})
|
||||
|
||||
action := readYAMLNode(t, filepath.Join(setupBazelActionDir, "action.yml"))
|
||||
walkYAML(action, "", func(path string, key bool, value string) {
|
||||
if key && value == "continue-on-error" {
|
||||
t.Errorf("setup-bazel: %s hides failures", path)
|
||||
}
|
||||
if !key && secretRef.MatchString(value) {
|
||||
t.Errorf("setup-bazel: %s reads secrets directly; the caller passes them in the step env", path)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// write-bazelrc.sh: no secrets means a local-only rc (fork PRs), a partial set
|
||||
// fails, and the full set enables remote-exec with the key material and rc
|
||||
// outside the workspace and the runner cache.
|
||||
func TestSetupBazelRCWriter(t *testing.T) {
|
||||
bash := requireHostTool(t, "bash")
|
||||
script := filepath.Join(sourceRepoRoot(t), setupBazelActionDir, "write-bazelrc.sh")
|
||||
pem := func(kind string) string {
|
||||
body := "-----BEGIN " + kind + "-----\nMIIBfake" + strings.ReplaceAll(kind, " ", "") + "\nline2\n-----END " + kind + "-----\n"
|
||||
return base64.StdEncoding.EncodeToString([]byte(body))
|
||||
}
|
||||
run := func(t *testing.T, extra ...string) (string, string, string, error) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
workspace := filepath.Join(dir, "ws")
|
||||
if err := os.MkdirAll(workspace, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out := filepath.Join(dir, "github_output")
|
||||
secret := filepath.Join(dir, "secret")
|
||||
cmd := exec.Command(bash, script)
|
||||
cmd.Dir = workspace
|
||||
cmd.Env = append([]string{
|
||||
"PATH=" + os.Getenv("PATH"),
|
||||
"GITHUB_WORKSPACE=" + workspace,
|
||||
"GITHUB_OUTPUT=" + out,
|
||||
"BAZEL_CI_CACHE_DIR=" + filepath.Join(dir, "cache"),
|
||||
"BAZEL_CI_SECRET_DIR=" + secret,
|
||||
}, extra...)
|
||||
logs, err := cmd.CombinedOutput()
|
||||
outputs, _ := os.ReadFile(out)
|
||||
rc, _ := os.ReadFile(filepath.Join(secret, "ci.bazelrc"))
|
||||
return string(outputs), string(rc), string(logs), err
|
||||
}
|
||||
executor := "BAZEL_REMOTE_EXECUTOR=grpcs://" + "farm.invalid:443"
|
||||
|
||||
t.Run("no secrets stays local", func(t *testing.T) {
|
||||
outputs, rc, logs, err := run(t)
|
||||
if err != nil {
|
||||
t.Fatalf("err=%v\n%s", err, logs)
|
||||
}
|
||||
if !strings.Contains(outputs, "remote=false") || strings.Contains(rc, "remote") || strings.Contains(rc, "tls") {
|
||||
t.Errorf("outputs=%q rc=%q; want remote=false and no remote flags", outputs, rc)
|
||||
}
|
||||
if !strings.Contains(rc, "--repository_cache=") || strings.Contains(rc, "--disk_cache=") {
|
||||
t.Errorf("local rc = %q; want --repository_cache and no --disk_cache (local runs never save it)", rc)
|
||||
}
|
||||
})
|
||||
for name, env := range map[string][]string{
|
||||
"executor only": {executor},
|
||||
"cert and key": {"RBE_TLS_CERT=" + pem("CERTIFICATE"), "RBE_TLS_KEY=" + pem("PRIVATE KEY")},
|
||||
"executor no key": {executor, "RBE_TLS_CERT=" + pem("CERTIFICATE")},
|
||||
"cert not pem b64": {executor, "RBE_TLS_CERT=bm90IGEgcGVt", "RBE_TLS_KEY=" + pem("PRIVATE KEY")},
|
||||
} {
|
||||
t.Run("rejects "+name, func(t *testing.T) {
|
||||
if _, _, logs, err := run(t, env...); err == nil {
|
||||
t.Errorf("want failure, got success:\n%s", logs)
|
||||
}
|
||||
})
|
||||
}
|
||||
t.Run("full set enables remote-exec", func(t *testing.T) {
|
||||
outputs, rc, logs, err := run(t, executor, "RBE_TLS_CERT="+pem("CERTIFICATE"), "RBE_TLS_KEY="+pem("PRIVATE KEY"))
|
||||
if err != nil {
|
||||
t.Fatalf("err=%v\n%s", err, logs)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"build:remote-exec --remote_executor=grpcs://farm.invalid:443", "--tls_client_certificate=", "--tls_client_key=",
|
||||
"build:remote-exec --noremote_upload_local_results", "build --config=remote-exec",
|
||||
} {
|
||||
if !strings.Contains(rc, want) {
|
||||
t.Errorf("rc lacks %q:\n%s", want, rc)
|
||||
}
|
||||
}
|
||||
if strings.Contains(rc, "--disk_cache") || strings.Contains(rc, "--remote_instance_name") {
|
||||
t.Errorf("rc sets --disk_cache or an instance nobody asked for:\n%s", rc)
|
||||
}
|
||||
if !strings.Contains(outputs, "remote=true") {
|
||||
t.Errorf("outputs = %q, want remote=true", outputs)
|
||||
}
|
||||
for _, want := range []string{"::add-mask::farm.invalid\n", "::add-mask::MIIBfakeCERTIFICATE\n", "::add-mask::MIIBfakePRIVATEKEY\n", "::add-mask::line2\n", "::warning title=RBE_INSTANCE not set::"} {
|
||||
if !strings.Contains(logs, want) {
|
||||
t.Errorf("log lacks %q:\n%s", want, logs)
|
||||
}
|
||||
}
|
||||
if strings.Contains(logs, "::add-mask::-----") {
|
||||
t.Errorf("PEM armor lines are masked (they would blank every PEM header in the log):\n%s", logs)
|
||||
}
|
||||
})
|
||||
t.Run("CA and instance", func(t *testing.T) {
|
||||
_, rc, logs, err := run(t, executor, "RBE_TLS_CERT="+pem("CERTIFICATE"), "RBE_TLS_KEY="+pem("PRIVATE KEY"),
|
||||
"RBE_TLS_CA="+pem("CA CERT"), "RBE_INSTANCE=beads")
|
||||
if err != nil {
|
||||
t.Fatalf("err=%v\n%s", err, logs)
|
||||
}
|
||||
for _, want := range []string{"build:remote-exec --tls_certificate=", "build:remote-exec --remote_instance_name=beads"} {
|
||||
if !strings.Contains(rc, want) {
|
||||
t.Errorf("rc lacks %q:\n%s", want, rc)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(logs, "::add-mask::MIIBfakeCACERT\n") || strings.Contains(logs, "RBE_INSTANCE not set") {
|
||||
t.Errorf("want the CA masked and no instance warning:\n%s", logs)
|
||||
}
|
||||
})
|
||||
t.Run("secret dir inside workspace fails", func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cmd := exec.Command(bash, script)
|
||||
cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "GITHUB_WORKSPACE=" + dir,
|
||||
"BAZEL_CI_CACHE_DIR=" + filepath.Join(t.TempDir(), "c"), "BAZEL_CI_SECRET_DIR=" + filepath.Join(dir, "s")}
|
||||
if out, err := cmd.CombinedOutput(); err == nil {
|
||||
t.Errorf("want failure for a secret dir inside the workspace:\n%s", out)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
|
|||
175
tools/bazel/critpath.py
Normal file
175
tools/bazel/critpath.py
Normal file
|
|
@ -0,0 +1,175 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Critical-path report and CI budget check for a `bazel --profile` JSON.
|
||||
|
||||
gap = elapsed - critical_path
|
||||
|
||||
The critical path is the SUM of the chained "critical path component" events
|
||||
Bazel records, not the single longest one: a cold race build is a long chain
|
||||
of compile/link actions, and reporting only its longest member understates the
|
||||
path and turns the difference into a phantom "scheduling gap".
|
||||
|
||||
Decision rule:
|
||||
gap < 15% of elapsed the chain itself is the bottleneck. Attack its
|
||||
longest member (shard/speed up a test, split or slim
|
||||
a build action, warm the CAS if input upload
|
||||
dominates).
|
||||
gap >= 15% of elapsed actions waited for capacity: raise --jobs, grow the
|
||||
executor pool, restore the runner disk cache.
|
||||
|
||||
Measurement tiers (caching makes single numbers lie):
|
||||
T1 steady: warm remote cache + restored runner disk cache (the PR loop).
|
||||
T2 cold client: fresh runner disk cache, warm remote cache.
|
||||
T3 cold farm: after worker recycle / CAS eviction. Tracked, never gated.
|
||||
local no remote execution (fork PRs). Reported, never gated.
|
||||
|
||||
Usage:
|
||||
critpath.py PROFILE [--tier T2] [--budget SECONDS] [--enforce]
|
||||
[--log BAZEL_OUTPUT_LOG] [--top N] [--summary FILE]
|
||||
|
||||
Without --enforce an over-budget run prints a GitHub warning annotation and
|
||||
exits 0 (report-only phase); with --enforce it exits 1.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
CP_CAT = "critical path component"
|
||||
DECISION_GAP_PCT = 15.0
|
||||
|
||||
|
||||
def load_events(path):
|
||||
with open(path, encoding="utf-8") as f:
|
||||
prof = json.load(f)
|
||||
if isinstance(prof, list):
|
||||
return prof
|
||||
return prof.get("traceEvents", [])
|
||||
|
||||
|
||||
def short_name(name, width=90):
|
||||
# "action 'GoLink cmd/bd/bd_test_/bd_test'" -> "GoLink cmd/bd/bd_test_/bd_test"
|
||||
m = re.match(r"action '(.*)'$", name)
|
||||
if m:
|
||||
name = m.group(1)
|
||||
return name if len(name) <= width else name[: width - 1] + "…"
|
||||
|
||||
|
||||
def processes_line(log_path):
|
||||
"""Return Bazel's final "N processes: ..." line from its console log."""
|
||||
if not log_path or not os.path.exists(log_path):
|
||||
return None
|
||||
found = None
|
||||
with open(log_path, encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
m = re.search(r"(\d[\d,]* processes?: .*)$", line.strip())
|
||||
if m:
|
||||
found = m.group(1)
|
||||
return found
|
||||
|
||||
|
||||
def analyze(events):
|
||||
elapsed = max((e.get("dur", 0) for e in events if e.get("name") == "buildTargets"), default=0) / 1e6
|
||||
stamps = [e["ts"] for e in events if e.get("ts")]
|
||||
wall = (max(stamps) - min(stamps)) / 1e6 if stamps else 0.0
|
||||
chain = [e for e in events if e.get("cat") == CP_CAT]
|
||||
cp = sum(e.get("dur", 0) for e in chain) / 1e6
|
||||
longest = sorted(chain, key=lambda e: e.get("dur", 0), reverse=True)
|
||||
return {
|
||||
"elapsed": elapsed,
|
||||
"wall": wall,
|
||||
"cp": cp,
|
||||
"chain_len": len(chain),
|
||||
"longest": longest,
|
||||
}
|
||||
|
||||
|
||||
def next_step(res):
|
||||
elapsed, cp = res["elapsed"], res["cp"]
|
||||
gap = max(elapsed - cp, 0.0)
|
||||
pct = gap / elapsed * 100 if elapsed else 0.0
|
||||
if not res["longest"]:
|
||||
return pct, "NEXT: no critical path recorded (nothing executed, or a fully cached run)"
|
||||
if pct < DECISION_GAP_PCT:
|
||||
top = res["longest"][0]
|
||||
name = short_name(top["name"], 200)
|
||||
secs = top.get("dur", 0) / 1e6
|
||||
m = re.search(r"Testing (\S+)", name)
|
||||
if m:
|
||||
return pct, f"NEXT: shard or speed up {m.group(1)} ({secs:.0f}s)"
|
||||
if "upload" in name.lower():
|
||||
return pct, "NEXT: input upload dominates; warm the remote CAS or slim the declared inputs"
|
||||
return pct, f"NEXT: attack {short_name(name, 70)} ({secs:.0f}s)"
|
||||
return pct, "NEXT: scheduling gap; raise --jobs, grow the executor pool, or restore the runner disk cache"
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("profile")
|
||||
ap.add_argument("--tier", default="T2")
|
||||
ap.add_argument("--budget", type=float, default=None, help="seconds; compared with max(elapsed, wall)")
|
||||
ap.add_argument("--enforce", action="store_true", help="exit 1 when over budget")
|
||||
ap.add_argument("--log", help="bazel console log, for the 'processes:' cache line")
|
||||
ap.add_argument("--top", type=int, default=8, help="longest chain members to list")
|
||||
ap.add_argument("--summary", default=None, help="append Markdown here (default: $GITHUB_STEP_SUMMARY)")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if not os.path.exists(args.profile):
|
||||
print(f"critpath: profile {args.profile} not found (bazel did not start?)", file=sys.stderr)
|
||||
return 1 if args.enforce else 0
|
||||
|
||||
res = analyze(load_events(args.profile))
|
||||
pct, advice = next_step(res)
|
||||
gap = max(res["elapsed"] - res["cp"], 0.0)
|
||||
procs = processes_line(args.log)
|
||||
tier = args.tier
|
||||
|
||||
lines = [
|
||||
f"[{tier}] wall clock {res['wall']:7.1f}s (server start to last event)",
|
||||
f"[{tier}] elapsed {res['elapsed']:7.1f}s (build graph)",
|
||||
f"[{tier}] critical path {res['cp']:7.1f}s (sum of {res['chain_len']} chained components)",
|
||||
f"[{tier}] gap {gap:7.1f}s ({pct:.0f}% of elapsed)",
|
||||
]
|
||||
if procs:
|
||||
lines.append(f"[{tier}] processes {procs}")
|
||||
top = [e for e in res["longest"][: args.top] if e.get("dur", 0) > 0]
|
||||
if top:
|
||||
lines.append("")
|
||||
lines.append(f"longest critical-path members (top {len(top)}):")
|
||||
for e in top:
|
||||
lines.append(f" {e['dur'] / 1e6:7.1f}s {short_name(e['name'])}")
|
||||
lines.append("")
|
||||
lines.append(advice)
|
||||
|
||||
verdict = None
|
||||
over = False
|
||||
if args.budget is not None:
|
||||
gated = max(res["elapsed"], res["wall"])
|
||||
over = gated > args.budget
|
||||
mode = "enforcing" if args.enforce else "report-only"
|
||||
if over:
|
||||
verdict = f"BUDGET ({mode}): over, {gated:.1f}s > {args.budget:.0f}s ({tier})"
|
||||
else:
|
||||
verdict = f"BUDGET ({mode}): within, {gated:.1f}s <= {args.budget:.0f}s ({tier})"
|
||||
lines.append("")
|
||||
lines.append(verdict)
|
||||
|
||||
print("\n".join(lines))
|
||||
if over and not args.enforce and os.environ.get("GITHUB_ACTIONS") == "true":
|
||||
print(f"::warning title=Bazel critical-path budget::{verdict}")
|
||||
|
||||
summary = args.summary or os.environ.get("GITHUB_STEP_SUMMARY")
|
||||
if summary:
|
||||
with open(summary, "a", encoding="utf-8") as f:
|
||||
f.write(
|
||||
f"**Tier {tier}**: elapsed {res['elapsed']:.0f}s, wall {res['wall']:.0f}s, "
|
||||
f"critical path {res['cp']:.0f}s ({res['chain_len']} components), gap {pct:.0f}%"
|
||||
)
|
||||
f.write(f"; {verdict}\n\n" if verdict else "\n\n")
|
||||
f.write("<details><summary>Critical path</summary>\n\n```\n" + "\n".join(lines) + "\n```\n</details>\n\n")
|
||||
|
||||
return 1 if (over and args.enforce) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
388
tools/bazel/equivalence.py
Normal file
388
tools/bazel/equivalence.py
Normal file
|
|
@ -0,0 +1,388 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Check that `bazel test --config=ci` ran the same Go tests `go test` would.
|
||||
|
||||
The Bazel lane is only a safe replacement for the `go test` lanes in pr.yml
|
||||
(PR Core runs `go test -race -short -skip '^TestEmbedded' ./...`) if it runs
|
||||
the same tests. A tag filter, a stray `manual` tag, a package gazelle never
|
||||
picked up, or a sharding bug that makes a shard run zero tests all shrink the
|
||||
Bazel set silently: every target Bazel did run is still green. This script
|
||||
makes that shrinkage a failure.
|
||||
|
||||
Two checks always run, a third when `go test -json` output is supplied:
|
||||
|
||||
1. Package set. Every package `go list` reports with test files must have a
|
||||
go_test target (`bazel query 'kind("go_test rule", //...)'`).
|
||||
2. Test-name set. For every such package, each top-level `func TestXxx(t
|
||||
*testing.T)` in the files `go list` selects (the same -race and
|
||||
gms_pure_go build context as PR Core, minus the `^TestEmbedded` skip) must
|
||||
appear as a <testcase> in the test.xml of a go_test target that this Bazel
|
||||
invocation tested (read from --build_event_json_file). Bazel testcases
|
||||
that `go test` would not run are reported too. This check only sees names:
|
||||
a test that runs and passes under `go test` but calls t.Skip under Bazel
|
||||
(a missing env var, no git checkout, a file that is not declared as data)
|
||||
still counts as present here.
|
||||
3. Skip parity (--go-test-json, nightly). Given the `go test -json` output of
|
||||
the PR Core command on the same commit, every top-level test that passed
|
||||
under `go test` and was skipped under Bazel is a divergence. The Bazel lane
|
||||
alone cannot see this (a skip looks the same whether or not `go test` also
|
||||
skips), so the nightly workflow runs PR Core's `go test -json` and passes
|
||||
it in. Other status differences (pass/fail either way, skipped under go
|
||||
test but run under Bazel) are reported, not failed: a failing test already
|
||||
fails its own lane.
|
||||
|
||||
Divergences listed in the allowlist (tools/bazel/equivalence_allowlist.txt)
|
||||
are expected; anything else fails. The allowlist format is one entry per line:
|
||||
|
||||
<package dir> <TestName or glob> # why Bazel does not run it
|
||||
<package dir> <TestName or glob> skip # why Bazel skips a test go test runs
|
||||
|
||||
The first form (kind "run") covers the name checks; `*` as the test name
|
||||
matches the whole package (including the package-set check). The `skip` form
|
||||
covers only check 3. Every entry needs a justification after `#`.
|
||||
|
||||
Requires: go (with the module cache the packages need), bazel for the query
|
||||
unless --no-query, and the BEP JSON plus bazel-testlogs of a `--config=ci`
|
||||
run (which sets GO_TEST_WRAP_TESTV=1 and downloads test.xml).
|
||||
"""
|
||||
import argparse
|
||||
import fnmatch
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
GO_LIST_ARGS = ["list", "-e", "-race", "-tags", "gms_pure_go", "-json", "./..."]
|
||||
# pr.yml's go test lanes pass -skip '^TestEmbedded' (as does test:prcore).
|
||||
SKIP_RE = re.compile(r"^TestEmbedded")
|
||||
TEST_FUNC_RE = re.compile(r"^func\s+(Test\w*)\s*\(\s*\w+\s+\*testing\.T\s*\)", re.M)
|
||||
STATUS_RANK = {"skipped": 0, "passed": 1, "failed": 2}
|
||||
GO_TEST_STATUS = {"pass": "passed", "fail": "failed", "skip": "skipped"}
|
||||
ALLOW_KINDS = ("run", "skip")
|
||||
DEFAULT_ALLOWLIST = os.path.join(os.path.dirname(os.path.abspath(__file__)), "equivalence_allowlist.txt")
|
||||
|
||||
|
||||
def is_go_test_name(name):
|
||||
# cmd/go: "TestXxx" where Xxx does not start with a lower-case letter.
|
||||
return name == "Test" or not name[4].islower()
|
||||
|
||||
|
||||
def go_expected(root, go_json_path=None):
|
||||
"""Return ({pkg_dir: set(test names)} for packages with test files,
|
||||
{import path: pkg_dir} for every package)."""
|
||||
if go_json_path:
|
||||
with open(go_json_path, encoding="utf-8") as f:
|
||||
raw = f.read()
|
||||
else:
|
||||
raw = subprocess.run(["go", *GO_LIST_ARGS], cwd=root, check=True, capture_output=True, text=True).stdout
|
||||
decoder = json.JSONDecoder()
|
||||
pkgs, idx = [], 0
|
||||
while idx < len(raw):
|
||||
while idx < len(raw) and raw[idx].isspace():
|
||||
idx += 1
|
||||
if idx >= len(raw):
|
||||
break
|
||||
obj, idx = decoder.raw_decode(raw, idx)
|
||||
pkgs.append(obj)
|
||||
|
||||
expected, dirs = {}, {}
|
||||
for p in pkgs:
|
||||
rel = os.path.relpath(p["Dir"], root)
|
||||
rel = "" if rel == "." else rel.replace(os.sep, "/")
|
||||
if p.get("ImportPath"):
|
||||
dirs[p["ImportPath"]] = rel
|
||||
files = (p.get("TestGoFiles") or []) + (p.get("XTestGoFiles") or [])
|
||||
if not files:
|
||||
continue
|
||||
names = set()
|
||||
for name in files:
|
||||
with open(os.path.join(p["Dir"], name), encoding="utf-8", errors="replace") as f:
|
||||
for m in TEST_FUNC_RE.finditer(f.read()):
|
||||
t = m.group(1)
|
||||
if is_go_test_name(t) and not SKIP_RE.search(t):
|
||||
names.add(t)
|
||||
expected[rel] = names
|
||||
return expected, dirs
|
||||
|
||||
|
||||
def go_test_statuses(path, dirs):
|
||||
"""Return {pkg_dir: {test: status}} for top-level tests in `go test -json`
|
||||
output. Packages go list does not know are skipped."""
|
||||
out = {}
|
||||
with open(path, encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
if not line.startswith("{"):
|
||||
continue
|
||||
try:
|
||||
ev = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
test, action = ev.get("Test"), ev.get("Action")
|
||||
if not test or "/" in test or action not in GO_TEST_STATUS:
|
||||
continue
|
||||
pkg = dirs.get(ev.get("Package", ""))
|
||||
if pkg is None:
|
||||
continue
|
||||
out.setdefault(pkg, {})[test] = GO_TEST_STATUS[action]
|
||||
return out
|
||||
|
||||
|
||||
def label_pkg(label):
|
||||
# "//cmd/bd:bd_test" -> "cmd/bd", "@@//:x" -> ""
|
||||
label = label.lstrip("@")
|
||||
return label[2:].split(":", 1)[0] if label.startswith("//") else label.split(":", 1)[0]
|
||||
|
||||
|
||||
def query_go_test_pkgs(root, bazel):
|
||||
out = subprocess.run(
|
||||
[bazel, "query", 'kind("go_test rule", //...)', "--output=label"],
|
||||
cwd=root, check=True, capture_output=True, text=True,
|
||||
).stdout
|
||||
return {label_pkg(l) for l in out.split()}
|
||||
|
||||
|
||||
def read_bep(path):
|
||||
"""Return ({label: shard_count} for tested go_test targets, set of go_test
|
||||
labels configured, this invocation's testlogs dir or None)."""
|
||||
go_tests, shards = set(), {}
|
||||
exec_root, testlogs_rel = None, None
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
if not line.strip():
|
||||
continue
|
||||
ev = json.loads(line)
|
||||
eid = ev.get("id", {})
|
||||
if "workspace" in eid:
|
||||
exec_root = ev.get("workspaceInfo", {}).get("localExecRoot")
|
||||
elif "convenienceSymlinksIdentified" in eid:
|
||||
for link in ev.get("convenienceSymlinksIdentified", {}).get("convenienceSymlinks", []):
|
||||
if link.get("path", "").endswith("testlogs") and link.get("target"):
|
||||
testlogs_rel = link["target"]
|
||||
elif "targetConfigured" in eid:
|
||||
if ev.get("configured", {}).get("targetKind") == "go_test rule":
|
||||
go_tests.add(eid["targetConfigured"]["label"])
|
||||
elif "testResult" in eid:
|
||||
tr = eid["testResult"]
|
||||
shards[tr["label"]] = max(shards.get(tr["label"], 0), int(tr.get("shard", 1) or 1))
|
||||
tested = {l: n for l, n in shards.items() if l in go_tests}
|
||||
testlogs = None
|
||||
if exec_root and testlogs_rel:
|
||||
# The symlink target is relative to the output base, two levels above
|
||||
# the exec root (<output_base>/execroot/<workspace>).
|
||||
testlogs = os.path.join(os.path.dirname(os.path.dirname(exec_root)), testlogs_rel)
|
||||
return tested, go_tests, testlogs
|
||||
|
||||
|
||||
def testlog_xmls(testlogs, label, shard_count):
|
||||
pkg = label_pkg(label)
|
||||
name = label.split(":", 1)[1]
|
||||
base = os.path.join(testlogs, pkg, name)
|
||||
if shard_count <= 1:
|
||||
return [os.path.join(base, "test.xml")]
|
||||
return [os.path.join(base, f"shard_{i}_of_{shard_count}", "test.xml") for i in range(1, shard_count + 1)]
|
||||
|
||||
|
||||
def bazel_observed(tested, testlogs):
|
||||
"""Return ({pkg: {test: status}}, [problems])."""
|
||||
observed, problems = {}, []
|
||||
for label, n in sorted(tested.items()):
|
||||
pkg = label_pkg(label)
|
||||
per = observed.setdefault(pkg, {})
|
||||
for xml_path in testlog_xmls(testlogs, label, n):
|
||||
if not os.path.exists(xml_path):
|
||||
problems.append(f"{label}: {xml_path} missing (was test.xml downloaded? run with --config=ci)")
|
||||
continue
|
||||
try:
|
||||
tree = ET.parse(xml_path)
|
||||
except ET.ParseError as e:
|
||||
problems.append(f"{label}: cannot parse {xml_path}: {e}")
|
||||
continue
|
||||
cases = tree.getroot().iter("testcase")
|
||||
count = 0
|
||||
for tc in cases:
|
||||
name = tc.get("name", "")
|
||||
if "/" in name or not name.startswith("Test"):
|
||||
continue
|
||||
count += 1
|
||||
if tc.find("failure") is not None or tc.find("error") is not None:
|
||||
status = "failed"
|
||||
elif tc.find("skipped") is not None:
|
||||
status = "skipped"
|
||||
else:
|
||||
status = "passed"
|
||||
# A test seen in several targets keeps its worst status.
|
||||
if name not in per or STATUS_RANK[status] > STATUS_RANK[per[name]]:
|
||||
per[name] = status
|
||||
if count == 0 and os.path.getsize(xml_path) < 64:
|
||||
# An empty <testsuites/> means the binary ran no tests at all
|
||||
# (or GO_TEST_WRAP_TESTV was off). The test-name diff reports
|
||||
# which tests are missing; note it here for the log.
|
||||
problems.append(f"{label}: {os.path.relpath(xml_path, testlogs)} lists no testcases")
|
||||
return observed, problems
|
||||
|
||||
|
||||
def load_allowlist(path):
|
||||
entries, errors = [], []
|
||||
if not path or not os.path.exists(path):
|
||||
return entries, errors
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for n, line in enumerate(f, 1):
|
||||
body, _, why = line.partition("#")
|
||||
body = body.strip()
|
||||
if not body:
|
||||
continue
|
||||
fields = body.split()
|
||||
kind = fields[2] if len(fields) == 3 else "run"
|
||||
if len(fields) not in (2, 3) or kind not in ALLOW_KINDS or not why.strip():
|
||||
errors.append(
|
||||
f"{path}:{n}: want '<package dir> <test glob> [skip] # justification', got {line.strip()!r}"
|
||||
)
|
||||
continue
|
||||
entries.append({"pkg": fields[0], "test": fields[1], "kind": kind, "why": why.strip(), "line": n, "used": False})
|
||||
return entries, errors
|
||||
|
||||
|
||||
def allowed(entries, pkg, test, kind="run"):
|
||||
for e in entries:
|
||||
if e["kind"] == kind and fnmatch.fnmatchcase(pkg, e["pkg"]) and fnmatch.fnmatchcase(test, e["test"]):
|
||||
e["used"] = True
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--bep", required=True, help="--build_event_json_file of the bazel test run")
|
||||
ap.add_argument("--root", default=os.getcwd(), help="workspace root (default: cwd)")
|
||||
ap.add_argument("--testlogs", default=None, help="default: from the BEP, else <root>/bazel-testlogs")
|
||||
ap.add_argument("--allowlist", default=DEFAULT_ALLOWLIST)
|
||||
ap.add_argument("--go-list-json", default=None, help="precomputed `go list -json` output (tests)")
|
||||
ap.add_argument(
|
||||
"--go-test-json", default=None,
|
||||
help="`go test -json` output of the PR Core command on the same commit; enables the skip-parity check",
|
||||
)
|
||||
ap.add_argument("--bazel", default=os.environ.get("BAZEL", "bazel"))
|
||||
ap.add_argument("--no-query", action="store_true", help="skip the bazel query package-set check")
|
||||
ap.add_argument("--details", action="store_true", help="list every divergence (nightly)")
|
||||
ap.add_argument("--summary", default=None, help="append Markdown here (default: $GITHUB_STEP_SUMMARY)")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
root = os.path.abspath(args.root)
|
||||
entries, allow_errors = load_allowlist(args.allowlist)
|
||||
|
||||
expected, dirs = go_expected(root, args.go_list_json)
|
||||
tested, configured, bep_testlogs = read_bep(args.bep)
|
||||
# Prefer the BEP's own testlogs dir: the bazel-testlogs symlink follows
|
||||
# whatever bazel command ran last, possibly in another output base.
|
||||
testlogs = args.testlogs or bep_testlogs or os.path.join(root, "bazel-testlogs")
|
||||
observed, problems = bazel_observed(tested, testlogs)
|
||||
target_pkgs = None if args.no_query else query_go_test_pkgs(root, args.bazel)
|
||||
|
||||
no_target, missing, extra, allowlisted = [], [], [], []
|
||||
if target_pkgs is not None:
|
||||
for pkg in sorted(set(expected) - target_pkgs):
|
||||
(allowlisted if allowed(entries, pkg, "*") else no_target).append((pkg, "*"))
|
||||
for pkg, names in sorted(expected.items()):
|
||||
seen = observed.get(pkg, {})
|
||||
for t in sorted(names - set(seen)):
|
||||
(allowlisted if allowed(entries, pkg, t) else missing).append((pkg, t))
|
||||
for pkg, seen in sorted(observed.items()):
|
||||
for t in sorted(set(seen) - expected.get(pkg, set())):
|
||||
if SKIP_RE.search(t):
|
||||
continue
|
||||
(allowlisted if allowed(entries, pkg, t) else extra).append((pkg, t))
|
||||
|
||||
# Skip parity: passed under go test, skipped under Bazel.
|
||||
skip_div, skip_allowed, parity_notes = [], [], []
|
||||
go_status = go_test_statuses(args.go_test_json, dirs) if args.go_test_json else None
|
||||
if go_status is not None:
|
||||
for pkg, seen in sorted(observed.items()):
|
||||
gs = go_status.get(pkg, {})
|
||||
for t, bs in sorted(seen.items()):
|
||||
g = gs.get(t)
|
||||
if g is None or g == bs:
|
||||
continue
|
||||
if g == "passed" and bs == "skipped":
|
||||
(skip_allowed if allowed(entries, pkg, t, "skip") else skip_div).append((pkg, t))
|
||||
else:
|
||||
parity_notes.append((pkg, f"{t} (go test {g}, bazel {bs})"))
|
||||
allowlisted += skip_allowed
|
||||
|
||||
# Dedupe allowlisted pairs (a package-level entry also covers its tests).
|
||||
allowlisted = sorted(set(allowlisted))
|
||||
# Skip entries can only be exercised when go test statuses are known.
|
||||
unused = [e for e in entries if not e["used"] and (e["kind"] == "run" or go_status is not None)]
|
||||
|
||||
statuses = {"passed": 0, "skipped": 0, "failed": 0}
|
||||
for seen in observed.values():
|
||||
for s in seen.values():
|
||||
statuses[s] += 1
|
||||
n_expected = sum(len(v) for v in expected.values())
|
||||
ok = not (no_target or missing or extra or skip_div or allow_errors)
|
||||
|
||||
limit = None if args.details else 40
|
||||
|
||||
def listing(title, pairs):
|
||||
out = [f"{title} ({len(pairs)}):"]
|
||||
shown = pairs if limit is None else pairs[:limit]
|
||||
out += [f" {p or '.'} {t}" for p, t in shown]
|
||||
if limit is not None and len(pairs) > limit:
|
||||
out.append(f" ... {len(pairs) - limit} more (run with --details)")
|
||||
return out
|
||||
|
||||
lines = [
|
||||
f"go test: {len(expected)} packages with tests, {n_expected} top-level tests (excluding ^TestEmbedded)",
|
||||
f"bazel: {len(configured)} go_test targets configured, {len(tested)} tested; "
|
||||
f"{sum(len(v) for v in observed.values())} top-level tests seen "
|
||||
f"({statuses['passed']} passed, {statuses['skipped']} skipped, {statuses['failed']} failed)",
|
||||
]
|
||||
if target_pkgs is not None:
|
||||
lines.append(f"query: {len(target_pkgs)} packages with a go_test target")
|
||||
if go_status is None:
|
||||
lines.append("skips: not compared (no --go-test-json; the nightly run checks skip parity)")
|
||||
else:
|
||||
n_go = sum(len(v) for v in go_status.values())
|
||||
lines.append(
|
||||
f"skips: compared with {n_go} go test results: {len(skip_div) + len(skip_allowed)} passed under "
|
||||
f"go test but skipped under Bazel ({len(skip_allowed)} allowlisted), {len(parity_notes)} other differences"
|
||||
)
|
||||
lines.append(f"allowlisted divergences: {len(allowlisted)}")
|
||||
for title, pairs in (
|
||||
("packages with Go tests but no go_test target", no_target),
|
||||
("tests go test runs that Bazel did not", missing),
|
||||
("tests Bazel ran that go test would not", extra),
|
||||
("tests go test passed that Bazel skipped", skip_div),
|
||||
):
|
||||
if pairs:
|
||||
lines += listing(title, pairs)
|
||||
if parity_notes:
|
||||
lines += listing("other status differences (reported, not failed)", parity_notes)
|
||||
if args.details and allowlisted:
|
||||
lines += listing("allowlisted", allowlisted)
|
||||
for e in unused:
|
||||
lines.append(f"note: allowlist line {e['line']} ({e['pkg']} {e['test']} {e['kind']}) matched nothing; remove it")
|
||||
for p in problems:
|
||||
lines.append(f"note: {p}")
|
||||
lines += allow_errors
|
||||
verdict = "EQUIVALENCE: ok" if ok else "EQUIVALENCE: FAIL (unexplained divergence; fix the BUILD tags/targets or add a justified allowlist entry)"
|
||||
lines.append(verdict)
|
||||
print("\n".join(lines))
|
||||
|
||||
summary = args.summary or os.environ.get("GITHUB_STEP_SUMMARY")
|
||||
if summary:
|
||||
head = (
|
||||
f"**Equivalence**: {'ok' if ok else 'FAIL'}; go test {n_expected} tests in {len(expected)} packages, "
|
||||
f"Bazel saw {sum(len(v) for v in observed.values())} "
|
||||
f"(missing {len(missing)}, extra {len(extra)}, packages without target {len(no_target)}, "
|
||||
f"skipped only under Bazel {len(skip_div) if go_status is not None else 'not compared'}, "
|
||||
f"allowlisted {len(allowlisted)})\n\n"
|
||||
)
|
||||
with open(summary, "a", encoding="utf-8") as f:
|
||||
f.write(head)
|
||||
f.write("<details><summary>Equivalence detail</summary>\n\n```\n" + "\n".join(lines) + "\n```\n</details>\n\n")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
31
tools/bazel/equivalence_allowlist.txt
Normal file
31
tools/bazel/equivalence_allowlist.txt
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# Expected divergences between the Go tests `go test` runs (pr.yml) and the
|
||||
# ones `bazel test //... --config=ci` runs, for tools/bazel/equivalence.py.
|
||||
#
|
||||
# One entry per line, each with a justification:
|
||||
#
|
||||
# <package dir> <TestName or glob> # why Bazel does not run it
|
||||
# <package dir> <TestName or glob> skip # why Bazel skips a test go test runs
|
||||
#
|
||||
# `*` as the test name covers the whole package, including a package with no
|
||||
# go_test target. `skip` entries are checked only when go test statuses are
|
||||
# supplied (--go-test-json, the nightly run). Keep this list short: every entry
|
||||
# is a test the Bazel lane does not exercise. The checker reports entries that
|
||||
# no longer match anything.
|
||||
|
||||
# The in-test `go build` of bd that this test guards does not happen under
|
||||
# Bazel (bd comes in through BEADS_TEST_BD_BINARY); go test still runs it.
|
||||
cmd/bd TestGOMODCACHENotUnderTestHome skip # no in-test go build under Bazel; guarded by go test
|
||||
# Needs TestAAAStdioBaseline in the same process; sharding splits them.
|
||||
# TestMain's checkStdioAfterRun still checks every shard.
|
||||
cmd/bd TestZZStdioNotLeaked skip # baseline test lands in another shard; checkStdioAfterRun covers each shard
|
||||
|
||||
# Repository-policy tests that walk the whole source checkout (git ls-files,
|
||||
# every BUILD file, .beads/ formulas). Bazel's runfiles hold only declared
|
||||
# data, so they skip under TEST_SRCDIR; PR Core's go test runs them.
|
||||
scripts TestBazelGoSrcsBlocksCurrent skip # walks the source checkout; runs under go test
|
||||
scripts TestBazelNoPackagesUnderDocsTrees skip # walks the source checkout; runs under go test
|
||||
scripts TestBazelNoRemoteEndpointsInTrackedFiles skip # scans git-tracked files; runs under go test
|
||||
scripts TestBazelTestTagsFollowTaxonomy skip # reads every BUILD file in the checkout; runs under go test
|
||||
scripts TestBazelTreeGoSrcsListsEveryPackage skip # walks the source checkout; runs under go test
|
||||
scripts TestCheckTestingShortPassesOnCleanRepoTree skip # walks the whole checkout; runs under go test
|
||||
scripts TestReleaseFormula* skip # .beads/ is in .bazelignore, so the formula cannot be data; runs under go test
|
||||
Loading…
Add table
Add a link
Reference in a new issue