ci(bazel): mode-cache readers use the remote repo contents cache via rbe-cache (#7476)

* ci(bazel): mode cache reads the remote repo contents cache through rbe-cache

Mode cache (fork and Dependabot PRs while rbe-fork is closed, rbe=cache
dispatches) now reads Bazel's remote repo contents cache: the trees rrc-seed
writes on main pushes, served by rbe-cache's anonymous GetActionResult and
ByteStream.Read (Caddy :8443 -> nativelink-anon -> AC_OSS, read only).
setup-bazel's write-bazelrc.sh adds the trusted readers' two lines to its
fork-cache rc (passed with --bazelrc, so the startup line applies):

  startup --experimental_remote_repo_contents_cache
  common --loading_phase_threads=64

The committed .bazelrc stays untouched (design R2): the flag is a startup
option and cannot live in build:fork-cache, and developer and macOS
machines never read.

Gated on the new .github/actions/setup-bazel/cache-rrc-probe.sh (a copy of
gascity's tools/rbe/cache-rrc-probe.sh): one GetCapabilities to rbe-cache
(connect 3s, total 5s). Only an answer (gRPC 0, cache_capabilities present)
writes the lines; anything else writes none and logs a ::notice:: with the
reason, and the script never fails on it. Kill switch: fork_rrc_read=on in
that script; set it to off in a one-line PR to stop mode cache reading.

Measured (scratch origin/main, no credentials, 8 CPUs, cold output base,
`build --nobuild //...`): repository rules 665-672 -> 38-41 (the rest are
bd_releases/dolt_release and local or toolchain repos, not reproducible),
analysis 26.5-26.7s -> 12.1-14.8s, wall about 36s -> 25-32s.

Tests: TestSetupBazelRCWriter runs the writer against a stand-in rbe-cache
(answers, answers with zstd, PERMISSION_DENIED, refused): exactly the
reader's lines after --config=fork-cache and any zstd line only when it
answers, a notice otherwise; remote-exec and rbe-fork never get them nor
probe. TestBazelCacheModeReachesTheRC checks no simulated mode carries them
with the probe refused. TestCacheRRCProbe runs the real probe (errors,
malformed, timeout, refused, untrusted cert, kill switch off without a
request).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): mode cache rrc review fixes: reader opt-in, containment, AC probe

Scope. write-bazelrc.sh enabled the remote repo contents cache for every
mode cache job, bazel-release-cross included, whose configs
rrc-lane-commands.txt does not seed. setup-bazel now takes BAZEL_RRC_READ
("true" or empty, anything else fails): only with it does mode cache probe
rbe-cache and add the reader's lines. The 8 trusted reader jobs pass it,
and nothing else. TestBazelRRCOptInMirrorsTheTrustedReaders pins that set
to bazelRRCReadLanes; TestBazelRRCLaneCommandsCoverEveryReader now checks
every job that enables the flag either way (reader step or opt-in) is a
reading lane whose bazel configs are all seeded. TestSetupBazelRCWriter
covers a job without the opt-in (no probe, no lines) and bad values.

Runner. The 5 readers that run in mode cache (test, pure, integration,
doltserver, cmd-dolt) fall back to ubuntu-24.04 instead of ubuntu-latest:
one runs-on line each; mode remote stays Blacksmith, and the remote-only
readers and every other job are unchanged.

Visibility. write-bazelrc.sh writes one step-summary line in mode cache:
whether the job reads, and why (the probe's reason, with rbe-cache's
GetCapabilities TCP connect time, DNS excluded).

Probe. cache-rrc-probe.sh is re-copied from gascity: it also asks
GetActionResult for a fixed key no action has and a ByteStream Read of
that digest as a blob, and passes only if both answer NOT_FOUND (gRPC 5),
proving Caddy, nativelink-anon, the action cache and the CAS answer; it
reports the TCP connect time minus the DNS lookup. serveCapabilities
answers both; TestCacheRRCProbe adds their failure answers.

Containment and limits. The rrc-verify issue body and
CI_REQUIRED_CHECK_TOPOLOGY.md no longer say to set RBE_REPO_CONTENTS_CACHE
off (mode cache never sees it, and off also stops rrc-verify): commit
fork_rrc_read=off in both probe scripts (beads has no fresh-merge, so an
open fork PR keeps its old switch until it is pushed again); for immediate
server-side containment, rbe-cache-gate close on core; and, because the
mode remote readers read the same oss action cache entries, set the
variable to seed on a mismatch (no lane reads, rrc-seed and
rrc-verify keep running), off only to stop seeding too, and leave it as
it is when the run only could not check. The doc also says a
closed or down cache only falls back at probe time (after loading, files
are fetched lazily and losing rbe-cache fails the command, bazel#30218),
and notes the 512 MiB anonymous CAS limit under gascity's R4.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): bazel-dolt-race opts in to the rrc read like the other trusted readers

#7464 added bazel-dolt-race as a trusted reader after this branch was cut, so
TestBazelRRCOptInMirrorsTheTrustedReaders now covers it: pass BAZEL_RRC_READ
(it runs only in mode remote, so this changes nothing today). It never runs in
mode cache, so it keeps the lanes' ubuntu-latest fallback: the cache-reader
runner rule (ubuntu-24.04) excludes it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-10-10 17:42:12 +00:00 • committed by GitHub
commit c1ccb728ec
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 691 additions and 50 deletions

View file

@ -8,7 +8,9 @@ description: >
BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY in the step env, that rc
enables --config=remote-exec for every bazel command; with none of them the
job runs locally; with BAZEL_FORK_CACHE=true (and no secrets) it enables
the read-only fork cache (--config=fork-cache); with BAZEL_FORK_REMOTE=true
the read-only fork cache (--config=fork-cache), and with BAZEL_RRC_READ=true
too (the remote repo contents cache readers) Bazel's remote repo contents
cache through rbe-cache while it answers; with BAZEL_FORK_REMOTE=true
(fork and Dependabot PRs, bazel.yml modes fork-ro/fork-rw; needs
RBE_FORK_PR and RBE_FORK_TIER) it gets a short-lived certificate from
rbe-fork-mint (fork-credential.sh, proving the run with a CSR artifact) and

View file

@ -0,0 +1,151 @@
#!/usr/bin/env bash
# May this fork-cache lane read Bazel's remote repo contents cache through
# rbe-west's anonymous read-only cache (rbe-cache :8443, .bazelrc's
# fork-cache) right now? Exits 0 only when the kill switch below is on,
# rbe-cache answers a REAPI GetCapabilities for instance oss with gRPC status
# 0 and a ServerCapabilities that has cache_capabilities, and then answers a
# GetActionResult for a key no action has, and a ByteStream Read for a blob
# the CAS does not hold, each with NOT_FOUND (gRPC 5): the whole read chain
# (Caddy, nativelink-anon, the action cache and the CAS) is up. Anything else
# exits nonzero: switched off, an HTTP or gRPC error, a refused connection,
# a timeout, an answer that does not parse, no curl or python3. The caller
# then writes no repo contents cache lines, and never fails on it. The
# verdict carries the GetCapabilities TCP connect time without the DNS
# lookup (about one round trip to rbe-cache), for the lane's step summary.
#
# Why a probe: with the startup flag set and the cache unreachable (closed
# by rbe-west's kill switch, or down), Bazel retries every repository
# against it before fetching it itself (engdocs/design/
# bazel-remote-repo-contents-cache.md, R6), which costs far more than the
# cache saves. Fork pull_request runs see no repository variables, so the
# opt-out is the committed line below and the server's own answer.
#
# Three gRPC calls over curl's HTTP/2, the first answer decoded by
# python3's standard library: no grpcurl, no new action. Nothing goes to
# stdout (callers write their rc from it); one verdict line goes to stderr. A copy lives in gascity
# (tools/rbe/) and beads (.github/actions/setup-bazel/); keep in sync, except
# the kill switch, which is each repository's own.
#
# Tests only: RBE_CACHE_PROBE_URL (a stand-in server) and
# RBE_CACHE_PROBE_MAX_TIME (seconds).
set -uo pipefail
# Kill switch: off stops this repository's fork-cache lanes reading the
# remote repo contents cache (they fetch external repositories themselves),
# with a one-line pull request; on lets rbe-cache's answer decide. To stop
# every reader of rbe-cache, set it off in both copies (gascity and beads).
# It reaches a lane when the lane's checkout carries it.
fork_rrc_read=on
url=${RBE_CACHE_PROBE_URL:-https://rbe-cache.ops.gascity.com:8443}
max_time=${RBE_CACHE_PROBE_MAX_TIME:-5}
rtt=
no() {
echo "rbe-cache rrc probe: $*${rtt:+ ($rtt)}; this lane fetches external repositories itself" >&2
exit 1
}
[ "$fork_rrc_read" = on ] || no "switched off (fork_rrc_read=$fork_rrc_read in $(basename "$0"))"
command -v curl >/dev/null 2>&1 || no "no curl"
command -v python3 >/dev/null 2>&1 || no "no python3"
tmp=$(mktemp -d) || no "no temp dir"
trap 'rm -rf "$tmp"' EXIT
# call WHAT METHOD REQ: one gRPC call (METHOD: service/method); the answer's
# messages go to $tmp/body, its grpc-status to $st, and the TCP connect time
# without the DNS lookup to $tc. Fails the probe (prefixed with WHAT) on a
# transport error or a non-200 answer.
call() {
local what=$1 out code dns
out=$(curl -sS --http2 --connect-timeout 3 --max-time "$max_time" \
-H 'content-type: application/grpc' -H 'te: trailers' \
--data-binary @"$3" -D "$tmp/head" -o "$tmp/body" -w '%{http_code} %{time_namelookup} %{time_connect}' \
"$url/$2" 2>"$tmp/err") ||
no "$what$(tr '\n' ' ' <"$tmp/err")"
read -r code dns tc <<<"$out"
tc=$(awk -v d="$dns" -v c="$tc" 'BEGIN { if (c > d) printf "%.6f", c - d }' 2>/dev/null) || tc=
[ "$code" = 200 ] || no "${what}HTTP $code"
# grpc-status is a trailer; curl writes trailers after the headers.
st=$(tr -d '\r' <"$tmp/head" | sed -n 's/^grpc-status: *//Ip' | tail -1)
}
# GetCapabilitiesRequest{instance_name: "oss"} (fork-cache's
# --remote_instance_name) as one uncompressed gRPC message.
printf '\000\000\000\000\005\012\003oss' >"$tmp/req" || no "cannot write the request"
call "" build.bazel.remote.execution.v2.Capabilities/GetCapabilities "$tmp/req"
rtt=$(awk -v s="$tc" 'BEGIN { if (s > 0) printf "tcp connect %.1f ms", s * 1000 }' 2>/dev/null) || rtt=
[ "$st" = 0 ] || no "gRPC status ${st:-missing}"
why=$(python3 -I - "$tmp/body" <<'PY'
import sys
def varint(b, i):
v = s = 0
while True:
if i >= len(b) or s > 63:
raise ValueError("truncated varint")
c = b[i]
i += 1
v |= (c & 0x7F) << s
s += 7
if c < 0x80:
return v, i
def fields(b):
i = 0
while i < len(b):
key, i = varint(b, i)
num, wire = key >> 3, key & 7
if wire == 0:
v, i = varint(b, i)
elif wire == 1:
v, i = b[i:i + 8], i + 8
elif wire == 2:
n, i = varint(b, i)
v, i = b[i:i + n], i + n
elif wire == 5:
v, i = b[i:i + 4], i + 4
else:
raise ValueError("wire type %d" % wire)
if i > len(b):
raise ValueError("truncated field %d" % num)
yield num, wire, v
try:
body = open(sys.argv[1], "rb").read()
if len(body) < 5 or body[0] != 0:
raise ValueError("not an uncompressed gRPC message")
n = int.from_bytes(body[1:5], "big")
if len(body) != 5 + n:
raise ValueError("gRPC length %d, %d bytes" % (n, len(body) - 5))
# ServerCapabilities.cache_capabilities
cache = [v for num, wire, v in fields(body[5:]) if num == 1 and wire == 2]
except (OSError, ValueError) as e:
print("unreadable answer (%s)" % e)
sys.exit(2)
if not cache:
print("no cache_capabilities in the answer")
sys.exit(1)
PY
) || no "${why:-python3 failed}"
# GetActionResultRequest{instance_name: "oss", action_digest: {hash:
# sha256("cache-rrc-probe: no action has this key"), size_bytes: 1}}: a key
# no action has, so the action cache behind rbe-cache must say NOT_FOUND.
key=ad12d7a2f1dec188fc1f2c4f5535a4a65eb630fa2f309eca423695e6ec7abee7
printf '\000\000\000\000\113\012\003oss\022\104\012\100%s\020\001' "$key" >"$tmp/req" ||
no "cannot write the request"
call "GetActionResult: " build.bazel.remote.execution.v2.ActionCache/GetActionResult "$tmp/req"
[ "$st" = 5 ] || no "GetActionResult: gRPC status ${st:-missing}, want 5 (NOT_FOUND)"
# ReadRequest{resource_name: "oss/blobs/<the same hash>/1"}: a blob the CAS
# does not hold (its content would have to hash to that key), so the CAS
# behind rbe-cache must say NOT_FOUND.
printf '\000\000\000\000\116\012\114oss/blobs/%s/1' "$key" >"$tmp/req" ||
no "cannot write the request"
call "ByteStream Read: " google.bytestream.ByteStream/Read "$tmp/req"
[ "$st" = 5 ] || no "ByteStream Read: gRPC status ${st:-missing}, want 5 (NOT_FOUND)"
echo "rbe-cache rrc probe: rbe-cache answers GetCapabilities, GetActionResult and ByteStream Read${rtt:+ ($rtt)}; this lane reads the remote repo contents cache" >&2

View file

@ -32,7 +32,13 @@
# means no) it also asks for zstd transfers (--remote_cache_compression). A
# probe, not a repository variable: fork pull_request runs see no vars. No
# other mode ever asks, because only the anonymous cache can advertise zstd,
# and Bazel refuses a remote that does not.
# and Bazel refuses a remote that does not. With BAZEL_RRC_READ=true too
# (bazel.yml's remote repo contents cache readers alone: the jobs whose
# configs rrc-lane-commands.txt seeds), and while rbe-cache answers
# (cache-rrc-probe.sh, which also holds the committed kill switch), it reads
# Bazel's remote repo contents cache, as those jobs' trusted readers do: the
# trees rrc-seed wrote on main pushes, through rbe-cache's anonymous AC and
# CAS reads. The job's step summary says whether it reads, and why.
#
# RBE_FORK_CERT_FILE + RBE_FORK_KEY_FILE + RBE_FORK_ENDPOINT + RBE_FORK_INSTANCE
# (bazel.yml modes fork-ro/fork-rw: fork-credential.sh's outputs, a
@ -94,6 +100,7 @@ key="${RBE_TLS_KEY:-}"
ca="${RBE_TLS_CA:-}"
instance="${RBE_INSTANCE:-}"
fork_cache="${BAZEL_FORK_CACHE:-}"
rrc_reader="${BAZEL_RRC_READ:-}"
fork_cert="${RBE_FORK_CERT_FILE:-}"
fork_key="${RBE_FORK_KEY_FILE:-}"
fork_endpoint="${RBE_FORK_ENDPOINT:-}"
@ -116,6 +123,13 @@ case "$fork_cache" in
exit 1
;;
esac
case "$rrc_reader" in
"" | true) ;;
*)
echo "setup-bazel: BAZEL_RRC_READ must be \"true\" or empty, got \"$rrc_reader\"" >&2
exit 1
;;
esac
rc="$secret_dir/ci.bazelrc"
umask 077
@ -227,14 +241,38 @@ elif [[ "$set_fields" -ne 0 ]]; then
echo "setup-bazel: remote execution is partially configured; BAZEL_REMOTE_EXECUTOR, RBE_TLS_CERT and RBE_TLS_KEY must be set together (or none, for local execution)" >&2
exit 1
elif [[ "$fork_cache" == true ]]; then
# The remote repo contents cache, read-only, for the jobs that opt in
# (BAZEL_RRC_READ: only their configs are seeded), while rbe-cache
# answers and the probe's kill switch (fork_rrc_read) is on. With the
# startup flag set and the cache unreachable, every repository first
# fails a lookup (gascity engdocs/design/bazel-remote-repo-contents-cache.md,
# R6), so a closed or down rbe-cache gets no lines, and a notice says
# why. That holds at probe time only: after loading, Bazel fetches the
# files lazily, and losing rbe-cache then fails the command (R4). Both
# lines are key neutral, as for the trusted readers; fork-cache uploads
# nothing, and rbe-cache refuses every write.
rrc_read=false
if [[ "$rrc_reader" != true ]]; then
why="not a remote repo contents cache reader (no BAZEL_RRC_READ)"
elif why=$(bash "$(dirname "${BASH_SOURCE[0]}")/cache-rrc-probe.sh" 2>&1); then
rrc_read=true
echo "$why"
else
echo "::notice title=No remote repo contents cache::$why"
fi
echo "Fork cache remote repo contents cache read: **$([[ "$rrc_read" == true ]] && echo on || echo off)** (${why#rbe-cache rrc probe: })" >>"${GITHUB_STEP_SUMMARY:-/dev/null}"
# zstd for the anonymous cache's transfers while it advertises zstd:
# transport only, so action keys are unchanged. The probe never fails
# this script; it only decides the line.
# transport only, so action keys are unchanged. The probes never fail
# this script; they only decide the lines.
{
echo "build --config=fork-cache"
if bash "$(dirname "${BASH_SOURCE[0]}")/cache-zstd-probe.sh"; then
echo "build:fork-cache --remote_cache_compression"
fi
if [[ "$rrc_read" == true ]]; then
echo "startup --experimental_remote_repo_contents_cache"
echo "common --loading_phase_threads=64"
fi
} >>"$rc"
cache=true
echo "setup-bazel: read-only remote cache (rbe-cache); executing locally"

View file

@ -512,10 +512,11 @@ jobs:
# canary seed, and bazel-test reads;
# on seed, and every lane in .github/scripts/rrc-lane-commands.txt
# reads.
# Mode remote only: fork pull_request runs see no vars, and only
# rbe-west's trusted edge serves the entries. Any other value is off,
# with a warning, so a typo never fails the required gate. Like decide,
# this step runs no repository code (it comes before the checkout).
# Mode remote only: fork pull_request runs see no vars (mode cache
# reads through setup-bazel's rc instead, in the jobs that pass
# BAZEL_RRC_READ, behind its probe of rbe-cache: write-bazelrc.sh). Any other value is off, with a warning, so a typo
# never fails the required gate. Like decide, this step runs no
# repository code (it comes before the checkout).
- name: Remote repo contents cache
id: rrc
env:
@ -763,7 +764,10 @@ jobs:
# and cache runs (forks, rbe=off/cache, no secrets) build everything
# they miss on the runner, so they get the larger GitHub-hosted one.
# Fork modes (fork-ro/fork-rw) execute remotely but stay GitHub-hosted,
# like every fork job: the runner is only a client.
# like every fork job: the runner is only a client. ubuntu-24.04, not
# ubuntu-latest, in every lane that reads the remote repo contents cache
# in mode cache: the cached trees are Linux x86_64 ones from a 24.04 host
# (design R2), so the image must not move under them.
#
# 8 vCPU, not the other lanes' 4: with every action a remote cache hit,
# this lane's `bazel test //... --config=ci` spent ~63 of 81s in
@ -772,7 +776,7 @@ jobs:
# parallelizes across the client's cores. Analysis-only, seeded runner
# cache (ga-vnycm2.8): 38.3s on 2 vCPU, 25.6s on 4, 16.3s on 8.
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
# Remote execution goes to rbe-west (instance "oss", the Blacksmith-backed
# worker pool shared with gascity). Skipped only in mode skip (same-repo
# runs while RBE_WEST_WORKERS is unset).
@ -847,6 +851,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -1061,7 +1066,7 @@ jobs:
name: pure-Go and js/wasm
# Same runner and skip rule as bazel-test.
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
if: ${{ needs.rbe.outputs.mode != 'skip' }}
timeout-minutes: 90
outputs:
@ -1116,6 +1121,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -1413,6 +1419,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -1560,7 +1567,7 @@ jobs:
# A caller may turn it off with integration: "off"; pr.yml and
# bazel-farm.yml pass "on".
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
if: ${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outputs.mode == 'cache') && inputs.integration != 'off' }}
timeout-minutes: 85
outputs:
@ -1594,6 +1601,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -1705,7 +1713,7 @@ jobs:
name: dolt-server lane
# Same runner and skip rule as bazel-test.
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
if: ${{ needs.rbe.outputs.mode != 'skip' }}
timeout-minutes: 90
outputs:
@ -1748,6 +1756,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -1877,6 +1886,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -2058,6 +2068,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -2205,6 +2216,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -2342,7 +2354,7 @@ jobs:
bazel-cmd-dolt:
name: cmd/bd Dolt-server tier
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
if: ${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outputs.mode == 'cache') && inputs.integration != 'off' }}
timeout-minutes: 85
continue-on-error: true
@ -2377,6 +2389,7 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
RBE_FORK_PR: ${{ github.event.pull_request.number }}
@ -2840,7 +2853,7 @@ jobs:
title="rrc-verify: remote repo contents cache differs from a cold fetch"
body="The nightly rrc-verify run found cached repository trees that differ from a cold fetch, or could not check them: $RUN_URL (summary and the rrc-verify report artifact).
A mismatch is a poisoned entry or a repository rule falsely marked reproducible (gascity engdocs/design/bazel-remote-repo-contents-cache.md, R1/R2). Stop readers first: set the repository variable RBE_REPO_CONTENTS_CACHE to off. Then follow gascity/infra nativelink-cas/west/README.md (remote repo contents cache writer, poisoned entries)."
A mismatch is a poisoned entry or a repository rule falsely marked reproducible (gascity engdocs/design/bazel-remote-repo-contents-cache.md, R1/R2). Contain the fork-cache readers first: they see no repository variables, so (1) commit fork_rrc_read=off in both probe scripts, beads .github/actions/setup-bazel/cache-rrc-probe.sh and gascity tools/rbe/cache-rrc-probe.sh (beads has no fresh-merge: an open fork PR keeps the old switch until it is pushed again); (2) for immediate server-side containment, an operator runs rbe-cache-gate close on core, which stops all fork-cache, and fails fork-cache lanes already reading; (3) on a mismatch, set RBE_REPO_CONTENTS_CACHE to seed: the trusted readers read the same entries (rbe-cache serves the oss action cache), and seed stops every one of them while rrc-seed and this check keep running; set it to off only to stop seeding too, which also stops rrc-verify; if this run only says it could not check, leave the variable as it is. Then follow gascity/infra nativelink-cas/west/README.md (remote repo contents cache writer, poisoned entries)."
gh label create rrc-verify -R "$GITHUB_REPOSITORY" --color B60205 \
--description "Remote repo contents cache verification failures" 2>/dev/null || true
n=$(gh issue list -R "$GITHUB_REPOSITORY" --label rrc-verify --state open --json number --jq '.[0].number // empty')

View file

@ -675,15 +675,64 @@ Bazel; both are key neutral, and the lanes keep
`--noremote_upload_local_results`. While the variable is not `off`,
nightly.yml's call also runs `rrc-verify`: a cold fetch of every lane,
compared with the cached entries by `tools/bazel/rrc_verify.py`; a mismatch
fails the job and opens an `rrc-verify` issue.
fails the job and opens an `rrc-verify` issue, which lists the containment
steps below.
Mode `cache` lanes (fork PRs while rbe-fork is closed, `rbe=cache`
dispatches) see no repository variables. The jobs with a reader step above
pass `BAZEL_RRC_READ: 'true'` to setup-bazel, and only those: their configs
are the ones `rrc-lane-commands.txt` seeds (`bazel-release-cross` builds for
other platforms and does not opt in). In mode `cache` they read the same
entries through rbe-cache's anonymous AC and CAS reads: `write-bazelrc.sh`
adds the same two lines to its rc whenever
`.github/actions/setup-bazel/cache-rrc-probe.sh` gets a GetCapabilities
answer from rbe-cache, then NOT_FOUND for a GetActionResult key no action
has and for a ByteStream Read of a blob the CAS does not hold. The reader
lanes that run in mode `cache` use a pinned `ubuntu-24.04` runner there,
not `ubuntu-latest`. Each job's step summary records whether it reads, and
rbe-cache's TCP connect time (DNS lookup excluded). rbe-cache refuses every
write, and fork-cache uploads nothing.
An unreachable cache with the startup flag set costs every repository a
failed lookup, so a cache that is closed or down when the probe runs gets
no lines and a notice instead, and the lane fetches as before. That holds
at probe time only. After loading, Bazel fetches a cached repository's
files lazily from rbe-cache (:8443). If the cache goes away mid-run
(rbe-cache-gate closes, the runner's IP is banned, or the breaker opens),
the command fails, and Bazel does not re-fetch the repository (bazel#30218,
gascity design R4).
To stop mode `cache` lanes reading:
1. Commit `fork_rrc_read=off` in both probe scripts: beads'
`.github/actions/setup-bazel/cache-rrc-probe.sh` and gascity's
`tools/rbe/cache-rrc-probe.sh`. This is the preferred way: lanes that
already read finish, and later runs fetch for themselves. beads has no
fresh-merge, so an open fork PR keeps its old switch until it is pushed
again.
2. For immediate server-side containment, an operator runs
`rbe-cache-gate close` on core. That stops all fork-cache, cache hits
included, and fails the lanes that are reading at that moment.
3. The mode `remote` lanes read the same entries: rbe-cache serves the same
`oss` action cache. On an `rrc-verify` mismatch, set
`RBE_REPO_CONTENTS_CACHE` to `seed`. That stops every mode `remote`
reader, while `rrc-seed` and `rrc-verify` keep running. Set it to `off`
only to stop seeding too; that also stops `rrc-verify`. If the run only
says it could not check the cache, leave the variable as it is.
Known limit (gascity design R4): rbe-cache's anonymous CAS is a size
partition whose upper store, for blobs of 512 MiB or more, is a noop. A
cached repository with a file that large is a hit whose download always
fails. rrc-gate does not refuse such entries today.
`rrc-seed` asks for `id-token: write` and `rrc-verify` for `issues: write`, so
every caller of `bazel.yml` (pr.yml, nightly.yml, bazel-farm.yml) grants both:
GitHub checks a called workflow's job permissions when the run starts, even
for jobs that skip. Neither job runs on a pull_request, merge_group or
pull_request_target event, and every other `bazel.yml` job keeps
`contents: read`. Rollback: set the variable to `off` (or `seed`, to keep the
cache warm without readers).
`contents: read`. Rollback for the mode `remote` readers: set the variable to
`off` (or `seed`, to keep the cache warm without readers). It does not reach
mode `cache`; use the steps above.
## Required Check Contract

View file

@ -71,6 +71,8 @@ func TestBazelCacheModeReachesTheRC(t *testing.T) {
return tier
case "${{ github.event.pull_request.number }}":
return "7123" // every start that reaches a fork mode is a pull_request
case "true": // BAZEL_RRC_READ, a literal (the remote repo contents cache readers)
return "true"
}
m := setupExpr.FindStringSubmatch(expr)
if m == nil {
@ -344,6 +346,15 @@ func checkModeRC(t *testing.T, lane, mode, outputs, rc string, files []string, l
if has("remote_cache_compression") {
t.Errorf("%s (mode %s): rc asks for compression:\n%s", lane, mode, rc)
}
// The remote repo contents cache: mode cache's probe is refused here too
// (TestSetupBazelRCWriter runs it answered), and no other mode probes,
// so no rc here reads it (the trusted lanes read through bazel.yml's own
// reader steps, in .bazelrc.local).
for _, line := range bazelRRCReadLines {
if has(line) {
t.Errorf("%s (mode %s): rc carries %q:\n%s", lane, mode, line, rc)
}
}
credentialFree := func() {
for _, bad := range []string{"remote_executor", "remote-exec", "tls_", "remote_instance_name", "--remote_upload_local_results", "--remote_cache", "--remote_header", "--bes_", "farm.invalid"} {
if has(bad) {
@ -408,6 +419,14 @@ func checkModeRC(t *testing.T, lane, mode, outputs, rc string, files []string, l
// of gascity's tools/rbe/cache-zstd-probe.sh).
const cacheZstdProbe = setupBazelActionDir + "/cache-zstd-probe.sh"
// cacheRRCProbe: setup-bazel's remote repo contents cache probe for the fork
// cache (a copy of gascity's tools/rbe/cache-rrc-probe.sh; the kill switch
// line, cacheRRCKillLine, is beads' own).
const (
cacheRRCProbe = setupBazelActionDir + "/cache-rrc-probe.sh"
cacheRRCKillLine = "fork_rrc_read=on"
)
// refusedProbeURL: a loopback port nothing listens on, the probe's URL in
// tests that do not serve one (a refused connection: no flag).
const refusedProbeURL = "https://127.0.0.1:1"
@ -448,17 +467,37 @@ func grpcMessage(m []byte) []byte {
// capsAnswer is how the stand-in rbe-cache answers GetCapabilities: HTTP
// status (0: 200), body, and grpc-status ("": none; with no body, in the
// headers, as gRPC's trailers-only errors are), after delay.
// headers, as gRPC's trailers-only errors are), after delay. ac and bs answer
// cache-rrc-probe.sh's GetActionResult and ByteStream Read the same way; nil
// is rbe-cache's answer for a key no action has, or a blob it does not hold:
// a trailers-only NOT_FOUND.
type capsAnswer struct {
status int
grpc string
body []byte
delay time.Duration
ac, bs *capsAnswer
}
// acNotFound: rbe-cache's answer to a GetActionResult for a key no action
// has, or a ByteStream Read for a blob it does not hold (capsAnswer.ac's and
// capsAnswer.bs's default).
var acNotFound = capsAnswer{grpc: "5"}
// acRequest: cache-rrc-probe.sh's GetActionResultRequest{instance_name:
// "oss", action_digest: {sha256("cache-rrc-probe: no action has this key"),
// 1}}, a key no action has.
var acRequest = grpcMessage(append(pbBytes(1, []byte("oss")),
pbBytes(2, append(pbBytes(1, []byte("ad12d7a2f1dec188fc1f2c4f5535a4a65eb630fa2f309eca423695e6ec7abee7")), pbVarint(2, 1)...))...))
// bsRequest: cache-rrc-probe.sh's ReadRequest{resource_name:
// "oss/blobs/<acRequest's hash>/1"}, a blob the CAS does not hold.
var bsRequest = grpcMessage(pbBytes(1, []byte("oss/blobs/ad12d7a2f1dec188fc1f2c4f5535a4a65eb630fa2f309eca423695e6ec7abee7/1")))
// serveCapabilities answers a like rbe-cache's Caddy, over TLS and HTTP/2,
// after checking the request is the probe's GetCapabilities for instance
// oss. It returns the probe's RBE_CACHE_PROBE_URL, a CA file for curl's
// after checking the request is a probe's GetCapabilities, or
// cache-rrc-probe.sh's GetActionResult or ByteStream Read, for instance oss.
// It returns the probe's RBE_CACHE_PROBE_URL, a CA file for curl's
// CURL_CA_BUNDLE, and the request count.
func serveCapabilities(t *testing.T, a capsAnswer) (url, caFile string, hits *atomic.Int32) {
t.Helper()
@ -466,37 +505,49 @@ func serveCapabilities(t *testing.T, a capsAnswer) (url, caFile string, hits *at
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits.Add(1)
body, _ := io.ReadAll(r.Body)
const path = "/build.bazel.remote.execution.v2.Capabilities/GetCapabilities"
path, want, answer := "/build.bazel.remote.execution.v2.Capabilities/GetCapabilities", capsRequest, a
switch r.URL.Path {
case "/build.bazel.remote.execution.v2.ActionCache/GetActionResult":
path, want, answer = r.URL.Path, acRequest, acNotFound
if a.ac != nil {
answer = *a.ac
}
case "/google.bytestream.ByteStream/Read":
path, want, answer = r.URL.Path, bsRequest, acNotFound
if a.bs != nil {
answer = *a.bs
}
}
if r.ProtoMajor != 2 || r.Method != http.MethodPost || r.URL.Path != path ||
r.Header.Get("Content-Type") != "application/grpc" || !bytes.Equal(body, capsRequest) {
r.Header.Get("Content-Type") != "application/grpc" || !bytes.Equal(body, want) {
t.Errorf("probe sent %s %s %s, content-type %q, body %x; want HTTP/2 POST %s, application/grpc, %x",
r.Proto, r.Method, r.URL.Path, r.Header.Get("Content-Type"), body, path, capsRequest)
r.Proto, r.Method, r.URL.Path, r.Header.Get("Content-Type"), body, path, want)
w.WriteHeader(http.StatusBadRequest)
return
}
select {
case <-time.After(a.delay):
case <-time.After(answer.delay):
case <-r.Context().Done():
return
}
w.Header().Set("Content-Type", "application/grpc")
trailer := a.grpc != "" && a.body != nil
trailer := answer.grpc != "" && answer.body != nil
if trailer {
w.Header().Set("Trailer", "Grpc-Status")
} else if a.grpc != "" {
w.Header().Set("Grpc-Status", a.grpc)
} else if answer.grpc != "" {
w.Header().Set("Grpc-Status", answer.grpc)
}
status := a.status
status := answer.status
if status == 0 {
status = http.StatusOK
}
w.WriteHeader(status)
_, _ = w.Write(a.body)
_, _ = w.Write(answer.body)
// Streamed, as gRPC servers answer: Go drops the trailer of a
// response it can give a content-length.
w.(http.Flusher).Flush()
if trailer {
w.Header().Set("Grpc-Status", a.grpc)
w.Header().Set("Grpc-Status", answer.grpc)
}
}))
srv.EnableHTTP2 = true
@ -633,3 +684,134 @@ func TestCacheZstdProbe(t *testing.T) {
}
})
}
// TestCacheRRCProbe runs cache-rrc-probe.sh against a stand-in rbe-cache:
// only a gRPC status 0 GetCapabilities answer with cache_capabilities, then
// NOT_FOUND for the probe's GetActionResult key and ByteStream Read blob,
// passes; every error, refusal, timeout or unreadable answer fails, and so
// does a copy with the kill switch off, without asking rbe-cache at all. The probe never writes
// stdout and always says why on stderr, with the TCP connect time (without
// the DNS lookup) once it has one.
func TestCacheRRCProbe(t *testing.T) {
bash := requireHostTool(t, "bash")
requireCacheZstdProbeTools(t)
root := sourceRepoRoot(t)
probeText := readPolicyFile(t, root, cacheRRCProbe)
for _, want := range []string{
"\n" + cacheRRCKillLine + "\n",
"url=${RBE_CACHE_PROBE_URL:-https://rbe-cache.ops.gascity.com:8443}\n",
`printf '\000\000\000\000\005\012\003oss'`,
"--connect-timeout 3 --max-time \"$max_time\"",
"max_time=${RBE_CACHE_PROBE_MAX_TIME:-5}\n",
} {
if strings.Count(probeText, want) != 1 {
t.Errorf("%s lacks %q (once)", cacheRRCProbe, want)
}
}
switchedOff := filepath.Join(t.TempDir(), "cache-rrc-probe.sh")
if err := os.WriteFile(switchedOff, []byte(strings.Replace(probeText, "\n"+cacheRRCKillLine+"\n", "\nfork_rrc_read=off\n", 1)), 0o644); err != nil {
t.Fatal(err)
}
run := func(t *testing.T, script string, env ...string) (bool, string, string) {
t.Helper()
cmd := exec.Command(bash, script)
cmd.Dir = t.TempDir()
cmd.Env = append([]string{"PATH=" + os.Getenv("PATH"), "RBE_CACHE_PROBE_MAX_TIME=1"}, env...)
var stdout, stderr bytes.Buffer
cmd.Stdout, cmd.Stderr = &stdout, &stderr
err := cmd.Run()
var exit *exec.ExitError
if err != nil && !errors.As(err, &exit) {
t.Fatalf("run %s: %v", script, err)
}
return err == nil, stdout.String(), stderr.String()
}
check := func(t *testing.T, ok bool, stdout, stderr string, want bool) {
t.Helper()
if ok != want {
t.Errorf("probe passed: %v, want %v; stderr:\n%s", ok, want, stderr)
}
if stdout != "" {
t.Errorf("probe wrote stdout %q; write-bazelrc.sh would mistake it for its reason", stdout)
}
verdict := "; this lane fetches external repositories itself\n"
if want {
verdict = "; this lane reads the remote repo contents cache\n"
}
if !strings.HasPrefix(stderr, "rbe-cache rrc probe: ") || !strings.HasSuffix(stderr, verdict) || strings.Count(stderr, "\n") != 1 {
t.Errorf("probe stderr %q; want one verdict line ending %q", stderr, verdict)
}
}
script := filepath.Join(root, cacheRRCProbe)
live := grpcMessage(capsZstd)
acWith := func(a capsAnswer) capsAnswer { return capsAnswer{grpc: "0", body: live, ac: &a} }
bsWith := func(a capsAnswer) capsAnswer { return capsAnswer{grpc: "0", body: live, bs: &a} }
for _, c := range []struct {
name string
answer capsAnswer
want bool
asks int32 // GetCapabilities, GetActionResult, ByteStream Read: each only after a good answer
}{
{"rbe-cache today", capsAnswer{grpc: "0", body: live}, true, 3},
{"rbe-cache before zstd", capsAnswer{grpc: "0", body: grpcMessage(capsLive)}, true, 3},
{"no cache_capabilities", capsAnswer{grpc: "0", body: grpcMessage(capsLiveAPIVersions)}, false, 1},
{"gRPC error after the answer", capsAnswer{grpc: "13", body: live}, false, 1},
{"no grpc-status", capsAnswer{body: live}, false, 1},
{"trailers-only PERMISSION_DENIED", capsAnswer{grpc: "7"}, false, 1},
{"HTTP 502", capsAnswer{status: http.StatusBadGateway, grpc: "0", body: live}, false, 1},
{"compressed message", capsAnswer{grpc: "0", body: append([]byte{1}, live[1:]...)}, false, 1},
{"truncated message", capsAnswer{grpc: "0", body: live[:len(live)-1]}, false, 1},
{"not gRPC", capsAnswer{grpc: "0", body: []byte("<html>bad gateway</html>")}, false, 1},
{"timeout", capsAnswer{grpc: "0", body: live, delay: 10 * time.Second}, false, 1},
{"action cache answers the key", acWith(capsAnswer{grpc: "0", body: grpcMessage(nil)}), false, 2},
{"action cache PERMISSION_DENIED", acWith(capsAnswer{grpc: "7"}), false, 2},
{"action cache UNAVAILABLE", acWith(capsAnswer{grpc: "14"}), false, 2},
{"action cache no grpc-status", acWith(capsAnswer{body: []byte{}}), false, 2},
{"action cache HTTP 502", acWith(capsAnswer{status: http.StatusBadGateway, grpc: "5"}), false, 2},
{"action cache timeout", acWith(capsAnswer{grpc: "5", delay: 10 * time.Second}), false, 2},
{"CAS serves the blob", bsWith(capsAnswer{grpc: "0", body: grpcMessage(pbBytes(10, []byte{0}))}), false, 3},
{"CAS UNAVAILABLE", bsWith(capsAnswer{grpc: "14"}), false, 3},
{"CAS PERMISSION_DENIED", bsWith(capsAnswer{grpc: "7"}), false, 3},
{"CAS HTTP 502", bsWith(capsAnswer{status: http.StatusBadGateway, grpc: "5"}), false, 3},
{"CAS timeout", bsWith(capsAnswer{grpc: "5", delay: 10 * time.Second}), false, 3},
} {
t.Run(c.name, func(t *testing.T) {
url, ca, hits := serveCapabilities(t, c.answer)
start := time.Now()
ok, stdout, stderr := run(t, script, "RBE_CACHE_PROBE_URL="+url, "CURL_CA_BUNDLE="+ca)
check(t, ok, stdout, stderr, c.want)
if hits.Load() != c.asks {
t.Errorf("probe asked %d times, want %d", hits.Load(), c.asks)
}
if c.asks > 1 && !regexp.MustCompile(` \(tcp connect [0-9]+\.[0-9] ms\); `).MatchString(stderr) {
t.Errorf("probe stderr %q lacks the GetCapabilities TCP connect time", stderr)
}
if d := time.Since(start); d > 5*time.Second {
t.Errorf("probe took %v; RBE_CACHE_PROBE_MAX_TIME=1 bounds each call", d)
}
})
}
t.Run("refused", func(t *testing.T) {
ok, stdout, stderr := run(t, script, "RBE_CACHE_PROBE_URL="+refusedProbeURL)
check(t, ok, stdout, stderr, false)
})
t.Run("untrusted certificate", func(t *testing.T) {
url, _, hits := serveCapabilities(t, capsAnswer{grpc: "0", body: live})
ok, stdout, stderr := run(t, script, "RBE_CACHE_PROBE_URL="+url)
check(t, ok, stdout, stderr, false)
if hits.Load() != 0 {
t.Errorf("probe completed a request through an untrusted certificate")
}
})
t.Run("kill switch off", func(t *testing.T) {
url, ca, hits := serveCapabilities(t, capsAnswer{grpc: "0", body: live})
ok, stdout, stderr := run(t, switchedOff, "RBE_CACHE_PROBE_URL="+url, "CURL_CA_BUNDLE="+ca)
check(t, ok, stdout, stderr, false)
if !strings.Contains(stderr, "switched off") {
t.Errorf("switched-off probe stderr %q does not say it is switched off", stderr)
}
if hits.Load() != 0 {
t.Errorf("switched-off probe asked rbe-cache %d times, want 0", hits.Load())
}
})
}

View file

@ -174,9 +174,10 @@ func TestBazelRRCModeStep(t *testing.T) {
}
}
// TestBazelRRCReadSteps: exactly bazelRRCReadLanes read, in mode remote only
// (fork and local runs never: only rbe-west's trusted edge serves the
// entries), bazel-test under canary and every one of them under on; each
// TestBazelRRCReadSteps: exactly bazelRRCReadLanes read through these steps,
// in mode remote only (fork modes and local never; mode cache reads through
// setup-bazel's rc instead, behind cache-rrc-probe.sh: TestSetupBazelRCWriter),
// bazel-test under canary and every one of them under on; each
// writes exactly bazelRRCReadLines, never mentions uploads, and comes before
// the job's Set up Bazel (so no Bazel server starts without it).
func TestBazelRRCReadSteps(t *testing.T) {
@ -242,10 +243,60 @@ func bazelRRCLaneCommands(t *testing.T, root string) []string {
return cmds
}
// bazelRRCEnablingJobs: bazel.yml's jobs that enable the remote repo
// contents cache flag in any mode: a trusted reader step (mode remote) or
// setup-bazel's BAZEL_RRC_READ opt-in (mode cache, write-bazelrc.sh), with
// the steps that do it.
func bazelRRCEnablingJobs(t *testing.T) map[string][]string {
t.Helper()
jobs := map[string][]string{}
for name, job := range readCIWorkflow(t, bazelWorkflowName).Jobs {
if isBazelRRCJob(name) {
continue // the writer and its check set the flag themselves
}
for _, s := range job.Steps {
if s.Name == bazelRRCReadStep {
jobs[name] = append(jobs[name], s.Name)
}
if s.Uses == bazelSetupBazelUses && s.Env["BAZEL_RRC_READ"] != "" {
jobs[name] = append(jobs[name], "setup-bazel BAZEL_RRC_READ="+s.Env["BAZEL_RRC_READ"])
}
}
}
return jobs
}
// TestBazelRRCOptInMirrorsTheTrustedReaders: setup-bazel's BAZEL_RRC_READ
// (mode cache reads through rbe-cache) is exactly "true" on exactly the
// trusted readers' jobs, bazelRRCReadLanes: not on every mode cache job
// (bazel-release-cross builds for platforms rrc-lane-commands.txt does not
// seed, and the package gates are not lanes).
func TestBazelRRCOptInMirrorsTheTrustedReaders(t *testing.T) {
var optIn []string
for name, job := range readCIWorkflow(t, bazelWorkflowName).Jobs {
for _, s := range job.Steps {
v, ok := s.Env["BAZEL_RRC_READ"]
if !ok {
continue
}
if s.Uses != bazelSetupBazelUses || v != "true" {
t.Errorf("%s step %q sets BAZEL_RRC_READ=%q; only setup-bazel takes it, as \"true\"", name, s.Name, v)
}
optIn = append(optIn, name)
}
}
sort.Strings(optIn)
if !slices.Equal(optIn, bazelRRCReadLanes) {
t.Errorf("jobs passing BAZEL_RRC_READ %v, want the trusted readers %v", optIn, bazelRRCReadLanes)
}
}
// TestBazelRRCLaneCommandsCoverEveryReader: rrc-seed seeds (and rrc-verify
// checks) what the readers fetch, so every --config a reading lane passes to
// `bazel test` or `bazel build` has a command in rrc-lane-commands.txt, and
// every command there is a reading lane's.
// checks) what the readers fetch, so every job that enables the flag
// (bazelRRCEnablingJobs: the trusted reader step or the mode cache opt-in)
// is a reading lane, every --config it passes to `bazel test` or `bazel
// build` has a command in rrc-lane-commands.txt, and every command there is
// a reading lane's.
func TestBazelRRCLaneCommandsCoverEveryReader(t *testing.T) {
root := sourceRepoRoot(t)
seeded := map[string]bool{}
@ -264,24 +315,33 @@ func TestBazelRRCLaneCommandsCoverEveryReader(t *testing.T) {
invocation := regexp.MustCompile(`\bbazel (?:test|build)\b[^\n]*(?:\\\n[^\n]*)*`)
config := regexp.MustCompile(`--config=([a-z0-9-]+)`)
wf := readCIWorkflow(t, bazelWorkflowName)
for _, lane := range bazelRRCReadLanes {
for _, s := range wf.Jobs[lane].Steps {
enabling := bazelRRCEnablingJobs(t)
for job, how := range enabling {
if !slices.Contains(bazelRRCReadLanes, job) {
t.Errorf("%s enables the remote repo contents cache (%v) but is not a reading lane whose configs %s seeds", job, how, bazelRRCLaneCmds)
}
var configs []string
for _, s := range wf.Jobs[job].Steps {
for _, inv := range invocation.FindAllString(s.Run, -1) {
for _, m := range config.FindAllStringSubmatch(inv, -1) {
if m[1] != "sole-run" && m[1] != "fresh" { // test-result options, nothing fetched
used[m[1]] = true
configs = append(configs, m[1])
}
}
}
}
}
if len(used) < len(bazelRRCReadLanes) {
t.Fatalf("found the configs %v in %d reading lanes; the scan is broken", used, len(bazelRRCReadLanes))
}
for c := range used {
if !seeded[c] {
t.Errorf("a reading lane runs bazel with --config=%s, which %s does not seed", c, bazelRRCLaneCmds)
if len(configs) == 0 {
t.Errorf("%s enables the remote repo contents cache (%v) but no `bazel test|build --config=` is visible in its steps, so its configs cannot be checked against %s", job, how, bazelRRCLaneCmds)
}
for _, c := range configs {
used[c] = true
if !seeded[c] {
t.Errorf("%s enables the remote repo contents cache (%v) and runs bazel with --config=%s, which %s does not seed", job, how, c, bazelRRCLaneCmds)
}
}
}
if len(enabling) < len(bazelRRCReadLanes) || len(used) < len(bazelRRCReadLanes) {
t.Fatalf("found %d jobs enabling the cache and the configs %v; the scan is broken", len(enabling), used)
}
for c := range seeded {
if !used[c] {
@ -518,6 +578,25 @@ func TestBazelRRCVerifyJob(t *testing.T) {
if strings.Contains(alert.Run, "${{") {
t.Errorf("alert step interpolates an expression into its script; pass it through env")
}
// The issue's containment: mode cache lanes see no repository
// variables, so the committed switches (both repositories) and
// rbe-cache-gate stop them. The mode remote readers read the same
// entries, so a mismatch sets the variable to seed (no reader, rrc-seed
// and this check keep running); off stops seeding and this check too.
for _, want := range []string{
"fork_rrc_read=off in both probe scripts", cacheRRCProbe, "gascity tools/rbe/cache-rrc-probe.sh",
"beads has no fresh-merge", "rbe-cache-gate close", "on a mismatch, set RBE_REPO_CONTENTS_CACHE to seed",
"set it to off only to stop seeding too", "could not check, leave the variable as it is",
} {
if !strings.Contains(alert.Run, want) {
t.Errorf("alert issue body lacks %q", want)
}
}
for _, bad := range []string{"Stop readers first: set the repository variable", "leave RBE_REPO_CONTENTS_CACHE on"} {
if strings.Contains(alert.Run, bad) {
t.Errorf("alert issue body still says %q; a mismatch sets the variable to seed", bad)
}
}
}
// permissionRank orders GitHub token permission levels so a grant can be

View file

@ -3,6 +3,7 @@ package scripts_test
import (
"fmt"
"math"
"slices"
"strconv"
"strings"
"testing"
@ -63,6 +64,24 @@ func bazelRemoteRunsOn(label string) string {
return "${{ needs.rbe.outputs.mode == 'remote' && '" + label + "' || 'ubuntu-latest' }}"
}
// bazelRRCCacheReaderRunsOn is bazelRemoteRunsOn for the lanes that read the
// remote repo contents cache in mode cache (isBazelRRCCacheReader): their
// GitHub-hosted fallback is ubuntu-24.04, not ubuntu-latest, because the
// cached trees come from a 24.04 host (design R2) and the image must not
// move under them.
func bazelRRCCacheReaderRunsOn(label string) string {
return "${{ needs.rbe.outputs.mode == 'remote' && '" + label + "' || 'ubuntu-24.04' }}"
}
// isBazelRRCCacheReader: a remote repo contents cache reader
// (bazelRRCReadLanes) that also runs in mode cache (not remote-only).
// bazel-dolt-race runs only in mode remote on PRs and merge groups (its own
// if, TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls), so it keeps the
// lanes' ubuntu-latest fallback like the remote-only jobs.
func isBazelRRCCacheReader(name string) bool {
return slices.Contains(bazelRRCReadLanes, name) && !bazelRemoteOnlyJobs[name] && name != bazelDoltRaceJobName
}
// The default lane size: every action executes on rbe-west, but the
// client's loading and analysis is CPU-bound and gates every action
// (ga-vnycm2.8: 17-38 s on 2 vCPU, 12-26 s on 4), so 4 vCPU.
@ -584,6 +603,9 @@ func TestBlacksmithBazelRunnerSizes(t *testing.T) {
continue // rrc jobs: push/schedule only, never a fork (literal label)
}
want := "ubuntu-latest"
if isBazelRRCCacheReader(name) {
want = "ubuntu-24.04"
}
if mode == "remote" {
want = "blacksmith-4vcpu-ubuntu-2404"
if label, ok := wantRemote[name]; ok {

View file

@ -163,9 +163,12 @@ func TestReleaseCrossCompileRunsInItsOwnBazelLane(t *testing.T) {
}
cross := bazel.job(t, bazelReleaseCrossJobName)
pure := bazel.job(t, bazelPureJobName)
if cross.RunsOn != pure.RunsOn || cross.If != pure.If || !slices.Equal(cross.Needs, pure.Needs) {
t.Errorf("%s runs-on/if/needs = %q / %q / %v, want %s's %q / %q / %v",
bazelReleaseCrossJobName, cross.RunsOn, cross.If, cross.Needs, bazelPureJobName, pure.RunsOn, pure.If, pure.Needs)
// Not pure's runs-on: pure reads the remote repo contents cache in mode
// cache (ubuntu-24.04, bazelRRCCacheReaderRunsOn), the release lane does
// not (its target platforms are not seeded).
if cross.RunsOn != bazelLaneRunsOn || cross.If != pure.If || !slices.Equal(cross.Needs, pure.Needs) {
t.Errorf("%s runs-on/if/needs = %q / %q / %v, want %q / %s's %q / %v",
bazelReleaseCrossJobName, cross.RunsOn, cross.If, cross.Needs, bazelLaneRunsOn, bazelPureJobName, pure.If, pure.Needs)
}
run := cross.step(t, stepName).Run
for _, required := range []string{
@ -2164,6 +2167,19 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) {
if name == bazelJobName {
wantJobRunsOn = bazelTestLaneRunsOn
}
// The remote repo contents cache readers opt in to reading it in
// mode cache (BAZEL_RRC_READ, bazel_rrc_test.go); those that run
// there fall back to ubuntu-24.04.
if slices.Contains(bazelRRCReadLanes, name) {
wantJobSetupEnv = copyMap(wantSetupEnv)
wantJobSetupEnv["BAZEL_RRC_READ"] = "true"
}
if isBazelRRCCacheReader(name) {
wantJobRunsOn = bazelRRCCacheReaderRunsOn("blacksmith-4vcpu-ubuntu-2404")
if name == bazelJobName {
wantJobRunsOn = bazelRRCCacheReaderRunsOn("blacksmith-8vcpu-ubuntu-2404")
}
}
if bazelPackageJobs[name] {
wantJobRunsOn, wantJobIf, wantJobSetupEnv = bazelPackageRunsOn[name], bazelPackageGatesIf, wantPackageSetupEnv
} else if bazelRemoteOnlyJobs[name] {
@ -5223,6 +5239,83 @@ func TestSetupBazelRCWriter(t *testing.T) {
}
}
})
// The fork cache of a job that opts in (BAZEL_RRC_READ=true: the remote
// repo contents cache readers, bazel_rrc_test.go) reads the remote repo
// contents cache exactly while rbe-cache answers (cache-rrc-probe.sh
// against a stand-in; TestCacheRRCProbe runs every other answer and the
// kill switch): the reader's two lines, after --config=fork-cache and
// any zstd line, and nothing else; otherwise none, with a notice saying
// why. A job that does not opt in never probes. The step summary says
// which, and why. The lane never fails on the probe.
t.Run("fork cache repo contents cache follows rbe-cache", func(t *testing.T) {
requireCacheZstdProbeTools(t)
for answer, wantRead := range map[string]bool{"answers": true, "zstd": true, "denied": false, "refused": false, "not a reader": false} {
summaryFile := filepath.Join(t.TempDir(), "summary.md")
env := []string{"BAZEL_FORK_CACHE=true", "BAZEL_RRC_READ=true", "GITHUB_STEP_SUMMARY=" + summaryFile}
switch answer {
case "answers":
url, ca, _ := serveCapabilities(t, capsAnswer{grpc: "0", body: grpcMessage(capsLive)})
env = append(env, "RBE_CACHE_PROBE_URL="+url, "CURL_CA_BUNDLE="+ca)
case "not a reader":
url, ca, _ := serveCapabilities(t, capsAnswer{grpc: "0", body: grpcMessage(capsLive)})
env = []string{"BAZEL_FORK_CACHE=true", "GITHUB_STEP_SUMMARY=" + summaryFile, "RBE_CACHE_PROBE_URL=" + url, "CURL_CA_BUNDLE=" + ca}
case "zstd":
url, ca, _ := serveCapabilities(t, capsAnswer{grpc: "0", body: grpcMessage(capsZstd)})
env = append(env, "RBE_CACHE_PROBE_URL="+url, "CURL_CA_BUNDLE="+ca)
case "denied":
url, ca, _ := serveCapabilities(t, capsAnswer{grpc: "7"})
env = append(env, "RBE_CACHE_PROBE_URL="+url, "CURL_CA_BUNDLE="+ca)
}
outputs, rc, logs, err := run(t, env...)
if err != nil {
t.Fatalf("%s: err=%v\n%s", answer, err, logs)
}
_, after, ok := strings.Cut(rc, "\nbuild --config=fork-cache\n")
if !ok {
t.Fatalf("%s: rc lacks build --config=fork-cache:\n%s", answer, rc)
}
var want, got []string
if answer == "zstd" {
want = append(want, forkCacheZstdLine)
}
if wantRead {
want = append(want, bazelRRCReadLines...)
}
if after = strings.TrimSpace(after); after != "" {
got = strings.Split(after, "\n")
}
if !slices.Equal(got, want) {
t.Errorf("rbe-cache %s: rc after --config=fork-cache %q, want exactly %q", answer, got, want)
}
for _, line := range bazelRRCReadLines {
if strings.Count(rc, line) != map[bool]int{true: 1}[wantRead] {
t.Errorf("rbe-cache %s: rc has %q %d times, want %v", answer, line, strings.Count(rc, line), wantRead)
}
}
if !strings.Contains(outputs, "cache=true") || strings.Contains(rc, "remote-exec") || strings.Contains(rc, "--remote_upload_local_results") {
t.Errorf("rbe-cache %s: outputs = %q rc = %q; want cache=true, no remote-exec and no upload", answer, outputs, rc)
}
probed := answer != "not a reader"
notice := "::notice title=No remote repo contents cache::rbe-cache rrc probe: "
if got := strings.Contains(logs, notice); got != (probed && !wantRead) {
t.Errorf("rbe-cache %s: notice %v, want %v (the probe's reason, only when it fails):\n%s", answer, got, probed && !wantRead, logs)
}
if got := strings.Contains(logs, "rbe-cache rrc probe: "); got != probed {
t.Errorf("rbe-cache %s: log has the probe's verdict %v, want %v:\n%s", answer, got, probed, logs)
}
summary, _ := os.ReadFile(summaryFile)
wantSummary := map[string]string{
"answers": "**on** (rbe-cache answers GetCapabilities, GetActionResult and ByteStream Read (tcp connect ",
"zstd": "**on** (rbe-cache answers GetCapabilities, GetActionResult and ByteStream Read (tcp connect ",
"denied": "**off** (gRPC status 7 (tcp connect ",
"refused": "**off** (",
"not a reader": "**off** (not a remote repo contents cache reader (no BAZEL_RRC_READ))\n",
}[answer]
if !strings.HasPrefix(string(summary), "Fork cache remote repo contents cache read: "+wantSummary) || strings.Count(string(summary), "\n") != 1 {
t.Errorf("rbe-cache %s: step summary %q, want one line starting %q", answer, summary, wantSummary)
}
}
})
t.Run("remote-exec and rbe-fork never ask for zstd", func(t *testing.T) {
requireCacheZstdProbeTools(t)
url, ca, hits := serveCapabilities(t, capsAnswer{grpc: "0", body: grpcMessage(capsZstd)})
@ -5239,6 +5332,13 @@ func TestSetupBazelRCWriter(t *testing.T) {
if strings.Contains(rc, "remote_cache_compression") {
t.Errorf("%s rc asks for compression; rbe-west's schedulers and rbe-fork advertise none:\n%s", name, rc)
}
// Their repo contents cache readers are bazel.yml's own steps
// (mode remote, TestBazelRRCReadSteps); never this rc.
for _, line := range bazelRRCReadLines {
if strings.Contains(rc, line) {
t.Errorf("%s rc carries %q; only the fork cache's rc reads through rbe-cache:\n%s", name, line, rc)
}
}
}
if n := hits.Load(); n != 0 {
t.Errorf("remote-exec or rbe-fork probed rbe-cache %d times; only the fork cache may", n)
@ -5259,6 +5359,11 @@ func TestSetupBazelRCWriter(t *testing.T) {
t.Errorf("want failure, got success:\n%s", logs)
}
})
t.Run("rejects BAZEL_RRC_READ="+bad, func(t *testing.T) {
if _, _, logs, err := run(t, "BAZEL_FORK_CACHE=true", "BAZEL_RRC_READ="+bad); err == nil || !strings.Contains(logs, "BAZEL_RRC_READ must be") {
t.Errorf("want failure naming BAZEL_RRC_READ, got %v:\n%s", err, logs)
}
})
}
for name, env := range map[string][]string{
"executor only": {executor},