* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2) In mode remote, folds bazel-server-storage (--config=doltserver-integration) and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job (--config=dolt-int), following the dolt-race pattern from #7514. The members have different gating categories (server-storage plain-gated, cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true and a bespoke --label-scoped Shard balance step. All other modes keep the two lanes running unchanged. Action keys are byte-identical between the member configs and the union (17 + 46 = 63 TestRunner actions), so cached results carry over either direction and a revert is a safe rollback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review) Coordinator review of #7535 found three must-fixes: integration:"off" was ignored in remote mode (the union's members already honor it); job-level continue-on-error made a server-storage-only failure read success on push/nightly/dispatch, where main stays red; and the PR's key-parity claim overstated byte-identical actionKeys when only the remote action digest (not the local actionKey) is unaffected by moving --test_env/--test_arg onto the targets' own env/args. Fixes: - bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching the members (#7491's review); bazelLaneRunModes and TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case. - Job-level continue-on-error removed; only the `bazel test` step carries it, and a new always() "Classify tier results" step (tools/bazel/classify_tier_results.py, with its own unit tests and a Go wrapper) computes server_storage_result/cmd_dolt_result from the BEP and decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on synthetic empty-output inputs that a dead job still fails the gate. - Corrected the local-actionKey-vs-remote-digest wording in .bazelrc, bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md. - Should-fix: test step 23->25 / job 58 minutes with reasoning; --label coverage added to shard_budget_test.py; stale lane comment fixed in tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's docstring corrected (members bake sole-run's line in directly, never via --config=sole-run). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review) Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack, julianknutsen): - classify_tier_results.py now fails every tier shut (exit 1) on a non-PASSED label outside every --tier list, so a newly tagged target added without updating the step's --tier args cannot silently read as success (must-fix 1). Its --tier/--check label sets are policy-tied to the BUILD files' own tag sets by a new TestBazelDoltIntTierLabelsMatchTheTags. - A missing or non-integer --rc (a timed-out or cancelled test step) takes the same fail-everything-shut branch instead of crashing argparse with no outputs written (bee-ghosttrack's should-fix 2). - check_testcases.py gained a repeatable --label flag, and bazel.yml's shared "Every target and shard ran tests" step is split into check-ss and check-cd, each continue-on-error and folded into its own tier via classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard no longer reddens the job on push/nightly (must-fix 2). - classify's run: argv is now pinned by exact equality (bazelDoltIntClassifyRun), and every flag except --tier/--check is --StoreOnce; a repeated --tier/--check name or an unrecognized flag is now a hard parser error (must-fix 3). - "Upload test logs" now also fires on steps.test.outcome == 'failure', not just failure() (must-fix 4). - shard_budget.py --label now fails closed (::error, exit 1) on a label with no executed shard in the BEP at all, instead of silently reporting "no sharded test target executed two or more shards" (should-fix 3, carried across both review rounds). - Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the nits: stale "Advisory (continue-on-error: true)" wording, the bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int, "byte-identical" without specifying remote action digest vs. local actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch for an empty union output. Verified: python3 -I on all four tools/bazel/*_test.py (classify 29, check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test ./scripts/... green; bazel --nohome_rc test //scripts:scripts_test --config=ci green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): shard_budget --label fail-closed only on a truly unknown label The previous commit's fail-closed --label check compared against read_shard_durations, which only records executed (non-cached) shards, so a target whose every shard was a cache hit this run had zero entries and was wrongly treated the same as a --label typo. This was not caught by the new tests (none exercised an all-cached target) and broke this PR's own dolt-int CI run: bazel-dolt-int's Shard balance step went red on //internal/storage/dolt:dolt_server_full_test with "no executed shard for it", even though that target was correctly, fully cache-hit. Fix: a new read_tested_labels() reads every label with any testResult event (cached or executed), and --label's fail-closed check now tests against that set instead of read_shard_durations. A fully-cached target still legitimately reports "nothing to judge" through evaluate(); only a label absent from the BEP altogether is now an error. Added the missing test row (every shard cached) that would have caught this before push. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * test(bazel): pin bazel-dolt-int's Upload test logs if-condition A full 42-mutation re-run for PR #7535's second review round found that nothing pinned this job's own "Upload test logs" if-condition (must-fix 4): the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's (unlike dolt-race, a brand-new job) had no equivalent assertion, so a future accidental revert to the old failure()-only condition would go uncaught. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
871 lines
36 KiB
Go
871 lines
36 KiB
Go
package scripts_test
|
|
|
|
import (
|
|
"crypto/sha256"
|
|
"encoding/hex"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"reflect"
|
|
"regexp"
|
|
"runtime"
|
|
"slices"
|
|
"sort"
|
|
"strings"
|
|
"testing"
|
|
|
|
"gopkg.in/yaml.v3"
|
|
)
|
|
|
|
// Bazel's remote repo contents cache on rbe-west (ga-vnycm2.29, ported from
|
|
// gascity's bazel.yml, gascity #7467 and #7478). bazel.yml's rbe job turns
|
|
// vars.RBE_REPO_CONTENTS_CACHE into its rrc output (off, seed, canary, on);
|
|
// each reading lane's reader step appends bazelRRCReadLines to
|
|
// .bazelrc.local; the rrc-seed job, on push to main only, runs every lane's
|
|
// loading and analysis (.github/scripts/rrc-lane-commands.txt, through
|
|
// rrc-run-lanes.sh) and uploads repository trees with a 30-minute
|
|
// rbe-rrc-writer-beads certificate (rrc-writer-credential.sh); the
|
|
// rrc-verify job, nightly, compares the cache with a cold fetch
|
|
// (tools/bazel/rrc_verify.py).
|
|
|
|
const (
|
|
bazelRRCReadStep = "Remote repo contents cache (read)"
|
|
bazelRRCModeStepID = "rrc"
|
|
bazelRRCVar = "vars.RBE_REPO_CONTENTS_CACHE"
|
|
bazelRRCSeedJob = "rrc-seed"
|
|
bazelRRCSeedJobIf = "${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }}"
|
|
bazelRRCVerifyJob = "rrc-verify"
|
|
bazelRRCVerifyJobIf = "${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }}"
|
|
bazelRRCCanaryLane = "bazel-test"
|
|
bazelRRCCredential = ".github/scripts/rrc-writer-credential.sh"
|
|
bazelRRCRunLanes = ".github/scripts/rrc-run-lanes.sh"
|
|
bazelRRCLaneCmds = ".github/scripts/rrc-lane-commands.txt"
|
|
bazelRRCVerifyTool = "tools/bazel/rrc_verify.py"
|
|
bazelRRCVerifyTest = "tools/bazel/rrc_verify_test.py"
|
|
bazelSetupBazelUses = "./.github/actions/setup-bazel"
|
|
)
|
|
|
|
// bazelRRCReadLanes: the lanes that read the cache under `on` (bazel-test
|
|
// alone under `canary`). Not bazel-release-cross (other target platforms,
|
|
// not seeded) nor the package gates.
|
|
var bazelRRCReadLanes = []string{
|
|
"bazel-cmd-dolt", "bazel-dolt-int", "bazel-dolt-race", "bazel-doltserver",
|
|
"bazel-embedded", "bazel-integration", "bazel-proxied", "bazel-pure",
|
|
"bazel-server-storage", "bazel-test",
|
|
}
|
|
|
|
// bazelRRCReadLines: the reader's .bazelrc.local lines; both are key neutral
|
|
// (a startup option, and loading parallelism).
|
|
var bazelRRCReadLines = []string{
|
|
"startup --experimental_remote_repo_contents_cache",
|
|
"common --loading_phase_threads=64",
|
|
}
|
|
|
|
// bazelRRCJobs: bazel.yml's jobs that are not lanes (no result output, never
|
|
// gated): the remote repo contents cache's writer and its nightly check.
|
|
var bazelRRCJobs = []string{bazelRRCSeedJob, bazelRRCVerifyJob}
|
|
|
|
// isBazelRRCJob reports whether a bazel.yml job is one of bazelRRCJobs.
|
|
func isBazelRRCJob(name string) bool { return slices.Contains(bazelRRCJobs, name) }
|
|
|
|
// evalRRCIf evaluates a job or step if: with ctx, failing the test on an
|
|
// expression the evaluator cannot read.
|
|
func evalRRCIf(t *testing.T, expr string, ctx map[string]string) bool {
|
|
t.Helper()
|
|
v, err := evalGHExpr(expr, ctx)
|
|
if err != nil {
|
|
t.Fatalf("evalGHExpr(%q): %v", expr, err)
|
|
}
|
|
return ghTruthy(v)
|
|
}
|
|
|
|
// runRRCScript runs a step's script (or a script file) under GitHub's bash
|
|
// flags in dir with env, returning its combined output.
|
|
func runRRCScript(t *testing.T, dir, script string, env map[string]string) (string, error) {
|
|
t.Helper()
|
|
cmd := exec.Command("bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", script)
|
|
cmd.Dir = dir
|
|
cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "HOME=" + dir}
|
|
for k, v := range env {
|
|
cmd.Env = append(cmd.Env, k+"="+v)
|
|
}
|
|
out, err := cmd.CombinedOutput()
|
|
return string(out), err
|
|
}
|
|
|
|
// rrcOutputs reads a $GITHUB_OUTPUT file (key=value lines).
|
|
func rrcOutputs(t *testing.T, path string) map[string]string {
|
|
t.Helper()
|
|
out := map[string]string{}
|
|
data, err := os.ReadFile(path)
|
|
if os.IsNotExist(err) {
|
|
return out
|
|
}
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") {
|
|
if k, v, ok := strings.Cut(line, "="); ok {
|
|
out[k] = v
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
func rrcStep(t *testing.T, job ciWorkflowJob, what string, pick func(ciWorkflowStep) bool) ciWorkflowStep {
|
|
t.Helper()
|
|
var found []ciWorkflowStep
|
|
for _, s := range job.Steps {
|
|
if pick(s) {
|
|
found = append(found, s)
|
|
}
|
|
}
|
|
if len(found) != 1 {
|
|
t.Fatalf("want one %s step, got %d", what, len(found))
|
|
}
|
|
return found[0]
|
|
}
|
|
|
|
// TestBazelRRCModeStep runs the rbe job's rrc step for every variable value
|
|
// and execution mode: only mode remote enables anything, unset and off are
|
|
// off, and an unknown value is off with a warning (a typo never fails the
|
|
// required gate). It reads no secret and runs before the checkout.
|
|
func TestBazelRRCModeStep(t *testing.T) {
|
|
rbe := readCIWorkflow(t, bazelWorkflowName).Jobs[bazelRBEJobName]
|
|
if got := rbe.Outputs["rrc"]; got != "${{ steps.rrc.outputs.rrc }}" {
|
|
t.Errorf("rbe job output rrc = %q", got)
|
|
}
|
|
step := rrcStep(t, rbe, "rrc", func(s ciWorkflowStep) bool { return s.ID == bazelRRCModeStepID })
|
|
if want := map[string]string{"RRC": "${{ " + bazelRRCVar + " }}", "MODE": "${{ steps.decide.outputs.mode }}"}; !reflect.DeepEqual(step.Env, want) {
|
|
t.Errorf("rrc step env %v, want %v", step.Env, want)
|
|
}
|
|
if step.If != "" {
|
|
t.Errorf("rrc step if %q; it must always set the output", step.If)
|
|
}
|
|
for i, s := range rbe.Steps {
|
|
if s.Uses != "" {
|
|
t.Errorf("rbe step %d (%s) precedes nothing it should: the rrc step must come before any checkout", i, s.Uses)
|
|
}
|
|
if s.ID == bazelRRCModeStepID {
|
|
break
|
|
}
|
|
}
|
|
for _, mode := range []string{"remote", "fork-ro", "fork-rw", "cache", "local", "skip"} {
|
|
for value, want := range map[string]string{"": "off", "off": "off", "seed": "seed", "canary": "canary", "on": "on", "On": "off", "true": "off", "on;x": "off"} {
|
|
if mode != "remote" {
|
|
want = "off"
|
|
}
|
|
dir := t.TempDir()
|
|
output := filepath.Join(dir, "output")
|
|
out, err := runRRCScript(t, dir, step.Run, map[string]string{
|
|
"RRC": value, "MODE": mode, "GITHUB_OUTPUT": output, "GITHUB_STEP_SUMMARY": filepath.Join(dir, "summary"),
|
|
})
|
|
if err != nil {
|
|
t.Errorf("mode %s value %q: %v\n%s", mode, value, err, out)
|
|
continue
|
|
}
|
|
if got := rrcOutputs(t, output)["rrc"]; got != want {
|
|
t.Errorf("mode %s value %q: rrc=%q, want %q", mode, value, got, want)
|
|
}
|
|
unknown := !slices.Contains([]string{"", "off", "seed", "canary", "on"}, value)
|
|
if warned := strings.Contains(out, "::warning"); warned != unknown {
|
|
t.Errorf("mode %s value %q: warning %v, want %v:\n%s", mode, value, warned, unknown, out)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestBazelRRCReadSteps: exactly bazelRRCReadLanes read through these steps,
|
|
// in mode remote only (fork modes and local never; mode cache reads through
|
|
// setup-bazel's rc instead, behind cache-rrc-probe.sh: TestSetupBazelRCWriter),
|
|
// bazel-test under canary and every one of them under on; each
|
|
// writes exactly bazelRRCReadLines, never mentions uploads, and comes before
|
|
// the job's Set up Bazel (so no Bazel server starts without it).
|
|
func TestBazelRRCReadSteps(t *testing.T) {
|
|
wf := readCIWorkflow(t, bazelWorkflowName)
|
|
var readers []string
|
|
for name, job := range wf.Jobs {
|
|
read, setup := -1, -1
|
|
for i, s := range job.Steps {
|
|
switch {
|
|
case s.Name == bazelRRCReadStep:
|
|
if read >= 0 {
|
|
t.Errorf("%s has two %q steps", name, bazelRRCReadStep)
|
|
}
|
|
read = i
|
|
case s.Uses == bazelSetupBazelUses && setup < 0:
|
|
setup = i
|
|
}
|
|
}
|
|
if read < 0 {
|
|
continue
|
|
}
|
|
readers = append(readers, name)
|
|
if setup < 0 || read > setup {
|
|
t.Errorf("%s: %q at step %d, Set up Bazel at %d; the reader must come first", name, bazelRRCReadStep, read, setup)
|
|
}
|
|
step := job.Steps[read]
|
|
for _, mode := range []string{"remote", "fork-ro", "fork-rw", "cache", "local"} {
|
|
for _, rrc := range []string{"off", "seed", "canary", "on"} {
|
|
want := mode == "remote" && (rrc == "on" || (rrc == "canary" && name == bazelRRCCanaryLane))
|
|
if got := evalRRCIf(t, step.If, map[string]string{"needs.rbe.outputs.mode": mode, "needs.rbe.outputs.rrc": rrc}); got != want {
|
|
t.Errorf("%s mode %s rrc %s: reader runs %v, want %v", name, mode, rrc, got, want)
|
|
}
|
|
}
|
|
}
|
|
dir := t.TempDir()
|
|
if out, err := runRRCScript(t, dir, step.Run, nil); err != nil {
|
|
t.Fatalf("%s reader: %v\n%s", name, err, out)
|
|
}
|
|
got := strings.Split(strings.TrimSpace(readPolicyFile(t, dir, ".bazelrc.local")), "\n")
|
|
if !slices.Equal(got, bazelRRCReadLines) {
|
|
t.Errorf("%s reader writes %q, want %q", name, got, bazelRRCReadLines)
|
|
}
|
|
if strings.Contains(step.Run, "upload") || len(step.Env) != 0 {
|
|
t.Errorf("%s reader mentions uploads or takes env; lanes only read", name)
|
|
}
|
|
}
|
|
sort.Strings(readers)
|
|
if !slices.Equal(readers, bazelRRCReadLanes) {
|
|
t.Errorf("lanes with a reader step %v, want %v", readers, bazelRRCReadLanes)
|
|
}
|
|
}
|
|
|
|
// bazelRRCLaneCommands parses rrc-lane-commands.txt as rrc-run-lanes.sh does.
|
|
func bazelRRCLaneCommands(t *testing.T, root string) []string {
|
|
t.Helper()
|
|
var cmds []string
|
|
for _, line := range strings.Split(readPolicyFile(t, root, bazelRRCLaneCmds), "\n") {
|
|
line, _, _ = strings.Cut(line, "#")
|
|
if line = strings.TrimSpace(line); line != "" {
|
|
cmds = append(cmds, line)
|
|
}
|
|
}
|
|
return cmds
|
|
}
|
|
|
|
// bazelRRCEnablingJobs: bazel.yml's jobs that enable the remote repo
|
|
// contents cache flag in any mode: a trusted reader step (mode remote) or
|
|
// setup-bazel's BAZEL_RRC_READ opt-in (mode cache, write-bazelrc.sh), with
|
|
// the steps that do it.
|
|
func bazelRRCEnablingJobs(t *testing.T) map[string][]string {
|
|
t.Helper()
|
|
jobs := map[string][]string{}
|
|
for name, job := range readCIWorkflow(t, bazelWorkflowName).Jobs {
|
|
if isBazelRRCJob(name) {
|
|
continue // the writer and its check set the flag themselves
|
|
}
|
|
for _, s := range job.Steps {
|
|
if s.Name == bazelRRCReadStep {
|
|
jobs[name] = append(jobs[name], s.Name)
|
|
}
|
|
if s.Uses == bazelSetupBazelUses && s.Env["BAZEL_RRC_READ"] != "" {
|
|
jobs[name] = append(jobs[name], "setup-bazel BAZEL_RRC_READ="+s.Env["BAZEL_RRC_READ"])
|
|
}
|
|
}
|
|
}
|
|
return jobs
|
|
}
|
|
|
|
// TestBazelRRCOptInMirrorsTheTrustedReaders: setup-bazel's BAZEL_RRC_READ
|
|
// (mode cache reads through rbe-cache) is exactly "true" on exactly the
|
|
// trusted readers' jobs, bazelRRCReadLanes: not on every mode cache job
|
|
// (bazel-release-cross builds for platforms rrc-lane-commands.txt does not
|
|
// seed, and the package gates are not lanes).
|
|
func TestBazelRRCOptInMirrorsTheTrustedReaders(t *testing.T) {
|
|
var optIn []string
|
|
for name, job := range readCIWorkflow(t, bazelWorkflowName).Jobs {
|
|
for _, s := range job.Steps {
|
|
v, ok := s.Env["BAZEL_RRC_READ"]
|
|
if !ok {
|
|
continue
|
|
}
|
|
if s.Uses != bazelSetupBazelUses || v != "true" {
|
|
t.Errorf("%s step %q sets BAZEL_RRC_READ=%q; only setup-bazel takes it, as \"true\"", name, s.Name, v)
|
|
}
|
|
optIn = append(optIn, name)
|
|
}
|
|
}
|
|
sort.Strings(optIn)
|
|
if !slices.Equal(optIn, bazelRRCReadLanes) {
|
|
t.Errorf("jobs passing BAZEL_RRC_READ %v, want the trusted readers %v", optIn, bazelRRCReadLanes)
|
|
}
|
|
}
|
|
|
|
// TestBazelRRCLaneCommandsCoverEveryReader: rrc-seed seeds (and rrc-verify
|
|
// checks) what the readers fetch, so every job that enables the flag
|
|
// (bazelRRCEnablingJobs: the trusted reader step or the mode cache opt-in)
|
|
// is a reading lane, every --config it passes to `bazel test` or `bazel
|
|
// build` has a command in rrc-lane-commands.txt, and every command there is
|
|
// a reading lane's.
|
|
func TestBazelRRCLaneCommandsCoverEveryReader(t *testing.T) {
|
|
root := sourceRepoRoot(t)
|
|
seeded := map[string]bool{}
|
|
for _, cmd := range bazelRRCLaneCommands(t, root) {
|
|
f := strings.Fields(cmd)
|
|
if f[0] != "test" && f[0] != "build" {
|
|
t.Errorf("lane command %q: want a test or build command", cmd)
|
|
}
|
|
for _, a := range f[1:] {
|
|
if c, ok := strings.CutPrefix(a, "--config="); ok {
|
|
seeded[c] = true
|
|
}
|
|
}
|
|
}
|
|
used := map[string]bool{}
|
|
invocation := regexp.MustCompile(`\bbazel (?:test|build)\b[^\n]*(?:\\\n[^\n]*)*`)
|
|
config := regexp.MustCompile(`--config=([a-z0-9-]+)`)
|
|
wf := readCIWorkflow(t, bazelWorkflowName)
|
|
enabling := bazelRRCEnablingJobs(t)
|
|
for job, how := range enabling {
|
|
if !slices.Contains(bazelRRCReadLanes, job) {
|
|
t.Errorf("%s enables the remote repo contents cache (%v) but is not a reading lane whose configs %s seeds", job, how, bazelRRCLaneCmds)
|
|
}
|
|
var configs []string
|
|
for _, s := range wf.Jobs[job].Steps {
|
|
for _, inv := range invocation.FindAllString(s.Run, -1) {
|
|
for _, m := range config.FindAllStringSubmatch(inv, -1) {
|
|
if m[1] != "sole-run" && m[1] != "fresh" { // test-result options, nothing fetched
|
|
configs = append(configs, m[1])
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if len(configs) == 0 {
|
|
t.Errorf("%s enables the remote repo contents cache (%v) but no `bazel test|build --config=` is visible in its steps, so its configs cannot be checked against %s", job, how, bazelRRCLaneCmds)
|
|
}
|
|
for _, c := range configs {
|
|
used[c] = true
|
|
if !seeded[c] {
|
|
t.Errorf("%s enables the remote repo contents cache (%v) and runs bazel with --config=%s, which %s does not seed", job, how, c, bazelRRCLaneCmds)
|
|
}
|
|
}
|
|
}
|
|
if len(enabling) < len(bazelRRCReadLanes) || len(used) < len(bazelRRCReadLanes) {
|
|
t.Fatalf("found %d jobs enabling the cache and the configs %v; the scan is broken", len(enabling), used)
|
|
}
|
|
for c := range seeded {
|
|
if !used[c] {
|
|
t.Errorf("%s seeds --config=%s, which no reading lane uses", bazelRRCLaneCmds, c)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestRRCRunLanes runs rrc-run-lanes.sh with a stub bazel: one --nobuild
|
|
// invocation per lane command, in order, startup flags first and command
|
|
// flags after; `test --nobuild`'s exit 1 after a successful analysis
|
|
// passes, a failed analysis fails, and an empty command list fails.
|
|
func TestRRCRunLanes(t *testing.T) {
|
|
// rrc-run-lanes.sh runs only in bazel.yml's rrc-seed and rrc-verify jobs,
|
|
// on Linux runners. It reads its lane list with `mapfile`, a bash 4
|
|
// builtin that macOS's /bin/bash 3.2 lacks (exit 127, "mapfile: command
|
|
// not found"); its owner keeps it Linux-only rather than porting it.
|
|
if runtime.GOOS != "linux" {
|
|
t.Skipf("rrc-run-lanes.sh is Linux-CI-only (bazel.yml rrc-seed/rrc-verify) and uses bash 4's mapfile; GOOS=%s", runtime.GOOS)
|
|
}
|
|
root := sourceRepoRoot(t)
|
|
script := filepath.Join(root, bazelRRCRunLanes)
|
|
dir := t.TempDir()
|
|
bin := filepath.Join(dir, "bin")
|
|
if err := os.MkdirAll(bin, 0o755); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
calls := filepath.Join(dir, "calls")
|
|
stub := "#!/usr/bin/env bash\nprintf '%s\\n' \"$*\" >>" + calls + "\n" +
|
|
"if [ -n \"${BAZEL_TEST_FAIL:-}\" ]; then echo 'ERROR: no such package'; echo 'ERROR: Build did NOT complete successfully'; exit 1; fi\n" +
|
|
"case \"$*\" in *' build '*) echo 'INFO: Build completed successfully, 0 total actions'; exit 0 ;; esac\n" +
|
|
"echo 'INFO: Build completed successfully, 0 total actions'\necho \"ERROR: Couldn't start the build. Unable to run tests\"\nexit 1\n"
|
|
if err := os.WriteFile(filepath.Join(bin, "bazel"), []byte(stub), 0o755); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
env := map[string]string{
|
|
"PATH": bin + string(os.PathListSeparator) + os.Getenv("PATH"),
|
|
"GITHUB_STEP_SUMMARY": filepath.Join(dir, "summary"),
|
|
}
|
|
run := "bash " + script + " --experimental_remote_repo_contents_cache -- --loading_phase_threads=64 --remote_upload_local_results"
|
|
if out, err := runRRCScript(t, dir, run, env); err != nil {
|
|
t.Fatalf("rrc-run-lanes.sh: %v\n%s", err, out)
|
|
}
|
|
var want []string
|
|
for _, cmd := range bazelRRCLaneCommands(t, root) {
|
|
want = append(want, "--experimental_remote_repo_contents_cache "+cmd+" --nobuild --loading_phase_threads=64 --remote_upload_local_results")
|
|
}
|
|
if got := strings.Split(strings.TrimSpace(readPolicyFile(t, dir, "calls")), "\n"); !slices.Equal(got, want) {
|
|
t.Errorf("bazel calls:\n%s\nwant:\n%s", strings.Join(got, "\n"), strings.Join(want, "\n"))
|
|
}
|
|
if err := os.Remove(calls); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if out, err := runRRCScript(t, dir, "bash "+script+" --", env); err != nil {
|
|
t.Fatalf("without flags: %v\n%s", err, out)
|
|
}
|
|
if got := strings.Split(strings.TrimSpace(readPolicyFile(t, dir, "calls")), "\n"); len(got) == 0 || !strings.HasPrefix(got[0], "test ") || !strings.HasSuffix(got[0], " --nobuild") {
|
|
t.Errorf("without flags, first call %q", got)
|
|
}
|
|
|
|
failEnv := map[string]string{"BAZEL_TEST_FAIL": "1"}
|
|
for k, v := range env {
|
|
failEnv[k] = v
|
|
}
|
|
if out, err := runRRCScript(t, dir, run, failEnv); err == nil {
|
|
t.Errorf("a failed analysis passed:\n%s", out)
|
|
}
|
|
empty := filepath.Join(dir, "empty.txt")
|
|
if err := os.WriteFile(empty, []byte("# nothing\n\n"), 0o644); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
emptyEnv := map[string]string{"RRC_LANE_COMMANDS": empty}
|
|
for k, v := range env {
|
|
emptyEnv[k] = v
|
|
}
|
|
if out, err := runRRCScript(t, dir, run, emptyEnv); err == nil {
|
|
t.Errorf("no lane commands passed; it must fail rather than seed nothing:\n%s", out)
|
|
}
|
|
if out, err := runRRCScript(t, dir, "bash "+script+" --flag", env); err == nil {
|
|
t.Errorf("no -- separator passed:\n%s", out)
|
|
}
|
|
}
|
|
|
|
// TestBazelRRCSeedJob: the only writer runs on push to main in mode remote
|
|
// with rrc not off, holds the only id-token permission in bazel.yml, reads
|
|
// no secret (its credential is the OIDC-minted certificate), sets Bazel up
|
|
// in local mode, seeds only with a certificate, uploads to the cache alone
|
|
// (no executor), and always removes the key.
|
|
func TestBazelRRCSeedJob(t *testing.T) {
|
|
wf := readCIWorkflow(t, bazelWorkflowName)
|
|
job, ok := wf.Jobs[bazelRRCSeedJob]
|
|
if !ok {
|
|
t.Fatalf("%s has no %s job", bazelWorkflowName, bazelRRCSeedJob)
|
|
}
|
|
if job.If != bazelRRCSeedJobIf {
|
|
t.Errorf("%s if %q, want %q", bazelRRCSeedJob, job.If, bazelRRCSeedJobIf)
|
|
}
|
|
if !slices.Equal(job.Needs, []string{bazelRBEJobName}) || job.Outputs != nil {
|
|
t.Errorf("%s needs %v outputs %v; it needs the rbe job alone and reports nothing to the gate", bazelRRCSeedJob, job.Needs, job.Outputs)
|
|
}
|
|
for _, c := range []struct {
|
|
event, ref, mode, rrc string
|
|
want bool
|
|
}{
|
|
{"push", "refs/heads/main", "remote", "seed", true},
|
|
{"push", "refs/heads/main", "remote", "canary", true},
|
|
{"push", "refs/heads/main", "remote", "on", true},
|
|
{"push", "refs/heads/main", "remote", "off", false},
|
|
{"push", "refs/heads/main", "cache", "on", false},
|
|
{"push", "refs/heads/other", "remote", "on", false},
|
|
{"pull_request", "refs/pull/1/merge", "remote", "on", false},
|
|
{"pull_request_target", "refs/heads/main", "remote", "on", false},
|
|
{"merge_group", "refs/heads/gh-readonly-queue/main/pr-1-abc", "remote", "on", false},
|
|
{"workflow_dispatch", "refs/heads/main", "remote", "on", false},
|
|
{"schedule", "refs/heads/main", "remote", "on", false},
|
|
} {
|
|
got := evalRRCIf(t, job.If, map[string]string{
|
|
"github.event_name": c.event, "github.ref": c.ref, "needs.rbe.outputs.mode": c.mode, "needs.rbe.outputs.rrc": c.rrc,
|
|
})
|
|
if got != c.want {
|
|
t.Errorf("%+v: seed runs %v, want %v", c, got, c.want)
|
|
}
|
|
}
|
|
if want := map[string]any{"contents": "read", "id-token": "write"}; !reflect.DeepEqual(job.Permissions, want) {
|
|
t.Errorf("%s permissions %v, want %v", bazelRRCSeedJob, job.Permissions, want)
|
|
}
|
|
raw := readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+bazelWorkflowName)
|
|
if text := bazelRRCJobText(t, raw, bazelRRCSeedJob); strings.Contains(text, "secrets.") {
|
|
t.Errorf("%s reads a secret; its only credential is the OIDC-minted writer certificate", bazelRRCSeedJob)
|
|
}
|
|
setup := rrcStep(t, job, "setup-bazel", func(s ciWorkflowStep) bool { return s.Uses == bazelSetupBazelUses })
|
|
if len(setup.Env) != 0 {
|
|
t.Errorf("%s setup-bazel env %v; local mode (no executor, no certificate) only", bazelRRCSeedJob, setup.Env)
|
|
}
|
|
cred := rrcStep(t, job, "writer certificate", func(s ciWorkflowStep) bool { return s.ID == "writer" })
|
|
if cred.Run != "bash "+bazelRRCCredential {
|
|
t.Errorf("writer step runs %q, want bash %s", cred.Run, bazelRRCCredential)
|
|
}
|
|
seed := rrcStep(t, job, "seed", func(s ciWorkflowStep) bool { return strings.Contains(s.Run, bazelRRCRunLanes) })
|
|
if seed.If != "${{ steps.writer.outputs.cert != '' }}" {
|
|
t.Errorf("seed step if %q; it must skip when the mint switched seeding off (no certificate)", seed.If)
|
|
}
|
|
for _, want := range []string{
|
|
"bash " + bazelRRCRunLanes + " --experimental_remote_repo_contents_cache --", "--loading_phase_threads=64",
|
|
`"--remote_cache=$ENDPOINT"`, `"--remote_instance_name=$INSTANCE"`,
|
|
`"--tls_client_certificate=$CERT"`, `"--tls_client_key=$KEY"`, "--remote_upload_local_results",
|
|
} {
|
|
if !strings.Contains(seed.Run, want) {
|
|
t.Errorf("seed step lacks %q:\n%s", want, seed.Run)
|
|
}
|
|
}
|
|
if strings.Contains(seed.Run, "remote_executor") {
|
|
t.Errorf("seed step names an executor; the writer certificate is cache only")
|
|
}
|
|
cleanup := rrcStep(t, job, "key removal", func(s ciWorkflowStep) bool { return strings.Contains(s.Run, "rrc-writer.key") })
|
|
if cleanup.If != "${{ always() }}" {
|
|
t.Errorf("the writer key removal runs if %q; it must always run", cleanup.If)
|
|
}
|
|
}
|
|
|
|
// bazelRRCJobText returns a job's raw text in a workflow.
|
|
func bazelRRCJobText(t *testing.T, raw, job string) string {
|
|
t.Helper()
|
|
start := strings.Index(raw, "\n "+job+":\n")
|
|
if start < 0 {
|
|
t.Fatalf("cannot find the %s job's text", job)
|
|
}
|
|
text := raw[start+1:]
|
|
if next := regexp.MustCompile(`\n [a-z0-9-]+:\n`).FindStringIndex(text); next != nil {
|
|
text = text[:next[0]]
|
|
}
|
|
return text
|
|
}
|
|
|
|
// TestBazelRRCVerifyJob: the nightly check runs from nightly.yml (schedule)
|
|
// or a dispatch while the cache is in use, fetches every lane cold without
|
|
// the repo contents cache, compares with rrc_verify.py using the lanes'
|
|
// read-only certificate and setup-bazel's endpoint (no secret outside
|
|
// setup-bazel), and alerts through an issue only when the comparison fails.
|
|
func TestBazelRRCVerifyJob(t *testing.T) {
|
|
wf := readCIWorkflow(t, bazelWorkflowName)
|
|
job, ok := wf.Jobs[bazelRRCVerifyJob]
|
|
if !ok {
|
|
t.Fatalf("%s has no %s job", bazelWorkflowName, bazelRRCVerifyJob)
|
|
}
|
|
if job.If != bazelRRCVerifyJobIf {
|
|
t.Errorf("%s if %q, want %q", bazelRRCVerifyJob, job.If, bazelRRCVerifyJobIf)
|
|
}
|
|
if job.Outputs != nil {
|
|
t.Errorf("%s outputs %v; it reports nothing to the gate", bazelRRCVerifyJob, job.Outputs)
|
|
}
|
|
for _, c := range []struct {
|
|
event, mode, rrc string
|
|
want bool
|
|
}{
|
|
{"schedule", "remote", "on", true},
|
|
{"schedule", "remote", "seed", true},
|
|
{"workflow_dispatch", "remote", "canary", true},
|
|
{"schedule", "remote", "off", false},
|
|
{"schedule", "cache", "on", false},
|
|
{"push", "remote", "on", false},
|
|
{"pull_request", "remote", "on", false},
|
|
{"pull_request_target", "remote", "on", false},
|
|
{"merge_group", "remote", "on", false},
|
|
} {
|
|
got := evalRRCIf(t, job.If, map[string]string{
|
|
"github.event_name": c.event, "needs.rbe.outputs.mode": c.mode, "needs.rbe.outputs.rrc": c.rrc,
|
|
})
|
|
if got != c.want {
|
|
t.Errorf("%+v: verify runs %v, want %v", c, got, c.want)
|
|
}
|
|
}
|
|
if want := map[string]any{"contents": "read", "issues": "write"}; !reflect.DeepEqual(job.Permissions, want) {
|
|
t.Errorf("%s permissions %v, want %v", bazelRRCVerifyJob, job.Permissions, want)
|
|
}
|
|
for _, s := range job.Steps {
|
|
if strings.Contains(s.Run, "experimental_remote_repo_contents_cache") || strings.Contains(s.Run, ".bazelrc.local") ||
|
|
strings.Contains(s.Run, "remote_upload_local_results") {
|
|
t.Errorf("%s step %q reads or writes the repo contents cache; the cold fetch must not", bazelRRCVerifyJob, s.Name)
|
|
}
|
|
for k, v := range s.Env {
|
|
if strings.Contains(v, "secrets.") && s.Uses != bazelSetupBazelUses {
|
|
t.Errorf("%s step %q env %s reads a secret; only setup-bazel may", bazelRRCVerifyJob, s.Name, k)
|
|
}
|
|
}
|
|
}
|
|
fetch := rrcStep(t, job, "cold fetch", func(s ciWorkflowStep) bool { return strings.HasPrefix(s.Name, "Cold fetch") })
|
|
if fetch.Run != "bash "+bazelRRCRunLanes+" -- --loading_phase_threads=64" {
|
|
t.Errorf("cold fetch runs %q; it must run every lane's command with no startup flag", fetch.Run)
|
|
}
|
|
verify := rrcStep(t, job, "verify", func(s ciWorkflowStep) bool { return s.ID == "verify" })
|
|
for _, want := range []string{"python3 " + bazelRRCVerifyTool, `--cert "$secret_dir/client.crt"`, `--key "$secret_dir/client.key"`, "--instance oss", ".bazelversion", "remote_executor="} {
|
|
if !strings.Contains(verify.Run, want) {
|
|
t.Errorf("verify step lacks %q", want)
|
|
}
|
|
}
|
|
alert := rrcStep(t, job, "alert", func(s ciWorkflowStep) bool { return strings.Contains(s.Run, "gh issue") })
|
|
if alert.If != "${{ failure() && steps.verify.outcome == 'failure' }}" {
|
|
t.Errorf("alert step if %q; it must alert only on a failed comparison", alert.If)
|
|
}
|
|
if strings.Contains(alert.Run, "${{") {
|
|
t.Errorf("alert step interpolates an expression into its script; pass it through env")
|
|
}
|
|
// The issue's containment: mode cache lanes see no repository
|
|
// variables, so the committed switches (both repositories) and
|
|
// rbe-cache-gate stop them. The mode remote readers read the same
|
|
// entries, so a mismatch sets the variable to seed (no reader, rrc-seed
|
|
// and this check keep running); off stops seeding and this check too.
|
|
for _, want := range []string{
|
|
"fork_rrc_read=off in both probe scripts", cacheRRCProbe, "gascity tools/rbe/cache-rrc-probe.sh",
|
|
"beads has no fresh-merge", "rbe-cache-gate close", "on a mismatch, set RBE_REPO_CONTENTS_CACHE to seed",
|
|
"set it to off only to stop seeding too", "could not check, leave the variable as it is",
|
|
} {
|
|
if !strings.Contains(alert.Run, want) {
|
|
t.Errorf("alert issue body lacks %q", want)
|
|
}
|
|
}
|
|
for _, bad := range []string{"Stop readers first: set the repository variable", "leave RBE_REPO_CONTENTS_CACHE on"} {
|
|
if strings.Contains(alert.Run, bad) {
|
|
t.Errorf("alert issue body still says %q; a mismatch sets the variable to seed", bad)
|
|
}
|
|
}
|
|
}
|
|
|
|
// permissionRank orders GitHub token permission levels so a grant can be
|
|
// compared with a request: write covers read, read covers none.
|
|
var permissionRank = map[string]int{"": 0, "none": 0, "read": 1, "write": 2}
|
|
|
|
type permWorkflow struct {
|
|
Jobs map[string]struct {
|
|
Uses string `yaml:"uses"`
|
|
Permissions map[string]string `yaml:"permissions"`
|
|
} `yaml:"jobs"`
|
|
}
|
|
|
|
// TestBazelWorkflowCallersGrantEveryRequestedPermission guards against
|
|
// startup_failure (gascity #7478): GitHub validates a reusable workflow's
|
|
// job permissions when the caller starts, including jobs whose `if:` would
|
|
// skip, so every workflow that calls bazel.yml must grant at least the
|
|
// highest level any bazel.yml job requests for each scope.
|
|
func TestBazelWorkflowCallersGrantEveryRequestedPermission(t *testing.T) {
|
|
root := sourceRepoRoot(t)
|
|
load := func(name string) permWorkflow {
|
|
var wf permWorkflow
|
|
if err := yaml.Unmarshal([]byte(readPolicyFile(t, root, ".github/workflows/"+name)), &wf); err != nil {
|
|
t.Fatalf("parse %s: %v", name, err)
|
|
}
|
|
return wf
|
|
}
|
|
need := map[string]string{}
|
|
for _, job := range load(bazelWorkflowName).Jobs {
|
|
for scope, level := range job.Permissions {
|
|
if permissionRank[level] > permissionRank[need[scope]] {
|
|
need[scope] = level
|
|
}
|
|
}
|
|
}
|
|
if need["id-token"] != "write" || need["issues"] != "write" {
|
|
t.Fatalf("bazel.yml requests %v; the parser is broken or the rrc jobs are gone", need)
|
|
}
|
|
entries, err := os.ReadDir(filepath.Join(root, ".github", "workflows"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
var callers []string
|
|
for _, e := range entries {
|
|
name := e.Name()
|
|
if name == bazelWorkflowName || (!strings.HasSuffix(name, ".yml") && !strings.HasSuffix(name, ".yaml")) {
|
|
continue
|
|
}
|
|
for jobName, job := range load(name).Jobs {
|
|
if job.Uses != "./.github/workflows/"+bazelWorkflowName {
|
|
continue
|
|
}
|
|
callers = append(callers, name)
|
|
var missing []string
|
|
for scope, level := range need {
|
|
if permissionRank[job.Permissions[scope]] < permissionRank[level] {
|
|
missing = append(missing, scope+": "+level)
|
|
}
|
|
}
|
|
sort.Strings(missing)
|
|
if len(missing) > 0 {
|
|
t.Errorf("%s job %q calls %s without granting %s (startup_failure)", name, jobName, bazelWorkflowName, strings.Join(missing, ", "))
|
|
}
|
|
}
|
|
}
|
|
sort.Strings(callers)
|
|
if want := []string{"bazel-farm.yml", "nightly.yml", "pr.yml"}; !slices.Equal(callers, want) {
|
|
t.Errorf("bazel.yml callers %v, want %v", callers, want)
|
|
}
|
|
}
|
|
|
|
// rrcMintCurlStub stands in for curl in rrc-writer-credential.sh. The OIDC
|
|
// request (its URL carries audience=) logs the audience and authorization
|
|
// and answers a token. A mint request logs its Authorization header and
|
|
// answers as the space-separated RBE_TEST_MINT says, one word per request
|
|
// (the last repeats): ok (a certificate for the runner's key, CN
|
|
// rbe-rrc-writer-beads O=gascity), cn, endpoint, instance, otherkey, nocert
|
|
// (each wrong in that one way), 000 (connection refused) or an HTTP status
|
|
// with an error body.
|
|
const rrcMintCurlStub = `#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
out= url= headers=()
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
-o) out=$2; shift 2 ;;
|
|
-H) headers+=("$2"); shift 2 ;;
|
|
-w | --data | --connect-timeout | --max-time | --retry) shift 2 ;;
|
|
-*) shift ;;
|
|
*) url=$1; shift ;;
|
|
esac
|
|
done
|
|
case "$url" in
|
|
*audience=*)
|
|
echo "oidc audience=${url##*audience=} auth=${headers[0]}" >>"$RBE_TEST_MINT_LOG"
|
|
echo '{"value": "oidc-jwt-value"}'
|
|
exit 0
|
|
;;
|
|
esac
|
|
echo "mint $url auth=${headers[0]}" >>"$RBE_TEST_MINT_LOG"
|
|
n=$(grep -c '^mint ' "$RBE_TEST_MINT_LOG")
|
|
read -r -a answers <<<"$RBE_TEST_MINT"
|
|
i=$((n - 1)); [ "$i" -lt "${#answers[@]}" ] || i=$((${#answers[@]} - 1))
|
|
answer=${answers[$i]}
|
|
key="$BAZEL_CI_SECRET_DIR/rrc-writer.key" cn=rbe-rrc-writer-beads endpoint=grpcs://rbe-west.ops.gascity.com:443 instance=oss
|
|
case "$answer" in
|
|
ok | cn | endpoint | instance | otherkey | nocert) ;;
|
|
000) echo "curl: (7) Failed to connect" >&2; exit 7 ;;
|
|
*) printf '{"error": "stub %s"}' "$answer" >"$out"; printf '%s' "$answer"; exit 0 ;;
|
|
esac
|
|
case "$answer" in
|
|
cn) cn=rbe-ci ;;
|
|
endpoint) endpoint=grpcs://attacker.example:443 ;;
|
|
instance) instance=main ;;
|
|
otherkey) openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out other.key 2>/dev/null; key=other.key ;;
|
|
esac
|
|
pem=
|
|
if [ "$answer" != nocert ]; then
|
|
pem=$(openssl req -new -x509 -key "$key" -subj "/O=gascity/CN=$cn" -days 1 2>/dev/null)
|
|
fi
|
|
jq -cn --arg pem "$pem" --arg e "$endpoint" --arg i "$instance" --arg cn "$cn" \
|
|
'{cert_pem: $pem, endpoint: $e, instance: $i, cn: $cn}' >"$out"
|
|
printf 200
|
|
`
|
|
|
|
// TestRRCWriterCredential runs rrc-writer-credential.sh against a stubbed
|
|
// curl (rrcMintCurlStub): it asks GitHub for an OIDC token with audience
|
|
// rbe-rrc-writer, masks it, sends it to the mint, retries only 429, 502 and
|
|
// the network, treats 503 or an unreachable mint as seeding off (no
|
|
// outputs, exit 0), and accepts only a certificate for the key it generated,
|
|
// CN rbe-rrc-writer[-x] O=gascity, for rbe-west's endpoint and instance oss.
|
|
func TestRRCWriterCredential(t *testing.T) {
|
|
for _, tool := range []string{"bash", "jq", "openssl"} {
|
|
requireHostTool(t, tool)
|
|
}
|
|
script := filepath.Join(sourceRepoRoot(t), bazelRRCCredential)
|
|
for _, c := range []struct {
|
|
answers string
|
|
ok, off bool
|
|
mints int
|
|
}{
|
|
{"ok", true, false, 1},
|
|
{"429 ok", true, false, 2},
|
|
{"502 ok", true, false, 2},
|
|
{"000 ok", true, false, 2},
|
|
{"502", false, false, 4},
|
|
{"403", false, false, 1},
|
|
{"401", false, false, 1},
|
|
{"409", false, false, 1},
|
|
{"cn", false, false, 1},
|
|
{"endpoint", false, false, 1},
|
|
{"instance", false, false, 1},
|
|
{"otherkey", false, false, 1},
|
|
{"nocert", false, false, 1},
|
|
{"503", true, true, 1},
|
|
{"000", true, true, 4},
|
|
} {
|
|
dir := t.TempDir()
|
|
bin := filepath.Join(dir, "bin")
|
|
if err := os.MkdirAll(bin, 0o755); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
for name, body := range map[string]string{"curl": rrcMintCurlStub, "sleep": "#!/bin/sh\nexit 0\n"} {
|
|
if err := os.WriteFile(filepath.Join(bin, name), []byte(body), 0o755); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
secret := filepath.Join(t.TempDir(), "secret")
|
|
output := filepath.Join(dir, "output")
|
|
out, err := runRRCScript(t, dir, "bash "+script, map[string]string{
|
|
"PATH": bin + string(os.PathListSeparator) + os.Getenv("PATH"),
|
|
"BAZEL_CI_SECRET_DIR": secret,
|
|
"GITHUB_OUTPUT": output,
|
|
"ACTIONS_ID_TOKEN_REQUEST_URL": "https://token.invalid/oidc?api-version=2.0",
|
|
"ACTIONS_ID_TOKEN_REQUEST_TOKEN": "request-token",
|
|
"RBE_TEST_MINT": c.answers,
|
|
"RBE_TEST_MINT_LOG": filepath.Join(dir, "mint.log"),
|
|
})
|
|
if (err == nil) != c.ok {
|
|
t.Errorf("%q: ok=%v, want %v\n%s", c.answers, err == nil, c.ok, out)
|
|
continue
|
|
}
|
|
log := readPolicyFile(t, dir, "mint.log")
|
|
if !strings.Contains(log, "oidc audience=rbe-rrc-writer auth=Authorization: bearer request-token\n") {
|
|
t.Errorf("%q: OIDC request %q", c.answers, log)
|
|
}
|
|
if got := strings.Count(log, "mint https://rbe-mint.ops.gascity.com:8444/v1/rrc-writer/cert auth=Authorization: Bearer oidc-jwt-value\n"); got != c.mints {
|
|
t.Errorf("%q: %d mint requests with the OIDC token, want %d:\n%s", c.answers, got, c.mints, log)
|
|
}
|
|
if !strings.Contains(out, "::add-mask::oidc-jwt-value") {
|
|
t.Errorf("%q: the OIDC token is not masked:\n%s", c.answers, out)
|
|
}
|
|
if !c.ok {
|
|
continue
|
|
}
|
|
if c.off {
|
|
if _, err := os.Stat(output); !os.IsNotExist(err) {
|
|
t.Errorf("%q: wrote outputs (%v); the seed must skip", c.answers, err)
|
|
}
|
|
if !strings.Contains(out, "seeding off (HTTP "+c.answers[len(c.answers)-3:]+")") {
|
|
t.Errorf("%q: no seeding-off warning:\n%s", c.answers, out)
|
|
}
|
|
continue
|
|
}
|
|
got := rrcOutputs(t, output)
|
|
for name, want := range map[string]string{
|
|
"cert": filepath.Join(secret, "rrc-writer.crt"), "key": filepath.Join(secret, "rrc-writer.key"),
|
|
"endpoint": "grpcs://rbe-west.ops.gascity.com:443", "instance": "oss",
|
|
} {
|
|
if got[name] != want {
|
|
t.Errorf("%q: output %s=%q, want %q", c.answers, name, got[name], want)
|
|
}
|
|
}
|
|
if key := readPolicyFile(t, secret, "rrc-writer.key"); !strings.Contains(key, "-----BEGIN PRIVATE KEY-----") {
|
|
t.Errorf("%q: key is not PKCS#8 (Bazel's TLS refuses SEC1)", c.answers)
|
|
}
|
|
if fi, err := os.Stat(filepath.Join(secret, "rrc-writer.key")); err != nil || fi.Mode().Perm() != 0o600 {
|
|
t.Errorf("%q: key mode %v (%v), want 0600", c.answers, fi, err)
|
|
}
|
|
if _, err := os.Stat(filepath.Join(secret, "rrc-mint.json")); !os.IsNotExist(err) {
|
|
t.Errorf("%q: the mint reply stays on disk (%v)", c.answers, err)
|
|
}
|
|
}
|
|
|
|
dir := t.TempDir()
|
|
if out, err := runRRCScript(t, dir, "bash "+script, map[string]string{
|
|
"BAZEL_CI_SECRET_DIR": filepath.Join(t.TempDir(), "secret"), "GITHUB_OUTPUT": filepath.Join(dir, "output"),
|
|
}); err == nil || !strings.Contains(out, "id-token: write") {
|
|
t.Errorf("without an OIDC request URL: err %v, want a failure naming id-token: write\n%s", err, out)
|
|
}
|
|
}
|
|
|
|
// The writer credential script and rrc_verify.py with its unit test are byte
|
|
// copies of gascity's (gascity 78c0e5c6f198306b96fe854c18b4d72f16395ce7,
|
|
// #7467), canonical there like tools/bazel/ci_analytics_extract.py: the
|
|
// pins catch a local edit that would drift out of lockstep.
|
|
var bazelRRCVendored = map[string]string{
|
|
bazelRRCCredential: "021788a724876bd1df7e5c117a4b3e1e60c5b25bb18bca472d61c35a08cb1fac",
|
|
bazelRRCVerifyTool: "00d50e569e2a50d8ef3d2740460de7337b71afd14dc76d2d2442dbd162236f94",
|
|
bazelRRCVerifyTest: "d934f13b466660f3148cf3c8dc7273803b4417f2a52f4edc2dd2f43afb93bfaa",
|
|
}
|
|
|
|
func TestBazelRRCVendoredFromGascity(t *testing.T) {
|
|
root := sourceRepoRoot(t)
|
|
for path, want := range bazelRRCVendored {
|
|
sum := sha256.Sum256([]byte(readPolicyFile(t, root, path)))
|
|
if got := hex.EncodeToString(sum[:]); got != want {
|
|
t.Errorf("%s sha256 = %s, want %s: it drifted from gascity's copy, or gascity changed it and this pin (and the copy) needs updating", path, got, want)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestRRCVerifyUnitTests runs gascity's rrc_verify_test.py in place: the
|
|
// AC key construction pinned against Bazel 9.3.0's own writes, the
|
|
// roll-forward of intermediate entries, a poisoned entry, and a GUID for
|
|
// .bazelversion's release.
|
|
func TestRRCVerifyUnitTests(t *testing.T) {
|
|
root := sourceRepoRoot(t)
|
|
python := requireHostTool(t, "python3")
|
|
cmd := exec.Command(python, filepath.Join(root, bazelRRCVerifyTest), "-v")
|
|
cmd.Dir = filepath.Join(root, "tools", "bazel")
|
|
if out, err := cmd.CombinedOutput(); err != nil {
|
|
t.Fatalf("%s failed: %v\n%s", bazelRRCVerifyTest, err, out)
|
|
}
|
|
}
|