beads/scripts/bazel_cache_mode_test.go
Julian Knutsen ebe3b6bcb0
test,ci(cmd/bd): run the Dolt-gated cmd/bd suite (advisory Bazel lane), fix in-process runner leaks and rotted tests; init --quiet stays quiet (#7185)
* test(cmd/bd): reset cobra flags between in-process runs; keep them out of parallel tests

runBDInProcess and runBDInProcessAllowError execute the package-level
rootCmd tree again and again in one process, and cobra never resets a
flag between Execute calls: a value one invocation parsed (create's
--title, --labels, list's --status, ...) and its Changed bit stayed on the
FlagSet, so the next invocation saw it, and a slice flag that had been
set appended to the stale value instead of replacing it. With a Dolt test
server (the only setup in which these tests run) that failed about 20
TestCLI_* tests depending on suite order.

TestMain now snapshots every flag of the tree (snapshotCommandFlags,
after all init()s registered them, before any test runs) and both runners
put the tree back before and after each run (resetCommandFlags): the
pointee of each flag.Value is restored (for pflag's scalar types that is
the bound variable; for slice types it also clears the internal "changed"
bit, so the next Set replaces), slice contents via SliceValue.Replace,
and Changed=false. TestResetCommandFlagsRestoresEveryFlag sets every flag
of the real tree and checks the reset. The ad-hoc --title reset in
TestCLI_CreateRejectsEmptyTitle_ProxiedServerMode is no longer needed.

Seven TestCLI_Comment* tests called t.Parallel() around the runners.
inProcessMutex serializes the runners only against each other, so their
chdir, stdio swap, env and config re-initialization raced every other
parallel test (data races on config globals; TestCobraParallelPolicyGuard
failing to read its own source files after a concurrent chdir). They run
serially now, and the guard treats the two runners like rootCmd.Execute.

Also two stale TestCLI_Create* cases: DryRunWithRigPrefix exercised a
create --prefix flag that no longer exists, and the flag-like-title cases
passed "--help" or "--foo-bar" where cobra parses them as flags; they now
pass the title after "--", the only way it reaches create's own guard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix the ADO round-trip and bulk-delete tests that rotted unseen

Both need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd (BEADS_TEST_SKIP=dolt everywhere), so they failed unnoticed:

- TestADORoundTrip* stored ado.pat in the database. ado.pat is a secret
  read only from config.yaml or AZURE_DEVOPS_PAT (config.IsYamlOnlyKey),
  so the tracker found no PAT. The fixture sets AZURE_DEVOPS_PAT.
- TestBulkDeleteNoResurrection expected GetIssue to return (nil, nil) for
  a deleted issue; the Dolt store reports storage.ErrNotFound.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix parity, completion and notion tests that only fail with a Dolt server

These are Dolt-gated (integration tag + test server), so no CI lane ran
them and they rotted:

- TestParityCreate*/TestParityReopenErrorExits1Silently: bd create
  auto-routes via routing.DetectUserRole("."), which warns "beads.role
  not configured (GH#2950)" on stderr when the working directory's git
  config has no beads.role. The suite pins the configured case, so the
  parity env now sets beads.role=maintainer in a test-owned global git
  config ($XDG_CONFIG_HOME/git/config; the role lookup scrubs every
  GIT_CONFIG_* variable, so those cannot be used).
- TestIssueIDCompletion: issueIDCompletion resolves a database path
  before looking at the open store and returns nothing when there is
  none; the test now sets dbPath to its store's path.
- TestIssueIDCompletion_UsesWorktreeFallbackWhenStoreNil and
  TestGetNotionConfigReadsDBPathWhenStoreUnset: the code under test
  reopens the store from metadata.json, which names a database but not
  the per-test branch of the shared database that newTestStore uses, so
  the reopened store saw none of the test's rows. They now use their own
  database (newTestStoreIsolatedDB).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix stale doctor/e2e/autocommit/readonly Dolt-gated tests

These need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd, so they rotted unseen:

- TestE2E_DoctorFixMetadataRoundtrip, TestE2E_MigrateDoltMetadata,
  TestDoctorCheckHealthReportsVersionMismatchOnRepoLocalPort: bd init now
  defaults to embedded mode, where `bd sql` and `bd dolt start` are
  refused. Init with --server on a repo-local server of their own
  (repoLocalServerEnv: the suite's shared-server port vars cleared, server
  stopped at cleanup) and start it explicitly, since BEADS_TEST_MODE=1 never
  auto-starts one. MigrateDoltMetadata's DELETE also needed `key` quoted.
- TestDoctor{Conventions,Pollution}MaxRows_EnvOnly_Exits2: bdRunRaw drops
  every BEADS_* variable, so the subprocess hit dolt.New's test-database
  firewall (exit 1) before the row cap; pass BEADS_TEST_SERVER=1.
- TestDoltAutoCommit_On_WritesAdvanceHead: the embedded engine keeps its
  data under .beads/embeddeddolt, not .beads/dolt; findDoltRepoDir looks
  in both.
- TestValidateCheck_AllClean: bd doctor validate runs five checks since
  the cross-table duplicates check was added.
- TestConfigValidateReadOnlyIsHermetic: readonly commands take a shared
  workspace gate, whose lock file (.beads/dolt.gate.lock) workspacegate
  creates on first use by design; compare the .beads tree without gate
  lock files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): undo each in-process execution's env and storage-mode leaks

Every in-process run of the bd command tree (runBDInProcess and the many
tests that call rootCmd.Execute directly) leaves process state behind:
PersistentPreRun exports the selected workspace as BEADS_DIR, loads its
.beads/.env, and records the workspace's mode in serverMode /
proxiedServerMode. With a Dolt test server, later tests in the same shard
then resolved a deleted fixture ("no active beads workspace found",
"migration sidecar is missing"), ran subprocess bd against the wrong
workspace ("recorded as proxied-server in .beads/metadata.json"), or took
the proxied path in RunE-level tests ("proxied-server UOW provider not
initialized", a nil-UOW panic in reopenProxiedResolve that killed a shard).

TestMain now registers a cobra initializer/finalizer pair
(installExecuteIsolation) that snapshots the environment and the two mode
globals when an execution starts and restores them when it ends; the
runners' restoreEnvSnapshot shares the helper. TestDoltConfigEnvironmentOverrides
used os.Setenv + defer os.Unsetenv, which deleted the suite's
BEADS_DOLT_SERVER_PORT (the test server's port) for every later test; it
uses t.Setenv now.

Full integration-tagged cmd/bd suite against a hermetic dolt (16 shards):
18 order-dependent failures and the shard panic gone; the 27 remaining
failures all fail alone too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(init): keep --quiet quiet through the Claude/Codex/Cursor auto-setup

`bd init --quiet` ("Suppress output") still printed the project
integration installers' progress to stdout ("Installing Claude hooks for
this project...", "Claude Code integration installed", ...): the
installers write to os.Stdout unconditionally. Point stdout at the null
device around them while quiet; their failures keep going to stderr as
before. Found by TestInitCommand's quiet case, which asserts no output and
had not run in CI (Dolt-gated).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(init): bring the Dolt-gated init tests up to the current init contract

These tests need a Dolt test server, so no CI lane ran them and they
drifted from the product:

- Embedded is init's default mode, so the server-mode assertions
  (.beads/dolt, opening the database on the test server, dolt_mode
  "server" in metadata.json) now pass --server (TestInitCommand,
  TestInitAlreadyInitialized, TestInitRedirect, TestInitBEADS_DIR,
  TestInitDoltMetadata, TestInitDoltMetadataNoGit, TestInitDatabaseFlag's
  metadata subtests). The in-process ones reset rootCmd's flags around
  themselves so --server cannot leak into later in-process inits.
- Each server init gets its own --database: the suite shares one test
  server, and a "test" database another test created belongs to another
  project (init refuses: PROJECT IDENTITY MISMATCH).
- The marker tests seed a current-era version witness: the legacy-upgrade
  guard refuses a non-empty .beads/dolt without one.
- The redirect test provisions the canonical target with config.yaml: bd
  ignores a redirect to a directory with no project files (#4692).
- Shared-server inits get a free port of their own
  (sharedServerTestEnv): the inherited BEADS_DOLT_SERVER_PORT is the
  suite's test server, which a shared-server start refuses as another
  project's, and the default shared port is 3308, a developer's live
  server.
- --shared-server's config.yaml is now nested YAML (dolt: shared-server:
  true); parse it instead of matching the old flat line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): run the Dolt-gated tests that skipped on a bare dolt.New

setupTestDB (config, kv and Linear round-trip tests), setupGatedTestDB
(mol ready --gated) and the rename-prefix and bd sql tests opened stores
with dolt.New(&dolt.Config{Path: ...}): no server port, no
CreateIfMissing, so against a running test server they got "database ...
not found" and skipped as "Dolt server not available". They now use the
suite's newTestStore helpers (skip only when no test server runs), and
what that exposed is fixed test-side:

- TestCustomStatusConfig: custom_statuses is read back ORDER BY name;
  look statuses up by name / expect name order.
- TestLinearRoundTripCoreFields, TestLinearPullMilestonesCreatesEpicHierarchy:
  linear.api_key is a yaml-only secret (LINEAR_API_KEY), push now needs an
  explicit linear.state_map, and batch push uses TeamLabels and
  issueBatchCreate, which the mock server now answers.
- TestSqlCommand: bd sql refuses in the default embedded mode; set
  serverMode for the server-backed store, and stop discarding RunE errors.
- TestE2E_AutoStartedRepoLocalServerPersistsAcrossCommands skipped on
  this process's embedded mode although every bd it runs is a subprocess
  on a --server workspace; gate on the dolt binary instead and stop the
  auto-started server at cleanup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): no fixed "unused" port, no swallowed JSON, no leaked front-door server

Three more things the Dolt-backed cmd/bd run exposed:

- TestTestServerConnection, TestSharedServerCfgNilHonorsSharedServer and
  the two TestLegacyGuard* e2e tests pointed bd at 127.0.0.1:59999 as a
  port nothing listens on. Any process on the host may hold it; a
  listener that accepts and resets turns the expected "connection
  refused" / "Dolt server unreachable" into another error. They take a
  just-freed port from closedLoopbackPort (testutil.FindFreePort) now.
- TestDoltSetConfigJSONOutput and TestDoltSetConfigWithUpdateConfig
  created .beads 0755, so bd's permissions warning landed in the JSON
  under test and the tests skipped ("output not pure JSON") every run.
  The fixture uses 0700, and non-JSON output fails instead of skipping.
- TestMigrateDoltModeFrontDoor's closing show/list auto-start a
  repo-local server after the reverse migration; nothing stopped it, so
  the suite's leak check failed (when the opt-in
  BEADS_TEST_MIGRATION_FRONTDOOR suite runs). It stops it at cleanup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): advisory cmd/bd Dolt-server tier (bazel-cmd-dolt)

No CI lane ran cmd/bd's Dolt-gated tests (TestCLI_*, the init and
store-backed suites, the doctor/e2e tests): main.yml's integration shards,
nightly's Full Test Suite, and Bazel's --config=prcore/ci and
--config=integration all pass BEADS_TEST_SKIP=dolt, so cmd/bd's TestMain
never starts a test server and those tests skip; the proxied and embedded
tiers run only their manifests' tests. They rotted unseen.

//cmd/bd:bd_dolt_server_test (tag dolt-server-cmd) runs the whole
integration-tagged cmd/bd suite, the integration build's bd_test, against
a hermetic dolt sql-server: BEADS_TEST_DOLT_SERVER=local starts it from
the pinned dolt in the test wrapper's runfiles, and
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 fails the target instead of skipping
into a cached pass. No test selection: the Go binary shards itself over
every top-level test (32 shards), so each runs exactly once, and
check_testcases.py fails a shard that ran none. --config=doltserver-cmd
has exactly --config=integration's build flags, so the two lanes share
bd_test's compile; prcore/ci and integration exclude the tag.

bazel.yml's new bazel-cmd-dolt job runs it exactly where bazel-integration
runs (same if: remotely on the farm for same-repo PRs, fork-ro/fork-rw,
and locally with the read-only cache in mode cache for forks). It is
advisory: job-level continue-on-error keeps a failure out of the call's
aggregate (BAZEL), and pr.yml's ci-gate reads its result (BAZEL_CMD_DOLT)
but requires it only while the committed top-level flag
BAZEL_CMD_DOLT_REQUIRED is "true" (now "false"; flipping it is the one
change needed). bazel-gate.sh accepts its skip in modes skip and local.

Policy tests: a third lane category in TestBazelLaneIsGatedAlongsideLegacy
(bazelFlagGatedLanes: gate env wired, never statically required, added by
the run script exactly when the pinned flag is "true"), the gate
simulation over every mode with the flag off (advisory: nothing it
reports reddens the gate) and on (exactly a gated lane), the cache-mode
and execution-mode checks, the tag taxonomy and prcore/integration
exclusions, and TestBazelCmdDoltJob (job steps, timeouts, .bazelrc
config equal to integration's build with no dolt skip or test selection,
the target's dolt-server rules, and bd_test's env repeated).

Measured locally (all fixes in this series, 32 shards, 16 at a time on a
shared host): 3267 top-level tests, all shards pass, max shard 313s (one
5001-issue export test, 214s, sets it), mean 100s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(init): hand the project installers a writer instead of swapping os.Stdout

The --quiet fix earlier in this series pointed the process-global
os.Stdout at the null device around the Claude/Codex/Cursor installers.
That works, but it mutates a global from inside a cobra Run: anything else
writing to stdout meanwhile (a goroutine, a future concurrent installer)
would vanish, and a concurrent read of os.Stdout is a data race.

The installers already write through env structs with stdout/stderr
fields. setup.ProjectInstallOutput carries the two writers, and
InstallClaudeProjectTo / InstallCodexProjectTo / InstallCursorProjectTo
use them (the old entry points keep the process streams). init passes
QuietProjectInstallOutput() under --quiet: progress discarded, error lines
still on stderr. The proxied-server init path (domain adapter
InstallClaudeProject now takes quiet) does the same, so its
"Installing Claude hooks..." is quiet too.

Consistent error policy under --quiet: errors still reach stderr. init's
own "Warning: failed to setup <installer>" lines were suppressed under
--quiet while the installers' "Error:" lines were not, and some installer
failures (Cursor rules) print nothing themselves; the warnings now print
whether or not quiet, like the installers' errors.

Coverage in required lanes: TestProjectInstallersReportToTheGivenOutput
(cmd/bd/setup) and TestEmbeddedInitA/basic, which now asserts that
`bd init --quiet` with agents enabled installs the Claude hooks and prints
none of the installers' progress (verified to fail with the quiet output
disabled).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(export): exercise the incremental-export fallback with a small threshold

TestTryIncrementalExport_ThresholdExceededFallsBack seeded
incrementalExportThreshold+1 = 5001 issues to cross the threshold. Batch
create is one transaction but still several statements per issue, so under
-race against the shared test server it took ~214 s, the longest test in
cmd/bd's Dolt-backed suite by far. The behaviour under test is only
"more changed issues than the threshold -> full export fallback".

incrementalExportThreshold becomes a var (same value), and the test, which
is not parallel, sets it to 20 and seeds 21 issues: 214 s -> 0.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): tighten the flag-reset self-test, guards and canaries

Review follow-ups:
- TestResetCommandFlagsRestoresEveryFlag no longer skips a flag it cannot
  set: a flag type without a sample value, a Set error, or a non-pointer
  Value (which resetCommandFlags could not restore) is an error, so a new
  flag type cannot go untested. resetCommandFlags documents that it also
  resets flag-bound package globals.
- TestCobraParallelPolicyGuard treats .ExecuteC/.ExecuteContext/
  .ExecuteContextC like .Execute (the per-execution isolation writes the
  mode globals; a parallel execution would race).
- TestConfigValidateReadOnlyIsHermetic ignores only the gate a readonly
  command takes (.beads/dolt.gate.lock and its .info sidecar), not any
  "*.gate.lock*" file, so a new unexpected gate still trips the canary.
- TestCLI_CreateRejectsFlagLikeTitles: a bare `bd create --foo-bar` must
  fail and create no issue (bd-2c0's intent), next to the "--" cases.
- TestCustomStatusConfig cites why statuses come back in name order: the
  normalized custom_statuses table (#2961, 8537d943b) has no ordinal and is
  read ORDER BY name, like custom_types; the test predates it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the cmd/bd Dolt-server tier in 16 shards

Each shard pays ~30-40 s of fixed cost (TestMain plus its own dolt
sql-server), so 32 shards spent roughly 40% of the lane's executor time on
setup and added 32 farm actions to every PR touching cmd/bd. With the
5001-issue export test cut to under a second, 16 shards measure (locally,
--nocache_test_results): max 186 s, mean 137 s, 2,187 executor-seconds
(vs ~2,800-3,200 at 32), all 3,267 top-level tests, every shard passing.
TestBazelCmdDoltJob pins the count; bazel.yml's timing comment is
restated (cache mode ~20-30 minutes).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 03:06:54 -07:00

384 lines
16 KiB
Go

package scripts_test
import (
"encoding/base64"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"testing"
)
// Mode cache, end to end, for every way a bazel.yml run can start: the rbe
// job's real decision step, each lane's `if:` and setup-bazel env (GitHub's
// expressions, evaluated) and the rc write-bazelrc.sh then writes. A cache
// run (fork and Dependabot PRs while rbe-fork is closed or unreachable,
// rbe=cache, a missing executor secret) must reach the rc with
// --config=fork-cache and nothing else remote: no executor, no TLS
// material, no instance, no upload. That holds even where the run could
// read the secrets (an rbe=cache dispatch on the base repo, and,
// adversarially, a fork PR whose secrets GitHub would never pass). Its
// lanes are the local ones plus bazel-integration; the remote-only tiers
// skip, and pr.yml's gate (bazel-gate.sh) accepts exactly those skips.
//
// The fork modes too (rbe-fork open, the mint answering ro or rw): every
// lane runs, setup-bazel's fork steps (setupBazelForkSim: key, CSR
// artifact, certificate from a stand-in mint) hand write-bazelrc.sh the
// minted certificate, and the rc executes on rbe-fork with instance
// oss-fork (ro) or oss (rw) and that certificate, never the CI secrets.
func TestBazelCacheModeReachesTheRC(t *testing.T) {
bash := requireHostTool(t, "bash")
root := sourceRepoRoot(t)
workflow := readCIWorkflow(t, bazelWorkflowName)
decide := workflow.job(t, bazelRBEJobName).Steps[0]
script := filepath.Join(root, setupBazelActionDir, "write-bazelrc.sh")
pem := func(kind string) string {
return base64.StdEncoding.EncodeToString([]byte("-----BEGIN " + kind + "-----\nMIIBfake\n-----END " + kind + "-----\n"))
}
secrets := map[string]string{
"RBE_WEST_EXECUTOR": "grpcs://" + "farm.invalid:443",
"RBE_TLS_CERT": pem("CERTIFICATE"),
"RBE_TLS_KEY": pem("PRIVATE KEY"),
"RBE_TLS_CA": pem("CERTIFICATE"),
}
// setup-bazel's env values: `needs.rbe.outputs.X == 'v' && RHS || ''`,
// RHS a secret or a literal. Anything else fails the test, so the
// simulation cannot drift from the workflow.
setupExpr := regexp.MustCompile(`^\$\{\{ needs\.rbe\.outputs\.(enabled|mode) == '([a-z]+)' && (?:secrets\.([A-Z_]+)|'([^']*)') \|\| '' \}\}$`)
evalSetup := func(t *testing.T, expr, mode, enabled, tier string, haveSecrets bool) string {
t.Helper()
switch expr {
case bazelForkRemoteValue:
if strings.HasPrefix(mode, "fork-") {
return "true"
}
return ""
case "${{ needs.rbe.outputs.tier }}":
return tier
case "${{ github.event.pull_request.number }}":
return "7123" // every start that reaches a fork mode is a pull_request
}
m := setupExpr.FindStringSubmatch(expr)
if m == nil {
t.Fatalf("setup-bazel env %q: teach TestBazelCacheModeReachesTheRC how GitHub evaluates it", expr)
}
got := map[string]string{"enabled": enabled, "mode": mode}[m[1]]
if !strings.EqualFold(got, m[2]) {
return ""
}
if m[3] != "" {
if !haveSecrets {
return ""
}
v, ok := secrets[m[3]]
if !ok {
t.Fatalf("setup-bazel env reads secrets.%s, which no caller passes", m[3])
}
return v
}
return m[4]
}
// writeRC: write-bazelrc.sh as the rc step runs it, in runner.temp (the
// fork steps put the key and certificate in the same secret dir).
writeRC := func(t *testing.T, dir string, env map[string]string) (outputs, rc string, files []string, logs string, err error) {
t.Helper()
ws := filepath.Join(dir, "ws")
if err := os.MkdirAll(ws, 0o755); err != nil {
t.Fatal(err)
}
secretDir := filepath.Join(dir, "bazel-ci-secret")
cmd := exec.Command(bash, script)
cmd.Dir = ws
cmd.Env = []string{
"PATH=" + os.Getenv("PATH"),
"GITHUB_WORKSPACE=" + ws,
"GITHUB_OUTPUT=" + filepath.Join(dir, "out"),
"BAZEL_CI_CACHE_DIR=" + filepath.Join(dir, "bazel-ci-cache"),
"BAZEL_CI_SECRET_DIR=" + secretDir,
}
for k, v := range env {
cmd.Env = append(cmd.Env, k+"="+v)
}
b, err := cmd.CombinedOutput()
o, _ := os.ReadFile(filepath.Join(dir, "out"))
r, _ := os.ReadFile(filepath.Join(secretDir, "ci.bazelrc"))
entries, _ := os.ReadDir(secretDir)
for _, e := range entries {
files = append(files, e.Name())
}
return string(o), string(r), files, string(b), err
}
type start struct {
name string
f rbeFacts
with map[string]string
secret bool // the run can read the RBE secrets
}
mint := newForkMint(t)
var starts []start
farm := map[string]string{"fork-farm": "authorized", "checkout-sha": "0123456789abcdef0123456789abcdef01234567"}
for _, rbeVar := range []string{"true", ""} {
v := "var=" + map[bool]string{true: "on", false: "unset"}[rbeVar != ""]
pr := func(fork, dependabot bool, secret bool) rbeFacts {
s := ""
if secret {
s = "x"
}
return rbeFacts{event: "pull_request", rbeVar: rbeVar, secret: s, fork: fork, dependabot: dependabot}
}
// prMint: the same, with rbe-fork-mint's /v1/status answer.
prMint := func(fork, dependabot, secret bool, answer string) rbeFacts {
f := pr(fork, dependabot, secret)
f.mint = answer
return f
}
starts = append(starts,
start{"same-repo PR, " + v, pr(false, false, true), nil, true},
start{"same-repo PR without the secret, " + v, pr(false, false, false), nil, false},
start{"Dependabot PR, " + v, pr(false, true, false), nil, false},
start{"fork PR, " + v, pr(true, false, false), nil, false},
start{"fork PR that somehow has the secrets, " + v, pr(true, false, true), nil, true},
start{"fork PR passing fork-farm from pull_request, " + v, pr(true, false, true), farm, true},
start{"push to main, " + v, rbeFacts{event: "push", rbeVar: rbeVar, secret: "x"}, nil, true},
start{"dispatch rbe=cache, " + v, rbeFacts{event: "workflow_dispatch", rbeVar: rbeVar, secret: "x"}, map[string]string{"rbe": "cache"}, true},
start{"dispatch rbe=off, " + v, rbeFacts{event: "workflow_dispatch", rbeVar: rbeVar, secret: "x"}, map[string]string{"rbe": "off"}, true},
start{"bazel-farm authorized fork, " + v, rbeFacts{event: "pull_request_target", rbeVar: rbeVar, secret: "x", fork: true}, farm, true},
start{"bazel-farm authorized fork without the secret, " + v, rbeFacts{event: "pull_request_target", rbeVar: rbeVar, fork: true}, farm, false},
// rbe-fork: the mint's answer decides fork and Dependabot runs.
start{"fork PR, rbe-fork ro, " + v, prMint(true, false, false, "ro"), nil, false},
start{"fork PR, rbe-fork rw, " + v, prMint(true, false, false, "rw"), nil, false},
start{"fork PR, rbe-fork closed, " + v, prMint(true, false, false, "closed"), nil, false},
start{"fork PR, rbe-fork rw tier closed, " + v, prMint(true, false, false, "rw-closed"), nil, false},
start{"fork PR, rbe-fork canary refuses, " + v, prMint(true, false, false, "canary"), nil, false},
start{"Dependabot PR, rbe-fork ro, " + v, prMint(false, true, false, "ro"), nil, false},
start{"fork PR that somehow has the secrets, rbe-fork ro, " + v, prMint(true, false, true, "ro"), nil, true},
start{"fork PR, rbe-fork ro, dispatch-style rbe=cache, " + v, prMint(true, false, false, "ro"), map[string]string{"rbe": "cache"}, false},
)
}
seen := map[string]bool{}
for _, s := range starts {
t.Run(s.name, func(t *testing.T) {
out, err := runDecisionStep(t, decide, s.f, s.with)
if err != nil {
t.Fatal(err)
}
mode, enabled, tier := out["mode"], out["enabled"], out["tier"]
seen[mode] = true
// Fork and Dependabot pull_request runs: fork-<tier> when the
// mint says open with tier ro or rw, else cache, whatever else
// holds (rbe=off: local). Other forks outside bazel-farm.yml's
// authorized call, and rbe=cache, are mode cache.
authorized := s.f.event == "pull_request_target" && s.with["fork-farm"] == "authorized"
forkPR := s.f.event == "pull_request" && (s.f.fork || s.f.dependabot)
want := ""
switch {
case s.with["rbe"] == "off":
want = "local"
case s.with["rbe"] == "cache":
want = "cache"
case forkPR && (s.f.mint == "ro" || s.f.mint == "rw"):
want = "fork-" + s.f.mint
case forkPR, s.f.fork && !authorized:
want = "cache"
}
if want != "" && mode != want {
t.Errorf("mode = %s, want %s", mode, want)
}
if wantTier := strings.TrimPrefix(want, "fork-"); strings.HasPrefix(want, "fork-") && tier != wantTier || !strings.HasPrefix(mode, "fork-") && tier != "" {
t.Errorf("mode %s with tier %q", mode, tier)
}
if mode == "remote" && (!s.secret || s.f.fork && !authorized || forkPR) {
t.Errorf("mode remote without the secret, for an unauthorized fork, or for a fork or Dependabot pull_request")
}
var lanes []string
for name, job := range workflow.Jobs {
if name == bazelRBEJobName {
continue
}
runs := bazelLaneRunModes(t, name, job.If, s.with)[mode]
switch {
// rbe-prewarm never runs in any fork mode (B1, security
// review of bdef342d5: gated on mode remote only). A fork or
// Dependabot pull_request run never carries a
// workflow_call secret regardless of mode, and
// bazel-farm.yml (the other path to a privileged fork tier)
// no longer forwards the app secrets either, so there is no
// audience left for pre-warming in fork-ro or fork-rw (see
// bazel.yml's comment on the job).
case strings.HasPrefix(mode, "fork-") && !runs && !bazelPackageJobs[name] && !(name == bazelRBEPrewarmJobName && strings.HasPrefix(mode, "fork-")):
t.Errorf("%s does not run in mode %s (every lane runs remotely)", name, mode)
case mode == "cache" && (name == bazelIntegJobName || name == bazelCmdDoltJobName) && !runs:
t.Errorf("%s does not run in mode cache", name)
case mode == "cache" && bazelRemoteOnlyJobs[name] && runs:
t.Errorf("remote-only %s runs in mode cache", name)
case mode == "local" && name == bazelIntegJobName && runs:
t.Errorf("%s runs in mode local (no cache: a cold tagged race build)", name)
}
if !runs {
continue
}
lanes = append(lanes, name)
for _, step := range job.Steps {
if step.Uses != "./"+setupBazelActionDir {
continue
}
env := map[string]string{}
for k, expr := range step.Env {
env[k] = evalSetup(t, expr, mode, enabled, tier, s.secret)
}
runnerTemp := t.TempDir()
env = setupBazelForkSim(t, mint, runnerTemp, name, env, tier)
outputs, rc, files, logs, err := writeRC(t, runnerTemp, env)
if err != nil {
t.Fatalf("%s: write-bazelrc.sh failed in mode %s: %v\n%s", name, mode, err, logs)
}
checkModeRC(t, name, mode, outputs, rc, files, logs)
if strings.HasPrefix(mode, "fork-") {
secretDir := filepath.Join(runnerTemp, "bazel-ci-secret")
for _, want := range []string{
"\nbuild:remote-exec --tls_client_certificate=" + filepath.Join(secretDir, "fork.crt") + "\n",
"\nbuild:remote-exec --tls_client_key=" + filepath.Join(secretDir, "fork.key") + "\n",
} {
if !strings.Contains(rc, want) {
t.Errorf("%s (mode %s): rc lacks the minted certificate %q:\n%s", name, mode, strings.TrimSpace(want), rc)
}
}
}
}
}
if mode == "skip" && len(lanes) != 0 {
t.Errorf("mode skip runs %v", lanes)
}
// pr.yml's gate accepts as skipped exactly the gated lanes that
// did not run (TestBazelGateSimulation runs the gate itself).
ran := map[string]bool{}
for _, l := range lanes {
ran[l] = true
}
var wantSkips []string
for lane, id := range bazelLaneGateIDs {
// F3: package-mcp/package-npm skip only when the caller's
// package-gates input is off, never because of rbe mode;
// bazel-gate.sh knows nothing about them (its skip list is
// mode-derived only), so this scenario's with (which never
// sets package-gates here) must not expect them either.
if bazelPackageJobs[lane] {
continue
}
if !ran[lane] {
wantSkips = append(wantSkips, id)
}
}
// Flag-gated lanes' skips are accepted whatever the flag says.
for lane, g := range bazelFlagGatedLanes {
if !ran[lane] {
wantSkips = append(wantSkips, g.id)
}
}
if len(lanes) == 0 {
wantSkips = append(wantSkips, bazelAggregateGateID)
}
cmd := exec.Command(bash, bazelGateScript, "skips")
cmd.Dir = root
cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "BAZEL_RBE_MODE=" + mode, "BAZEL_RBE_ENABLED=" + enabled}
b, err := cmd.Output()
if err != nil {
t.Fatal(err)
}
got := strings.Fields(string(b))
sort.Strings(got)
sort.Strings(wantSkips)
if strings.Join(got, " ") != strings.Join(wantSkips, " ") {
t.Errorf("mode %s: gate skips %v, want %v (lanes run: %v)", mode, got, wantSkips, lanes)
}
})
}
for _, mode := range bazelRBEModes {
if !seen[mode] {
t.Errorf("no start reaches mode %s", mode)
}
}
}
// checkModeRC: what one lane's generated rc may hold in its mode.
func checkModeRC(t *testing.T, lane, mode, outputs, rc string, files []string, logs string) {
t.Helper()
has := func(s string) bool { return strings.Contains(rc, s) }
// Every mode: the fetch hardening (key neutral, repository fetching only)
// and the client heap (R4: a startup option, key neutral).
for _, want := range []string{
"\ncommon --repo_env=GOPROXY=https://proxy.golang.org|https://proxy.golang.org|direct\n",
"\ncommon --http_timeout_scaling=2.0\n",
"\nstartup --host_jvm_args=-Xmx4g\n",
} {
if !has(want) {
t.Errorf("%s (mode %s): rc lacks %q:\n%s", lane, mode, strings.TrimSpace(want), rc)
}
}
credentialFree := func() {
for _, bad := range []string{"remote_executor", "remote-exec", "tls_", "remote_instance_name", "--remote_upload_local_results", "--remote_cache", "--remote_header", "--bes_", "farm.invalid"} {
if has(bad) {
t.Errorf("%s (mode %s): rc carries %q:\n%s", lane, mode, bad, rc)
}
}
if len(files) != 1 || files[0] != "ci.bazelrc" {
t.Errorf("%s (mode %s): secret dir holds %v, want only ci.bazelrc (no key material)", lane, mode, files)
}
if strings.Contains(logs, "::add-mask::") {
t.Errorf("%s (mode %s): masked something, so it saw a secret:\n%s", lane, mode, logs)
}
}
switch mode {
case "cache":
credentialFree()
if !has("\nbuild --config=fork-cache\n") || !strings.Contains(outputs, "remote=false") || !strings.Contains(outputs, "cache=true") {
t.Errorf("%s (mode cache): outputs %q rc %q; want --config=fork-cache, remote=false, cache=true", lane, outputs, rc)
}
case "local":
credentialFree()
if has("fork-cache") || !strings.Contains(outputs, "cache=false") {
t.Errorf("%s (mode local): outputs %q rc %q; want neither cache nor remote", lane, outputs, rc)
}
case "remote":
if !has("\nbuild --config=remote-exec\n") || has("fork-cache") || !strings.Contains(outputs, "remote=true") {
t.Errorf("%s (mode remote): outputs %q rc %q; want remote-exec and no fork-cache", lane, outputs, rc)
}
case "fork-ro", "fork-rw":
// rbe-fork with the minted certificate: the mint's endpoint and
// instance, no upload from the runner, never the CI secrets (their
// endpoint is farm.invalid here) or the fork cache.
instance := rbeForkInstance[strings.TrimPrefix(mode, "fork-")]
for _, want := range []string{
"\nbuild:remote-exec --remote_executor=" + rbeForkEndpoint + "\n",
"\nbuild:remote-exec --remote_instance_name=" + instance + "\n",
"\nbuild:remote-exec --noremote_upload_local_results\n",
"\nbuild --config=remote-exec\n",
} {
if !has(want) {
t.Errorf("%s (mode %s): rc lacks %q:\n%s", lane, mode, strings.TrimSpace(want), rc)
}
}
for _, bad := range []string{"fork-cache", "farm.invalid", "client.crt", "client.key", "--tls_certificate", "--remote_upload_local_results", "--remote_cache", "--remote_header", "--bes_"} {
if has(bad) {
t.Errorf("%s (mode %s): rc carries %q:\n%s", lane, mode, bad, rc)
}
}
sort.Strings(files)
if strings.Join(files, " ") != "ci.bazelrc fork.crt fork.key mint.json" {
t.Errorf("%s (mode %s): secret dir holds %v, want the rc, the minted certificate, its key and the mint's reply", lane, mode, files)
}
if !strings.Contains(outputs, "remote=true") || !strings.Contains(outputs, "cache=false") || strings.Contains(logs, "::add-mask::") {
t.Errorf("%s (mode %s): outputs %q logs %q; want remote=true, cache=false and nothing masked (no secret seen)", lane, mode, outputs, logs)
}
default:
t.Errorf("%s runs in mode %q", lane, mode)
}
}