beads/scripts/ci_merge_queue_test.go
Julian Knutsen 873162dafa
ci(bazel): dolt-int gates server-storage + cmd-dolt in mode remote (item 5 A2) (#7535)
* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2)

In mode remote, folds bazel-server-storage (--config=doltserver-integration)
and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job
(--config=dolt-int), following the dolt-race pattern from #7514. The
members have different gating categories (server-storage plain-gated,
cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true
and a bespoke --label-scoped Shard balance step. All other modes keep the
two lanes running unchanged. Action keys are byte-identical between the
member configs and the union (17 + 46 = 63 TestRunner actions), so cached
results carry over either direction and a revert is a safe rollback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review)

Coordinator review of #7535 found three must-fixes: integration:"off" was
ignored in remote mode (the union's members already honor it); job-level
continue-on-error made a server-storage-only failure read success on
push/nightly/dispatch, where main stays red; and the PR's key-parity claim
overstated byte-identical actionKeys when only the remote action digest
(not the local actionKey) is unaffected by moving --test_env/--test_arg
onto the targets' own env/args.

Fixes:
- bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching
  the members (#7491's review); bazelLaneRunModes and
  TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case.
- Job-level continue-on-error removed; only the `bazel test` step carries
  it, and a new always() "Classify tier results" step
  (tools/bazel/classify_tier_results.py, with its own unit tests and a Go
  wrapper) computes server_storage_result/cmd_dolt_result from the BEP and
  decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on
  synthetic empty-output inputs that a dead job still fails the gate.
- Corrected the local-actionKey-vs-remote-digest wording in .bazelrc,
  bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md.
- Should-fix: test step 23->25 / job 58 minutes with reasoning; --label
  coverage added to shard_budget_test.py; stale lane comment fixed in
  tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's
  docstring corrected (members bake sole-run's line in directly, never via
  --config=sole-run).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review)

Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack,
julianknutsen):

- classify_tier_results.py now fails every tier shut (exit 1) on a
  non-PASSED label outside every --tier list, so a newly tagged target
  added without updating the step's --tier args cannot silently read as
  success (must-fix 1). Its --tier/--check label sets are policy-tied to
  the BUILD files' own tag sets by a new
  TestBazelDoltIntTierLabelsMatchTheTags.
- A missing or non-integer --rc (a timed-out or cancelled test step)
  takes the same fail-everything-shut branch instead of crashing argparse
  with no outputs written (bee-ghosttrack's should-fix 2).
- check_testcases.py gained a repeatable --label flag, and bazel.yml's
  shared "Every target and shard ran tests" step is split into check-ss
  and check-cd, each continue-on-error and folded into its own tier via
  classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard
  no longer reddens the job on push/nightly (must-fix 2).
- classify's run: argv is now pinned by exact equality
  (bazelDoltIntClassifyRun), and every flag except --tier/--check is
  --StoreOnce; a repeated --tier/--check name or an unrecognized flag is
  now a hard parser error (must-fix 3).
- "Upload test logs" now also fires on steps.test.outcome == 'failure',
  not just failure() (must-fix 4).
- shard_budget.py --label now fails closed (::error, exit 1) on a label
  with no executed shard in the BEP at all, instead of silently reporting
  "no sharded test target executed two or more shards" (should-fix 3,
  carried across both review rounds).
- Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and
  scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the
  nits: stale "Advisory (continue-on-error: true)" wording, the
  bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int,
  "byte-identical" without specifying remote action digest vs. local
  actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch
  for an empty union output.

Verified: python3 -I on all four tools/bazel/*_test.py (classify 29,
check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test
./scripts/... green; bazel --nohome_rc test //scripts:scripts_test
--config=ci green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): shard_budget --label fail-closed only on a truly unknown label

The previous commit's fail-closed --label check compared against
read_shard_durations, which only records executed (non-cached) shards, so
a target whose every shard was a cache hit this run had zero entries and
was wrongly treated the same as a --label typo. This was not caught by
the new tests (none exercised an all-cached target) and broke this PR's
own dolt-int CI run: bazel-dolt-int's Shard balance step went red on
//internal/storage/dolt:dolt_server_full_test with "no executed shard for
it", even though that target was correctly, fully cache-hit.

Fix: a new read_tested_labels() reads every label with any testResult
event (cached or executed), and --label's fail-closed check now tests
against that set instead of read_shard_durations. A fully-cached target
still legitimately reports "nothing to judge" through evaluate(); only a
label absent from the BEP altogether is now an error. Added the missing
test row (every shard cached) that would have caught this before push.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(bazel): pin bazel-dolt-int's Upload test logs if-condition

A full 42-mutation re-run for PR #7535's second review round found that
nothing pinned this job's own "Upload test logs" if-condition (must-fix 4):
the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's
(unlike dolt-race, a brand-new job) had no equivalent assertion, so a future
accidental revert to the old failure()-only condition would go uncaught.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-13 09:17:54 +00:00

586 lines
26 KiB
Go

package scripts_test
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"strconv"
"strings"
"testing"
"gopkg.in/yaml.v3"
)
// Merge queue readiness (engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "Merge
// Queue"). The default-branch ruleset requires "CI Gate / Required" (pr.yml)
// and "PR Risk Gate / Required" (pr-risk.yml); with a merge queue both must
// also report on every queue entry's merge-group commit, from a merge_group
// run that has no github.event.pull_request at all. These tests simulate
// that event with the shared evalGHExpr evaluator (ci_blacksmith_runner_test.go)
// and the workflows' real step scripts.
const (
// Test results are cached (read and written) on every run but
// nightly's: nightly.yml passes fresh-test-results: true, and bazel.yml
// appends BAZEL_FRESH (--config=fresh, --nocache_test_results) last to
// every `bazel test`.
bazelFreshEnv = "${{ inputs.fresh-test-results && '--config=fresh' || '' }}"
bazelFreshArg = `${BAZEL_FRESH:+"$BAZEL_FRESH"}`
bazelFreshRCLine = "test:fresh --nocache_test_results"
mergeQueueActor = "github-merge-queue[bot]"
mergeQueueOwnRepo = "gastownhall/beads"
)
// mergeGroupCtx: an evalGHExpr context for a merge_group run of entry n
// (no github.event.pull_request.* key: null on this event).
func mergeGroupCtx(n int, extra map[string]string) map[string]string {
ctx := map[string]string{
"github.event_name": "merge_group",
"github.repository": mergeQueueOwnRepo,
"github.actor": mergeQueueActor,
"github.ref": fmt.Sprintf("refs/heads/gh-readonly-queue/main/pr-%d-%040d", 7000+n, n),
"github.run_attempt": "1",
"github.event.merge_group.base_sha": fmt.Sprintf("%040d", 100+n),
"github.event.merge_group.head_sha": fmt.Sprintf("%040d", 200+n),
"github.event.merge_group.base_ref": "refs/heads/main",
"github.event.merge_group.head_ref": fmt.Sprintf("refs/heads/gh-readonly-queue/main/pr-%d-%040d", 7000+n, n),
}
for k, v := range extra {
ctx[k] = v
}
return ctx
}
// pullRequestCtx: an evalGHExpr context for a same-repo pull_request run.
func pullRequestCtx(n int, extra map[string]string) map[string]string {
ctx := map[string]string{
"github.event_name": "pull_request",
"github.repository": mergeQueueOwnRepo,
"github.actor": "alice",
"github.ref": fmt.Sprintf("refs/pull/%d/merge", 7000+n),
"github.run_attempt": "1",
"github.event.pull_request.number": strconv.Itoa(7000 + n),
"github.event.pull_request.base.sha": fmt.Sprintf("%040d", 300+n),
"github.event.pull_request.head.sha": fmt.Sprintf("%040d", 400+n),
"github.event.pull_request.head.repo.full_name": mergeQueueOwnRepo,
"github.event.pull_request.head.repo.fork": "false",
}
for k, v := range extra {
ctx[k] = v
}
return ctx
}
// interpolateGH replaces every ${{ ... }} in s with its evalGHExpr value.
func interpolateGH(t *testing.T, s string, ctx map[string]string) string {
t.Helper()
re := regexp.MustCompile(`\$\{\{(.*?)\}\}`)
return re.ReplaceAllStringFunc(s, func(m string) string {
v, err := evalGHExpr(m, ctx)
if err != nil {
t.Fatalf("evalGHExpr(%q): %v", m, err)
}
switch x := v.(type) {
case nil:
return ""
case bool:
return strconv.FormatBool(x)
default:
return fmt.Sprint(x)
}
})
}
// The two required gates: their workflows run on merge_group (exactly the
// checks_requested type), the gate jobs keep their ruleset names and always
// run, and no job's or step's condition in either workflow keys on the
// event, so the merge group runs (or skips) exactly what the decision jobs
// say, like a PR.
func TestMergeQueueRequiredGatesRunOnMergeGroup(t *testing.T) {
gates := map[string]string{"pr.yml": "CI Gate / Required", prRiskWorkflowName: "PR Risk Gate / Required"}
for name, gateName := range gates {
var doc struct {
On map[string]yaml.Node `yaml:"on"`
}
if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+name)), &doc); err != nil {
t.Fatal(err)
}
mg, ok := doc.On["merge_group"]
if !ok {
t.Fatalf("%s has no merge_group trigger; a queued PR would wait forever for %q", name, gateName)
}
var trig struct {
Types []string `yaml:"types"`
}
if err := mg.Decode(&trig); err != nil || strings.Join(trig.Types, ",") != "checks_requested" {
t.Errorf("%s merge_group trigger types = %v (%v), want exactly [checks_requested]", name, trig.Types, err)
}
w := readCIWorkflow(t, name)
gate := w.job(t, "ci-gate")
if gate.Name != gateName || gate.If != "${{ always() }}" {
t.Errorf("%s ci-gate: name %q, if %q; want %q (the ruleset's exact context), if ${{ always() }}", name, gate.Name, gate.If, gateName)
}
// The gate runs on Blacksmith for a merge group (same-repo trust).
if got := mustEvalGHRunsOn(t, gate.RunsOn, mergeGroupCtx(1, nil)); !strings.HasPrefix(got, "blacksmith-") {
t.Errorf("%s ci-gate runs-on for merge_group = %q, want a Blacksmith runner", name, got)
}
event := regexp.MustCompile(`github\.event_name|github\.event\.|github\.base_ref|github\.head_ref`)
for jobName, job := range w.Jobs {
if event.MatchString(job.If) {
t.Errorf("%s job %s if %q keys on the event; decide in a job (bazel.yml's rbe job) and read its outputs", name, jobName, job.If)
}
for _, step := range job.Steps {
if event.MatchString(step.If) {
t.Errorf("%s job %s step %q if %q keys on the event; a merge group must run it like a PR", name, jobName, step.Name, step.If)
}
if strings.Contains(step.Run, "github.base_ref") || strings.Contains(step.Run, "github.head_ref") {
t.Errorf("%s job %s step %q interpolates base_ref/head_ref, which a merge group lacks", name, jobName, step.Name)
}
}
}
}
}
// Concurrency: a merge group's runs key on its own gh-readonly-queue/* ref,
// so two queue entries never share a group (cancel-in-progress would
// otherwise cancel the earlier entry's required checks), and a PR's own
// runs never cancel its queue entry's or the other way round. bazel.yml's
// called-workflow group never cancels a merge group either.
func TestMergeQueueConcurrencyNeverCancelsQueueRuns(t *testing.T) {
for _, name := range []string{"pr.yml", prRiskWorkflowName, bazelWorkflowName} {
var doc struct {
Concurrency struct {
Group string `yaml:"group"`
CancelInProgress any `yaml:"cancel-in-progress"`
} `yaml:"concurrency"`
}
if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+name)), &doc); err != nil {
t.Fatal(err)
}
group := func(ctx map[string]string) string {
c := map[string]string{"github.workflow": strings.TrimSuffix(name, ".yml")}
for k, v := range ctx {
c[k] = v
}
return interpolateGH(t, doc.Concurrency.Group, c)
}
mg1, mg2 := group(mergeGroupCtx(1, nil)), group(mergeGroupCtx(2, nil))
pr1 := group(pullRequestCtx(1, nil))
if mg1 == mg2 || mg1 == pr1 || !strings.Contains(mg1, "gh-readonly-queue/main/pr-7001") {
t.Errorf("%s concurrency group: merge group 1 %q, merge group 2 %q, PR %q; want one group per queue entry, apart from the PR's", name, mg1, mg2, pr1)
}
if name == bazelWorkflowName {
if got := interpolateGH(t, fmt.Sprint(doc.Concurrency.CancelInProgress), mergeGroupCtx(1, nil)); got != "false" {
t.Errorf("%s cancel-in-progress on merge_group = %q, want false", name, got)
}
}
}
}
// Every github.event.pull_request.* read in the required topology either
// falls back to the merge_group field (evaluated below: the queue's SHAs on
// merge_group, the PR's on pull_request) or is listed here with why null is
// right on a merge group. A new read fails until it is classified.
func TestMergeQueuePullRequestFieldsHandleMergeGroup(t *testing.T) {
// path suffix (workflow:YAML path) -> why null on merge_group is right.
nullOK := map[string]string{
"pr.yml:.concurrency.group": "falls back to github.ref (TestMergeQueueConcurrencyNeverCancelsQueueRuns)",
"pr-risk.yml:.concurrency.group": "falls back to github.ref",
"bazel.yml:.concurrency.group": "falls back to github.ref",
"rbe.steps[0].env.FORK": "null == true is false: a merge group is never a fork",
"rbe.steps[0].env.PR_NUMBER": "read only for fork/Dependabot pull_request runs (the rbe-fork mint)",
".env.RBE_FORK_PR": "setup-bazel reads it only in the fork modes, which a merge group never takes",
"bazel-test.steps[*].env.PR_NUMBER": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs",
"bazel-test.steps[*].env.PR_HEAD_SHA": "bazel-sync patch metadata; bazel-autofix.yml ignores non-pull_request runs",
// bazel.yml's CI analytics summary step (rbe-ci-bep-analytics-
// design.md, S3), one entry per lane job (ciAnalyticsLaneJobs in
// ci_analytics_workflow_test.go) rather than a bare "steps[*]...",
// so this stays scoped to bazel.yml's own PR_HINT and can't
// silently also cover a future PR_HINT read elsewhere.
"bazel-test.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-pure.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-embedded.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-integration.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-doltserver.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-proxied.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-server-storage.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-cmd-dolt.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-dolt-race.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
"bazel-dolt-int.steps[*].env.PR_HINT": "ci-analytics-summary metadata only (--pr-hint); `|| 0` already covers merge_group same as a non-PR push",
}
used := map[string]bool{}
var fallbacks []string
for _, name := range []string{"pr.yml", prRiskWorkflowName, bazelWorkflowName} {
walkYAML(readYAMLNode(t, filepath.Join(".github", "workflows", name)), "", func(path string, key bool, value string) {
if key || !strings.Contains(value, "github.event.pull_request.") {
return
}
// The Blacksmith venue ternaries name merge_group explicitly
// (TestSameRepoBlacksmithExpressionSemantics).
if strings.HasSuffix(path, ".runs-on") && strings.Contains(value, "github.event_name == 'merge_group' ||") {
return
}
if strings.Contains(value, "github.event.merge_group.") {
fallbacks = append(fallbacks, value)
return
}
for suffix := range nullOK {
// steps[*] matches any step index (the sync step moves when
// steps are added before it).
re := regexp.MustCompile(regexp.QuoteMeta(suffix) + "$")
if strings.Contains(suffix, "[*]") {
re = regexp.MustCompile(strings.ReplaceAll(regexp.QuoteMeta(suffix), `\[\*\]`, `\[[0-9]+\]`) + "$")
}
if re.MatchString(name + ":" + path) {
used[suffix] = true
return
}
}
t.Errorf("%s %s reads %q, which is null on merge_group: add a github.event.merge_group.* fallback or classify it here", name, path, value)
})
}
for suffix, why := range nullOK {
if !used[suffix] {
t.Errorf("stale merge_group classification %q (%s): nothing matches it", suffix, why)
}
}
if len(fallbacks) < 3 {
t.Fatalf("found %d merge_group fallbacks, want at least the migration-hygiene BASE_SHA and both package gates' bounds", len(fallbacks))
}
for _, expr := range fallbacks {
mg, pr := interpolateGH(t, expr, mergeGroupCtx(1, nil)), interpolateGH(t, expr, pullRequestCtx(1, nil))
if !strings.HasPrefix(mg, "0000") || mg == pr || !strings.HasPrefix(pr, "0000") {
t.Errorf("%q: merge_group %q, pull_request %q; want the queue's SHA and the PR's", expr, mg, pr)
}
}
}
// Merge groups: bazel.yml's rbe step, evaluated by evalGHExpr for a merge
// group, takes mode remote with the CI secrets, so pr.yml's gate (which
// requires every lane to have run remotely, BAZEL_REMOTE) can pass. Without
// the secret or with the farm switch off the mode is cache or skip, and the
// gate is red (TestPRGateRequiresRemoteBazelLanes).
func TestMergeQueueBazelRunsRemotely(t *testing.T) {
requireHostTool(t, "bash")
rbe := readCIWorkflow(t, bazelWorkflowName).job(t, bazelRBEJobName)
for _, c := range []struct {
name, varOn, secret, mode string
}{
{"farm on, secrets", "true", "grpcs://rbe.example:443", "remote"},
{"secret missing", "true", "", "cache"},
{"farm switch off", "", "grpcs://rbe.example:443", "skip"},
} {
ctx := mergeGroupCtx(1, map[string]string{"vars.RBE_WEST_WORKERS": c.varOn, "secrets.RBE_WEST_EXECUTOR": c.secret, "inputs.rbe": "on", "inputs.fork-farm": "off"})
env := map[string]string{}
for k, v := range rbe.Steps[0].Env {
env[k] = interpolateGH(t, v, ctx)
}
out, log, err := runBazelRBEDecisionLog(t, rbe.Steps[0].Run, env)
if err != nil {
t.Fatalf("rbe on merge_group (%s): %v", c.name, err)
}
if out["mode"] != c.mode || log != "" {
t.Errorf("rbe on merge_group (%s): mode %q (mint asked: %q), want %s without asking rbe-fork", c.name, out["mode"], log, c.mode)
}
if got := mustEvalGHRunsOn(t, rbe.RunsOn, ctx); !strings.HasPrefix(got, "blacksmith-") {
t.Errorf("rbe runs-on for merge_group = %q, want Blacksmith", got)
}
}
}
// Test result caching: every `bazel test` in bazel.yml reuses cached
// results except where the caller asks for fresh ones. BAZEL_FRESH is one
// workflow-level env, --config=fresh only when inputs.fresh-test-results is
// true (nightly.yml passes it; nothing else does), appended after every
// other --config of each `bazel test` so its --nocache_test_results wins;
// no lane config turns caching off; and nothing adds retries (eviction retries stay 0 where
// set, TestBazelRetiredLanesArePinned / TestBazelPRLanesArePinned).
func TestBazelTestResultCachingPolicy(t *testing.T) {
var doc struct {
Env map[string]string `yaml:"env"`
On struct {
Dispatch struct {
Inputs map[string]struct {
Type string `yaml:"type"`
Default any `yaml:"default"`
} `yaml:"inputs"`
} `yaml:"workflow_dispatch"`
Call struct {
Inputs map[string]struct {
Type string `yaml:"type"`
Default any `yaml:"default"`
} `yaml:"inputs"`
} `yaml:"workflow_call"`
} `yaml:"on"`
}
if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+bazelWorkflowName)), &doc); err != nil {
t.Fatal(err)
}
if doc.Env["BAZEL_FRESH"] != bazelFreshEnv {
t.Errorf("%s env BAZEL_FRESH = %q, want %q", bazelWorkflowName, doc.Env["BAZEL_FRESH"], bazelFreshEnv)
}
for name, in := range map[string]struct {
Type string `yaml:"type"`
Default any `yaml:"default"`
}{"workflow_call": doc.On.Call.Inputs["fresh-test-results"], "workflow_dispatch": doc.On.Dispatch.Inputs["fresh-test-results"]} {
if in.Type != "boolean" || in.Default != false {
t.Errorf("%s input fresh-test-results: type %q default %v; want boolean, default false", name, in.Type, in.Default)
}
}
for _, c := range []struct {
name string
v any
want string
}{
{"nightly (true)", true, "--config=fresh"},
{"default (false)", false, ""},
{"push (null)", nil, ""},
} {
got, err := evalGHExprTyped(bazelFreshEnv, map[string]any{"inputs.fresh-test-results": c.v})
if err != nil {
t.Fatal(err)
}
if got != c.want {
t.Errorf("%s: BAZEL_FRESH = %#v, want %q", c.name, got, c.want)
}
}
w := readCIWorkflow(t, bazelWorkflowName)
bazelTest := regexp.MustCompile(`^\s*bazel\s+test\b`)
tests := 0
for name, job := range w.Jobs {
if _, ok := job.Env["BAZEL_FRESH"]; ok {
t.Errorf("%s sets BAZEL_FRESH; only the workflow env may", name)
}
for _, step := range job.Steps {
if strings.Contains(step.Run, "--config=fresh") || strings.Contains(step.Run, "cache_test_results") || strings.Contains(step.Run, "BAZEL_REUSE_RESULTS") {
t.Errorf("%s step %q names result caching directly; only BAZEL_FRESH may", name, step.Name)
}
for _, line := range strings.Split(step.Run, "\n") {
if !bazelTest.MatchString(line) {
continue
}
tests++
i := strings.Index(line, bazelFreshArg)
if i < 0 || strings.Contains(line[i:], "--config") || strings.Contains(line[i:], "SOLE_RUN") {
t.Errorf("%s step %q: %q must pass %s after every --config", name, step.Name, line, bazelFreshArg)
}
}
}
}
if tests < 9 {
t.Errorf("found %d `bazel test` lines in %s, want every lane's (at least 9)", tests, bazelWorkflowName)
}
nightly := readCIWorkflow(t, "nightly.yml").job(t, "bazel")
if nightly.With["fresh-test-results"] != "true" {
t.Errorf("nightly.yml's bazel call with fresh-test-results = %q, want true (the run that samples flakes)", nightly.With["fresh-test-results"])
}
for _, name := range []string{"pr.yml", "bazel-farm.yml"} {
for jobName, job := range readCIWorkflow(t, name).Jobs {
if _, ok := job.With["fresh-test-results"]; ok {
t.Errorf("%s %s passes fresh-test-results; only nightly re-executes every test", name, jobName)
}
}
}
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
var lines []string
for _, line := range strings.Split(rc, "\n") {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "#") {
continue
}
head, _, _ := strings.Cut(line, " ")
if strings.HasSuffix(head, ":fresh") {
lines = append(lines, line)
}
if strings.Contains(line, "cache_test_results") && line != bazelFreshRCLine {
t.Errorf(".bazelrc %q: only test:fresh sets test result caching", line)
}
}
if strings.Join(lines, "\n") != bazelFreshRCLine {
t.Errorf(".bazelrc --config=fresh = %q, want exactly %q", lines, bazelFreshRCLine)
}
}
// detect-package-gates.sh on merge_group diffs the queue's own bounds
// (merge_group.base_sha..head_sha), not HEAD^: a rebase-method entry
// stacks every commit of the PR, and only the first touches a package.
func TestMergeQueueDetectPackageGatesUsesQueueBounds(t *testing.T) {
requireHostTool(t, "git")
requireHostTool(t, "bash")
dir, base, head := mergeQueueRepo(t, []string{"npm-package/package.json"}, []string{"README.md"})
script := filepath.Join(sourceRepoRoot(t), "scripts/ci/detect-package-gates.sh")
run := func(env ...string) map[string]string {
t.Helper()
out := filepath.Join(t.TempDir(), "out")
cmd := exec.Command("bash", script)
cmd.Dir = dir
cmd.Env = append([]string{"PATH=" + os.Getenv("PATH"), "GITHUB_OUTPUT=" + out, "GITHUB_EVENT_NAME=merge_group"}, env...)
if b, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("detect-package-gates.sh: %v\n%s", err, b)
}
data, err := os.ReadFile(out)
if err != nil {
t.Fatal(err)
}
got := map[string]string{}
for _, l := range strings.Split(strings.TrimSpace(string(data)), "\n") {
k, v, _ := strings.Cut(l, "=")
got[k] = v
}
return got
}
if got := run("PR_BASE_SHA="+base, "PR_HEAD_SHA="+head); got["npm_package"] != "true" || got["mcp_package"] != "false" {
t.Errorf("merge group whose first commit changes npm-package/: %v, want npm_package only", got)
}
if got := run(); got["npm_package"] != "true" || got["mcp_package"] != "true" {
t.Errorf("merge group without bounds: %v, want every gate", got)
}
if got := run("PR_BASE_SHA="+head+"^", "PR_HEAD_SHA="+head); got["npm_package"] != "false" {
t.Errorf("merge group touching only README.md: %v, want no gate", got)
}
}
// fast-checks' .beads guard runs on merge_group against the queue's base
// commit and on pull_request against origin/<base branch>.
func TestMergeQueueBeadsGuardRuns(t *testing.T) {
requireHostTool(t, "git")
requireHostTool(t, "bash")
step := readCIWorkflow(t, "pr.yml").job(t, "fast-checks").step(t, "Check for .beads/issues.jsonl changes")
for _, c := range []struct {
name string
changed string
pass bool
}{
{"unrelated change", "README.md", true},
{".beads change", ".beads/issues.jsonl", false},
} {
dir, base, _ := mergeQueueRepo(t, []string{c.changed})
gitIn(t, dir, "update-ref", "refs/remotes/origin/main", base)
for event, env := range map[string][]string{
"merge_group": {"MERGE_GROUP_BASE_SHA=" + interpolateGH(t, step.Env["MERGE_GROUP_BASE_SHA"], mergeGroupCtx(1, map[string]string{"github.event.merge_group.base_sha": base}))},
"pull_request": {"MERGE_GROUP_BASE_SHA=" + interpolateGH(t, step.Env["MERGE_GROUP_BASE_SHA"], pullRequestCtx(1, nil)), "GITHUB_BASE_REF=main"},
} {
// GitHub's default shell for a run step without `shell:`.
cmd := exec.Command("bash", "--noprofile", "--norc", "-e", "-c", step.Run)
cmd.Dir = dir
cmd.Env = append([]string{"PATH=" + os.Getenv("PATH")}, env...)
out, err := cmd.CombinedOutput()
if (err == nil) != c.pass {
t.Errorf("%s, %s: err %v, want pass %v\n%s", event, c.name, err, c.pass, out)
}
}
}
}
// The guard fails closed when its diff base is missing (a failed git diff
// piped into grep -q would otherwise pass).
func TestMergeQueueBeadsGuardFailsWithoutBase(t *testing.T) {
requireHostTool(t, "git")
requireHostTool(t, "bash")
step := readCIWorkflow(t, "pr.yml").job(t, "fast-checks").step(t, "Check for .beads/issues.jsonl changes")
dir, _, _ := mergeQueueRepo(t, []string{"README.md"})
for name, env := range map[string][]string{
"merge_group, unknown base_sha": {"MERGE_GROUP_BASE_SHA=" + strings.Repeat("e", 40)},
"pull_request, no origin/main": {"GITHUB_BASE_REF=main"},
} {
cmd := exec.Command("bash", "--noprofile", "--norc", "-e", "-c", step.Run)
cmd.Dir = dir
cmd.Env = append([]string{"PATH=" + os.Getenv("PATH")}, env...)
if out, err := cmd.CombinedOutput(); err == nil || !strings.Contains(string(out), "unavailable") {
t.Errorf("%s: err %v, want a failure naming the missing base\n%s", name, err, out)
}
}
}
// check-doc-flags' CLI-docs drift attribution needs a diff base; a merge
// group has no GITHUB_BASE_REF, so the step passes the queue's base commit
// (and nothing on pull_request, where origin/$GITHUB_BASE_REF applies).
// Without it, drift already on main fails every queue entry.
func TestMergeQueueDocDriftHasDiffBase(t *testing.T) {
step := readCIWorkflow(t, "pr.yml").job(t, "check-doc-flags").step(t, "Validate docs against CLI")
expr := step.Env["BD_DOCS_DIFF_BASE"]
if got := interpolateGH(t, expr, mergeGroupCtx(1, nil)); got != fmt.Sprintf("%040d", 101) {
t.Errorf("check-doc-flags BD_DOCS_DIFF_BASE on merge_group = %q (%q), want merge_group.base_sha", got, expr)
}
if got := interpolateGH(t, expr, pullRequestCtx(1, nil)); got != "" {
t.Errorf("check-doc-flags BD_DOCS_DIFF_BASE on pull_request = %q, want empty (origin/$GITHUB_BASE_REF)", got)
}
drift := readPolicyFile(t, sourceRepoRoot(t), "scripts/check-cli-docs-drift.sh")
if !strings.Contains(drift, `BASE_REF="${BD_DOCS_DIFF_BASE:-}"`) {
t.Error("check-cli-docs-drift.sh no longer reads BD_DOCS_DIFF_BASE")
}
}
// The shared evaluator follows GitHub's loose equality, so a boolean input
// compared with a string ('true') is false, as on GitHub, instead of
// masking that classic workflow_call bug.
func TestGHExprLooseEquality(t *testing.T) {
for _, c := range []struct {
expr string
ctx map[string]any
want bool
}{
{"${{ inputs.b == 'true' }}", map[string]any{"inputs.b": true}, false},
{"${{ inputs.b == true }}", map[string]any{"inputs.b": true}, true},
{"${{ inputs.b == false }}", map[string]any{"inputs.b": false}, true},
{"${{ inputs.b == '' }}", map[string]any{"inputs.b": false}, true},
{"${{ github.event.pull_request.head.repo.fork == true }}", map[string]any{}, false},
{"${{ github.event.pull_request.number == '' }}", map[string]any{}, true},
{"${{ vars.X == 'true' }}", map[string]any{"vars.X": "TRUE"}, true},
{"${{ github.run_attempt != '1' }}", map[string]any{"github.run_attempt": "2"}, true},
} {
got, err := evalGHExprTyped(c.expr, c.ctx)
if err != nil {
t.Fatal(err)
}
if got != c.want {
t.Errorf("%s with %v = %v, want %v", c.expr, c.ctx, got, c.want)
}
}
}
// mergeQueueRepo: a git repo with a base commit and one commit per file
// group on top; returns its dir, the base SHA and the head SHA.
func mergeQueueRepo(t *testing.T, commits ...[]string) (dir, base, head string) {
t.Helper()
dir = t.TempDir()
gitIn(t, dir, "init", "-q", "-b", "main")
write := func(rel, content string) {
p := filepath.Join(dir, rel)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
write("base.txt", "base\n")
gitIn(t, dir, "add", "-A")
gitIn(t, dir, "commit", "-q", "-m", "base")
base = strings.TrimSpace(gitIn(t, dir, "rev-parse", "HEAD"))
for i, files := range commits {
for _, f := range files {
write(f, fmt.Sprintf("change %d\n", i))
}
gitIn(t, dir, "add", "-A")
gitIn(t, dir, "commit", "-q", "-m", fmt.Sprintf("change %d", i))
}
head = strings.TrimSpace(gitIn(t, dir, "rev-parse", "HEAD"))
return dir, base, head
}
func gitIn(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL="+os.DevNull,
"GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.com", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.com")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return string(out)
}