Compare commits

...
Sign in to create a new pull request.

1 commit

Author SHA1 Message Date
Gas City Adopt-PR
7252978fca ci(bazel): require the integration lane on same-repo PRs
pr.yml's bazel.yml call now passes integration: "on", and its ci-gate
requires BAZEL_INTEGRATION (bazel-integration's job.status output, a
missing result reading as skipped). The lane stays remote-only:
bazel-gate.sh allows it to skip in modes local (fork and Dependabot PRs,
until the farm has a read-only cache for them) and skip, never in remote.
bazel-farm.yml's advisory call runs it too.

The integration test step gets 45 minutes (job 55) for cold PR compiles;
the policy test keeps it at or above a 20-minute cold compile plus
.bazelrc's test:integration --test_timeout (the 1200s per-action cap).
The integration input (default "on") stays for callers that want the lane
off; pr.yml's call inputs are pinned, and the gate simulation now covers
the lane skipped, failed and cancelled in each mode.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 04:55:45 +00:00
7 changed files with 123 additions and 69 deletions

View file

@ -8,7 +8,8 @@
# in this mode, for CI_GATE_SKIPPED_OK:
# skip: every lane and the aggregate (BAZEL)
# local: the remote-only BAZEL_EMBEDDED,
# BAZEL_PROXIED, BAZEL_SERVER_STORAGE
# BAZEL_INTEGRATION, BAZEL_PROXIED,
# BAZEL_SERVER_STORAGE
# remote: none
# bazel-gate.sh aggregate the value to gate on for BAZEL: the call's
# aggregate result, or, when the mode is missing
@ -36,8 +37,8 @@ case "${1:-}" in
skips=()
if [[ "$valid" == true ]]; then
case "$mode" in
skip) skips+=(BAZEL BAZEL_TEST BAZEL_PURE BAZEL_EMBEDDED BAZEL_DOLTSERVER BAZEL_PROXIED BAZEL_SERVER_STORAGE) ;;
local) skips+=(BAZEL_EMBEDDED BAZEL_PROXIED BAZEL_SERVER_STORAGE) ;;
skip) skips+=(BAZEL BAZEL_TEST BAZEL_PURE BAZEL_EMBEDDED BAZEL_INTEGRATION BAZEL_DOLTSERVER BAZEL_PROXIED BAZEL_SERVER_STORAGE) ;;
local) skips+=(BAZEL_EMBEDDED BAZEL_INTEGRATION BAZEL_PROXIED BAZEL_SERVER_STORAGE) ;;
esac
fi
echo "${skips[*]-}"

View file

@ -8,10 +8,11 @@
#
# ADVISORY. pr.yml is unchanged: every fork PR, listed or not, still runs the
# local Bazel lanes inside the enforced `CI Gate / Required`. This run adds a
# fast remote signal, including the remote-only embedded, proxied-server and
# server-storage tiers, that no required check and no privileged workflow
# reads. Making it authoritative needs a trusted reporter; see
# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "Trusted-author fork PRs".
# fast remote signal, including the remote-only embedded, integration,
# proxied-server and server-storage tiers, that no required check and no
# privileged workflow reads. Making it authoritative needs a trusted
# reporter; see engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "Trusted-author fork
# PRs".
#
# SECURITY MODEL (policy-tested in scripts/ci_workflow_test.go,
# TestBazelFarmWorkflowSecurity; threat model in the engdoc):
@ -128,5 +129,5 @@ jobs:
with:
checkout-sha: ${{ github.event.pull_request.head.sha }}
fork-farm: authorized
integration: "off"
integration: "on"
build-artifact-name: bazel-farm-build-artifacts

View file

@ -10,16 +10,19 @@
# runs the lane once. Every lane reports its job.status as a workflow_call
# output (named after the job) from an always() last step. pr.yml's ci-gate
# requires the call's aggregate result (BAZEL) and bazel-test, bazel-pure,
# bazel-embedded, bazel-doltserver, bazel-proxied and bazel-server-storage
# (BAZEL_TEST, BAZEL_PURE, BAZEL_EMBEDDED, BAZEL_DOLTSERVER, BAZEL_PROXIED,
# BAZEL_SERVER_STORAGE); .github/scripts/bazel-gate.sh turns the rbe job's
# rbe-mode output into the skips it accepts, so a lane skipped by its `if:`
# below passes and one that should have run and failed, was cancelled or
# reported nothing fails the gate. bazel-integration is not part of the PR
# call (pr.yml passes integration: "off"): its legacy counterparts, main.yml's
# integration jobs, run only on push to main, so it runs on push to main,
# dispatch and nightly, like them. Direct runs (push to main,
# workflow_dispatch, nightly.yml's call) gate nothing.
# bazel-embedded, bazel-integration, bazel-doltserver, bazel-proxied and
# bazel-server-storage (BAZEL_TEST, BAZEL_PURE, BAZEL_EMBEDDED,
# BAZEL_INTEGRATION, BAZEL_DOLTSERVER, BAZEL_PROXIED, BAZEL_SERVER_STORAGE);
# .github/scripts/bazel-gate.sh turns the rbe job's rbe-mode output into the
# skips it accepts, so a lane skipped by its `if:` below passes and one that
# should have run and failed, was cancelled or reported nothing fails the
# gate. bazel-integration runs in the PR call (pr.yml passes integration:
# "on") although its legacy counterparts, main.yml's integration jobs, run
# only on push to main: remotely it is minutes, and it catches on the PR what
# main.yml would catch after the merge. Like every remote-only lane it skips
# for fork and Dependabot PRs (mode local); forks get it once the farm has a
# read-only cache for them. Direct runs (push to main, workflow_dispatch,
# nightly.yml's call) gate nothing.
#
# Execution mode, decided once by the rbe job (every other job needs it and
# reads its outputs; none re-derives the condition) and passed to
@ -130,8 +133,8 @@ on:
# test would silently stop the lane on main.
integration:
description: >
"off" skips bazel-integration (pr.yml: its legacy counterparts do
not run on PRs); anything else runs it where remote execution is on.
"off" skips bazel-integration; anything else (pr.yml and
bazel-farm.yml pass "on") runs it where remote execution is on.
type: string
required: false
default: "on"
@ -192,7 +195,7 @@ on:
description: "bazel-embedded's job.status"
value: ${{ jobs.bazel-embedded.outputs.result }}
bazel-integration:
description: "bazel-integration's job.status (empty in pr.yml's call, which turns the lane off)"
description: "bazel-integration's job.status"
value: ${{ jobs.bazel-integration.outputs.result }}
bazel-doltserver:
description: "bazel-doltserver's job.status"
@ -737,17 +740,19 @@ jobs:
# selection (go list without the integration tag, -short, ^TestEmbedded
# skipped), and its job mode needs the integration jobs' go test -json,
# which no run here has. check_testcases.py still catches a target or
# shard that ran no tests.
# shard that ran no tests. pr.yml's gate requires it (BAZEL_INTEGRATION)
# on same-repo PRs; see the header.
bazel-integration:
name: integration (advisory)
name: integration
# Remote execution only, like bazel-embedded: locally the tagged race
# build of //... plus 14 jobs' worth of tests on one runner is well over
# the timeout, for tests main.yml's integration jobs already run. Off in
# pr.yml's call (integration: "off"), as those jobs are off on PRs.
# the timeout, for tests main.yml's integration jobs already run. A
# caller may turn it off with integration: "off"; pr.yml and
# bazel-farm.yml pass "on".
needs: rbe
runs-on: ${{ needs.rbe.outputs.enabled == 'true' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
if: ${{ needs.rbe.outputs.enabled == 'true' && inputs.integration != 'off' }}
timeout-minutes: 40
timeout-minutes: 55
outputs:
result: ${{ steps.result.outputs.result }}
steps:
@ -770,11 +775,13 @@ jobs:
RBE_TLS_CA: ${{ needs.rbe.outputs.enabled == 'true' && secrets.RBE_TLS_CA || '' }}
RBE_INSTANCE: ${{ needs.rbe.outputs.enabled == 'true' && 'oss' || '' }}
# Remote: a cold compile plus the longest test action (1200s cap, see
# .bazelrc) fits well inside 30 minutes.
# Remote: a cold compile (a PR touching a widely imported package; the
# first cold run took 17 minutes end to end, warm ones ~3.5) plus the
# longest test action (1200s cap, see .bazelrc) if it starts last.
# scripts/ci_workflow_test.go keeps this at or above that sum.
- name: bazel test //... --config=integration
id: test
timeout-minutes: 30
timeout-minutes: 45
run: |
set -o pipefail
start=$(date +%s)
@ -803,8 +810,9 @@ jobs:
if-no-files-found: ignore
retention-days: 7
# Last step: the job's result as a workflow_call output (for
# symmetry; pr.yml's call turns this lane off).
# Last step: the job's result as a workflow_call output for pr.yml's
# ci-gate. always() runs it after a failure, a timeout or a
# cancellation too.
- name: Record job result
id: result
if: ${{ always() }}

View file

@ -1764,10 +1764,11 @@ jobs:
# this call is its only PR run. It gets the four RBE secrets and nothing
# else; its rbe job decides remote, local (forks, Dependabot: no
# remote-only lanes) or skip, and exports that for ci-gate. Its
# bazel-integration lane is off here (main.yml's integration jobs, its
# legacy counterpart, run only on push to main; bazel.yml runs it there).
# Its bd artifact is renamed so it cannot collide with build-artifacts'
# upload.
# bazel-integration lane (main.yml's integration jobs, which run only on
# push to main) is on and required here, remote only: same-repo PRs run
# it, fork and Dependabot PRs (mode local) skip it until the farm has a
# read-only cache for them. Its bd artifact is renamed so it cannot
# collide with build-artifacts' upload.
bazel:
name: Bazel
permissions:
@ -1780,7 +1781,7 @@ jobs:
RBE_TLS_CA: ${{ secrets.RBE_TLS_CA }}
with:
build-artifact-name: bazel-ci-build-artifacts
integration: "off"
integration: "on"
ci-gate:
name: CI Gate / Required
@ -1845,6 +1846,7 @@ jobs:
BAZEL_TEST
BAZEL_PURE
BAZEL_EMBEDDED
BAZEL_INTEGRATION
BAZEL_DOLTSERVER
BAZEL_PROXIED
BAZEL_SERVER_STORAGE
@ -1877,6 +1879,7 @@ jobs:
BAZEL_TEST: ${{ needs.bazel.outputs.bazel-test || 'skipped' }}
BAZEL_PURE: ${{ needs.bazel.outputs.bazel-pure || 'skipped' }}
BAZEL_EMBEDDED: ${{ needs.bazel.outputs.bazel-embedded || 'skipped' }}
BAZEL_INTEGRATION: ${{ needs.bazel.outputs.bazel-integration || 'skipped' }}
BAZEL_DOLTSERVER: ${{ needs.bazel.outputs.bazel-doltserver || 'skipped' }}
BAZEL_PROXIED: ${{ needs.bazel.outputs.bazel-proxied || 'skipped' }}
BAZEL_SERVER_STORAGE: ${{ needs.bazel.outputs.bazel-server-storage || 'skipped' }}
@ -1892,9 +1895,9 @@ jobs:
skipped_ok="CHECK_NO_BEADS_CHANGES"
fi
# Bazel lanes skipped by design in the call's execution mode (all
# of them in mode skip, the remote-only embedded, proxied and
# server-storage tiers in mode local), and the aggregate unless the
# mode is missing or invalid.
# of them in mode skip, the remote-only embedded, integration,
# proxied and server-storage tiers in mode local), and the
# aggregate unless the mode is missing or invalid.
skipped_ok="$skipped_ok $(bash .github/scripts/bazel-gate.sh skips)"
BAZEL="$(bash .github/scripts/bazel-gate.sh aggregate)"
export BAZEL

View file

@ -47,20 +47,32 @@ Current PR-related workflow names:
exports it as the `rbe-mode` / `rbe-enabled` outputs; every lane exports its
`job.status` as an output named after the job. `pr.yml`'s gate requires the
call's result (`BAZEL`) and `BAZEL_TEST`, `BAZEL_PURE`, `BAZEL_EMBEDDED`,
`BAZEL_DOLTSERVER`, `BAZEL_PROXIED` and `BAZEL_SERVER_STORAGE`. The legacy jobs these mirror stay required in `pr.yml`
and `pr-risk.yml`.
`BAZEL_INTEGRATION`, `BAZEL_DOLTSERVER`, `BAZEL_PROXIED` and
`BAZEL_SERVER_STORAGE`. The legacy jobs these mirror stay required in
`pr.yml` and `pr-risk.yml`.
`.github/scripts/bazel-gate.sh` reads the exported mode, never the variable
or the fork flag: mode `skip` allows every Bazel id to skip, mode `local`
allows only the remote-only `BAZEL_EMBEDDED`, `BAZEL_PROXIED` and
`BAZEL_SERVER_STORAGE` (fork and Dependabot PRs rely on `pr-risk.yml`'s
legacy tiers for those), and mode `remote` allows none.
allows only the remote-only `BAZEL_EMBEDDED`, `BAZEL_INTEGRATION`,
`BAZEL_PROXIED` and `BAZEL_SERVER_STORAGE` (fork and Dependabot PRs rely on
`pr-risk.yml`'s legacy tiers for the embedded, proxied and server-storage
tiers), and mode `remote` allows none.
A lane that should run and fails, is cancelled, or reports no result fails
the gate, and so does a missing or invalid mode.
`bazel-integration` is not part of the PR call (`pr.yml` passes
`integration: "off"`) and has no gate id: its legacy counterparts
(`main.yml`'s integration jobs) run only on push to `main`, and so does it
(plus dispatch and nightly). `scripts/ci_workflow_test.go` fails when a new
`bazel.yml` job has neither a gate id nor an advisory entry.
`bazel-integration` (`bazel test //... --config=integration`, the Bazel
twin of `main.yml`'s Linux integration shards) is required on same-repo
PRs: `pr.yml` passes `integration: "on"` (policy-pinned), so in mode
`remote` it runs, and a failure, cancellation or missing result turns the
gate red. Its legacy twins still run only on push to `main`, so this lane
is the only PR-time run of the full integration-tagged suite. Fork and
Dependabot PRs (mode `local`) skip it, with no legacy fallback on the PR:
they get integration coverage only after merge, from `main.yml`. Running it
for forks waits on a read-only remote cache they may use; that change is
making the lane local-capable for forks in `bazel.yml` and dropping
`BAZEL_INTEGRATION` from `bazel-gate.sh`'s mode-`local` skips. Its step
timeout (45 minutes) covers a cold compile plus the 1200 s per-action cap
(policy-tested against `.bazelrc`); warm runs take about 3.5 minutes.
`scripts/ci_workflow_test.go` fails when a new `bazel.yml` job has neither
a gate id nor an advisory entry.
Runbook: if `main`'s Bazel BUILD files drift (every PR's Bazel lanes go
red, and autofix only patches packages the PR itself changed), the RBE
farm is down, or the beads CI RBE client certificate expires (about
@ -152,7 +164,7 @@ fork PRs are unchanged.
- `farm` job: calls `bazel.yml` only when `authorize` allowed it, with
`contents: read`, exactly the four RBE secrets, `checkout-sha` =
`github.event.pull_request.head.sha`, `fork-farm: authorized`, and
`integration: "off"`. `bazel.yml` checks out that SHA in every lane with
`integration: "on"`. `bazel.yml` checks out that SHA in every lane with
`persist-credentials: false`. actions/checkout v7 refuses to check out a
fork PR's head on `pull_request_target` unless `allow-unsafe-pr-checkout`
is true. Each lane sets that input to
@ -170,7 +182,7 @@ fork PRs are unchanged.
- Gate: advisory. `pr.yml` is unchanged. Every fork PR, listed or not, still
runs the local Bazel lanes inside the enforced `CI Gate / Required`, and
the farm run adds a faster signal that also covers the remote-only
embedded, proxied-server and server-storage tiers.
embedded, integration, proxied-server and server-storage tiers.
- Policy tests: `scripts/bazel_farm_workflow_test.go` pins the trigger,
both jobs' conditions, the allowlist source, the pinned checkout, the
permissions, where secrets appear, and that no expression is interpolated

View file

@ -215,7 +215,7 @@ func TestBazelFarmWorkflowSecurity(t *testing.T) {
wantWith := map[string]string{
"checkout-sha": "${{ github.event.pull_request.head.sha }}",
"fork-farm": "authorized",
"integration": "off",
"integration": "on",
"build-artifact-name": "bazel-farm-build-artifacts",
}
if !reflect.DeepEqual(farm.With, wantWith) {

View file

@ -1745,23 +1745,31 @@ var bazelLaneGateIDs = map[string]string{
// pr-risk.yml's legacy jobs, like the embedded tier's).
bazelProxiedJobName: "BAZEL_PROXIED",
bazelServerJobName: "BAZEL_SERVER_STORAGE",
// main.yml's integration jobs, required on same-repo PRs although their
// legacy twins run only on push to main. Remote-only: fork and
// Dependabot PRs skip it until the farm has a read-only cache for them
// (then it becomes local-capable for forks, here and in bazel-gate.sh).
bazelIntegJobName: "BAZEL_INTEGRATION",
}
var bazelAdvisoryLanes = map[string]string{
bazelIntegJobName: "its legacy counterparts, main.yml's integration jobs, run only on push to main",
}
// None today: every lane is gated. Kept so a future lane that pr.yml's call
// turns off has a place to record why.
var bazelAdvisoryLanes = map[string]string{}
// bazel-integration's if: remote only, and off when the caller passes
// integration: "off" (pr.yml). A string input: on push and dispatch it is
// null, and null != 'off', so the lane keeps running on main.
// bazel-integration's if: remote only, and off when a caller passes
// integration: "off" (pr.yml and bazel-farm.yml pass "on"). A string input:
// on push and dispatch it is null, and null != 'off', so the lane keeps
// running on main.
const bazelIntegIf = "${{ needs.rbe.outputs.enabled == 'true' && inputs.integration != 'off' }}"
// pr.yml's call of bazel.yml: exactly these inputs (review D1 v2 N3). An rbe
// override would put every PR in local mode and ungate the embedded tier
// while the gate stays self-consistent.
// while the gate stays self-consistent; integration: "off" would drop the
// required integration lane (explicit "on", so the pin and the gate
// simulation, which reads these inputs, do not depend on the default).
var bazelPRCallWith = map[string]string{
"build-artifact-name": "bazel-ci-build-artifacts",
"integration": "off",
"integration": "on",
}
// The call's aggregate result (needs.bazel.result, through bazel-gate.sh).
@ -2259,9 +2267,10 @@ func runPRGateStep(t *testing.T, step ciWorkflowStep, sc bazelGateScenario) (boo
// The gate over every execution mode (review D1 F3): the skip script allows
// exactly the skips the lanes' own `if:`s produce in that mode, and for every
// lane that should run, that lane alone skipped, failed or cancelled turns
// pr.yml's actual gate step red, whatever the aggregate says. The advisory
// integration lane's failure alone keeps it green; a missing or inconsistent
// mode, or an aggregate failure no lane explains, turns it red.
// pr.yml's actual gate step red, whatever the aggregate says (in mode remote
// that includes the integration lane, which skips green in mode local); a
// missing or inconsistent mode, or an aggregate failure no lane explains,
// turns it red.
func TestBazelGateSimulation(t *testing.T) {
requireHostTool(t, "bash")
root := sourceRepoRoot(t)
@ -3003,12 +3012,32 @@ func bazelRuleBlock(build, name string) string {
func TestBazelIntegrationJob(t *testing.T) {
workflow := readCIWorkflow(t, bazelWorkflowName)
job := workflow.job(t, bazelIntegJobName)
if job.TimeoutMinutes > 45 {
t.Errorf("%s timeout-minutes = %d; it runs remotely only, keep it near the step's 30", bazelIntegJobName, job.TimeoutMinutes)
}
test := job.step(t, "bazel test //... --config=integration")
if test.TimeoutMinutes == 0 || test.TimeoutMinutes >= job.TimeoutMinutes {
t.Errorf("%s test step timeout-minutes = %d, want set and below the job's %d", bazelIntegJobName, test.TimeoutMinutes, job.TimeoutMinutes)
// A required PR lane: the step must fit a cold compile (the first
// GitHub run took 17 minutes end to end) followed by the longest test
// action, which .bazelrc caps at its test:integration --test_timeout
// (rbe-west's 1200s limit). The job adds setup and log upload, but
// stays remote-only short.
const coldCompileMinutes = 20
actionCapMinutes := 0
for line := range bazelrcLines(t) {
if v, ok := strings.CutPrefix(line, "test:integration --test_timeout="); ok {
secs, err := strconv.Atoi(v)
if err != nil || secs <= 0 || secs > 1200 {
t.Fatalf(".bazelrc test:integration --test_timeout=%s, want one value of at most 1200 (rbe-west's cap)", v)
}
actionCapMinutes = (secs + 59) / 60
}
}
if actionCapMinutes == 0 {
t.Fatal(".bazelrc has no test:integration --test_timeout")
}
if want := coldCompileMinutes + actionCapMinutes; test.TimeoutMinutes < want {
t.Errorf("%s test step timeout-minutes = %d, want at least %d (cold compile %d + action cap %d)",
bazelIntegJobName, test.TimeoutMinutes, want, coldCompileMinutes, actionCapMinutes)
}
if job.TimeoutMinutes <= test.TimeoutMinutes || job.TimeoutMinutes > test.TimeoutMinutes+15 {
t.Errorf("%s timeout-minutes = %d, want above the test step's %d by at most 15", bazelIntegJobName, job.TimeoutMinutes, test.TimeoutMinutes)
}
cmd := regexp.MustCompile(`\s*\\\n\s*`).ReplaceAllString(test.Run, " ")
const wantCmd = `bazel test //... --config=integration --build_event_json_file="$RUNNER_TEMP/bazel-bep.json" 2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?`
@ -3268,7 +3297,7 @@ func TestBazelDoltServerTiersMirrorPRRisk(t *testing.T) {
// pr.yml's gate requires it: bazelLaneGateIDs): not a step of
// bazel-doltserver (which also runs locally, and whose job a gate may
// require) or of bazel-integration (which a caller may switch off,
// although the server tier is a PR-time tier).
// while the server tier always runs remotely).
workflow := readCIWorkflow(t, bazelWorkflowName)
for _, c := range []struct{ job, config, logs string }{
{bazelProxiedJobName, "doltserver-proxied", "bazel-proxied-testlogs"},