ci(bazel): dolt-race gates the dolt-server and proxied tiers in mode remote (item 5 A1b) (#7514)

* ci(bazel): dolt-race gates the dolt-server and proxied tiers in mode remote (item 5 A1b)

In mode remote, bazel-dolt-race (`bazel test //... --config=dolt-race`, the
union of --config=doltserver and --config=doltserver-proxied) runs both tiers
in place of bazel-doltserver and bazel-proxied, and is no longer advisory:
its job.status is both the bazel-doltserver and the bazel-proxied
workflow_call output (union result first, member's otherwise). In the fork
modes, cache and local, the two member jobs run exactly as before: there one
client does both tiers' work without in-flight deduplication, so a union
would cost the sum of the tiers rather than the max. Exactly one arm runs in
any mode, so gate ids, pr.yml, ci-gate.sh and bazel-gate.sh are unchanged.

- bazel.yml: dolt-race if: mode == 'remote' (every event that reaches it);
  no continue-on-error; result output and always() recorder; test step
  timeout 25 -> 20 (job 55 >= step sum 46 + 5, < 60). Members' if: their old
  rule && mode != 'remote'.
- scripts/bazel_dolt_race_test.go (was bazel_dolt_race_shadow_test.go):
  union-of-members config, sole-run lines in the union, gating shape, step
  and job timeouts, exactly one arm per mode x event with the call outputs
  evaluated as GitHub does, and the union's 62 test actions within
  --config=remote-exec's --jobs=64.
- Gate simulations evaluate the workflow_call output expressions, and a
  gated lane's reporter is the lane or its union lane.
- aquery: all 62 TestRunner actions under --config=dolt-race are
  byte-identical (key, argv, env, outputs) to --config=doltserver (with and
  without sole-run) and --config=doltserver-proxied on origin/main.
- Docs: CI_REQUIRED_CHECK_TOPOLOGY.md "dolt-race Union Lane", TESTING.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): dolt-race test step 23 min so Bazel logs a 1200 s TIMEOUT first

The proxied tier caps each test at 1200 s; a 20-minute step let GitHub
kill a hung shard before Bazel wrote its TIMEOUT test.log and goroutine
dump. Step 23, job 56 (step sum + 5, under the merge queue's 60).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): say the dolt-race job timeout bounds the job, not the run

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-10-13 04:35:17 +00:00 • committed by GitHub
commit 85464f567a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 736 additions and 474 deletions

View file

@ -178,20 +178,21 @@ test:doltserver-proxied --experimental_remote_cache_eviction_retries=0
# above.
test:doltserver-proxied --norun_validations
# dolt-server + proxied union (item 5, Phase 1 shadow): bazel.yml's advisory
# bazel-dolt-race job runs both tiers in one invocation, beside their own
# lanes, to measure one client per group of lanes before any cutover. Exactly
# the union of --config=doltserver's and --config=doltserver-proxied's lines,
# with one tag filter naming both tiers
# (scripts/bazel_dolt_race_shadow_test.go). Every other flag is the same in
# both members or outside any action key for the other: the race build,
# dolt-server + proxied union (item 5): bazel.yml's bazel-dolt-race job runs
# both tiers in one invocation in mode remote, in place of the bazel-doltserver
# and bazel-proxied lanes, which still run --config=doltserver and
# --config=doltserver-proxied in every other mode. Exactly the union of the two
# member configs' lines, with one tag filter naming both tiers
# (scripts/bazel_dolt_race_test.go). Every other flag is the same in both
# members or outside any action key for the other: the race build,
# -test.parallel=4 and GO_TEST_WRAP_TESTV=1 are common; the eternal timeout
# applies to no dolt-server target; the eviction retry count is what
# doltserver already gets from --config=sole-run in mode remote, the only mode
# the shadow runs in; test_summary, local_test_jobs and remote_download_regex
# are not in any action key. So each member target's test action is
# unchanged, and the two lanes' cached results carry over. The member configs
# stay for make, developers and the member lanes.
# doltserver gets from --config=sole-run wherever it executes remotely;
# test_summary, local_test_jobs and remote_download_regex are not in any action
# key. So each member target's test action is unchanged (aquery: all 62
# TestRunner actions byte-identical), and whichever arm the mode picks shares
# cached results with the other. The member configs stay for those modes, make
# and developers.
test:dolt-race --@rules_go//go/config:race
test:dolt-race --test_tag_filters=dolt-server,dolt-server-proxied
test:dolt-race --build_tests_only

View file

@ -46,11 +46,14 @@
# only what the PR changed is built and tested on the runner, and it is
# their only PR-time integration run (no legacy job runs it on PRs). Mode
# local (rbe=off) still skips it. Direct runs (push to main,
# workflow_dispatch, nightly.yml's call) gate nothing. bazel-dolt-race (the
# dolt-server + proxied union shadow, mode remote on PRs and merge groups
# only) cannot fail the gate either: no output, job-level continue-on-error.
# The gate still waits for it, as for every job of the call (see its comment
# and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "dolt-race Shadow").
# workflow_dispatch, nightly.yml's call) gate nothing. In mode remote the
# dolt-server and proxied-server tiers run as one union job, bazel-dolt-race
# (`bazel test --config=dolt-race`), whose job.status is both the
# bazel-doltserver and the bazel-proxied output; in every other mode
# bazel-doltserver and bazel-proxied run as before and report their own.
# Exactly one of the two arms runs in any mode, so the gate ids
# (BAZEL_DOLTSERVER, BAZEL_PROXIED) and bazel-gate.sh's skips are unchanged
# (engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "dolt-race Union Lane").
#
# Execution mode, decided once by the rbe job (every other job needs it and
# reads its outputs; none re-derives the condition) and passed to
@ -346,12 +349,14 @@ on:
bazel-integration:
description: "bazel-integration's job.status"
value: ${{ jobs.bazel-integration.outputs.result }}
# Mode remote: bazel-dolt-race runs both tiers and the member jobs
# skip; every other mode: the reverse. One side is always empty.
bazel-doltserver:
description: "bazel-doltserver's job.status"
value: ${{ jobs.bazel-doltserver.outputs.result }}
description: "the dolt-server tier's job.status: bazel-dolt-race's in mode remote, bazel-doltserver's otherwise"
value: ${{ jobs.bazel-dolt-race.outputs.result || jobs.bazel-doltserver.outputs.result }}
bazel-proxied:
description: "bazel-proxied's job.status"
value: ${{ jobs.bazel-proxied.outputs.result }}
description: "the proxied-server tier's job.status: bazel-dolt-race's in mode remote, bazel-proxied's otherwise"
value: ${{ jobs.bazel-dolt-race.outputs.result || jobs.bazel-proxied.outputs.result }}
bazel-server-storage:
description: "bazel-server-storage's job.status"
value: ${{ jobs.bazel-server-storage.outputs.result }}
@ -1743,12 +1748,18 @@ jobs:
# doctor/fix, protocol in 8 shards, testutil), each of which starts its
# own dolt sql-server from the pinned dolt, so they execute on the remote
# workers like any other test (locally with rbe=off).
# Not in mode remote: there bazel-dolt-race runs this tier together with
# the proxied-server tier and reports for both (item 5). In the fork
# modes, cache and local there is no in-flight deduplication of one
# client's actions and the client does the work, so a union would cost the
# sum of the two tiers instead of the max: this job runs there as before.
bazel-doltserver:
name: dolt-server lane
# Same runner and skip rule as bazel-test.
# Same runner as bazel-test; bazel-test's skip rule, plus mode remote
# (bazel-dolt-race's).
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
if: ${{ needs.rbe.outputs.mode != 'skip' }}
if: ${{ needs.rbe.outputs.mode != 'skip' && needs.rbe.outputs.mode != 'remote' }}
timeout-minutes: 90
outputs:
result: ${{ steps.result.outputs.result }}
@ -1879,7 +1890,8 @@ jobs:
# (formerly proxied-local-smoke.yml) in a loopback-only network
# namespace, which needs the workers' unprivileged user namespaces. A job
# of its own, not gated by any integration switch: it runs in pr.yml's
# call and its gate requires it (BAZEL_PROXIED).
# call and its gate requires it (BAZEL_PROXIED). Fork modes only: in mode
# remote bazel-dolt-race runs this tier (see bazel-doltserver's comment).
bazel-proxied:
name: proxied-server tier
# Remote execution only, like bazel-embedded: 30 concurrent shards, each
@ -1889,7 +1901,7 @@ jobs:
# retired) and pr.yml's gate requires it to have run here and passed.
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
if: ${{ needs.rbe.outputs.enabled == 'true' }}
if: ${{ needs.rbe.outputs.enabled == 'true' && needs.rbe.outputs.mode != 'remote' }}
timeout-minutes: 30
outputs:
result: ${{ steps.result.outputs.result }}
@ -2033,45 +2045,37 @@ jobs:
JOB_STATUS: ${{ job.status }}
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
# Item 5, Phase 1 (scripts/bazel_dolt_race_shadow_test.go): the advisory
# shadow of a union lane. One `bazel test --config=dolt-race` runs the
# dolt-server and proxied-server tiers together (.bazelrc: exactly the
# union of --config=doltserver's and --config=doltserver-proxied's lines,
# one tag filter for both), beside bazel-doltserver and bazel-proxied, so
# one client per group of lanes can be compared with the two it would
# replace (end time, runner minutes, test counts and results) before any
# cutover. Each member target's test action key is unchanged, so the shadow
# and the member lanes share cached results; and rbe-west deduplicates
# identical in-flight executions (confirmed by the farm maintainer:
# NativeLink 1.7.1 subscribes a second identical Execute of a cacheable
# action to the operation already in flight), so the shadow adds no
# executions.
# Advisory: it cannot fail the gate. Job-level continue-on-error (a failed
# job otherwise fails the whole call, whose result pr.yml's ci-gate
# requires as BAZEL), no workflow_call output, no job outputs, no job in
# this file needs it, and no gate script or caller names it. The gate does
# wait for it: ci-gate needs pr.yml's bazel call, which completes only when
# every job in it has finished (success, failure or timeout), so a slow
# shadow delays the gate and the merge queue. continue-on-error does not
# cover a job GitHub cancels at its own timeout-minutes, which could turn
# the call red: every step has a timeout, and the job's exceeds their sum
# (TestBazelDoltRaceShadowIsAdvisory). Mode remote only, on
# same-repo pull requests and merge groups (mode remote on a pull_request
# is never a fork): no fork mode, no cache or local mode, and not on push,
# nightly, dispatch or bazel-farm.yml's call. Its ci-analytics LANE is
# dolt-race. Removing it is the rollback; nothing reads it.
# Item 5 (scripts/bazel_dolt_race_test.go): the dolt-server and
# proxied-server tiers in one `bazel test --config=dolt-race` (.bazelrc:
# exactly the union of --config=doltserver's and
# --config=doltserver-proxied's lines, one tag filter for both), mode
# remote only, in place of bazel-doltserver and bazel-proxied, which run
# in every other mode. One client waits on rbe-west for both tiers'
# shards at once (27 + 35 test actions, under the remote-exec config's 64
# jobs), so the job ends with the slower tier and saves the other's
# runner. Each member target's test action key is unchanged (aquery-
# verified), so cached results carry over both ways, and a revert of the
# cutover re-executes nothing. Gated: its job.status is both the
# bazel-doltserver and the bazel-proxied workflow_call output (a failure
# in either tier fails both ids, which are both required). Every step has
# a timeout and the job's exceeds their sum by 5 minutes while staying
# under the merge queue's 60-minute check timeout
# (TestBazelDoltRaceJobTimeouts), so a hang fails a named step instead of
# dropping the merge group. The member jobs' configs and the union's must
# stay key-identical: the mode decides which runs (TestBazelDoltRaceConfigIsMemberUnion).
# Its ci-analytics LANE is dolt-race.
bazel-dolt-race:
name: dolt-race
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
if: ${{ needs.rbe.outputs.mode == 'remote' && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }}
# At least every step's timeout-minutes plus 5 (and the test step's plus
# 10): a hung step then fails the job (continue-on-error) before GitHub
# cancels it, which continue-on-error does not cover
# (TestBazelDoltRaceShadowIsAdvisory). bazel-proxied's steps have no
# timeouts but its test step's; these bound the rest.
timeout-minutes: 55
continue-on-error: true
if: ${{ needs.rbe.outputs.mode == 'remote' }}
# At least every step's timeout-minutes plus 5, and below the merge
# queue's 60-minute check timeout so this job alone cannot exhaust it
# (TestBazelDoltRaceJobTimeouts). Not a whole-run bound: the rbe job runs
# first and sibling lanes have longer timeouts.
timeout-minutes: 56
outputs:
result: ${{ steps.result.outputs.result }}
steps:
# inputs.checkout-sha: see the header (empty: the event's default ref).
# checkout v7 refuses a fork head on pull_request_target unless opted
@ -2120,11 +2124,15 @@ jobs:
fi
# Both tiers' shards at once: 27 dolt-server + 35 proxied test actions,
# under the remote-exec config's 64 jobs. The step timeout is
# bazel-proxied's (the longer tier on rbe-west).
# under the remote-exec config's 64 jobs. No --config=sole-run: its one
# line (no retry on a remote cache eviction) is already in the union.
# Shards are expected around 55-95s on rbe-west (see bazel-proxied's
# test step), the proxied tier's 1200s cap included. The step timeout is
# >= proxied test_timeout 1200 s + about 3 min so Bazel writes TIMEOUT
# logs before GitHub kills the step.
- name: bazel test //... --config=dolt-race
id: test
timeout-minutes: 25
timeout-minutes: 23
run: |
set -o pipefail
start=$(date +%s)
@ -2138,12 +2146,11 @@ jobs:
echo "bazel test --config=dolt-race: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
exit "$rc"
# The member lanes' checks over the union's BEP: bazel-proxied's (the
# dolt-server lane has none). check_shard_coverage ignores targets it
# does not list; check_testcases checks every target, so it covers the
# dolt-server tier too, which bazel-doltserver does not check: a
# dolt-server shard that runs no test turns this job red while both
# member lanes stay green.
# The member lanes' checks, once over the union's BEP: bazel-proxied's
# (the dolt-server lane has none). check_shard_coverage ignores targets
# it does not list; check_testcases checks every target, so it covers
# the dolt-server tier too, which bazel-doltserver does not check: a
# dolt-server shard that runs no test turns this job red.
- name: Every target and shard ran tests
if: ${{ always() && steps.test.outcome != 'skipped' }}
timeout-minutes: 2
@ -2201,6 +2208,17 @@ jobs:
retention-days: 7
if-no-files-found: ignore
# Last step: the job's result as the bazel-doltserver and
# bazel-proxied workflow_call outputs for pr.yml's ci-gate. always()
# runs it after a failure, a timeout or a cancellation too.
- name: Record job result
id: result
if: ${{ always() }}
timeout-minutes: 1
env:
JOB_STATUS: ${{ job.status }}
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
# pr-risk.yml's server-Dolt storage tier, "Test (Server Dolt Conformance)"
# and "Test (Server Dolt Full Suite N/16)", as --config=doltserver-integration:
# the conformance and 16-shard full-suite variants of

View file

@ -24,8 +24,8 @@ env:
# The legacy Go test tiers this workflow used to run (embedded Dolt storage,
# conformance and cmd shards; proxied-server cmd shards; server-Dolt storage
# and full suite) are pr.yml's Bazel lanes on rbe-west now: `embedded Dolt
# tier`, `proxied-server tier` and `server-Dolt storage tier` (bazel.yml),
# which pr.yml's `CI Gate / Required` requires to have run remotely and
# tier`, `proxied-server tier` (`dolt-race`, with the dolt-server lane, in
# mode remote) and `server-Dolt storage tier` (bazel.yml), which pr.yml's `CI Gate / Required` requires to have run remotely and
# passed on every pull_request and merge group, fork and Dependabot PRs
# included (ga-96smfk.22). What remains is the Nix packaging check, which no
# Bazel target can stand in for: it builds the package Nix users install.

View file

@ -54,7 +54,10 @@ Current PR-related workflow names:
repo variable is unset) and exports it as the `rbe-mode` / `rbe-enabled`
outputs (`rbe-enabled` is `true` in `remote`, `fork-ro` and `fork-rw`);
every lane exports its
`job.status` as an output named after the job. `pr.yml`'s gate requires the
`job.status` as an output named after the job (in mode `remote`, the
`bazel-doltserver` and `bazel-proxied` outputs carry the `bazel-dolt-race`
union job's status instead; see [dolt-race Union Lane](#dolt-race-union-lane)).
`pr.yml`'s gate requires the
call's result (`BAZEL`) and `BAZEL_TEST`, `BAZEL_PURE`,
`BAZEL_RELEASE_CROSS`, `BAZEL_EMBEDDED`, `BAZEL_INTEGRATION`,
`BAZEL_DOLTSERVER`, `BAZEL_PROXIED` and `BAZEL_SERVER_STORAGE`. These lanes are the only CI run of the Linux Go
@ -620,38 +623,41 @@ job's `if:` or rotating the key, in either of two ways:
step's own kill switch, checked before any network call. Deleting the
variable does not disable pre-warming; it restores the default of 1.
## dolt-race Shadow (Not Required)
## dolt-race Union Lane
`bazel.yml`'s `bazel-dolt-race` job (check name `Bazel / dolt-race`) is an
advisory shadow, not a required check. It runs
`bazel test //... --config=dolt-race`, the union of `--config=doltserver` and
`--config=doltserver-proxied` with one tag filter for both tiers, beside the
`bazel-doltserver` and `bazel-proxied` lanes. Its purpose is to measure one
client per group of lanes against the two lanes it would replace (end time,
runner minutes, test counts and results) before any cutover. Each member
target's test action key is unchanged, so the shadow shares cached results
with the member lanes.
In mode `remote`, `bazel.yml`'s `bazel-dolt-race` job (check name
`Bazel / dolt-race`) runs the dolt-server and proxied-server tiers in one
`bazel test //... --config=dolt-race`, in place of the `bazel-doltserver`
(`dolt-server lane`) and `bazel-proxied` (`proxied-server tier`) jobs.
`--config=dolt-race` is exactly the union of `--config=doltserver` and
`--config=doltserver-proxied`, with one tag filter for both tiers, so every
member target's test action is unchanged (`bazel aquery`: the 27 + 35 = 62
`TestRunner` actions are byte-identical under the member configs and the
union). One client waits on rbe-west for both tiers at once, so the job ends
with the slower tier and the other tier's runner is saved.
- It runs in mode `remote` on `pull_request` and `merge_group` events only.
It never runs in a fork mode (`fork-ro`, `fork-rw`), in mode `cache` or
`local`, on `bazel-farm.yml`'s call, on push, nightly or dispatch.
- It cannot fail the gate. It has job-level `continue-on-error`, so its
failure cannot fail the call's aggregate result (`BAZEL`). It has no
`workflow_call` output, no `bazel.yml` job `needs` it, and `pr.yml`,
`ci-gate.sh` and `bazel-gate.sh` never name it.
- The gate still waits for it. `CI Gate` needs `pr.yml`'s `bazel` call, and
a reusable-workflow call completes only when all of its jobs have finished
(success, failure or timeout), `continue-on-error` ones included. A slow
shadow therefore delays `CI Gate` and the merge queue, up to its
timeout. `continue-on-error` does not cover a job GitHub cancels at its
own `timeout-minutes`, so every step has its own timeout and the job's
(65 minutes) stays at least 5 minutes above their sum and 10 above the
test step's (policy-tested). A hung step then fails the job, which
`continue-on-error` covers, before GitHub would cancel it.
- Mode `remote` only (same-repo pull requests, merge groups, push, nightly,
dispatch and `bazel-farm.yml`'s authorized call). In the fork modes
(`fork-ro`, `fork-rw`), `cache` and `local`, `bazel-doltserver` and
`bazel-proxied` run as before: there a single client does both tiers' work
without in-flight deduplication, so a union would cost the sum of the
tiers rather than the max, on the run's critical path.
- Exactly one arm runs in any mode. The union's `job.status` is both the
`bazel-doltserver` and the `bazel-proxied` `workflow_call` output (each
reads the union's result first, then its member's), so `pr.yml`'s gate
ids (`BAZEL_DOLTSERVER`, `BAZEL_PROXIED`), `ci-gate.sh` and
`bazel-gate.sh` are unchanged: a failed or cancelled union reds both ids,
a union that reported nothing reads as `skipped`, which mode `remote`
never allows, and the per-mode skips are the members' as before.
- `check_testcases.py` and `check_shard_coverage.py` run once over the
union's BEP (the proxied tier's `--suite`; the dolt-server tier has no
shard-coverage check). The `bazel test` step has a 20-minute timeout,
every step has one, and the job's (55 minutes) is at least their sum plus
5 and below the merge queue's 60-minute check timeout.
- Its ci-analytics `LANE` is `dolt-race`.
- Policy: `scripts/bazel_dolt_race_shadow_test.go`. Removing the job, its
`.bazelrc` config and its line in `.github/scripts/rrc-lane-commands.txt`
is the rollback; nothing reads them.
- Policy: `scripts/bazel_dolt_race_test.go`. Rollback is a revert of the
cutover PR: the gate ids never changed and the action keys are equal both
ways, so cached results carry back.
## Remote Repo Contents Cache
@ -1032,7 +1038,7 @@ Required` requires them to have run remotely and passed.
`nogoConfigurations` table (race, pure, js/wasm) validates in exactly
one required lane, not beside every compile of every lane: the race
configuration's nogo is owned by `test` above, and `embedded`,
`doltserver`, `doltserver-proxied`, `dolt-race` (shadow) and the
`doltserver`, `doltserver-proxied`, `dolt-race` and the
package gates pass `--norun_validations`, since they compile a strict
subset of what `test` already validates on that configuration
(`scripts/nogo_lint_policy_test.go`, engdocs/LINTING.md). Since F5 S2

View file

@ -43,6 +43,7 @@ Each lane, as bazel.yml runs it (add your `--config=fork-cache` or
| cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 32 shards of the integration-tagged cmd/bd suite (the slowest tests pinned to 16 of them). |
| Embedded Dolt (`bazel-embedded`) | `bazel test //... --config=embedded` | Remote execution only in CI; locally it is slow. Also runs the served http tiers with `BEADS_HTTP_TEST_REQUIRED=1`: `//backend/http:http_served_test`, `//internal/httpclient:httpclient_served_test` and the command corpus `//backend/http/corpus:corpus_served_test`. |
| Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 34 shards, each with a Dolt server. |
| Dolt server + proxied (`bazel-dolt-race`) | `bazel test //... --config=dolt-race` | Both rows above in one invocation; CI runs this instead of them in mode `remote`. |
| Server-Dolt storage (`bazel-server-storage`) | `bazel test //... --config=doltserver-integration` | Remote execution only. |
| Docs | `make check-docs` | `//test/docsync:docsync_test` and `//scripts/repochecks:doc_freshness_test` in the test lane's configuration, then `scripts/check-doc-flags.sh` against the pinned release. |
@ -199,7 +200,8 @@ remotely with `--config=remote-exec`. No Bazel target uses the `container`
backend; run `go test` with docker and the pulled image to exercise it.
PR Risk's heavier server tiers have configs of their own, run by bazel.yml
only with remote execution, each in a job of its own (`bazel-proxied`,
`bazel-server-storage`): `--config=doltserver-proxied` is the
which in mode `remote` is folded with `bazel-doltserver` into
`bazel-dolt-race`, and `bazel-server-storage`): `--config=doltserver-proxied` is the
proxied-server cmd/bd tier ("Test (Proxied Dolt Cmd N/15)",
`//cmd/bd:bd_proxied_test`), and `--config=doltserver-integration` the
server-Dolt storage tier ("Test (Server Dolt Conformance)", "Test (Server

View file

@ -35,7 +35,7 @@ go_test(
srcs = [
"bazel_autofix_push_test.go",
"bazel_cache_mode_test.go",
"bazel_dolt_race_shadow_test.go",
"bazel_dolt_race_test.go",
"bazel_farm_workflow_test.go",
"bazel_fork_mode_test.go",
"bazel_go_repository_tools_test.go",

View file

@ -227,9 +227,9 @@ func TestBazelCacheModeReachesTheRC(t *testing.T) {
// no longer forwards the app secrets either, so there is no
// audience left for pre-warming in fork-ro or fork-rw (see
// bazel.yml's comment on the job).
// Nor does a shadow lane (bazelShadowLanes): mode remote on
// same-repo PRs and merge groups only.
case strings.HasPrefix(mode, "fork-") && !runs && !bazelPackageJobs[name] && !(name == bazelRBEPrewarmJobName && strings.HasPrefix(mode, "fork-")) && bazelShadowLanes[name] == "":
// Nor does a union lane (bazelUnionLanes): mode remote only,
// its members run in the fork modes.
case strings.HasPrefix(mode, "fork-") && !runs && !bazelPackageJobs[name] && !(name == bazelRBEPrewarmJobName && strings.HasPrefix(mode, "fork-")) && bazelUnionLanes[name] == nil:
t.Errorf("%s does not run in mode %s (every lane runs remotely)", name, mode)
case mode == "cache" && (name == bazelIntegJobName || name == bazelCmdDoltJobName) && !runs:
t.Errorf("%s does not run in mode cache", name)
@ -289,7 +289,9 @@ func TestBazelCacheModeReachesTheRC(t *testing.T) {
if bazelPackageJobs[lane] {
continue
}
if !ran[lane] {
// A union lane's run reports for its members
// (bazel_dolt_race_test.go).
if !ran[lane] && !ran[bazelUnionOf(lane)] {
wantSkips = append(wantSkips, id)
}
}

View file

@ -1,305 +0,0 @@
package scripts_test
import (
"cmp"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"gopkg.in/yaml.v3"
)
// The dolt-race shadow (item 5, Phase 1): bazel.yml's bazel-dolt-race job
// runs --config=dolt-race, the union of --config=doltserver and
// --config=doltserver-proxied, in one `bazel test` beside the two member
// lanes, so the union-job model can be measured against them before any
// cutover. It must be advisory: no workflow_call output, nothing in pr.yml's
// gates or bazel-gate.sh/ci-gate.sh, and a failure that cannot fail the
// call's aggregate result (BAZEL) that pr.yml's ci-gate requires. It cannot
// fail the gate, but the gate still waits for it: ci-gate needs the whole
// bazel call, which completes only when every job in it has finished.
const (
bazelDoltRaceJobName = "bazel-dolt-race"
bazelDoltRaceConfig = "dolt-race"
// Mode remote only, and only on same-repo pull requests and merge
// groups (mode remote on a pull_request event is never a fork: forks
// get fork-ro, fork-rw or cache). Not on push, nightly, dispatch or
// bazel-farm.yml's pull_request_target call.
bazelDoltRaceJobIf = "${{ needs.rbe.outputs.mode == 'remote' && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }}"
)
// bazelShadowLanes: bazel.yml jobs that run beside gated lanes only to
// measure a change to them, and report into nothing (no job or
// workflow_call output, no gate id, job-level continue-on-error), with the
// reason. TestBazelLaneIsGated and the continue-on-error checks carve them
// out by this map.
var bazelShadowLanes = map[string]string{
bazelDoltRaceJobName: "item 5 Phase 1: the dolt-server + proxied union beside its member lanes, advisory until a cutover PR",
}
// bazelDoltRaceJobTimeoutHeadroom: the minutes the shadow job's
// timeout-minutes must exceed its test step's (TestBazelDoltRaceShadowIsAdvisory).
const bazelDoltRaceJobTimeoutHeadroom = 10
// bazelDoltRaceJobTimeoutMargin: the minutes the shadow job's timeout-minutes
// must exceed the sum of all its step timeouts.
const bazelDoltRaceJobTimeoutMargin = 5
// bazelDoltRaceMergeQueueTimeout: the merge queue's check timeout (ruleset
// merge_queue.check_response_timeout_minutes). CI Gate waits for the shadow,
// so the shadow's job timeout must stay below it or a hung shadow drops the
// group from the queue.
const bazelDoltRaceMergeQueueTimeout = 60
// The union's members, and the tag filter each owns.
var bazelDoltRaceMembers = []struct{ config, tag string }{
{"doltserver", "dolt-server"},
{"doltserver-proxied", "dolt-server-proxied"},
}
// bazelRCConfigLines returns .bazelrc's option lines for --config=name
// (any command: test:name, build:name, ...), as "<command> <option>"
// with the ":name" suffix stripped, in file order.
func bazelRCConfigLines(rc, name string) []string {
var out []string
for _, line := range strings.Split(rc, "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
head, opt, ok := strings.Cut(line, " ")
cmd, cfg, hasCfg := strings.Cut(head, ":")
if !ok || !hasCfg || cfg != name {
continue
}
out = append(out, cmd+" "+strings.TrimSpace(opt))
}
return out
}
// TestBazelDoltRaceConfigIsMemberUnion: --config=dolt-race is exactly the
// set union of its members' lines, except that each member's
// --test_tag_filters is replaced by one filter naming both members' tags.
// Every flag stays a member's own, so each member target's test action is
// unchanged (verified with aquery in the PR that added it), and a flag added
// to a member must be added to the union too.
func TestBazelDoltRaceConfigIsMemberUnion(t *testing.T) {
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
const tagPrefix = "test --test_tag_filters="
var want []string
var tags []string
for _, m := range bazelDoltRaceMembers {
lines := bazelRCConfigLines(rc, m.config)
if len(lines) == 0 {
t.Fatalf(".bazelrc has no --config=%s lines", m.config)
}
var memberTags []string
for _, l := range lines {
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
memberTags = append(memberTags, v)
continue
}
if !slices.Contains(want, l) {
want = append(want, l)
}
}
if len(memberTags) != 1 || memberTags[0] != m.tag {
t.Errorf("--config=%s tag filters %v, want exactly [%s]", m.config, memberTags, m.tag)
}
tags = append(tags, m.tag)
}
got := bazelRCConfigLines(rc, bazelDoltRaceConfig)
var gotTags []string
var gotRest []string
for _, l := range got {
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
gotTags = append(gotTags, v)
continue
}
gotRest = append(gotRest, l)
}
if wantTag := strings.Join(tags, ","); len(gotTags) != 1 || gotTags[0] != wantTag {
t.Errorf("--config=%s tag filters %v, want exactly [%s]", bazelDoltRaceConfig, gotTags, wantTag)
}
if !sameStringSet(gotRest, want) {
t.Errorf("--config=%s is not the union of its members' lines (tag filter aside):\ngot:\n%s\nwant (any order):\n%s",
bazelDoltRaceConfig, strings.Join(gotRest, "\n"), strings.Join(want, "\n"))
}
if len(gotRest) != len(want) {
t.Errorf("--config=%s repeats a line: %d lines for %d distinct member lines", bazelDoltRaceConfig, len(gotRest), len(want))
}
// No nested --config: the union is spelled out, so the policy above
// sees every flag.
for _, l := range got {
if strings.Contains(l, "--config=") {
t.Errorf("--config=%s line %q: spell the members' flags out, no nested --config", bazelDoltRaceConfig, l)
}
}
}
func sameStringSet(a, b []string) bool {
as := map[string]bool{}
for _, s := range a {
as[s] = true
}
bs := map[string]bool{}
for _, s := range b {
bs[s] = true
}
if len(as) != len(bs) {
return false
}
for s := range as {
if !bs[s] {
return false
}
}
return true
}
// TestBazelDoltRaceShadowIsAdvisory pins what keeps the shadow from failing
// any gate: remote-only, job-level continue-on-error (a job's failure
// otherwise fails the reusable-workflow call, whose result pr.yml's ci-gate
// requires as BAZEL), timeout headroom (a job cancelled at its own timeout
// is not covered by continue-on-error), no workflow_call output, no
// bazel.yml job needs it, and no gate script or caller names it. ci-gate
// still waits for it to finish, through its needs on the whole call.
func TestBazelDoltRaceShadowIsAdvisory(t *testing.T) {
workflow := readCIWorkflow(t, bazelWorkflowName)
job := workflow.job(t, bazelDoltRaceJobName)
if job.If != bazelDoltRaceJobIf {
t.Errorf("%s if: %q, want %q", bazelDoltRaceJobName, job.If, bazelDoltRaceJobIf)
}
if !job.ContinueOnError {
t.Errorf("%s: continue-on-error false; the shadow must never fail the bazel call pr.yml's ci-gate requires", bazelDoltRaceJobName)
}
if !slices.Equal(job.Needs, []string{bazelRBEJobName}) {
t.Errorf("%s needs %v, want [%s]", bazelDoltRaceJobName, job.Needs, bazelRBEJobName)
}
for name, other := range workflow.Jobs {
if slices.Contains(other.Needs, bazelDoltRaceJobName) {
t.Errorf("job %s needs %s; no bazel.yml job may depend on the shadow", name, bazelDoltRaceJobName)
}
}
// The test step: the union config, as the member lanes run theirs.
test := job.step(t, "bazel test //... --config="+bazelDoltRaceConfig)
if strings.TrimSpace(test.Run) != bazelTierTestRun(bazelDoltRaceConfig) {
t.Errorf("%s test step run changed; want exactly:\n%s\ngot:\n%s", bazelDoltRaceJobName, bazelTierTestRun(bazelDoltRaceConfig), test.Run)
}
// Timeout headroom. continue-on-error covers a failed or timed-out step,
// not a job GitHub cancels at its own timeout-minutes; a cancelled job
// can make the call (needs.bazel.result) non-success and turn ci-gate
// red. So the job outlives its test step by at least
// bazelDoltRaceJobTimeoutHeadroom minutes, for setup before it and the
// checks, uploads and analytics after it. Both sides come from the
// workflow: a step without timeout-minutes would default to 360.
if test.TimeoutMinutes <= 0 {
t.Errorf("%s test step has no timeout-minutes; it would default to GitHub's 360 and outlive the job", bazelDoltRaceJobName)
}
if job.TimeoutMinutes < test.TimeoutMinutes+bazelDoltRaceJobTimeoutHeadroom {
t.Errorf("%s timeout-minutes %d, want at least the test step's %d + %d: a job cancelled at its own timeout can turn ci-gate red",
bazelDoltRaceJobName, job.TimeoutMinutes, test.TimeoutMinutes, bazelDoltRaceJobTimeoutHeadroom)
}
// And no step can run the job into its own timeout: every step has a
// timeout-minutes (a step timeout is a step failure, which
// continue-on-error covers), and their sum plus
// bazelDoltRaceJobTimeoutMargin minutes (runner setup, post-job
// cleanup) fits in the job's.
sum := 0
for i, step := range job.Steps {
if step.TimeoutMinutes <= 0 {
t.Errorf("%s step %d (%q) has no timeout-minutes; a hang would run into the job timeout, a cancellation continue-on-error does not cover",
bazelDoltRaceJobName, i, cmp.Or(step.Name, step.Uses))
}
sum += step.TimeoutMinutes
}
if job.TimeoutMinutes < sum+bazelDoltRaceJobTimeoutMargin {
t.Errorf("%s timeout-minutes %d, want at least the sum of its step timeouts (%d) + %d",
bazelDoltRaceJobName, job.TimeoutMinutes, sum, bazelDoltRaceJobTimeoutMargin)
}
if job.TimeoutMinutes >= bazelDoltRaceMergeQueueTimeout {
t.Errorf("%s timeout-minutes %d, want below the merge queue's %d-minute check timeout: CI Gate waits for the shadow",
bazelDoltRaceJobName, job.TimeoutMinutes, bazelDoltRaceMergeQueueTimeout)
}
// No workflow_call output reads the job.
var doc struct {
On struct {
WorkflowCall struct {
Outputs map[string]struct {
Value string `yaml:"value"`
} `yaml:"outputs"`
} `yaml:"workflow_call"`
} `yaml:"on"`
}
if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+bazelWorkflowName)), &doc); err != nil {
t.Fatal(err)
}
if len(doc.On.WorkflowCall.Outputs) == 0 {
t.Fatalf("%s: no workflow_call outputs parsed; the check is broken", bazelWorkflowName)
}
for name, out := range doc.On.WorkflowCall.Outputs {
if strings.Contains(out.Value, "jobs."+bazelDoltRaceJobName+".") || strings.Contains(name, bazelDoltRaceConfig) {
t.Errorf("%s workflow_call output %s (%q) reads the shadow; it reports into no gate", bazelWorkflowName, name, out.Value)
}
}
// No gate or caller names the job, its config or a gate id for it.
root := sourceRepoRoot(t)
for _, f := range []string{
".github/workflows/pr.yml",
".github/workflows/pr-risk.yml",
".github/workflows/nightly.yml",
".github/workflows/bazel-farm.yml",
".github/scripts/bazel-gate.sh",
".github/scripts/ci-gate.sh",
} {
b, err := os.ReadFile(filepath.Join(root, f))
if err != nil {
t.Fatal(err)
}
s := strings.ToLower(string(b))
for _, needle := range []string{"dolt-race", "dolt_race", "doltrace"} {
if strings.Contains(s, needle) {
t.Errorf("%s mentions %q; the dolt-race shadow must stay out of every gate", f, needle)
}
}
}
}
// TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls evaluates the job's if:
// for every execution mode and every event bazel.yml runs on. It runs only
// in mode remote on a pull_request or merge_group event: never in a fork
// mode (fork-ro, fork-rw), cache or local, never on bazel-farm.yml's
// pull_request_target call (fork code with trusted credentials), and not on
// push, nightly (schedule) or dispatch, so it never doubles nightly's fresh
// execution of the two tiers.
func TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls(t *testing.T) {
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltRaceJobName)
for _, mode := range []string{"remote", "fork-ro", "fork-rw", "cache", "local", "skip"} {
for _, event := range []string{"pull_request", "merge_group", "pull_request_target", "push", "schedule", "workflow_dispatch"} {
want := mode == "remote" && (event == "pull_request" || event == "merge_group")
got := evalRRCIf(t, job.If, map[string]string{
"needs.rbe.outputs.mode": mode,
"github.event_name": event,
})
if got != want {
t.Errorf("%s runs in mode %s on %s: %v, want %v", bazelDoltRaceJobName, mode, event, got, want)
}
}
}
// The runner is the lanes' ternary, which picks Blacksmith only in mode
// remote: the only mode the job runs in.
if job.RunsOn != bazelLaneRunsOn {
t.Errorf("%s runs-on = %q, want %q", bazelDoltRaceJobName, job.RunsOn, bazelLaneRunsOn)
}
}

View file

@ -0,0 +1,508 @@
package scripts_test
import (
"cmp"
"maps"
"path"
"reflect"
"regexp"
"slices"
"strconv"
"strings"
"sync"
"testing"
)
// The dolt-race union lane (item 5): in mode remote, bazel.yml's
// bazel-dolt-race job runs --config=dolt-race, the union of
// --config=doltserver and --config=doltserver-proxied, in one `bazel test`,
// in place of the bazel-doltserver and bazel-proxied jobs; in every other
// mode those two run as before. Its job.status is both members' workflow_call
// output, so pr.yml's gate ids (BAZEL_DOLTSERVER, BAZEL_PROXIED), ci-gate.sh
// and bazel-gate.sh are unchanged. Outside mode remote one client would do
// both tiers' work with no in-flight deduplication, so the union would cost
// the sum of the two tiers rather than the max: the members stay there.
const (
bazelDoltRaceJobName = "bazel-dolt-race"
bazelDoltRaceConfig = "dolt-race"
// Mode remote, on every event that reaches it (same-repo pull requests,
// merge groups, push, nightly, dispatch, bazel-farm.yml's authorized
// call): exactly where the members do not run.
bazelDoltRaceJobIf = "${{ needs.rbe.outputs.mode == 'remote' }}"
// The members: their pre-union rules (bazel-test's skip rule; remote
// execution only), minus mode remote.
bazelDoltMemberIf = "${{ needs.rbe.outputs.mode != 'skip' && needs.rbe.outputs.mode != 'remote' }}"
bazelProxiedMemberIf = "${{ needs.rbe.outputs.enabled == 'true' && needs.rbe.outputs.mode != 'remote' }}"
)
// bazelUnionLanes: a union job -> the gated member jobs it runs in place of
// in mode remote (bazelDoltRaceJobIf). Each member keeps its gate id
// (bazelLaneGateIDs) and its workflow_call output, which reads the union's
// result first and the member's own otherwise (bazelLaneCallOutput).
var bazelUnionLanes = map[string][]string{
bazelDoltRaceJobName: {bazelDoltJobName, bazelProxiedJobName},
}
// bazelUnionOf: the union job a gated member belongs to, or "".
func bazelUnionOf(member string) string {
for union, members := range bazelUnionLanes {
if slices.Contains(members, member) {
return union
}
}
return ""
}
// bazelLaneCallOutput: the workflow_call output value bazel.yml must declare
// for a lane that reports its job.status.
func bazelLaneCallOutput(lane string) string {
if union := bazelUnionOf(lane); union != "" {
return "${{ jobs." + union + ".outputs.result || jobs." + lane + ".outputs.result }}"
}
return "${{ jobs." + lane + ".outputs.result }}"
}
// bazelLaneReporter: the job whose result a gated lane's output carries in
// this mode (lanes: job -> modes it runs in), or "" when neither the lane
// nor its union runs (an expected skip).
func bazelLaneReporter(lanes map[string]map[string]bool, lane, mode string) string {
if lanes[lane][mode] {
return lane
}
if union := bazelUnionOf(lane); union != "" && lanes[union][mode] {
return union
}
return ""
}
var (
bazelCallOutputsMu sync.Mutex
bazelCallOutputsCache map[string]string
)
// bazelCallOutputValue evaluates bazel.yml's workflow_call output name with
// jobs (job -> its result output; missing or "" = not reported), the way
// GitHub does: what pr.yml's needs.bazel.outputs.<name> reads.
func bazelCallOutputValue(t *testing.T, name string, jobs map[string]string) string {
t.Helper()
bazelCallOutputsMu.Lock()
if bazelCallOutputsCache == nil {
bazelCallOutputsCache = map[string]string{}
for n, out := range readBazelWorkflowCall(t).Outputs {
bazelCallOutputsCache[n] = out.Value
}
}
expr, ok := bazelCallOutputsCache[name]
bazelCallOutputsMu.Unlock()
if !ok {
t.Fatalf("%s has no workflow_call output %s", bazelWorkflowName, name)
}
ctx := map[string]string{}
for _, m := range regexp.MustCompile(`jobs\.([A-Za-z0-9_-]+)\.outputs\.([A-Za-z0-9_-]+)`).FindAllStringSubmatch(expr, -1) {
v := ""
if m[2] == "result" {
v = jobs[m[1]]
}
ctx[m[0]] = v
}
v, err := evalGHExpr(expr, ctx)
if err != nil {
t.Fatalf("workflow_call output %s = %q: %v", name, expr, err)
}
str, _ := v.(string)
return str
}
// bazelDoltRaceJobTimeoutMargin: the minutes the union job's timeout-minutes
// must exceed the sum of all its step timeouts (runner setup, post-job
// cleanup), so a hang fails a named step, not the job.
const bazelDoltRaceJobTimeoutMargin = 5
// bazelDoltRaceMergeQueueTimeout: the merge queue's check timeout (ruleset
// merge_queue.check_response_timeout_minutes). The union job is required in
// every merge group, so its own timeout stays below it.
const bazelDoltRaceMergeQueueTimeout = 60
// bazelDoltRaceTestStepTimeout: the union's `bazel test` step timeout. Its
// shards run about 55-95 s on rbe-west, each capped at 1200 s; the step allows
// about 3 minutes past that cap so Bazel writes TIMEOUT logs before GitHub
// kills the step.
const bazelDoltRaceTestStepTimeout = 23
// The union's members, and the tag filter each owns.
var bazelDoltRaceMembers = []struct{ config, tag string }{
{"doltserver", "dolt-server"},
{"doltserver-proxied", "dolt-server-proxied"},
}
// bazelRCConfigLines returns .bazelrc's option lines for --config=name
// (any command: test:name, build:name, ...), as "<command> <option>"
// with the ":name" suffix stripped, in file order.
func bazelRCConfigLines(rc, name string) []string {
var out []string
for _, line := range strings.Split(rc, "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
head, opt, ok := strings.Cut(line, " ")
cmd, cfg, hasCfg := strings.Cut(head, ":")
if !ok || !hasCfg || cfg != name {
continue
}
out = append(out, cmd+" "+strings.TrimSpace(opt))
}
return out
}
// TestBazelDoltRaceConfigIsMemberUnion: --config=dolt-race is exactly the
// set union of its members' lines, except that each member's
// --test_tag_filters is replaced by one filter naming both members' tags.
// Every flag stays a member's own, so each member target's test action is
// unchanged: `bazel aquery 'mnemonic("TestRunner", ...)'` over each member's
// targets gives byte-identical actions (key, argv, env, outputs) under the
// member config and under the union, 27 + 35 = 62 (item 5 A1b). Both arms
// run, the mode deciding which, so drift between them would split results
// by mode: a flag added to a member must be added to the union too.
func TestBazelDoltRaceConfigIsMemberUnion(t *testing.T) {
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
const tagPrefix = "test --test_tag_filters="
var want []string
var tags []string
for _, m := range bazelDoltRaceMembers {
lines := bazelRCConfigLines(rc, m.config)
if len(lines) == 0 {
t.Fatalf(".bazelrc has no --config=%s lines", m.config)
}
var memberTags []string
for _, l := range lines {
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
memberTags = append(memberTags, v)
continue
}
if !slices.Contains(want, l) {
want = append(want, l)
}
}
if len(memberTags) != 1 || memberTags[0] != m.tag {
t.Errorf("--config=%s tag filters %v, want exactly [%s]", m.config, memberTags, m.tag)
}
tags = append(tags, m.tag)
}
got := bazelRCConfigLines(rc, bazelDoltRaceConfig)
var gotTags []string
var gotRest []string
for _, l := range got {
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
gotTags = append(gotTags, v)
continue
}
gotRest = append(gotRest, l)
}
if wantTag := strings.Join(tags, ","); len(gotTags) != 1 || gotTags[0] != wantTag {
t.Errorf("--config=%s tag filters %v, want exactly [%s]", bazelDoltRaceConfig, gotTags, wantTag)
}
if !sameStringSet(gotRest, want) {
t.Errorf("--config=%s is not the union of its members' lines (tag filter aside):\ngot:\n%s\nwant (any order):\n%s",
bazelDoltRaceConfig, strings.Join(gotRest, "\n"), strings.Join(want, "\n"))
}
if len(gotRest) != len(want) {
t.Errorf("--config=%s repeats a line: %d lines for %d distinct member lines", bazelDoltRaceConfig, len(gotRest), len(want))
}
// No nested --config: the union is spelled out, so the policy above
// sees every flag.
for _, l := range got {
if strings.Contains(l, "--config=") {
t.Errorf("--config=%s line %q: spell the members' flags out, no nested --config", bazelDoltRaceConfig, l)
}
}
}
func sameStringSet(a, b []string) bool {
as := map[string]bool{}
for _, s := range a {
as[s] = true
}
bs := map[string]bool{}
for _, s := range b {
bs[s] = true
}
if len(as) != len(bs) {
return false
}
for s := range as {
if !bs[s] {
return false
}
}
return true
}
// TestBazelDoltRaceSoleRunIsInTheUnion: where bazel-doltserver executes
// remotely (the fork modes) it adds --config=sole-run; the union carries that
// config's lines itself, so the dolt-server tier's actions are the same in
// either arm. Every sole-run line is a dolt-race line.
func TestBazelDoltRaceSoleRunIsInTheUnion(t *testing.T) {
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
sole := bazelRCConfigLines(rc, "sole-run")
if len(sole) == 0 {
t.Fatal(".bazelrc has no --config=sole-run lines")
}
union := bazelRCConfigLines(rc, bazelDoltRaceConfig)
for _, l := range sole {
if !slices.Contains(union, l) {
t.Errorf("--config=sole-run line %q is not in --config=%s: bazel-doltserver's remote runs and the union would differ", l, bazelDoltRaceConfig)
}
}
// And the union job takes no sole-run of its own: nothing to add.
test := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltRaceJobName).step(t, "bazel test //... --config="+bazelDoltRaceConfig)
if strings.Contains(test.Run, "BAZEL_SOLE_RUN") || strings.Contains(test.Run, "sole-run") {
t.Errorf("%s test step adds sole-run; --config=%s already has its lines", bazelDoltRaceJobName, bazelDoltRaceConfig)
}
}
// TestBazelDoltRaceIsGated pins the cutover: the union job is a required
// lane like its members (no continue-on-error, the always() job.status
// recorder), it runs the union config with the members' checks once over its
// BEP, and both members' workflow_call outputs read its result first.
func TestBazelDoltRaceIsGated(t *testing.T) {
workflow := readCIWorkflow(t, bazelWorkflowName)
job := workflow.job(t, bazelDoltRaceJobName)
if job.If != bazelDoltRaceJobIf {
t.Errorf("%s if: %q, want %q", bazelDoltRaceJobName, job.If, bazelDoltRaceJobIf)
}
if job.ContinueOnError {
t.Errorf("%s: continue-on-error hides the dolt-server and proxied tiers' failures from pr.yml's ci-gate", bazelDoltRaceJobName)
}
if job.Name != "dolt-race" {
// ci-analytics' collector maps the check name "Bazel / dolt-race"
// to lane dolt-race.
t.Errorf("%s name = %q, want dolt-race", bazelDoltRaceJobName, job.Name)
}
if !slices.Equal(job.Needs, []string{bazelRBEJobName}) {
t.Errorf("%s needs %v, want [%s]", bazelDoltRaceJobName, job.Needs, bazelRBEJobName)
}
if !reflect.DeepEqual(job.Outputs, map[string]string{"result": "${{ steps.result.outputs.result }}"}) {
t.Errorf("%s outputs = %v, want only result from the result step", bazelDoltRaceJobName, job.Outputs)
}
// The test step: the union config, as the member lanes run theirs.
test := job.step(t, "bazel test //... --config="+bazelDoltRaceConfig)
if strings.TrimSpace(test.Run) != bazelTierTestRun(bazelDoltRaceConfig) {
t.Errorf("%s test step run changed; want exactly:\n%s\ngot:\n%s", bazelDoltRaceJobName, bazelTierTestRun(bazelDoltRaceConfig), test.Run)
}
// The members' per-lane checks, once over the union's BEP: bazel-proxied's
// two (bazel-doltserver has none). check_shard_coverage takes the union
// of the members' --suite/--whole arguments, which is bazel-proxied's.
for member, steps := range map[string][]string{
bazelProxiedJobName: {"Every target and shard ran tests", "Every listed test ran in its shard", "No test target is marked flaky"},
} {
m := workflow.job(t, member)
for _, name := range steps {
want, got := m.step(t, name), job.step(t, name)
if strings.TrimSpace(got.Run) != strings.TrimSpace(want.Run) || got.If != want.If {
t.Errorf("%s step %q = (if %q) %q, want %s's (if %q) %q", bazelDoltRaceJobName, name, got.If, got.Run, member, want.If, want.Run)
}
}
}
for _, member := range bazelUnionLanes[bazelDoltRaceJobName] {
for _, step := range workflow.job(t, member).Steps {
if strings.Contains(step.Run, "shard_budget.py") || strings.Contains(step.Run, "check_shard_coverage.py") && member != bazelProxiedJobName {
t.Errorf("%s step %q: a per-lane check the union does not run", member, step.Name)
}
}
}
// No job depends on it, and every member's output reads it first.
for name, other := range workflow.Jobs {
if slices.Contains(other.Needs, bazelDoltRaceJobName) {
t.Errorf("job %s needs %s", name, bazelDoltRaceJobName)
}
}
call := readBazelWorkflowCall(t)
for _, member := range bazelUnionLanes[bazelDoltRaceJobName] {
if got, want := call.Outputs[member].Value, bazelLaneCallOutput(member); got != want {
t.Errorf("workflow_call output %s = %q, want %q", member, got, want)
}
}
for name := range call.Outputs {
if strings.Contains(name, bazelDoltRaceConfig) {
t.Errorf("workflow_call output %s: the union reports through its members' outputs, not one of its own (the gate ids stay)", name)
}
}
}
// TestBazelDoltRaceJobTimeouts: the union's test step has a 23-minute
// timeout, every step has one, and the job's exceeds their sum by
// bazelDoltRaceJobTimeoutMargin minutes (a hang fails a named step) while
// staying under the merge queue's check timeout (a hung required job would
// otherwise drop the group).
func TestBazelDoltRaceJobTimeouts(t *testing.T) {
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltRaceJobName)
test := job.step(t, "bazel test //... --config="+bazelDoltRaceConfig)
if test.TimeoutMinutes != bazelDoltRaceTestStepTimeout {
t.Errorf("%s test step timeout-minutes = %d, want %d", bazelDoltRaceJobName, test.TimeoutMinutes, bazelDoltRaceTestStepTimeout)
}
sum := 0
for i, step := range job.Steps {
if step.TimeoutMinutes <= 0 {
t.Errorf("%s step %d (%q) has no timeout-minutes; a hang would run into the job timeout",
bazelDoltRaceJobName, i, cmp.Or(step.Name, step.Uses))
}
sum += step.TimeoutMinutes
}
if job.TimeoutMinutes < sum+bazelDoltRaceJobTimeoutMargin {
t.Errorf("%s timeout-minutes %d, want at least the sum of its step timeouts (%d) + %d",
bazelDoltRaceJobName, job.TimeoutMinutes, sum, bazelDoltRaceJobTimeoutMargin)
}
if job.TimeoutMinutes >= bazelDoltRaceMergeQueueTimeout {
t.Errorf("%s timeout-minutes %d, want below the merge queue's %d-minute check timeout",
bazelDoltRaceJobName, job.TimeoutMinutes, bazelDoltRaceMergeQueueTimeout)
}
}
// TestBazelDoltRaceArmsExactlyOnePerMode evaluates the union's and the
// members' if: for every execution mode and every event bazel.yml runs on:
// the union runs exactly in mode remote, never beside a member, and each
// member's gate id is reported by exactly one job wherever that member ran
// before the union (bazel-doltserver: every mode but skip; bazel-proxied:
// the remote-executing modes) and by none elsewhere, so bazel-gate.sh's
// per-mode skips are unchanged. The workflow_call outputs, evaluated as
// GitHub does, carry the running arm's result: a failed or cancelled union
// reds both ids, and a union that never reported reads as skipped, which no
// mode it runs in allows.
func TestBazelDoltRaceArmsExactlyOnePerMode(t *testing.T) {
workflow := readCIWorkflow(t, bazelWorkflowName)
union := workflow.job(t, bazelDoltRaceJobName)
members := map[string]struct {
ifExpr string
before func(mode string) bool // where the member ran before the union
}{
bazelDoltJobName: {bazelDoltMemberIf, func(m string) bool { return m != "skip" }},
bazelProxiedJobName: {bazelProxiedMemberIf, func(m string) bool { return bazelRemoteModes[m] }},
}
if got := slices.Sorted(maps.Keys(members)); !slices.Equal(got, slices.Sorted(slices.Values(bazelUnionLanes[bazelDoltRaceJobName]))) {
t.Fatalf("members %v, want bazelUnionLanes' %v", got, bazelUnionLanes[bazelDoltRaceJobName])
}
for name, m := range members {
if got := workflow.job(t, name).If; got != m.ifExpr {
t.Errorf("%s if = %q, want %q", name, got, m.ifExpr)
}
}
for _, mode := range bazelRBEModes {
for _, event := range []string{"pull_request", "merge_group", "pull_request_target", "push", "schedule", "workflow_dispatch"} {
ctx := map[string]string{
"needs.rbe.outputs.mode": mode,
"needs.rbe.outputs.enabled": bazelModeEnabled(mode),
"github.event_name": event,
}
unionRuns := evalRRCIf(t, union.If, ctx)
if unionRuns != (mode == "remote") {
t.Errorf("mode %s, %s: %s runs = %v, want %v", mode, event, bazelDoltRaceJobName, unionRuns, mode == "remote")
}
for name, m := range members {
runs := evalRRCIf(t, workflow.job(t, name).If, ctx)
if unionRuns && runs {
t.Errorf("mode %s, %s: %s and %s both run", mode, event, bazelDoltRaceJobName, name)
}
if got, want := unionRuns || runs, m.before(mode); got != want {
t.Errorf("mode %s, %s: %s's tier reported by a job = %v, want %v (where it ran before the union)", mode, event, name, got, want)
}
for _, res := range []string{"success", "failure", "cancelled", ""} {
jobs := map[string]string{}
switch {
case unionRuns:
jobs[bazelDoltRaceJobName] = res
case runs:
jobs[name] = res
}
want := res
if !unionRuns && !runs {
want = ""
}
if got := bazelCallOutputValue(t, name, jobs); got != want {
t.Errorf("mode %s, %s, reporting job %q: output %s = %q, want %q", mode, event, res, name, got, want)
}
}
}
}
}
// The union's runner is the lanes' ternary: Blacksmith in mode remote,
// the only mode it runs in.
if union.RunsOn != bazelLaneRunsOn {
t.Errorf("%s runs-on = %q, want %q", bazelDoltRaceJobName, union.RunsOn, bazelLaneRunsOn)
}
}
// bazelDoltRaceTestActions: the union's test actions, 27 dolt-server + 35
// proxied-server (aquery, item 5 A1b), each a target's shard.
const bazelDoltRaceTestActions = 62
// TestBazelDoltRaceFitsTheJobsBudget: the union runs both tiers' shards in
// one invocation under --config=remote-exec's --jobs, so all of them must
// fit at once or the slower tier's shards queue behind the other's and the
// union stops ending with the slower member. Every rule tagged with one of
// the union's tags, read from the BUILD files: its shard_count (1 when
// unsharded), summed. A tag outside a one-line `tags = [...]` list cannot
// be counted and fails the test.
func TestBazelDoltRaceFitsTheJobsBudget(t *testing.T) {
root := bazelPolicyRoot(t)
rc := readPolicyFile(t, root, ".bazelrc")
m := regexp.MustCompile(`(?m)^build:remote-exec --jobs=(\d+)$`).FindStringSubmatch(rc)
if m == nil {
t.Fatal(".bazelrc has no build:remote-exec --jobs=N line")
}
budget, _ := strconv.Atoi(m[1])
var tags []string
for _, mem := range bazelDoltRaceMembers {
tags = append(tags, `"`+mem.tag+`"`)
}
ruleStart := regexp.MustCompile(`(?m)^[A-Za-z_][A-Za-z0-9_]*\($`)
total, targets := 0, 0
for _, f := range repoFiles(t, root, 300) {
if path.Base(f) != "BUILD.bazel" {
continue
}
build := readPolicyFile(t, root, f)
for _, line := range strings.Split(build, "\n") {
for _, tag := range tags {
if strings.Contains(line, tag) && !strings.HasPrefix(line, " tags = [") {
t.Errorf("%s: %q outside a one-line tags list: %q", f, tag, line)
}
}
}
for _, loc := range ruleStart.FindAllStringIndex(build, -1) {
end := strings.Index(build[loc[0]:], "\n)\n")
if end < 0 {
continue
}
rule := build[loc[0] : loc[0]+end]
tagged := false
for _, line := range strings.Split(rule, "\n") {
if strings.HasPrefix(line, " tags = [") && slices.ContainsFunc(tags, func(tag string) bool { return strings.Contains(line, tag) }) {
tagged = true
}
}
if !tagged {
continue
}
shards := 1
if sm := shardCountPattern.FindStringSubmatch(rule); sm != nil {
shards, _ = strconv.Atoi(sm[1])
}
total += shards
targets++
}
}
if total != bazelDoltRaceTestActions {
t.Errorf("--config=%s: %d test actions over %d targets, want %d (aquery-verified at the cutover): re-check against --jobs=%d and update", bazelDoltRaceConfig, total, targets, bazelDoltRaceTestActions, budget)
}
if total > budget {
t.Errorf("--config=%s: %d test actions, more than --config=remote-exec's --jobs=%d: the tiers would no longer run at once", bazelDoltRaceConfig, total, budget)
}
}

View file

@ -22,7 +22,7 @@ var ciAnalyticsLaneJobs = map[string]string{
"bazel-proxied": "proxied-server",
"bazel-server-storage": "server-storage",
"bazel-cmd-dolt": "cmd-dolt",
// The advisory dolt-server + proxied union shadow (item 5 Phase 1): its
// The dolt-server + proxied union lane (item 5; mode remote): its
// own lane value, so the A/B queries can pair it with its members.
"bazel-dolt-race": "dolt-race",
}

View file

@ -75,9 +75,9 @@ func bazelRRCCacheReaderRunsOn(label string) string {
// isBazelRRCCacheReader: a remote repo contents cache reader
// (bazelRRCReadLanes) that also runs in mode cache (not remote-only).
// bazel-dolt-race runs only in mode remote on PRs and merge groups (its own
// if, TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls), so it keeps the
// lanes' ubuntu-latest fallback like the remote-only jobs.
// bazel-dolt-race runs only in mode remote (its own if,
// TestBazelDoltRaceArmsExactlyOnePerMode), so it keeps the lanes'
// ubuntu-latest fallback like the remote-only jobs.
func isBazelRRCCacheReader(name string) bool {
return slices.Contains(bazelRRCReadLanes, name) && !bazelRemoteOnlyJobs[name] && name != bazelDoltRaceJobName
}

View file

@ -1881,15 +1881,9 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) {
// bazel.yml's own run - and thus pr.yml's ci-gate - red for a job
// nothing requires.
_, flagGated := bazelFlagGatedLanes[name]
// So is each shadow lane (bazelShadowLanes): it reports into no
// gate, so its failure must not fail the call (BAZEL) either.
_, shadow := bazelShadowLanes[name]
if job.ContinueOnError && name != bazelRBEPrewarmJobName && !flagGated && !shadow {
if job.ContinueOnError && name != bazelRBEPrewarmJobName && !flagGated {
t.Errorf("%s continue-on-error hides failures from pr.yml's ci-gate", name)
}
if shadow && !job.ContinueOnError {
t.Errorf("%s is a shadow lane (%s) but lacks job-level continue-on-error", name, bazelShadowLanes[name])
}
// A flag-gated lane is advisory until its flag says otherwise, and
// an advisory job's failure must not fail the call (BAZEL).
if flagGated && !job.ContinueOnError {
@ -1994,11 +1988,19 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) {
// below only fires for a setup-bazel step).
wantJobRunsOn, wantJobIf = bazelPrewarmRunsOn, bazelRBEPrewarmIf
} else if name == bazelDoltRaceJobName {
// Mode remote, on pull_request and merge_group events only
// (bazel_dolt_race_shadow_test.go); the lanes' runner and
// setup-bazel env otherwise.
// Mode remote only, in place of its members
// (bazel_dolt_race_test.go); the lanes' runner and setup-bazel
// env otherwise.
wantJobIf = bazelDoltRaceJobIf
}
// The union's members: their own rule, minus mode remote
// (TestBazelDoltRaceArmsExactlyOnePerMode).
switch name {
case bazelDoltJobName:
wantJobIf = bazelDoltMemberIf
case bazelProxiedJobName:
wantJobIf = bazelProxiedMemberIf
}
if job.RunsOn != wantJobRunsOn {
t.Errorf("%s runs-on = %q, want %q", name, job.RunsOn, wantJobRunsOn)
}
@ -2543,21 +2545,15 @@ func TestBazelLaneIsGated(t *testing.T) {
if name == bazelRBEJobName || isBazelRRCJob(name) {
continue // the rrc jobs never run in a PR call (bazel_rrc_test.go)
}
if _, shadow := bazelShadowLanes[name]; shadow {
// A shadow lane reports into nothing: no job outputs, so no
// workflow_call output and no gate id
// (TestBazelDoltRaceShadowIsAdvisory).
if len(job.Outputs) != 0 {
t.Errorf("shadow lane %s outputs = %v, want none", name, job.Outputs)
}
continue
}
_, gated := bazelLaneGateIDs[name]
_, advisory := bazelAdvisoryLanes[name]
_, flagGated := bazelFlagGatedLanes[name]
if n := btoi(gated) + btoi(advisory) + btoi(flagGated); n != 1 {
t.Errorf("%s job %s: gated=%v advisory=%v flag-gated=%v; add it to exactly one of bazelLaneGateIDs (with a CI_GATE_REQUIRED id in pr.yml), bazelFlagGatedLanes (with its pr.yml flag) or bazelAdvisoryLanes (with the reason)",
bazelWorkflowName, name, gated, advisory, flagGated)
// A union lane reports through its gated members' ids and outputs
// (bazel_dolt_race_test.go).
_, union := bazelUnionLanes[name]
if n := btoi(gated) + btoi(advisory) + btoi(flagGated) + btoi(union); n != 1 {
t.Errorf("%s job %s: gated=%v advisory=%v flag-gated=%v union=%v; add it to exactly one of bazelLaneGateIDs (with a CI_GATE_REQUIRED id in pr.yml), bazelFlagGatedLanes (with its pr.yml flag), bazelUnionLanes (with the gated members it runs in place of) or bazelAdvisoryLanes (with the reason)",
bazelWorkflowName, name, gated, advisory, flagGated, union)
}
// Each lane reports its job.status from an always() last step.
if !reflect.DeepEqual(job.Outputs, map[string]string{"result": "${{ steps.result.outputs.result }}"}) {
@ -2568,7 +2564,19 @@ func TestBazelLaneIsGated(t *testing.T) {
last.Run != `echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"` {
t.Errorf("%s last step = %+v; want the always() job.status recorder", name, last)
}
wantCallOutputs[name] = "${{ jobs." + name + ".outputs.result }}"
if !union {
wantCallOutputs[name] = bazelLaneCallOutput(name)
}
}
for union, members := range bazelUnionLanes {
if _, ok := workflow.Jobs[union]; !ok {
t.Errorf("bazelUnionLanes lists %s, which %s does not have", union, bazelWorkflowName)
}
for _, m := range members {
if _, gated := bazelLaneGateIDs[m]; !gated {
t.Errorf("union lane %s member %s is not a gated lane", union, m)
}
}
}
for name := range bazelLaneGateIDs {
if _, ok := workflow.Jobs[name]; !ok {
@ -2858,6 +2866,12 @@ func bazelLaneRunModes(t *testing.T, lane, ifExpr string, with map[string]string
return map[string]bool{"remote": true, "fork-ro": true, "fork-rw": true, "cache": true, "local": true}
case "${{ needs.rbe.outputs.enabled == 'true' }}":
return map[string]bool{"remote": true, "fork-ro": true, "fork-rw": true}
case bazelDoltMemberIf:
// The dolt-server tier's own job, where bazel-dolt-race does not run
// (TestBazelDoltRaceArmsExactlyOnePerMode).
return map[string]bool{"fork-ro": true, "fork-rw": true, "cache": true, "local": true}
case bazelProxiedMemberIf:
return map[string]bool{"fork-ro": true, "fork-rw": true}
case bazelIntegIf:
if strings.EqualFold(with["integration"], "off") {
return map[string]bool{}
@ -2884,14 +2898,8 @@ func bazelLaneRunModes(t *testing.T, lane, ifExpr string, with map[string]string
// on the job: B1 removed fork-rw). Unlike the historical advisory
// lanes, this one legitimately runs in pr.yml's own call (see
// bazelAdvisoryLanes' name-specific exception in
// TestBazelGateSimulation below).
return map[string]bool{"remote": true}
case bazelDoltRaceJobIf:
// Mode remote, and only on pull_request and merge_group events:
// within pr.yml's call (the only PR-time caller these simulations
// model) that is mode remote. bazel-farm.yml's pull_request_target
// call, push, nightly and dispatch never run it
// (TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls).
// TestBazelGateSimulation below). bazel-dolt-race's if
// (bazelDoltRaceJobIf) is the same expression: mode remote.
return map[string]bool{"remote": true}
case bazelRRCSeedJobIf, bazelRRCVerifyJobIf:
// The remote repo contents cache's writer (push to main) and
@ -2909,7 +2917,7 @@ type bazelGateScenario struct {
event string
mode, enabled string // the call's rbe-mode / rbe-enabled outputs
call string // needs.bazel.result
outputs map[string]string // the lanes' outputs ("" = not reported)
outputs map[string]string // the jobs' result outputs ("" = not reported)
// Non-Bazel needs' results that differ from success.
results map[string]string
// pr.yml's top-level flags the run script reads (the workflow env is
@ -2977,7 +2985,9 @@ func runPRGateStep(t *testing.T, step ciWorkflowStep, sc bazelGateScenario) (boo
case m[3] == "rbe-enabled":
got = sc.enabled
default:
got = sc.outputs[m[3]]
// The call's output as GitHub evaluates it over the jobs'
// results (a union lane's members read its result first).
got = bazelCallOutputValue(t, m[3], sc.outputs)
}
if got == "" && m[4] != "" {
got = "skipped"
@ -3021,8 +3031,10 @@ func TestBazelGateSimulation(t *testing.T) {
if _, advisory := bazelAdvisoryLanes[name]; advisory && name != bazelRBEPrewarmJobName && len(lanes[name]) != 0 {
t.Errorf("advisory lane %s runs in pr.yml's call in modes %v; turn it off there (%s)", name, lanes[name], bazelAdvisoryLanes[name])
}
if _, gated := bazelLaneGateIDs[name]; gated && !lanes[name]["remote"] {
t.Errorf("gated lane %s does not run in pr.yml's call even in remote mode", name)
}
for lane := range bazelLaneGateIDs {
if bazelLaneReporter(lanes, lane, "remote") == "" {
t.Errorf("gated lane %s does not run (nor does its union lane) in pr.yml's call even in remote mode", lane)
}
}
enabledFor := bazelModeEnabled
@ -3054,9 +3066,14 @@ func TestBazelGateSimulation(t *testing.T) {
}
if modes[mode] {
anyRuns = true
} else if id, gated := bazelLaneGateIDs[lane]; gated {
wantSkips = append(wantSkips, id)
} else if g, flagGated := bazelFlagGatedLanes[lane]; flagGated {
}
if id, gated := bazelLaneGateIDs[lane]; gated {
// Skipped by design only when neither the lane nor its
// union lane runs in this mode.
if bazelLaneReporter(lanes, lane, mode) == "" {
wantSkips = append(wantSkips, id)
}
} else if g, flagGated := bazelFlagGatedLanes[lane]; !modes[mode] && flagGated {
// Listed even while advisory, so flipping the flag is the
// only change that makes it required.
wantSkips = append(wantSkips, g.id)
@ -3144,6 +3161,9 @@ func TestBazelGateSimulation(t *testing.T) {
}
continue
}
if _, union := bazelUnionLanes[lane]; union {
continue // through its members' ids, below
}
id, gated := bazelLaneGateIDs[lane]
if !gated {
// Advisory: not in the call, so it cannot report, and
@ -3153,18 +3173,31 @@ func TestBazelGateSimulation(t *testing.T) {
}
continue
}
// The job whose result the lane's id reads in this mode: the
// lane itself, or its union lane (bazel-dolt-race in mode
// remote, for bazel-doltserver and bazel-proxied).
reporter := bazelLaneReporter(lanes, lane, mode)
if !modes[mode] {
// Skipped by design (baseline); if it ran anyway and failed,
// the gate still sees it.
sc(lane+" ran and failed", "failure", with(lane, "failure"), false, id)
// Not its own job in this mode (skipped by design, or its
// union reports); if it ran anyway and failed, the gate still
// sees it: through its id, or, where the union's result
// comes first in the output, through the aggregate.
out := with(lane, "failure")
mention := id
if bazelCallOutputValue(t, lane, out) != "failure" {
mention = bazelAggregateGateID
}
sc(lane+" ran and failed", "failure", out, false, mention)
}
if reporter == "" {
continue
}
for _, call := range []string{"success", "failure"} {
sc(lane+" alone skipped, aggregate "+call, call, with(lane, ""), false, id)
sc(lane+" failed, aggregate "+call, call, with(lane, "failure"), false, id)
sc(lane+" cancelled, aggregate "+call, call, with(lane, "cancelled"), false, id)
sc(reporter+" (for "+lane+") alone skipped, aggregate "+call, call, with(reporter, ""), false, id)
sc(reporter+" (for "+lane+") failed, aggregate "+call, call, with(reporter, "failure"), false, id)
sc(reporter+" (for "+lane+") cancelled, aggregate "+call, call, with(reporter, "cancelled"), false, id)
}
sc(lane+" cancelled, aggregate cancelled", "cancelled", with(lane, "cancelled"), false, id)
sc(reporter+" (for "+lane+") cancelled, aggregate cancelled", "cancelled", with(reporter, "cancelled"), false, id)
}
}
@ -4590,11 +4623,8 @@ func TestBazelWorkflowSecretsAndFailureSurface(t *testing.T) {
// (bazelCIAnalyticsStepNames): reporting only, never a `needs`
// of anything and never read back by this workflow or its
// callers.
// And each shadow lane's (bazelShadowLanes): it reports into no
// gate, so its failure must not fail the call either.
_, flagGated := bazelFlagGatedLanes[strings.TrimSuffix(strings.TrimPrefix(path, ".jobs."), ".continue-on-error")]
_, shadow := bazelShadowLanes[strings.TrimSuffix(strings.TrimPrefix(path, ".jobs."), ".continue-on-error")]
if path != bazelRBEPrewarmContinueOnErrorPath && !flagGated && !shadow && !ciAnalyticsPaths[path] {
if path != bazelRBEPrewarmContinueOnErrorPath && !flagGated && !ciAnalyticsPaths[path] {
t.Errorf("%s: %s hides failures from pr.yml's ci-gate", bazelWorkflowName, path)
}
}

View file

@ -197,8 +197,8 @@ func TestPRGateRequiresRemoteBazelLanes(t *testing.T) {
t.Errorf("pr.yml's ci-gate env sets %s; only its run script may", bazelRemoteGateID)
}
for lane, id := range bazelLaneGateIDs {
if !lanes[lane]["remote"] {
t.Errorf("gated lane %s does not run in mode remote", lane)
if bazelLaneReporter(lanes, lane, "remote") == "" {
t.Errorf("gated lane %s does not run (nor does its union lane) in mode remote", lane)
}
if !contains(required, id) && !bazelPackageJobs[lane] {
t.Errorf("pr.yml's ci-gate does not require %s", id)
@ -266,13 +266,13 @@ func TestPRGateRequiresRemoteBazelLanes(t *testing.T) {
}
}
// Remote, but one gated lane failed, was cancelled or reported nothing:
// red, naming that lane.
// Remote, but one gated lane (or the union lane reporting for it)
// failed, was cancelled or reported nothing: red, naming that lane.
for lane, id := range bazelLaneGateIDs {
for _, mode := range []string{"remote", "fork-ro"} {
for _, res := range []string{"failure", "cancelled", ""} {
sc := prGateFor(t, lanes, "pull_request", mode)
sc.outputs[lane] = res
sc.outputs[bazelLaneReporter(lanes, lane, mode)] = res
if pass, log := runPRGateStep(t, gateStep, sc); pass || !regexp.MustCompile(`::error::`+id+`\b`).MatchString(log) {
t.Errorf("mode %s, lane %s %q: gate pass = %v, want red naming %s\n%s", mode, lane, res, pass, id, log)
}
@ -1170,7 +1170,7 @@ func TestBazelRetiredLanesArePinned(t *testing.T) {
}
evictionAllowed[want] = true
}
// The dolt-race shadow's union config carries its proxied member's
// The dolt-race union config carries its proxied member's
// line (TestBazelDoltRaceConfigIsMemberUnion pins the union).
evictionAllowed["test:"+bazelDoltRaceConfig+" --experimental_remote_cache_eviction_retries=0"] = true
for _, line := range strings.Split(rc, "\n") {