ci(bazel): dolt-race gates the dolt-server and proxied tiers in mode remote (item 5 A1b) (#7514)
* ci(bazel): dolt-race gates the dolt-server and proxied tiers in mode remote (item 5 A1b) In mode remote, bazel-dolt-race (`bazel test //... --config=dolt-race`, the union of --config=doltserver and --config=doltserver-proxied) runs both tiers in place of bazel-doltserver and bazel-proxied, and is no longer advisory: its job.status is both the bazel-doltserver and the bazel-proxied workflow_call output (union result first, member's otherwise). In the fork modes, cache and local, the two member jobs run exactly as before: there one client does both tiers' work without in-flight deduplication, so a union would cost the sum of the tiers rather than the max. Exactly one arm runs in any mode, so gate ids, pr.yml, ci-gate.sh and bazel-gate.sh are unchanged. - bazel.yml: dolt-race if: mode == 'remote' (every event that reaches it); no continue-on-error; result output and always() recorder; test step timeout 25 -> 20 (job 55 >= step sum 46 + 5, < 60). Members' if: their old rule && mode != 'remote'. - scripts/bazel_dolt_race_test.go (was bazel_dolt_race_shadow_test.go): union-of-members config, sole-run lines in the union, gating shape, step and job timeouts, exactly one arm per mode x event with the call outputs evaluated as GitHub does, and the union's 62 test actions within --config=remote-exec's --jobs=64. - Gate simulations evaluate the workflow_call output expressions, and a gated lane's reporter is the lane or its union lane. - aquery: all 62 TestRunner actions under --config=dolt-race are byte-identical (key, argv, env, outputs) to --config=doltserver (with and without sole-run) and --config=doltserver-proxied on origin/main. - Docs: CI_REQUIRED_CHECK_TOPOLOGY.md "dolt-race Union Lane", TESTING.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(bazel): dolt-race test step 23 min so Bazel logs a 1200 s TIMEOUT first The proxied tier caps each test at 1200 s; a 20-minute step let GitHub kill a hung shard before Bazel wrote its TIMEOUT test.log and goroutine dump. Step 23, job 56 (step sum + 5, under the merge queue's 60). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(bazel): say the dolt-race job timeout bounds the job, not the run Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
ec55baa425
commit
85464f567a
13 changed files with 736 additions and 474 deletions
25
.bazelrc
25
.bazelrc
|
|
@ -178,20 +178,21 @@ test:doltserver-proxied --experimental_remote_cache_eviction_retries=0
|
|||
# above.
|
||||
test:doltserver-proxied --norun_validations
|
||||
|
||||
# dolt-server + proxied union (item 5, Phase 1 shadow): bazel.yml's advisory
|
||||
# bazel-dolt-race job runs both tiers in one invocation, beside their own
|
||||
# lanes, to measure one client per group of lanes before any cutover. Exactly
|
||||
# the union of --config=doltserver's and --config=doltserver-proxied's lines,
|
||||
# with one tag filter naming both tiers
|
||||
# (scripts/bazel_dolt_race_shadow_test.go). Every other flag is the same in
|
||||
# both members or outside any action key for the other: the race build,
|
||||
# dolt-server + proxied union (item 5): bazel.yml's bazel-dolt-race job runs
|
||||
# both tiers in one invocation in mode remote, in place of the bazel-doltserver
|
||||
# and bazel-proxied lanes, which still run --config=doltserver and
|
||||
# --config=doltserver-proxied in every other mode. Exactly the union of the two
|
||||
# member configs' lines, with one tag filter naming both tiers
|
||||
# (scripts/bazel_dolt_race_test.go). Every other flag is the same in both
|
||||
# members or outside any action key for the other: the race build,
|
||||
# -test.parallel=4 and GO_TEST_WRAP_TESTV=1 are common; the eternal timeout
|
||||
# applies to no dolt-server target; the eviction retry count is what
|
||||
# doltserver already gets from --config=sole-run in mode remote, the only mode
|
||||
# the shadow runs in; test_summary, local_test_jobs and remote_download_regex
|
||||
# are not in any action key. So each member target's test action is
|
||||
# unchanged, and the two lanes' cached results carry over. The member configs
|
||||
# stay for make, developers and the member lanes.
|
||||
# doltserver gets from --config=sole-run wherever it executes remotely;
|
||||
# test_summary, local_test_jobs and remote_download_regex are not in any action
|
||||
# key. So each member target's test action is unchanged (aquery: all 62
|
||||
# TestRunner actions byte-identical), and whichever arm the mode picks shares
|
||||
# cached results with the other. The member configs stay for those modes, make
|
||||
# and developers.
|
||||
test:dolt-race --@rules_go//go/config:race
|
||||
test:dolt-race --test_tag_filters=dolt-server,dolt-server-proxied
|
||||
test:dolt-race --build_tests_only
|
||||
|
|
|
|||
132
.github/workflows/bazel.yml
vendored
132
.github/workflows/bazel.yml
vendored
|
|
@ -46,11 +46,14 @@
|
|||
# only what the PR changed is built and tested on the runner, and it is
|
||||
# their only PR-time integration run (no legacy job runs it on PRs). Mode
|
||||
# local (rbe=off) still skips it. Direct runs (push to main,
|
||||
# workflow_dispatch, nightly.yml's call) gate nothing. bazel-dolt-race (the
|
||||
# dolt-server + proxied union shadow, mode remote on PRs and merge groups
|
||||
# only) cannot fail the gate either: no output, job-level continue-on-error.
|
||||
# The gate still waits for it, as for every job of the call (see its comment
|
||||
# and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "dolt-race Shadow").
|
||||
# workflow_dispatch, nightly.yml's call) gate nothing. In mode remote the
|
||||
# dolt-server and proxied-server tiers run as one union job, bazel-dolt-race
|
||||
# (`bazel test --config=dolt-race`), whose job.status is both the
|
||||
# bazel-doltserver and the bazel-proxied output; in every other mode
|
||||
# bazel-doltserver and bazel-proxied run as before and report their own.
|
||||
# Exactly one of the two arms runs in any mode, so the gate ids
|
||||
# (BAZEL_DOLTSERVER, BAZEL_PROXIED) and bazel-gate.sh's skips are unchanged
|
||||
# (engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "dolt-race Union Lane").
|
||||
#
|
||||
# Execution mode, decided once by the rbe job (every other job needs it and
|
||||
# reads its outputs; none re-derives the condition) and passed to
|
||||
|
|
@ -346,12 +349,14 @@ on:
|
|||
bazel-integration:
|
||||
description: "bazel-integration's job.status"
|
||||
value: ${{ jobs.bazel-integration.outputs.result }}
|
||||
# Mode remote: bazel-dolt-race runs both tiers and the member jobs
|
||||
# skip; every other mode: the reverse. One side is always empty.
|
||||
bazel-doltserver:
|
||||
description: "bazel-doltserver's job.status"
|
||||
value: ${{ jobs.bazel-doltserver.outputs.result }}
|
||||
description: "the dolt-server tier's job.status: bazel-dolt-race's in mode remote, bazel-doltserver's otherwise"
|
||||
value: ${{ jobs.bazel-dolt-race.outputs.result || jobs.bazel-doltserver.outputs.result }}
|
||||
bazel-proxied:
|
||||
description: "bazel-proxied's job.status"
|
||||
value: ${{ jobs.bazel-proxied.outputs.result }}
|
||||
description: "the proxied-server tier's job.status: bazel-dolt-race's in mode remote, bazel-proxied's otherwise"
|
||||
value: ${{ jobs.bazel-dolt-race.outputs.result || jobs.bazel-proxied.outputs.result }}
|
||||
bazel-server-storage:
|
||||
description: "bazel-server-storage's job.status"
|
||||
value: ${{ jobs.bazel-server-storage.outputs.result }}
|
||||
|
|
@ -1743,12 +1748,18 @@ jobs:
|
|||
# doctor/fix, protocol in 8 shards, testutil), each of which starts its
|
||||
# own dolt sql-server from the pinned dolt, so they execute on the remote
|
||||
# workers like any other test (locally with rbe=off).
|
||||
# Not in mode remote: there bazel-dolt-race runs this tier together with
|
||||
# the proxied-server tier and reports for both (item 5). In the fork
|
||||
# modes, cache and local there is no in-flight deduplication of one
|
||||
# client's actions and the client does the work, so a union would cost the
|
||||
# sum of the two tiers instead of the max: this job runs there as before.
|
||||
bazel-doltserver:
|
||||
name: dolt-server lane
|
||||
# Same runner and skip rule as bazel-test.
|
||||
# Same runner as bazel-test; bazel-test's skip rule, plus mode remote
|
||||
# (bazel-dolt-race's).
|
||||
needs: rbe
|
||||
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
|
||||
if: ${{ needs.rbe.outputs.mode != 'skip' }}
|
||||
if: ${{ needs.rbe.outputs.mode != 'skip' && needs.rbe.outputs.mode != 'remote' }}
|
||||
timeout-minutes: 90
|
||||
outputs:
|
||||
result: ${{ steps.result.outputs.result }}
|
||||
|
|
@ -1879,7 +1890,8 @@ jobs:
|
|||
# (formerly proxied-local-smoke.yml) in a loopback-only network
|
||||
# namespace, which needs the workers' unprivileged user namespaces. A job
|
||||
# of its own, not gated by any integration switch: it runs in pr.yml's
|
||||
# call and its gate requires it (BAZEL_PROXIED).
|
||||
# call and its gate requires it (BAZEL_PROXIED). Fork modes only: in mode
|
||||
# remote bazel-dolt-race runs this tier (see bazel-doltserver's comment).
|
||||
bazel-proxied:
|
||||
name: proxied-server tier
|
||||
# Remote execution only, like bazel-embedded: 30 concurrent shards, each
|
||||
|
|
@ -1889,7 +1901,7 @@ jobs:
|
|||
# retired) and pr.yml's gate requires it to have run here and passed.
|
||||
needs: rbe
|
||||
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
||||
if: ${{ needs.rbe.outputs.enabled == 'true' }}
|
||||
if: ${{ needs.rbe.outputs.enabled == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
||||
timeout-minutes: 30
|
||||
outputs:
|
||||
result: ${{ steps.result.outputs.result }}
|
||||
|
|
@ -2033,45 +2045,37 @@ jobs:
|
|||
JOB_STATUS: ${{ job.status }}
|
||||
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Item 5, Phase 1 (scripts/bazel_dolt_race_shadow_test.go): the advisory
|
||||
# shadow of a union lane. One `bazel test --config=dolt-race` runs the
|
||||
# dolt-server and proxied-server tiers together (.bazelrc: exactly the
|
||||
# union of --config=doltserver's and --config=doltserver-proxied's lines,
|
||||
# one tag filter for both), beside bazel-doltserver and bazel-proxied, so
|
||||
# one client per group of lanes can be compared with the two it would
|
||||
# replace (end time, runner minutes, test counts and results) before any
|
||||
# cutover. Each member target's test action key is unchanged, so the shadow
|
||||
# and the member lanes share cached results; and rbe-west deduplicates
|
||||
# identical in-flight executions (confirmed by the farm maintainer:
|
||||
# NativeLink 1.7.1 subscribes a second identical Execute of a cacheable
|
||||
# action to the operation already in flight), so the shadow adds no
|
||||
# executions.
|
||||
# Advisory: it cannot fail the gate. Job-level continue-on-error (a failed
|
||||
# job otherwise fails the whole call, whose result pr.yml's ci-gate
|
||||
# requires as BAZEL), no workflow_call output, no job outputs, no job in
|
||||
# this file needs it, and no gate script or caller names it. The gate does
|
||||
# wait for it: ci-gate needs pr.yml's bazel call, which completes only when
|
||||
# every job in it has finished (success, failure or timeout), so a slow
|
||||
# shadow delays the gate and the merge queue. continue-on-error does not
|
||||
# cover a job GitHub cancels at its own timeout-minutes, which could turn
|
||||
# the call red: every step has a timeout, and the job's exceeds their sum
|
||||
# (TestBazelDoltRaceShadowIsAdvisory). Mode remote only, on
|
||||
# same-repo pull requests and merge groups (mode remote on a pull_request
|
||||
# is never a fork): no fork mode, no cache or local mode, and not on push,
|
||||
# nightly, dispatch or bazel-farm.yml's call. Its ci-analytics LANE is
|
||||
# dolt-race. Removing it is the rollback; nothing reads it.
|
||||
# Item 5 (scripts/bazel_dolt_race_test.go): the dolt-server and
|
||||
# proxied-server tiers in one `bazel test --config=dolt-race` (.bazelrc:
|
||||
# exactly the union of --config=doltserver's and
|
||||
# --config=doltserver-proxied's lines, one tag filter for both), mode
|
||||
# remote only, in place of bazel-doltserver and bazel-proxied, which run
|
||||
# in every other mode. One client waits on rbe-west for both tiers'
|
||||
# shards at once (27 + 35 test actions, under the remote-exec config's 64
|
||||
# jobs), so the job ends with the slower tier and saves the other's
|
||||
# runner. Each member target's test action key is unchanged (aquery-
|
||||
# verified), so cached results carry over both ways, and a revert of the
|
||||
# cutover re-executes nothing. Gated: its job.status is both the
|
||||
# bazel-doltserver and the bazel-proxied workflow_call output (a failure
|
||||
# in either tier fails both ids, which are both required). Every step has
|
||||
# a timeout and the job's exceeds their sum by 5 minutes while staying
|
||||
# under the merge queue's 60-minute check timeout
|
||||
# (TestBazelDoltRaceJobTimeouts), so a hang fails a named step instead of
|
||||
# dropping the merge group. The member jobs' configs and the union's must
|
||||
# stay key-identical: the mode decides which runs (TestBazelDoltRaceConfigIsMemberUnion).
|
||||
# Its ci-analytics LANE is dolt-race.
|
||||
bazel-dolt-race:
|
||||
name: dolt-race
|
||||
needs: rbe
|
||||
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
||||
if: ${{ needs.rbe.outputs.mode == 'remote' && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }}
|
||||
# At least every step's timeout-minutes plus 5 (and the test step's plus
|
||||
# 10): a hung step then fails the job (continue-on-error) before GitHub
|
||||
# cancels it, which continue-on-error does not cover
|
||||
# (TestBazelDoltRaceShadowIsAdvisory). bazel-proxied's steps have no
|
||||
# timeouts but its test step's; these bound the rest.
|
||||
timeout-minutes: 55
|
||||
continue-on-error: true
|
||||
if: ${{ needs.rbe.outputs.mode == 'remote' }}
|
||||
# At least every step's timeout-minutes plus 5, and below the merge
|
||||
# queue's 60-minute check timeout so this job alone cannot exhaust it
|
||||
# (TestBazelDoltRaceJobTimeouts). Not a whole-run bound: the rbe job runs
|
||||
# first and sibling lanes have longer timeouts.
|
||||
timeout-minutes: 56
|
||||
outputs:
|
||||
result: ${{ steps.result.outputs.result }}
|
||||
steps:
|
||||
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
||||
# checkout v7 refuses a fork head on pull_request_target unless opted
|
||||
|
|
@ -2120,11 +2124,15 @@ jobs:
|
|||
fi
|
||||
|
||||
# Both tiers' shards at once: 27 dolt-server + 35 proxied test actions,
|
||||
# under the remote-exec config's 64 jobs. The step timeout is
|
||||
# bazel-proxied's (the longer tier on rbe-west).
|
||||
# under the remote-exec config's 64 jobs. No --config=sole-run: its one
|
||||
# line (no retry on a remote cache eviction) is already in the union.
|
||||
# Shards are expected around 55-95s on rbe-west (see bazel-proxied's
|
||||
# test step), the proxied tier's 1200s cap included. The step timeout is
|
||||
# >= proxied test_timeout 1200 s + about 3 min so Bazel writes TIMEOUT
|
||||
# logs before GitHub kills the step.
|
||||
- name: bazel test //... --config=dolt-race
|
||||
id: test
|
||||
timeout-minutes: 25
|
||||
timeout-minutes: 23
|
||||
run: |
|
||||
set -o pipefail
|
||||
start=$(date +%s)
|
||||
|
|
@ -2138,12 +2146,11 @@ jobs:
|
|||
echo "bazel test --config=dolt-race: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
||||
exit "$rc"
|
||||
|
||||
# The member lanes' checks over the union's BEP: bazel-proxied's (the
|
||||
# dolt-server lane has none). check_shard_coverage ignores targets it
|
||||
# does not list; check_testcases checks every target, so it covers the
|
||||
# dolt-server tier too, which bazel-doltserver does not check: a
|
||||
# dolt-server shard that runs no test turns this job red while both
|
||||
# member lanes stay green.
|
||||
# The member lanes' checks, once over the union's BEP: bazel-proxied's
|
||||
# (the dolt-server lane has none). check_shard_coverage ignores targets
|
||||
# it does not list; check_testcases checks every target, so it covers
|
||||
# the dolt-server tier too, which bazel-doltserver does not check: a
|
||||
# dolt-server shard that runs no test turns this job red.
|
||||
- name: Every target and shard ran tests
|
||||
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
||||
timeout-minutes: 2
|
||||
|
|
@ -2201,6 +2208,17 @@ jobs:
|
|||
retention-days: 7
|
||||
if-no-files-found: ignore
|
||||
|
||||
# Last step: the job's result as the bazel-doltserver and
|
||||
# bazel-proxied workflow_call outputs for pr.yml's ci-gate. always()
|
||||
# runs it after a failure, a timeout or a cancellation too.
|
||||
- name: Record job result
|
||||
id: result
|
||||
if: ${{ always() }}
|
||||
timeout-minutes: 1
|
||||
env:
|
||||
JOB_STATUS: ${{ job.status }}
|
||||
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# pr-risk.yml's server-Dolt storage tier, "Test (Server Dolt Conformance)"
|
||||
# and "Test (Server Dolt Full Suite N/16)", as --config=doltserver-integration:
|
||||
# the conformance and 16-shard full-suite variants of
|
||||
|
|
|
|||
4
.github/workflows/pr-risk.yml
vendored
4
.github/workflows/pr-risk.yml
vendored
|
|
@ -24,8 +24,8 @@ env:
|
|||
# The legacy Go test tiers this workflow used to run (embedded Dolt storage,
|
||||
# conformance and cmd shards; proxied-server cmd shards; server-Dolt storage
|
||||
# and full suite) are pr.yml's Bazel lanes on rbe-west now: `embedded Dolt
|
||||
# tier`, `proxied-server tier` and `server-Dolt storage tier` (bazel.yml),
|
||||
# which pr.yml's `CI Gate / Required` requires to have run remotely and
|
||||
# tier`, `proxied-server tier` (`dolt-race`, with the dolt-server lane, in
|
||||
# mode remote) and `server-Dolt storage tier` (bazel.yml), which pr.yml's `CI Gate / Required` requires to have run remotely and
|
||||
# passed on every pull_request and merge group, fork and Dependabot PRs
|
||||
# included (ga-96smfk.22). What remains is the Nix packaging check, which no
|
||||
# Bazel target can stand in for: it builds the package Nix users install.
|
||||
|
|
|
|||
|
|
@ -54,7 +54,10 @@ Current PR-related workflow names:
|
|||
repo variable is unset) and exports it as the `rbe-mode` / `rbe-enabled`
|
||||
outputs (`rbe-enabled` is `true` in `remote`, `fork-ro` and `fork-rw`);
|
||||
every lane exports its
|
||||
`job.status` as an output named after the job. `pr.yml`'s gate requires the
|
||||
`job.status` as an output named after the job (in mode `remote`, the
|
||||
`bazel-doltserver` and `bazel-proxied` outputs carry the `bazel-dolt-race`
|
||||
union job's status instead; see [dolt-race Union Lane](#dolt-race-union-lane)).
|
||||
`pr.yml`'s gate requires the
|
||||
call's result (`BAZEL`) and `BAZEL_TEST`, `BAZEL_PURE`,
|
||||
`BAZEL_RELEASE_CROSS`, `BAZEL_EMBEDDED`, `BAZEL_INTEGRATION`,
|
||||
`BAZEL_DOLTSERVER`, `BAZEL_PROXIED` and `BAZEL_SERVER_STORAGE`. These lanes are the only CI run of the Linux Go
|
||||
|
|
@ -620,38 +623,41 @@ job's `if:` or rotating the key, in either of two ways:
|
|||
step's own kill switch, checked before any network call. Deleting the
|
||||
variable does not disable pre-warming; it restores the default of 1.
|
||||
|
||||
## dolt-race Shadow (Not Required)
|
||||
## dolt-race Union Lane
|
||||
|
||||
`bazel.yml`'s `bazel-dolt-race` job (check name `Bazel / dolt-race`) is an
|
||||
advisory shadow, not a required check. It runs
|
||||
`bazel test //... --config=dolt-race`, the union of `--config=doltserver` and
|
||||
`--config=doltserver-proxied` with one tag filter for both tiers, beside the
|
||||
`bazel-doltserver` and `bazel-proxied` lanes. Its purpose is to measure one
|
||||
client per group of lanes against the two lanes it would replace (end time,
|
||||
runner minutes, test counts and results) before any cutover. Each member
|
||||
target's test action key is unchanged, so the shadow shares cached results
|
||||
with the member lanes.
|
||||
In mode `remote`, `bazel.yml`'s `bazel-dolt-race` job (check name
|
||||
`Bazel / dolt-race`) runs the dolt-server and proxied-server tiers in one
|
||||
`bazel test //... --config=dolt-race`, in place of the `bazel-doltserver`
|
||||
(`dolt-server lane`) and `bazel-proxied` (`proxied-server tier`) jobs.
|
||||
`--config=dolt-race` is exactly the union of `--config=doltserver` and
|
||||
`--config=doltserver-proxied`, with one tag filter for both tiers, so every
|
||||
member target's test action is unchanged (`bazel aquery`: the 27 + 35 = 62
|
||||
`TestRunner` actions are byte-identical under the member configs and the
|
||||
union). One client waits on rbe-west for both tiers at once, so the job ends
|
||||
with the slower tier and the other tier's runner is saved.
|
||||
|
||||
- It runs in mode `remote` on `pull_request` and `merge_group` events only.
|
||||
It never runs in a fork mode (`fork-ro`, `fork-rw`), in mode `cache` or
|
||||
`local`, on `bazel-farm.yml`'s call, on push, nightly or dispatch.
|
||||
- It cannot fail the gate. It has job-level `continue-on-error`, so its
|
||||
failure cannot fail the call's aggregate result (`BAZEL`). It has no
|
||||
`workflow_call` output, no `bazel.yml` job `needs` it, and `pr.yml`,
|
||||
`ci-gate.sh` and `bazel-gate.sh` never name it.
|
||||
- The gate still waits for it. `CI Gate` needs `pr.yml`'s `bazel` call, and
|
||||
a reusable-workflow call completes only when all of its jobs have finished
|
||||
(success, failure or timeout), `continue-on-error` ones included. A slow
|
||||
shadow therefore delays `CI Gate` and the merge queue, up to its
|
||||
timeout. `continue-on-error` does not cover a job GitHub cancels at its
|
||||
own `timeout-minutes`, so every step has its own timeout and the job's
|
||||
(65 minutes) stays at least 5 minutes above their sum and 10 above the
|
||||
test step's (policy-tested). A hung step then fails the job, which
|
||||
`continue-on-error` covers, before GitHub would cancel it.
|
||||
- Mode `remote` only (same-repo pull requests, merge groups, push, nightly,
|
||||
dispatch and `bazel-farm.yml`'s authorized call). In the fork modes
|
||||
(`fork-ro`, `fork-rw`), `cache` and `local`, `bazel-doltserver` and
|
||||
`bazel-proxied` run as before: there a single client does both tiers' work
|
||||
without in-flight deduplication, so a union would cost the sum of the
|
||||
tiers rather than the max, on the run's critical path.
|
||||
- Exactly one arm runs in any mode. The union's `job.status` is both the
|
||||
`bazel-doltserver` and the `bazel-proxied` `workflow_call` output (each
|
||||
reads the union's result first, then its member's), so `pr.yml`'s gate
|
||||
ids (`BAZEL_DOLTSERVER`, `BAZEL_PROXIED`), `ci-gate.sh` and
|
||||
`bazel-gate.sh` are unchanged: a failed or cancelled union reds both ids,
|
||||
a union that reported nothing reads as `skipped`, which mode `remote`
|
||||
never allows, and the per-mode skips are the members' as before.
|
||||
- `check_testcases.py` and `check_shard_coverage.py` run once over the
|
||||
union's BEP (the proxied tier's `--suite`; the dolt-server tier has no
|
||||
shard-coverage check). The `bazel test` step has a 20-minute timeout,
|
||||
every step has one, and the job's (55 minutes) is at least their sum plus
|
||||
5 and below the merge queue's 60-minute check timeout.
|
||||
- Its ci-analytics `LANE` is `dolt-race`.
|
||||
- Policy: `scripts/bazel_dolt_race_shadow_test.go`. Removing the job, its
|
||||
`.bazelrc` config and its line in `.github/scripts/rrc-lane-commands.txt`
|
||||
is the rollback; nothing reads them.
|
||||
- Policy: `scripts/bazel_dolt_race_test.go`. Rollback is a revert of the
|
||||
cutover PR: the gate ids never changed and the action keys are equal both
|
||||
ways, so cached results carry back.
|
||||
|
||||
## Remote Repo Contents Cache
|
||||
|
||||
|
|
@ -1032,7 +1038,7 @@ Required` requires them to have run remotely and passed.
|
|||
`nogoConfigurations` table (race, pure, js/wasm) validates in exactly
|
||||
one required lane, not beside every compile of every lane: the race
|
||||
configuration's nogo is owned by `test` above, and `embedded`,
|
||||
`doltserver`, `doltserver-proxied`, `dolt-race` (shadow) and the
|
||||
`doltserver`, `doltserver-proxied`, `dolt-race` and the
|
||||
package gates pass `--norun_validations`, since they compile a strict
|
||||
subset of what `test` already validates on that configuration
|
||||
(`scripts/nogo_lint_policy_test.go`, engdocs/LINTING.md). Since F5 S2
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@ Each lane, as bazel.yml runs it (add your `--config=fork-cache` or
|
|||
| cmd/bd Dolt server (`bazel-cmd-dolt`) | `bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd` | 32 shards of the integration-tagged cmd/bd suite (the slowest tests pinned to 16 of them). |
|
||||
| Embedded Dolt (`bazel-embedded`) | `bazel test //... --config=embedded` | Remote execution only in CI; locally it is slow. Also runs the served http tiers with `BEADS_HTTP_TEST_REQUIRED=1`: `//backend/http:http_served_test`, `//internal/httpclient:httpclient_served_test` and the command corpus `//backend/http/corpus:corpus_served_test`. |
|
||||
| Proxied server (`bazel-proxied`) | `bazel test //... --config=doltserver-proxied` | Remote execution only: 34 shards, each with a Dolt server. |
|
||||
| Dolt server + proxied (`bazel-dolt-race`) | `bazel test //... --config=dolt-race` | Both rows above in one invocation; CI runs this instead of them in mode `remote`. |
|
||||
| Server-Dolt storage (`bazel-server-storage`) | `bazel test //... --config=doltserver-integration` | Remote execution only. |
|
||||
| Docs | `make check-docs` | `//test/docsync:docsync_test` and `//scripts/repochecks:doc_freshness_test` in the test lane's configuration, then `scripts/check-doc-flags.sh` against the pinned release. |
|
||||
|
||||
|
|
@ -199,7 +200,8 @@ remotely with `--config=remote-exec`. No Bazel target uses the `container`
|
|||
backend; run `go test` with docker and the pulled image to exercise it.
|
||||
PR Risk's heavier server tiers have configs of their own, run by bazel.yml
|
||||
only with remote execution, each in a job of its own (`bazel-proxied`,
|
||||
`bazel-server-storage`): `--config=doltserver-proxied` is the
|
||||
which in mode `remote` is folded with `bazel-doltserver` into
|
||||
`bazel-dolt-race`, and `bazel-server-storage`): `--config=doltserver-proxied` is the
|
||||
proxied-server cmd/bd tier ("Test (Proxied Dolt Cmd N/15)",
|
||||
`//cmd/bd:bd_proxied_test`), and `--config=doltserver-integration` the
|
||||
server-Dolt storage tier ("Test (Server Dolt Conformance)", "Test (Server
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ go_test(
|
|||
srcs = [
|
||||
"bazel_autofix_push_test.go",
|
||||
"bazel_cache_mode_test.go",
|
||||
"bazel_dolt_race_shadow_test.go",
|
||||
"bazel_dolt_race_test.go",
|
||||
"bazel_farm_workflow_test.go",
|
||||
"bazel_fork_mode_test.go",
|
||||
"bazel_go_repository_tools_test.go",
|
||||
|
|
|
|||
|
|
@ -227,9 +227,9 @@ func TestBazelCacheModeReachesTheRC(t *testing.T) {
|
|||
// no longer forwards the app secrets either, so there is no
|
||||
// audience left for pre-warming in fork-ro or fork-rw (see
|
||||
// bazel.yml's comment on the job).
|
||||
// Nor does a shadow lane (bazelShadowLanes): mode remote on
|
||||
// same-repo PRs and merge groups only.
|
||||
case strings.HasPrefix(mode, "fork-") && !runs && !bazelPackageJobs[name] && !(name == bazelRBEPrewarmJobName && strings.HasPrefix(mode, "fork-")) && bazelShadowLanes[name] == "":
|
||||
// Nor does a union lane (bazelUnionLanes): mode remote only,
|
||||
// its members run in the fork modes.
|
||||
case strings.HasPrefix(mode, "fork-") && !runs && !bazelPackageJobs[name] && !(name == bazelRBEPrewarmJobName && strings.HasPrefix(mode, "fork-")) && bazelUnionLanes[name] == nil:
|
||||
t.Errorf("%s does not run in mode %s (every lane runs remotely)", name, mode)
|
||||
case mode == "cache" && (name == bazelIntegJobName || name == bazelCmdDoltJobName) && !runs:
|
||||
t.Errorf("%s does not run in mode cache", name)
|
||||
|
|
@ -289,7 +289,9 @@ func TestBazelCacheModeReachesTheRC(t *testing.T) {
|
|||
if bazelPackageJobs[lane] {
|
||||
continue
|
||||
}
|
||||
if !ran[lane] {
|
||||
// A union lane's run reports for its members
|
||||
// (bazel_dolt_race_test.go).
|
||||
if !ran[lane] && !ran[bazelUnionOf(lane)] {
|
||||
wantSkips = append(wantSkips, id)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,305 +0,0 @@
|
|||
package scripts_test
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// The dolt-race shadow (item 5, Phase 1): bazel.yml's bazel-dolt-race job
|
||||
// runs --config=dolt-race, the union of --config=doltserver and
|
||||
// --config=doltserver-proxied, in one `bazel test` beside the two member
|
||||
// lanes, so the union-job model can be measured against them before any
|
||||
// cutover. It must be advisory: no workflow_call output, nothing in pr.yml's
|
||||
// gates or bazel-gate.sh/ci-gate.sh, and a failure that cannot fail the
|
||||
// call's aggregate result (BAZEL) that pr.yml's ci-gate requires. It cannot
|
||||
// fail the gate, but the gate still waits for it: ci-gate needs the whole
|
||||
// bazel call, which completes only when every job in it has finished.
|
||||
|
||||
const (
|
||||
bazelDoltRaceJobName = "bazel-dolt-race"
|
||||
bazelDoltRaceConfig = "dolt-race"
|
||||
// Mode remote only, and only on same-repo pull requests and merge
|
||||
// groups (mode remote on a pull_request event is never a fork: forks
|
||||
// get fork-ro, fork-rw or cache). Not on push, nightly, dispatch or
|
||||
// bazel-farm.yml's pull_request_target call.
|
||||
bazelDoltRaceJobIf = "${{ needs.rbe.outputs.mode == 'remote' && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }}"
|
||||
)
|
||||
|
||||
// bazelShadowLanes: bazel.yml jobs that run beside gated lanes only to
|
||||
// measure a change to them, and report into nothing (no job or
|
||||
// workflow_call output, no gate id, job-level continue-on-error), with the
|
||||
// reason. TestBazelLaneIsGated and the continue-on-error checks carve them
|
||||
// out by this map.
|
||||
var bazelShadowLanes = map[string]string{
|
||||
bazelDoltRaceJobName: "item 5 Phase 1: the dolt-server + proxied union beside its member lanes, advisory until a cutover PR",
|
||||
}
|
||||
|
||||
// bazelDoltRaceJobTimeoutHeadroom: the minutes the shadow job's
|
||||
// timeout-minutes must exceed its test step's (TestBazelDoltRaceShadowIsAdvisory).
|
||||
const bazelDoltRaceJobTimeoutHeadroom = 10
|
||||
|
||||
// bazelDoltRaceJobTimeoutMargin: the minutes the shadow job's timeout-minutes
|
||||
// must exceed the sum of all its step timeouts.
|
||||
const bazelDoltRaceJobTimeoutMargin = 5
|
||||
|
||||
// bazelDoltRaceMergeQueueTimeout: the merge queue's check timeout (ruleset
|
||||
// merge_queue.check_response_timeout_minutes). CI Gate waits for the shadow,
|
||||
// so the shadow's job timeout must stay below it or a hung shadow drops the
|
||||
// group from the queue.
|
||||
const bazelDoltRaceMergeQueueTimeout = 60
|
||||
|
||||
// The union's members, and the tag filter each owns.
|
||||
var bazelDoltRaceMembers = []struct{ config, tag string }{
|
||||
{"doltserver", "dolt-server"},
|
||||
{"doltserver-proxied", "dolt-server-proxied"},
|
||||
}
|
||||
|
||||
// bazelRCConfigLines returns .bazelrc's option lines for --config=name
|
||||
// (any command: test:name, build:name, ...), as "<command> <option>"
|
||||
// with the ":name" suffix stripped, in file order.
|
||||
func bazelRCConfigLines(rc, name string) []string {
|
||||
var out []string
|
||||
for _, line := range strings.Split(rc, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
head, opt, ok := strings.Cut(line, " ")
|
||||
cmd, cfg, hasCfg := strings.Cut(head, ":")
|
||||
if !ok || !hasCfg || cfg != name {
|
||||
continue
|
||||
}
|
||||
out = append(out, cmd+" "+strings.TrimSpace(opt))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceConfigIsMemberUnion: --config=dolt-race is exactly the
|
||||
// set union of its members' lines, except that each member's
|
||||
// --test_tag_filters is replaced by one filter naming both members' tags.
|
||||
// Every flag stays a member's own, so each member target's test action is
|
||||
// unchanged (verified with aquery in the PR that added it), and a flag added
|
||||
// to a member must be added to the union too.
|
||||
func TestBazelDoltRaceConfigIsMemberUnion(t *testing.T) {
|
||||
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
|
||||
const tagPrefix = "test --test_tag_filters="
|
||||
|
||||
var want []string
|
||||
var tags []string
|
||||
for _, m := range bazelDoltRaceMembers {
|
||||
lines := bazelRCConfigLines(rc, m.config)
|
||||
if len(lines) == 0 {
|
||||
t.Fatalf(".bazelrc has no --config=%s lines", m.config)
|
||||
}
|
||||
var memberTags []string
|
||||
for _, l := range lines {
|
||||
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
|
||||
memberTags = append(memberTags, v)
|
||||
continue
|
||||
}
|
||||
if !slices.Contains(want, l) {
|
||||
want = append(want, l)
|
||||
}
|
||||
}
|
||||
if len(memberTags) != 1 || memberTags[0] != m.tag {
|
||||
t.Errorf("--config=%s tag filters %v, want exactly [%s]", m.config, memberTags, m.tag)
|
||||
}
|
||||
tags = append(tags, m.tag)
|
||||
}
|
||||
|
||||
got := bazelRCConfigLines(rc, bazelDoltRaceConfig)
|
||||
var gotTags []string
|
||||
var gotRest []string
|
||||
for _, l := range got {
|
||||
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
|
||||
gotTags = append(gotTags, v)
|
||||
continue
|
||||
}
|
||||
gotRest = append(gotRest, l)
|
||||
}
|
||||
if wantTag := strings.Join(tags, ","); len(gotTags) != 1 || gotTags[0] != wantTag {
|
||||
t.Errorf("--config=%s tag filters %v, want exactly [%s]", bazelDoltRaceConfig, gotTags, wantTag)
|
||||
}
|
||||
if !sameStringSet(gotRest, want) {
|
||||
t.Errorf("--config=%s is not the union of its members' lines (tag filter aside):\ngot:\n%s\nwant (any order):\n%s",
|
||||
bazelDoltRaceConfig, strings.Join(gotRest, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
if len(gotRest) != len(want) {
|
||||
t.Errorf("--config=%s repeats a line: %d lines for %d distinct member lines", bazelDoltRaceConfig, len(gotRest), len(want))
|
||||
}
|
||||
// No nested --config: the union is spelled out, so the policy above
|
||||
// sees every flag.
|
||||
for _, l := range got {
|
||||
if strings.Contains(l, "--config=") {
|
||||
t.Errorf("--config=%s line %q: spell the members' flags out, no nested --config", bazelDoltRaceConfig, l)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func sameStringSet(a, b []string) bool {
|
||||
as := map[string]bool{}
|
||||
for _, s := range a {
|
||||
as[s] = true
|
||||
}
|
||||
bs := map[string]bool{}
|
||||
for _, s := range b {
|
||||
bs[s] = true
|
||||
}
|
||||
if len(as) != len(bs) {
|
||||
return false
|
||||
}
|
||||
for s := range as {
|
||||
if !bs[s] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceShadowIsAdvisory pins what keeps the shadow from failing
|
||||
// any gate: remote-only, job-level continue-on-error (a job's failure
|
||||
// otherwise fails the reusable-workflow call, whose result pr.yml's ci-gate
|
||||
// requires as BAZEL), timeout headroom (a job cancelled at its own timeout
|
||||
// is not covered by continue-on-error), no workflow_call output, no
|
||||
// bazel.yml job needs it, and no gate script or caller names it. ci-gate
|
||||
// still waits for it to finish, through its needs on the whole call.
|
||||
func TestBazelDoltRaceShadowIsAdvisory(t *testing.T) {
|
||||
workflow := readCIWorkflow(t, bazelWorkflowName)
|
||||
job := workflow.job(t, bazelDoltRaceJobName)
|
||||
|
||||
if job.If != bazelDoltRaceJobIf {
|
||||
t.Errorf("%s if: %q, want %q", bazelDoltRaceJobName, job.If, bazelDoltRaceJobIf)
|
||||
}
|
||||
if !job.ContinueOnError {
|
||||
t.Errorf("%s: continue-on-error false; the shadow must never fail the bazel call pr.yml's ci-gate requires", bazelDoltRaceJobName)
|
||||
}
|
||||
if !slices.Equal(job.Needs, []string{bazelRBEJobName}) {
|
||||
t.Errorf("%s needs %v, want [%s]", bazelDoltRaceJobName, job.Needs, bazelRBEJobName)
|
||||
}
|
||||
for name, other := range workflow.Jobs {
|
||||
if slices.Contains(other.Needs, bazelDoltRaceJobName) {
|
||||
t.Errorf("job %s needs %s; no bazel.yml job may depend on the shadow", name, bazelDoltRaceJobName)
|
||||
}
|
||||
}
|
||||
|
||||
// The test step: the union config, as the member lanes run theirs.
|
||||
test := job.step(t, "bazel test //... --config="+bazelDoltRaceConfig)
|
||||
if strings.TrimSpace(test.Run) != bazelTierTestRun(bazelDoltRaceConfig) {
|
||||
t.Errorf("%s test step run changed; want exactly:\n%s\ngot:\n%s", bazelDoltRaceJobName, bazelTierTestRun(bazelDoltRaceConfig), test.Run)
|
||||
}
|
||||
|
||||
// Timeout headroom. continue-on-error covers a failed or timed-out step,
|
||||
// not a job GitHub cancels at its own timeout-minutes; a cancelled job
|
||||
// can make the call (needs.bazel.result) non-success and turn ci-gate
|
||||
// red. So the job outlives its test step by at least
|
||||
// bazelDoltRaceJobTimeoutHeadroom minutes, for setup before it and the
|
||||
// checks, uploads and analytics after it. Both sides come from the
|
||||
// workflow: a step without timeout-minutes would default to 360.
|
||||
if test.TimeoutMinutes <= 0 {
|
||||
t.Errorf("%s test step has no timeout-minutes; it would default to GitHub's 360 and outlive the job", bazelDoltRaceJobName)
|
||||
}
|
||||
if job.TimeoutMinutes < test.TimeoutMinutes+bazelDoltRaceJobTimeoutHeadroom {
|
||||
t.Errorf("%s timeout-minutes %d, want at least the test step's %d + %d: a job cancelled at its own timeout can turn ci-gate red",
|
||||
bazelDoltRaceJobName, job.TimeoutMinutes, test.TimeoutMinutes, bazelDoltRaceJobTimeoutHeadroom)
|
||||
}
|
||||
|
||||
// And no step can run the job into its own timeout: every step has a
|
||||
// timeout-minutes (a step timeout is a step failure, which
|
||||
// continue-on-error covers), and their sum plus
|
||||
// bazelDoltRaceJobTimeoutMargin minutes (runner setup, post-job
|
||||
// cleanup) fits in the job's.
|
||||
sum := 0
|
||||
for i, step := range job.Steps {
|
||||
if step.TimeoutMinutes <= 0 {
|
||||
t.Errorf("%s step %d (%q) has no timeout-minutes; a hang would run into the job timeout, a cancellation continue-on-error does not cover",
|
||||
bazelDoltRaceJobName, i, cmp.Or(step.Name, step.Uses))
|
||||
}
|
||||
sum += step.TimeoutMinutes
|
||||
}
|
||||
if job.TimeoutMinutes < sum+bazelDoltRaceJobTimeoutMargin {
|
||||
t.Errorf("%s timeout-minutes %d, want at least the sum of its step timeouts (%d) + %d",
|
||||
bazelDoltRaceJobName, job.TimeoutMinutes, sum, bazelDoltRaceJobTimeoutMargin)
|
||||
}
|
||||
if job.TimeoutMinutes >= bazelDoltRaceMergeQueueTimeout {
|
||||
t.Errorf("%s timeout-minutes %d, want below the merge queue's %d-minute check timeout: CI Gate waits for the shadow",
|
||||
bazelDoltRaceJobName, job.TimeoutMinutes, bazelDoltRaceMergeQueueTimeout)
|
||||
}
|
||||
|
||||
// No workflow_call output reads the job.
|
||||
var doc struct {
|
||||
On struct {
|
||||
WorkflowCall struct {
|
||||
Outputs map[string]struct {
|
||||
Value string `yaml:"value"`
|
||||
} `yaml:"outputs"`
|
||||
} `yaml:"workflow_call"`
|
||||
} `yaml:"on"`
|
||||
}
|
||||
if err := yaml.Unmarshal([]byte(readPolicyFile(t, sourceRepoRoot(t), ".github/workflows/"+bazelWorkflowName)), &doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(doc.On.WorkflowCall.Outputs) == 0 {
|
||||
t.Fatalf("%s: no workflow_call outputs parsed; the check is broken", bazelWorkflowName)
|
||||
}
|
||||
for name, out := range doc.On.WorkflowCall.Outputs {
|
||||
if strings.Contains(out.Value, "jobs."+bazelDoltRaceJobName+".") || strings.Contains(name, bazelDoltRaceConfig) {
|
||||
t.Errorf("%s workflow_call output %s (%q) reads the shadow; it reports into no gate", bazelWorkflowName, name, out.Value)
|
||||
}
|
||||
}
|
||||
|
||||
// No gate or caller names the job, its config or a gate id for it.
|
||||
root := sourceRepoRoot(t)
|
||||
for _, f := range []string{
|
||||
".github/workflows/pr.yml",
|
||||
".github/workflows/pr-risk.yml",
|
||||
".github/workflows/nightly.yml",
|
||||
".github/workflows/bazel-farm.yml",
|
||||
".github/scripts/bazel-gate.sh",
|
||||
".github/scripts/ci-gate.sh",
|
||||
} {
|
||||
b, err := os.ReadFile(filepath.Join(root, f))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := strings.ToLower(string(b))
|
||||
for _, needle := range []string{"dolt-race", "dolt_race", "doltrace"} {
|
||||
if strings.Contains(s, needle) {
|
||||
t.Errorf("%s mentions %q; the dolt-race shadow must stay out of every gate", f, needle)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls evaluates the job's if:
|
||||
// for every execution mode and every event bazel.yml runs on. It runs only
|
||||
// in mode remote on a pull_request or merge_group event: never in a fork
|
||||
// mode (fork-ro, fork-rw), cache or local, never on bazel-farm.yml's
|
||||
// pull_request_target call (fork code with trusted credentials), and not on
|
||||
// push, nightly (schedule) or dispatch, so it never doubles nightly's fresh
|
||||
// execution of the two tiers.
|
||||
func TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls(t *testing.T) {
|
||||
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltRaceJobName)
|
||||
for _, mode := range []string{"remote", "fork-ro", "fork-rw", "cache", "local", "skip"} {
|
||||
for _, event := range []string{"pull_request", "merge_group", "pull_request_target", "push", "schedule", "workflow_dispatch"} {
|
||||
want := mode == "remote" && (event == "pull_request" || event == "merge_group")
|
||||
got := evalRRCIf(t, job.If, map[string]string{
|
||||
"needs.rbe.outputs.mode": mode,
|
||||
"github.event_name": event,
|
||||
})
|
||||
if got != want {
|
||||
t.Errorf("%s runs in mode %s on %s: %v, want %v", bazelDoltRaceJobName, mode, event, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
// The runner is the lanes' ternary, which picks Blacksmith only in mode
|
||||
// remote: the only mode the job runs in.
|
||||
if job.RunsOn != bazelLaneRunsOn {
|
||||
t.Errorf("%s runs-on = %q, want %q", bazelDoltRaceJobName, job.RunsOn, bazelLaneRunsOn)
|
||||
}
|
||||
}
|
||||
508
scripts/bazel_dolt_race_test.go
Normal file
508
scripts/bazel_dolt_race_test.go
Normal file
|
|
@ -0,0 +1,508 @@
|
|||
package scripts_test
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
"maps"
|
||||
"path"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The dolt-race union lane (item 5): in mode remote, bazel.yml's
|
||||
// bazel-dolt-race job runs --config=dolt-race, the union of
|
||||
// --config=doltserver and --config=doltserver-proxied, in one `bazel test`,
|
||||
// in place of the bazel-doltserver and bazel-proxied jobs; in every other
|
||||
// mode those two run as before. Its job.status is both members' workflow_call
|
||||
// output, so pr.yml's gate ids (BAZEL_DOLTSERVER, BAZEL_PROXIED), ci-gate.sh
|
||||
// and bazel-gate.sh are unchanged. Outside mode remote one client would do
|
||||
// both tiers' work with no in-flight deduplication, so the union would cost
|
||||
// the sum of the two tiers rather than the max: the members stay there.
|
||||
|
||||
const (
|
||||
bazelDoltRaceJobName = "bazel-dolt-race"
|
||||
bazelDoltRaceConfig = "dolt-race"
|
||||
// Mode remote, on every event that reaches it (same-repo pull requests,
|
||||
// merge groups, push, nightly, dispatch, bazel-farm.yml's authorized
|
||||
// call): exactly where the members do not run.
|
||||
bazelDoltRaceJobIf = "${{ needs.rbe.outputs.mode == 'remote' }}"
|
||||
// The members: their pre-union rules (bazel-test's skip rule; remote
|
||||
// execution only), minus mode remote.
|
||||
bazelDoltMemberIf = "${{ needs.rbe.outputs.mode != 'skip' && needs.rbe.outputs.mode != 'remote' }}"
|
||||
bazelProxiedMemberIf = "${{ needs.rbe.outputs.enabled == 'true' && needs.rbe.outputs.mode != 'remote' }}"
|
||||
)
|
||||
|
||||
// bazelUnionLanes: a union job -> the gated member jobs it runs in place of
|
||||
// in mode remote (bazelDoltRaceJobIf). Each member keeps its gate id
|
||||
// (bazelLaneGateIDs) and its workflow_call output, which reads the union's
|
||||
// result first and the member's own otherwise (bazelLaneCallOutput).
|
||||
var bazelUnionLanes = map[string][]string{
|
||||
bazelDoltRaceJobName: {bazelDoltJobName, bazelProxiedJobName},
|
||||
}
|
||||
|
||||
// bazelUnionOf: the union job a gated member belongs to, or "".
|
||||
func bazelUnionOf(member string) string {
|
||||
for union, members := range bazelUnionLanes {
|
||||
if slices.Contains(members, member) {
|
||||
return union
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// bazelLaneCallOutput: the workflow_call output value bazel.yml must declare
|
||||
// for a lane that reports its job.status.
|
||||
func bazelLaneCallOutput(lane string) string {
|
||||
if union := bazelUnionOf(lane); union != "" {
|
||||
return "${{ jobs." + union + ".outputs.result || jobs." + lane + ".outputs.result }}"
|
||||
}
|
||||
return "${{ jobs." + lane + ".outputs.result }}"
|
||||
}
|
||||
|
||||
// bazelLaneReporter: the job whose result a gated lane's output carries in
|
||||
// this mode (lanes: job -> modes it runs in), or "" when neither the lane
|
||||
// nor its union runs (an expected skip).
|
||||
func bazelLaneReporter(lanes map[string]map[string]bool, lane, mode string) string {
|
||||
if lanes[lane][mode] {
|
||||
return lane
|
||||
}
|
||||
if union := bazelUnionOf(lane); union != "" && lanes[union][mode] {
|
||||
return union
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
var (
|
||||
bazelCallOutputsMu sync.Mutex
|
||||
bazelCallOutputsCache map[string]string
|
||||
)
|
||||
|
||||
// bazelCallOutputValue evaluates bazel.yml's workflow_call output name with
|
||||
// jobs (job -> its result output; missing or "" = not reported), the way
|
||||
// GitHub does: what pr.yml's needs.bazel.outputs.<name> reads.
|
||||
func bazelCallOutputValue(t *testing.T, name string, jobs map[string]string) string {
|
||||
t.Helper()
|
||||
bazelCallOutputsMu.Lock()
|
||||
if bazelCallOutputsCache == nil {
|
||||
bazelCallOutputsCache = map[string]string{}
|
||||
for n, out := range readBazelWorkflowCall(t).Outputs {
|
||||
bazelCallOutputsCache[n] = out.Value
|
||||
}
|
||||
}
|
||||
expr, ok := bazelCallOutputsCache[name]
|
||||
bazelCallOutputsMu.Unlock()
|
||||
if !ok {
|
||||
t.Fatalf("%s has no workflow_call output %s", bazelWorkflowName, name)
|
||||
}
|
||||
ctx := map[string]string{}
|
||||
for _, m := range regexp.MustCompile(`jobs\.([A-Za-z0-9_-]+)\.outputs\.([A-Za-z0-9_-]+)`).FindAllStringSubmatch(expr, -1) {
|
||||
v := ""
|
||||
if m[2] == "result" {
|
||||
v = jobs[m[1]]
|
||||
}
|
||||
ctx[m[0]] = v
|
||||
}
|
||||
v, err := evalGHExpr(expr, ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("workflow_call output %s = %q: %v", name, expr, err)
|
||||
}
|
||||
str, _ := v.(string)
|
||||
return str
|
||||
}
|
||||
|
||||
// bazelDoltRaceJobTimeoutMargin: the minutes the union job's timeout-minutes
|
||||
// must exceed the sum of all its step timeouts (runner setup, post-job
|
||||
// cleanup), so a hang fails a named step, not the job.
|
||||
const bazelDoltRaceJobTimeoutMargin = 5
|
||||
|
||||
// bazelDoltRaceMergeQueueTimeout: the merge queue's check timeout (ruleset
|
||||
// merge_queue.check_response_timeout_minutes). The union job is required in
|
||||
// every merge group, so its own timeout stays below it.
|
||||
const bazelDoltRaceMergeQueueTimeout = 60
|
||||
|
||||
// bazelDoltRaceTestStepTimeout: the union's `bazel test` step timeout. Its
|
||||
// shards run about 55-95 s on rbe-west, each capped at 1200 s; the step allows
|
||||
// about 3 minutes past that cap so Bazel writes TIMEOUT logs before GitHub
|
||||
// kills the step.
|
||||
const bazelDoltRaceTestStepTimeout = 23
|
||||
|
||||
// The union's members, and the tag filter each owns.
|
||||
var bazelDoltRaceMembers = []struct{ config, tag string }{
|
||||
{"doltserver", "dolt-server"},
|
||||
{"doltserver-proxied", "dolt-server-proxied"},
|
||||
}
|
||||
|
||||
// bazelRCConfigLines returns .bazelrc's option lines for --config=name
|
||||
// (any command: test:name, build:name, ...), as "<command> <option>"
|
||||
// with the ":name" suffix stripped, in file order.
|
||||
func bazelRCConfigLines(rc, name string) []string {
|
||||
var out []string
|
||||
for _, line := range strings.Split(rc, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
head, opt, ok := strings.Cut(line, " ")
|
||||
cmd, cfg, hasCfg := strings.Cut(head, ":")
|
||||
if !ok || !hasCfg || cfg != name {
|
||||
continue
|
||||
}
|
||||
out = append(out, cmd+" "+strings.TrimSpace(opt))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceConfigIsMemberUnion: --config=dolt-race is exactly the
|
||||
// set union of its members' lines, except that each member's
|
||||
// --test_tag_filters is replaced by one filter naming both members' tags.
|
||||
// Every flag stays a member's own, so each member target's test action is
|
||||
// unchanged: `bazel aquery 'mnemonic("TestRunner", ...)'` over each member's
|
||||
// targets gives byte-identical actions (key, argv, env, outputs) under the
|
||||
// member config and under the union, 27 + 35 = 62 (item 5 A1b). Both arms
|
||||
// run, the mode deciding which, so drift between them would split results
|
||||
// by mode: a flag added to a member must be added to the union too.
|
||||
func TestBazelDoltRaceConfigIsMemberUnion(t *testing.T) {
|
||||
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
|
||||
const tagPrefix = "test --test_tag_filters="
|
||||
|
||||
var want []string
|
||||
var tags []string
|
||||
for _, m := range bazelDoltRaceMembers {
|
||||
lines := bazelRCConfigLines(rc, m.config)
|
||||
if len(lines) == 0 {
|
||||
t.Fatalf(".bazelrc has no --config=%s lines", m.config)
|
||||
}
|
||||
var memberTags []string
|
||||
for _, l := range lines {
|
||||
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
|
||||
memberTags = append(memberTags, v)
|
||||
continue
|
||||
}
|
||||
if !slices.Contains(want, l) {
|
||||
want = append(want, l)
|
||||
}
|
||||
}
|
||||
if len(memberTags) != 1 || memberTags[0] != m.tag {
|
||||
t.Errorf("--config=%s tag filters %v, want exactly [%s]", m.config, memberTags, m.tag)
|
||||
}
|
||||
tags = append(tags, m.tag)
|
||||
}
|
||||
|
||||
got := bazelRCConfigLines(rc, bazelDoltRaceConfig)
|
||||
var gotTags []string
|
||||
var gotRest []string
|
||||
for _, l := range got {
|
||||
if v, ok := strings.CutPrefix(l, tagPrefix); ok {
|
||||
gotTags = append(gotTags, v)
|
||||
continue
|
||||
}
|
||||
gotRest = append(gotRest, l)
|
||||
}
|
||||
if wantTag := strings.Join(tags, ","); len(gotTags) != 1 || gotTags[0] != wantTag {
|
||||
t.Errorf("--config=%s tag filters %v, want exactly [%s]", bazelDoltRaceConfig, gotTags, wantTag)
|
||||
}
|
||||
if !sameStringSet(gotRest, want) {
|
||||
t.Errorf("--config=%s is not the union of its members' lines (tag filter aside):\ngot:\n%s\nwant (any order):\n%s",
|
||||
bazelDoltRaceConfig, strings.Join(gotRest, "\n"), strings.Join(want, "\n"))
|
||||
}
|
||||
if len(gotRest) != len(want) {
|
||||
t.Errorf("--config=%s repeats a line: %d lines for %d distinct member lines", bazelDoltRaceConfig, len(gotRest), len(want))
|
||||
}
|
||||
// No nested --config: the union is spelled out, so the policy above
|
||||
// sees every flag.
|
||||
for _, l := range got {
|
||||
if strings.Contains(l, "--config=") {
|
||||
t.Errorf("--config=%s line %q: spell the members' flags out, no nested --config", bazelDoltRaceConfig, l)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func sameStringSet(a, b []string) bool {
|
||||
as := map[string]bool{}
|
||||
for _, s := range a {
|
||||
as[s] = true
|
||||
}
|
||||
bs := map[string]bool{}
|
||||
for _, s := range b {
|
||||
bs[s] = true
|
||||
}
|
||||
if len(as) != len(bs) {
|
||||
return false
|
||||
}
|
||||
for s := range as {
|
||||
if !bs[s] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceSoleRunIsInTheUnion: where bazel-doltserver executes
|
||||
// remotely (the fork modes) it adds --config=sole-run; the union carries that
|
||||
// config's lines itself, so the dolt-server tier's actions are the same in
|
||||
// either arm. Every sole-run line is a dolt-race line.
|
||||
func TestBazelDoltRaceSoleRunIsInTheUnion(t *testing.T) {
|
||||
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
|
||||
sole := bazelRCConfigLines(rc, "sole-run")
|
||||
if len(sole) == 0 {
|
||||
t.Fatal(".bazelrc has no --config=sole-run lines")
|
||||
}
|
||||
union := bazelRCConfigLines(rc, bazelDoltRaceConfig)
|
||||
for _, l := range sole {
|
||||
if !slices.Contains(union, l) {
|
||||
t.Errorf("--config=sole-run line %q is not in --config=%s: bazel-doltserver's remote runs and the union would differ", l, bazelDoltRaceConfig)
|
||||
}
|
||||
}
|
||||
// And the union job takes no sole-run of its own: nothing to add.
|
||||
test := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltRaceJobName).step(t, "bazel test //... --config="+bazelDoltRaceConfig)
|
||||
if strings.Contains(test.Run, "BAZEL_SOLE_RUN") || strings.Contains(test.Run, "sole-run") {
|
||||
t.Errorf("%s test step adds sole-run; --config=%s already has its lines", bazelDoltRaceJobName, bazelDoltRaceConfig)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceIsGated pins the cutover: the union job is a required
|
||||
// lane like its members (no continue-on-error, the always() job.status
|
||||
// recorder), it runs the union config with the members' checks once over its
|
||||
// BEP, and both members' workflow_call outputs read its result first.
|
||||
func TestBazelDoltRaceIsGated(t *testing.T) {
|
||||
workflow := readCIWorkflow(t, bazelWorkflowName)
|
||||
job := workflow.job(t, bazelDoltRaceJobName)
|
||||
|
||||
if job.If != bazelDoltRaceJobIf {
|
||||
t.Errorf("%s if: %q, want %q", bazelDoltRaceJobName, job.If, bazelDoltRaceJobIf)
|
||||
}
|
||||
if job.ContinueOnError {
|
||||
t.Errorf("%s: continue-on-error hides the dolt-server and proxied tiers' failures from pr.yml's ci-gate", bazelDoltRaceJobName)
|
||||
}
|
||||
if job.Name != "dolt-race" {
|
||||
// ci-analytics' collector maps the check name "Bazel / dolt-race"
|
||||
// to lane dolt-race.
|
||||
t.Errorf("%s name = %q, want dolt-race", bazelDoltRaceJobName, job.Name)
|
||||
}
|
||||
if !slices.Equal(job.Needs, []string{bazelRBEJobName}) {
|
||||
t.Errorf("%s needs %v, want [%s]", bazelDoltRaceJobName, job.Needs, bazelRBEJobName)
|
||||
}
|
||||
if !reflect.DeepEqual(job.Outputs, map[string]string{"result": "${{ steps.result.outputs.result }}"}) {
|
||||
t.Errorf("%s outputs = %v, want only result from the result step", bazelDoltRaceJobName, job.Outputs)
|
||||
}
|
||||
|
||||
// The test step: the union config, as the member lanes run theirs.
|
||||
test := job.step(t, "bazel test //... --config="+bazelDoltRaceConfig)
|
||||
if strings.TrimSpace(test.Run) != bazelTierTestRun(bazelDoltRaceConfig) {
|
||||
t.Errorf("%s test step run changed; want exactly:\n%s\ngot:\n%s", bazelDoltRaceJobName, bazelTierTestRun(bazelDoltRaceConfig), test.Run)
|
||||
}
|
||||
// The members' per-lane checks, once over the union's BEP: bazel-proxied's
|
||||
// two (bazel-doltserver has none). check_shard_coverage takes the union
|
||||
// of the members' --suite/--whole arguments, which is bazel-proxied's.
|
||||
for member, steps := range map[string][]string{
|
||||
bazelProxiedJobName: {"Every target and shard ran tests", "Every listed test ran in its shard", "No test target is marked flaky"},
|
||||
} {
|
||||
m := workflow.job(t, member)
|
||||
for _, name := range steps {
|
||||
want, got := m.step(t, name), job.step(t, name)
|
||||
if strings.TrimSpace(got.Run) != strings.TrimSpace(want.Run) || got.If != want.If {
|
||||
t.Errorf("%s step %q = (if %q) %q, want %s's (if %q) %q", bazelDoltRaceJobName, name, got.If, got.Run, member, want.If, want.Run)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, member := range bazelUnionLanes[bazelDoltRaceJobName] {
|
||||
for _, step := range workflow.job(t, member).Steps {
|
||||
if strings.Contains(step.Run, "shard_budget.py") || strings.Contains(step.Run, "check_shard_coverage.py") && member != bazelProxiedJobName {
|
||||
t.Errorf("%s step %q: a per-lane check the union does not run", member, step.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// No job depends on it, and every member's output reads it first.
|
||||
for name, other := range workflow.Jobs {
|
||||
if slices.Contains(other.Needs, bazelDoltRaceJobName) {
|
||||
t.Errorf("job %s needs %s", name, bazelDoltRaceJobName)
|
||||
}
|
||||
}
|
||||
call := readBazelWorkflowCall(t)
|
||||
for _, member := range bazelUnionLanes[bazelDoltRaceJobName] {
|
||||
if got, want := call.Outputs[member].Value, bazelLaneCallOutput(member); got != want {
|
||||
t.Errorf("workflow_call output %s = %q, want %q", member, got, want)
|
||||
}
|
||||
}
|
||||
for name := range call.Outputs {
|
||||
if strings.Contains(name, bazelDoltRaceConfig) {
|
||||
t.Errorf("workflow_call output %s: the union reports through its members' outputs, not one of its own (the gate ids stay)", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceJobTimeouts: the union's test step has a 23-minute
|
||||
// timeout, every step has one, and the job's exceeds their sum by
|
||||
// bazelDoltRaceJobTimeoutMargin minutes (a hang fails a named step) while
|
||||
// staying under the merge queue's check timeout (a hung required job would
|
||||
// otherwise drop the group).
|
||||
func TestBazelDoltRaceJobTimeouts(t *testing.T) {
|
||||
job := readCIWorkflow(t, bazelWorkflowName).job(t, bazelDoltRaceJobName)
|
||||
test := job.step(t, "bazel test //... --config="+bazelDoltRaceConfig)
|
||||
if test.TimeoutMinutes != bazelDoltRaceTestStepTimeout {
|
||||
t.Errorf("%s test step timeout-minutes = %d, want %d", bazelDoltRaceJobName, test.TimeoutMinutes, bazelDoltRaceTestStepTimeout)
|
||||
}
|
||||
sum := 0
|
||||
for i, step := range job.Steps {
|
||||
if step.TimeoutMinutes <= 0 {
|
||||
t.Errorf("%s step %d (%q) has no timeout-minutes; a hang would run into the job timeout",
|
||||
bazelDoltRaceJobName, i, cmp.Or(step.Name, step.Uses))
|
||||
}
|
||||
sum += step.TimeoutMinutes
|
||||
}
|
||||
if job.TimeoutMinutes < sum+bazelDoltRaceJobTimeoutMargin {
|
||||
t.Errorf("%s timeout-minutes %d, want at least the sum of its step timeouts (%d) + %d",
|
||||
bazelDoltRaceJobName, job.TimeoutMinutes, sum, bazelDoltRaceJobTimeoutMargin)
|
||||
}
|
||||
if job.TimeoutMinutes >= bazelDoltRaceMergeQueueTimeout {
|
||||
t.Errorf("%s timeout-minutes %d, want below the merge queue's %d-minute check timeout",
|
||||
bazelDoltRaceJobName, job.TimeoutMinutes, bazelDoltRaceMergeQueueTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBazelDoltRaceArmsExactlyOnePerMode evaluates the union's and the
|
||||
// members' if: for every execution mode and every event bazel.yml runs on:
|
||||
// the union runs exactly in mode remote, never beside a member, and each
|
||||
// member's gate id is reported by exactly one job wherever that member ran
|
||||
// before the union (bazel-doltserver: every mode but skip; bazel-proxied:
|
||||
// the remote-executing modes) and by none elsewhere, so bazel-gate.sh's
|
||||
// per-mode skips are unchanged. The workflow_call outputs, evaluated as
|
||||
// GitHub does, carry the running arm's result: a failed or cancelled union
|
||||
// reds both ids, and a union that never reported reads as skipped, which no
|
||||
// mode it runs in allows.
|
||||
func TestBazelDoltRaceArmsExactlyOnePerMode(t *testing.T) {
|
||||
workflow := readCIWorkflow(t, bazelWorkflowName)
|
||||
union := workflow.job(t, bazelDoltRaceJobName)
|
||||
members := map[string]struct {
|
||||
ifExpr string
|
||||
before func(mode string) bool // where the member ran before the union
|
||||
}{
|
||||
bazelDoltJobName: {bazelDoltMemberIf, func(m string) bool { return m != "skip" }},
|
||||
bazelProxiedJobName: {bazelProxiedMemberIf, func(m string) bool { return bazelRemoteModes[m] }},
|
||||
}
|
||||
if got := slices.Sorted(maps.Keys(members)); !slices.Equal(got, slices.Sorted(slices.Values(bazelUnionLanes[bazelDoltRaceJobName]))) {
|
||||
t.Fatalf("members %v, want bazelUnionLanes' %v", got, bazelUnionLanes[bazelDoltRaceJobName])
|
||||
}
|
||||
for name, m := range members {
|
||||
if got := workflow.job(t, name).If; got != m.ifExpr {
|
||||
t.Errorf("%s if = %q, want %q", name, got, m.ifExpr)
|
||||
}
|
||||
}
|
||||
for _, mode := range bazelRBEModes {
|
||||
for _, event := range []string{"pull_request", "merge_group", "pull_request_target", "push", "schedule", "workflow_dispatch"} {
|
||||
ctx := map[string]string{
|
||||
"needs.rbe.outputs.mode": mode,
|
||||
"needs.rbe.outputs.enabled": bazelModeEnabled(mode),
|
||||
"github.event_name": event,
|
||||
}
|
||||
unionRuns := evalRRCIf(t, union.If, ctx)
|
||||
if unionRuns != (mode == "remote") {
|
||||
t.Errorf("mode %s, %s: %s runs = %v, want %v", mode, event, bazelDoltRaceJobName, unionRuns, mode == "remote")
|
||||
}
|
||||
for name, m := range members {
|
||||
runs := evalRRCIf(t, workflow.job(t, name).If, ctx)
|
||||
if unionRuns && runs {
|
||||
t.Errorf("mode %s, %s: %s and %s both run", mode, event, bazelDoltRaceJobName, name)
|
||||
}
|
||||
if got, want := unionRuns || runs, m.before(mode); got != want {
|
||||
t.Errorf("mode %s, %s: %s's tier reported by a job = %v, want %v (where it ran before the union)", mode, event, name, got, want)
|
||||
}
|
||||
for _, res := range []string{"success", "failure", "cancelled", ""} {
|
||||
jobs := map[string]string{}
|
||||
switch {
|
||||
case unionRuns:
|
||||
jobs[bazelDoltRaceJobName] = res
|
||||
case runs:
|
||||
jobs[name] = res
|
||||
}
|
||||
want := res
|
||||
if !unionRuns && !runs {
|
||||
want = ""
|
||||
}
|
||||
if got := bazelCallOutputValue(t, name, jobs); got != want {
|
||||
t.Errorf("mode %s, %s, reporting job %q: output %s = %q, want %q", mode, event, res, name, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// The union's runner is the lanes' ternary: Blacksmith in mode remote,
|
||||
// the only mode it runs in.
|
||||
if union.RunsOn != bazelLaneRunsOn {
|
||||
t.Errorf("%s runs-on = %q, want %q", bazelDoltRaceJobName, union.RunsOn, bazelLaneRunsOn)
|
||||
}
|
||||
}
|
||||
|
||||
// bazelDoltRaceTestActions: the union's test actions, 27 dolt-server + 35
|
||||
// proxied-server (aquery, item 5 A1b), each a target's shard.
|
||||
const bazelDoltRaceTestActions = 62
|
||||
|
||||
// TestBazelDoltRaceFitsTheJobsBudget: the union runs both tiers' shards in
|
||||
// one invocation under --config=remote-exec's --jobs, so all of them must
|
||||
// fit at once or the slower tier's shards queue behind the other's and the
|
||||
// union stops ending with the slower member. Every rule tagged with one of
|
||||
// the union's tags, read from the BUILD files: its shard_count (1 when
|
||||
// unsharded), summed. A tag outside a one-line `tags = [...]` list cannot
|
||||
// be counted and fails the test.
|
||||
func TestBazelDoltRaceFitsTheJobsBudget(t *testing.T) {
|
||||
root := bazelPolicyRoot(t)
|
||||
rc := readPolicyFile(t, root, ".bazelrc")
|
||||
m := regexp.MustCompile(`(?m)^build:remote-exec --jobs=(\d+)$`).FindStringSubmatch(rc)
|
||||
if m == nil {
|
||||
t.Fatal(".bazelrc has no build:remote-exec --jobs=N line")
|
||||
}
|
||||
budget, _ := strconv.Atoi(m[1])
|
||||
|
||||
var tags []string
|
||||
for _, mem := range bazelDoltRaceMembers {
|
||||
tags = append(tags, `"`+mem.tag+`"`)
|
||||
}
|
||||
ruleStart := regexp.MustCompile(`(?m)^[A-Za-z_][A-Za-z0-9_]*\($`)
|
||||
total, targets := 0, 0
|
||||
for _, f := range repoFiles(t, root, 300) {
|
||||
if path.Base(f) != "BUILD.bazel" {
|
||||
continue
|
||||
}
|
||||
build := readPolicyFile(t, root, f)
|
||||
for _, line := range strings.Split(build, "\n") {
|
||||
for _, tag := range tags {
|
||||
if strings.Contains(line, tag) && !strings.HasPrefix(line, " tags = [") {
|
||||
t.Errorf("%s: %q outside a one-line tags list: %q", f, tag, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, loc := range ruleStart.FindAllStringIndex(build, -1) {
|
||||
end := strings.Index(build[loc[0]:], "\n)\n")
|
||||
if end < 0 {
|
||||
continue
|
||||
}
|
||||
rule := build[loc[0] : loc[0]+end]
|
||||
tagged := false
|
||||
for _, line := range strings.Split(rule, "\n") {
|
||||
if strings.HasPrefix(line, " tags = [") && slices.ContainsFunc(tags, func(tag string) bool { return strings.Contains(line, tag) }) {
|
||||
tagged = true
|
||||
}
|
||||
}
|
||||
if !tagged {
|
||||
continue
|
||||
}
|
||||
shards := 1
|
||||
if sm := shardCountPattern.FindStringSubmatch(rule); sm != nil {
|
||||
shards, _ = strconv.Atoi(sm[1])
|
||||
}
|
||||
total += shards
|
||||
targets++
|
||||
}
|
||||
}
|
||||
if total != bazelDoltRaceTestActions {
|
||||
t.Errorf("--config=%s: %d test actions over %d targets, want %d (aquery-verified at the cutover): re-check against --jobs=%d and update", bazelDoltRaceConfig, total, targets, bazelDoltRaceTestActions, budget)
|
||||
}
|
||||
if total > budget {
|
||||
t.Errorf("--config=%s: %d test actions, more than --config=remote-exec's --jobs=%d: the tiers would no longer run at once", bazelDoltRaceConfig, total, budget)
|
||||
}
|
||||
}
|
||||
|
|
@ -22,7 +22,7 @@ var ciAnalyticsLaneJobs = map[string]string{
|
|||
"bazel-proxied": "proxied-server",
|
||||
"bazel-server-storage": "server-storage",
|
||||
"bazel-cmd-dolt": "cmd-dolt",
|
||||
// The advisory dolt-server + proxied union shadow (item 5 Phase 1): its
|
||||
// The dolt-server + proxied union lane (item 5; mode remote): its
|
||||
// own lane value, so the A/B queries can pair it with its members.
|
||||
"bazel-dolt-race": "dolt-race",
|
||||
}
|
||||
|
|
|
|||
|
|
@ -75,9 +75,9 @@ func bazelRRCCacheReaderRunsOn(label string) string {
|
|||
|
||||
// isBazelRRCCacheReader: a remote repo contents cache reader
|
||||
// (bazelRRCReadLanes) that also runs in mode cache (not remote-only).
|
||||
// bazel-dolt-race runs only in mode remote on PRs and merge groups (its own
|
||||
// if, TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls), so it keeps the
|
||||
// lanes' ubuntu-latest fallback like the remote-only jobs.
|
||||
// bazel-dolt-race runs only in mode remote (its own if,
|
||||
// TestBazelDoltRaceArmsExactlyOnePerMode), so it keeps the lanes'
|
||||
// ubuntu-latest fallback like the remote-only jobs.
|
||||
func isBazelRRCCacheReader(name string) bool {
|
||||
return slices.Contains(bazelRRCReadLanes, name) && !bazelRemoteOnlyJobs[name] && name != bazelDoltRaceJobName
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1881,15 +1881,9 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) {
|
|||
// bazel.yml's own run - and thus pr.yml's ci-gate - red for a job
|
||||
// nothing requires.
|
||||
_, flagGated := bazelFlagGatedLanes[name]
|
||||
// So is each shadow lane (bazelShadowLanes): it reports into no
|
||||
// gate, so its failure must not fail the call (BAZEL) either.
|
||||
_, shadow := bazelShadowLanes[name]
|
||||
if job.ContinueOnError && name != bazelRBEPrewarmJobName && !flagGated && !shadow {
|
||||
if job.ContinueOnError && name != bazelRBEPrewarmJobName && !flagGated {
|
||||
t.Errorf("%s continue-on-error hides failures from pr.yml's ci-gate", name)
|
||||
}
|
||||
if shadow && !job.ContinueOnError {
|
||||
t.Errorf("%s is a shadow lane (%s) but lacks job-level continue-on-error", name, bazelShadowLanes[name])
|
||||
}
|
||||
// A flag-gated lane is advisory until its flag says otherwise, and
|
||||
// an advisory job's failure must not fail the call (BAZEL).
|
||||
if flagGated && !job.ContinueOnError {
|
||||
|
|
@ -1994,11 +1988,19 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) {
|
|||
// below only fires for a setup-bazel step).
|
||||
wantJobRunsOn, wantJobIf = bazelPrewarmRunsOn, bazelRBEPrewarmIf
|
||||
} else if name == bazelDoltRaceJobName {
|
||||
// Mode remote, on pull_request and merge_group events only
|
||||
// (bazel_dolt_race_shadow_test.go); the lanes' runner and
|
||||
// setup-bazel env otherwise.
|
||||
// Mode remote only, in place of its members
|
||||
// (bazel_dolt_race_test.go); the lanes' runner and setup-bazel
|
||||
// env otherwise.
|
||||
wantJobIf = bazelDoltRaceJobIf
|
||||
}
|
||||
// The union's members: their own rule, minus mode remote
|
||||
// (TestBazelDoltRaceArmsExactlyOnePerMode).
|
||||
switch name {
|
||||
case bazelDoltJobName:
|
||||
wantJobIf = bazelDoltMemberIf
|
||||
case bazelProxiedJobName:
|
||||
wantJobIf = bazelProxiedMemberIf
|
||||
}
|
||||
if job.RunsOn != wantJobRunsOn {
|
||||
t.Errorf("%s runs-on = %q, want %q", name, job.RunsOn, wantJobRunsOn)
|
||||
}
|
||||
|
|
@ -2543,21 +2545,15 @@ func TestBazelLaneIsGated(t *testing.T) {
|
|||
if name == bazelRBEJobName || isBazelRRCJob(name) {
|
||||
continue // the rrc jobs never run in a PR call (bazel_rrc_test.go)
|
||||
}
|
||||
if _, shadow := bazelShadowLanes[name]; shadow {
|
||||
// A shadow lane reports into nothing: no job outputs, so no
|
||||
// workflow_call output and no gate id
|
||||
// (TestBazelDoltRaceShadowIsAdvisory).
|
||||
if len(job.Outputs) != 0 {
|
||||
t.Errorf("shadow lane %s outputs = %v, want none", name, job.Outputs)
|
||||
}
|
||||
continue
|
||||
}
|
||||
_, gated := bazelLaneGateIDs[name]
|
||||
_, advisory := bazelAdvisoryLanes[name]
|
||||
_, flagGated := bazelFlagGatedLanes[name]
|
||||
if n := btoi(gated) + btoi(advisory) + btoi(flagGated); n != 1 {
|
||||
t.Errorf("%s job %s: gated=%v advisory=%v flag-gated=%v; add it to exactly one of bazelLaneGateIDs (with a CI_GATE_REQUIRED id in pr.yml), bazelFlagGatedLanes (with its pr.yml flag) or bazelAdvisoryLanes (with the reason)",
|
||||
bazelWorkflowName, name, gated, advisory, flagGated)
|
||||
// A union lane reports through its gated members' ids and outputs
|
||||
// (bazel_dolt_race_test.go).
|
||||
_, union := bazelUnionLanes[name]
|
||||
if n := btoi(gated) + btoi(advisory) + btoi(flagGated) + btoi(union); n != 1 {
|
||||
t.Errorf("%s job %s: gated=%v advisory=%v flag-gated=%v union=%v; add it to exactly one of bazelLaneGateIDs (with a CI_GATE_REQUIRED id in pr.yml), bazelFlagGatedLanes (with its pr.yml flag), bazelUnionLanes (with the gated members it runs in place of) or bazelAdvisoryLanes (with the reason)",
|
||||
bazelWorkflowName, name, gated, advisory, flagGated, union)
|
||||
}
|
||||
// Each lane reports its job.status from an always() last step.
|
||||
if !reflect.DeepEqual(job.Outputs, map[string]string{"result": "${{ steps.result.outputs.result }}"}) {
|
||||
|
|
@ -2568,7 +2564,19 @@ func TestBazelLaneIsGated(t *testing.T) {
|
|||
last.Run != `echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"` {
|
||||
t.Errorf("%s last step = %+v; want the always() job.status recorder", name, last)
|
||||
}
|
||||
wantCallOutputs[name] = "${{ jobs." + name + ".outputs.result }}"
|
||||
if !union {
|
||||
wantCallOutputs[name] = bazelLaneCallOutput(name)
|
||||
}
|
||||
}
|
||||
for union, members := range bazelUnionLanes {
|
||||
if _, ok := workflow.Jobs[union]; !ok {
|
||||
t.Errorf("bazelUnionLanes lists %s, which %s does not have", union, bazelWorkflowName)
|
||||
}
|
||||
for _, m := range members {
|
||||
if _, gated := bazelLaneGateIDs[m]; !gated {
|
||||
t.Errorf("union lane %s member %s is not a gated lane", union, m)
|
||||
}
|
||||
}
|
||||
}
|
||||
for name := range bazelLaneGateIDs {
|
||||
if _, ok := workflow.Jobs[name]; !ok {
|
||||
|
|
@ -2858,6 +2866,12 @@ func bazelLaneRunModes(t *testing.T, lane, ifExpr string, with map[string]string
|
|||
return map[string]bool{"remote": true, "fork-ro": true, "fork-rw": true, "cache": true, "local": true}
|
||||
case "${{ needs.rbe.outputs.enabled == 'true' }}":
|
||||
return map[string]bool{"remote": true, "fork-ro": true, "fork-rw": true}
|
||||
case bazelDoltMemberIf:
|
||||
// The dolt-server tier's own job, where bazel-dolt-race does not run
|
||||
// (TestBazelDoltRaceArmsExactlyOnePerMode).
|
||||
return map[string]bool{"fork-ro": true, "fork-rw": true, "cache": true, "local": true}
|
||||
case bazelProxiedMemberIf:
|
||||
return map[string]bool{"fork-ro": true, "fork-rw": true}
|
||||
case bazelIntegIf:
|
||||
if strings.EqualFold(with["integration"], "off") {
|
||||
return map[string]bool{}
|
||||
|
|
@ -2884,14 +2898,8 @@ func bazelLaneRunModes(t *testing.T, lane, ifExpr string, with map[string]string
|
|||
// on the job: B1 removed fork-rw). Unlike the historical advisory
|
||||
// lanes, this one legitimately runs in pr.yml's own call (see
|
||||
// bazelAdvisoryLanes' name-specific exception in
|
||||
// TestBazelGateSimulation below).
|
||||
return map[string]bool{"remote": true}
|
||||
case bazelDoltRaceJobIf:
|
||||
// Mode remote, and only on pull_request and merge_group events:
|
||||
// within pr.yml's call (the only PR-time caller these simulations
|
||||
// model) that is mode remote. bazel-farm.yml's pull_request_target
|
||||
// call, push, nightly and dispatch never run it
|
||||
// (TestBazelDoltRaceShadowRunsOnlyInRemotePRCalls).
|
||||
// TestBazelGateSimulation below). bazel-dolt-race's if
|
||||
// (bazelDoltRaceJobIf) is the same expression: mode remote.
|
||||
return map[string]bool{"remote": true}
|
||||
case bazelRRCSeedJobIf, bazelRRCVerifyJobIf:
|
||||
// The remote repo contents cache's writer (push to main) and
|
||||
|
|
@ -2909,7 +2917,7 @@ type bazelGateScenario struct {
|
|||
event string
|
||||
mode, enabled string // the call's rbe-mode / rbe-enabled outputs
|
||||
call string // needs.bazel.result
|
||||
outputs map[string]string // the lanes' outputs ("" = not reported)
|
||||
outputs map[string]string // the jobs' result outputs ("" = not reported)
|
||||
// Non-Bazel needs' results that differ from success.
|
||||
results map[string]string
|
||||
// pr.yml's top-level flags the run script reads (the workflow env is
|
||||
|
|
@ -2977,7 +2985,9 @@ func runPRGateStep(t *testing.T, step ciWorkflowStep, sc bazelGateScenario) (boo
|
|||
case m[3] == "rbe-enabled":
|
||||
got = sc.enabled
|
||||
default:
|
||||
got = sc.outputs[m[3]]
|
||||
// The call's output as GitHub evaluates it over the jobs'
|
||||
// results (a union lane's members read its result first).
|
||||
got = bazelCallOutputValue(t, m[3], sc.outputs)
|
||||
}
|
||||
if got == "" && m[4] != "" {
|
||||
got = "skipped"
|
||||
|
|
@ -3021,8 +3031,10 @@ func TestBazelGateSimulation(t *testing.T) {
|
|||
if _, advisory := bazelAdvisoryLanes[name]; advisory && name != bazelRBEPrewarmJobName && len(lanes[name]) != 0 {
|
||||
t.Errorf("advisory lane %s runs in pr.yml's call in modes %v; turn it off there (%s)", name, lanes[name], bazelAdvisoryLanes[name])
|
||||
}
|
||||
if _, gated := bazelLaneGateIDs[name]; gated && !lanes[name]["remote"] {
|
||||
t.Errorf("gated lane %s does not run in pr.yml's call even in remote mode", name)
|
||||
}
|
||||
for lane := range bazelLaneGateIDs {
|
||||
if bazelLaneReporter(lanes, lane, "remote") == "" {
|
||||
t.Errorf("gated lane %s does not run (nor does its union lane) in pr.yml's call even in remote mode", lane)
|
||||
}
|
||||
}
|
||||
enabledFor := bazelModeEnabled
|
||||
|
|
@ -3054,9 +3066,14 @@ func TestBazelGateSimulation(t *testing.T) {
|
|||
}
|
||||
if modes[mode] {
|
||||
anyRuns = true
|
||||
} else if id, gated := bazelLaneGateIDs[lane]; gated {
|
||||
wantSkips = append(wantSkips, id)
|
||||
} else if g, flagGated := bazelFlagGatedLanes[lane]; flagGated {
|
||||
}
|
||||
if id, gated := bazelLaneGateIDs[lane]; gated {
|
||||
// Skipped by design only when neither the lane nor its
|
||||
// union lane runs in this mode.
|
||||
if bazelLaneReporter(lanes, lane, mode) == "" {
|
||||
wantSkips = append(wantSkips, id)
|
||||
}
|
||||
} else if g, flagGated := bazelFlagGatedLanes[lane]; !modes[mode] && flagGated {
|
||||
// Listed even while advisory, so flipping the flag is the
|
||||
// only change that makes it required.
|
||||
wantSkips = append(wantSkips, g.id)
|
||||
|
|
@ -3144,6 +3161,9 @@ func TestBazelGateSimulation(t *testing.T) {
|
|||
}
|
||||
continue
|
||||
}
|
||||
if _, union := bazelUnionLanes[lane]; union {
|
||||
continue // through its members' ids, below
|
||||
}
|
||||
id, gated := bazelLaneGateIDs[lane]
|
||||
if !gated {
|
||||
// Advisory: not in the call, so it cannot report, and
|
||||
|
|
@ -3153,18 +3173,31 @@ func TestBazelGateSimulation(t *testing.T) {
|
|||
}
|
||||
continue
|
||||
}
|
||||
// The job whose result the lane's id reads in this mode: the
|
||||
// lane itself, or its union lane (bazel-dolt-race in mode
|
||||
// remote, for bazel-doltserver and bazel-proxied).
|
||||
reporter := bazelLaneReporter(lanes, lane, mode)
|
||||
if !modes[mode] {
|
||||
// Skipped by design (baseline); if it ran anyway and failed,
|
||||
// the gate still sees it.
|
||||
sc(lane+" ran and failed", "failure", with(lane, "failure"), false, id)
|
||||
// Not its own job in this mode (skipped by design, or its
|
||||
// union reports); if it ran anyway and failed, the gate still
|
||||
// sees it: through its id, or, where the union's result
|
||||
// comes first in the output, through the aggregate.
|
||||
out := with(lane, "failure")
|
||||
mention := id
|
||||
if bazelCallOutputValue(t, lane, out) != "failure" {
|
||||
mention = bazelAggregateGateID
|
||||
}
|
||||
sc(lane+" ran and failed", "failure", out, false, mention)
|
||||
}
|
||||
if reporter == "" {
|
||||
continue
|
||||
}
|
||||
for _, call := range []string{"success", "failure"} {
|
||||
sc(lane+" alone skipped, aggregate "+call, call, with(lane, ""), false, id)
|
||||
sc(lane+" failed, aggregate "+call, call, with(lane, "failure"), false, id)
|
||||
sc(lane+" cancelled, aggregate "+call, call, with(lane, "cancelled"), false, id)
|
||||
sc(reporter+" (for "+lane+") alone skipped, aggregate "+call, call, with(reporter, ""), false, id)
|
||||
sc(reporter+" (for "+lane+") failed, aggregate "+call, call, with(reporter, "failure"), false, id)
|
||||
sc(reporter+" (for "+lane+") cancelled, aggregate "+call, call, with(reporter, "cancelled"), false, id)
|
||||
}
|
||||
sc(lane+" cancelled, aggregate cancelled", "cancelled", with(lane, "cancelled"), false, id)
|
||||
sc(reporter+" (for "+lane+") cancelled, aggregate cancelled", "cancelled", with(reporter, "cancelled"), false, id)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -4590,11 +4623,8 @@ func TestBazelWorkflowSecretsAndFailureSurface(t *testing.T) {
|
|||
// (bazelCIAnalyticsStepNames): reporting only, never a `needs`
|
||||
// of anything and never read back by this workflow or its
|
||||
// callers.
|
||||
// And each shadow lane's (bazelShadowLanes): it reports into no
|
||||
// gate, so its failure must not fail the call either.
|
||||
_, flagGated := bazelFlagGatedLanes[strings.TrimSuffix(strings.TrimPrefix(path, ".jobs."), ".continue-on-error")]
|
||||
_, shadow := bazelShadowLanes[strings.TrimSuffix(strings.TrimPrefix(path, ".jobs."), ".continue-on-error")]
|
||||
if path != bazelRBEPrewarmContinueOnErrorPath && !flagGated && !shadow && !ciAnalyticsPaths[path] {
|
||||
if path != bazelRBEPrewarmContinueOnErrorPath && !flagGated && !ciAnalyticsPaths[path] {
|
||||
t.Errorf("%s: %s hides failures from pr.yml's ci-gate", bazelWorkflowName, path)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -197,8 +197,8 @@ func TestPRGateRequiresRemoteBazelLanes(t *testing.T) {
|
|||
t.Errorf("pr.yml's ci-gate env sets %s; only its run script may", bazelRemoteGateID)
|
||||
}
|
||||
for lane, id := range bazelLaneGateIDs {
|
||||
if !lanes[lane]["remote"] {
|
||||
t.Errorf("gated lane %s does not run in mode remote", lane)
|
||||
if bazelLaneReporter(lanes, lane, "remote") == "" {
|
||||
t.Errorf("gated lane %s does not run (nor does its union lane) in mode remote", lane)
|
||||
}
|
||||
if !contains(required, id) && !bazelPackageJobs[lane] {
|
||||
t.Errorf("pr.yml's ci-gate does not require %s", id)
|
||||
|
|
@ -266,13 +266,13 @@ func TestPRGateRequiresRemoteBazelLanes(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// Remote, but one gated lane failed, was cancelled or reported nothing:
|
||||
// red, naming that lane.
|
||||
// Remote, but one gated lane (or the union lane reporting for it)
|
||||
// failed, was cancelled or reported nothing: red, naming that lane.
|
||||
for lane, id := range bazelLaneGateIDs {
|
||||
for _, mode := range []string{"remote", "fork-ro"} {
|
||||
for _, res := range []string{"failure", "cancelled", ""} {
|
||||
sc := prGateFor(t, lanes, "pull_request", mode)
|
||||
sc.outputs[lane] = res
|
||||
sc.outputs[bazelLaneReporter(lanes, lane, mode)] = res
|
||||
if pass, log := runPRGateStep(t, gateStep, sc); pass || !regexp.MustCompile(`::error::`+id+`\b`).MatchString(log) {
|
||||
t.Errorf("mode %s, lane %s %q: gate pass = %v, want red naming %s\n%s", mode, lane, res, pass, id, log)
|
||||
}
|
||||
|
|
@ -1170,7 +1170,7 @@ func TestBazelRetiredLanesArePinned(t *testing.T) {
|
|||
}
|
||||
evictionAllowed[want] = true
|
||||
}
|
||||
// The dolt-race shadow's union config carries its proxied member's
|
||||
// The dolt-race union config carries its proxied member's
|
||||
// line (TestBazelDoltRaceConfigIsMemberUnion pins the union).
|
||||
evictionAllowed["test:"+bazelDoltRaceConfig+" --experimental_remote_cache_eviction_retries=0"] = true
|
||||
for _, line := range strings.Split(rc, "\n") {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue