* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2) In mode remote, folds bazel-server-storage (--config=doltserver-integration) and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job (--config=dolt-int), following the dolt-race pattern from #7514. The members have different gating categories (server-storage plain-gated, cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true and a bespoke --label-scoped Shard balance step. All other modes keep the two lanes running unchanged. Action keys are byte-identical between the member configs and the union (17 + 46 = 63 TestRunner actions), so cached results carry over either direction and a revert is a safe rollback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review) Coordinator review of #7535 found three must-fixes: integration:"off" was ignored in remote mode (the union's members already honor it); job-level continue-on-error made a server-storage-only failure read success on push/nightly/dispatch, where main stays red; and the PR's key-parity claim overstated byte-identical actionKeys when only the remote action digest (not the local actionKey) is unaffected by moving --test_env/--test_arg onto the targets' own env/args. Fixes: - bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching the members (#7491's review); bazelLaneRunModes and TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case. - Job-level continue-on-error removed; only the `bazel test` step carries it, and a new always() "Classify tier results" step (tools/bazel/classify_tier_results.py, with its own unit tests and a Go wrapper) computes server_storage_result/cmd_dolt_result from the BEP and decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on synthetic empty-output inputs that a dead job still fails the gate. - Corrected the local-actionKey-vs-remote-digest wording in .bazelrc, bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md. - Should-fix: test step 23->25 / job 58 minutes with reasoning; --label coverage added to shard_budget_test.py; stale lane comment fixed in tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's docstring corrected (members bake sole-run's line in directly, never via --config=sole-run). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review) Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack, julianknutsen): - classify_tier_results.py now fails every tier shut (exit 1) on a non-PASSED label outside every --tier list, so a newly tagged target added without updating the step's --tier args cannot silently read as success (must-fix 1). Its --tier/--check label sets are policy-tied to the BUILD files' own tag sets by a new TestBazelDoltIntTierLabelsMatchTheTags. - A missing or non-integer --rc (a timed-out or cancelled test step) takes the same fail-everything-shut branch instead of crashing argparse with no outputs written (bee-ghosttrack's should-fix 2). - check_testcases.py gained a repeatable --label flag, and bazel.yml's shared "Every target and shard ran tests" step is split into check-ss and check-cd, each continue-on-error and folded into its own tier via classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard no longer reddens the job on push/nightly (must-fix 2). - classify's run: argv is now pinned by exact equality (bazelDoltIntClassifyRun), and every flag except --tier/--check is --StoreOnce; a repeated --tier/--check name or an unrecognized flag is now a hard parser error (must-fix 3). - "Upload test logs" now also fires on steps.test.outcome == 'failure', not just failure() (must-fix 4). - shard_budget.py --label now fails closed (::error, exit 1) on a label with no executed shard in the BEP at all, instead of silently reporting "no sharded test target executed two or more shards" (should-fix 3, carried across both review rounds). - Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the nits: stale "Advisory (continue-on-error: true)" wording, the bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int, "byte-identical" without specifying remote action digest vs. local actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch for an empty union output. Verified: python3 -I on all four tools/bazel/*_test.py (classify 29, check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test ./scripts/... green; bazel --nohome_rc test //scripts:scripts_test --config=ci green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): shard_budget --label fail-closed only on a truly unknown label The previous commit's fail-closed --label check compared against read_shard_durations, which only records executed (non-cached) shards, so a target whose every shard was a cache hit this run had zero entries and was wrongly treated the same as a --label typo. This was not caught by the new tests (none exercised an all-cached target) and broke this PR's own dolt-int CI run: bazel-dolt-int's Shard balance step went red on //internal/storage/dolt:dolt_server_full_test with "no executed shard for it", even though that target was correctly, fully cache-hit. Fix: a new read_tested_labels() reads every label with any testResult event (cached or executed), and --label's fail-closed check now tests against that set instead of read_shard_durations. A fully-cached target still legitimately reports "nothing to judge" through evaluate(); only a label absent from the BEP altogether is now an error. Added the missing test row (every shard cached) that would have caught this before push. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * test(bazel): pin bazel-dolt-int's Upload test logs if-condition A full 42-mutation re-run for PR #7535's second review round found that nothing pinned this job's own "Upload test logs" if-condition (must-fix 4): the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's (unlike dolt-race, a brand-new job) had no equivalent assertion, so a future accidental revert to the old failure()-only condition would go uncaught. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
3223 lines
171 KiB
YAML
3223 lines
171 KiB
YAML
# The Bazel lanes: the only CI run of the Linux Go test tiers. Runs
|
|
# `bazel test //... --config=ci`, the PR-core selection (race, -short, dolt
|
|
# skip), plus one job per tier (pure-Go and js/wasm, the release
|
|
# cross-compile, the dolt-server lane, the embedded-Dolt, proxied-server and
|
|
# server-Dolt storage tiers that pr-risk.yml used to run, integration, the
|
|
# cmd/bd Dolt-server tier).
|
|
#
|
|
# PRs and merge groups run it through pr.yml's `bazel` job (workflow_call).
|
|
# The legacy go test jobs these lanes mirrored are retired (ga-96smfk.22),
|
|
# so pr.yml's ci-gate requires the lanes to have run remotely on rbe-west
|
|
# and passed on every pull_request and merge group (BAZEL_REMOTE): modes
|
|
# skip, local and cache are red there (scripts/pr_risk_bazel_coverage_test.go).
|
|
# Where a lane executes remotely its `bazel test`s add --config=sole-run: it
|
|
# is the tier's only run, so a remote cache eviction fails it rather than
|
|
# re-running the invocation (whose retry could turn a failed test green).
|
|
# Test results are cached (read and written) on every run but nightly's: a
|
|
# test whose action key is unchanged reuses the PASS the farm recorded for
|
|
# it, on a PR, a merge group (merge queue) or a push to main alike, and only
|
|
# what changed runs. nightly.yml passes fresh-test-results: true, so every
|
|
# `bazel test` here appends --config=fresh (BAZEL_FRESH below;
|
|
# --nocache_test_results) and re-executes every test once a night, which
|
|
# keeps sampling flaky and -race failures. A cached failure is never reused.
|
|
# Policy: scripts/ci_merge_queue_test.go.
|
|
# This file has no pull_request or merge_group trigger of its own, so a PR
|
|
# runs the lane once. Every lane reports its job.status as a workflow_call
|
|
# output (named after the job) from an always() last step. pr.yml's ci-gate
|
|
# requires the call's aggregate result (BAZEL) and bazel-test, bazel-pure,
|
|
# bazel-release-cross, bazel-embedded, bazel-integration, bazel-doltserver,
|
|
# bazel-proxied and bazel-server-storage (BAZEL_TEST, BAZEL_PURE,
|
|
# BAZEL_RELEASE_CROSS, BAZEL_EMBEDDED, BAZEL_INTEGRATION, BAZEL_DOLTSERVER,
|
|
# BAZEL_PROXIED, BAZEL_SERVER_STORAGE);
|
|
# bazel-cmd-dolt (the cmd/bd Dolt-server tier, BAZEL_CMD_DOLT) is advisory:
|
|
# job-level continue-on-error, and pr.yml requires it only while its
|
|
# BAZEL_CMD_DOLT_REQUIRED flag is "true" (pr.yml then also passes
|
|
# cmd-dolt-required: true, and bazel-integration leaves //cmd/bd:bd_test,
|
|
# which bazel-cmd-dolt runs too, to it);
|
|
# .github/scripts/bazel-gate.sh turns the rbe job's rbe-mode output into the
|
|
# skips it accepts, so a lane skipped by its `if:` below passes and one that
|
|
# should have run and failed, was cancelled or reported nothing fails the
|
|
# gate. bazel-integration runs in the PR call (pr.yml passes integration:
|
|
# "on") although its legacy counterparts, main.yml's integration jobs, run
|
|
# only on push to main: remotely it is minutes, and it catches on the PR what
|
|
# main.yml would catch after the merge. Fork and Dependabot PRs run it
|
|
# remotely in modes fork-ro/fork-rw like every lane, and in mode cache
|
|
# (rbe-fork closed) unlike the remote-only lanes: with the read-only cache
|
|
# only what the PR changed is built and tested on the runner, and it is
|
|
# their only PR-time integration run (no legacy job runs it on PRs). Mode
|
|
# local (rbe=off) still skips it. Direct runs (push to main,
|
|
# workflow_dispatch, nightly.yml's call) gate nothing. In mode remote the
|
|
# dolt-server and proxied-server tiers run as one union job, bazel-dolt-race
|
|
# (`bazel test --config=dolt-race`), whose job.status is both the
|
|
# bazel-doltserver and the bazel-proxied output; in every other mode
|
|
# bazel-doltserver and bazel-proxied run as before and report their own.
|
|
# Exactly one of the two arms runs in any mode, so the gate ids
|
|
# (BAZEL_DOLTSERVER, BAZEL_PROXIED) and bazel-gate.sh's skips are unchanged
|
|
# (engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "dolt-race Union Lane").
|
|
#
|
|
# Execution mode, decided once by the rbe job (every other job needs it and
|
|
# reads its outputs; none re-derives the condition) and passed to
|
|
# .github/actions/setup-bazel:
|
|
# remote: repo variable RBE_WEST_WORKERS=true, rbe not off, not a fork or
|
|
# Dependabot PR (or a fork PR bazel-farm.yml authorized, below), and the
|
|
# secret RBE_WEST_EXECUTOR (grpcs:// endpoint) present, plus
|
|
# RBE_TLS_CERT, RBE_TLS_KEY (base64; RBE_TLS_CA optional): remote
|
|
# execution on rbe-west, instance "oss" (shared with gascity), from the
|
|
# Blacksmith runner.
|
|
# fork-ro, fork-rw: fork and Dependabot pull_request runs while rbe-west's
|
|
# certificate mint (rbe-fork-mint, infra nativelink-cas/west README
|
|
# "rbe-fork") says open for this run. Such runs get no secrets, no OIDC
|
|
# token and no repository variables, so the farm decides: the rbe job
|
|
# asks the mint's /v1/status, and every lane then proves it is a
|
|
# running job of this run (a CSR uploaded as an artifact of it) and gets
|
|
# its own certificate of about two hours (setup-bazel,
|
|
# fork-credential.sh). fork-ro: instance "oss-fork", the fork pool,
|
|
# nothing cached for anyone; fork-rw (PRs from an allowlisted author's
|
|
# own fork, triggered by an allowlisted user; the mint reads the
|
|
# default branch's .github/bazel-farm-allowlist.txt): instance "oss",
|
|
# the trusted OSS pool, results cached by its workers. Every lane runs,
|
|
# remotely, as in mode remote, from a GitHub-hosted runner (the client
|
|
# only). A lane whose certificate says another tier than this job fails
|
|
# ("re-run all jobs").
|
|
# cache: rbe=cache dispatches, fork and Dependabot PRs while rbe-fork is
|
|
# closed, unreachable or refuses the run (the mint's canary list, its
|
|
# switches, the farm's gate), and same-repo runs without the secret:
|
|
# like local (ubuntu-latest, bazel-embedded, bazel-proxied and
|
|
# bazel-server-storage skipped), plus rbe-west's anonymous read-only
|
|
# cache (.bazelrc --config=fork-cache): hits for every action trusted CI
|
|
# already executed, no credentials, no uploads; and bazel-integration
|
|
# runs too (a warm run hits all but what the PR changed). The farm can
|
|
# close the cache at any time; Bazel then runs everything locally.
|
|
# local: rbe=off dispatches (fork simulation without the cache): local
|
|
# execution on ubuntu-latest, slower but the same results, for every job
|
|
# but the remote-only ones (bazel-embedded, bazel-proxied,
|
|
# bazel-server-storage) and bazel-integration, which are skipped.
|
|
# skip: same-repo runs while RBE_WEST_WORKERS is unset (the farm switch),
|
|
# and authorized fork-farm runs without it or without the secret. Fork
|
|
# and Dependabot pull_request runs cannot read the variable; rbe-fork's
|
|
# own switches decide for them.
|
|
# Trusted-author fork PRs: bazel-farm.yml (pull_request_target, advisory)
|
|
# calls this workflow for fork PRs whose author and triggering user are on
|
|
# the base branch's .github/bazel-farm-allowlist.txt, with fork-farm:
|
|
# authorized and checkout-sha: the event's pinned head SHA. Only that
|
|
# combination, on a pull_request_target event, lets a fork run remotely; a
|
|
# fork's pull_request run (pr.yml, which the fork may edit) stays in mode
|
|
# cache (or local, or fork-ro/fork-rw as the mint decides) whatever it
|
|
# passes, and has no secrets anyway. See engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md,
|
|
# "Trusted-author fork PRs". (rbe-fork's tier rw replaces this workflow
|
|
# once allowlisted forks have run it.)
|
|
# workflow_dispatch with rbe=cache simulates a fork run; rbe=off also drops
|
|
# the cache. The decision is also a workflow_call output (rbe-mode,
|
|
# rbe-enabled), so pr.yml's gate can tell an expected skip from a missing
|
|
# lane without re-deriving it. Callers pass secrets explicitly, never
|
|
# `secrets: inherit`: pr.yml and nightly.yml pass all five declared secrets
|
|
# (the four RBE ones plus rbe-prewarm's bazel-allocator app private key);
|
|
# bazel-farm.yml passes only the four RBE secrets (its pull_request_target
|
|
# run never gets the app credential - see "rbe-west Pre-warm" below and B1
|
|
# in this task's security review).
|
|
#
|
|
# Accepted risk (same model as gascity's bazel-test.yml): on push, merge_group
|
|
# and same-repo pull_request runs, code from the PR head runs with the RBE
|
|
# client certificate. setup-bazel is itself PR code, the key file is readable
|
|
# by every later step and by locally executed Bazel actions, and a .bazelrc
|
|
# edit can point the BEP or the executor anywhere. Anyone who can push a
|
|
# branch here can therefore use or copy the certificate and write to the
|
|
# farm's cache. Fork PRs never get it on their own runs, except those
|
|
# bazel-farm.yml authorizes (allowlisted author and sender, accepted as
|
|
# same-repo-level trust; that workflow's header lists what it adds), and
|
|
# except once queued: a merge_group run loads every workflow file and
|
|
# script from the queue entry's commit, so a fork (or Dependabot) PR that a
|
|
# maintainer queues runs its own edits to .github/, .bazelrc, tools/bazel,
|
|
# MODULE.bazel, BUILD files and CI scripts with all CI secrets, the
|
|
# pre-warm app key included, even if the entry is later ejected. That is
|
|
# the trust of merging it (push to main would run the same code), so the
|
|
# maintainer rule is: review a fork PR's changes to those paths before
|
|
# queueing it, as before merging (engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md,
|
|
# "Merge Queue"). Otherwise: pull_request from a
|
|
# fork receives no secrets, bazel-farm.yml is the only pull_request_target
|
|
# caller (policy-tested), and forks cannot write the runner cache. Nor do
|
|
# Dependabot PRs, which run as same-repo but get no Actions secrets. Fork
|
|
# and Dependabot runs in modes fork-ro/fork-rw hold a certificate of their
|
|
# own instead, worth what their run gets anyway (remote execution under the
|
|
# farm's quotas, AC_OSS reads, hash-verified CAS uploads; no action-cache
|
|
# writes) for about two hours: anyone who copies it from the run gets no
|
|
# more than the PR's author already has. In place: the four RBE secrets are
|
|
# referenced by the setup-bazel step alone (plus the rbe job's emptiness test
|
|
# of RBE_WEST_EXECUTOR, and rbe-prewarm's own has-app/mint steps for the
|
|
# app private key, below), never by a workflow- or job-level env (policy-tested),
|
|
# decoded key material is masked in logs, and CI uploads only remotely
|
|
# executed results (--noremote_upload_local_results in .bazelrc's
|
|
# remote-exec config and in the generated rc), and the certificate
|
|
# is issued to beads CI alone, so it can be revoked without affecting other
|
|
# clients. Maintainers decided (2026-09-28) that same-repo PRs keep remote
|
|
# execution under this model. Still open: a beads-only RBE_INSTANCE. The farm
|
|
# refuses client action-cache writes (only its workers write; rbe-west D2).
|
|
#
|
|
# Accepted risk, rbe-prewarm's credential (see "rbe-west Pre-warm",
|
|
# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, and B1 in this task's security
|
|
# review): the bazel-allocator private key is shared with gascity's own
|
|
# rbe-west scaler (a deliberate, user-approved reuse, not a beads-only
|
|
# credential), so same-repo pull_request, merge_group, push and nightly.yml
|
|
# runs - the only contexts that ever receive RBE_POOL_APP_PRIVATE_KEY - each
|
|
# reach that shared key for the duration of the mint step alone (nothing
|
|
# executes from the checked-out repository tree while it, or the token it
|
|
# produces, is in scope: see the rbe-prewarm job below). That holds only for
|
|
# this file as committed: a merge_group run uses the queued commit's copy,
|
|
# so a queued fork PR that edits this job (or pr.yml) reaches the key with
|
|
# whatever it wrote. Queueing a fork PR is therefore merging-level trust;
|
|
# review its .github/ changes first (see the header above). Unlike the RBE
|
|
# client certificate above, this key cannot be rotated or revoked for beads
|
|
# alone without also breaking gascity's scaler; the kill switch
|
|
# (RBE_POOL_APP_PRIVATE_KEY unset, or RBE_PREWARM_WORKERS=0) disables
|
|
# pre-warming without touching the key itself, so it is the first response to
|
|
# a suspected leak, not a rotation.
|
|
# In pr.yml's run the bd artifact is bazel-ci-build-artifacts, so it cannot
|
|
# collide with the build-artifacts job's ci-build-artifacts. F3: the package
|
|
# gates no longer consume this artifact (they build their own bd); it is
|
|
# kept for the F3.5.3 SHA256SUMS comparison and for debugging.
|
|
#
|
|
# Runner cache: one entry per (.bazelversion, MODULE.bazel, MODULE.bazel.lock)
|
|
# triple, about 1 GB (Bazel's content-addressable --repository_cache, re-hashed on every
|
|
# hit; never the Bazel binary or the repo contents cache), written
|
|
# only by a push to main that missed it. No --disk_cache: remote runs use the
|
|
# remote cache, and local runs would need about 7 GB more.
|
|
#
|
|
# Workflow-file changes suppress pull_request runs under the org approval
|
|
# policy (merge_group runs are not suppressed: they run the queued copy);
|
|
# validate edits to this file with workflow_dispatch on the branch.
|
|
name: Bazel
|
|
|
|
on:
|
|
push:
|
|
branches: [ main ]
|
|
workflow_dispatch:
|
|
inputs:
|
|
rbe:
|
|
description: "Remote execution: on (use secrets), cache (fork simulation: anonymous read-only cache) or off (local only)"
|
|
required: true
|
|
type: choice
|
|
default: "on"
|
|
options: [ "on", "cache", "off" ]
|
|
details:
|
|
description: "List every equivalence divergence"
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
fresh-test-results:
|
|
description: "Re-execute every test (--config=fresh: no cached test results), as nightly does"
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
workflow_call:
|
|
inputs:
|
|
rbe:
|
|
type: string
|
|
required: false
|
|
default: "on"
|
|
details:
|
|
type: boolean
|
|
required: false
|
|
default: true
|
|
# A string, not a boolean: on push and dispatch inputs.integration is
|
|
# null, and null == false in GitHub's loose comparison, so a boolean
|
|
# test would silently stop the lane on main.
|
|
integration:
|
|
description: >
|
|
"off" skips bazel-integration, bazel-cmd-dolt and
|
|
bazel-server-storage; anything else (pr.yml and bazel-farm.yml
|
|
pass "on") runs them where remote execution is on.
|
|
type: string
|
|
required: false
|
|
default: "on"
|
|
# true (pr.yml only, policy-tested to equal its committed
|
|
# BAZEL_CMD_DOLT_REQUIRED flag): pr.yml's ci-gate requires
|
|
# bazel-cmd-dolt, which runs the whole integration-tagged cmd/bd suite
|
|
# (//cmd/bd:bd_dolt_server_test wraps the integration build's
|
|
# bd_test), so bazel-integration leaves //cmd/bd:bd_test to it and
|
|
# each of those tests runs once. A boolean: on push and dispatch it is
|
|
# null, which is not == true, so bazel-farm.yml, nightly.yml, push and
|
|
# dispatch (bazel-cmd-dolt advisory there) keep bd_test in
|
|
# bazel-integration.
|
|
cmd-dolt-required:
|
|
description: >
|
|
true (pr.yml only) drops //cmd/bd:bd_test from bazel-integration:
|
|
the required bazel-cmd-dolt lane runs those tests.
|
|
type: boolean
|
|
required: false
|
|
default: false
|
|
# true (nightly.yml only): every `bazel test` re-executes every test
|
|
# (--config=fresh), the run that samples flaky and -race failures.
|
|
# Everyone else reuses cached results for unchanged tests. A boolean:
|
|
# null on push, so push to main reuses too.
|
|
fresh-test-results:
|
|
description: >
|
|
true (nightly.yml) re-executes every test: no cached test results.
|
|
type: boolean
|
|
required: false
|
|
default: false
|
|
build-artifact-name:
|
|
description: >
|
|
Name of the uploaded bd artifact (ci-build-artifacts layout). pr.yml
|
|
sets bazel-ci-build-artifacts: its build-artifacts job uploads
|
|
ci-build-artifacts in the same run.
|
|
type: string
|
|
required: false
|
|
default: "ci-build-artifacts"
|
|
# Set only by bazel-farm.yml (policy-tested): the fork PR's head SHA
|
|
# from its pull_request_target event, pinned, and "authorized" once
|
|
# that workflow's authorize job has checked the base branch's
|
|
# allowlist. See the header.
|
|
checkout-sha:
|
|
description: >
|
|
Commit every lane checks out (bazel-farm.yml: the trusted fork PR's
|
|
head SHA); empty checks out the event's default ref.
|
|
type: string
|
|
required: false
|
|
default: ""
|
|
fork-farm:
|
|
description: >
|
|
"authorized" (bazel-farm.yml only) lets a pull_request_target fork
|
|
run execute remotely; anything else keeps forks local.
|
|
type: string
|
|
required: false
|
|
default: "off"
|
|
# "on" (pr.yml only) runs package-mcp and package-npm in this call,
|
|
# which waits on nothing but the rbe job. bazel-farm.yml, nightly.yml
|
|
# and direct runs (push, dispatch) keep the default "off": main.yml
|
|
# keeps its own package jobs, and a fork-farm run has no bd artifact
|
|
# consumer waiting on these.
|
|
package-gates:
|
|
description: >
|
|
"on" (pr.yml only) runs the MCP and npm package gates in this call.
|
|
type: string
|
|
required: false
|
|
default: "off"
|
|
# Only what the rbe job and setup-bazel need; callers pass them
|
|
# explicitly. None is required: fork and Dependabot runs have none.
|
|
secrets:
|
|
RBE_WEST_EXECUTOR:
|
|
required: false
|
|
RBE_TLS_CERT:
|
|
required: false
|
|
RBE_TLS_KEY:
|
|
required: false
|
|
RBE_TLS_CA:
|
|
required: false
|
|
# rbe-prewarm only (never read by setup-bazel or the rbe job): the
|
|
# "bazel-allocator" GitHub App's private key (its public client ID is
|
|
# a literal in the mint step, not a secret). The app is installed on
|
|
# gastownhall/gascity alone with Actions (read/write) and Metadata
|
|
# (read) permissions, nothing else; rbe-prewarm's mint step reads the
|
|
# private key directly (it IS a long-lived credential, held in the
|
|
# runner's memory for that one step) to produce a short-lived
|
|
# installation token, which is all any later step ever sees. Unset
|
|
# disables pre-warming. Declared here so fork-rw could in principle
|
|
# request it, but no caller ever does: pr.yml and nightly.yml pass it
|
|
# straight through (both are same-repo/trusted triggers only);
|
|
# bazel-farm.yml's pull_request_target call does not, so no pre-warm
|
|
# credential reaches a fork-authored run before a maintainer queues
|
|
# it (B1, security review of bdef342d5 - accepted risk is same-repo
|
|
# runs reaching the shared gascity scaler key, not forks). Exception:
|
|
# a merge_group run executes the queued commit's workflows, so a
|
|
# queued fork PR's edits run with this key (merging-level trust; see
|
|
# the header). See
|
|
# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "rbe-west Pre-warm".
|
|
RBE_POOL_APP_PRIVATE_KEY:
|
|
required: false
|
|
# The rbe job's decision, and each lane's job.status (success, failure or
|
|
# cancelled; empty when the lane was skipped or never reported). pr.yml's
|
|
# ci-gate reads them through .github/scripts/bazel-gate.sh.
|
|
outputs:
|
|
rbe-enabled:
|
|
description: "'true' when the lanes executed remotely on rbe-west, else 'false'"
|
|
value: ${{ jobs.rbe.outputs.enabled }}
|
|
rbe-mode:
|
|
description: "remote, fork-ro or fork-rw (every lane runs remotely), cache (no remote-only jobs but integration), local (no remote-only jobs), or skip (no jobs): the rbe job's decision"
|
|
value: ${{ jobs.rbe.outputs.mode }}
|
|
bazel-test:
|
|
description: "bazel-test's job.status"
|
|
value: ${{ jobs.bazel-test.outputs.result }}
|
|
bazel-pure:
|
|
description: "bazel-pure's job.status"
|
|
value: ${{ jobs.bazel-pure.outputs.result }}
|
|
bazel-release-cross:
|
|
description: "bazel-release-cross's job.status"
|
|
value: ${{ jobs.bazel-release-cross.outputs.result }}
|
|
bazel-embedded:
|
|
description: "bazel-embedded's job.status"
|
|
value: ${{ jobs.bazel-embedded.outputs.result }}
|
|
bazel-integration:
|
|
description: "bazel-integration's job.status"
|
|
value: ${{ jobs.bazel-integration.outputs.result }}
|
|
# Mode remote: bazel-dolt-race runs both tiers and the member jobs
|
|
# skip; every other mode: the reverse. One side is always empty.
|
|
bazel-doltserver:
|
|
description: "the dolt-server tier's job.status: bazel-dolt-race's in mode remote, bazel-doltserver's otherwise"
|
|
value: ${{ jobs.bazel-dolt-race.outputs.result || jobs.bazel-doltserver.outputs.result }}
|
|
bazel-proxied:
|
|
description: "the proxied-server tier's job.status: bazel-dolt-race's in mode remote, bazel-proxied's otherwise"
|
|
value: ${{ jobs.bazel-dolt-race.outputs.result || jobs.bazel-proxied.outputs.result }}
|
|
# Mode remote: bazel-dolt-int runs both tiers and the member jobs
|
|
# skip; every other mode: the reverse. One side is always empty
|
|
# (item 5 A2). bazel-dolt-int reports its own per-tier classification
|
|
# (classify_tier_results.py), not job.status: the job carries no
|
|
# continue-on-error, so job.status alone would read failure for both
|
|
# tiers on a cmd-dolt-only failure, unlike bazel-cmd-dolt's own
|
|
# advisory, continue-on-error nature outside mode remote.
|
|
bazel-server-storage:
|
|
description: "the server-Dolt storage tier's result: bazel-dolt-int's server_storage_result in mode remote, bazel-server-storage's job.status otherwise"
|
|
value: ${{ jobs.bazel-dolt-int.outputs.server_storage_result || jobs.bazel-server-storage.outputs.result }}
|
|
bazel-cmd-dolt:
|
|
description: "the cmd/bd Dolt-server tier's result (advisory unless pr.yml's BAZEL_CMD_DOLT_REQUIRED is \"true\"): bazel-dolt-int's cmd_dolt_result in mode remote, bazel-cmd-dolt's job.status otherwise"
|
|
value: ${{ jobs.bazel-dolt-int.outputs.cmd_dolt_result || jobs.bazel-cmd-dolt.outputs.result }}
|
|
package-mcp:
|
|
description: "package-mcp's job.status (empty unless package-gates is 'on')"
|
|
value: ${{ jobs.package-mcp.outputs.result }}
|
|
package-npm:
|
|
description: "package-npm's job.status (empty unless package-gates is 'on')"
|
|
value: ${{ jobs.package-npm.outputs.result }}
|
|
rbe-prewarm:
|
|
description: "rbe-prewarm's job.status (advisory: continue-on-error, never gates or blocks a lane)"
|
|
value: ${{ jobs.rbe-prewarm.outputs.result }}
|
|
|
|
# Main runs always finish (they may save the runner cache); PR runs (called
|
|
# from pr.yml or bazel-farm.yml, whose own groups cancel the calling run too)
|
|
# supersede. The group differs from each caller's, as a called workflow's
|
|
# must.
|
|
concurrency:
|
|
group: bazel-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }}
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
env:
|
|
BD_DISABLE_METRICS: "1"
|
|
BD_DISABLE_EVENT_FLUSH: "1"
|
|
# Appended last to every `bazel test` (after the lane's config and
|
|
# sole-run): --config=fresh (.bazelrc, --nocache_test_results) only when
|
|
# the caller asks for fresh results (nightly.yml, or a dispatch); empty
|
|
# otherwise, so results are cached and reused. See the header.
|
|
BAZEL_FRESH: ${{ inputs.fresh-test-results && '--config=fresh' || '' }}
|
|
# Phase B report-only budget for T1/T2 (tools/bazel/critpath.py).
|
|
BAZEL_BUDGET_SECONDS: "420"
|
|
|
|
jobs:
|
|
# The one place the execution mode is decided (see the header), so every
|
|
# job sees the same answer for the whole run. Its decide step reads one
|
|
# secret, and only to test it for emptiness: the step's env holds a
|
|
# boolean, never the value, and the step runs no repository code (it comes
|
|
# before the job's checkout). Fork and Dependabot pull_request runs get no
|
|
# secrets: rbe-fork-mint decides theirs (fork-ro, fork-rw, or cache while
|
|
# it is closed).
|
|
#
|
|
# Then, only when the lanes will execute remotely (enabled: modes remote,
|
|
# fork-ro, fork-rw; merge groups are mode remote), the worker-env preflight:
|
|
# rbe-west's pool workers run gascity main's tools/rbe/worker-env and
|
|
# advertise its pin, and the schedulers match //platforms:rbe_worker's pin
|
|
# exactly, so when gascity re-pins and beads' copy (tools/rbe/worker-env.txt
|
|
# and the pin in platforms/BUILD.bazel) falls behind, every remote action
|
|
# queues until the lanes time out. tools/rbe/worker-env-sync compares the
|
|
# copy with gascity main and, out of step, fails this job: every lane needs
|
|
# it, so they all skip and pr.yml's gate is red within a minute, with the
|
|
# diff and the pin to commit in this job's summary. A fallback to mode
|
|
# cache would not keep the PR green (pr.yml's ci-gate requires a remote
|
|
# run, BAZEL_REMOTE) and would only make it slower, so fail fast. gascity
|
|
# unreachable (exit 2: unknown) warns and proceeds: GitHub's availability
|
|
# never blocks a PR. The steps after decide run repository code (the PR's
|
|
# script, sparse-checked-out) but hold no secret and set no output: the
|
|
# job's outputs are decide's alone. nightly.yml runs the same check every
|
|
# night and opens a ci/infra issue.
|
|
rbe:
|
|
name: execution mode
|
|
# Same-repo PRs (and merge_group/push/dispatch/schedule runs, which are
|
|
# never forks) get the org's Blacksmith pool; a pull_request_target farm
|
|
# run and any fork or Dependabot PR stay on GitHub-hosted runners. This
|
|
# job reads no repository code and holds no secret value (see below), so
|
|
# Blacksmith sees nothing sensitive either way.
|
|
runs-on: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch' || github.event_name == 'schedule' || github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
timeout-minutes: 5
|
|
outputs:
|
|
enabled: ${{ steps.decide.outputs.enabled }}
|
|
mode: ${{ steps.decide.outputs.mode }}
|
|
tier: ${{ steps.decide.outputs.tier }}
|
|
rrc: ${{ steps.rrc.outputs.rrc }}
|
|
steps:
|
|
# The expressions' == is case-insensitive (RBE_WEST_WORKERS=True counts).
|
|
# A push, merge_group or dispatch has no pull_request, so FORK is false.
|
|
# Fork and Dependabot pull_request runs see no secrets and no vars: for
|
|
# them rbe-west decides (rbe-fork-mint's status: open, and the tier from
|
|
# the PR's author and this run's triggering actor). Its answer is the
|
|
# whole run's mode; every lane then gets its own certificate
|
|
# (setup-bazel, BAZEL_FORK_REMOTE) and fails if the tier changed. A
|
|
# mint that is closed, refuses or cannot be reached leaves mode cache,
|
|
# as before rbe-fork.
|
|
- name: Decide remote execution
|
|
id: decide
|
|
env:
|
|
RBE_VAR_ON: ${{ vars.RBE_WEST_WORKERS == 'true' }}
|
|
RBE_INPUT_OFF: ${{ inputs.rbe == 'off' }}
|
|
RBE_INPUT_CACHE: ${{ inputs.rbe == 'cache' }}
|
|
PULL_REQUEST: ${{ github.event_name == 'pull_request' }}
|
|
FORK: ${{ github.event.pull_request.head.repo.fork == true }}
|
|
DEPENDABOT: ${{ github.actor == 'dependabot[bot]' }}
|
|
FORK_FARM: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' && inputs.checkout-sha != '' }}
|
|
HAS_EXECUTOR: ${{ secrets.RBE_WEST_EXECUTOR != '' }}
|
|
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
run: |
|
|
for v in RBE_VAR_ON RBE_INPUT_OFF RBE_INPUT_CACHE PULL_REQUEST FORK DEPENDABOT FORK_FARM HAS_EXECUTOR; do
|
|
case "${!v}" in
|
|
true|false) ;;
|
|
*) echo "::error::$v is '${!v}', want true or false"; exit 1 ;;
|
|
esac
|
|
done
|
|
# A fork or Dependabot pull_request run executes remotely only with
|
|
# rbe-fork's certificate (mint open for this run), else locally with
|
|
# the read-only cache. Any other fork run (pull_request_target)
|
|
# stays local unless bazel-farm.yml authorized it; an authorized
|
|
# fork run is remote or nothing (pr.yml's call already runs it).
|
|
tier=
|
|
if [ "$RBE_INPUT_OFF" = true ]; then
|
|
mode=local
|
|
elif [ "$RBE_INPUT_CACHE" = true ]; then
|
|
mode=cache
|
|
elif [ "$PULL_REQUEST" = true ] && { [ "$FORK" = true ] || [ "$DEPENDABOT" = true ]; }; then
|
|
[[ "$PR_NUMBER" =~ ^[1-9][0-9]{0,8}$ ]] || { echo "::error::PR number '$PR_NUMBER'"; exit 1; }
|
|
url="https://rbe-mint.ops.gascity.com:8444/v1/status?repo=${GITHUB_REPOSITORY#*/}&run=$GITHUB_RUN_ID&attempt=$GITHUB_RUN_ATTEMPT&pr=$PR_NUMBER"
|
|
st=$(curl -sS --connect-timeout 5 --max-time 30 --retry 3 --retry-all-errors "$url" || echo '{}')
|
|
tier=$(jq -r 'if .open == true then .tier else "" end' <<<"$st" 2>/dev/null || true)
|
|
case "$tier" in
|
|
ro|rw) mode="fork-$tier" ;;
|
|
*)
|
|
tier=
|
|
mode=cache
|
|
echo "::warning title=rbe-fork closed for this run::$(jq -r '.reason // .error // "rbe-fork-mint unreachable"' <<<"$st" 2>/dev/null || echo unreachable)"
|
|
;;
|
|
esac
|
|
elif [ "$FORK" = true ] && [ "$FORK_FARM" != true ]; then
|
|
mode=cache
|
|
elif [ "$RBE_VAR_ON" != true ]; then
|
|
mode=skip
|
|
elif [ "$HAS_EXECUTOR" = true ]; then
|
|
mode=remote
|
|
elif [ "$FORK_FARM" = true ]; then
|
|
mode=skip
|
|
else
|
|
mode=cache
|
|
fi
|
|
enabled=false
|
|
case "$mode" in remote|fork-ro|fork-rw) enabled=true ;; esac
|
|
{
|
|
echo "mode=$mode"
|
|
echo "enabled=$enabled"
|
|
echo "tier=$tier"
|
|
} >> "$GITHUB_OUTPUT"
|
|
echo "Bazel execution mode: **$mode** (RBE_WEST_WORKERS on: $RBE_VAR_ON, rbe=off: $RBE_INPUT_OFF, rbe=cache: $RBE_INPUT_CACHE, fork: $FORK, dependabot: $DEPENDABOT, fork farm: $FORK_FARM, executor secret: $HAS_EXECUTOR)" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
|
|
# Bazel's remote repo contents cache on rbe-west (ga-vnycm2.29, ported
|
|
# from gascity's bazel.yml; gascity
|
|
# engdocs/design/bazel-remote-repo-contents-cache.md): extracted
|
|
# external repositories (go_deps, the Go SDK, the LLVM slice) come from
|
|
# rbe-west's action cache instead of being re-fetched by every cold lane
|
|
# client. vars.RBE_REPO_CONTENTS_CACHE rolls it out:
|
|
# unset/off nothing (the default);
|
|
# seed main pushes seed the cache (rrc-seed); no lane reads it;
|
|
# canary seed, and bazel-test reads;
|
|
# on seed, and every lane in .github/scripts/rrc-lane-commands.txt
|
|
# reads.
|
|
# Mode remote only: fork pull_request runs see no vars (mode cache
|
|
# reads through setup-bazel's rc instead, in the jobs that pass
|
|
# BAZEL_RRC_READ, behind its probe of rbe-cache: write-bazelrc.sh). Any other value is off, with a warning, so a typo
|
|
# never fails the required gate. Like decide, this step runs no
|
|
# repository code (it comes before the checkout).
|
|
- name: Remote repo contents cache
|
|
id: rrc
|
|
env:
|
|
RRC: ${{ vars.RBE_REPO_CONTENTS_CACHE }}
|
|
MODE: ${{ steps.decide.outputs.mode }}
|
|
run: |
|
|
rrc=off
|
|
case "$RRC" in
|
|
"" | off) ;;
|
|
seed | canary | on)
|
|
if [ "$MODE" = remote ]; then rrc=$RRC; fi ;;
|
|
*) echo "::warning title=RBE_REPO_CONTENTS_CACHE::unknown value '$RRC' (want off, seed, canary or on); treated as off" ;;
|
|
esac
|
|
echo "rrc=$rrc" >> "$GITHUB_OUTPUT"
|
|
echo "Remote repo contents cache: **$rrc**" >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# The worker-env preflight (see the job's comment): the PR's own copy,
|
|
# so a PR that syncs it passes. Only the two paths it compares.
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
if: ${{ steps.decide.outputs.enabled == 'true' }}
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
sparse-checkout: |
|
|
tools/rbe
|
|
platforms
|
|
|
|
- name: worker-env in step with gascity main
|
|
if: ${{ steps.decide.outputs.enabled == 'true' }}
|
|
env:
|
|
MODE: ${{ steps.decide.outputs.mode }}
|
|
run: |
|
|
rc=0
|
|
tools/rbe/worker-env-sync main || rc=$?
|
|
case "$rc" in
|
|
0) ;;
|
|
1)
|
|
echo "::error title=Bazel lanes stopped before queueing::beads' worker-env pin is out of step with gascity main, so rbe-west has no worker for this run's remote actions (mode $MODE): copy tools/rbe/worker-env.txt and the worker-env pin in platforms/BUILD.bazel from gastownhall/gascity main (this job's summary has the diff and the line to commit; see tools/rbe/worker-env-sync)."
|
|
exit 1
|
|
;;
|
|
*)
|
|
echo "::warning title=rbe worker-env not checked::could not compare beads' worker-env pin with gascity main (exit $rc); proceeding in mode $MODE. If the lanes sit at 0 running actions, run tools/rbe/worker-env-sync."
|
|
;;
|
|
esac
|
|
|
|
# Best-effort pre-warm of gastownhall/gascity's rbe-west OSS worker pool
|
|
# (one Blacksmith-hosted NativeLink worker, rbe-worker-pool.yml) so it is
|
|
# already booting instead of the lanes below waiting ~120s at "N actions, 0
|
|
# running" while the pool scales from zero. A job of its own (not a step in
|
|
# `rbe`), so it never delays the mode decision every other job needs, and
|
|
# it is never a `needs` of any lane and never in pr.yml's ci-gate required
|
|
# list (advisory; .github/scripts/bazel-gate.sh does not read it). See
|
|
# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md, "rbe-west Pre-warm".
|
|
#
|
|
# Only when the run's lanes will actually execute on instance "oss": mode
|
|
# remote (the CI secrets) or fork-rw (rbe-fork-mint's trusted-author tier,
|
|
# cached by the same OSS workers per the header above). Mode fork-ro targets
|
|
# instance "oss-fork", a separate, uncached pool rbe-worker-pool.yml does
|
|
# not serve, so it is excluded (TestBazelCacheModeReachesTheRC's one named
|
|
# exception to "every lane runs remotely" in a fork mode). Like every other
|
|
# lane, the `if:` reads only needs.rbe.outputs - TestBazelRBEJobDecidesOnce
|
|
# forbids re-deriving fork/event facts outside the rbe job itself - and
|
|
# runs-on is the same mode-ternary every other lane uses, so a genuine fork
|
|
# or Dependabot PR that reaches mode fork-rw still never touches Blacksmith:
|
|
# the runner is only a client there, like every other fork-mode lane (see
|
|
# bazel-test's comment above). That PR also carries no repository secrets
|
|
# regardless of runs-on (GitHub strips them from a fork's pull_request
|
|
# event), so the credential-check step below finds nothing to mint even if
|
|
# it runs. The app secret (secrets.RBE_POOL_APP_PRIVATE_KEY) is never
|
|
# read in this `if:` or anywhere else in this file - only inside this job's own step env/with, same as the
|
|
# rbe job's HAS_EXECUTOR pattern - so a missing credential (the app not
|
|
# provisioned yet, or the kill switch) fails soft (no mint attempt, then
|
|
# the script's own ::warning::, exit 0) rather than changing whether the
|
|
# job runs.
|
|
rbe-prewarm:
|
|
name: pre-warm rbe-west pool
|
|
needs: rbe
|
|
# Only mode remote (B1, security review of bdef342d5: was remote or
|
|
# fork-rw). fork-rw is reached only by a fork or Dependabot
|
|
# pull_request run, which GitHub never hands a workflow_call secret
|
|
# regardless of this job's `if:`, so that half of the old condition
|
|
# always found no-app and no-op'd; bazel-farm.yml (the one caller with a
|
|
# distinct pull_request_target trust tier that could otherwise have
|
|
# reached fork-rw with secrets attached) no longer forwards the app
|
|
# secrets either, below. See "rbe-west Pre-warm",
|
|
# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md.
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' }}
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
timeout-minutes: 5
|
|
# Job-level, not step-level: the only continue-on-error in this workflow
|
|
# (TestBazelWorkflowSecretsAndFailureSurface and
|
|
# TestBazelWorkflowJobsAndExecutionMode both carve out this one path by
|
|
# name). Every other job here is a gated lane whose real failure must
|
|
# reach pr.yml's ci-gate; this one is advisory and must never fail the
|
|
# call bazel.yml's own run represents (a job's failure fails the whole
|
|
# reusable-workflow run regardless of `needs`, which would otherwise turn
|
|
# pr.yml's ci-gate red for a pre-warm hiccup nothing requires).
|
|
continue-on-error: true
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# No checkout in this job (B1, security review of bdef342d5; policy-
|
|
# tested that it has none): a step that checked out PR-head content
|
|
# and then ran a script from that tree while this job's credential was
|
|
# live would let that content - or any same-repo branch push, since
|
|
# editing a .github/scripts/*.sh file does not trip the org's
|
|
# workflow-file approval gate the way editing this YAML does - read
|
|
# the shared gascity scaler key. The dispatch logic therefore lives
|
|
# entirely in this trusted workflow definition's own `run:` below,
|
|
# never a checked-out file.
|
|
|
|
# Same env-boolean pattern as the rbe job's HAS_EXECUTOR: the secret is
|
|
# tested for emptiness, never referenced in an `if:` directly, and the
|
|
# next step's `if:` reads this step's output (a job-level env can't be
|
|
# read from another step's `if:`, only its own `env:`).
|
|
- name: Check for the pre-warm app credential
|
|
id: has-app
|
|
env:
|
|
HAS_POOL_APP: ${{ secrets.RBE_POOL_APP_PRIVATE_KEY != '' }}
|
|
run: echo "has-app=$HAS_POOL_APP" >> "$GITHUB_OUTPUT"
|
|
|
|
# Mints a short-lived installation token for the "bazel-allocator"
|
|
# GitHub App (installed on gastownhall/gascity only, Actions
|
|
# read/write + Metadata read): skipped entirely while the app's
|
|
# private key is not provisioned, leaving steps.mint.outputs.token empty, which
|
|
# the dispatch step below treats the same as any other missing-token
|
|
# case (::warning::, exit 0). permission-actions narrows the minted
|
|
# token to Actions even if the app is ever granted more. This is the
|
|
# only step that ever reads the long-lived private key directly
|
|
# (below); every step after it sees at most the short-lived token this
|
|
# one produces. client-id is the app's public Client ID (GET
|
|
# /apps/bazel-allocator; not a credential), committed as a literal so
|
|
# it is reviewed and policy-tested with the rest of this step rather
|
|
# than living in an unversioned repository variable; it replaces the
|
|
# action's deprecated app-id input.
|
|
- name: Mint gastownhall/gascity installation token
|
|
id: mint
|
|
if: ${{ steps.has-app.outputs.has-app == 'true' }}
|
|
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
|
|
with:
|
|
client-id: Iv23ligrqVEhlamZnoPU
|
|
private-key: ${{ secrets.RBE_POOL_APP_PRIVATE_KEY }}
|
|
owner: gastownhall
|
|
repositories: gascity
|
|
permission-actions: write
|
|
|
|
# Inlined rather than checked out and run from a script file (B1
|
|
# above): every byte this step executes comes from this workflow
|
|
# definition, which pull_request_target always loads from the base
|
|
# branch and which editing suppresses pull_request runs under the
|
|
# org's approval policy (see the header). Not on merge_group: a queue
|
|
# entry runs its own commit's copy of this file with no approval
|
|
# gate, so a queued fork PR's edit here runs with the key (review
|
|
# .github/ changes before queueing). steps.mint.outputs.token
|
|
# appears nowhere else in this file (policy-tested): this is the only
|
|
# step with the credential in its environment, and no step after it
|
|
# touches a repository path or runs a local composite action.
|
|
- name: Pre-warm rbe-west OSS worker pool (gastownhall/gascity)
|
|
env:
|
|
GH_TOKEN: ${{ steps.mint.outputs.token }}
|
|
RBE_PREWARM_WORKERS: ${{ vars.RBE_PREWARM_WORKERS }}
|
|
run: |
|
|
set -u
|
|
|
|
POOL_REPO="gastownhall/gascity"
|
|
POOL_WORKFLOW="rbe-worker-pool.yml"
|
|
POOL_REF="main"
|
|
|
|
n="${RBE_PREWARM_WORKERS:-1}"
|
|
case "$n" in
|
|
'') n=1 ;;
|
|
*[!0-9]*) n=1 ;;
|
|
# Two or more digits is over the cap unless they are all zeros
|
|
# (still the kill switch); deciding it here keeps a string too
|
|
# long for the shell's integer compare from slipping past the
|
|
# -gt below unclamped.
|
|
??*)
|
|
case "$n" in
|
|
*[!0]*) n=4 ;;
|
|
*) n=0 ;;
|
|
esac
|
|
;;
|
|
esac
|
|
if [ "$n" -gt 4 ]; then
|
|
n=4
|
|
fi
|
|
|
|
if [ "$n" -le 0 ]; then
|
|
echo "RBE_PREWARM_WORKERS=0: rbe-west pre-warm disabled (kill switch)"
|
|
exit 0
|
|
fi
|
|
|
|
if [ -z "${GH_TOKEN:-}" ]; then
|
|
echo "::warning title=rbe-west pre-warm::no dispatch token (bazel-allocator app not provisioned, or the mint step failed); skipping"
|
|
exit 0
|
|
fi
|
|
|
|
active="$(gh run list -R "$POOL_REPO" --workflow "$POOL_WORKFLOW" --limit 50 \
|
|
--json status --jq '[.[] | select(.status != "completed")] | length' 2>/dev/null)" || true
|
|
case "$active" in
|
|
'' | *[!0-9]*)
|
|
echo "::warning title=rbe-west pre-warm::could not read $POOL_REPO/$POOL_WORKFLOW run status; skipping dispatch"
|
|
exit 0
|
|
;;
|
|
esac
|
|
|
|
missing=$((n - active))
|
|
if [ "$missing" -le 0 ]; then
|
|
echo "${active} $POOL_REPO/$POOL_WORKFLOW worker(s) already active (want ${n}); nothing to dispatch"
|
|
exit 0
|
|
fi
|
|
|
|
dispatched=0
|
|
i=1
|
|
while [ "$i" -le "$missing" ]; do
|
|
if gh workflow run "$POOL_WORKFLOW" -R "$POOL_REPO" --ref "$POOL_REF"; then
|
|
dispatched=$((dispatched + 1))
|
|
else
|
|
echo "::warning title=rbe-west pre-warm::gh workflow run $POOL_WORKFLOW -R $POOL_REPO failed (attempt $i/$missing)"
|
|
fi
|
|
i=$((i + 1))
|
|
done
|
|
|
|
echo "dispatched ${dispatched}/${missing} $POOL_REPO/$POOL_WORKFLOW worker(s) (active before: ${active}, want: ${n})"
|
|
exit 0
|
|
|
|
# Every lane reports its job.status from an always() last step
|
|
# (TestBazelLaneIsGatedAlongsideLegacy). The dispatch step above always
|
|
# exits 0, so this normally reads success regardless of what it found;
|
|
# the job-level continue-on-error above is the backstop for a step
|
|
# that fails outside it (the mint action).
|
|
- name: Record result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
bazel-test:
|
|
name: test
|
|
# Remote-execution runs use the org's Blacksmith pool (the farm's edge
|
|
# admits any runner holding the client certificate; there is no IP
|
|
# allowlist) and do not count against GitHub's concurrent-job cap. Local
|
|
# and cache runs (forks, rbe=off/cache, no secrets) build everything
|
|
# they miss on the runner, so they get the larger GitHub-hosted one.
|
|
# Fork modes (fork-ro/fork-rw) execute remotely but stay GitHub-hosted,
|
|
# like every fork job: the runner is only a client. ubuntu-24.04, not
|
|
# ubuntu-latest, in every lane that reads the remote repo contents cache
|
|
# in mode cache: the cached trees are Linux x86_64 ones from a 24.04 host
|
|
# (design R2), so the image must not move under them.
|
|
#
|
|
# 8 vCPU, not the other lanes' 4: with every action a remote cache hit,
|
|
# this lane's `bazel test //... --config=ci` spent ~63 of 81s in
|
|
# client-side loading and analysis (1798 packages, 66k configured
|
|
# targets; critical path 7.65s, merge group 37721729916), which Bazel
|
|
# parallelizes across the client's cores. Analysis-only, seeded runner
|
|
# cache (ga-vnycm2.8): 38.3s on 2 vCPU, 25.6s on 4, 16.3s on 8.
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
|
|
# Remote execution goes to rbe-west (instance "oss", the Blacksmith-backed
|
|
# worker pool shared with gascity). Skipped only in mode skip (same-repo
|
|
# runs while RBE_WEST_WORKERS is unset).
|
|
if: ${{ needs.rbe.outputs.mode != 'skip' }}
|
|
timeout-minutes: 100
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
env:
|
|
# This lane is the only pre-merge run of the PR-core tests (pr.yml's
|
|
# retired PR Core job never retried): where it executes remotely
|
|
# (mode remote, and the fork modes) --config=sole-run fails on a
|
|
# remote cache eviction instead of re-running the whole invocation
|
|
# (whose retry could turn a failed test green). Test results are
|
|
# cached and reused like every lane's; only nightly re-executes them
|
|
# all (BAZEL_FRESH).
|
|
BAZEL_SOLE_RUN: ${{ needs.rbe.outputs.enabled == 'true' && '--config=sole-run' || '' }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# Go is for tools/bazel/equivalence.py's `go list` and for make; Bazel
|
|
# uses its own hermetic SDK.
|
|
- name: Set up Go
|
|
id: setup-go
|
|
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
cache: false
|
|
|
|
# Read-only: main.yml's build-artifacts job owns this cache.
|
|
- name: Restore Go module cache
|
|
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ~/go/pkg/mod
|
|
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
|
|
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
|
|
|
|
# Go does not re-hash extracted module directories on use; a restored
|
|
# cache written by an untrusted run is checked against go.sum first.
|
|
- name: Verify restored Go modules
|
|
run: go mod verify
|
|
|
|
# The remote repo contents cache, read-only (the rbe job's rrc output:
|
|
# canary, bazel-test; on, every lane). Each cached repository costs
|
|
# several sequential round trips, resolved on Skyframe's loading
|
|
# threads (default: the CPU count), so without 64 of them a cached read
|
|
# is slower than fetching. Both lines are key neutral (startup and
|
|
# loading only). Reading only: setup-bazel's rc keeps
|
|
# --noremote_upload_local_results, so nothing here writes; rrc-seed
|
|
# writes, on push to main. Before Set up Bazel: no server starts first.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && (needs.rbe.outputs.rrc == 'on' || needs.rbe.outputs.rrc == 'canary') }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# T1: remote + restored repository cache; T2: remote, cold runner; local:
|
|
# no remote execution (reported, never budgeted).
|
|
- name: Measurement tier
|
|
id: tier
|
|
env:
|
|
REMOTE: ${{ steps.bazel.outputs.remote }}
|
|
RESTORED: ${{ steps.bazel.outputs.cache-restored }}
|
|
run: |
|
|
if [ "$REMOTE" = "true" ] && [ "$RESTORED" = "true" ]; then tier=T1
|
|
elif [ "$REMOTE" = "true" ]; then tier=T2
|
|
else tier=local; fi
|
|
echo "tier=$tier" >> "$GITHUB_OUTPUT"
|
|
{
|
|
echo "### Bazel / test"
|
|
echo
|
|
echo "Tier **$tier**: remote execution $REMOTE, runner cache restored $RESTORED."
|
|
echo
|
|
} >> "$GITHUB_STEP_SUMMARY"
|
|
|
|
# --profile and --build_event_json_file are command options, so they go
|
|
# after `test` (a startup position fails).
|
|
- name: bazel test //... --config=ci
|
|
id: test
|
|
timeout-minutes: 90
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=ci ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
echo >> "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
# pr.yml's retired build-artifacts job, built by Bazel: the same
|
|
# artifact name and layout (bd-linux-gms-pure, SHA256SUMS,
|
|
# build-manifest.txt). F3: the package gates build
|
|
# their own bd and no longer consume this artifact; it is kept for the
|
|
# F3.5.3 SHA256SUMS comparison and for debugging. The script documents
|
|
# which binary and how it differs from `go build`'s.
|
|
- name: Package bd (ci-build-artifacts layout)
|
|
id: bd-artifact
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: ./scripts/ci/package-bazel-bd.sh "$RUNNER_TEMP/bd-artifacts"
|
|
|
|
- name: Upload build artifacts
|
|
if: ${{ always() && steps.bd-artifact.outcome == 'success' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ${{ inputs.build-artifact-name || 'ci-build-artifacts' }}
|
|
path: ${{ runner.temp }}/bd-artifacts/
|
|
retention-days: 1
|
|
if-no-files-found: error
|
|
|
|
- name: Critical-path report
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
env:
|
|
TIER: ${{ steps.tier.outputs.tier }}
|
|
run: |
|
|
budget=()
|
|
if [ "$TIER" != "local" ]; then budget=(--budget "$BAZEL_BUDGET_SECONDS"); fi
|
|
python3 tools/bazel/critpath.py "$RUNNER_TEMP/bazel-profile.json" \
|
|
--tier "$TIER" --log "$RUNNER_TEMP/bazel-test.log" "${budget[@]}"
|
|
|
|
# Fails on any Go test that `go test` runs and Bazel did not (tag filter,
|
|
# manual, missing target, empty shard) unless it is allowlisted.
|
|
# It reads test.xml from bazel-testlogs, which --config=integration
|
|
# shares with --config=ci (same output directory): if a job ever runs
|
|
# both lanes, run each lane's equivalence right after its own
|
|
# `bazel test`, before the other lane overwrites the logs.
|
|
- name: Go test equivalence
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 15
|
|
env:
|
|
DETAILS: ${{ inputs.details && '--details' || '' }}
|
|
run: |
|
|
args=()
|
|
if [ -n "$DETAILS" ]; then args+=("$DETAILS"); fi
|
|
python3 tools/bazel/equivalence.py --bep "$RUNNER_TEMP/bazel-bep.json" "${args[@]}"
|
|
|
|
# Runs last: it rewrites BUILD files in the checkout. main moves BUILD
|
|
# inputs (imports, go.mod) without touching Bazel, so drift shows up here
|
|
# first. On drift the step still fails, and the BUILD.bazel /
|
|
# MODULE.bazel / MODULE.bazel.lock part of the fix is written as a patch
|
|
# for the bazel-sync-patch artifact, which bazel-autofix.yml pushes to
|
|
# same-repo PR branches (or turns into an apply recipe for forks).
|
|
- name: BUILD files in sync (gazelle, go_srcs, MODULE.bazel)
|
|
id: sync
|
|
if: ${{ always() && steps.bazel.outcome == 'success' }}
|
|
timeout-minutes: 15
|
|
env:
|
|
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
|
run: |
|
|
start=$(date +%s)
|
|
# A stale go_srcs block is drift like any other: record it, then let
|
|
# bazel-sync fix it so the patch covers it too.
|
|
check=0
|
|
make bazel-sync-check || check=$?
|
|
make bazel-sync
|
|
drift="$(git status --porcelain --untracked-files=all)"
|
|
echo "BUILD sync check: $(( $(date +%s) - start ))s" >> "$GITHUB_STEP_SUMMARY"
|
|
if [ -n "$drift" ]; then
|
|
echo "::error title=Bazel BUILD files out of sync::run 'make bazel-sync' and commit the result"
|
|
echo "$drift"
|
|
git diff | head -200
|
|
echo "BUILD files **out of sync**; run \`make bazel-sync\` and commit the result." >> "$GITHUB_STEP_SUMMARY"
|
|
./scripts/ci/bazel-sync-patch.sh "$RUNNER_TEMP/bazel-sync-patch"
|
|
exit 1
|
|
fi
|
|
exit "$check"
|
|
|
|
# Data only: bazel-autofix.yml validates every path before applying it.
|
|
# Not from bazel-farm.yml's pull_request_target runs: autofix reads only
|
|
# "PR" runs, and a trusted-fork run publishes nothing meant for a
|
|
# privileged consumer.
|
|
- name: Upload BUILD sync patch
|
|
if: ${{ always() && steps.sync.outcome == 'failure' && github.event_name != 'pull_request_target' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-sync-patch
|
|
path: ${{ runner.temp }}/bazel-sync-patch/
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome == 'failure' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Only a push to main that missed the exact key writes the runner cache.
|
|
# Its content (--repository_cache only, never the rc or credentials) is
|
|
# fixed by .bazelversion, MODULE.bazel and MODULE.bazel.lock, so one
|
|
# entry per triple is enough and later pushes do not churn the quota.
|
|
- name: Save Bazel runner cache
|
|
if: ${{ always() && github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.bazel.outcome == 'success' && steps.bazel.outputs.cache-hit != 'true' }}
|
|
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ${{ runner.temp }}/bazel-ci-cache
|
|
key: bazel-repo-v4-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel', 'MODULE.bazel.lock') }}
|
|
|
|
# A redacted, schema'd summary of this invocation's BEP, exec log and
|
|
# profile: timing, per-mnemonic cache classes, cold actions, test
|
|
# results, repo-fetch time. tools/bazel/ci_analytics_extract.py reads
|
|
# only a field whitelist and validates every emitted string, so this
|
|
# (unlike the raw BEP file) is safe to upload unredacted; see its
|
|
# module docstring. Mirrors gascity's ci/analytics-extractor lane step.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: test
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-test-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# The CGO_ENABLED=0 portability boundaries (pr.yml's retired
|
|
# check-cmd-bd-puregeo-tests job) with Bazel-built binaries: --config=pure
|
|
# builds bd and three pure test binaries and runs the -short cmd/bd subset
|
|
# (PURE_CMD_BD_TESTS, pinned by scripts/ci_workflow_test.go);
|
|
# --config=js-wasm builds the hooks test for js/wasm, which the Go SDK's
|
|
# go_js_wasm_exec runs under node with an exact pass-count guard.
|
|
bazel-pure:
|
|
name: pure-Go and js/wasm
|
|
# Same runner and skip rule as bazel-test.
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
|
|
if: ${{ needs.rbe.outputs.mode != 'skip' }}
|
|
timeout-minutes: 90
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
env:
|
|
PURE_CMD_BD_TESTS: '^Test(Help|CheckRemoteSafety|FormatDestroyToken|ShouldWireInitRemote|ExtractPrefix|IsNumericID|GetWorktreeGitDir|DriftItemStatuses|RunDriftChecks|IsServerProbablyRunning|CheckHooksDriftNotGitRepo|CheckServerDriftNoBeadsDir|CheckRemoteDriftNoBeadsDir|NocgoFactoriesHonorSharedServer|NocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|NocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag)'
|
|
# The only pre-merge run of these boundaries: where it executes
|
|
# remotely --config=sole-run fails on a remote cache eviction instead
|
|
# of re-running the whole invocation (whose retry could turn a failed
|
|
# test green). Test results are cached and reused like every lane's;
|
|
# only nightly re-executes them all (BAZEL_FRESH).
|
|
BAZEL_SOLE_RUN: ${{ needs.rbe.outputs.enabled == 'true' && '--config=sole-run' || '' }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# Only for go_js_wasm_exec and wasm_exec_node.js, which must match the
|
|
# Go version Bazel compiles with (go.mod's; policy-tested).
|
|
- name: Set up Go
|
|
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
cache: false
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6
|
|
with:
|
|
node-version: '24'
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
- name: Run js/wasm hook boundary
|
|
id: build-wasm
|
|
run: |
|
|
set -euo pipefail
|
|
bazel build --config=js-wasm //internal/hooks:hooks_test \
|
|
--profile="$RUNNER_TEMP/bazel-profile-build-wasm.json" \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep-build-wasm.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec-build-wasm.log.zst" \
|
|
--experimental_build_event_upload_strategy=local
|
|
bin="$(bazel cquery --config=js-wasm --output=files //internal/hooks:hooks_test)"
|
|
cd internal/hooks
|
|
|
|
# Same selector and count guard as pr.yml's job.
|
|
test_status=0
|
|
# wasm_exec.js caps argv+env size; the Blacksmith runner's environment
|
|
# exceeds it, so run with only what the hook test needs.
|
|
wasm_exec="$(go env GOROOT)/lib/wasm/go_js_wasm_exec"
|
|
test_output="$(env -i PATH="$PATH" HOME="$HOME" TMPDIR="$RUNNER_TEMP" \
|
|
BD_DISABLE_METRICS=1 BD_DISABLE_EVENT_FLUSH=1 \
|
|
"$wasm_exec" "$GITHUB_WORKSPACE/$bin" \
|
|
-test.run '^TestRunHookReportsUnsupportedExecution$' \
|
|
-test.v 2>&1)" || test_status=$?
|
|
printf '%s\n' "$test_output"
|
|
(( test_status == 0 )) || exit "$test_status"
|
|
|
|
run_count=0
|
|
pass_count=0
|
|
nonpass_count=0
|
|
pass_pattern='^--- PASS: TestRunHookReportsUnsupportedExecution \([0-9.]+s\)$'
|
|
nonpass_pattern='^[[:space:]]*--- (FAIL|SKIP): '
|
|
while IFS= read -r line; do
|
|
if [[ "$line" == "=== RUN TestRunHookReportsUnsupportedExecution" ]]; then
|
|
((run_count += 1))
|
|
fi
|
|
if [[ "$line" =~ $pass_pattern ]]; then
|
|
((pass_count += 1))
|
|
fi
|
|
if [[ "$line" =~ $nonpass_pattern ]]; then
|
|
((nonpass_count += 1))
|
|
fi
|
|
done <<< "$test_output"
|
|
if (( run_count != 1 || pass_count != 1 || nonpass_count != 0 )); then
|
|
printf 'expected one exact js/wasm hook test run and pass with no failures or skips; got runs=%s passes=%s nonpasses=%s\n' \
|
|
"$run_count" "$pass_count" "$nonpass_count" >&2
|
|
exit 1
|
|
fi
|
|
|
|
- name: Build cmd/bd and pure-Go test binaries (--config=pure)
|
|
id: build-pure
|
|
run: >-
|
|
bazel build --config=pure
|
|
//cmd/bd:bd
|
|
//cmd/bd:bd_test
|
|
//internal/storage/embeddeddolt:embeddeddolt_test
|
|
//internal/tracker:tracker_test
|
|
--profile="$RUNNER_TEMP/bazel-profile-build-pure.json"
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep-build-pure.json"
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec-build-pure.log.zst"
|
|
--experimental_build_event_upload_strategy=local
|
|
|
|
# `go build` with CGO_ENABLED=0 rejects a pure binary whose imports reach
|
|
# dolt's cgo-only store/nbs (gozstd). Bazel compiles gozstd's pure stubs
|
|
# instead, whose init panics (third_party/patches/gozstd_nocgo.patch), so
|
|
# the check happens here: every pure artifact is started once. bd_test
|
|
# starts in the next step. -test.run=^$ runs no test (TestMain still
|
|
# runs; BEADS_TEST_SKIP=dolt keeps tracker's from starting a container).
|
|
- name: Start every pure-Go artifact (gozstd contamination check)
|
|
id: test-pure
|
|
run: |
|
|
set -euo pipefail
|
|
# Not instrumented: no BEP/exec-log/profile is captured for this
|
|
# one-off smoke invocation (design's bazel-pure invocation list
|
|
# names build-wasm, build-pure, test-pure and test-bd only).
|
|
bazel run --config=pure //cmd/bd:bd -- version
|
|
bazel test --config=pure ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
//internal/storage/embeddeddolt:embeddeddolt_test \
|
|
//internal/tracker:tracker_test \
|
|
--test_sharding_strategy=disabled \
|
|
'--test_arg=-test.run=^$' \
|
|
--test_env=BEADS_TEST_SKIP=dolt \
|
|
--profile="$RUNNER_TEMP/bazel-profile-test-pure.json" \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep-test-pure.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec-test-pure.log.zst" \
|
|
--experimental_build_event_upload_strategy=local
|
|
|
|
# One unsharded process, like `go test -run`; the test.xml check keeps a
|
|
# selector that matches nothing from passing.
|
|
- name: Run pure-Go cmd/bd test subset (--config=pure)
|
|
id: test-bd
|
|
run: |
|
|
set -euo pipefail
|
|
bazel test --config=pure ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} //cmd/bd:bd_test \
|
|
--test_sharding_strategy=disabled \
|
|
"--test_arg=-test.run=$PURE_CMD_BD_TESTS" \
|
|
--profile="$RUNNER_TEMP/bazel-profile-test-bd.json" \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep-test-bd.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec-test-bd.log.zst" \
|
|
--experimental_build_event_upload_strategy=local
|
|
n="$(grep -c '<testcase ' bazel-testlogs/cmd/bd/bd_test/test.xml || true)"
|
|
echo "pure cmd/bd subset: $n test cases"
|
|
(( n > 0 ))
|
|
|
|
# A redacted, schema'd summary of this job's four BEP-tracked
|
|
# invocations (build-wasm, build-pure, test-pure, test-bd; `bazel run
|
|
# ... -- version` above is deliberately not tracked, see its comment).
|
|
# Same contract as bazel-test's step; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled()"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: pure
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
args=()
|
|
for step in build-wasm build-pure test-pure test-bd; do
|
|
bep="$RUNNER_TEMP/bazel-bep-$step.json"
|
|
[ -f "$bep" ] || continue
|
|
args+=(--invocation "$step" "$bep" "$RUNNER_TEMP/bazel-exec-$step.log.zst" "$RUNNER_TEMP/bazel-profile-$step.json")
|
|
done
|
|
[ "${#args[@]}" -gt 0 ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
"${args[@]}" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-pure-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# Every release target (scripts/ci/release-targets.txt) on every PR, so a
|
|
# platform-gating mistake is caught here instead of at tag time (#5662):
|
|
# the whole tree, cgo off, one remote `bazel build --platforms=...` per
|
|
# target (the script's header has the coverage boundary: the cgo release
|
|
# builds are not compiled here). nogo (//tools/nogo) validates every one of
|
|
# those compiles, so this is also the lint and vet gate for files only
|
|
# other platforms build (//go:build windows, darwin, !linux ...;
|
|
# engdocs/LINTING.md). A job of its own rather than a step of bazel-pure:
|
|
# release_cross is a split transition that shares no analysis with the pure
|
|
# invocations, and as a step it was about a minute on the longest cached
|
|
# lane. Same runner and skip rule as bazel-pure (it runs in every mode but
|
|
# skip).
|
|
bazel-release-cross:
|
|
name: release cross-compile
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ needs.rbe.outputs.mode != 'skip' }}
|
|
timeout-minutes: 90
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
- name: Cross-compile every release target (--config=release-cross)
|
|
run: |
|
|
set -euo pipefail
|
|
./scripts/ci/bazel-release-cross-compile.sh
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# pr-risk.yml's retired embedded-Dolt tier ("Test (Embedded Dolt Cmd N/20)",
|
|
# "Test (Embedded Dolt Storage N/5)", "Test (Embedded Dolt Conformance -
|
|
# core/audit)") as --config=embedded: those jobs, and the frozen 20-shard
|
|
# cmd block they read, no longer exist (see engdocs/TESTING.md) — this runs
|
|
# the embedded-tagged variants of
|
|
# //cmd/bd:bd_test and //internal/storage/embeddeddolt:embeddeddolt_test.
|
|
# The shard variants run the jobs' own shard scripts and manifests, so Bazel
|
|
# shard k runs exactly the tests of job k. A job of its own rather than a
|
|
# second invocation in bazel-test: the tier's long race shards stay off
|
|
# bazel-test's critical path and budget, and it compiles nothing bazel-test
|
|
# does not (same race configuration, shared through the remote cache).
|
|
bazel-embedded:
|
|
name: embedded Dolt tier
|
|
# Remote execution only: skipped unless the rbe job chose remote,
|
|
# fork-ro or fork-rw (enabled; not in modes cache, local or skip).
|
|
# Locally the tier is 72 race test
|
|
# processes (50 cmd/bd shards + 20 embeddeddolt storage shards + 2
|
|
# conformance partitions; ~24k test-seconds measured locally -- see
|
|
# scripts/ci/embedded_cmd_test_durations.json and
|
|
# scripts/ci/embedded_storage_test_durations.json -- up from an earlier,
|
|
# smaller test surface's ~7-10k estimate) on one runner, several hours
|
|
# per run on the pool whose job cap this track exists to relieve, and it
|
|
# proves nothing pr-risk.yml's embedded jobs (which do run for forks) do
|
|
# not.
|
|
#
|
|
# Shard counts: storage (40): its tests are serial, so a shard's wall
|
|
# time is the sum of its tests, and CI shards run ~1.3-1.6x the rbe-west
|
|
# measurements; 40 is where the LPT-packed heaviest shard
|
|
# (scripts/ci/gen_embedded_storage_shard_manifest.py --weights=duration
|
|
# --repack) is its heaviest single top-level test, ~71s
|
|
# (TestLifecycleCloseReopenContract). cmd is a 100-shard block split
|
|
# over two targets, //cmd/bd:bd_embedded_test (shards 1-50) and
|
|
# //cmd/bd:bd_embedded_part2_test (51-100), because Bazel caps a rule at
|
|
# 50 shards ("Having more than 50 shards is indicative of poor test
|
|
# organization. Please reduce the number of shards." -- a native
|
|
# analysis-time error, not a style lint); the step below checks that
|
|
# the two ranges tile the block exactly once. After its slowest
|
|
# functions were split into smaller top-level tests, a cmd shard's wall
|
|
# time is its slowest test: ~45s median, and at most the *Concurrent
|
|
# stress tests' 65-145s (their bd subprocesses queue on one workspace's
|
|
# writer lock, so they cannot be split without changing what they test). On RBE the tier's wall time ~= that floor
|
|
# + fixed cost (checkout, build, shard dispatch), not ~=
|
|
# sum(durations)/shard_count the way a CPU-core-bound local sweep is --
|
|
# each shard is its own remote action and runs concurrently with every
|
|
# other shard, farm capacity permitting.
|
|
# This lane is the tier's only PR run (pr-risk.yml's legacy jobs are
|
|
# retired) and pr.yml's gate requires it to have run here and passed.
|
|
# The runs-on expression is the shared one: Blacksmith in mode remote,
|
|
# GitHub-hosted in the fork modes (a client only).
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ needs.rbe.outputs.enabled == 'true' }}
|
|
timeout-minutes: 60
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# Bazel retries a flaky = True test (up to three attempts) and reports a
|
|
# later pass as PASSED. The policy test greps BUILD files for it; this
|
|
# asks Bazel itself, so a macro or a variable cannot hide one. Cheap:
|
|
# loading only, with the repository cache the test step needs anyway.
|
|
- name: No test target is marked flaky
|
|
run: |
|
|
flaky="$(bazel query 'attr(flaky, 1, tests(//...))')"
|
|
if [ -n "$flaky" ]; then
|
|
echo "::error::flaky = True on ${flaky//$'\n'/ }: the gated lanes run each test once"
|
|
exit 1
|
|
fi
|
|
|
|
# --build_event_json_file is a command option, so it goes after `test`.
|
|
- name: bazel test //... --config=embedded
|
|
id: test
|
|
timeout-minutes: 50
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=embedded ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=embedded: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
# A shard the manifest assigns nothing, or a selector that matches
|
|
# nothing, exits 0: fail if any target's or shard's test.xml lists no
|
|
# tests (or is missing).
|
|
- name: Every target and shard ran tests
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json"
|
|
|
|
# This lane is the tier's only pre-merge run. The shard scripts discover tests from source, but each
|
|
# Bazel binary holds only its target's srcs: a listed test the binary
|
|
# lacks matches nothing and passes silently. Fail unless every Bazel
|
|
# shard ran exactly the tests its script lists (list-only mode, cheap),
|
|
# and unless every shard and conformance partition ran at least one
|
|
# top-level test that was not skipped (an injected -test.short or
|
|
# BEADS_TEST_SKIP turns the tier into t.Skip calls).
|
|
- name: Every listed test ran in its shard
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: >-
|
|
python3 tools/bazel/check_shard_coverage.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--suite //cmd/bd:bd_embedded_test .github/scripts/embedded-test-shard.sh 50@0/100
|
|
--suite //cmd/bd:bd_embedded_part2_test .github/scripts/embedded-test-shard.sh 50@50/100
|
|
--suite //internal/storage/embeddeddolt:embeddeddolt_embedded_test .github/scripts/embedded-storage-test-shard.sh 40
|
|
--whole //internal/storage/embeddeddolt:embeddeddolt_conformance_core_test
|
|
--whole //internal/storage/embeddeddolt:embeddeddolt_conformance_core_slow_test
|
|
--whole //internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome != 'skipped' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-embedded-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: embedded
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-embedded-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# main.yml's "Main Linux integration packages (N/6)" and "Main Linux
|
|
# integration cmd/bd (N/8)" jobs (and nightly.yml's retired Full Test
|
|
# Suite, the same go test in one job) as --config=integration: every package with
|
|
# -tags=integration,gms_pure_go, race, no -short, BEADS_TEST_SKIP=dolt (see
|
|
# .bazelrc). A job of its own: ci and integration share bazel-testlogs, so
|
|
# bazel-test's equivalence step would read this lane's test.xml if both ran
|
|
# in one job, and the tagged race build is a second full compile.
|
|
# No equivalence step: equivalence.py's default mode is the --config=ci
|
|
# selection (go list without the integration tag, -short, ^TestEmbedded
|
|
# skipped), and its job mode needs the integration jobs' go test -json,
|
|
# which no run here has. check_testcases.py still catches a target or
|
|
# shard that ran no tests. pr.yml's gate requires it (BAZEL_INTEGRATION)
|
|
# on same-repo PRs; see the header.
|
|
# Exactly-once cmd/bd: where the caller says bazel-cmd-dolt is required
|
|
# (inputs.cmd-dolt-required true: pr.yml while its
|
|
# BAZEL_CMD_DOLT_REQUIRED is "true"), the target set is //... minus
|
|
# //cmd/bd:bd_test. bazel-cmd-dolt has this job's `if:`, so it runs in
|
|
# every mode this one does (remote, fork-ro, fork-rw, cache; Dependabot
|
|
# too), bazel-gate.sh accepts the two lanes' skips in the same modes, and
|
|
# its //cmd/bd:bd_dolt_server_test runs that same integration-build
|
|
# bd_test without BEADS_TEST_SKIP=dolt (a superset: the skip only skips).
|
|
# Rollback: commit BAZEL_CMD_DOLT_REQUIRED "false" and pr.yml's
|
|
# cmd-dolt-required: false together (policy-tested equal); bd_test is
|
|
# back here on the next run.
|
|
bazel-integration:
|
|
name: integration
|
|
# Remote (modes remote, fork-ro, fork-rw), or local with the read-only
|
|
# cache (mode cache: fork and Dependabot PRs while rbe-fork is closed,
|
|
# rbe=cache dispatches), which serves every action
|
|
# trusted CI already ran, so only what the PR changed builds and tests
|
|
# on the GitHub-hosted runner. Never plain local (rbe=off): the tagged
|
|
# race build of //... plus 14 jobs' worth of tests on one runner, cold.
|
|
# A caller may turn it off with integration: "off"; pr.yml and
|
|
# bazel-farm.yml pass "on".
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
|
|
if: ${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outputs.mode == 'cache') && inputs.integration != 'off' }}
|
|
# F5 S2 Variant A added the "Nogo: integration library files" step after
|
|
# `test`; +5 over the previous 85 keeps it headroom without exceeding
|
|
# TestBazelIntegrationJob's test-step-timeout+15 ceiling (75+15=90).
|
|
timeout-minutes: 90
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# Remote: a cold compile (a PR touching a widely imported package; the
|
|
# first cold run took 17 minutes end to end, warm ones ~3.5) plus the
|
|
# longest test action (1200s cap, see .bazelrc) if it starts last.
|
|
# scripts/ci_workflow_test.go keeps this at or above that sum. Mode
|
|
# cache on a 4-CPU runner (simulated, 2026-10-02): ~2.5 minutes warm,
|
|
# 3-5 for a cmd/bd change, plus up to ~11 more when slow blob reads
|
|
# make Bazel restart the build ("Lost inputs"), and ~30 with the cache
|
|
# closed (the farm's kill switch: everything local, cold), which must
|
|
# still fit so closing the cache does not turn fork PRs red.
|
|
# Where bazel-cmd-dolt is required, //cmd/bd:bd_test is excluded (see
|
|
# above): on a PR that changes cmd/bd or what it imports, its race
|
|
# link and 16 shards were this step's last ~2-3.5 minutes (runs of
|
|
# 2026-10-05: 215-314s wall, 113-208s of it after every other test
|
|
# had finished).
|
|
- name: bazel test //... --config=integration
|
|
id: test
|
|
timeout-minutes: 75
|
|
env:
|
|
EXCLUDE_TARGETS: ${{ inputs.cmd-dolt-required == true && '-//cmd/bd:bd_test' || '' }}
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
if [[ -n "$EXCLUDE_TARGETS" ]]; then
|
|
echo "bazel-cmd-dolt is required: excluding $EXCLUDE_TARGETS (it runs there)" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
fi
|
|
bazel test //... --config=integration ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
${EXCLUDE_TARGETS:+-- "$EXCLUDE_TARGETS"} \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=integration: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
# As in bazel-embedded. dolt_version_test and the repository guards in
|
|
# //scripts/repochecks are plain sh_tests (no Go testcases in their
|
|
# test.xml), which --config=integration's tag filter keeps, unlike the
|
|
# embedded one.
|
|
- name: Every target and shard ran tests
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json" --not-go //tools/bazel:dolt_version_test --not-go //scripts/repochecks:all
|
|
|
|
# F5 S2 Variant A: the integration/doltserver-integration/doltserver-cmd
|
|
# configuration (--config=integration plus race) has no full-lane nogo
|
|
# owner (every lane above passes --norun_validations, .bazelrc). This
|
|
# narrow build keeps full nogo (vet + golangci-lint) on the
|
|
# configuration's non-test library package,
|
|
# //internal/testutil/integration, which every one of those lanes
|
|
# compiles; the 56 integration-tagged _test.go files this does not
|
|
# cover are the accepted loss (tools/nogo/unvalidated_integration_tests.txt,
|
|
# f5-spec.md §3). --keep_going so one finding does not hide another.
|
|
# scripts/nogo_lint_policy_test.go's
|
|
# TestNogoConfigurationsHaveOneValidatingLane pins this step as the
|
|
# configuration's sole validating lane.
|
|
- name: "Nogo: integration library files"
|
|
id: nogo-integration
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 10
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel build --config=integration --@rules_go//go/config:race --keep_going \
|
|
--output_groups=nogo_fix \
|
|
-- //internal/testutil/integration:integration \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-nogo-integration.log" || rc=$?
|
|
procs=$(grep -oE 'INFO: [0-9]+ process' "$RUNNER_TEMP/bazel-nogo-integration.log" | tail -1 || true)
|
|
echo "nogo-integration: exit $rc, $(( $(date +%s) - start ))s wall${procs:+, $procs}" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome != 'skipped' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-integration-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: integration
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-integration-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# pr.yml's retired container-backed jobs, "Test (storage domain + uow)"
|
|
# and "Contract corpus", as Bazel's dolt-server lane: --config=doltserver
|
|
# runs the dolt-server variants (domain/..., uow in 8 shards, tracker/...,
|
|
# doctor/fix, protocol in 8 shards, testutil), each of which starts its
|
|
# own dolt sql-server from the pinned dolt, so they execute on the remote
|
|
# workers like any other test (locally with rbe=off).
|
|
# Not in mode remote: there bazel-dolt-race runs this tier together with
|
|
# the proxied-server tier and reports for both (item 5). In the fork
|
|
# modes, cache and local there is no in-flight deduplication of one
|
|
# client's actions and the client does the work, so a union would cost the
|
|
# sum of the two tiers instead of the max: this job runs there as before.
|
|
bazel-doltserver:
|
|
name: dolt-server lane
|
|
# Same runner as bazel-test; bazel-test's skip rule, plus mode remote
|
|
# (bazel-dolt-race's).
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
|
|
if: ${{ needs.rbe.outputs.mode != 'skip' && needs.rbe.outputs.mode != 'remote' }}
|
|
timeout-minutes: 90
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
env:
|
|
# This lane is the only pre-merge run of these suites (the go test
|
|
# jobs it replaced never retried): where it executes remotely
|
|
# --config=sole-run fails on a remote cache eviction instead of
|
|
# re-running the whole invocation (whose retry could turn a failed
|
|
# test green).
|
|
# Test results are cached and reused like every lane's; only nightly
|
|
# re-executes them all (BAZEL_FRESH).
|
|
BAZEL_SOLE_RUN: ${{ needs.rbe.outputs.enabled == 'true' && '--config=sole-run' || '' }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
- name: bazel test //... --config=doltserver
|
|
id: test
|
|
timeout-minutes: 80
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=doltserver ${BAZEL_SOLE_RUN:+"$BAZEL_SOLE_RUN"} ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=doltserver: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome == 'failure' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-doltserver-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: doltserver
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-doltserver-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# Like pr-risk.yml's retired proxied-server tier, "Test (Proxied Dolt Cmd
|
|
# N/15)" (that job, and the frozen 15-shard block it read, no longer exist;
|
|
# see engdocs/TESTING.md), as --config=doltserver-proxied:
|
|
# //cmd/bd:bd_proxied_test runs the shard script, so shard k runs exactly
|
|
# that manifest entry's tests. This lane runs 34 shards of its own
|
|
# (.github/scripts/proxied-cmd-test-shards.txt's duration-balanced 34-block;
|
|
# see scripts/ci/proxied_test_durations.json) — shard k here was never the
|
|
# same split as the retired job's shard k. Same build as --config=doltserver
|
|
# (race bd_test, non-race
|
|
# bd_for_tests), shared with bazel-doltserver through the remote cache.
|
|
# The test binary is race, as PR Risk's retired jobs built it. It also runs
|
|
# //cmd/bd:bd_managed_local_test, the managed-local proxied lifecycle
|
|
# (formerly proxied-local-smoke.yml) in a loopback-only network
|
|
# namespace, which needs the workers' unprivileged user namespaces. A job
|
|
# of its own, not gated by any integration switch: it runs in pr.yml's
|
|
# call and its gate requires it (BAZEL_PROXIED). Fork modes only: in mode
|
|
# remote bazel-dolt-race runs this tier (see bazel-doltserver's comment).
|
|
bazel-proxied:
|
|
name: proxied-server tier
|
|
# Remote execution only, like bazel-embedded: 30 concurrent shards, each
|
|
# running its own dolt sql-server busy on ~4 cores, do not fit a
|
|
# GitHub-hosted runner regardless of any single shard's wall time.
|
|
# This lane is the tier's only PR run (pr-risk.yml's legacy jobs are
|
|
# retired) and pr.yml's gate requires it to have run here and passed.
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ needs.rbe.outputs.enabled == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
timeout-minutes: 30
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# Bazel retries a flaky = True test (up to three attempts) and reports a
|
|
# later pass as PASSED. The policy test greps BUILD files for it; this
|
|
# asks Bazel itself, so a macro or a variable cannot hide one. Cheap:
|
|
# loading only, with the repository cache the test step needs anyway.
|
|
- name: No test target is marked flaky
|
|
run: |
|
|
flaky="$(bazel query 'attr(flaky, 1, tests(//...))')"
|
|
if [ -n "$flaky" ]; then
|
|
echo "::error::flaky = True on ${flaky//$'\n'/ }: the gated lanes run each test once"
|
|
exit 1
|
|
fi
|
|
|
|
# Remote: each shard's Go timeout is its job's 15m, 1200s Bazel cap,
|
|
# plus a compile. The 34-shard block is LPT-packed by
|
|
# scripts/ci/proxied_test_durations.json, a rbe-west capture of each
|
|
# top-level test's summed leaf-test time (see its header), to ~187
|
|
# cost-seconds per shard; shards are expected around 55-95s (the same
|
|
# packed cost has run up to 1.45x the median wall time). Nearly
|
|
# all of that is `bd init --proxied-server` against the shard's dolt
|
|
# sql-server, ~one per subtest, each as a user scoped to its own
|
|
# database (cmd/bd/proxied_shared_harness_test.go); a shard running far
|
|
# longer is worth comparing against this job's recent history before
|
|
# assuming a hang.
|
|
- name: bazel test //... --config=doltserver-proxied
|
|
id: test
|
|
timeout-minutes: 25
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=doltserver-proxied ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=doltserver-proxied: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
# As in bazel-embedded.
|
|
- name: Every target and shard ran tests
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json"
|
|
|
|
# This lane is the tier's only pre-merge run: as in bazel-embedded, every Bazel shard must run
|
|
# exactly the tests its shard script lists, and not only skips.
|
|
- name: Every listed test ran in its shard
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: >-
|
|
python3 tools/bazel/check_shard_coverage.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--suite //cmd/bd:bd_proxied_test .github/scripts/proxied-test-shard.sh 34
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome != 'skipped' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-proxied-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: proxied-server
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-proxied-server-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# Item 5 (scripts/bazel_dolt_race_test.go): the dolt-server and
|
|
# proxied-server tiers in one `bazel test --config=dolt-race` (.bazelrc:
|
|
# exactly the union of --config=doltserver's and
|
|
# --config=doltserver-proxied's lines, one tag filter for both), mode
|
|
# remote only, in place of bazel-doltserver and bazel-proxied, which run
|
|
# in every other mode. One client waits on rbe-west for both tiers'
|
|
# shards at once (27 + 35 test actions, under the remote-exec config's 64
|
|
# jobs), so the job ends with the slower tier and saves the other's
|
|
# runner. Each member target's test action key is unchanged (aquery-
|
|
# verified), so cached results carry over both ways, and a revert of the
|
|
# cutover re-executes nothing. Gated: its job.status is both the
|
|
# bazel-doltserver and the bazel-proxied workflow_call output (a failure
|
|
# in either tier fails both ids, which are both required). Every step has
|
|
# a timeout and the job's exceeds their sum by 5 minutes while staying
|
|
# under the merge queue's 60-minute check timeout
|
|
# (TestBazelDoltRaceJobTimeouts), so a hang fails a named step instead of
|
|
# dropping the merge group. The member jobs' configs and the union's must
|
|
# stay key-identical: the mode decides which runs (TestBazelDoltRaceConfigIsMemberUnion).
|
|
# Its ci-analytics LANE is dolt-race.
|
|
bazel-dolt-race:
|
|
name: dolt-race
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' }}
|
|
# At least every step's timeout-minutes plus 5, and below the merge
|
|
# queue's 60-minute check timeout so this job alone cannot exhaust it
|
|
# (TestBazelDoltRaceJobTimeouts). Not a whole-run bound: the rbe job runs
|
|
# first and sibling lanes have longer timeouts.
|
|
timeout-minutes: 56
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
timeout-minutes: 5
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
timeout-minutes: 1
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
timeout-minutes: 5
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# As in bazel-proxied.
|
|
- name: No test target is marked flaky
|
|
timeout-minutes: 3
|
|
run: |
|
|
flaky="$(bazel query 'attr(flaky, 1, tests(//...))')"
|
|
if [ -n "$flaky" ]; then
|
|
echo "::error::flaky = True on ${flaky//$'\n'/ }: the gated lanes run each test once"
|
|
exit 1
|
|
fi
|
|
|
|
# Both tiers' shards at once: 27 dolt-server + 35 proxied test actions,
|
|
# under the remote-exec config's 64 jobs. No --config=sole-run: its one
|
|
# line (no retry on a remote cache eviction) is already in the union.
|
|
# Shards are expected around 55-95s on rbe-west (see bazel-proxied's
|
|
# test step), the proxied tier's 1200s cap included. The step timeout is
|
|
# >= proxied test_timeout 1200 s + about 3 min so Bazel writes TIMEOUT
|
|
# logs before GitHub kills the step.
|
|
- name: bazel test //... --config=dolt-race
|
|
id: test
|
|
timeout-minutes: 23
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=dolt-race ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=dolt-race: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
# The member lanes' checks, once over the union's BEP: bazel-proxied's
|
|
# (the dolt-server lane has none). check_shard_coverage ignores targets
|
|
# it does not list; check_testcases checks every target, so it covers
|
|
# the dolt-server tier too, which bazel-doltserver does not check: a
|
|
# dolt-server shard that runs no test turns this job red.
|
|
- name: Every target and shard ran tests
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 2
|
|
run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json"
|
|
|
|
- name: Every listed test ran in its shard
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 2
|
|
run: >-
|
|
python3 tools/bazel/check_shard_coverage.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--suite //cmd/bd:bd_proxied_test .github/scripts/proxied-test-shard.sh 34
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 3
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-dolt-race-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
env:
|
|
LANE: dolt-race
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-dolt-race-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as the bazel-doltserver and
|
|
# bazel-proxied workflow_call outputs for pr.yml's ci-gate. always()
|
|
# runs it after a failure, a timeout or a cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
timeout-minutes: 1
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# pr-risk.yml's server-Dolt storage tier, "Test (Server Dolt Conformance)"
|
|
# and "Test (Server Dolt Full Suite N/16)", as --config=doltserver-integration:
|
|
# the conformance and 16-shard full-suite variants of
|
|
# //internal/storage/dolt over a non-race dolt_test build, each shard
|
|
# running the job's own shard script. --config=doltserver-integration has
|
|
# exactly --config=integration's build flags, so it compiles nothing
|
|
# bazel-integration does not (shared through the remote cache). This is a
|
|
# PR-time tier, unlike main.yml's integration jobs, and pr.yml's gate
|
|
# requires it (BAZEL_SERVER_STORAGE); it also honors the same
|
|
# `integration` input as bazel-integration/bazel-cmd-dolt so an explicit
|
|
# "off" skips it too, even though no caller passes that today (pr.yml and
|
|
# bazel-farm.yml both pass "on") -- see the `integration` input's own
|
|
# description.
|
|
# Not in mode remote: there bazel-dolt-int runs this tier together with
|
|
# the cmd/bd Dolt-server tier and reports for both (item 5 A2). In the
|
|
# fork modes and cache there is no in-flight deduplication of one client's
|
|
# actions, so a union would cost the sum of the two tiers instead of the
|
|
# max: this job runs there as before.
|
|
bazel-server-storage:
|
|
name: server-Dolt storage tier
|
|
# Remote execution only, like bazel-integration: locally the tagged race
|
|
# build of //... alone is most of the timeout. Plus mode remote
|
|
# (bazel-dolt-int's).
|
|
# This lane is the tier's only PR run (pr-risk.yml's legacy jobs are
|
|
# retired) and pr.yml's gate requires it to have run here and passed.
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ needs.rbe.outputs.enabled == 'true' && needs.rbe.outputs.mode != 'remote' && inputs.integration != 'off' }}
|
|
timeout-minutes: 30
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# Bazel retries a flaky = True test (up to three attempts) and reports a
|
|
# later pass as PASSED. The policy test greps BUILD files for it; this
|
|
# asks Bazel itself, so a macro or a variable cannot hide one. Cheap:
|
|
# loading only, with the repository cache the test step needs anyway.
|
|
- name: No test target is marked flaky
|
|
run: |
|
|
flaky="$(bazel query 'attr(flaky, 1, tests(//...))')"
|
|
if [ -n "$flaky" ]; then
|
|
echo "::error::flaky = True on ${flaky//$'\n'/ }: the gated lanes run each test once"
|
|
exit 1
|
|
fi
|
|
|
|
# Remote: the longest shard takes ~2-3 minutes (the storage suite's Go
|
|
# timeout is 15m per shard, 1200s Bazel cap), plus a compile.
|
|
- name: bazel test //... --config=doltserver-integration
|
|
id: test
|
|
timeout-minutes: 25
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=doltserver-integration ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=doltserver-integration: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
# As in bazel-embedded.
|
|
- name: Every target and shard ran tests
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json"
|
|
|
|
# This lane is the tier's only pre-merge run: as in bazel-embedded, every full-suite shard must run
|
|
# exactly the tests its shard script lists, and neither it nor the
|
|
# conformance target only skips.
|
|
- name: Every listed test ran in its shard
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: >-
|
|
python3 tools/bazel/check_shard_coverage.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--suite //internal/storage/dolt:dolt_server_full_test .github/scripts/server-storage-test-shard.sh 16
|
|
--whole //internal/storage/dolt:dolt_server_conformance_test
|
|
|
|
# rbe-ci-cost-latency-study.md recommendation 4: this tier's last
|
|
# shard has trailed the rest by 4.3-4.5 min in 2 of 8 runs. Warns
|
|
# (::warning) when the slowest shard this run is far above the
|
|
# median; does not fail the job yet (--fail-on-imbalance switches
|
|
# that on once the budget has run clean for a while).
|
|
- name: Shard balance
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: python3 tools/bazel/shard_budget.py --bep "$RUNNER_TEMP/bazel-bep.json"
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome != 'skipped' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-server-storage-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: server-storage
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-server-storage-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# The cmd/bd Dolt-server tier, as --config=doltserver-cmd: the whole
|
|
# integration-tagged cmd/bd suite (//cmd/bd:bd_dolt_server_test, 32
|
|
# shards, the slowest tests pinned to 16 of them) against a hermetic dolt sql-server, i.e. the Dolt-gated cmd/bd
|
|
# tests (TestCLI_*, TestInit*, the store-backed suites) that every other
|
|
# lane and legacy job skips with BEADS_TEST_SKIP=dolt or leaves out of its
|
|
# manifest; plus the differential regression suite
|
|
# (//tests/regression:regression_test, 14 shards: the candidate
|
|
# bd_for_tests against the catalog-pinned v0.49.6 release, formerly
|
|
# regression.yml). --config=doltserver-cmd has exactly
|
|
# --config=integration's build flags, so it shares bazel-integration's
|
|
# compile through the remote cache. Runs exactly where bazel-integration runs (the same if):
|
|
# remotely (modes remote, fork-ro, fork-rw), or locally with the
|
|
# read-only cache (mode cache: fork and Dependabot PRs while rbe-fork is
|
|
# closed), never plain local (rbe=off), and not when a caller passes
|
|
# integration: "off".
|
|
# Advisory: job-level continue-on-error keeps a failure out of this
|
|
# call's aggregate result (BAZEL), and pr.yml's ci-gate requires the
|
|
# lane's own result (BAZEL_CMD_DOLT) only while pr.yml's committed
|
|
# BAZEL_CMD_DOLT_REQUIRED flag is "true" (scripts/ci_workflow_test.go,
|
|
# TestBazelCmdDoltJob); pr.yml then passes cmd-dolt-required: true and
|
|
# this lane is the integration build's only bd_test run
|
|
# (TestBazelIntegrationExcludesBdTestOnlyWhereCmdDoltCovers).
|
|
bazel-cmd-dolt:
|
|
name: cmd/bd Dolt-server tier
|
|
# Not in mode remote: there bazel-dolt-int runs this tier together with
|
|
# the server-Dolt storage tier (item 5 A2). See bazel-server-storage's
|
|
# comment above.
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
|
|
if: ${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outputs.mode == 'cache') && inputs.integration != 'off' && needs.rbe.outputs.mode != 'remote' }}
|
|
timeout-minutes: 85
|
|
continue-on-error: true
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# Remote: a compile shared with bazel-integration plus the longest
|
|
# of 32 shards (~1 minute: tools/bazel/go_test_pinned_shard.sh pins
|
|
# the slowest tests to shards of their own; each shard pays a few
|
|
# seconds of TestMain and its own dolt sql-server; Go's
|
|
# -test.timeout=19m, Bazel's 1200s cap). Mode cache on a 4-CPU runner: the compile hits
|
|
# the read-only cache, and a PR that changes cmd/bd or what it imports
|
|
# runs every shard locally, four at a time (~2,200 test-seconds
|
|
# measured unloaded, slower four-up: ~20-30 minutes); with the cache
|
|
# closed, add the cold race build (~30).
|
|
- name: bazel test //... --config=doltserver-cmd
|
|
id: test
|
|
timeout-minutes: 75
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=doltserver-cmd ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=doltserver-cmd: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
exit "$rc"
|
|
|
|
# As in bazel-embedded: every shard of the target ran tests (the Go
|
|
# binary shards itself over every top-level test, so together the
|
|
# shards run each exactly once).
|
|
- name: Every target and shard ran tests
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
run: python3 tools/bazel/check_testcases.py --bep "$RUNNER_TEMP/bazel-bep.json"
|
|
|
|
- name: Upload test logs
|
|
if: ${{ failure() && steps.test.outcome != 'skipped' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-cmd-dolt-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 3
|
|
env:
|
|
LANE: cmd-dolt
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-cmd-dolt-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too; job.status is the steps' status, unaffected by the
|
|
# job-level continue-on-error.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# Item 5 A2 (scripts/bazel_dolt_int_test.go): the server-Dolt storage and
|
|
# cmd/bd Dolt-server tiers in one `bazel test --config=dolt-int` (.bazelrc:
|
|
# exactly the union of --config=doltserver-integration's and
|
|
# --config=doltserver-cmd's lines, one tag filter for both), mode remote
|
|
# only, in place of bazel-server-storage and bazel-cmd-dolt, which run in
|
|
# every other mode (there a single client does both tiers' work without
|
|
# in-flight deduplication, so a union would cost the sum of the two tiers
|
|
# rather than the max). One client waits on rbe-west for both tiers' shards
|
|
# at once (17 + 46 = 63 test actions, under the remote-exec config's 64
|
|
# jobs), so the job ends with the slower tier and the other tier's runner
|
|
# is saved. cmd-dolt's Go-level -test.timeout=19m arg and its
|
|
# BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION env moved from --config=doltserver-cmd
|
|
# onto //cmd/bd:bd_dolt_server_test's and //tests/regression:regression_test's
|
|
# own args/env (their comments), so the union's tag filter alone decides
|
|
# which targets see them: without the move, server-storage's per-shard 15m
|
|
# Go timeout would become 19m under the union. The final argv and
|
|
# environment either way are identical (aquery-verified for all 4 affected
|
|
# targets, item 5 A2 PR), so remote action digests -- what RBE's cache keys
|
|
# on -- are unchanged and cached results should carry over; this is a local
|
|
# actionKey change for cmd-dolt's 46 actions, not a remote one (--test_env
|
|
# is part of the build configuration's own fingerprint, which every target
|
|
# under that configuration picks up, unlike a target's own env attribute),
|
|
# confirmed in practice by the first merge group's tests_cached ==
|
|
# tests_total in ci-analytics, not claimed here as already observed.
|
|
#
|
|
# Unlike dolt-race, the members have different gating categories:
|
|
# bazel-server-storage is plain-gated (BAZEL_SERVER_STORAGE), bazel-cmd-dolt
|
|
# is flag-gated and advisory (BAZEL_CMD_DOLT, continue-on-error) until
|
|
# pr.yml's BAZEL_CMD_DOLT_REQUIRED is "true". This job therefore carries no
|
|
# job-level continue-on-error: on push, nightly and dispatch nothing reads
|
|
# BAZEL_SERVER_STORAGE off a workflow_call output, so job-level
|
|
# continue-on-error (job.status reads success regardless of the test step's
|
|
# own result) would let a server-storage failure end such a run green,
|
|
# where it must stay red exactly as it does on main today. Instead only the
|
|
# `bazel test` step carries continue-on-error, and an always() step
|
|
# (tools/bazel/classify_tier_results.py) reads the BEP by target label,
|
|
# computes each tier's own result, and fails the job itself iff
|
|
# server-storage's tier failed (a cmd-dolt-only test failure leaves the job
|
|
# green, as bazel-cmd-dolt's own continue-on-error already does outside
|
|
# mode remote) or bazel's exit code was neither 0 nor 3 (a build or infra
|
|
# failure is not attributable to one tier, so both are marked failed and
|
|
# the job fails regardless). Both members' workflow_call outputs read this
|
|
# job's matching per-tier output first, then their own member's; a job that
|
|
# dies before the classify step leaves both outputs empty, which mode
|
|
# remote's gate (bazel-gate.sh, ci-gate.sh) already treats as skipped and
|
|
# fails, same as an empty bazel-server-storage/bazel-cmd-dolt result would.
|
|
# Every step has a timeout and the job's exceeds their sum by 5 minutes
|
|
# while staying under the merge queue's 60-minute check timeout
|
|
# (TestBazelDoltIntJobTimeouts): 58 leaves only about 2 minutes for runner
|
|
# pickup and the rbe job inside that 60-minute budget, so a hang that runs
|
|
# to this job's own timeout can drop the merge group; accepted, because
|
|
# such a run is red anyway and the 58-minute job timeout bounds only this
|
|
# job, not the whole run (as #7514's dolt-race job comment). The member
|
|
# jobs' configs and the union's must stay key-identical: the mode decides
|
|
# which runs (TestBazelDoltIntConfigIsMemberUnion). Its ci-analytics LANE
|
|
# is dolt-int.
|
|
bazel-dolt-int:
|
|
name: dolt-int
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && inputs.integration != 'off' }}
|
|
# At least every step's timeout-minutes plus 5, and below the merge
|
|
# queue's 60-minute check timeout so this job alone cannot exhaust it
|
|
# (TestBazelDoltIntJobTimeouts). 58 leaves only about 2 minutes for
|
|
# runner pickup and the rbe job inside the merge queue's 60-minute check;
|
|
# a hang that runs to this timeout can therefore drop the group, which is
|
|
# accepted because such a run is red anyway and the 58-minute job timeout
|
|
# bounds this job, not the whole run (as #7514's dolt-race job comment).
|
|
timeout-minutes: 58
|
|
outputs:
|
|
server_storage_result: ${{ steps.classify.outputs.server_storage_result }}
|
|
cmd_dolt_result: ${{ steps.classify.outputs.cmd_dolt_result }}
|
|
steps:
|
|
# inputs.checkout-sha: see the header (empty: the event's default ref).
|
|
# checkout v7 refuses a fork head on pull_request_target unless opted
|
|
# in; only bazel-farm.yml's authorized call opts in (false elsewhere).
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
timeout-minutes: 5
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
# The remote repo contents cache, read-only, as in bazel-test.
|
|
- name: Remote repo contents cache (read)
|
|
if: ${{ needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc == 'on' }}
|
|
timeout-minutes: 1
|
|
run: |
|
|
{
|
|
echo 'startup --experimental_remote_repo_contents_cache'
|
|
echo 'common --loading_phase_threads=64'
|
|
} >> .bazelrc.local
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
timeout-minutes: 5
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
BAZEL_RRC_READ: 'true' # mode cache reads the repo contents cache through rbe-cache
|
|
BAZEL_FORK_REMOTE: ${{ startsWith(needs.rbe.outputs.mode, 'fork-') && 'true' || '' }}
|
|
RBE_FORK_TIER: ${{ needs.rbe.outputs.tier }}
|
|
RBE_FORK_PR: ${{ github.event.pull_request.number }}
|
|
|
|
# As in bazel-server-storage and bazel-proxied.
|
|
- name: No test target is marked flaky
|
|
timeout-minutes: 3
|
|
run: |
|
|
flaky="$(bazel query 'attr(flaky, 1, tests(//...))')"
|
|
if [ -n "$flaky" ]; then
|
|
echo "::error::flaky = True on ${flaky//$'\n'/ }: the gated lanes run each test once"
|
|
exit 1
|
|
fi
|
|
|
|
# Both tiers' shards at once: 17 server-storage + 46 cmd-dolt test
|
|
# actions, under the remote-exec config's 64 jobs. No --config=sole-run:
|
|
# its one line (no retry on a remote cache eviction) is already in the
|
|
# union. The step timeout is >= the union's eternal-size test_timeout
|
|
# cap (1200s, shared by bd_dolt_server_test and regression_test) + about
|
|
# 3 min so Bazel writes TIMEOUT logs before GitHub kills the step, plus
|
|
# the margin server-storage's own step budgets for a cold compile (its
|
|
# own step was 25). continue-on-error: a cmd-dolt-only test failure
|
|
# must not fail this step (and so the job) outright; the classify step
|
|
# below decides that per-tier, from this step's own exit code (written
|
|
# to its "rc" output) and the BEP.
|
|
- name: bazel test //... --config=dolt-int
|
|
id: test
|
|
timeout-minutes: 25
|
|
continue-on-error: true
|
|
run: |
|
|
set -o pipefail
|
|
start=$(date +%s)
|
|
rc=0
|
|
bazel test //... --config=dolt-int ${BAZEL_FRESH:+"$BAZEL_FRESH"} \
|
|
--build_event_json_file="$RUNNER_TEMP/bazel-bep.json" \
|
|
--execution_log_compact_file="$RUNNER_TEMP/bazel-exec.log.zst" \
|
|
--profile="$RUNNER_TEMP/bazel-profile.json" \
|
|
--experimental_build_event_upload_strategy=local \
|
|
2>&1 | tee "$RUNNER_TEMP/bazel-test.log" || rc=$?
|
|
echo "bazel test --config=dolt-int: exit $rc, $(( $(date +%s) - start ))s wall" | tee -a "$GITHUB_STEP_SUMMARY"
|
|
echo "rc=$rc" >> "$GITHUB_OUTPUT"
|
|
exit "$rc"
|
|
|
|
# check_testcases split per tier (should-fix 2, PR #7535 review): a
|
|
# single shared step with no continue-on-error would fail this job
|
|
# outright on a cmd-dolt-only zero-testcase shard (e.g. TestMain dying
|
|
# on dolt server start under BEADS_TEST_REQUIRE_DOLT_CONTAINER=1),
|
|
# where main (bazel-cmd-dolt alone) stays green, and the divergence
|
|
# would only show up under the documented rollback to
|
|
# BAZEL_CMD_DOLT_REQUIRED=false. Each step is --label-scoped to its own
|
|
# tier (so a --label typo or a label the run did not test fails
|
|
# closed, not open: check_testcases.py's own --label section) and
|
|
# carries continue-on-error so neither can fail the job by itself; the
|
|
# classify step below folds each step's own outcome into that tier's
|
|
# result with --check, so a cmd-dolt failure here reaches
|
|
# cmd_dolt_result only, same as a cmd-dolt BEP test failure does.
|
|
- name: Every target and shard ran tests (server-storage)
|
|
id: check-ss
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
continue-on-error: true
|
|
timeout-minutes: 1
|
|
run: >-
|
|
python3 tools/bazel/check_testcases.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--label //internal/storage/dolt:dolt_server_full_test
|
|
--label //internal/storage/dolt:dolt_server_conformance_test
|
|
|
|
- name: Every target and shard ran tests (cmd-dolt)
|
|
id: check-cd
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
continue-on-error: true
|
|
timeout-minutes: 1
|
|
run: >-
|
|
python3 tools/bazel/check_testcases.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--label //cmd/bd:bd_dolt_server_test
|
|
--label //tests/regression:regression_test
|
|
|
|
# bazel-server-storage's own check (cmd-dolt has none): every
|
|
# full-suite shard ran exactly the tests its shard script lists, and
|
|
# neither it nor the conformance target only skips.
|
|
- name: Every listed test ran in its shard
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 2
|
|
run: >-
|
|
python3 tools/bazel/check_shard_coverage.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--suite //internal/storage/dolt:dolt_server_full_test .github/scripts/server-storage-test-shard.sh 16
|
|
--whole //internal/storage/dolt:dolt_server_conformance_test
|
|
|
|
# bazel-server-storage's own check, scoped with --label to the one
|
|
# target it has always budgeted: cmd-dolt's bd_dolt_server_test and
|
|
# regression_test shards are sharded too but no job budgeted them
|
|
# before the union, and this script has never judged them (see its
|
|
# docstring).
|
|
- name: Shard balance
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 2
|
|
run: >-
|
|
python3 tools/bazel/shard_budget.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--label //internal/storage/dolt:dolt_server_full_test
|
|
|
|
# failure() alone misses an ordinary test failure here: the `bazel
|
|
# test` step carries its own continue-on-error (must-fix 2), so a
|
|
# failing test does not make failure() true until the classify step
|
|
# (which runs after this one) exits non-zero. steps.test.outcome ==
|
|
# 'failure' catches that case directly (nit, PR #7535 review).
|
|
- name: Upload test logs
|
|
if: ${{ (failure() || steps.test.outcome == 'failure') && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 3
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: bazel-dolt-int-testlogs
|
|
path: |
|
|
bazel-testlogs/**/test.log
|
|
bazel-testlogs/**/test.xml
|
|
if-no-files-found: ignore
|
|
retention-days: 7
|
|
|
|
# Same redacted summary as bazel-test's; see its comment.
|
|
- name: CI analytics summary
|
|
id: ci-analytics
|
|
if: "!cancelled() && steps.test.outcome != 'skipped'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
env:
|
|
LANE: dolt-int
|
|
MODE: ${{ needs.rbe.outputs.mode }}
|
|
CHECK_RUN_ID: ${{ job.check_run_id }}
|
|
PR_HINT: ${{ github.event.pull_request.number || 0 }}
|
|
run: |
|
|
[ -f "$RUNNER_TEMP/bazel-bep.json" ] || exit 0
|
|
zstd --version || echo "::warning::zstd missing: no per-mnemonic data in this lane's summary"
|
|
python3 tools/bazel/ci_analytics_extract.py \
|
|
--lane "$LANE" --mode "$MODE" --check-run-id "$CHECK_RUN_ID" --pr-hint "$PR_HINT" \
|
|
--invocation test "$RUNNER_TEMP/bazel-bep.json" "$RUNNER_TEMP/bazel-exec.log.zst" "$RUNNER_TEMP/bazel-profile.json" \
|
|
--out "$RUNNER_TEMP/ci-analytics/ci-analytics.json"
|
|
|
|
- name: Upload CI analytics summary
|
|
if: "!cancelled() && steps.ci-analytics.outcome == 'success'"
|
|
continue-on-error: true
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: ci-analytics-dolt-int-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/ci-analytics/ci-analytics.json
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
# Last step: each tier's own result (not job.status: this job carries
|
|
# no continue-on-error, see the job's own comment) as the
|
|
# bazel-server-storage and bazel-cmd-dolt workflow_call outputs for
|
|
# pr.yml's ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too (steps.test.outcome stays "skipped" then, so the
|
|
# script's --bep file does not exist; it is only read when the test
|
|
# step actually ran). This step's own exit code decides job.status: it
|
|
# fails the job iff server-storage's tier failed, or `bazel test`'s
|
|
# exit code was neither 0 nor 3 (a build or infra failure, not
|
|
# attributable to one tier, fails both and the job).
|
|
- name: Classify tier results
|
|
id: classify
|
|
if: ${{ always() && steps.test.outcome != 'skipped' }}
|
|
timeout-minutes: 1
|
|
env:
|
|
TEST_RC: ${{ steps.test.outputs.rc }}
|
|
CHECK_SS_OUTCOME: ${{ steps.check-ss.outcome }}
|
|
CHECK_CD_OUTCOME: ${{ steps.check-cd.outcome }}
|
|
run: >-
|
|
python3 tools/bazel/classify_tier_results.py
|
|
--bep "$RUNNER_TEMP/bazel-bep.json"
|
|
--rc "$TEST_RC"
|
|
--tier server-storage=//internal/storage/dolt:dolt_server_full_test,//internal/storage/dolt:dolt_server_conformance_test
|
|
--tier cmd-dolt=//cmd/bd:bd_dolt_server_test,//tests/regression:regression_test
|
|
--gate-tier server-storage
|
|
--check "server-storage=$CHECK_SS_OUTCOME"
|
|
--check "cmd-dolt=$CHECK_CD_OUTCOME"
|
|
--github-output "$GITHUB_OUTPUT"
|
|
|
|
# F3: moved here from pr.yml so the gate depends on nothing but the rbe
|
|
# job's decision (not the whole bazel call). Only pr.yml opts in (package-gates: "on");
|
|
# bazel-farm.yml and nightly.yml keep the default "off" and this job does
|
|
# not run for them. Same-repo PRs get a Bazel-built bd (no --config=ci:
|
|
# only test:ci is defined in .bazelrc, so bazel build //cmd/bd:bd_for_tests
|
|
# is run directly); forks, Dependabot (whatever rbe-fork says: these gates
|
|
# take no fork certificate) and rbe-off/skip/cache runs fall back to
|
|
# `go build`, same as package-mcp.sh's own prepare_bd_binary() does when
|
|
# BEADS_TEST_BD_BINARY is unset. Hence mode == 'remote' below, not enabled
|
|
# (the fork modes are enabled too).
|
|
package-mcp:
|
|
name: Package Gate (MCP)
|
|
needs: rbe
|
|
# 8 vCPU with 16 xdist workers (BEADS_MCP_PYTEST_WORKERS below): pytest
|
|
# was 70s of this job at -n 8 on 4 vCPU (merge group 37721729916),
|
|
# dominated by per-test `bd init` fixture setup, which is CPU-bound.
|
|
# Was 4 vCPU / -n 8 (tools/f3 timing, 2026-10: ~2.7-3x on 4 CPUs);
|
|
# revert to that if the A/B does not save at least 15s.
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ inputs.package-gates == 'on' }}
|
|
timeout-minutes: 30
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
fetch-depth: 0
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
- name: Decide applicability
|
|
id: detect
|
|
env:
|
|
PR_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
|
|
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.event.merge_group.head_sha }}
|
|
PUSH_BEFORE_SHA: ${{ github.event.before }}
|
|
PUSH_AFTER_SHA: ${{ github.sha }}
|
|
run: ./scripts/ci/detect-package-gates.sh
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
|
|
# No --config=ci: only test:ci is defined in .bazelrc, and `bazel
|
|
# build --config=foo` fails when only test:foo exists. The explicit
|
|
# --@rules_go//go/config:race matches test:ci's top-level build
|
|
# setting (test:ci -> test:prcore), so bd_for_tests' race="off"
|
|
# transition lands in the same k8-fastbuild-ST-... output directory
|
|
# and action-cache key `bazel test --config=ci` already populated,
|
|
# instead of plain k8-fastbuild's; race itself still builds a non-race
|
|
# bd (verified with cquery: both configs cquery to the identical
|
|
# bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path).
|
|
- name: bazel build //cmd/bd:bd_for_tests
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
|
|
run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations
|
|
|
|
- name: Package the Bazel-built bd
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
|
|
run: |
|
|
./scripts/ci/package-bazel-bd.sh "$RUNNER_TEMP/bd-artifacts"
|
|
echo "BEADS_TEST_BD_BINARY=$RUNNER_TEMP/bd-artifacts/bd-linux-gms-pure" >> "$GITHUB_ENV"
|
|
# Defense in depth: nothing after this step needs the Bazel server
|
|
# or its RBE client key (uv sync and pytest run PR-controlled
|
|
# code). Shutting down also frees the JVM's memory for the 16
|
|
# xdist workers that follow on this 8 vCPU runner.
|
|
bazel shutdown
|
|
rm -rf "$RUNNER_TEMP/bazel-ci-secret"
|
|
|
|
- name: Set up Go
|
|
id: setup-go
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
cache: false
|
|
|
|
- name: Restore Go module cache
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ~/go/pkg/mod
|
|
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
|
|
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
|
|
|
|
- name: Verify Go modules
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
run: go mod verify
|
|
|
|
- name: Restore non-race Go build cache
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ${{ runner.temp }}/go-cache/non-race
|
|
key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }}
|
|
restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-
|
|
|
|
- name: Set up Python
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' }}
|
|
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
|
|
with:
|
|
python-version: '3.14'
|
|
|
|
- name: Install uv
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' }}
|
|
run: python -m pip install uv==0.11.16
|
|
|
|
- name: Run MCP package gate
|
|
if: ${{ steps.detect.outputs.mcp_package == 'true' }}
|
|
env:
|
|
GOCACHE: ${{ runner.temp }}/go-cache/non-race
|
|
# 16 workers on the 8 vCPU Blacksmith runner; package-mcp.sh's
|
|
# default 8 on the GitHub-hosted fallback.
|
|
BEADS_MCP_PYTEST_WORKERS: ${{ needs.rbe.outputs.mode == 'remote' && '16' || '8' }}
|
|
run: make ci-package-mcp
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too; package-gates == 'on' is the only skip condition
|
|
# (the job itself), so PACKAGE_MCP is never "skipped" on pr.yml.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# F3: npm's twin of package-mcp above. Structurally identical apart from
|
|
# the language setup (Node instead of Python/uv) and the `make` target.
|
|
package-npm:
|
|
name: Package Gate (npm)
|
|
needs: rbe
|
|
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
|
|
if: ${{ inputs.package-gates == 'on' }}
|
|
timeout-minutes: 30
|
|
outputs:
|
|
result: ${{ steps.result.outputs.result }}
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
ref: ${{ inputs.checkout-sha }}
|
|
fetch-depth: 0
|
|
persist-credentials: false
|
|
allow-unsafe-pr-checkout: ${{ inputs.fork-farm == 'authorized' && github.event_name == 'pull_request_target' }}
|
|
|
|
- name: Decide applicability
|
|
id: detect
|
|
env:
|
|
PR_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
|
|
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.event.merge_group.head_sha }}
|
|
PUSH_BEFORE_SHA: ${{ github.event.before }}
|
|
PUSH_AFTER_SHA: ${{ github.sha }}
|
|
run: ./scripts/ci/detect-package-gates.sh
|
|
|
|
- name: Set up Bazel
|
|
id: bazel
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
BAZEL_FORK_CACHE: ${{ needs.rbe.outputs.mode == 'cache' && 'true' || '' }}
|
|
|
|
# No --config=ci: only test:ci is defined in .bazelrc, and `bazel
|
|
# build --config=foo` fails when only test:foo exists. The explicit
|
|
# --@rules_go//go/config:race matches test:ci's top-level build
|
|
# setting (test:ci -> test:prcore), so bd_for_tests' race="off"
|
|
# transition lands in the same k8-fastbuild-ST-... output directory
|
|
# and action-cache key `bazel test --config=ci` already populated,
|
|
# instead of plain k8-fastbuild's; race itself still builds a non-race
|
|
# bd (verified with cquery: both configs cquery to the identical
|
|
# bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path).
|
|
- name: bazel build //cmd/bd:bd_for_tests
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
|
|
run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations
|
|
|
|
- name: Package the Bazel-built bd
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
|
|
run: |
|
|
./scripts/ci/package-bazel-bd.sh "$RUNNER_TEMP/bd-artifacts"
|
|
echo "BEADS_TEST_BD_BINARY=$RUNNER_TEMP/bd-artifacts/bd-linux-gms-pure" >> "$GITHUB_ENV"
|
|
# Defense in depth: nothing after this step needs the Bazel server
|
|
# or its RBE client key (npm install runs PR-controlled code).
|
|
bazel shutdown
|
|
rm -rf "$RUNNER_TEMP/bazel-ci-secret"
|
|
|
|
- name: Set up Go
|
|
id: setup-go
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
|
|
with:
|
|
go-version-file: 'go.mod'
|
|
cache: false
|
|
|
|
- name: Restore Go module cache
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ~/go/pkg/mod
|
|
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
|
|
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
|
|
|
|
- name: Verify Go modules
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
run: go mod verify
|
|
|
|
- name: Restore non-race Go build cache
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode != 'remote' }}
|
|
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
with:
|
|
path: ${{ runner.temp }}/go-cache/non-race
|
|
key: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-${{ github.sha }}
|
|
restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-non-race-
|
|
|
|
- name: Set up Node.js
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' }}
|
|
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6
|
|
with:
|
|
node-version: '24'
|
|
|
|
- name: Run npm package gate
|
|
if: ${{ steps.detect.outputs.npm_package == 'true' }}
|
|
env:
|
|
GOCACHE: ${{ runner.temp }}/go-cache/non-race
|
|
run: make ci-package-npm
|
|
|
|
# Last step: the job's result as a workflow_call output for pr.yml's
|
|
# ci-gate. always() runs it after a failure, a timeout or a
|
|
# cancellation too; package-gates == 'on' is the only skip condition
|
|
# (the job itself), so PACKAGE_NPM is never "skipped" on pr.yml.
|
|
- name: Record job result
|
|
id: result
|
|
if: ${{ always() }}
|
|
env:
|
|
JOB_STATUS: ${{ job.status }}
|
|
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
|
|
|
# The remote repo contents cache's only writer (ga-vnycm2.29, ported from
|
|
# gascity's bazel.yml). On a push to main, with the rbe job's rrc not off,
|
|
# run every lane's loading and analysis (.github/scripts/rrc-lane-commands.txt,
|
|
# --nobuild): only main's repository rules and Starlark run (gazelle's
|
|
# fetch_repo, the Go SDK and LLVM downloads, extraction: what every lane
|
|
# already runs on its runner), never a build or test action. Bazel uploads
|
|
# each reproducible repository's tree (--remote_upload_local_results) with
|
|
# a 30-minute rbe-rrc-writer-beads certificate that rbe-west's mint signs
|
|
# only for this workflow on a push to main
|
|
# (.github/scripts/rrc-writer-credential.sh); rbe-west's rrc-gate admits
|
|
# repo-contents entries from it and refuses every other action cache
|
|
# write, and the certificate cannot Execute. Cache only (--remote_cache,
|
|
# no executor). Not a lane: no result output, never in pr.yml's gate
|
|
# (which never sees it anyway: push only), so a failed seed only leaves the
|
|
# lanes fetching. setup-bazel runs in local mode (no secrets): its rc, the
|
|
# runner cache and the pinned Bazel, nothing else. The only bazel.yml job
|
|
# with id-token: write; every caller must grant it (GitHub checks a called
|
|
# workflow's job permissions at startup, even for a job that skips), and
|
|
# the mint refuses any token but a main-push bazel.yml one.
|
|
rrc-seed:
|
|
name: repo contents cache seed
|
|
needs: rbe
|
|
if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }}
|
|
runs-on: blacksmith-8vcpu-ubuntu-2404
|
|
timeout-minutes: 30
|
|
permissions:
|
|
contents: read
|
|
id-token: write # the rbe-rrc-writer-beads certificate (GitHub OIDC)
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
- name: Set up Bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
- name: rbe-rrc-writer certificate
|
|
id: writer
|
|
env:
|
|
BAZEL_CI_SECRET_DIR: ${{ runner.temp }}/bazel-ci-secret
|
|
run: bash .github/scripts/rrc-writer-credential.sh
|
|
- name: Seed the remote repo contents cache
|
|
if: ${{ steps.writer.outputs.cert != '' }}
|
|
env:
|
|
CERT: ${{ steps.writer.outputs.cert }}
|
|
KEY: ${{ steps.writer.outputs.key }}
|
|
ENDPOINT: ${{ steps.writer.outputs.endpoint }}
|
|
INSTANCE: ${{ steps.writer.outputs.instance }}
|
|
run: >-
|
|
bash .github/scripts/rrc-run-lanes.sh --experimental_remote_repo_contents_cache --
|
|
--loading_phase_threads=64
|
|
"--remote_cache=$ENDPOINT" "--remote_instance_name=$INSTANCE"
|
|
"--tls_client_certificate=$CERT" "--tls_client_key=$KEY"
|
|
--remote_upload_local_results
|
|
- name: Remove the writer key
|
|
if: ${{ always() }}
|
|
run: rm -f -- "$RUNNER_TEMP/bazel-ci-secret/rrc-writer.key" "$RUNNER_TEMP/bazel-ci-secret/rrc-writer.crt"
|
|
|
|
# Nightly check of the remote repo contents cache (gascity design R1/R2). A
|
|
# cached repository tree is trusted by every reader, and nothing on the
|
|
# reader checks it against what its rule produces. So, from nightly.yml (or
|
|
# a dispatch), fetch every lane's repositories cold (rrc-lane-commands.txt
|
|
# with --nobuild and no repo contents cache, Bazel set up exactly as a
|
|
# lane) and compare each repository's tree and marker with the cached
|
|
# entry a lane would read for the same key (tools/bazel/rrc_verify.py,
|
|
# reads only, with the lanes' certificate and the endpoint from
|
|
# setup-bazel's rc: no secret is referenced outside setup-bazel). A
|
|
# mismatch is a poisoned entry or a rule falsely marked reproducible: the
|
|
# job fails and opens (or comments on) an rrc-verify issue. Not a lane:
|
|
# never gated. The only bazel.yml job with issues: write; every caller
|
|
# must grant it, as for rrc-seed's id-token.
|
|
rrc-verify:
|
|
name: repo contents cache verify
|
|
needs: rbe
|
|
if: ${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && needs.rbe.outputs.mode == 'remote' && needs.rbe.outputs.rrc != 'off' }}
|
|
runs-on: blacksmith-8vcpu-ubuntu-2404
|
|
timeout-minutes: 45
|
|
permissions:
|
|
contents: read
|
|
issues: write # the rrc-verify alert issue
|
|
steps:
|
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
with:
|
|
persist-credentials: false
|
|
- name: Set up Bazel
|
|
uses: ./.github/actions/setup-bazel
|
|
env:
|
|
BAZEL_REMOTE_EXECUTOR: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_WEST_EXECUTOR || '' }}
|
|
RBE_TLS_CERT: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CERT || '' }}
|
|
RBE_TLS_KEY: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_KEY || '' }}
|
|
RBE_TLS_CA: ${{ needs.rbe.outputs.mode == 'remote' && secrets.RBE_TLS_CA || '' }}
|
|
RBE_INSTANCE: ${{ needs.rbe.outputs.mode == 'remote' && 'oss' || '' }}
|
|
- name: Cold fetch (every lane, no repo contents cache)
|
|
run: bash .github/scripts/rrc-run-lanes.sh -- --loading_phase_threads=64
|
|
- name: Compare with the cache
|
|
id: verify
|
|
run: |
|
|
set -euo pipefail
|
|
secret_dir="$RUNNER_TEMP/bazel-ci-secret"
|
|
prefix='build:remote-exec --remote_executor='
|
|
endpoint=$(grep -F -m1 "$prefix" "$secret_dir/ci.bazelrc" || true)
|
|
endpoint=${endpoint#"$prefix"}
|
|
[ -n "$endpoint" ] || { echo "::error::setup-bazel's rc names no executor"; exit 1; }
|
|
python3 tools/bazel/rrc_verify.py \
|
|
--output-base "$(bazel info output_base)" \
|
|
--endpoint "$endpoint" --instance oss \
|
|
--cert "$secret_dir/client.crt" --key "$secret_dir/client.key" \
|
|
--bazel-version "$(tr -d '[:space:]' < .bazelversion)" \
|
|
--json "$RUNNER_TEMP/rrc-verify.json" --summary "$GITHUB_STEP_SUMMARY"
|
|
- name: Upload the report
|
|
if: ${{ always() && hashFiles(format('{0}/rrc-verify.json', runner.temp)) != '' }}
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
|
|
with:
|
|
overwrite: true # a job re-run is the same run: replace, not 409
|
|
name: rrc-verify-${{ github.run_attempt }}
|
|
path: ${{ runner.temp }}/rrc-verify.json
|
|
retention-days: 30
|
|
- name: Alert (rrc-verify issue)
|
|
if: ${{ failure() && steps.verify.outcome == 'failure' }}
|
|
env:
|
|
GH_TOKEN: ${{ github.token }}
|
|
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
|
run: |
|
|
set -euo pipefail
|
|
title="rrc-verify: remote repo contents cache differs from a cold fetch"
|
|
body="The nightly rrc-verify run found cached repository trees that differ from a cold fetch, or could not check them: $RUN_URL (summary and the rrc-verify report artifact).
|
|
|
|
A mismatch is a poisoned entry or a repository rule falsely marked reproducible (gascity engdocs/design/bazel-remote-repo-contents-cache.md, R1/R2). Contain the fork-cache readers first: they see no repository variables, so (1) commit fork_rrc_read=off in both probe scripts, beads .github/actions/setup-bazel/cache-rrc-probe.sh and gascity tools/rbe/cache-rrc-probe.sh (beads has no fresh-merge: an open fork PR keeps the old switch until it is pushed again); (2) for immediate server-side containment, an operator runs rbe-cache-gate close on core, which stops all fork-cache, and fails fork-cache lanes already reading; (3) on a mismatch, set RBE_REPO_CONTENTS_CACHE to seed: the trusted readers read the same entries (rbe-cache serves the oss action cache), and seed stops every one of them while rrc-seed and this check keep running; set it to off only to stop seeding too, which also stops rrc-verify; if this run only says it could not check, leave the variable as it is. Then follow gascity/infra nativelink-cas/west/README.md (remote repo contents cache writer, poisoned entries)."
|
|
gh label create rrc-verify -R "$GITHUB_REPOSITORY" --color B60205 \
|
|
--description "Remote repo contents cache verification failures" 2>/dev/null || true
|
|
n=$(gh issue list -R "$GITHUB_REPOSITORY" --label rrc-verify --state open --json number --jq '.[0].number // empty')
|
|
if [ -n "$n" ]; then
|
|
gh issue comment "$n" -R "$GITHUB_REPOSITORY" --body "Again: $RUN_URL"
|
|
else
|
|
gh issue create -R "$GITHUB_REPOSITORY" --label rrc-verify --title "$title" --body "$body"
|
|
fi
|