fix(bazel): one nogo-validating lane per race configuration (F5 S1) (#7479)

* fix(bazel): one nogo-validating lane per race configuration (F5 S1)

The race configuration (--@rules_go//go/config:race) is built by five
lanes that duplicate the same RunNogo/ValidateNogo work: bazel-test
(--config=ci), bazel-embedded, bazel-doltserver, bazel-proxied, and
the advisory bazel-dolt-race shadow, plus the package-mcp/package-npm
gates' explicit race build. --norun_validations is a build-request
option, not a configuration flag (proven here by aquery: GoCompilePkg/
GoLink/RunNogo/ValidateNogo action keys are byte-identical with and
without it), so passing it on every lane but bazel-test removes the
duplicate work without touching a single action key or dropping any
lint coverage: bazel-test's --config=ci already validates every file
any of the others compiles.

scripts/nogo_lint_policy_test.go replaces the old blanket
"--norun_validations never appears in .bazelrc" check with
TestNogoConfigurationsHaveOneValidatingLane (T1: one owner per
configuration fingerprint), TestNogoOwnersAreRequired (T2: the owner
is a required CI Gate check whose `if:` covers every non-owner's),
TestNogoRaceOwnerBuildsEverything (T3: the owner builds //... whole,
no exclusions) and TestNoUnconditionalValidationOff (T5: nothing else
may disable validation). tools/bazel/nogo_ownership.py (make
nogo-ownership) is a dynamic, advisory aquery cross-check of the same
property, run on demand rather than from inside a Bazel action.

Verified locally: aquery shows identical action keys for the affected
lanes with/without --norun_validations; a planted printf-format vet
finding in internal/idgen/hash.go failed the owner build and passed
the non-owner build (then reverted, never committed);
scripts/bazel_dolt_race_shadow_test.go and the full //scripts suite
stay green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): close S1 nogo-ownership review gaps (allowlist, owner pin, real aquery)

Addresses maintainer-city-c3's review of #7479:

1. T6 TestRunValidationsOnlyOnAllowlistedLines replaces the coverage gap
   left when main's blanket run_validations ban was dropped: only the four
   race-group non-owners' "test:<cfg> --norun_validations" lines in
   .bazelrc, and the two package-gate bazel-build steps in bazel.yml, are
   permitted. A later test:integration/doltserver-cmd --norun_validations
   (out of T1's table, S2) now fails here instead of landing silently.
   Verified by mutation: adding test:integration --norun_validations fails
   the new test; reverting passes.

3. TestNogoRaceOwnerBuildsEverything (T3) now also pins that the race
   owner's "bazel test //... --config=ci" step has no step-level if: or
   continue-on-error, and that its job has none either. Verified by
   mutation: adding if: ${{ always() }} to that step fails; reverting
   passes.

2. tools/bazel/nogo_ownership.py's aquery comparison was querying the
   identical mnemonic("RunNogo", //...) for the owner and every non-owner,
   so `missing` could never be non-empty. It now derives each non-owner's
   actual built target set the way --build_tests_only + --test_tag_filters
   does: the deps closure of tests(//...) filtered to targets carrying one
   of the lane's tags, read exactly from `query --output=xml` (not attr()'s
   substring regex match, which would wrongly fold "dolt-server-proxied"
   and "dolt-server-cmd" into a "dolt-server" filter). Confirmed against a
   real (offline, --nohome_rc) bazel run: reports OK at head, and reports
   100+ missing targets when the owner's query is artificially narrowed to
   a subpackage -- it can now actually fail.

Also: refreshed a stale .bazelrc comment claiming nogo runs beside every
lane's compile, which S1 made untrue for race-group non-owners.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-10-10 18:05:17 +00:00 • committed by GitHub
commit b8e6ba1791
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 753 additions and 33 deletions

View file

@ -104,6 +104,16 @@ test:embedded --remote_download_regex=.*/test\.xml$
# jobs, which never retried. Evicted inputs fail the step instead; a
# visible "Re-run failed jobs" retries it.
test:embedded --experimental_remote_cache_eviction_retries=0
# The race configuration's nogo validations run once, in bazel.yml's `test`
# lane (`bazel test //... --config=ci`), which is a strict superset of every
# race-group lane's RunNogo/ValidateNogo keys. --norun_validations is a
# build-request option, not a configuration flag: it changes no action key
# (compiles, links and test PASSes are shared with `test` through the
# action cache) and discards no analysis; it only skips re-running nogo's
# validation actions that the owner lane already runs on this configuration.
# scripts/nogo_lint_policy_test.go's TestNogoConfigurationsHaveOneValidatingLane
# pins `test` as this configuration's sole validating lane.
test:embedded --norun_validations
# Dolt-server lane: what pr.yml's retired container-backed jobs ("Test
# (storage domain + uow)" and "Contract corpus") ran, but every dolt
@ -127,6 +137,9 @@ test:doltserver --test_arg=-test.parallel=4
# runner shape; remote workers are not limited by it.
test:doltserver --local_test_jobs=4
test:doltserver --remote_download_regex=.*/test\.(log|xml)$
# Race-group nogo is owned by the `test` lane; see test:embedded's comment
# above.
test:doltserver --norun_validations
# Proxied-server tier: PR Risk's "Test (Proxied Dolt Cmd N/15)" jobs, as
# //cmd/bd:bd_proxied_test and //cmd/bd:bd_managed_local_test (the
@ -160,6 +173,9 @@ test:doltserver-proxied --remote_download_regex=.*/test\.(log|xml)$
# jobs, which never retried. Evicted inputs fail the step instead; a
# visible "Re-run failed jobs" retries it.
test:doltserver-proxied --experimental_remote_cache_eviction_retries=0
# Race-group nogo is owned by the `test` lane; see test:embedded's comment
# above.
test:doltserver-proxied --norun_validations
# dolt-server + proxied union (item 5, Phase 1 shadow): bazel.yml's advisory
# bazel-dolt-race job runs both tiers in one invocation, beside their own
@ -186,6 +202,10 @@ test:dolt-race --test_arg=-test.parallel=4
test:dolt-race --local_test_jobs=4
test:dolt-race --remote_download_regex=.*/test\.(log|xml)$
test:dolt-race --experimental_remote_cache_eviction_retries=0
# The union carries its members' --norun_validations line
# (TestBazelDoltRaceConfigIsMemberUnion needs no change: both members and the
# union carry the same line).
test:dolt-race --norun_validations
# Server-Dolt storage tier: PR Risk's "Test (Server Dolt Conformance)" and
# "Test (Server Dolt Full Suite N/16)" jobs, as
@ -320,11 +340,15 @@ build:js-wasm --use_target_platform_for_tests
build:js-wasm --remote_download_outputs=toplevel
# nogo (//tools/nogo: go test's vet checks plus the golangci-lint linters
# .golangci.yml enables) runs beside every Go compile of every lane, so any
# `bazel build`/`bazel test` fails on a finding. --config=nogo asks for the
# analysis alone (the nogo_fix output group; nothing is linked), in the race
# configuration bazel.yml's unit lane (test:ci) builds, so a local run reuses
# CI's remote cache. make lint, scripts/pr-lint and the pre-commit hook use it.
# .golangci.yml enables) runs beside every Go compile, so `bazel build`/
# `bazel test` fails on a finding -- in bazel.yml's one validating (owner)
# lane per configuration (since F5 S1: a race-group non-owner lane reuses
# its owner's nogo results instead of re-running them; see
# scripts/nogo_lint_policy_test.go's nogoConfigurations table).
# --config=nogo asks for the analysis alone (the nogo_fix output group;
# nothing is linked), in the race configuration bazel.yml's unit lane
# (test:ci) builds, so a local run reuses CI's remote cache. make lint,
# scripts/pr-lint and the pre-commit hook use it.
build:nogo --@rules_go//go/config:race
build:nogo --keep_going
build:nogo --output_groups=nogo_fix

View file

@ -2543,7 +2543,7 @@ jobs:
# bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path).
- name: bazel build //cmd/bd:bd_for_tests
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$'
run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations
- name: Package the Bazel-built bd
if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
@ -2665,7 +2665,7 @@ jobs:
# bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path).
- name: bazel build //cmd/bd:bd_for_tests
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode == 'remote' }}
run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$'
run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations
- name: Package the Bazel-built bd
if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode == 'remote' }}

View file

@ -46,7 +46,7 @@ endif
.PHONY: all build doctor-build test test-icu-path test-full-cgo test-regression test-upgrade test-cross-version test-migration corpus-regen githooks-regen bench bench-quick clean clean-test-tmp install install-force help check-up-to-date fmt fmt-check check-testing-short
.PHONY: lint lint-changed vet
.PHONY: check check-go test-go check-docs-go
.PHONY: ci-pr-core ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm
.PHONY: ci-pr-core ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm nogo-ownership
.PHONY: api-gen api-check
.PHONY: bazel-sync bazel-sync-check
@ -237,6 +237,16 @@ ci-pr-core:
ci-pr-lint:
@./scripts/ci/pr-lint.sh
# nogo-ownership is a dynamic, advisory cross-check (not part of the required
# //scripts static suite, which cannot shell out to Bazel): it asks real
# `bazel aquery` whether the nogo owner lane for each configuration group
# (today: race, bazel-test/--config=ci) really does validate every file a
# non-owner lane (embedded, doltserver, doltserver-proxied, dolt-race, which
# pass --norun_validations) compiles. See tools/bazel/nogo_ownership.py and
# scripts/nogo_lint_policy_test.go's nogoConfigurations table.
nogo-ownership:
@python3 tools/bazel/nogo_ownership.py
# The CI lint gate: native, plus //tools/bazel:release_cross for windows/amd64
# and darwin/arm64 (BD_LINT_TARGETS selects; scripts/pr-lint).
lint: ci-pr-lint

View file

@ -1025,10 +1025,15 @@ Required` requires them to have run remotely and passed.
the job it replaces.
- `go test`'s own vet checks (cmd/go's `defaultVetFlags`, policy-tested
equal to the toolchain's) and the golangci-lint linters `.golangci.yml`
enables run as nogo (`//tools/nogo`) beside every compile of every Bazel
lane: natively in `bazel test //... --config=ci`, and for every release
platform in the `bazel-release-cross` lane
(engdocs/LINTING.md). The former
enables run as nogo (`//tools/nogo`): natively in `bazel test //...
--config=ci`, and for every release platform in the
`bazel-release-cross` lane (engdocs/LINTING.md). Each Bazel configuration
validates in exactly one required lane, not beside every compile of
every lane: the race configuration's nogo is owned by `test` above, and
`embedded`, `doltserver`, `doltserver-proxied`, `dolt-race` (shadow) and
the package gates pass `--norun_validations`, since they compile a
strict subset of what `test` already validates on that configuration
(`scripts/nogo_lint_policy_test.go`, engdocs/LINTING.md). The former
`scripts-go-checks` (`Go checks (vet)`) and `pr-lint-wrapper`
(`PR Lint (native|windows|darwin)`) jobs are retired.
- The repository policy tests (`./scripts/...`, including the D2 guards)

View file

@ -49,8 +49,14 @@ Where it gates:
validates each of those compiles. So `//go:build windows`, `darwin`,
`!linux` (and freebsd, android, arm64) files are analyzed from Linux: a
superset of golangci-lint's former `GOOS=windows`/`GOOS=darwin` legs.
- **Every other Bazel lane** (integration, embedded, dolt-server) validates
what it compiles, including files only its build tags select.
- The race configuration's nogo validations run once, owned by the `test`
lane above: `embedded`, `doltserver`, `doltserver-proxied`, `dolt-race`
(shadow) and the package gates (`package-mcp`, `package-npm`) all pass
`--norun_validations`, since each compiles a strict subset of what `test`
already validates on that configuration (`scripts/nogo_lint_policy_test.go`'s
`TestNogoConfigurationsHaveOneValidatingLane` pins this).
- **Every other Bazel lane** (integration) still validates what it compiles,
including files only its build tags select.
Every PR is checked against the whole tree, not only its diff: the tree has
no findings, so there is no baseline to scope against.

View file

@ -4310,6 +4310,8 @@ var bazelEmbeddedRCLines = []string{
"test:embedded --test_env=GO_TEST_WRAP_TESTV=1",
"test:embedded --remote_download_regex=.*/test\\.xml$",
"test:embedded --experimental_remote_cache_eviction_retries=0",
// F5 S1: the race configuration's nogo is owned by the test lane.
"test:embedded --norun_validations",
}
func TestBazelEmbeddedJobRunsEmbeddedTier(t *testing.T) {

View file

@ -4,6 +4,8 @@ import (
"os"
"path/filepath"
"regexp"
"slices"
"sort"
"strings"
"testing"
)
@ -15,6 +17,18 @@ import (
// platforms compile inside the pure lane's release cross-compile
// (//tools/bazel:release_cross, every release platform). These pin that
// wiring and that golangci-lint is gone from CI, hooks and make.
//
// nogo validation is duplicated across lanes that compile the same Bazel
// configuration (race: test, embedded, doltserver, doltserver-proxied,
// dolt-race and the package gates). TestLintAndVetRunAsNogo used to forbid
// `run_validations` anywhere in .bazelrc ("every lane validates"); that
// blanket ban is replaced by TestNogoConfigurationsHaveOneValidatingLane
// (T1), TestNogoOwnersAreRequired (T2), TestNogoRaceOwnerBuildsEverything
// (T3), TestNoUnconditionalValidationOff (T5) and
// TestRunValidationsOnlyOnAllowlistedLines (T6), which together pin "every
// configuration's nogo runs in exactly one required lane" instead.
// --norun_validations is a build-request option, not a configuration flag:
// it changes no action key and discards no analysis.
// nogoBazelrcLines are .bazelrc's nogo configs, exactly.
var nogoBazelrcLines = []string{
@ -38,24 +52,6 @@ func TestLintAndVetRunAsNogo(t *testing.T) {
if !equalStrings(got, nogoBazelrcLines) {
t.Errorf(".bazelrc nogo configs = %q, want %q", got, nogoBazelrcLines)
}
// Every lane validates: a lane that skipped nogo would let findings in
// the files only it compiles (integration-tagged, pure) through.
if strings.Contains(rc, "run_validations") {
t.Error(".bazelrc turns nogo validation off for some lane")
}
// The cross-platform half: the pure lane's release cross-compile builds
// every library and binary for every release platform, and nogo
// validates each compile (TestReleaseCrossCompileRunsInBazelPureLane
// pins the step). Nothing on that path may opt out of validation.
if script := readPolicyFile(t, root, "scripts/ci/bazel-release-cross-compile.sh"); strings.Contains(script, "run_validations") {
t.Error("scripts/ci/bazel-release-cross-compile.sh turns nogo validation off")
}
for _, line := range strings.Split(rc, "\n") {
if strings.HasPrefix(strings.TrimSpace(line), "build:release-cross") && strings.Contains(line, "validation") {
t.Errorf(".bazelrc %q changes validation for the release cross-compile", line)
}
}
// No workflow installs or runs golangci-lint.
entries, err := os.ReadDir(filepath.Join(root, ".github", "workflows"))
@ -107,3 +103,399 @@ func TestLintAndVetRunAsNogo(t *testing.T) {
t.Error("scripts/ci/pr-lint.sh runs gofmt; //scripts/repochecks:fmt_test gates formatting")
}
}
// --- F5 S1: one validating lane per nogo configuration -----------------
// nogoKeyFlagPattern matches the build-affecting flags f5-spec.md §6 T1
// calls out as fingerprint-relevant: they select a distinct Bazel
// configuration (and so a distinct set of nogo action keys). Flags like
// --test_tag_filters, --keep_going or --test_arg narrow which tests run but
// do not change what is compiled, so they are not in the fingerprint.
var nogoKeyFlagPattern = regexp.MustCompile(`--@rules_go//go/config:(race|pure|tags=\S+)|--platforms=\S+|//tools/bazel:release_platforms\S*`)
// expandBazelrcConfig returns every "<cmd> <opt>" line (as bazelRCConfigLines
// formats them) that --config=name pulls in, recursively following any
// nested --config=Y reference within those lines (as .bazelrc's own comments
// describe, e.g. test:ci --config=prcore). seen prevents infinite recursion
// on a cycle.
func expandBazelrcConfig(rc, name string, seen map[string]bool) []string {
if seen[name] {
return nil
}
seen[name] = true
var out []string
for _, l := range bazelRCConfigLines(rc, name) {
if v, ok := strings.CutPrefix(l, "build --config="); ok {
out = append(out, expandBazelrcConfig(rc, v, seen)...)
continue
}
if v, ok := strings.CutPrefix(l, "test --config="); ok {
out = append(out, expandBazelrcConfig(rc, v, seen)...)
continue
}
out = append(out, l)
}
return out
}
// nogoFingerprint reduces a config's expanded lines to its key-affecting
// flags, sorted and deduplicated, so two configs that build the same thing
// compare equal regardless of line order or non-key-affecting flags.
func nogoFingerprint(lines []string) []string {
seen := map[string]bool{}
var out []string
for _, l := range lines {
for _, m := range nogoKeyFlagPattern.FindAllString(l, -1) {
if !seen[m] {
seen[m] = true
out = append(out, m)
}
}
}
sort.Strings(out)
return out
}
// bazelrcConfigSkipsValidations reports whether --config=name's own
// (non-recursive) .bazelrc lines pass --norun_validations. Non-recursive
// because each non-owner config sets the flag itself (D1); inheriting it
// through a nested --config would make the policy table here miss a
// config that silently stopped validating by referencing one that did.
func bazelrcConfigSkipsValidations(rc, name string) bool {
for _, l := range bazelRCConfigLines(rc, name) {
if strings.Contains(l, "--norun_validations") {
return true
}
}
return false
}
// nogoConfigurations: f5-spec.md §6 T1's table. Each row is one Bazel
// configuration (one nogo fingerprint): the .bazelrc --config names (or, for
// the package gates, a pseudo-config checked against their pinned command
// line) that build it, the bazel.yml job that owns validating it, and the
// jobs that must pass --norun_validations instead.
//
// Only the race group is classified here (F5 S1). The race+integration
// group (integration, doltserver-integration, doltserver-cmd) and `pure`'s
// optional D4a are later slices (f5-spec.md §5 S2, S3): every lane in them
// still validates today, so they are intentionally left out of this table
// rather than added as a row that TestNogoConfigurationsHaveOneValidatingLane
// could never satisfy. `pure`, js/wasm and release-cross are unaffected by
// any slice (single-lane configurations: there is no other lane to
// deduplicate against), so they are included as trivial one-member rows for
// completeness and as a guard against a future lane quietly joining them.
var nogoConfigurations = []struct {
name string
configs []string // .bazelrc --config names sharing this fingerprint, owner first
owner string // bazel.yml job name required to validate
nonOwners []string // bazel.yml job names required to pass --norun_validations
}{
{
name: "race",
configs: []string{"ci", "embedded", "doltserver", "doltserver-proxied", "dolt-race"},
owner: bazelJobName,
nonOwners: []string{bazelEmbedJobName, bazelDoltJobName, bazelProxiedJobName, bazelDoltRaceJobName},
},
{
name: "pure",
configs: []string{"pure"},
owner: bazelPureJobName,
},
{
name: "js-wasm",
configs: []string{"js-wasm"},
owner: bazelPureJobName,
},
}
// nogoPackageGateJobs: the package gates do not select a named .bazelrc
// --config; their `bazel build` command line spells the race flag out
// (pr_lanes_bazel_coverage_test.go pins the exact command). They are race
// non-owners, checked against the "race" row's fingerprint and
// --norun_validations requirement directly from that pinned command.
var nogoPackageGateJobs = []string{bazelPackageMCPJobName, bazelPackageNPMJobName}
// T1 TestNogoConfigurationsHaveOneValidatingLane: every classified Bazel
// configuration has exactly one validating (required) lane; every lane that
// builds the same configuration passes --norun_validations instead; and
// every config within a row shares that row's fingerprint, so a key-affecting
// flag added to a non-owner (which would split it into a different, now
// unvalidated, configuration) fails here instead of silently losing
// coverage.
func TestNogoConfigurationsHaveOneValidatingLane(t *testing.T) {
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
workflow := readCIWorkflow(t, bazelWorkflowName)
seenConfig := map[string]string{} // config -> row name, to catch a config in two rows
for _, row := range nogoConfigurations {
if row.owner == "" {
t.Fatalf("nogoConfigurations row %q has no owner", row.name)
}
if len(row.configs) == 0 {
t.Fatalf("nogoConfigurations row %q has no configs", row.name)
}
ownerFP := nogoFingerprint(expandBazelrcConfig(rc, row.configs[0], map[string]bool{}))
for _, c := range row.configs {
if prev, ok := seenConfig[c]; ok {
t.Errorf("--config=%s appears in both row %q and row %q", c, prev, row.name)
}
seenConfig[c] = row.name
fp := nogoFingerprint(expandBazelrcConfig(rc, c, map[string]bool{}))
if !sameStringSet(fp, ownerFP) {
t.Errorf("row %q: --config=%s fingerprint %v != owner %s's %v (a key-affecting flag split this config out of the row)",
row.name, c, fp, row.configs[0], ownerFP)
}
}
ownerJob := workflow.job(t, row.owner)
if ownerJob.ContinueOnError {
t.Errorf("row %q owner %s has continue-on-error; a skipped finding there would never fail anything", row.name, row.owner)
}
if bazelrcConfigSkipsValidations(rc, row.configs[0]) {
t.Errorf("row %q owner config --config=%s passes --norun_validations; it is the row's only validating lane", row.name, row.configs[0])
}
nonOwners := append([]string{}, row.nonOwners...)
if row.name == "race" {
nonOwners = append(nonOwners, nogoPackageGateJobs...)
}
for i, job := range nonOwners {
if job == row.owner {
t.Errorf("row %q lists its own owner %s as a non-owner", row.name, job)
}
isPackageGate := slices.Contains(nogoPackageGateJobs, job)
if isPackageGate {
// The package gates spell their flags out on the command
// line rather than through a named --config
// (pr_lanes_bazel_coverage_test.go pins the exact string);
// confirmed here against their job's steps instead of
// .bazelrc.
step := workflow.job(t, job).step(t, "bazel build //cmd/bd:bd_for_tests")
if !strings.Contains(step.Run, "--@rules_go//go/config:race") {
t.Errorf("row %q non-owner %s's build step does not build the race configuration: %q", row.name, job, step.Run)
}
if !strings.Contains(step.Run, "--norun_validations") {
t.Errorf("row %q non-owner %s's build step does not pass --norun_validations", row.name, job)
}
continue
}
// Map job -> its own --config, which is row.configs[1+i'] in
// declaration order (owner's config is row.configs[0]).
idx := i + 1
if idx >= len(row.configs) {
t.Fatalf("row %q has more non-owner jobs than configs (idx %d, configs %v)", row.name, idx, row.configs)
}
cfg := row.configs[idx]
if !bazelrcConfigSkipsValidations(rc, cfg) {
t.Errorf("row %q non-owner %s (--config=%s) does not pass --norun_validations; it duplicates %s's nogo", row.name, job, cfg, row.owner)
}
}
}
}
// T2 TestNogoOwnersAreRequired: each owner job is required by pr.yml's CI
// Gate (or, for a flag-gated lane, the flag that is pinned true) with no
// job-level continue-on-error, and its `if:` runs at least everywhere its
// non-owners' do — otherwise a mode could run a non-owner's compile with no
// owner validating it at all (f5-spec.md §9).
func TestNogoOwnersAreRequired(t *testing.T) {
workflow := readCIWorkflow(t, bazelWorkflowName)
gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate")
gateEnv := gate.step(t, "Evaluate CI gate").Env
required := strings.Fields(gateEnv["CI_GATE_REQUIRED"])
ownerGateToken := map[string]string{
bazelJobName: "BAZEL_TEST",
bazelPureJobName: "BAZEL_PURE",
}
events := []string{"pull_request", "merge_group", "pull_request_target", "push", "schedule", "workflow_dispatch"}
for _, row := range nogoConfigurations {
ownerJob := workflow.job(t, row.owner)
if token, ok := ownerGateToken[row.owner]; ok && !contains(required, token) {
t.Errorf("row %q owner %s: CI_GATE_REQUIRED %v lacks %s", row.name, row.owner, required, token)
}
if ownerJob.ContinueOnError {
t.Errorf("row %q owner %s has continue-on-error", row.name, row.owner)
}
nonOwners := append([]string{}, row.nonOwners...)
if row.name == "race" {
nonOwners = append(nonOwners, nogoPackageGateJobs...)
}
for _, job := range nonOwners {
if slices.Contains(nogoPackageGateJobs, job) {
// The package gates' bazel build only ever runs in mode
// remote (TestPackageGateJobs pins the surrounding `if:`),
// and the owner's `if:` (mode != 'skip') is true whenever
// mode == 'remote': "remote" is never "skip".
step := workflow.job(t, job).step(t, "bazel build //cmd/bd:bd_for_tests")
if !strings.Contains(step.If, "mode == 'remote'") {
t.Errorf("%s's bazel-build step if %q no longer implies the owner's mode != 'skip'", job, step.If)
}
continue
}
nonOwnerJob := workflow.job(t, job)
for _, mode := range bazelRBEModes {
for _, event := range events {
ctx := map[string]string{
"needs.rbe.outputs.mode": mode,
"needs.rbe.outputs.enabled": bazelModeEnabled(mode),
"github.event_name": event,
}
if evalRRCIf(t, nonOwnerJob.If, ctx) && !evalRRCIf(t, ownerJob.If, ctx) {
t.Errorf("row %q: %s runs in mode=%s event=%s but owner %s does not (owner if %q, non-owner if %q)",
row.name, job, mode, event, row.owner, ownerJob.If, nonOwnerJob.If)
}
}
}
}
}
}
// T3 TestNogoRaceOwnerBuildsEverything: the race owner's target set covers
// every file any race-group lane compiles (f5-spec.md §2.1): its test step
// runs over the whole tree with no "-//" exclusion, and test:ci's expansion
// narrows no target selection (--build_tests_only, --build_tag_filters)
// — only test execution (--test_tag_filters etc., which T1's fingerprint
// deliberately ignores).
func TestNogoRaceOwnerBuildsEverything(t *testing.T) {
workflow := readCIWorkflow(t, bazelWorkflowName)
ownerJob := workflow.job(t, bazelJobName)
test := ownerJob.step(t, "bazel test //... --config=ci")
if !strings.Contains(test.Run, "bazel test //... --config=ci") {
t.Fatalf("%s step does not run bazel test //... --config=ci:\n%s", bazelJobName, test.Run)
}
if regexp.MustCompile(`bazel test[^\n]*\s-//`).MatchString(test.Run) {
t.Errorf("%s excludes a target (-//...) from //...: it would stop validating what it excludes", bazelJobName)
}
// The race owner's validating step must be unconditional: a step-level
// `if:` or continue-on-error here would let the suite report green while
// race-group nogo silently did not run, with nothing else in this file
// (or T1/T2, which only check job-level fields) noticing.
if test.If != "" {
t.Errorf("%s step %q has if %q; the race owner's validating step must be unconditional", bazelJobName, test.Name, test.If)
}
if test.ContinueOnError != nil && test.ContinueOnError != false {
t.Errorf("%s step %q has continue-on-error %v; a failed race-group nogo finding there would never fail the job", bazelJobName, test.Name, test.ContinueOnError)
}
if ownerJob.ContinueOnError {
t.Errorf("%s job has continue-on-error; the race owner's job must be able to fail", bazelJobName)
}
rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc")
for _, l := range expandBazelrcConfig(rc, "ci", map[string]bool{}) {
if strings.Contains(l, "build_tests_only") || strings.Contains(l, "build_tag_filters") {
t.Errorf("test:ci (or a config it includes) narrows the target set: %q; the race owner must build //... whole", l)
}
}
}
// T5 TestNoUnconditionalValidationOff: no common/build/test line without a
// :config suffix touches run_validations (that would turn validation off
// everywhere, defeating every row's owner), and neither does build:nogo*,
// build:release-cross, build:js-wasm, scripts/ci/bazel-release-cross-compile.sh,
// or any row's own owner config (an owner opting itself out would leave its
// configuration with no validating lane at all).
func TestNoUnconditionalValidationOff(t *testing.T) {
root := sourceRepoRoot(t)
rc := readPolicyFile(t, root, ".bazelrc")
for _, raw := range strings.Split(rc, "\n") {
line := strings.TrimSpace(raw)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
head, _, _ := strings.Cut(line, " ")
cmd, _, hasCfg := strings.Cut(head, ":")
if cmd != "common" && cmd != "build" && cmd != "test" {
continue
}
if !hasCfg && strings.Contains(line, "run_validations") {
t.Errorf(".bazelrc %q: an unconfigured line must not touch run_validations; it would apply to every lane, including every row's owner", line)
}
}
for _, prefix := range []string{"build:nogo", "build:release-cross", "build:js-wasm"} {
for _, raw := range strings.Split(rc, "\n") {
line := strings.TrimSpace(raw)
if strings.HasPrefix(line, prefix) && strings.Contains(line, "validation") {
t.Errorf(".bazelrc %q changes validation for %s, which has no non-owner duplicate to remove", line, prefix)
}
}
}
if script := readPolicyFile(t, root, "scripts/ci/bazel-release-cross-compile.sh"); strings.Contains(script, "run_validations") {
t.Error("scripts/ci/bazel-release-cross-compile.sh turns nogo validation off; release-cross has no non-owner duplicate to remove")
}
for _, row := range nogoConfigurations {
if bazelrcConfigSkipsValidations(rc, row.configs[0]) {
t.Errorf("row %q owner config --config=%s passes --norun_validations; it is the row's only validating lane", row.name, row.configs[0])
}
}
}
// T6 TestRunValidationsOnlyOnAllowlistedLines: the only
// run_validations/norun_validations lines anywhere in .bazelrc or bazel.yml
// are exactly the ones T1's race row already requires: one
// "test:<non-owner config> --norun_validations" line per non-owner in
// .bazelrc, and the two package-gate jobs' bazel-build steps. Main's
// TestLintAndVetRunAsNogo used to ban run_validations outright, which caught
// a configuration outside T1's table (e.g. test:integration,
// test:doltserver-cmd -- the race+integration group is a later slice,
// f5-spec.md §5 S2) quietly adding --norun_validations with no owner to
// notice. T5 does not re-check that case: it only forbids unconfigured
// lines and a short prefix list. This allowlist restores the ban for every
// other line without re-banning the four the race row requires.
func TestRunValidationsOnlyOnAllowlistedLines(t *testing.T) {
root := sourceRepoRoot(t)
var raceNonOwners []string
for _, row := range nogoConfigurations {
if row.name == "race" {
raceNonOwners = row.configs[1:]
}
}
if len(raceNonOwners) == 0 {
t.Fatal("race row has no non-owner configs to allowlist")
}
allowed := map[string]bool{}
for _, cfg := range raceNonOwners {
allowed["test:"+cfg+" --norun_validations"] = true
}
rc := readPolicyFile(t, root, ".bazelrc")
for _, raw := range strings.Split(rc, "\n") {
line := strings.TrimSpace(raw)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
if strings.Contains(line, "run_validations") && !allowed[line] {
t.Errorf(".bazelrc %q: run_validations is only allowed as one of %v; any other line (an unlisted config, an --integration config, or a bare --run_validations toggle) would turn validation off with no owner lane catching it", line, sortedKeys(allowed))
}
}
// bazel.yml: only the two package-gate jobs' bazel-build steps may pass
// --norun_validations (they spell the race build flags out on the
// command line rather than through a .bazelrc --config; T1 pins their
// content). Anywhere else -- a new job, a new step, a non-package-gate
// job -- would turn a lane's validation off unnoticed.
workflow := readCIWorkflow(t, bazelWorkflowName)
for name, j := range workflow.Jobs {
for _, st := range j.Steps {
if !strings.Contains(st.Run, "run_validations") {
continue
}
if !slices.Contains(nogoPackageGateJobs, name) {
stepLabel := st.Name
if stepLabel == "" {
stepLabel = st.Uses
}
t.Errorf("%s job %q step %q: run_validations outside the package-gate build steps", bazelWorkflowName, name, stepLabel)
}
}
}
}

View file

@ -64,6 +64,8 @@ var bazelPRLaneRCLines = map[string][]string{
"test:doltserver --test_arg=-test.parallel=4",
"test:doltserver --local_test_jobs=4",
"test:doltserver --remote_download_regex=.*/test\\.(log|xml)$",
// F5 S1: the race configuration's nogo is owned by the test lane.
"test:doltserver --norun_validations",
},
"pure": {
"build:pure --@rules_go//go/config:pure",
@ -318,7 +320,11 @@ func testPackageGateJobs(t *testing.T, prGateRequired []string) {
// e2b78f7d7a: both configs cquery to the identical
// bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path).
buildStep := job.step(t, "bazel build //cmd/bd:bd_for_tests")
if buildStep.If != bazelPathIf || buildStep.Run != `bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$'` {
// F5 S1: the package gates' build is a race-configuration non-owner
// (bd_for_tests is also built, with its nogo already validated, by
// bazel-test's `bazel test //... --config=ci`), so it passes
// --norun_validations too.
if buildStep.If != bazelPathIf || buildStep.Run != `bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations` {
t.Errorf("%s bazel build step: if %q, run %q; want if %q", lane.job, buildStep.If, buildStep.Run, bazelPathIf)
}
pkgBD := job.step(t, "Package the Bazel-built bd")

View file

@ -1092,6 +1092,8 @@ var bazelDoltServerRCLines = map[string][]string{
"test:doltserver-proxied --local_test_jobs=4",
"test:doltserver-proxied --remote_download_regex=.*/test\\.(log|xml)$",
"test:doltserver-proxied --experimental_remote_cache_eviction_retries=0",
// F5 S1: the race configuration's nogo is owned by the test lane.
"test:doltserver-proxied --norun_validations",
},
"doltserver-integration": {
"build:doltserver-integration --@rules_go//go/config:tags=gms_pure_go,integration",

View file

@ -0,0 +1,273 @@
#!/usr/bin/env python3
"""Dynamic cross-check for the nogo-ownership policy (scripts/nogo_lint_policy_test.go).
The static policy tests (T1-T3, T5-T6) pin .bazelrc/bazel.yml *text*: a
config's flags, and which lane owns its validations. This script instead
asks Bazel itself whether the owner's RunNogo action set for a configuration
really is a superset of every non-owner's RunNogo action set for that same
configuration -- i.e. the owner genuinely analyzes every first-party file a
non-owner compiles, so skipping validation on the non-owner loses no lint
coverage.
Scope (S1): the race configuration group only -- bazel-test (owner, race)
vs. embedded/doltserver/doltserver-proxied/dolt-race (non-owners, which pass
--norun_validations). The integration group (S2) is out of scope until its
lanes gain the same split; see scripts/nogo_lint_policy_test.go's
nogoConfigurations table, which this script's RACE_GROUP mirrors.
Each config's key-affecting flags (the same fingerprint
scripts/nogo_lint_policy_test.go's nogoKeyFlagPattern extracts) are read
straight out of .bazelrc and passed to `aquery`/`query` directly: those are
"build"-family commands and do not see flags defined under a bare `test:X`
.bazelrc stanza, so `--config=X` itself cannot be used here.
The owner (`test:ci`) builds //... whole (T3 pins that: no
--build_tests_only, no --build_tag_filters), so its RunNogo set is queried
directly over //.... Every non-owner here instead passes --build_tests_only
with a --test_tag_filters=<tag[,tag...]> value, which narrows what `bazel
test` actually builds to deps(the tagged tests) -- querying RunNogo over
//... whole for a non-owner would silently compare the *owner's* universe
against itself and always report full coverage (ga-f5s1-nogo-ownership-1:
this was exactly that bug). So each non-owner's target set here is derived
the same way `bazel test --build_tests_only --test_tag_filters=...` derives
it: the deps closure of `tests(//...)` filtered down to targets carrying one
of the lane's tags, read as an *exact* tag (not a substring -- "dolt-server"
is a prefix of both "dolt-server-proxied" and "dolt-server-cmd", so a naive
substring match over `attr(tags, ...)` would wrongly fold those in; tags are
read from `--output=xml` instead, which lists each tag as its own exact
string, to sidestep that).
This is advisory and dynamic (runs real `bazel query`/`aquery`, a few
minutes locally): it is intentionally not part of the required `//scripts`
static test suite, which cannot shell out to Bazel from inside a Bazel
action. Run it with `make nogo-ownership`, locally or on demand in a
slice's own PR.
Exit status: 0 if every non-owner's RunNogo target set is covered by the
owner's, 2 otherwise. Prints the offending targets.
"""
from __future__ import annotations
import argparse
import json
import re
import subprocess
import sys
import xml.etree.ElementTree as ET
from pathlib import Path
# (name, owner config, [non-owner config, ...]) for the race group.
# Keep in sync with scripts/nogo_lint_policy_test.go's nogoConfigurations.
RACE_GROUP = (
"race",
"ci",
["embedded", "doltserver", "doltserver-proxied", "dolt-race"],
)
# Mirrors scripts/nogo_lint_policy_test.go's nogoKeyFlagPattern.
KEY_FLAG_RE = re.compile(
r"--@rules_go//go/config:(?:race|pure|tags=\S+)"
r"|--platforms=\S+"
r"|//tools/bazel:release_platforms\S*"
)
CONFIG_LINE_RE = re.compile(r"^(build|test):([A-Za-z0-9_-]+)\s+(.*)$")
TEST_TAG_FILTERS_RE = re.compile(r"^(?:build|test) --test_tag_filters=(\S+)$")
BUILD_TESTS_ONLY_RE = re.compile(r"^(?:build|test) --build_tests_only$")
def bazelrc_lines(rc_text: str, name: str) -> list[str]:
"""Every raw ("build"|"test", opt) line for --config=name, in file order."""
out = []
for line in rc_text.splitlines():
line = line.split(" #", 1)[0].rstrip()
m = CONFIG_LINE_RE.match(line)
if m and m.group(2) == name:
out.append(f"{m.group(1)} {m.group(3)}")
return out
def expand_config(rc_text: str, name: str, seen: set[str]) -> list[str]:
"""Recursively follows nested --config=Y references, as the Go test does."""
if name in seen:
return []
seen.add(name)
out = []
for line in bazelrc_lines(rc_text, name):
for prefix in ("build --config=", "test --config="):
if line.startswith(prefix):
out.extend(expand_config(rc_text, line[len(prefix):], seen))
break
else:
out.append(line)
return out
def fingerprint(rc_text: str, name: str) -> list[str]:
lines = expand_config(rc_text, name, set())
flags = set()
for line in lines:
flags.update(m.group(0) for m in KEY_FLAG_RE.finditer(line))
return sorted(flags)
def test_tag_filters(rc_text: str, name: str) -> list[str] | None:
"""The lane's --test_tag_filters value, as a list of exact tags, if any.
None means the lane sets no --test_tag_filters (the owner: it builds
//... whole, per T3). A --test_tag_filters value with a leading '-'
(tag exclusion) is not used by any race-group lane today; this raises
rather than silently mis-scoping a future lane that added one.
"""
for line in expand_config(rc_text, name, set()):
m = TEST_TAG_FILTERS_RE.match(line)
if not m:
continue
tags = m.group(1).split(",")
for tag in tags:
if tag.startswith("-"):
raise NotImplementedError(
f"--config={name}: --test_tag_filters={m.group(1)} has an "
"exclusion tag; this script only understands inclusion filters"
)
return tags
return None
def lane_builds_tests_only(rc_text: str, name: str) -> bool:
return any(BUILD_TESTS_ONLY_RE.match(line) for line in expand_config(rc_text, name, set()))
def run_query_xml(bazel: str, query: str) -> ET.Element:
cmd = [bazel, "--nohome_rc", "query", query, "--output=xml", "--noimplicit_deps"]
out = subprocess.run(cmd, capture_output=True, text=True, check=True).stdout
return ET.fromstring(out)
def tagged_test_targets(bazel: str, tags: list[str]) -> set[str]:
"""Labels of tests(//...) carrying at least one of tags, read exactly.
Tags are read from --output=xml's own <list name="tags"><string .../></list>
elements rather than matched with attr()'s regex against a stringified
list: "dolt-server" is a prefix of both "dolt-server-proxied" and
"dolt-server-cmd" (and attr()'s per-element regex match is a substring
match, not an exact one), so attr(tags, "dolt-server", //...) also
returns proxied- and cmd-tagged targets -- the exact bug this script
exists to avoid reintroducing one level up.
"""
root = run_query_xml(bazel, "tests(//...)")
wanted = set(tags)
labels = set()
for rule in root.findall("rule"):
rule_tags = {
s.get("value")
for list_el in rule.findall("list[@name='tags']")
for s in list_el.findall("string")
}
if rule_tags & wanted:
labels.add(rule.get("name"))
return labels
def lane_universe(bazel: str, targets: set[str] | None) -> str:
"""The query-language target expression this lane builds.
None (the owner, or any lane with no --build_tests_only): //... whole.
Otherwise (a lane that builds only its tagged tests and their deps, as
--build_tests_only + --test_tag_filters does): the deps closure of just
those targets.
"""
if targets is None:
return "//..."
if not targets:
raise ValueError("lane_universe: empty tagged target set")
return "deps(set(" + " ".join(sorted(targets)) + "))"
def run_nogo_targets(bazel: str, flags: list[str], universe: str) -> set[str]:
"""Returns the set of target labels with a RunNogo action under flags."""
cmd = [bazel, "--nohome_rc", "aquery", f'mnemonic("RunNogo", {universe})', *flags,
"--output=jsonproto"]
out = subprocess.run(cmd, capture_output=True, text=True, check=True).stdout
data = json.loads(out)
targets_by_id = {t["id"]: t["label"] for t in data.get("targets", [])}
labels = set()
for action in data.get("actions", []):
if action.get("mnemonic") != "RunNogo":
continue
label = targets_by_id.get(action.get("targetId"))
if label and not label.startswith("@"):
labels.add(label)
return labels
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--bazel", default="bazel", help="bazel/bazelisk binary to invoke")
parser.add_argument("--bazelrc", default=".bazelrc")
args = parser.parse_args()
rc_text = Path(args.bazelrc).read_text()
name, owner, non_owners = RACE_GROUP
owner_fp = fingerprint(rc_text, owner)
if lane_builds_tests_only(rc_text, owner):
raise NotImplementedError(
f"--config={owner} now passes --build_tests_only, narrowing its target "
"set; this script assumes the race owner builds //... whole "
"(scripts/nogo_lint_policy_test.go's T3). --test_tag_filters alone "
"(exclusion-only, as today's owner has) only narrows test execution, "
"not the build, so it is not checked here."
)
owner_targets = run_nogo_targets(args.bazel, owner_fp, "//...")
problems = []
for non_owner in non_owners:
non_owner_fp = fingerprint(rc_text, non_owner)
if non_owner_fp != owner_fp:
problems.append(
f"{name}: --config={non_owner}'s fingerprint {non_owner_fp} "
f"differs from owner --config={owner}'s {owner_fp}; it is no "
"longer safe to skip its validations"
)
continue
tags = test_tag_filters(rc_text, non_owner)
if tags is None or not lane_builds_tests_only(rc_text, non_owner):
# No narrowing this script knows how to replicate: fall back to
# the owner's own universe, which is always a safe (if coarser)
# comparison.
non_owner_universe = "//..."
else:
tagged = tagged_test_targets(args.bazel, tags)
if not tagged:
problems.append(
f"{name}: --config={non_owner}'s --test_tag_filters={','.join(tags)} "
"matches no test in //...; its coverage could not be checked"
)
continue
non_owner_universe = lane_universe(args.bazel, tagged)
non_owner_targets = run_nogo_targets(args.bazel, non_owner_fp, non_owner_universe)
missing = non_owner_targets - owner_targets
if missing:
problems.append(
f"{name}: --config={non_owner} compiles {len(missing)} target(s) "
f"with no RunNogo action under the owner --config={owner}: "
+ ", ".join(sorted(missing)[:10])
+ (" ..." if len(missing) > 10 else "")
)
if problems:
print("nogo_ownership: FAIL", file=sys.stderr)
for p in problems:
print(f" {p}", file=sys.stderr)
return 2
print(f"nogo_ownership: OK ({name}: --config={owner} covers "
f"{', '.join(non_owners)})")
return 0
if __name__ == "__main__":
sys.exit(main())