From b8e6ba17916fd3cc42f511fea143e34191f8b1f2 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sat, 10 Oct 2026 18:05:17 +0000 Subject: [PATCH] fix(bazel): one nogo-validating lane per race configuration (F5 S1) (#7479) * fix(bazel): one nogo-validating lane per race configuration (F5 S1) The race configuration (--@rules_go//go/config:race) is built by five lanes that duplicate the same RunNogo/ValidateNogo work: bazel-test (--config=ci), bazel-embedded, bazel-doltserver, bazel-proxied, and the advisory bazel-dolt-race shadow, plus the package-mcp/package-npm gates' explicit race build. --norun_validations is a build-request option, not a configuration flag (proven here by aquery: GoCompilePkg/ GoLink/RunNogo/ValidateNogo action keys are byte-identical with and without it), so passing it on every lane but bazel-test removes the duplicate work without touching a single action key or dropping any lint coverage: bazel-test's --config=ci already validates every file any of the others compiles. scripts/nogo_lint_policy_test.go replaces the old blanket "--norun_validations never appears in .bazelrc" check with TestNogoConfigurationsHaveOneValidatingLane (T1: one owner per configuration fingerprint), TestNogoOwnersAreRequired (T2: the owner is a required CI Gate check whose `if:` covers every non-owner's), TestNogoRaceOwnerBuildsEverything (T3: the owner builds //... whole, no exclusions) and TestNoUnconditionalValidationOff (T5: nothing else may disable validation). tools/bazel/nogo_ownership.py (make nogo-ownership) is a dynamic, advisory aquery cross-check of the same property, run on demand rather than from inside a Bazel action. Verified locally: aquery shows identical action keys for the affected lanes with/without --norun_validations; a planted printf-format vet finding in internal/idgen/hash.go failed the owner build and passed the non-owner build (then reverted, never committed); scripts/bazel_dolt_race_shadow_test.go and the full //scripts suite stay green. Co-Authored-By: Claude Sonnet 5 * fix(bazel): close S1 nogo-ownership review gaps (allowlist, owner pin, real aquery) Addresses maintainer-city-c3's review of #7479: 1. T6 TestRunValidationsOnlyOnAllowlistedLines replaces the coverage gap left when main's blanket run_validations ban was dropped: only the four race-group non-owners' "test: --norun_validations" lines in .bazelrc, and the two package-gate bazel-build steps in bazel.yml, are permitted. A later test:integration/doltserver-cmd --norun_validations (out of T1's table, S2) now fails here instead of landing silently. Verified by mutation: adding test:integration --norun_validations fails the new test; reverting passes. 3. TestNogoRaceOwnerBuildsEverything (T3) now also pins that the race owner's "bazel test //... --config=ci" step has no step-level if: or continue-on-error, and that its job has none either. Verified by mutation: adding if: ${{ always() }} to that step fails; reverting passes. 2. tools/bazel/nogo_ownership.py's aquery comparison was querying the identical mnemonic("RunNogo", //...) for the owner and every non-owner, so `missing` could never be non-empty. It now derives each non-owner's actual built target set the way --build_tests_only + --test_tag_filters does: the deps closure of tests(//...) filtered to targets carrying one of the lane's tags, read exactly from `query --output=xml` (not attr()'s substring regex match, which would wrongly fold "dolt-server-proxied" and "dolt-server-cmd" into a "dolt-server" filter). Confirmed against a real (offline, --nohome_rc) bazel run: reports OK at head, and reports 100+ missing targets when the owner's query is artificially narrowed to a subpackage -- it can now actually fail. Also: refreshed a stale .bazelrc comment claiming nogo runs beside every lane's compile, which S1 made untrue for race-group non-owners. Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Sonnet 5 --- .bazelrc | 34 +- .github/workflows/bazel.yml | 4 +- Makefile | 12 +- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 13 +- engdocs/LINTING.md | 10 +- scripts/ci_workflow_test.go | 2 + scripts/nogo_lint_policy_test.go | 428 +++++++++++++++++++++++- scripts/pr_lanes_bazel_coverage_test.go | 8 +- scripts/pr_risk_bazel_coverage_test.go | 2 + tools/bazel/nogo_ownership.py | 273 +++++++++++++++ 10 files changed, 753 insertions(+), 33 deletions(-) create mode 100644 tools/bazel/nogo_ownership.py diff --git a/.bazelrc b/.bazelrc index 61e7e0207..d78d7a842 100644 --- a/.bazelrc +++ b/.bazelrc @@ -104,6 +104,16 @@ test:embedded --remote_download_regex=.*/test\.xml$ # jobs, which never retried. Evicted inputs fail the step instead; a # visible "Re-run failed jobs" retries it. test:embedded --experimental_remote_cache_eviction_retries=0 +# The race configuration's nogo validations run once, in bazel.yml's `test` +# lane (`bazel test //... --config=ci`), which is a strict superset of every +# race-group lane's RunNogo/ValidateNogo keys. --norun_validations is a +# build-request option, not a configuration flag: it changes no action key +# (compiles, links and test PASSes are shared with `test` through the +# action cache) and discards no analysis; it only skips re-running nogo's +# validation actions that the owner lane already runs on this configuration. +# scripts/nogo_lint_policy_test.go's TestNogoConfigurationsHaveOneValidatingLane +# pins `test` as this configuration's sole validating lane. +test:embedded --norun_validations # Dolt-server lane: what pr.yml's retired container-backed jobs ("Test # (storage domain + uow)" and "Contract corpus") ran, but every dolt @@ -127,6 +137,9 @@ test:doltserver --test_arg=-test.parallel=4 # runner shape; remote workers are not limited by it. test:doltserver --local_test_jobs=4 test:doltserver --remote_download_regex=.*/test\.(log|xml)$ +# Race-group nogo is owned by the `test` lane; see test:embedded's comment +# above. +test:doltserver --norun_validations # Proxied-server tier: PR Risk's "Test (Proxied Dolt Cmd N/15)" jobs, as # //cmd/bd:bd_proxied_test and //cmd/bd:bd_managed_local_test (the @@ -160,6 +173,9 @@ test:doltserver-proxied --remote_download_regex=.*/test\.(log|xml)$ # jobs, which never retried. Evicted inputs fail the step instead; a # visible "Re-run failed jobs" retries it. test:doltserver-proxied --experimental_remote_cache_eviction_retries=0 +# Race-group nogo is owned by the `test` lane; see test:embedded's comment +# above. +test:doltserver-proxied --norun_validations # dolt-server + proxied union (item 5, Phase 1 shadow): bazel.yml's advisory # bazel-dolt-race job runs both tiers in one invocation, beside their own @@ -186,6 +202,10 @@ test:dolt-race --test_arg=-test.parallel=4 test:dolt-race --local_test_jobs=4 test:dolt-race --remote_download_regex=.*/test\.(log|xml)$ test:dolt-race --experimental_remote_cache_eviction_retries=0 +# The union carries its members' --norun_validations line +# (TestBazelDoltRaceConfigIsMemberUnion needs no change: both members and the +# union carry the same line). +test:dolt-race --norun_validations # Server-Dolt storage tier: PR Risk's "Test (Server Dolt Conformance)" and # "Test (Server Dolt Full Suite N/16)" jobs, as @@ -320,11 +340,15 @@ build:js-wasm --use_target_platform_for_tests build:js-wasm --remote_download_outputs=toplevel # nogo (//tools/nogo: go test's vet checks plus the golangci-lint linters -# .golangci.yml enables) runs beside every Go compile of every lane, so any -# `bazel build`/`bazel test` fails on a finding. --config=nogo asks for the -# analysis alone (the nogo_fix output group; nothing is linked), in the race -# configuration bazel.yml's unit lane (test:ci) builds, so a local run reuses -# CI's remote cache. make lint, scripts/pr-lint and the pre-commit hook use it. +# .golangci.yml enables) runs beside every Go compile, so `bazel build`/ +# `bazel test` fails on a finding -- in bazel.yml's one validating (owner) +# lane per configuration (since F5 S1: a race-group non-owner lane reuses +# its owner's nogo results instead of re-running them; see +# scripts/nogo_lint_policy_test.go's nogoConfigurations table). +# --config=nogo asks for the analysis alone (the nogo_fix output group; +# nothing is linked), in the race configuration bazel.yml's unit lane +# (test:ci) builds, so a local run reuses CI's remote cache. make lint, +# scripts/pr-lint and the pre-commit hook use it. build:nogo --@rules_go//go/config:race build:nogo --keep_going build:nogo --output_groups=nogo_fix diff --git a/.github/workflows/bazel.yml b/.github/workflows/bazel.yml index 063d27b5e..308e97a0b 100644 --- a/.github/workflows/bazel.yml +++ b/.github/workflows/bazel.yml @@ -2543,7 +2543,7 @@ jobs: # bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path). - name: bazel build //cmd/bd:bd_for_tests if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode == 'remote' }} - run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' + run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations - name: Package the Bazel-built bd if: ${{ steps.detect.outputs.mcp_package == 'true' && needs.rbe.outputs.mode == 'remote' }} @@ -2665,7 +2665,7 @@ jobs: # bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path). - name: bazel build //cmd/bd:bd_for_tests if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode == 'remote' }} - run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' + run: bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations - name: Package the Bazel-built bd if: ${{ steps.detect.outputs.npm_package == 'true' && needs.rbe.outputs.mode == 'remote' }} diff --git a/Makefile b/Makefile index fe5ada1c1..d0bfbbdc6 100644 --- a/Makefile +++ b/Makefile @@ -46,7 +46,7 @@ endif .PHONY: all build doctor-build test test-icu-path test-full-cgo test-regression test-upgrade test-cross-version test-migration corpus-regen githooks-regen bench bench-quick clean clean-test-tmp install install-force help check-up-to-date fmt fmt-check check-testing-short .PHONY: lint lint-changed vet .PHONY: check check-go test-go check-docs-go -.PHONY: ci-pr-core ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm +.PHONY: ci-pr-core ci-pr-lint ci-complexity ci-complexity-diff ci-complexity-check ci-package-mcp ci-package-npm nogo-ownership .PHONY: api-gen api-check .PHONY: bazel-sync bazel-sync-check @@ -237,6 +237,16 @@ ci-pr-core: ci-pr-lint: @./scripts/ci/pr-lint.sh +# nogo-ownership is a dynamic, advisory cross-check (not part of the required +# //scripts static suite, which cannot shell out to Bazel): it asks real +# `bazel aquery` whether the nogo owner lane for each configuration group +# (today: race, bazel-test/--config=ci) really does validate every file a +# non-owner lane (embedded, doltserver, doltserver-proxied, dolt-race, which +# pass --norun_validations) compiles. See tools/bazel/nogo_ownership.py and +# scripts/nogo_lint_policy_test.go's nogoConfigurations table. +nogo-ownership: + @python3 tools/bazel/nogo_ownership.py + # The CI lint gate: native, plus //tools/bazel:release_cross for windows/amd64 # and darwin/arm64 (BD_LINT_TARGETS selects; scripts/pr-lint). lint: ci-pr-lint diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 93cacf9f4..b68ec81d2 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -1025,10 +1025,15 @@ Required` requires them to have run remotely and passed. the job it replaces. - `go test`'s own vet checks (cmd/go's `defaultVetFlags`, policy-tested equal to the toolchain's) and the golangci-lint linters `.golangci.yml` - enables run as nogo (`//tools/nogo`) beside every compile of every Bazel - lane: natively in `bazel test //... --config=ci`, and for every release - platform in the `bazel-release-cross` lane - (engdocs/LINTING.md). The former + enables run as nogo (`//tools/nogo`): natively in `bazel test //... + --config=ci`, and for every release platform in the + `bazel-release-cross` lane (engdocs/LINTING.md). Each Bazel configuration + validates in exactly one required lane, not beside every compile of + every lane: the race configuration's nogo is owned by `test` above, and + `embedded`, `doltserver`, `doltserver-proxied`, `dolt-race` (shadow) and + the package gates pass `--norun_validations`, since they compile a + strict subset of what `test` already validates on that configuration + (`scripts/nogo_lint_policy_test.go`, engdocs/LINTING.md). The former `scripts-go-checks` (`Go checks (vet)`) and `pr-lint-wrapper` (`PR Lint (native|windows|darwin)`) jobs are retired. - The repository policy tests (`./scripts/...`, including the D2 guards) diff --git a/engdocs/LINTING.md b/engdocs/LINTING.md index 991a6d0a3..c44c6de4b 100644 --- a/engdocs/LINTING.md +++ b/engdocs/LINTING.md @@ -49,8 +49,14 @@ Where it gates: validates each of those compiles. So `//go:build windows`, `darwin`, `!linux` (and freebsd, android, arm64) files are analyzed from Linux: a superset of golangci-lint's former `GOOS=windows`/`GOOS=darwin` legs. -- **Every other Bazel lane** (integration, embedded, dolt-server) validates - what it compiles, including files only its build tags select. +- The race configuration's nogo validations run once, owned by the `test` + lane above: `embedded`, `doltserver`, `doltserver-proxied`, `dolt-race` + (shadow) and the package gates (`package-mcp`, `package-npm`) all pass + `--norun_validations`, since each compiles a strict subset of what `test` + already validates on that configuration (`scripts/nogo_lint_policy_test.go`'s + `TestNogoConfigurationsHaveOneValidatingLane` pins this). +- **Every other Bazel lane** (integration) still validates what it compiles, + including files only its build tags select. Every PR is checked against the whole tree, not only its diff: the tree has no findings, so there is no baseline to scope against. diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 87cf96061..fef8f7fd1 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -4310,6 +4310,8 @@ var bazelEmbeddedRCLines = []string{ "test:embedded --test_env=GO_TEST_WRAP_TESTV=1", "test:embedded --remote_download_regex=.*/test\\.xml$", "test:embedded --experimental_remote_cache_eviction_retries=0", + // F5 S1: the race configuration's nogo is owned by the test lane. + "test:embedded --norun_validations", } func TestBazelEmbeddedJobRunsEmbeddedTier(t *testing.T) { diff --git a/scripts/nogo_lint_policy_test.go b/scripts/nogo_lint_policy_test.go index 7e06407ca..9252c2ed4 100644 --- a/scripts/nogo_lint_policy_test.go +++ b/scripts/nogo_lint_policy_test.go @@ -4,6 +4,8 @@ import ( "os" "path/filepath" "regexp" + "slices" + "sort" "strings" "testing" ) @@ -15,6 +17,18 @@ import ( // platforms compile inside the pure lane's release cross-compile // (//tools/bazel:release_cross, every release platform). These pin that // wiring and that golangci-lint is gone from CI, hooks and make. +// +// nogo validation is duplicated across lanes that compile the same Bazel +// configuration (race: test, embedded, doltserver, doltserver-proxied, +// dolt-race and the package gates). TestLintAndVetRunAsNogo used to forbid +// `run_validations` anywhere in .bazelrc ("every lane validates"); that +// blanket ban is replaced by TestNogoConfigurationsHaveOneValidatingLane +// (T1), TestNogoOwnersAreRequired (T2), TestNogoRaceOwnerBuildsEverything +// (T3), TestNoUnconditionalValidationOff (T5) and +// TestRunValidationsOnlyOnAllowlistedLines (T6), which together pin "every +// configuration's nogo runs in exactly one required lane" instead. +// --norun_validations is a build-request option, not a configuration flag: +// it changes no action key and discards no analysis. // nogoBazelrcLines are .bazelrc's nogo configs, exactly. var nogoBazelrcLines = []string{ @@ -38,24 +52,6 @@ func TestLintAndVetRunAsNogo(t *testing.T) { if !equalStrings(got, nogoBazelrcLines) { t.Errorf(".bazelrc nogo configs = %q, want %q", got, nogoBazelrcLines) } - // Every lane validates: a lane that skipped nogo would let findings in - // the files only it compiles (integration-tagged, pure) through. - if strings.Contains(rc, "run_validations") { - t.Error(".bazelrc turns nogo validation off for some lane") - } - - // The cross-platform half: the pure lane's release cross-compile builds - // every library and binary for every release platform, and nogo - // validates each compile (TestReleaseCrossCompileRunsInBazelPureLane - // pins the step). Nothing on that path may opt out of validation. - if script := readPolicyFile(t, root, "scripts/ci/bazel-release-cross-compile.sh"); strings.Contains(script, "run_validations") { - t.Error("scripts/ci/bazel-release-cross-compile.sh turns nogo validation off") - } - for _, line := range strings.Split(rc, "\n") { - if strings.HasPrefix(strings.TrimSpace(line), "build:release-cross") && strings.Contains(line, "validation") { - t.Errorf(".bazelrc %q changes validation for the release cross-compile", line) - } - } // No workflow installs or runs golangci-lint. entries, err := os.ReadDir(filepath.Join(root, ".github", "workflows")) @@ -107,3 +103,399 @@ func TestLintAndVetRunAsNogo(t *testing.T) { t.Error("scripts/ci/pr-lint.sh runs gofmt; //scripts/repochecks:fmt_test gates formatting") } } + +// --- F5 S1: one validating lane per nogo configuration ----------------- + +// nogoKeyFlagPattern matches the build-affecting flags f5-spec.md §6 T1 +// calls out as fingerprint-relevant: they select a distinct Bazel +// configuration (and so a distinct set of nogo action keys). Flags like +// --test_tag_filters, --keep_going or --test_arg narrow which tests run but +// do not change what is compiled, so they are not in the fingerprint. +var nogoKeyFlagPattern = regexp.MustCompile(`--@rules_go//go/config:(race|pure|tags=\S+)|--platforms=\S+|//tools/bazel:release_platforms\S*`) + +// expandBazelrcConfig returns every " " line (as bazelRCConfigLines +// formats them) that --config=name pulls in, recursively following any +// nested --config=Y reference within those lines (as .bazelrc's own comments +// describe, e.g. test:ci --config=prcore). seen prevents infinite recursion +// on a cycle. +func expandBazelrcConfig(rc, name string, seen map[string]bool) []string { + if seen[name] { + return nil + } + seen[name] = true + var out []string + for _, l := range bazelRCConfigLines(rc, name) { + if v, ok := strings.CutPrefix(l, "build --config="); ok { + out = append(out, expandBazelrcConfig(rc, v, seen)...) + continue + } + if v, ok := strings.CutPrefix(l, "test --config="); ok { + out = append(out, expandBazelrcConfig(rc, v, seen)...) + continue + } + out = append(out, l) + } + return out +} + +// nogoFingerprint reduces a config's expanded lines to its key-affecting +// flags, sorted and deduplicated, so two configs that build the same thing +// compare equal regardless of line order or non-key-affecting flags. +func nogoFingerprint(lines []string) []string { + seen := map[string]bool{} + var out []string + for _, l := range lines { + for _, m := range nogoKeyFlagPattern.FindAllString(l, -1) { + if !seen[m] { + seen[m] = true + out = append(out, m) + } + } + } + sort.Strings(out) + return out +} + +// bazelrcConfigSkipsValidations reports whether --config=name's own +// (non-recursive) .bazelrc lines pass --norun_validations. Non-recursive +// because each non-owner config sets the flag itself (D1); inheriting it +// through a nested --config would make the policy table here miss a +// config that silently stopped validating by referencing one that did. +func bazelrcConfigSkipsValidations(rc, name string) bool { + for _, l := range bazelRCConfigLines(rc, name) { + if strings.Contains(l, "--norun_validations") { + return true + } + } + return false +} + +// nogoConfigurations: f5-spec.md §6 T1's table. Each row is one Bazel +// configuration (one nogo fingerprint): the .bazelrc --config names (or, for +// the package gates, a pseudo-config checked against their pinned command +// line) that build it, the bazel.yml job that owns validating it, and the +// jobs that must pass --norun_validations instead. +// +// Only the race group is classified here (F5 S1). The race+integration +// group (integration, doltserver-integration, doltserver-cmd) and `pure`'s +// optional D4a are later slices (f5-spec.md §5 S2, S3): every lane in them +// still validates today, so they are intentionally left out of this table +// rather than added as a row that TestNogoConfigurationsHaveOneValidatingLane +// could never satisfy. `pure`, js/wasm and release-cross are unaffected by +// any slice (single-lane configurations: there is no other lane to +// deduplicate against), so they are included as trivial one-member rows for +// completeness and as a guard against a future lane quietly joining them. +var nogoConfigurations = []struct { + name string + configs []string // .bazelrc --config names sharing this fingerprint, owner first + owner string // bazel.yml job name required to validate + nonOwners []string // bazel.yml job names required to pass --norun_validations +}{ + { + name: "race", + configs: []string{"ci", "embedded", "doltserver", "doltserver-proxied", "dolt-race"}, + owner: bazelJobName, + nonOwners: []string{bazelEmbedJobName, bazelDoltJobName, bazelProxiedJobName, bazelDoltRaceJobName}, + }, + { + name: "pure", + configs: []string{"pure"}, + owner: bazelPureJobName, + }, + { + name: "js-wasm", + configs: []string{"js-wasm"}, + owner: bazelPureJobName, + }, +} + +// nogoPackageGateJobs: the package gates do not select a named .bazelrc +// --config; their `bazel build` command line spells the race flag out +// (pr_lanes_bazel_coverage_test.go pins the exact command). They are race +// non-owners, checked against the "race" row's fingerprint and +// --norun_validations requirement directly from that pinned command. +var nogoPackageGateJobs = []string{bazelPackageMCPJobName, bazelPackageNPMJobName} + +// T1 TestNogoConfigurationsHaveOneValidatingLane: every classified Bazel +// configuration has exactly one validating (required) lane; every lane that +// builds the same configuration passes --norun_validations instead; and +// every config within a row shares that row's fingerprint, so a key-affecting +// flag added to a non-owner (which would split it into a different, now +// unvalidated, configuration) fails here instead of silently losing +// coverage. +func TestNogoConfigurationsHaveOneValidatingLane(t *testing.T) { + rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc") + workflow := readCIWorkflow(t, bazelWorkflowName) + + seenConfig := map[string]string{} // config -> row name, to catch a config in two rows + for _, row := range nogoConfigurations { + if row.owner == "" { + t.Fatalf("nogoConfigurations row %q has no owner", row.name) + } + if len(row.configs) == 0 { + t.Fatalf("nogoConfigurations row %q has no configs", row.name) + } + ownerFP := nogoFingerprint(expandBazelrcConfig(rc, row.configs[0], map[string]bool{})) + for _, c := range row.configs { + if prev, ok := seenConfig[c]; ok { + t.Errorf("--config=%s appears in both row %q and row %q", c, prev, row.name) + } + seenConfig[c] = row.name + fp := nogoFingerprint(expandBazelrcConfig(rc, c, map[string]bool{})) + if !sameStringSet(fp, ownerFP) { + t.Errorf("row %q: --config=%s fingerprint %v != owner %s's %v (a key-affecting flag split this config out of the row)", + row.name, c, fp, row.configs[0], ownerFP) + } + } + + ownerJob := workflow.job(t, row.owner) + if ownerJob.ContinueOnError { + t.Errorf("row %q owner %s has continue-on-error; a skipped finding there would never fail anything", row.name, row.owner) + } + if bazelrcConfigSkipsValidations(rc, row.configs[0]) { + t.Errorf("row %q owner config --config=%s passes --norun_validations; it is the row's only validating lane", row.name, row.configs[0]) + } + + nonOwners := append([]string{}, row.nonOwners...) + if row.name == "race" { + nonOwners = append(nonOwners, nogoPackageGateJobs...) + } + for i, job := range nonOwners { + if job == row.owner { + t.Errorf("row %q lists its own owner %s as a non-owner", row.name, job) + } + isPackageGate := slices.Contains(nogoPackageGateJobs, job) + if isPackageGate { + // The package gates spell their flags out on the command + // line rather than through a named --config + // (pr_lanes_bazel_coverage_test.go pins the exact string); + // confirmed here against their job's steps instead of + // .bazelrc. + step := workflow.job(t, job).step(t, "bazel build //cmd/bd:bd_for_tests") + if !strings.Contains(step.Run, "--@rules_go//go/config:race") { + t.Errorf("row %q non-owner %s's build step does not build the race configuration: %q", row.name, job, step.Run) + } + if !strings.Contains(step.Run, "--norun_validations") { + t.Errorf("row %q non-owner %s's build step does not pass --norun_validations", row.name, job) + } + continue + } + // Map job -> its own --config, which is row.configs[1+i'] in + // declaration order (owner's config is row.configs[0]). + idx := i + 1 + if idx >= len(row.configs) { + t.Fatalf("row %q has more non-owner jobs than configs (idx %d, configs %v)", row.name, idx, row.configs) + } + cfg := row.configs[idx] + if !bazelrcConfigSkipsValidations(rc, cfg) { + t.Errorf("row %q non-owner %s (--config=%s) does not pass --norun_validations; it duplicates %s's nogo", row.name, job, cfg, row.owner) + } + } + } +} + +// T2 TestNogoOwnersAreRequired: each owner job is required by pr.yml's CI +// Gate (or, for a flag-gated lane, the flag that is pinned true) with no +// job-level continue-on-error, and its `if:` runs at least everywhere its +// non-owners' do — otherwise a mode could run a non-owner's compile with no +// owner validating it at all (f5-spec.md §9). +func TestNogoOwnersAreRequired(t *testing.T) { + workflow := readCIWorkflow(t, bazelWorkflowName) + gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") + gateEnv := gate.step(t, "Evaluate CI gate").Env + required := strings.Fields(gateEnv["CI_GATE_REQUIRED"]) + + ownerGateToken := map[string]string{ + bazelJobName: "BAZEL_TEST", + bazelPureJobName: "BAZEL_PURE", + } + + events := []string{"pull_request", "merge_group", "pull_request_target", "push", "schedule", "workflow_dispatch"} + + for _, row := range nogoConfigurations { + ownerJob := workflow.job(t, row.owner) + if token, ok := ownerGateToken[row.owner]; ok && !contains(required, token) { + t.Errorf("row %q owner %s: CI_GATE_REQUIRED %v lacks %s", row.name, row.owner, required, token) + } + if ownerJob.ContinueOnError { + t.Errorf("row %q owner %s has continue-on-error", row.name, row.owner) + } + + nonOwners := append([]string{}, row.nonOwners...) + if row.name == "race" { + nonOwners = append(nonOwners, nogoPackageGateJobs...) + } + for _, job := range nonOwners { + if slices.Contains(nogoPackageGateJobs, job) { + // The package gates' bazel build only ever runs in mode + // remote (TestPackageGateJobs pins the surrounding `if:`), + // and the owner's `if:` (mode != 'skip') is true whenever + // mode == 'remote': "remote" is never "skip". + step := workflow.job(t, job).step(t, "bazel build //cmd/bd:bd_for_tests") + if !strings.Contains(step.If, "mode == 'remote'") { + t.Errorf("%s's bazel-build step if %q no longer implies the owner's mode != 'skip'", job, step.If) + } + continue + } + nonOwnerJob := workflow.job(t, job) + for _, mode := range bazelRBEModes { + for _, event := range events { + ctx := map[string]string{ + "needs.rbe.outputs.mode": mode, + "needs.rbe.outputs.enabled": bazelModeEnabled(mode), + "github.event_name": event, + } + if evalRRCIf(t, nonOwnerJob.If, ctx) && !evalRRCIf(t, ownerJob.If, ctx) { + t.Errorf("row %q: %s runs in mode=%s event=%s but owner %s does not (owner if %q, non-owner if %q)", + row.name, job, mode, event, row.owner, ownerJob.If, nonOwnerJob.If) + } + } + } + } + } +} + +// T3 TestNogoRaceOwnerBuildsEverything: the race owner's target set covers +// every file any race-group lane compiles (f5-spec.md §2.1): its test step +// runs over the whole tree with no "-//" exclusion, and test:ci's expansion +// narrows no target selection (--build_tests_only, --build_tag_filters) +// — only test execution (--test_tag_filters etc., which T1's fingerprint +// deliberately ignores). +func TestNogoRaceOwnerBuildsEverything(t *testing.T) { + workflow := readCIWorkflow(t, bazelWorkflowName) + ownerJob := workflow.job(t, bazelJobName) + test := ownerJob.step(t, "bazel test //... --config=ci") + if !strings.Contains(test.Run, "bazel test //... --config=ci") { + t.Fatalf("%s step does not run bazel test //... --config=ci:\n%s", bazelJobName, test.Run) + } + if regexp.MustCompile(`bazel test[^\n]*\s-//`).MatchString(test.Run) { + t.Errorf("%s excludes a target (-//...) from //...: it would stop validating what it excludes", bazelJobName) + } + // The race owner's validating step must be unconditional: a step-level + // `if:` or continue-on-error here would let the suite report green while + // race-group nogo silently did not run, with nothing else in this file + // (or T1/T2, which only check job-level fields) noticing. + if test.If != "" { + t.Errorf("%s step %q has if %q; the race owner's validating step must be unconditional", bazelJobName, test.Name, test.If) + } + if test.ContinueOnError != nil && test.ContinueOnError != false { + t.Errorf("%s step %q has continue-on-error %v; a failed race-group nogo finding there would never fail the job", bazelJobName, test.Name, test.ContinueOnError) + } + if ownerJob.ContinueOnError { + t.Errorf("%s job has continue-on-error; the race owner's job must be able to fail", bazelJobName) + } + + rc := readPolicyFile(t, bazelPolicyRoot(t), ".bazelrc") + for _, l := range expandBazelrcConfig(rc, "ci", map[string]bool{}) { + if strings.Contains(l, "build_tests_only") || strings.Contains(l, "build_tag_filters") { + t.Errorf("test:ci (or a config it includes) narrows the target set: %q; the race owner must build //... whole", l) + } + } +} + +// T5 TestNoUnconditionalValidationOff: no common/build/test line without a +// :config suffix touches run_validations (that would turn validation off +// everywhere, defeating every row's owner), and neither does build:nogo*, +// build:release-cross, build:js-wasm, scripts/ci/bazel-release-cross-compile.sh, +// or any row's own owner config (an owner opting itself out would leave its +// configuration with no validating lane at all). +func TestNoUnconditionalValidationOff(t *testing.T) { + root := sourceRepoRoot(t) + rc := readPolicyFile(t, root, ".bazelrc") + + for _, raw := range strings.Split(rc, "\n") { + line := strings.TrimSpace(raw) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + head, _, _ := strings.Cut(line, " ") + cmd, _, hasCfg := strings.Cut(head, ":") + if cmd != "common" && cmd != "build" && cmd != "test" { + continue + } + if !hasCfg && strings.Contains(line, "run_validations") { + t.Errorf(".bazelrc %q: an unconfigured line must not touch run_validations; it would apply to every lane, including every row's owner", line) + } + } + + for _, prefix := range []string{"build:nogo", "build:release-cross", "build:js-wasm"} { + for _, raw := range strings.Split(rc, "\n") { + line := strings.TrimSpace(raw) + if strings.HasPrefix(line, prefix) && strings.Contains(line, "validation") { + t.Errorf(".bazelrc %q changes validation for %s, which has no non-owner duplicate to remove", line, prefix) + } + } + } + + if script := readPolicyFile(t, root, "scripts/ci/bazel-release-cross-compile.sh"); strings.Contains(script, "run_validations") { + t.Error("scripts/ci/bazel-release-cross-compile.sh turns nogo validation off; release-cross has no non-owner duplicate to remove") + } + + for _, row := range nogoConfigurations { + if bazelrcConfigSkipsValidations(rc, row.configs[0]) { + t.Errorf("row %q owner config --config=%s passes --norun_validations; it is the row's only validating lane", row.name, row.configs[0]) + } + } +} + +// T6 TestRunValidationsOnlyOnAllowlistedLines: the only +// run_validations/norun_validations lines anywhere in .bazelrc or bazel.yml +// are exactly the ones T1's race row already requires: one +// "test: --norun_validations" line per non-owner in +// .bazelrc, and the two package-gate jobs' bazel-build steps. Main's +// TestLintAndVetRunAsNogo used to ban run_validations outright, which caught +// a configuration outside T1's table (e.g. test:integration, +// test:doltserver-cmd -- the race+integration group is a later slice, +// f5-spec.md §5 S2) quietly adding --norun_validations with no owner to +// notice. T5 does not re-check that case: it only forbids unconfigured +// lines and a short prefix list. This allowlist restores the ban for every +// other line without re-banning the four the race row requires. +func TestRunValidationsOnlyOnAllowlistedLines(t *testing.T) { + root := sourceRepoRoot(t) + + var raceNonOwners []string + for _, row := range nogoConfigurations { + if row.name == "race" { + raceNonOwners = row.configs[1:] + } + } + if len(raceNonOwners) == 0 { + t.Fatal("race row has no non-owner configs to allowlist") + } + allowed := map[string]bool{} + for _, cfg := range raceNonOwners { + allowed["test:"+cfg+" --norun_validations"] = true + } + + rc := readPolicyFile(t, root, ".bazelrc") + for _, raw := range strings.Split(rc, "\n") { + line := strings.TrimSpace(raw) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + if strings.Contains(line, "run_validations") && !allowed[line] { + t.Errorf(".bazelrc %q: run_validations is only allowed as one of %v; any other line (an unlisted config, an --integration config, or a bare --run_validations toggle) would turn validation off with no owner lane catching it", line, sortedKeys(allowed)) + } + } + + // bazel.yml: only the two package-gate jobs' bazel-build steps may pass + // --norun_validations (they spell the race build flags out on the + // command line rather than through a .bazelrc --config; T1 pins their + // content). Anywhere else -- a new job, a new step, a non-package-gate + // job -- would turn a lane's validation off unnoticed. + workflow := readCIWorkflow(t, bazelWorkflowName) + for name, j := range workflow.Jobs { + for _, st := range j.Steps { + if !strings.Contains(st.Run, "run_validations") { + continue + } + if !slices.Contains(nogoPackageGateJobs, name) { + stepLabel := st.Name + if stepLabel == "" { + stepLabel = st.Uses + } + t.Errorf("%s job %q step %q: run_validations outside the package-gate build steps", bazelWorkflowName, name, stepLabel) + } + } + } +} diff --git a/scripts/pr_lanes_bazel_coverage_test.go b/scripts/pr_lanes_bazel_coverage_test.go index b53fdfb69..2084b8928 100644 --- a/scripts/pr_lanes_bazel_coverage_test.go +++ b/scripts/pr_lanes_bazel_coverage_test.go @@ -64,6 +64,8 @@ var bazelPRLaneRCLines = map[string][]string{ "test:doltserver --test_arg=-test.parallel=4", "test:doltserver --local_test_jobs=4", "test:doltserver --remote_download_regex=.*/test\\.(log|xml)$", + // F5 S1: the race configuration's nogo is owned by the test lane. + "test:doltserver --norun_validations", }, "pure": { "build:pure --@rules_go//go/config:pure", @@ -318,7 +320,11 @@ func testPackageGateJobs(t *testing.T, prGateRequired []string) { // e2b78f7d7a: both configs cquery to the identical // bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path). buildStep := job.step(t, "bazel build //cmd/bd:bd_for_tests") - if buildStep.If != bazelPathIf || buildStep.Run != `bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$'` { + // F5 S1: the package gates' build is a race-configuration non-owner + // (bd_for_tests is also built, with its nogo already validated, by + // bazel-test's `bazel test //... --config=ci`), so it passes + // --norun_validations too. + if buildStep.If != bazelPathIf || buildStep.Run != `bazel build --@rules_go//go/config:race //cmd/bd:bd_for_tests --remote_download_regex='.*/bin/cmd/bd/bd_for_tests/bd$' --norun_validations` { t.Errorf("%s bazel build step: if %q, run %q; want if %q", lane.job, buildStep.If, buildStep.Run, bazelPathIf) } pkgBD := job.step(t, "Package the Bazel-built bd") diff --git a/scripts/pr_risk_bazel_coverage_test.go b/scripts/pr_risk_bazel_coverage_test.go index 19d524a5d..e22bde3de 100644 --- a/scripts/pr_risk_bazel_coverage_test.go +++ b/scripts/pr_risk_bazel_coverage_test.go @@ -1092,6 +1092,8 @@ var bazelDoltServerRCLines = map[string][]string{ "test:doltserver-proxied --local_test_jobs=4", "test:doltserver-proxied --remote_download_regex=.*/test\\.(log|xml)$", "test:doltserver-proxied --experimental_remote_cache_eviction_retries=0", + // F5 S1: the race configuration's nogo is owned by the test lane. + "test:doltserver-proxied --norun_validations", }, "doltserver-integration": { "build:doltserver-integration --@rules_go//go/config:tags=gms_pure_go,integration", diff --git a/tools/bazel/nogo_ownership.py b/tools/bazel/nogo_ownership.py new file mode 100644 index 000000000..a279310c0 --- /dev/null +++ b/tools/bazel/nogo_ownership.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 +"""Dynamic cross-check for the nogo-ownership policy (scripts/nogo_lint_policy_test.go). + +The static policy tests (T1-T3, T5-T6) pin .bazelrc/bazel.yml *text*: a +config's flags, and which lane owns its validations. This script instead +asks Bazel itself whether the owner's RunNogo action set for a configuration +really is a superset of every non-owner's RunNogo action set for that same +configuration -- i.e. the owner genuinely analyzes every first-party file a +non-owner compiles, so skipping validation on the non-owner loses no lint +coverage. + +Scope (S1): the race configuration group only -- bazel-test (owner, race) +vs. embedded/doltserver/doltserver-proxied/dolt-race (non-owners, which pass +--norun_validations). The integration group (S2) is out of scope until its +lanes gain the same split; see scripts/nogo_lint_policy_test.go's +nogoConfigurations table, which this script's RACE_GROUP mirrors. + +Each config's key-affecting flags (the same fingerprint +scripts/nogo_lint_policy_test.go's nogoKeyFlagPattern extracts) are read +straight out of .bazelrc and passed to `aquery`/`query` directly: those are +"build"-family commands and do not see flags defined under a bare `test:X` +.bazelrc stanza, so `--config=X` itself cannot be used here. + +The owner (`test:ci`) builds //... whole (T3 pins that: no +--build_tests_only, no --build_tag_filters), so its RunNogo set is queried +directly over //.... Every non-owner here instead passes --build_tests_only +with a --test_tag_filters= value, which narrows what `bazel +test` actually builds to deps(the tagged tests) -- querying RunNogo over +//... whole for a non-owner would silently compare the *owner's* universe +against itself and always report full coverage (ga-f5s1-nogo-ownership-1: +this was exactly that bug). So each non-owner's target set here is derived +the same way `bazel test --build_tests_only --test_tag_filters=...` derives +it: the deps closure of `tests(//...)` filtered down to targets carrying one +of the lane's tags, read as an *exact* tag (not a substring -- "dolt-server" +is a prefix of both "dolt-server-proxied" and "dolt-server-cmd", so a naive +substring match over `attr(tags, ...)` would wrongly fold those in; tags are +read from `--output=xml` instead, which lists each tag as its own exact +string, to sidestep that). + +This is advisory and dynamic (runs real `bazel query`/`aquery`, a few +minutes locally): it is intentionally not part of the required `//scripts` +static test suite, which cannot shell out to Bazel from inside a Bazel +action. Run it with `make nogo-ownership`, locally or on demand in a +slice's own PR. + +Exit status: 0 if every non-owner's RunNogo target set is covered by the +owner's, 2 otherwise. Prints the offending targets. +""" +from __future__ import annotations + +import argparse +import json +import re +import subprocess +import sys +import xml.etree.ElementTree as ET +from pathlib import Path + +# (name, owner config, [non-owner config, ...]) for the race group. +# Keep in sync with scripts/nogo_lint_policy_test.go's nogoConfigurations. +RACE_GROUP = ( + "race", + "ci", + ["embedded", "doltserver", "doltserver-proxied", "dolt-race"], +) + +# Mirrors scripts/nogo_lint_policy_test.go's nogoKeyFlagPattern. +KEY_FLAG_RE = re.compile( + r"--@rules_go//go/config:(?:race|pure|tags=\S+)" + r"|--platforms=\S+" + r"|//tools/bazel:release_platforms\S*" +) + +CONFIG_LINE_RE = re.compile(r"^(build|test):([A-Za-z0-9_-]+)\s+(.*)$") +TEST_TAG_FILTERS_RE = re.compile(r"^(?:build|test) --test_tag_filters=(\S+)$") +BUILD_TESTS_ONLY_RE = re.compile(r"^(?:build|test) --build_tests_only$") + + +def bazelrc_lines(rc_text: str, name: str) -> list[str]: + """Every raw ("build"|"test", opt) line for --config=name, in file order.""" + out = [] + for line in rc_text.splitlines(): + line = line.split(" #", 1)[0].rstrip() + m = CONFIG_LINE_RE.match(line) + if m and m.group(2) == name: + out.append(f"{m.group(1)} {m.group(3)}") + return out + + +def expand_config(rc_text: str, name: str, seen: set[str]) -> list[str]: + """Recursively follows nested --config=Y references, as the Go test does.""" + if name in seen: + return [] + seen.add(name) + out = [] + for line in bazelrc_lines(rc_text, name): + for prefix in ("build --config=", "test --config="): + if line.startswith(prefix): + out.extend(expand_config(rc_text, line[len(prefix):], seen)) + break + else: + out.append(line) + return out + + +def fingerprint(rc_text: str, name: str) -> list[str]: + lines = expand_config(rc_text, name, set()) + flags = set() + for line in lines: + flags.update(m.group(0) for m in KEY_FLAG_RE.finditer(line)) + return sorted(flags) + + +def test_tag_filters(rc_text: str, name: str) -> list[str] | None: + """The lane's --test_tag_filters value, as a list of exact tags, if any. + + None means the lane sets no --test_tag_filters (the owner: it builds + //... whole, per T3). A --test_tag_filters value with a leading '-' + (tag exclusion) is not used by any race-group lane today; this raises + rather than silently mis-scoping a future lane that added one. + """ + for line in expand_config(rc_text, name, set()): + m = TEST_TAG_FILTERS_RE.match(line) + if not m: + continue + tags = m.group(1).split(",") + for tag in tags: + if tag.startswith("-"): + raise NotImplementedError( + f"--config={name}: --test_tag_filters={m.group(1)} has an " + "exclusion tag; this script only understands inclusion filters" + ) + return tags + return None + + +def lane_builds_tests_only(rc_text: str, name: str) -> bool: + return any(BUILD_TESTS_ONLY_RE.match(line) for line in expand_config(rc_text, name, set())) + + +def run_query_xml(bazel: str, query: str) -> ET.Element: + cmd = [bazel, "--nohome_rc", "query", query, "--output=xml", "--noimplicit_deps"] + out = subprocess.run(cmd, capture_output=True, text=True, check=True).stdout + return ET.fromstring(out) + + +def tagged_test_targets(bazel: str, tags: list[str]) -> set[str]: + """Labels of tests(//...) carrying at least one of tags, read exactly. + + Tags are read from --output=xml's own + elements rather than matched with attr()'s regex against a stringified + list: "dolt-server" is a prefix of both "dolt-server-proxied" and + "dolt-server-cmd" (and attr()'s per-element regex match is a substring + match, not an exact one), so attr(tags, "dolt-server", //...) also + returns proxied- and cmd-tagged targets -- the exact bug this script + exists to avoid reintroducing one level up. + """ + root = run_query_xml(bazel, "tests(//...)") + wanted = set(tags) + labels = set() + for rule in root.findall("rule"): + rule_tags = { + s.get("value") + for list_el in rule.findall("list[@name='tags']") + for s in list_el.findall("string") + } + if rule_tags & wanted: + labels.add(rule.get("name")) + return labels + + +def lane_universe(bazel: str, targets: set[str] | None) -> str: + """The query-language target expression this lane builds. + + None (the owner, or any lane with no --build_tests_only): //... whole. + Otherwise (a lane that builds only its tagged tests and their deps, as + --build_tests_only + --test_tag_filters does): the deps closure of just + those targets. + """ + if targets is None: + return "//..." + if not targets: + raise ValueError("lane_universe: empty tagged target set") + return "deps(set(" + " ".join(sorted(targets)) + "))" + + +def run_nogo_targets(bazel: str, flags: list[str], universe: str) -> set[str]: + """Returns the set of target labels with a RunNogo action under flags.""" + cmd = [bazel, "--nohome_rc", "aquery", f'mnemonic("RunNogo", {universe})', *flags, + "--output=jsonproto"] + out = subprocess.run(cmd, capture_output=True, text=True, check=True).stdout + data = json.loads(out) + targets_by_id = {t["id"]: t["label"] for t in data.get("targets", [])} + labels = set() + for action in data.get("actions", []): + if action.get("mnemonic") != "RunNogo": + continue + label = targets_by_id.get(action.get("targetId")) + if label and not label.startswith("@"): + labels.add(label) + return labels + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--bazel", default="bazel", help="bazel/bazelisk binary to invoke") + parser.add_argument("--bazelrc", default=".bazelrc") + args = parser.parse_args() + + rc_text = Path(args.bazelrc).read_text() + name, owner, non_owners = RACE_GROUP + + owner_fp = fingerprint(rc_text, owner) + if lane_builds_tests_only(rc_text, owner): + raise NotImplementedError( + f"--config={owner} now passes --build_tests_only, narrowing its target " + "set; this script assumes the race owner builds //... whole " + "(scripts/nogo_lint_policy_test.go's T3). --test_tag_filters alone " + "(exclusion-only, as today's owner has) only narrows test execution, " + "not the build, so it is not checked here." + ) + owner_targets = run_nogo_targets(args.bazel, owner_fp, "//...") + + problems = [] + for non_owner in non_owners: + non_owner_fp = fingerprint(rc_text, non_owner) + if non_owner_fp != owner_fp: + problems.append( + f"{name}: --config={non_owner}'s fingerprint {non_owner_fp} " + f"differs from owner --config={owner}'s {owner_fp}; it is no " + "longer safe to skip its validations" + ) + continue + + tags = test_tag_filters(rc_text, non_owner) + if tags is None or not lane_builds_tests_only(rc_text, non_owner): + # No narrowing this script knows how to replicate: fall back to + # the owner's own universe, which is always a safe (if coarser) + # comparison. + non_owner_universe = "//..." + else: + tagged = tagged_test_targets(args.bazel, tags) + if not tagged: + problems.append( + f"{name}: --config={non_owner}'s --test_tag_filters={','.join(tags)} " + "matches no test in //...; its coverage could not be checked" + ) + continue + non_owner_universe = lane_universe(args.bazel, tagged) + + non_owner_targets = run_nogo_targets(args.bazel, non_owner_fp, non_owner_universe) + missing = non_owner_targets - owner_targets + if missing: + problems.append( + f"{name}: --config={non_owner} compiles {len(missing)} target(s) " + f"with no RunNogo action under the owner --config={owner}: " + + ", ".join(sorted(missing)[:10]) + + (" ..." if len(missing) > 10 else "") + ) + + if problems: + print("nogo_ownership: FAIL", file=sys.stderr) + for p in problems: + print(f" {p}", file=sys.stderr) + return 2 + + print(f"nogo_ownership: OK ({name}: --config={owner} covers " + f"{', '.join(non_owners)})") + return 0 + + +if __name__ == "__main__": + sys.exit(main())