* ci: F5 S2 Variant A nogo ownership for the integration configuration group No full Bazel lane validates the integration / doltserver-integration / doltserver-cmd configurations anymore (all three now pass --norun_validations). Instead bazel-integration gains a narrow "Nogo: integration library files" step that keeps full nogo (vet + golangci-lint) on the configuration's one non-test library package, //internal/testutil/integration, via `bazel build --config=integration --@rules_go//go/config:race --keep_going --output_groups=nogo_fix -- //internal/testutil/integration:integration`. This accepts a scoped coverage loss (f5-spec.md §3/§5 S2, Variant A): the 56 integration-tagged _test.go files (listed in tools/nogo/unvalidated_integration_tests.txt, derived from source by the new T4 test) no longer get a vet pass in any lane, since golangci-lint already skips all _test.go files repo-wide (.golangci.yml's run.tests: false) and no lane builds those test binaries with validations on. Verified directly with real `bazel build` runs: a planted vet finding in the library package still fails the new step; the same finding in an integration-tagged _test.go file no longer fails anything; a planted finding in a plain race-configuration library still fails bazel-test; and GoCompilePkg action keys for the integration configuration are identical with and without --norun_validations (it is a build-request option, not a configuration flag, so no farm cache invalidation). scripts/nogo_lint_policy_test.go: the nogoConfigurations table gains a "integ" row with an ownerIsNogoOnlyStep flag and a dedicated checkNogoOnlyStepRow, generalizing T1/T2/T5/T6 to cover an owner that is a narrow build step rather than a full lane; adds T4 (TestNogoIntegrationStepCoversItsConfiguration), which derives the step's expected target set from source via go/build's own tag matching instead of a hardcoded list. Also fixes four should-fix items from julianknutsen's review of #7482: 1. expandBazelrcConfig now follows `common:` lines and --config= tokens anywhere on a line, not just a first-position `build:`/`test:` one. 2. TestNogoRaceOwnerBuildsEverything now expands every config the owner job can reach (including shell-variable-gated ones like sole-run), not just the statically-named "ci". 3. The package-gate check now compares full key-flag fingerprints instead of substring-matching --@rules_go//go/config:race / --norun_validations. 4. tools/bazel/nogo_ownership.py now fails loudly (exit 2) when both the owner's and non-owners' action-key sets are empty, instead of reporting a vacuous pass. Each fix has new mutation-test coverage in TestNogoPolicyRejectsMutations (should-fixes 1-3) or nogo_ownership_test.py (should-fix 4). engdocs/LINTING.md and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md updated to remove the stale "integration still validates in all three lanes" claim and document the Variant A trade-off. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(ci): make T4's source scan work under Bazel, not just plain go test scripts:scripts_test only declares //:repo_other_files as data (every tracked file but Go and Markdown source, by design: a Go-only change should not re-run the CI/workflow policy tests). TestNogoIntegrationStep CoversItsConfiguration's source scan needs every non-test .go file, so under `bazel test //scripts:scripts_test` its runfiles tree had none and the scan always came back empty. Fixes: - scripts/BUILD.bazel: add //:repo_go_srcs to scripts_test's data, with a comment explaining why (this is the one test in the target that needs it). - integrationOnlyLibraryPackages: use bazelPolicyRoot(t) + repoFiles(t, ...) (the existing Bazel-safe, runfiles-aware file listing other policy tests in this package already use) instead of sourceRepoRoot(t) + filepath.WalkDir, which only sees what //scripts's own go_test target declares. Verified with real `bazel test --config=integration --@rules_go//go/config:race` and `--config=ci --@rules_go//go/config:race` runs of //scripts:scripts_test (both pass); CI caught the original gap (PR #7486's first push failed Bazel / integration and Bazel / test with exactly this). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): address independent review + bee-ghosttrack findings on F5 S2 Variant A Independent review of #7486, must-fixes 1-7: pipefail on the nogo-only step's tee pipeline; exact source-scan equality check for tools/nogo/unvalidated_integration_tests.txt; expandBazelrcConfig follows common:-lines and non-first-position --config= tokens in both the Go policy test and nogo_ownership.py; mirrored nogo_ownership.py should-fix guarding against a non-owner action-key query silently matching nothing. Cost fix: split T4/T6's whole-repo .go source scan into its own go_test target (//scripts:nogo_integration_scan_test) so a Go-only change no longer re-runs all of scripts_test; extracted the shared bazelInvocations and ciWorkflow helpers into helper-only files so both targets can depend on them without violating the one-target-per-test-file partition rule. bee-ghosttrack review: checkNogoOnlyStepRow's own fingerprint now excludes test:-sourced .bazelrc lines pulled in via --config=X (buildOrCommonLines), since the nogo-only step is a `bazel build` invocation that never reads a test: line; added mutation s2i reproducing the gap this closes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(nogo): pin integration nogo-only step's target, fingerprint its lane, add behavioral test Addresses the remaining should-fixes from the independent review of #7486: 1. TestNogoIntegrationStepScriptFailsWithoutMaskingBazelsExitCode (scripts/nogo_integration_step_script_test.go) extracts the real "Nogo: integration library files" step script from bazel.yml and runs it under plain `bash -e` (GitHub's actual run: semantics, no pipefail added) against a stub bazel, closing the gap a text-only pin can't: it catches rc=0 before `exit "$rc"`, `|| :`/`|| true` on the tee pipeline, or a missing/negated `set -o pipefail`, regardless of how the mutation is spelled. 2. The nogo-only step's target set is now pinned by exact go_library label, not just package: nogo_integration_scan_test.go derives the label(s) from source + BUILD.bazel and nogo_lint_policy_test.go pins them literally (its own go_test target has no //:repo_go_srcs). Both reject a `-//` exclusion pattern outright. 3. checkNonOwnerCommandLineAddsNoKeyFlag is now also applied to the integ row's owner's own full lane: under Variant A that lane validates nothing, so it is effectively a non-owner for this check's purpose, and must not override a key-affecting flag on its command line either. 4. Latent items: TestNoUnconditionalValidationOff now also scans .github/actions/setup-bazel/write-bazelrc.sh for run_validations (every line it emits lacks a :config suffix, so any occurrence would turn validation off everywhere with no owner lane to catch it). tools/bazel/nogo_ownership.py's docstring is corrected: it had said S2 was simply "out of scope," which is now stale since S2 has its own validating lane (the nogo-only step, checked statically); the script's dynamic aquery-based cross-check remains S1-only because S2's owner has no non-owner test lane building the same configuration to compare actions against. Skipped (not cheap / not reachable): - bazel-server-storage's `if:` not modeling an "integration" input: no such input exists to model; the gate is RBE-enabled-only by design, and resolving this requires understanding pr.yml's broader gating semantics, not a textual fix. - defaults.run.shell parsing: bazel.yml has no `defaults:` block today (confirmed by grep), so there is nothing to parse; the gap is latent only for a future change, not a live bug. - race-off scan's future-wrong-fix edge case: the review itself characterizes this as not reachable with today's file set. New mutation cases s2j/s2k/s2l in TestNogoPolicyRejectsMutations cover items 2 and 3. All mutation-verifications performed by hand-editing the real files, confirming FAIL, and reverting via Edit (never git stash/checkout). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
318 lines
14 KiB
Go
318 lines
14 KiB
Go
package scripts_test
|
|
|
|
// T4 (TestNogoIntegrationStepCoversItsConfiguration), the integration/test
|
|
// file source scans it shares with T6
|
|
// (TestUnvalidatedIntegrationTestsFileIsExact), and the scan's own unit test
|
|
// live in their own file and go_test target (cost fix, independent review of
|
|
// #7486): both scans read //:repo_go_srcs and //:repo_go_test_srcs (every
|
|
// tracked .go file, test and non-test), so putting them in scripts_test
|
|
// alongside everything else made ANY Go source change re-run the whole
|
|
// nogo-policy suite (scripts_test's rerun rate rose from 73% to 91% of main
|
|
// commits). This target alone depends on those two data partitions; nothing
|
|
// else in scripts/BUILD.bazel needs to.
|
|
|
|
import (
|
|
"fmt"
|
|
"go/build"
|
|
"os"
|
|
"path/filepath"
|
|
"regexp"
|
|
"slices"
|
|
"sort"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// goLibraryNamePattern finds a go_library rule's own "name" attribute,
|
|
// assumed to be the first go_library block in the file (every scanned
|
|
// package here is a single-package Go directory with at most one
|
|
// non-test go_library). Used by integrationOnlyLibraryTargets (independent
|
|
// review of #7486, must-fix 2) to pin the nogo-only step's exact target
|
|
// label, not just its package: a package-only comparison would accept the
|
|
// step retargeted at any other rule in the same package (e.g. the
|
|
// package's :repo_doc_files filegroup), silently validating nothing.
|
|
var goLibraryNamePattern = regexp.MustCompile(`go_library\(\s*\n\s*name = "([^"]+)"`)
|
|
|
|
// integrationOnlyLibraryTargets pairs each of integrationOnlyLibraryPackages'
|
|
// packages with its go_library target name, read from the package's own
|
|
// BUILD.bazel, returning full Bazel labels ("//dir:name"). A bare package
|
|
// label does not pin which target within it the nogo-only step must build.
|
|
func integrationOnlyLibraryTargets(t *testing.T, root string) []string {
|
|
t.Helper()
|
|
var out []string
|
|
for _, pkg := range integrationOnlyLibraryPackages(t, root) {
|
|
dir := strings.TrimPrefix(pkg, "//")
|
|
build := readPolicyFile(t, root, filepath.Join(filepath.FromSlash(dir), "BUILD.bazel"))
|
|
m := goLibraryNamePattern.FindStringSubmatch(build)
|
|
if m == nil {
|
|
t.Fatalf("%s has no go_library rule in its BUILD.bazel; integrationOnlyLibraryPackages found it by source files, but there is no target to pin a label for", pkg)
|
|
}
|
|
out = append(out, pkg+":"+m[1])
|
|
}
|
|
sort.Strings(out)
|
|
return out
|
|
}
|
|
|
|
// T4 TestNogoIntegrationStepCoversItsConfiguration: f5-spec.md §6 T4
|
|
// (Variant A). bazel-integration's nogo-only step's target set must be
|
|
// exactly the integration configuration's non-test library packages, found
|
|
// by a source scan (go/build's own tag matching, diffed with and without
|
|
// the "integration" tag) rather than hardcoded, so a new integration-tagged
|
|
// library file lands in the nogo-only step automatically instead of
|
|
// silently losing coverage, and a file that stops needing the tag drops
|
|
// out instead of the step validating a target gazelle no longer builds that
|
|
// way.
|
|
//
|
|
// The comparison is by exact label ("//pkg:name"), not package alone, and a
|
|
// "-//" exclusion pattern on the step's command line is rejected outright,
|
|
// as TestNogoRaceOwnerBuildsEverything (T3) already does for the race
|
|
// owner's own target set (independent review of #7486, must-fix 2): a
|
|
// package-only match would accept the step retargeted at a different rule
|
|
// in the same package (e.g. :repo_doc_files), or let a "-//..." pattern
|
|
// silently exclude a target the source scan found, with nothing here
|
|
// noticing either way.
|
|
func TestNogoIntegrationStepCoversItsConfiguration(t *testing.T) {
|
|
root := bazelPolicyRoot(t)
|
|
workflow := readCIWorkflow(t, bazelWorkflowName)
|
|
job := workflow.job(t, bazelIntegJobName)
|
|
|
|
var nogoArgs []string
|
|
for _, st := range job.Steps {
|
|
for _, args := range bazelInvocations(st.Run, "build") {
|
|
if slices.Contains(args, "--output_groups=nogo_fix") {
|
|
nogoArgs = args
|
|
}
|
|
}
|
|
}
|
|
if nogoArgs == nil {
|
|
t.Fatalf("%s has no `bazel build --output_groups=nogo_fix` step", bazelIntegJobName)
|
|
}
|
|
|
|
got := map[string]bool{}
|
|
for _, a := range nogoArgs {
|
|
switch {
|
|
case strings.HasPrefix(a, "-//"):
|
|
t.Errorf("%s's nogo-only step excludes a target (%s) with a negative pattern; every target the source scan found must be built, not excluded", bazelIntegJobName, a)
|
|
case strings.HasPrefix(a, "//"):
|
|
got[a] = true
|
|
}
|
|
}
|
|
|
|
want := integrationOnlyLibraryTargets(t, root)
|
|
if len(want) == 0 {
|
|
t.Fatal("source scan found no integration-tagged library target; the scan itself is broken")
|
|
}
|
|
for _, target := range want {
|
|
if !got[target] {
|
|
t.Errorf("source scan found integration-only go_library target %s with no exact match in %s's nogo-only step (targets: %v)", target, bazelIntegJobName, nogoArgs)
|
|
}
|
|
}
|
|
for target := range got {
|
|
if !slices.Contains(want, target) {
|
|
t.Errorf("%s's nogo-only step targets %s, which is not one of the exact go_library labels the source scan found (scan found: %v); a retargeted label (even one in the same package) would silently stop validating the real target", bazelIntegJobName, target, want)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestIntegrationOnlyLibraryPackagesScanConstraints (independent review of
|
|
// #7486, must-fix 3 and the bee-ghosttrack nit): unit-tests
|
|
// scanIntegrationOnlyLibraryPackages' constraint handling directly, against
|
|
// a small synthetic file tree, rather than only indirectly through whatever
|
|
// the real repository happens to contain today.
|
|
func TestIntegrationOnlyLibraryPackagesScanConstraints(t *testing.T) {
|
|
root := t.TempDir()
|
|
write := func(rel, constraint string) {
|
|
dir := filepath.Join(root, filepath.FromSlash(filepath.Dir(rel)))
|
|
if err := os.MkdirAll(dir, 0o755); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
pkg := filepath.Base(filepath.Dir(rel))
|
|
src := fmt.Sprintf("package %s\n", pkg)
|
|
if constraint != "" {
|
|
src = fmt.Sprintf("//go:build %s\n\n%s", constraint, src)
|
|
}
|
|
if err := os.WriteFile(filepath.Join(root, filepath.FromSlash(rel)), []byte(src), 0o644); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
|
|
// cgo: only shows up once CgoEnabled is true (must-fix 3).
|
|
write("cgocase/f.go", "cgo && integration")
|
|
// race: only shows up once the race-on scan's tags include "race"
|
|
// (must-fix 3).
|
|
write("racecase/f.go", "race && integration")
|
|
// racezero: never compiled by any of the "integ" row's members (they all
|
|
// set race=true), but still a real integration-tagged library file no
|
|
// lane validates; only the race-off scan surfaces it (must-fix 3).
|
|
write("racezerocase/f.go", "!race && integration")
|
|
// gmsonly: requires gms_pure_go but not integration; must NOT be
|
|
// reported, or the "without" context's missing gms_pure_go tag would
|
|
// make it look integration-only (bee-ghosttrack nit on #7486).
|
|
write("gmsonlycase/f.go", "gms_pure_go && !integration")
|
|
// plain: no constraint at all; compiles either way, must NOT be
|
|
// reported.
|
|
write("plaincase/f.go", "")
|
|
|
|
files := []string{
|
|
"cgocase/f.go", "racecase/f.go", "racezerocase/f.go",
|
|
"gmsonlycase/f.go", "plaincase/f.go",
|
|
}
|
|
got := scanIntegrationOnlyLibraryPackages(root, files)
|
|
|
|
want := []string{"//cgocase", "//racecase", "//racezerocase"}
|
|
sort.Strings(want)
|
|
if !equalStrings(got, want) {
|
|
t.Errorf("scanIntegrationOnlyLibraryPackages = %v, want %v", got, want)
|
|
}
|
|
}
|
|
|
|
// integrationOnlyLibraryPackages scans root for non-test .go files that
|
|
// compile only when the "integration" build tag is set (go/build.Context's
|
|
// MatchFile, diffed with and without the tag), and returns their Bazel
|
|
// package labels ("//dir", repo root relative), sorted and deduplicated. A
|
|
// file that compiles either way (no integration constraint), or that exists
|
|
// only as a _test.go file, is not included: a _test.go file is not a
|
|
// library's ValidateNogo target either way (f5-spec.md §3's accepted loss).
|
|
//
|
|
// Uses repoFiles (//:repo_go_srcs under Bazel, BUILD.bazel's scripts_test
|
|
// data) rather than its own filepath.WalkDir: under Bazel, root is a
|
|
// runfiles symlink forest holding only what the go_test declares as data,
|
|
// not an arbitrary subtree of the checkout.
|
|
func integrationOnlyLibraryPackages(t *testing.T, root string) []string {
|
|
t.Helper()
|
|
return scanIntegrationOnlyLibraryPackages(root, repoFiles(t, root, 500))
|
|
}
|
|
|
|
// scanIntegrationOnlyLibraryPackages is integrationOnlyLibraryPackages' pure
|
|
// scan logic, kept apart so TestIntegrationOnlyLibraryPackagesScanConstraints
|
|
// can feed it a small synthetic file list instead of the whole repository.
|
|
//
|
|
// It scans twice, with race on and with race off, and returns the union:
|
|
// every member of nogoConfigurations' "integ" row sets
|
|
// --@rules_go//go/config:race (so a file gated `race && integration` is only
|
|
// ever actually compiled that way, and matching race=off for it would miss
|
|
// it), but a file gated `!race && integration` would never be compiled by
|
|
// any of that row's members at all -- an uncovered configuration no lane
|
|
// validates -- and the race-off scan is what surfaces it here instead of
|
|
// silently losing it (independent review of #7486, must-fix 3). Each scan's
|
|
// own "without" context carries the other scan's non-integration tags
|
|
// (gms_pure_go, and race for the race-on scan) so the with/without diff
|
|
// isolates the integration tag alone: an earlier version's "without" had no
|
|
// tags at all, so a file gated on gms_pure_go alone (no integration
|
|
// constraint) would have shown up as integration-only too (review of #7486,
|
|
// bee-ghosttrack nit). CgoEnabled is true in every context: race requires
|
|
// cgo, so a file additionally gated `cgo && integration` only shows up with
|
|
// it set (independent review of #7486, must-fix 3).
|
|
func scanIntegrationOnlyLibraryPackages(root string, files []string) []string {
|
|
return scanIntegrationTaggedFiles(root, files, false)
|
|
}
|
|
|
|
// integrationOnlyTestFiles is scanIntegrationOnlyLibraryPackages' mirror
|
|
// image: _test.go files (instead of library files) that require the
|
|
// integration build tag, returned as sorted repo-relative paths (instead of
|
|
// Bazel package labels, since many such files share a package and T6 of
|
|
// must-fix 6 (independent review of #7486) checks the exact file list, not
|
|
// the package set). This is tools/nogo/unvalidated_integration_tests.txt's
|
|
// own source of truth: TestUnvalidatedIntegrationTestsFileIsExact pins the
|
|
// file's content equal to it, so a newly integration-tagged test file lands
|
|
// there automatically instead of silently losing coverage with nothing
|
|
// making the loss explicit.
|
|
func integrationOnlyTestFiles(t *testing.T, root string) []string {
|
|
t.Helper()
|
|
return scanIntegrationTaggedFiles(root, repoFiles(t, root, 500), true)
|
|
}
|
|
|
|
// scanIntegrationTaggedFiles is integrationOnlyLibraryPackages' and
|
|
// integrationOnlyTestFiles' shared pure scan logic: see
|
|
// scanIntegrationOnlyLibraryPackages' doc comment for the race-on/race-off
|
|
// union and CgoEnabled rationale (independent review of #7486, must-fix 3).
|
|
// testFiles selects which half of the source tree is scanned (non-test
|
|
// library files, or _test.go files) and what each match is reported as
|
|
// (a Bazel package label, or a repo-relative file path).
|
|
func scanIntegrationTaggedFiles(root string, files []string, testFiles bool) []string {
|
|
scan := func(raceOn bool) []string {
|
|
base := []string{"gms_pure_go"}
|
|
if raceOn {
|
|
base = append(base, "race")
|
|
}
|
|
withTags := append(append([]string{}, base...), "integration")
|
|
without := &build.Context{GOOS: "linux", GOARCH: "amd64", Compiler: "gc", CgoEnabled: true, BuildTags: base}
|
|
with := &build.Context{GOOS: "linux", GOARCH: "amd64", Compiler: "gc", CgoEnabled: true, BuildTags: withTags}
|
|
|
|
var out []string
|
|
for _, rel := range files {
|
|
if !strings.HasSuffix(rel, ".go") || strings.HasSuffix(rel, "_test.go") != testFiles {
|
|
continue
|
|
}
|
|
dir := filepath.ToSlash(filepath.Dir(rel))
|
|
name := filepath.Base(rel)
|
|
dirPath := filepath.Join(root, filepath.FromSlash(dir)) + string(filepath.Separator)
|
|
matchesWithout, errW := without.MatchFile(dirPath, name)
|
|
matchesWith, errT := with.MatchFile(dirPath, name)
|
|
if errW != nil || errT != nil {
|
|
continue // an unparsable file is not this scan's concern
|
|
}
|
|
if !matchesWith || matchesWithout {
|
|
continue
|
|
}
|
|
if testFiles {
|
|
out = append(out, rel)
|
|
} else {
|
|
out = append(out, "//"+dir)
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
seen := map[string]bool{}
|
|
var out []string
|
|
for _, pkg := range append(scan(true), scan(false)...) {
|
|
if !seen[pkg] {
|
|
seen[pkg] = true
|
|
out = append(out, pkg)
|
|
}
|
|
}
|
|
sort.Strings(out)
|
|
return out
|
|
}
|
|
|
|
// TestUnvalidatedIntegrationTestsFileIsExact (independent review of #7486,
|
|
// must-fix 6): tools/nogo/unvalidated_integration_tests.txt -- f5-spec.md
|
|
// §3's "lost_2a.txt", the list of integration-tagged _test.go files no nogo
|
|
// lane validates under Variant A -- was otherwise unread by any test: a file
|
|
// could drift arbitrarily out of date with no check noticing. This pins it
|
|
// exactly equal to the same source scan TestNogoIntegrationStepCoversItsConfiguration
|
|
// uses, so a newly integration-tagged test file (a new loss of coverage)
|
|
// fails here explicitly instead of silently not being tracked anywhere.
|
|
func TestUnvalidatedIntegrationTestsFileIsExact(t *testing.T) {
|
|
root := bazelPolicyRoot(t)
|
|
const path = "tools/nogo/unvalidated_integration_tests.txt"
|
|
var got []string
|
|
for _, line := range strings.Split(readPolicyFile(t, root, path), "\n") {
|
|
line = strings.TrimSpace(line)
|
|
if line == "" || strings.HasPrefix(line, "#") {
|
|
continue
|
|
}
|
|
got = append(got, line)
|
|
}
|
|
sort.Strings(got)
|
|
|
|
want := integrationOnlyTestFiles(t, root)
|
|
if len(want) == 0 {
|
|
t.Fatal("source scan found no integration-tagged _test.go file; the scan itself is broken")
|
|
}
|
|
if !equalStrings(got, want) {
|
|
var missing, extra []string
|
|
for _, f := range want {
|
|
if !slices.Contains(got, f) {
|
|
missing = append(missing, f)
|
|
}
|
|
}
|
|
for _, f := range got {
|
|
if !slices.Contains(want, f) {
|
|
extra = append(extra, f)
|
|
}
|
|
}
|
|
t.Errorf("%s does not match the source scan of integration-tagged _test.go files:\n missing (scan found, file does not list): %v\n extra (file lists, scan did not find): %v", path, missing, extra)
|
|
}
|
|
}
|