ci: fix red nightly (Bazel-skip allowlist, credential helper init); rerun-safe Bazel artifacts (#7096)

* fix(ci): allowlist the scripts tests that skip under Bazel, and check new ones before merge

nightly.yml's skip-parity check (equivalence.py --go-test-json) went red
on 2026-10-02 (run 36955306415): TestBazelGatedLanesNeverRetryFlakyTests
and TestCheckShardCoverageScript pass under go test and deliberately
t.Skip under Bazel (their inputs are not in //scripts:scripts_test's
runfiles), but had no `skip` allowlist entry. Step 2's
TestShardScriptsListOnlyRealTests and TestShardScriptNarrowingCheck, now
on main, have the same shape and would have turned the next nightly red.

Add the four entries, and TestBazelOnlySkipsAreAllowlisted: it parses
every _test.go file of the module and requires each top-level test whose
t.Skip is guarded by TEST_SRCDIR or bazeltest.IsBazel() (directly or
through a helper) to have an entry, so a new one fails under go test
before the merge instead of in the next nightly. It skips under Bazel
itself and is allowlisted too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(test): keep cmd/bd's integration init out of the credential-command helper process

nightly.yml's Full Test Suite (go test -tags=integration ./...) failed
TestApplyInitGatewayCredentialAdoptsToken and ...FailsClosed on
2026-10-02 (run 36955306415): credentialcmd re-execs the test binary as
the credential helper, and cli_fast_test.go's integration-tagged init()
runs in that child before TestMain can dispatch it, tries `go build .`
from the helper's working directory and panics, so the helper exits 2.
Add credentialcmd.IsHelperProcess (Dispatch's two gates) and return from
that init early, as it already does for the worktree-remove helper.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): overwrite fixed-name artifacts so "Re-run failed jobs" does not 409

A job re-run is the same workflow run, and upload-artifact v4+ refuses a
second artifact of the same name there (409 Conflict). bazel-test uploads
the bd package even after its tests fail, so re-running a failed Bazel /
test lane, the documented recovery for an eviction or a flake, failed on
the upload even when every test passed. Set overwrite: true on all eight
uploads in bazel.yml (the bd package, BUILD sync patch, every lane's test
logs) and policy-test it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(test): keep skip-scan fixture functions out of line-based test discovery

The fixture source in bazel_skip_allowlist_test.go had top-level
`func TestXxx(t *testing.T)` lines, which equivalence.py (and the CI shard
scripts) count as tests of the scripts package; write them with a FUNC
placeholder and substitute at write time.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-10-02 05:42:05 -07:00 • committed by GitHub
commit fe6bd6301f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 459 additions and 4 deletions

View file

@ -402,6 +402,7 @@ jobs:
if: ${{ always() && steps.bd-artifact.outcome == 'success' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: ${{ inputs.build-artifact-name || 'ci-build-artifacts' }}
path: ${{ runner.temp }}/bd-artifacts/
retention-days: 1
@ -485,6 +486,7 @@ jobs:
if: ${{ always() && steps.sync.outcome == 'failure' && github.event_name != 'pull_request_target' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: bazel-sync-patch
path: ${{ runner.temp }}/bazel-sync-patch/
if-no-files-found: ignore
@ -494,6 +496,7 @@ jobs:
if: ${{ failure() && steps.test.outcome == 'failure' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: bazel-testlogs
path: |
bazel-testlogs/**/test.log
@ -766,6 +769,7 @@ jobs:
if: ${{ failure() && steps.test.outcome != 'skipped' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: bazel-embedded-testlogs
path: |
bazel-testlogs/**/test.log
@ -858,6 +862,7 @@ jobs:
if: ${{ failure() && steps.test.outcome != 'skipped' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: bazel-integration-testlogs
path: |
bazel-testlogs/**/test.log
@ -936,6 +941,7 @@ jobs:
if: ${{ failure() && steps.test.outcome == 'failure' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: bazel-${{ env.BAZEL_DOLT_LANE }}-testlogs
path: |
bazel-testlogs/**/test.log
@ -1043,6 +1049,7 @@ jobs:
if: ${{ failure() && steps.test.outcome != 'skipped' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: bazel-proxied-testlogs
path: |
bazel-testlogs/**/test.log
@ -1152,6 +1159,7 @@ jobs:
if: ${{ failure() && steps.test.outcome != 'skipped' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
overwrite: true # a job re-run is the same run: replace, not 409
name: bazel-server-storage-testlogs
path: |
bazel-testlogs/**/test.log

View file

@ -16,6 +16,7 @@ import (
"time"
"github.com/steveyegge/beads/internal/testutil/bazeltest"
"github.com/steveyegge/beads/internal/testutil/credentialcmd"
)
// Fast CLI tests converted from scripttest suite
@ -1015,6 +1016,12 @@ func init() {
if os.Getenv(worktreeRemoveHelperEnv) != "" {
return
}
// Likewise the credential-command helper (internal/testutil/credentialcmd):
// this init runs before TestMain dispatches it, in the helper's cwd, where
// the `go build .` fallback panics and the fixture reports exit 2.
if credentialcmd.IsHelperProcess() {
return
}
// Prebuilt fast path (scripts/test.sh and CI export this), else build
// once. No repo-root ./bd reuse: a stale checkout-root binary silently
// substitutes itself for the source under test (wy-4mtr0).

View file

@ -52,18 +52,33 @@ type preparedFixture struct {
// without the sentinel is simply not ours, so it falls through to its own
// TestMain rather than being refused.
func Dispatch() (int, bool) {
if os.Getenv(helperProcessEnv) != "1" {
return 0, false
}
for i, arg := range os.Args {
if arg == helperSentinel {
return runProtocol(os.Args[i+1:], os.Stdout, os.Stderr), true
}
if i := helperArgIndex(); i >= 0 {
return runProtocol(os.Args[i+1:], os.Stdout, os.Stderr), true
}
return 0, false
}
// IsHelperProcess reports whether this process is a credential helper
// re-exec (the same two gates as Dispatch). Package init functions, which run
// before TestMain can call Dispatch, use it to skip suite setup the helper
// never needs: cmd/bd's integration-tagged init would otherwise try to build
// bd from the helper's working directory and panic.
func IsHelperProcess() bool {
return helperArgIndex() >= 0
}
func helperArgIndex() int {
if os.Getenv(helperProcessEnv) != "1" {
return -1
}
for i, arg := range os.Args {
if arg == helperSentinel {
return i
}
}
return -1
}
// Constructor contract — Emit, Exit23 and Marker all share it, and it is more
// than "returns a string":
//

View file

@ -20,6 +20,7 @@ go_test(
"bazel_autofix_push_test.go",
"bazel_farm_workflow_test.go",
"bazel_policy_test.go",
"bazel_skip_allowlist_test.go",
"check_build_tags_test.go",
"check_doc_freshness_test.go",
"check_testing_short_test.go",

View file

@ -0,0 +1,390 @@
package scripts_test
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path"
"path/filepath"
"regexp"
"sort"
"strings"
"testing"
)
// Every Go test that deliberately skips itself under Bazel (a t.Skip guarded
// by TEST_SRCDIR or bazeltest.IsBazel(), directly or through a helper it
// calls) has a `skip` entry in tools/bazel/equivalence_allowlist.txt.
//
// The PR run of the Bazel lane cannot tell a test that skips under Bazel from
// one that runs: only nightly.yml's skip-parity check (equivalence.py
// --go-test-json) can, after the merge, and a missing entry turns that
// nightly red (as TestBazelGatedLanesNeverRetryFlakyTests and
// TestCheckShardCoverageScript did on 2026-10-02). The entries are also what
// keeps those tests running under go test where the Bazel lane replaces a
// go test job. This check finds them before the merge, from the source.
//
// Only top-level tests are considered: a subtest's skip is invisible to
// equivalence.py, which compares top-level tests.
func TestBazelOnlySkipsAreAllowlisted(t *testing.T) {
if os.Getenv("TEST_SRCDIR") != "" {
t.Skip("walks every _test.go file in the source checkout; runs under go test")
}
root := sourceRepoRoot(t)
entries := readSkipAllowlist(t, filepath.Join(root, "tools", "bazel", "equivalence_allowlist.txt"))
found, err := bazelSkippingTests(root)
if err != nil {
t.Fatal(err)
}
if len(found) < 8 {
t.Fatalf("found only %d Bazel-skipping tests (%v); the scan is broken", len(found), found)
}
for _, f := range found {
if !skipAllowlisted(entries, f.pkg, f.test) {
t.Errorf("%s %s skips under Bazel (%s) but tools/bazel/equivalence_allowlist.txt has no `%s %s skip # why` entry",
f.pkg, f.test, f.where, f.pkg, f.test)
}
}
}
// The scan itself, on fixtures: direct guards of either form, a skip helper,
// and the shapes it must not report.
func TestBazelSkippingTestsScan(t *testing.T) {
dir := t.TempDir()
write := func(rel, src string) {
p := filepath.Join(dir, rel)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(src), 0o644); err != nil {
t.Fatal(err)
}
}
write("go.mod", "module example.com/m\n")
write("a/a_test.go", fixtureFuncs(`package a
import (
"os"
"testing"
"example.com/m/bazeltest"
)
FUNC skipUnderBazel(t *testing.T) {
if bazeltest.IsBazel() {
t.Skip("no")
}
}
FUNC TestEnvGuard(t *testing.T) {
if os.Getenv("TEST_SRCDIR") != "" {
t.Skip("runfiles")
}
}
FUNC TestIsBazelGuard(t *testing.T) {
if bazeltest.IsBazel() {
t.Skipf("%s", "x")
}
}
FUNC TestViaHelper(t *testing.T) {
skipUnderBazel(t)
}
FUNC TestLaterGuard(t *testing.T) {
_ = 1
if srcdir := os.Getenv("TEST_SRCDIR"); srcdir != "" {
t.SkipNow()
}
}
FUNC TestGoTestOnlySkip(t *testing.T) {
if os.Getenv("TEST_SRCDIR") == "" {
t.Skip("only under go test")
}
}
FUNC TestNotBazel(t *testing.T) {
if !bazeltest.IsBazel() {
t.Skip("go test only")
}
}
FUNC TestSubtestOnly(t *testing.T) {
t.Run("x", func(t *testing.T) {
if bazeltest.IsBazel() {
t.Skip("subtest")
}
})
}
FUNC TestNoSkip(t *testing.T) {
if bazeltest.IsBazel() {
return
}
}
FUNC helperTakingM(m *testing.M) {}
`))
write("node_modules/x/x_test.go", "package x\nimport \"testing\"\nfunc TestIgnored(t *testing.T) { if bazeltest.IsBazel() { t.Skip() } }\n")
write("sub/go.mod", "module example.com/sub\n")
write("sub/s_test.go", "package s\nimport \"testing\"\nfunc TestOtherModule(t *testing.T) { if bazeltest.IsBazel() { t.Skip() } }\n")
found, err := bazelSkippingTests(dir)
if err != nil {
t.Fatal(err)
}
var got []string
for _, f := range found {
got = append(got, f.pkg+" "+f.test)
}
sort.Strings(got)
want := []string{"a TestEnvGuard", "a TestIsBazelGuard", "a TestLaterGuard", "a TestViaHelper"}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Errorf("scan found %v, want %v", got, want)
}
entries := []skipEntry{{"a", "TestEnv*"}, {"a", "TestIsBazelGuard"}}
for test, want := range map[string]bool{"TestEnvGuard": true, "TestIsBazelGuard": true, "TestViaHelper": false} {
if got := skipAllowlisted(entries, "a", test); got != want {
t.Errorf("skipAllowlisted(a, %s) = %v, want %v", test, got, want)
}
}
}
type bazelSkip struct{ pkg, test, where string }
type skipEntry struct{ pkg, test string }
// readSkipAllowlist: the allowlist's `skip` entries, in equivalence.py's
// format (<package dir> <test glob> skip # why).
func readSkipAllowlist(t *testing.T, file string) []skipEntry {
t.Helper()
data, err := os.ReadFile(file)
if err != nil {
t.Fatal(err)
}
var out []skipEntry
for _, line := range strings.Split(string(data), "\n") {
body, _, _ := strings.Cut(line, "#")
f := strings.Fields(body)
if len(f) == 3 && f[2] == "skip" {
out = append(out, skipEntry{f[0], f[1]})
}
}
return out
}
// skipAllowlisted matches like equivalence.py's allowed() (fnmatch, which
// path.Match equals for these names).
func skipAllowlisted(entries []skipEntry, pkg, test string) bool {
for _, e := range entries {
pm, _ := path.Match(e.pkg, pkg)
tm, _ := path.Match(e.test, test)
if pm && tm {
return true
}
}
return false
}
var bazelGuardTestFunc = regexp.MustCompile(`^Test([^a-z].*)?$`)
// bazelSkippingTests parses every _test.go file of the module rooted at
// root (not nested modules, node_modules, .git, .beads or bazel-* output
// trees) and returns the top-level tests that skip under Bazel.
func bazelSkippingTests(root string) ([]bazelSkip, error) {
byDir := map[string][]*ast.File{}
fset := token.NewFileSet()
err := filepath.WalkDir(root, func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
name := d.Name()
if p != root && (name == ".git" || name == "node_modules" || name == ".beads" || strings.HasPrefix(name, "bazel-")) {
return filepath.SkipDir
}
if p != root {
if _, err := os.Stat(filepath.Join(p, "go.mod")); err == nil {
return filepath.SkipDir
}
}
return nil
}
if !strings.HasSuffix(p, "_test.go") || d.Type()&os.ModeSymlink != 0 {
return nil
}
f, err := parser.ParseFile(fset, p, nil, 0)
if err != nil {
return err
}
byDir[filepath.Dir(p)] = append(byDir[filepath.Dir(p)], f)
return nil
})
if err != nil {
return nil, err
}
var out []bazelSkip
for dir, files := range byDir {
rel, _ := filepath.Rel(root, dir)
pkg := filepath.ToSlash(rel)
// Package-level helpers (per package name: _test packages share the
// directory) that skip under Bazel.
helpers := map[string]bool{}
for _, f := range files {
for _, decl := range f.Decls {
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Body != nil && hasBazelSkip(fn.Body) {
helpers[f.Name.Name+"."+fn.Name.Name] = true
}
}
}
for _, f := range files {
for _, decl := range f.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Recv != nil || fn.Body == nil || !bazelGuardTestFunc.MatchString(fn.Name.Name) || !takesTestingT(fn) {
continue
}
where := ""
if hasBazelSkip(fn.Body) {
where = "guarded t.Skip"
} else if h := calledHelper(fn.Body, f.Name.Name, helpers); h != "" {
where = "via " + h
}
if where != "" {
out = append(out, bazelSkip{pkg, fn.Name.Name, where})
}
}
}
}
sort.Slice(out, func(i, j int) bool { return out[i].pkg+" "+out[i].test < out[j].pkg+" "+out[j].test })
return out, nil
}
func takesTestingT(fn *ast.FuncDecl) bool {
params := fn.Type.Params.List
if len(params) != 1 {
return false
}
star, ok := params[0].Type.(*ast.StarExpr)
if !ok {
return false
}
sel, ok := star.X.(*ast.SelectorExpr)
return ok && sel.Sel.Name == "T"
}
// hasBazelSkip: an if statement outside any closure whose condition is
// true under Bazel (TEST_SRCDIR non-empty, or bazeltest.IsBazel()) and
// whose body calls a Skip method.
func hasBazelSkip(body *ast.BlockStmt) bool {
found := false
ast.Inspect(body, func(n ast.Node) bool {
if found {
return false
}
switch n := n.(type) {
case *ast.FuncLit:
return false
case *ast.IfStmt:
if bazelCond(n.Cond) && callsSkip(n.Body) {
found = true
return false
}
}
return true
})
return found
}
func bazelCond(cond ast.Expr) bool {
switch c := cond.(type) {
case *ast.ParenExpr:
return bazelCond(c.X)
case *ast.CallExpr:
sel, ok := c.Fun.(*ast.SelectorExpr)
return ok && sel.Sel.Name == "IsBazel"
case *ast.BinaryExpr:
if c.Op == token.LAND {
return bazelCond(c.X) || bazelCond(c.Y)
}
if c.Op != token.NEQ {
return false
}
lit, ok := c.Y.(*ast.BasicLit)
if !ok || lit.Value != `""` {
return false
}
return mentionsTestSrcdir(c.X)
}
return false
}
// mentionsTestSrcdir: os.Getenv("TEST_SRCDIR") itself, or a variable (the
// `if srcdir := os.Getenv("TEST_SRCDIR"); srcdir != ""` form; such a
// variable is only ever bound that way in these tests).
func mentionsTestSrcdir(e ast.Expr) bool {
switch x := e.(type) {
case *ast.CallExpr:
for _, a := range x.Args {
if lit, ok := a.(*ast.BasicLit); ok && lit.Value == `"TEST_SRCDIR"` {
return true
}
}
case *ast.Ident:
if x.Obj != nil {
if as, ok := x.Obj.Decl.(*ast.AssignStmt); ok && len(as.Rhs) == 1 {
return mentionsTestSrcdir(as.Rhs[0])
}
}
}
return false
}
func callsSkip(body *ast.BlockStmt) bool {
found := false
ast.Inspect(body, func(n ast.Node) bool {
if _, ok := n.(*ast.FuncLit); ok {
return false
}
if call, ok := n.(*ast.CallExpr); ok {
if sel, ok := call.Fun.(*ast.SelectorExpr); ok && (sel.Sel.Name == "Skip" || sel.Sel.Name == "Skipf" || sel.Sel.Name == "SkipNow") {
found = true
}
}
return !found
})
return found
}
// calledHelper: a skip helper of the same package that the test calls
// outside any closure.
func calledHelper(body *ast.BlockStmt, pkgName string, helpers map[string]bool) string {
name := ""
ast.Inspect(body, func(n ast.Node) bool {
if name != "" {
return false
}
if _, ok := n.(*ast.FuncLit); ok {
return false
}
if call, ok := n.(*ast.CallExpr); ok {
if id, ok := call.Fun.(*ast.Ident); ok && helpers[pkgName+"."+id.Name] {
name = id.Name
}
}
return true
})
return name
}
// fixtureFuncs turns the "FUNC " placeholders of fixture source back into
// "func ": written literally, the fixture's `func TestXxx(t *testing.T)` lines
// would be counted as real tests of this package by line-based test discovery
// (tools/bazel/equivalence.py, the CI shard scripts).
func fixtureFuncs(src string) string {
return strings.ReplaceAll(src, "\nFUNC ", "\nfunc ")
}

View file

@ -4759,3 +4759,27 @@ func TestBazelGatedLanesNeverRetryFlakyTests(t *testing.T) {
t.Fatalf("checked only %d files; is the repository root right?", checked)
}
}
// Every artifact bazel.yml uploads has a fixed name within its run, and a
// job re-run ("Re-run failed jobs", the documented recovery for a Bazel lane
// that hit a remote cache eviction or a flake) is the same workflow run:
// upload-artifact v4+ refuses a second artifact of the same name there
// (409 Conflict) unless overwrite is set, which would turn a passing retry
// red. bazel-test uploads the bd package even after its tests fail.
func TestBazelWorkflowUploadsSurviveJobReruns(t *testing.T) {
uploads := 0
for name, job := range readCIWorkflow(t, bazelWorkflowName).Jobs {
for _, step := range job.Steps {
if !strings.HasPrefix(step.Uses, "actions/upload-artifact@") {
continue
}
uploads++
if step.With["overwrite"] != "true" {
t.Errorf("%s step %q uploads %q without overwrite: true; a job re-run's upload would fail with 409", name, step.Name, step.With["name"])
}
}
}
if uploads < 8 {
t.Errorf("found %d upload steps in %s, want at least 8", uploads, bazelWorkflowName)
}
}

View file

@ -29,3 +29,13 @@ scripts TestBazelTestTagsFollowTaxonomy skip # reads every BUILD file in the ch
scripts TestBazelTreeGoSrcsListsEveryPackage skip # walks the source checkout; runs under go test
scripts TestCheckTestingShortPassesOnCleanRepoTree skip # walks the whole checkout; runs under go test
scripts TestReleaseFormula* skip # .beads/ is in .bazelignore, so the formula cannot be data; runs under go test
# Policy tests whose inputs (other packages' BUILD files, .github/scripts,
# tools/bazel Python) are not in //scripts:scripts_test's runfiles. Missing
# here, they turned nightly.yml's skip-parity check red (2026-10-02).
# scripts TestBazelOnlySkipsAreAllowlisted keeps this list complete.
scripts TestBazelGatedLanesNeverRetryFlakyTests skip # reads every BUILD and .bzl file in the checkout; runs under go test
scripts TestCheckShardCoverageScript skip # runs tools/bazel/check_shard_coverage.py, not in the runfiles; runs under go test
scripts TestShardScriptsListOnlyRealTests skip # runs the real .github/scripts shard scripts over the source tree; runs under go test
scripts TestShardScriptNarrowingCheck skip # reads the real .github/scripts shard scripts; runs under go test
scripts TestBazelOnlySkipsAreAllowlisted skip # parses every _test.go file in the checkout; runs under go test