fix(ci-policy): close review leftovers from #7485/#7486 (#7491)

* fix(ci-policy): close review leftovers from #7485/#7486

Follow-ups from maintainer review on the host-tool exec/shell policy
tests and the bazel.yml integration gate:

- host_tool_shell_policy_test.go: recognize indented function headers
  and `function name {` forms when finding a function's closing brace,
  so scripts/test.sh's indented cleanup_shared_server() no longer
  mis-attributes its trailing lines to the wrong function; classify
  same-line `command -v gofmt && gofmt` (and `go`) as the guarded
  presence-check idiom instead of flagging it.
- host_tool_exec_policy_test.go: make condPolarity reject mixed-polarity
  && / || compounds (one recognized side, one not) instead of guessing
  at the recognized side's polarity, which was unsound in both
  directions; add pinning tests documenting the exec-before-gate and
  function-value-helper cases as known, safety-conservative limitations
  rather than bugs. Fixing this exposed a real ungated fallback in
  cmd/bd/scripttest_test.go (a `go build` that could still run under
  Bazel on a prebuilt-binary basename mismatch); fixed by symlinking the
  prebuilt binary under the expected name instead of rebuilding it.
- bazel.yml: bazel-server-storage now honors the `integration` input
  like bazel-integration/bazel-cmd-dolt do, so an explicit "off" skips
  it too (latent today: no caller passes "off"). Updated
  ci_workflow_test.go's policy tests to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci-policy): close should-fix B (go command -v guard evasion) on #7491

shellCommandVGuard previously excused go the same as gofmt, but under an
exit-127 host-tool stub `command -v` finds the stub and the guarded exec
still runs, and on a worker genuinely lacking go the exec was silently
skipped -- the worker-dependent behavior this policy exists to ban. Drop
go from the guard entirely, require the guarded exec to sit directly after
the guard's own `&&` (no intervening `;`, `||`, `|`, or leading `!`), and
require an exact gofmt match so a same-prefixed sibling like `gofmt-foo`/
`go-junit-report` can't borrow the guard. Add every should-fix B shape as
a fixture that must still be flagged, plus the generalized gofmt-side
cases (leading `!`, exact-name, post-`;` exec).

Also: correct host_tool_exec_policy_test.go's doc claim that a
function-value helper call site is "not a soundness hole" -- mixed with
one gated direct call site, the invisible function-value call site's
ungated-ness is silently lost and the combined verdict flips to gated;
pinned as review M22. And recognize a function closed by `} >&2` as
ended, so abort-style safety computed inside it does not leak into the
rest of the file (no scoped script has this shape today, but a bare `}`
equality check would silently miss it the moment one does).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-10-11 01:22:48 +00:00 • committed by GitHub
commit 17127ee544
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 629 additions and 47 deletions

View file

@ -220,8 +220,9 @@ on:
# test would silently stop the lane on main.
integration:
description: >
"off" skips bazel-integration; anything else (pr.yml and
bazel-farm.yml pass "on") runs it where remote execution is on.
"off" skips bazel-integration, bazel-cmd-dolt and
bazel-server-storage; anything else (pr.yml and bazel-farm.yml
pass "on") runs them where remote execution is on.
type: string
required: false
default: "on"
@ -2206,10 +2207,13 @@ jobs:
# //internal/storage/dolt over a non-race dolt_test build, each shard
# running the job's own shard script. --config=doltserver-integration has
# exactly --config=integration's build flags, so it compiles nothing
# bazel-integration does not (shared through the remote cache). A job of
# its own, not gated by any integration switch: this is a PR-time tier,
# unlike main.yml's integration jobs, and pr.yml's gate requires it
# (BAZEL_SERVER_STORAGE).
# bazel-integration does not (shared through the remote cache). This is a
# PR-time tier, unlike main.yml's integration jobs, and pr.yml's gate
# requires it (BAZEL_SERVER_STORAGE); it also honors the same
# `integration` input as bazel-integration/bazel-cmd-dolt so an explicit
# "off" skips it too, even though no caller passes that today (pr.yml and
# bazel-farm.yml both pass "on") -- see the `integration` input's own
# description.
bazel-server-storage:
name: server-Dolt storage tier
# Remote execution only, like bazel-integration: locally the tagged race
@ -2218,7 +2222,7 @@ jobs:
# retired) and pr.yml's gate requires it to have run here and passed.
needs: rbe
runs-on: ${{ needs.rbe.outputs.mode == 'remote' && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
if: ${{ needs.rbe.outputs.enabled == 'true' }}
if: ${{ needs.rbe.outputs.enabled == 'true' && inputs.integration != 'off' }}
timeout-minutes: 30
outputs:
result: ${{ steps.result.outputs.result }}

View file

@ -28,9 +28,25 @@ func TestScripts(t *testing.T) {
exeName := "bd"
binDir := t.TempDir()
exe := filepath.Join(binDir, exeName)
if prebuilt, err := findPrebuiltBDBinary(); err == nil && prebuilt != "" && filepath.Base(prebuilt) == exeName {
exe = prebuilt
binDir = filepath.Dir(prebuilt)
prebuilt, err := findPrebuiltBDBinary()
if err != nil {
t.Fatal(err)
}
if prebuilt != "" {
// Under Bazel this is always non-empty (bazeltest.PrebuiltBD), so the
// `go build` fallback below never runs there -- but its basename may
// not be literally "bd" (the injected //cmd/bd:bd_for_tests runfile),
// and the scripts below invoke plain `bd` via sh -c, so symlink it
// under the expected name rather than falling through to a host `go
// build` that would otherwise still run under Bazel on a basename
// mismatch (the exact worker-dependent failure this scan guards
// against: f4-spec.md §9).
if filepath.Base(prebuilt) == exeName {
exe = prebuilt
binDir = filepath.Dir(prebuilt)
} else if err := os.Symlink(prebuilt, exe); err != nil {
t.Fatalf("symlink prebuilt bd binary: %v", err)
}
} else if err := goBuildBDCommand(exe).Run(); err != nil {
t.Fatal(err)
}

View file

@ -1722,6 +1722,13 @@ var bazelAdvisoryLanes = map[string]string{
// lane keeps running on main.
const bazelIntegIf = "${{ (needs.rbe.outputs.enabled == 'true' || needs.rbe.outputs.mode == 'cache') && inputs.integration != 'off' }}"
// bazelServerStorageIf: like bazelIntegIf, bazel-server-storage honors the
// `integration` input (an explicit "off" skips it too, gastownhall/beads
// #7486 review), but unlike bazel-integration/bazel-cmd-dolt it is
// remote-only (wantRemoteOnlyIf), not mode cache -- there is no PR-time
// warm-cache run of this tier, only remote or nothing.
const bazelServerStorageIf = "${{ needs.rbe.outputs.enabled == 'true' && inputs.integration != 'off' }}"
// F3: package-mcp and package-npm's if. Unlike every other lane's, it does
// not read the rbe job's outputs at all - only the caller's package-gates
// input - so these two never skip because of execution mode, only because a
@ -1969,6 +1976,13 @@ func TestBazelWorkflowJobsAndExecutionMode(t *testing.T) {
wantJobRunsOn, wantJobIf, wantJobSetupEnv = bazelPackageRunsOn[name], bazelPackageGatesIf, wantPackageSetupEnv
} else if bazelRemoteOnlyJobs[name] {
wantJobIf = wantRemoteOnlyIf
if name == bazelServerJobName {
// Remote-only like bazel-embedded/bazel-proxied, but also
// honors the `integration` input like
// bazel-integration/bazel-cmd-dolt (gastownhall/beads#7486
// review).
wantJobIf = bazelServerStorageIf
}
} else if name == bazelIntegJobName || name == bazelCmdDoltJobName {
// The cmd/bd Dolt-server tier runs exactly where the integration
// lane does: its build is --config=integration's.
@ -2849,6 +2863,14 @@ func bazelLaneRunModes(t *testing.T, lane, ifExpr string, with map[string]string
return map[string]bool{}
}
return map[string]bool{"remote": true, "fork-ro": true, "fork-rw": true, "cache": true}
case bazelServerStorageIf:
// Remote-only (unlike bazelIntegIf, no mode cache run), but also
// honors `integration` like bazel-integration/bazel-cmd-dolt
// (gastownhall/beads#7486 review).
if strings.EqualFold(with["integration"], "off") {
return map[string]bool{}
}
return map[string]bool{"remote": true, "fork-ro": true, "fork-rw": true}
case bazelPackageGatesIf:
// F3: unlike every other lane, these two never skip because of the
// rbe job's decision - only because the caller turned them off

View file

@ -251,13 +251,28 @@ func condPolarity(cond ast.Expr, prebuiltVars map[string]bool) (bazelWhenTrue bo
if e.Op == token.LAND || e.Op == token.LOR {
lp, lok := condPolarity(e.X, prebuiltVars)
rp, rok := condPolarity(e.Y, prebuiltVars)
switch {
case lok && rok && lp == rp:
// A mixed compound -- one side recognized, the other not -- is
// left unmatched rather than guessed at, for EITHER operator
// (review: "mixed compounds with one unrecognized side are
// taken as a gate", against this function's own doc comment).
// An earlier revision returned the recognized side's polarity
// in that case, which is unsound in both directions:
// - `IsBazel() && testing.Short()` (lok=true, rok=false)
// taken as bazelWhenTrue=true lets `if IsBazel() &&
// testing.Short() { t.Skip() }` count as containment-style
// gated even though, without -short, the condition is false
// under Bazel and the exec still runs;
// - `os.Getenv("TEST_SRCDIR") == "" || os.Getenv(
// "FORCE_HOST_GO") != ""` (lok=true, rok=false, since the
// FORCE_HOST_GO side names no recognized Bazel signal) taken
// as bazelWhenTrue=false/matched=true lets abort-style
// wrongly fire when FORCE_HOST_GO is unset off-Bazel.
// The sound rule is operator-dependent (containment needs
// &&-with-unknown to be unmatched; abort needs ||-with-unknown
// to be unmatched), so requiring BOTH sides recognized, for
// both operators, is what the doc comment already promised.
if lok && rok && lp == rp {
return lp, true
case lok && !rok:
return lp, true
case rok && !lok:
return rp, true
}
}
}
@ -783,6 +798,185 @@ func bazelOnlyFixture(t *testing.T) {
}
})
t.Run("a mixed && compound with one unrecognized side is not a gate (mutation check, review item C)", func(t *testing.T) {
// `IsBazel() && testing.Short()` is true under Bazel only when
// -short was also passed: an earlier revision of condPolarity
// treated the unrecognized `testing.Short()` side as absent and
// returned IsBazel()'s own polarity (bazelWhenTrue=true), which
// wrongly makes this containment-style-gate the exec below for
// every run, including a non-short run under Bazel where the
// condition is false and the exec is NOT skipped.
fset, f, lines := parse(t, `package p
import "os/exec"
func build(t *testing.T) {
if bazeltest.IsBazel() && extraCondition() {
t.Skip()
}
exec.Command("go", "build")
}
`)
hits := findHostToolExecCalls(fset, f, lines)
if len(hits) != 1 || hits[0].gated {
t.Fatalf("hits = %+v, want one ungated hit: the compound is mixed-polarity "+
"(one side unrecognized), so it must not be treated as a gate", hits)
}
})
t.Run("a mixed || compound with one unrecognized side is not a gate (mutation check, review item C)", func(t *testing.T) {
// `os.Getenv("TEST_SRCDIR") == "" || os.Getenv("FORCE_HOST_GO") !=
// ""` has a recognized left side (true off-Bazel) and an
// unrecognized right side. An earlier revision returned the
// recognized side's polarity (bazelWhenTrue=false, matched=true),
// which wrongly makes this abort-style-gate the exec: with
// FORCE_HOST_GO unset, off-Bazel, the condition is already true and
// aborts, so the abort-style rule would (wrongly) call the exec
// below safe under Bazel too, when in fact FORCE_HOST_GO could be
// set under Bazel and reach it there.
fset, f, lines := parse(t, `package p
import (
"os"
"os/exec"
)
func build() {
if os.Getenv("TEST_SRCDIR") == "" || os.Getenv("FORCE_HOST_GO") != "" {
return
}
exec.Command("go", "build")
}
`)
hits := findHostToolExecCalls(fset, f, lines)
if len(hits) != 1 || hits[0].gated {
t.Fatalf("hits = %+v, want one ungated hit: the compound is mixed-polarity "+
"(one side unrecognized), so it must not be treated as a gate", hits)
}
})
t.Run("exec placed before its gate, in sequence (not nested), still counts as gated -- documented limitation, not a bug (review item C)", func(t *testing.T) {
// This is NOT the M17 false-safe (an exec inside the SAME
// true-under-Bazel branch, ahead of that branch's own abort): here
// the exec is a sibling statement, textually before an unrelated
// `if` later in the function that aborts under Bazel. The package
// comment already documents this scan as deliberately
// position-agnostic ("it does not check that the `if` lexically
// precedes the call"), and "same-function IsBazel gate, textually
// after the exec, still counts" above already pins the single-if
// case. This fixture pins the same behavior for an abort-style gate
// specifically, as its own named case, since gastownhall/beads#7485
// review flagged "an exec placed before its gate" as still open.
// Fixing it for real would require control-flow-ordering analysis
// this line/AST-position scan does not do; left as a known,
// documented limitation (conservative in the safe direction only
// when paired with the M17 same-branch check above, which this
// fixture does not exercise).
fset, f, lines := parse(t, `package p
import "os/exec"
func build() {
exec.Command("go", "build")
if bazeltest.IsBazel() {
return
}
}
`)
hits := findHostToolExecCalls(fset, f, lines)
if len(hits) != 1 || !hits[0].gated {
t.Fatalf("hits = %+v, want one hit reported GATED: pinning the documented "+
"position-agnostic limitation (not asserting this is safe)", hits)
}
})
t.Run("an exec reached only through a function-value helper is not recognized -- documented limitation, not a bug (review item C)", func(t *testing.T) {
// helperIsGatedByAllCallers (via findCallSitesOf) only recognizes a
// bare `name(...)` call (an *ast.Ident). A call through a function
// VALUE -- a variable or struct field holding the helper, or the
// helper passed to another function and invoked indirectly -- is
// not a call site findCallSitesOf's walk can name, so
// helperIsGatedByAllCallers reports zero call sites for it (not
// gated), even when every real invocation happens to be gated.
// This under-approximates safety (it would require an allowlist
// entry it does not actually need), not over-approximates it, so
// it is a usability gap, not a soundness hole IN ISOLATION -- but
// see the next fixture (review M22): mixed with even one call site
// findCallSitesOf DOES see and that one happens to be gated, the
// invisible function-value call site's ungated-ness is silently
// lost, and the combined result flips to "gated". That combined
// case is a soundness hole, not merely a usability gap.
_, helperFile, _ := parse(t, `package p
import "os/exec"
func helper() {
exec.Command("go", "build")
}
`)
_, callerFile, _ := parse(t, `package p
func caller() {
if bazeltest.IsBazel() {
return
}
fn := helper
fn()
}
`)
files := map[string]*ast.File{"helper.go": helperFile, "caller.go": callerFile}
gated, n := helperIsGatedByAllCallers(files, "helper")
if gated || n != 0 {
t.Fatalf("helperIsGatedByAllCallers = (%v, %d), want (false, 0): pinning that a "+
"function-value call site is invisible to this scan (documented limitation)", gated, n)
}
})
t.Run("a helper with one gated direct call and one ungated function-value call is wrongly reported gated (review M22, pinned false-safe)", func(t *testing.T) {
// This is the soundness hole the previous fixture's doc comment
// understated as "not a soundness hole": findCallSitesOf only ever
// sees callerDirect's bare `helper()` call (gated=true). It cannot
// see callerIndirect's `fn := helper; fn()` call at all, so that
// call site -- reached with NO gate -- never lowers the combined
// verdict. helperIsGatedByAllCallers returns gated=true from a
// single visible, gated call site, even though helper's exec also
// runs, unconditionally, through callerIndirect. A caller of this
// helper that relies on helperIsGatedByAllCallers("helper") to
// decide the exec needs no allowlist entry is told it is safe when
// it is not. Pinned as a known, documented limitation (fixing it
// for real needs points-to analysis for a same-package helper
// value), not asserted to be correct.
_, helperFile, _ := parse(t, `package p
import "os/exec"
func helper() {
exec.Command("go", "build")
}
`)
_, callerFile, _ := parse(t, `package p
func callerDirect() {
if bazeltest.IsBazel() {
return
}
helper()
}
func callerIndirect() {
fn := helper
fn()
}
`)
files := map[string]*ast.File{"helper.go": helperFile, "caller.go": callerFile}
gated, n := helperIsGatedByAllCallers(files, "helper")
if !gated || n != 1 {
t.Fatalf("helperIsGatedByAllCallers = (%v, %d), want (true, 1): pinning the false-safe "+
"(review M22) -- callerIndirect's ungated function-value call site is invisible, "+
"so the one visible, gated call site wins even though the exec is not always gated", gated, n)
}
})
t.Run("TEST_SRCDIR gate", func(t *testing.T) {
fset, f, lines := parse(t, `package p

View file

@ -243,24 +243,25 @@ var shellExecAllowlist = map[string][]shellAllowlistEntry{
why: "same fake-go-first-on-PATH gate as the GOEXE line above",
},
{
// fn is "cleanup_shared_server", not "": these two lines are
// top-level script code, textually after that indented,
// fn is "" (top level), not "cleanup_shared_server": this line
// is top-level script code, textually after that indented,
// nested `cleanup_shared_server() { ... }` definition (around
// test.sh:191-195). findShellHostToolExecs' end-of-function
// detection only recognizes an UNINDENTED closing `}`/`)` (the
// column-0 convention every other in-scope script's functions
// follow); cleanup_shared_server's own closing brace is
// indented, so the scanner never sees this function end and
// attributes every later line in the file to it. Keying this
// entry by that attribution (rather than fixing the shared
// scanner for one nested-function script) keeps this fix
// scoped to what review item 1 asked for.
fn: "cleanup_shared_server",
// test.sh:191-195) has already closed. An earlier revision of
// findShellHostToolExecs's end-of-function detection recognized
// only an UNINDENTED closing `}`/`)`, so it never saw
// cleanup_shared_server's own indented closing brace and
// attributed every later line in the file to that function
// (review: "the scanner never closes an indented function
// header... two allowlist entries are keyed to the wrong
// function"). The scanner now matches a closing brace/paren at
// the SAME indentation as the function's opening line, so this
// line is correctly attributed to top level.
fn: "",
snippet: `CMD=(go test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -timeout "$TIMEOUT")`,
why: "same fake-go-first-on-PATH gate as the GOEXE line above",
},
{
fn: "cleanup_shared_server", // see the fn note on the CMD= entry above
fn: "", // see the fn note on the CMD= entry above
snippet: `total=$(go tool cover -func="$COVERPROFILE" | awk '/^total:/ {print $NF}')`,
why: "same fake-go-first-on-PATH gate; test_script_test.go's fake go's `tool` " +
"subcommand (COVERAGE is never requested by the test, so this line only " +
@ -315,6 +316,69 @@ var shellGofmtExec = regexp.MustCompile("(^|[\\s;|&(`])gofmt([\\s;|&)`]|$)")
// gofmt, which is not itself an exec of the tool.
var shellToolLookup = regexp.MustCompile(`\b(command\s+-v|which|type)\s+(go|gofmt)\b`)
// shellCommandVGuard matches the presence-guard idiom `command -v gofmt
// >/dev/null 2>&1 && gofmt -l .` on a single line (review: "a same-line
// `command -v gofmt && gofmt` is skipped"), and ONLY that exact shape:
//
// - `go` is deliberately not covered. A should-fix found that guarding
// `go` this way is an evasion, not a safety idiom: `command -v go &&
// go build` is reported safe here, but under an exit-127 host-tool
// stub `command -v` finds the stub and the exec still runs, and on a
// worker that genuinely lacks `go` the exec is silently skipped --
// exactly the worker-dependent behavior this policy exists to ban
// (review: "B is an evasion for go"; "the 'safe by construction'
// premise is also false"). For `gofmt` specifically the guard stays
// sound under this policy's own premise (package comment: a host
// gofmt is not guaranteed on the hermetic-first `bazel test` PATH), so
// `command -v gofmt && gofmt` really is safe by construction.
// - The lookup's own clause must not be negated by a leading `!`, and
// must not be separated from its `&&` -- or the `&&` from the exec it
// guards -- by a `;`, `||`, or a bare `|` (review: "… && echo; go
// build" and the rest of the `… && echo; go build` family must still
// be flagged; the fix is to require the guarded exec to come directly
// after the guard's own `&&`, with nothing intervening). The
// `(^|[;&|])` lead-in only starts a match right after a clause
// boundary, which also rules out a leading `!` on the lookup: `!
// command -v gofmt` has no `;`/`&`/`|`/line-start immediately before
// `command`, so it never matches.
// - Both the lookup and the exec must name EXACTLY `gofmt`, not a
// same-prefixed sibling like `gofmt-bin` or (for the `go` case this
// guard no longer covers at all) `go-junit-report` (review: "require
// exact tool-name match (not `go-*` prefixes)"). The boundary class
// mirrors shellGofmtExec's own word-boundary set rather than `\b`,
// since `\b` alone treats the `-` in `gofmt-bin` as a boundary too.
var shellCommandVGuard = regexp.MustCompile(
"(^|[;&|])\\s*(?:command\\s+-v|which|type)\\s+gofmt([\\s;|&)`]|$)[^;|]*?&&\\s*(gofmt)([\\s;|&)`]|$)",
)
// commandVGuardedExecEnd returns the byte offset of the single gofmt exec
// guarded by a same-line `command -v gofmt ... && gofmt ...`-style idiom on
// line, and whether one was found. Only that EXACT occurrence -- the gofmt
// token immediately following the guard's own `&&` -- is guarded; any other
// gofmt on the line (before the guard, after a `;`/`||`/`|`, or anywhere
// else) is a separate, ungated exec and must still be reported. There is no
// `tool` parameter any more: the guard only ever recognizes gofmt (see
// shellCommandVGuard's comment for why `go` is excluded entirely).
func commandVGuardedExecEnd(line string) (int, bool) {
m := shellCommandVGuard.FindStringSubmatchIndex(line)
if m == nil || m[6] < 0 {
return 0, false
}
return m[6], true
}
// insideAnyIntSpan reports whether pos lies in any of spans, each a
// [2]int-shaped []int as returned by regexp's FindAllStringIndex /
// FindAllStringSubmatchIndex.
func insideAnyIntSpan(spans [][]int, pos int) bool {
for _, s := range spans {
if pos >= s[0] && pos < s[1] {
return true
}
}
return false
}
// heredocStart matches a `<<` (optionally `<<-`) redirection naming its
// terminator, with or without quotes: `<<EOF`, `<<-EOF`, `<<'EOF'`,
// `<<"EOF"`. The delimiter must start with a letter or underscore, so this
@ -481,6 +545,7 @@ func findShellHostToolExecs(src string) []shellExecHit {
inHeredoc := false
heredocDelim := ""
currentFn := ""
currentFnIndent := ""
for scanner.Scan() {
lineno++
line := scanner.Text()
@ -496,11 +561,20 @@ func findShellHostToolExecs(src string) []shellExecHit {
}
if fn, ok := matchFuncStart(trimmed); ok {
currentFn = fn
} else if currentFn != "" && (line == "}" || line == ")") {
// An unindented closing brace/paren alone on a line ends the
// function, mirroring matchFuncStart's own column-0 anchoring:
// every function body line in this policy's scope is indented.
currentFnIndent = leadingWhitespace(line)
} else if currentFn != "" && isFuncCloseLine(trimmed) && leadingWhitespace(line) == currentFnIndent {
// A closing brace/paren alone on a line, at the SAME indentation
// as the function's own opening line, ends the function. Column
// 0 alone (the previous rule) never closes an indented function
// header like ` cleanup_shared_server() {` -- the function's
// own closing brace is indented to match -- so every later line
// in the file was wrongly attributed to that function (review:
// "the scanner never closes an indented function header").
// Matching the start line's indentation instead of hard-coding
// column 0 handles both the common top-level case and a nested,
// indented function definition the same way.
currentFn = ""
currentFnIndent = ""
}
if m := heredocStart.FindStringSubmatch(line); m != nil {
inHeredoc = true
@ -510,37 +584,80 @@ func findShellHostToolExecs(src string) []shellExecHit {
}
spans := quotedSpans(line)
codeSpans := dashCCodeSpans(line)
lookupSpans := shellToolLookup.FindAllStringIndex(line, -1)
for _, m := range shellHostToolVerbs.FindAllStringSubmatchIndex(line, -1) {
if quoted(spans, codeSpans, m[0]) {
continue
}
// No same-line `command -v go && go ...` guard exists any more
// (shellCommandVGuard only recognizes gofmt): every `go` hit is
// reported here regardless of what else shares its line.
hits = append(hits, shellExecHit{line: lineno, fn: currentFn, snippet: lines[lineno-1], verb: line[m[2]:m[3]]})
}
if shellToolLookup.MatchString(line) {
continue
}
for _, m := range shellGofmtExec.FindAllStringSubmatchIndex(line, -1) {
start := strings.Index(line[m[0]:m[1]], "gofmt") + m[0]
if quoted(spans, codeSpans, start) {
continue
}
if insideAnyIntSpan(lookupSpans, start) {
// This "gofmt" occurrence IS the `command -v`/`which`/`type`
// lookup phrase itself, not a separate exec.
continue
}
if guarded, ok := commandVGuardedExecEnd(line); ok && start == guarded {
// A same-line `command -v gofmt ... && gofmt ...` (review:
// "a same-line `command -v gofmt && gofmt` is skipped"): now
// classified explicitly as the presence-guarded idiom it is,
// and only that EXACT occurrence (the gofmt directly after
// the guard's own `&&`) is excused -- not a range, so a
// second, unrelated gofmt elsewhere on the line is still
// reported.
continue
}
hits = append(hits, shellExecHit{line: lineno, fn: currentFn, snippet: lines[lineno-1], verb: ""})
}
}
return hits
}
// shellFuncStart matches a shell function definition's opening line, either
// style this repo uses: `name() {` or `name() (`.
var shellFuncStart = regexp.MustCompile(`^([A-Za-z_][A-Za-z0-9_]*)\s*\(\)\s*[{(]\s*$`)
// funcCloseLine matches a bare function-closing `}`/`)`, optionally followed
// by a trailing output redirection such as `>&2` or `2>&1` (review: "a
// function closed by `} >&2` isn't detected as ended"). No scoped script has
// one today, but a plain `trimmed == "}"` equality check would silently stop
// recognizing the function's end -- and leak whatever safety was computed
// inside it into the rest of the file -- the moment one is added.
var funcCloseLine = regexp.MustCompile(`^[})](\s+\S*>\S*)*$`)
// isFuncCloseLine reports whether trimmed is a function-closing line: a bare
// `}`/`)`, or one followed only by an output redirection.
func isFuncCloseLine(trimmed string) bool {
return funcCloseLine.MatchString(trimmed)
}
// leadingWhitespace returns s's leading run of spaces/tabs.
func leadingWhitespace(s string) string {
i := 0
for i < len(s) && (s[i] == ' ' || s[i] == '\t') {
i++
}
return s[:i]
}
// shellFuncStart matches a shell function definition's opening line, any
// style this repo uses: `name() {`, `name() (`, `function name {`, or
// `function name() {`.
var shellFuncStart = regexp.MustCompile(`^(?:function\s+([A-Za-z_][A-Za-z0-9_]*)\s*(?:\(\))?|([A-Za-z_][A-Za-z0-9_]*)\s*\(\))\s*[{(]\s*$`)
func matchFuncStart(trimmed string) (string, bool) {
m := shellFuncStart.FindStringSubmatch(trimmed)
if m == nil {
return "", false
}
return m[1], true
if m[1] != "" {
return m[1], true // `function name {` / `function name() {`
}
return m[2], true // `name() {` / `name() (`
}
// condVarCheck matches a condition fragment testing one of TEST_SRCDIR or a
@ -652,6 +769,7 @@ func shellLineGating(src string) map[int]bool {
var stack []*ifFrame
abortSafeFrom := -1 // line after which abort-style safety applies, -1 if none
inFunc := false
funcIndent := ""
for i, raw := range lines {
lineno := i + 1
trimmed := strings.TrimSpace(raw)
@ -659,19 +777,23 @@ func shellLineGating(src string) map[int]bool {
stack = nil
abortSafeFrom = -1
inFunc = true
} else if inFunc && (raw == "}" || raw == ")") {
// An unindented closing brace/paren alone on a line ends the
// function (matchFuncStart's own column-0 anchoring assumes
// every function body line in this policy's scope is indented,
// the same convention findShellHostToolExecs's currentFn
// tracking uses). Without this reset, abort-style safety
funcIndent = leadingWhitespace(raw)
} else if inFunc && isFuncCloseLine(trimmed) && leadingWhitespace(raw) == funcIndent {
// A closing brace/paren alone on a line, at the SAME indentation
// as the function's own opening line, ends the function
// (findShellHostToolExecs's currentFn tracking uses the same
// rule). Column 0 alone (the previous rule) never closed an
// indented function header like ` cleanup_shared_server() {`,
// so this reset never fired for it and abort-style safety
// computed inside one function (e.g. upgrade-smoke-test.sh's
// build_candidate) leaked into whatever top-level code follows
// the function, long after it closed (review: "the abort/skip
// gate state isn't reset at function end").
// gate state isn't reset at function end"; "the scanner never
// closes an indented function header").
stack = nil
abortSafeFrom = -1
inFunc = false
funcIndent = ""
}
switch {
case strings.HasPrefix(trimmed, "if ") || trimmed == "if":
@ -1019,6 +1141,230 @@ func TestShellHostToolExecPolicyScan(t *testing.T) {
t.Fatal("a disjunctive condition is not reasoned about; must not be reported safe")
}
})
t.Run("an indented function's own indented closing brace ends it (mutation check, review: test.sh cleanup_shared_server)", func(t *testing.T) {
// Mirrors scripts/test.sh:191-195's shape: a function defined,
// indented, inside other indented code, whose own closing brace is
// indented to match. A scanner that only recognizes a column-0 `}`
// would never see this function end, and would wrongly attribute
// the top-level `go build` below to cleanup_shared_server instead
// of to top level.
src := "#!/usr/bin/env bash\n" +
"if true; then\n" +
" cleanup_shared_server() {\n" +
" rm -rf \"$DIR\"\n" +
" }\n" +
"fi\n" +
"go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
found := false
for _, h := range hits {
if h.line == 7 {
found = true
if h.fn != "" {
t.Fatalf("line 7 hit fn = %q, want \"\" (top level, past the indented function's "+
"own closing brace), got hits %+v", h.fn, hits)
}
}
}
if !found {
t.Fatalf("got %+v, want a hit at line 7", hits)
}
// shellLineGating's own function-boundary tracking must reset the
// same way: abort-style safety computed (if any) inside
// cleanup_shared_server must not leak past its indented close.
src2 := "#!/usr/bin/env bash\n" +
"if true; then\n" +
" cleanup_shared_server() {\n" +
" if [ -n \"${CANDIDATE_BIN:-}\" ]; then\n" +
" return\n" +
" fi\n" +
" }\n" +
"fi\n" +
"go build -o out ./cmd/bd\n"
if shellLineGating(src2)[9] {
t.Fatal("line 9 is top-level code after the indented function's indented close; " +
"must not inherit its abort-style safety")
}
})
t.Run("a `function name {` header is recognized (POSIX/ksh style)", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"function cleanup {\n" +
" true\n" +
"}\n" +
"go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 5 || hits[0].fn != "" {
t.Fatalf("got %+v, want one hit at line 5 attributed to top level", hits)
}
})
t.Run("a `function name() {` header is recognized", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"function cleanup() {\n" +
" go build -o out ./cmd/bd\n" +
"}\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].fn != "cleanup" {
t.Fatalf("got %+v, want one hit attributed to fn \"cleanup\"", hits)
}
})
t.Run("a function closed by `} >&2` is recognized as ended (review follow-up)", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"log() {\n" +
" echo \"$*\"\n" +
"} >&2\n" +
"go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 5 || hits[0].fn != "" {
t.Fatalf("got %+v, want one hit at line 5 attributed to top level: `} >&2` ends log(), "+
"a bare `}`-only check never would", hits)
}
src2 := "#!/usr/bin/env bash\n" +
"log() {\n" +
" if [ -n \"${CANDIDATE_BIN:-}\" ]; then\n" +
" return\n" +
" fi\n" +
"} >&2\n" +
"go build -o out ./cmd/bd\n"
if shellLineGating(src2)[7] {
t.Fatal("line 7 is top-level code after log()'s `} >&2` close; must not inherit " +
"its abort-style safety")
}
})
t.Run("same-line `command -v gofmt && gofmt` is classified as the guarded idiom, not flagged (review)", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"command -v gofmt >/dev/null 2>&1 && gofmt -l .\n"
if hits := findShellHostToolExecs(src); len(hits) != 0 {
t.Fatalf("got %+v, want no hits: command -v gofmt && gofmt is safe by construction "+
"(gofmt is not guaranteed on the hermetic bazel-test PATH)", hits)
}
})
t.Run("same-line `command -v go && go build` is NOT a guarded idiom (should-fix B)", func(t *testing.T) {
// go is deliberately not covered by shellCommandVGuard at all (see
// its doc comment): a command -v go guard is a worker-dependent
// skip this policy bans, not a safe-by-construction idiom.
src := "#!/usr/bin/env bash\n" +
"command -v go >/dev/null 2>&1 && go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 || hits[0].verb != "build" {
t.Fatalf("got %+v, want one `build` hit at line 2: a command -v go guard must not excuse it", hits)
}
})
t.Run("should-fix B: command -v go-junit-report && go test is still flagged", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"command -v go-junit-report >/dev/null 2>&1 && go test ./...\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 || hits[0].verb != "test" {
t.Fatalf("got %+v, want one `test` hit at line 2: a go-junit-report lookup must not "+
"excuse an unrelated go exec (exact tool-name match, not a go-* prefix)", hits)
}
})
t.Run("should-fix B: command -v gofmt && echo; go build is still flagged", func(t *testing.T) {
// The `… && echo; go build` family: the lookup's own && really does
// guard `echo`, but the go build after the `;` is a separate,
// unconditional command that must still be reported.
src := "#!/usr/bin/env bash\n" +
"command -v gofmt >/dev/null 2>&1 && echo; go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 || hits[0].verb != "build" {
t.Fatalf("got %+v, want one `build` hit at line 2", hits)
}
})
t.Run("should-fix B: command -v gofmt && true || go build is still flagged", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"command -v gofmt >/dev/null 2>&1 && true || go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 || hits[0].verb != "build" {
t.Fatalf("got %+v, want one `build` hit at line 2", hits)
}
})
t.Run("should-fix B: command -v gofmt && echo || true && go build is still flagged", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"command -v gofmt >/dev/null 2>&1 && echo || true && go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 || hits[0].verb != "build" {
t.Fatalf("got %+v, want one `build` hit at line 2", hits)
}
})
t.Run("should-fix B: ! command -v go && go build is still flagged", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"! command -v go >/dev/null 2>&1 && go build -o out ./cmd/bd\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 || hits[0].verb != "build" {
t.Fatalf("got %+v, want one `build` hit at line 2: a negated lookup guards nothing", hits)
}
})
t.Run("should-fix B: which go; [ x ] && go test is still flagged", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"which go; [ -x /usr/bin/go ] && go test ./...\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 || hits[0].verb != "test" {
t.Fatalf("got %+v, want one `test` hit at line 2: the lookup is separated from the "+
"exec's && by a ; and an unrelated [ x ] test", hits)
}
})
t.Run("should-fix B generalized: ! command -v gofmt && gofmt is still flagged", func(t *testing.T) {
// Same leading-! requirement applies to the gofmt guard that remains.
src := "#!/usr/bin/env bash\n" +
"! command -v gofmt >/dev/null 2>&1 && gofmt -l .\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 {
t.Fatalf("got %+v, want one hit at line 2: a negated lookup guards nothing", hits)
}
})
t.Run("should-fix B generalized: command -v gofmt-foo && gofmt is still flagged (exact tool-name match)", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"command -v gofmt-foo >/dev/null 2>&1 && gofmt -l .\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 {
t.Fatalf("got %+v, want one hit at line 2: gofmt-foo is not gofmt", hits)
}
})
t.Run("should-fix B generalized: command -v gofmt && echo; gofmt is still flagged", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"command -v gofmt >/dev/null 2>&1 && echo; gofmt -l .\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 {
t.Fatalf("got %+v, want one hit at line 2: the gofmt exec after the ; is unguarded", hits)
}
})
t.Run("an unrelated exec sharing a line with an unrelated lookup is still reported (guard is tool- and position-specific)", func(t *testing.T) {
// The guard must not degrade back into a blanket same-line skip:
// looking up `go` must not excuse an unrelated `gofmt` exec later
// on the same line (different tool), and a lookup with no `&&` at
// all must not excuse anything.
src := "#!/usr/bin/env bash\n" +
"command -v go >/dev/null 2>&1 && gofmt -l .\n"
hits := findShellHostToolExecs(src)
if len(hits) != 1 || hits[0].line != 2 {
t.Fatalf("got %+v, want one hit at line 2: the gofmt exec is not guarded by a `go` lookup", hits)
}
})
t.Run("a bare command -v gofmt lookup, with no following &&, is still just a lookup", func(t *testing.T) {
src := "#!/usr/bin/env bash\n" +
"if command -v gofmt >/dev/null 2>&1; then\n" +
" echo found\n" +
"fi\n"
if hits := findShellHostToolExecs(src); len(hits) != 0 {
t.Fatalf("got %+v, want no hits (plain presence lookup, no exec)", hits)
}
})
}
// TestNoUngatedBareGoInBazelReachedShellScripts scans shellPolicyScope and