beads/scripts/ci_workflow_apt_get_timeout_test.go
Julian Knutsen c1fc671fcf
ci: cross-build Windows test binaries on Linux (advisory side-by-side), drop PR macOS full suite (F4) (#7135)
* ci(F4): advisory Linux-cross-built Windows test binaries, macOS cache restore

Side-by-side rollout slice of the PR CI -> 5 min plan (F4):

- Add pr.yml `windows-test-binaries`: a Linux job that cross-compiles
  bd.exe and two go-test-c binaries (cgo and nocgo) for windows/amd64
  via mingw (CC/CXX=x86_64-w64-mingw32-*, matching release.yml's
  goreleaser setup), driven by a committed manifest
  (scripts/ci/windows-test-binaries.txt) and build script
  (scripts/ci/build-windows-test-binaries.sh). Artifacts are uploaded
  and consumed by two new advisory "-prebuilt" twins
  (test-windows-liveness-prebuilt, worktree-remove-windows-prebuilt)
  that run the identical test selection/flags/env (including
  BEADS_TEST_REPO_ROOT) against the prebuilt binaries via
  scripts/ci/run-go-test-binary.sh and scripts/test.sh's new
  BEADS_TEST_PREBUILT_TEST_BINARY short-circuit.
- The original native Windows jobs are untouched and remain required;
  the new jobs are advisory only (not in CI_GATE_REQUIRED) for this
  rollout phase. Flipping to the cross-built path later is a one-line
  CI_GATE_REQUIRED change.
- Add main.yml `windows-test-binaries-cache`: seeds the Go build cache
  for the cross-compile toolchain/packages on a schedule, mirroring
  the existing module/build cache warmers; reorder build-artifacts'
  warm-cache step to precede the module-cache save for a cleaner
  build -> warm -> save grouping.
- Restore macOS's native build-cache save step in main.yml (lost
  during an earlier job consolidation).
- Remove the non-required "Test (macos-latest)" full-suite job from
  PRs; the native macOS gate jobs still run, so required coverage is
  unaffected. This does drop a non-required pre-merge macOS race
  signal; see docs/notes for the accepted tradeoff.
- Add scripts/windows_test_binaries_manifest_test.go: policy tests
  pinning the manifest's three advisory-phase entries, pinning the
  mingw toolchain to release.yml's, checking consumer jobs only
  reference manifest-listed binaries, checking upload/download
  artifact names and `needs` match, checking the prebuilt jobs select
  exactly the same tests (-run literals) as their native counterparts,
  and checking none of the new jobs reference secrets (fork safety).
  All mutation-tested.
- Add two scripts/ci_workflow_apt_get_timeout_test.go budget entries
  for the new "Install mingw cross-compiler" steps.
- Fix two latent Bazel-only scripts_test failures surfaced while
  wiring the new test file into scripts/BUILD.bazel: materialize
  cmd/bd's sources via the existing //cmd/bd:go_srcs filegroup so
  sourceRepoRoot(t)-based package-directory checks work under Bazel's
  runfiles root, and strip Bazel's own TEST_TIMEOUT env var before
  exercising scripts/test.sh's hardcoded default in
  test_script_test.go (name collision with test.sh's own override
  knob).

Verified: go test ./scripts/...; bazel --nohome_rc test
//scripts:scripts_test; actionlint; shellcheck; make ci-pr-policy;
make bazel-sync-check; a from-scratch /tmp clone cross-build using
the exact job commands (cold 130.2s, warm/cache-hit 30.3s, PE32+
x86-64 validity via file/objdump, byte-identical SHA-256 across
cold/warm/traced runs); wine -test.list confirms
TestPrimeBinaryPortfolio/TestIsServerProbablyRunning* present in the
cgo binary and the full TestWorktreeRemove* family (including
Windows-specific variants) present in the nocgo binary.

Scoping note: only 2 of the spec's 4 consumer jobs (liveness,
worktree-remove) get prebuilt twins, and only 3 of 8 spec-listed
manifest binaries are built this slice; preflight, doc-freshness and
credential-fixture consumers are deferred as documented follow-up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): address F4 review findings (SF-1..SF-6 + nits)

Addresses the F4 review (~/beads-bazel-plan/f4/review.md): no blockers, six
substantive findings plus several nits.

- SF-1: main.yml's windows-test-binaries-cache job now seeds the Windows
  cross-compile cache from both runner selections pr.yml's consumer job can
  land on (Blacksmith for same-repo PRs/merge_group, ubuntu-latest for
  forks/Dependabot) via a 2-leg matrix with identical keys/paths, instead of
  a single ubuntu-latest leg Blacksmith runners can never see.
- SF-2: removed the two consumer-less "Warm non-race Go build cache
  (test-only packages)" steps from main.yml (build-artifacts, test macOS
  leg); they had no real consumer and cost ~270s on main's macOS leg for
  nothing.
- SF-3: scripts/ci/run-go-test-binary.sh now injects the same
  -test.timeout=10m default (when -timeout is omitted) and unconditional
  -test.paniconexit0 flag `go test` itself injects; added
  timeout-minutes to test-windows-liveness-prebuilt; added a direct
  regression test (TestRunGoTestBinaryDefaultsTimeoutWhenOmitted) for the
  omitted-timeout branch, which no path through scripts/test.sh can reach
  (test.sh always supplies its own -timeout default first).
- SF-4: scripts/test.sh's BEADS_TEST_PREBUILT_TEST_BINARY branch now runs
  run-go-test-binary.sh as a child process (capturing and re-exiting its
  status) instead of `exec`-ing it, so the EXIT trap (beads_test_env_cleanup)
  still runs and the per-run BEADS_TEST_ENV_ROOT temp dir (and, with
  BEADS_TEST_SHARED_SERVER=1, the shared dolt sql-server) is cleaned up
  instead of leaked/orphaned.
- SF-5: closed the five surviving mutations (cgo/nocgo swap between the
  Windows prebuilt twins, dropped -count=1, an unguarded twin in
  ci-gate.needs, a goreleaser compiler change, a PKGDIR change) with
  dedicated policy tests, and replaced the three-edits-per-job required/
  advisory toggle with a single committed WINDOWS_PREBUILT_REQUIRED flag
  (pr.yml top-level env) in the style of the existing BAZEL_RETIRES_LEGACY_*
  flags: both Windows pairs are now unconditionally in ci-gate.needs and
  always evaluated, and only which pair lands in CI_GATE_REQUIRED vs.
  CI_GATE_SKIPPED_OK is decided by the flag, in one place.
- SF-6: dropped the //cmd/bd:go_srcs data dep from scripts_test; the
  prebuilt-binary contract test now exercises ./scripts/ci (already a data
  dep of this target) instead of ./cmd/bd, since it never builds or runs the
  real package.
- Nits: removed dangling ~/beads-bazel-plan references and a nonexistent
  test name from committed comments; log the resolved mingw-w64 version
  when cross-compiling; removed a no-op shellcheck disable comment;
  documented why run-go-test-binary.sh's BEADS_TEST_REPO_ROOT re-export
  doesn't race beads_test_env_enter's unset; rewrote the pr.yml tombstone
  comment for the removed macOS PR job to state the actual, narrower
  tradeoff instead of an overclaiming "main.yml covers this" statement.

Verified: bazel --nohome_rc test //scripts:scripts_test passes; actionlint
clean on pr.yml/main.yml; shellcheck clean on all edited scripts (only
pre-existing, unrelated warnings remain elsewhere); make bazel-sync-check
and make ci-pr-policy both pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(scripts): isolate prebuilt-binary contract test from ambient hermetic env

TestTestScriptPrebuiltTestBinaryContract's "cleans up BEADS_TEST_ENV_ROOT
after the binary exits" subtest (added for F4 review finding SF-4) passed
locally but failed in CI's scripts-go-checks/scripts-test leg.

scripts/ci/scripts-go-test.sh (the CI driver for this package's `go test
./scripts/...` run) calls beads_test_env_enter in its own shell before
invoking go test, so BEADS_TEST_ENV_ACTIVE=1 and an outer
BEADS_TEST_ENV_ROOT are already present in this test binary's own
os.Environ(). runTestScriptPrebuiltBinary built the scripts/test.sh
subprocess's env from os.Environ() (minus TEST_TIMEOUT), so that ambient
state leaked through. Inside the subprocess, scripts/test.sh's own
beads_test_env_enter call (test.sh:17) saw the already-active guard and
no-opped: no new root, no new EXIT trap armed in that process - so the
fake binary observed the outer, still-alive root, and the subtest's
"directory is gone after test.sh exits" assertion failed, even though the
SF-4 fix (run the prebuilt binary as a child instead of exec'ing it) is
correct.

Every real production caller of scripts/test.sh's prebuilt path (pr.yml's
Windows liveness/worktree-remove prebuilt steps) runs it as the first
command of a fresh GitHub Actions `run:` step, never nested under an
already-entered hermetic env, so this nesting is purely an artifact of
this test binary's own ambient environment under scripts-go-test.sh, not
a real CI behavior gap.

Strip BEADS_TEST_ENV_ACTIVE/_ROOT/_DISABLE/_KEEP (alongside the existing
TEST_TIMEOUT filter) from the child env in runTestScriptPrebuiltBinary so
the subtest deterministically exercises scripts/test.sh's real top-level,
non-nested invocation regardless of what ambient hermetic-env state this
test binary itself happens to be running under.

Verified: reproduced the CI failure locally by exporting
BEADS_TEST_ENV_ACTIVE=1/BEADS_TEST_ENV_ROOT before `go test ./scripts/...`;
confirmed the fix resolves it; re-ran bazel test //scripts:scripts_test
and scripts/ci/scripts-go-test.sh directly, both pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 11:49:56 -07:00

165 lines
5.9 KiB
Go

package scripts_test
import (
"fmt"
"strconv"
"strings"
"testing"
)
// aptStepBudget describes one bounded `apt-get` step: how long its own inner
// `timeout` calls may run, and the step-level `timeout-minutes` that backstops
// them.
//
// The components are declared here rather than only pinning the final minute
// count so that the two halves are checked against each other. A cap that
// lands below the step's own worst case reproduces exactly the undiagnosable
// mid-flight kill this hardening exists to remove, just at a smaller number --
// and that is invisible if the test only asserts the cap's literal value.
type aptStepBudget struct {
workflow string
job string
step string
updateAttempts int // iterations of the `for attempt in ...` loop
updateTimeout int // seconds allowed per `apt-get update` attempt
backoff int // seconds slept between attempts (never after the last)
installTimeout int // seconds allowed for `apt-get install`
installPkgs string
capMinutes int // the step's `timeout-minutes`
}
// worstCaseSeconds is the longest the step's own bounds permit it to run.
func (b aptStepBudget) worstCaseSeconds() int {
return b.updateAttempts*b.updateTimeout +
(b.updateAttempts-1)*b.backoff +
b.installTimeout
}
// loopHeader is the `for` header the declared attempt count implies.
func (b aptStepBudget) loopHeader() string {
attempts := make([]string, 0, b.updateAttempts)
for i := 1; i <= b.updateAttempts; i++ {
attempts = append(attempts, strconv.Itoa(i))
}
return "for attempt in " + strings.Join(attempts, " ")
}
func aptStepBudgets() []aptStepBudget {
return []aptStepBudget{
{
workflow: "migration-test.yml",
job: "historical-upgrades",
step: "Install test dependencies",
updateAttempts: 3,
updateTimeout: 120,
backoff: 10,
installTimeout: 180,
installPkgs: "jq libicu74",
capMinutes: 11,
},
{
workflow: "release.yml",
job: "goreleaser",
step: "Install cross-compilation toolchains and signing tools",
updateAttempts: 3,
updateTimeout: 120,
backoff: 10,
installTimeout: 300,
installPkgs: "gcc-mingw-w64-x86-64 gcc-aarch64-linux-gnu osslsigncode",
capMinutes: 13,
},
{
workflow: "pr.yml",
job: "windows-test-binaries",
step: "Install mingw cross-compiler",
updateAttempts: 3,
updateTimeout: 120,
backoff: 10,
installTimeout: 300,
installPkgs: "gcc-mingw-w64-x86-64",
capMinutes: 12,
},
{
workflow: "main.yml",
job: "windows-test-binaries-cache",
step: "Install mingw cross-compiler",
updateAttempts: 3,
updateTimeout: 120,
backoff: 10,
installTimeout: 300,
installPkgs: "gcc-mingw-w64-x86-64",
capMinutes: 12,
},
}
}
// TestAptGetStepsAreBoundedAndRetried pins the bounded/retried invocations, so
// a revert to a bare `apt-get update` -- the unbounded shape that burned whole
// job budgets on a stalled mirror -- goes red.
func TestAptGetStepsAreBoundedAndRetried(t *testing.T) {
for _, b := range aptStepBudgets() {
t.Run(b.workflow+"/"+b.step, func(t *testing.T) {
step := readCIWorkflow(t, b.workflow).job(t, b.job).step(t, b.step)
for _, required := range []string{
fmt.Sprintf("sudo timeout %d apt-get", b.updateTimeout),
"-o Acquire::Retries=3",
"-o Acquire::http::Timeout=30",
"-o Acquire::https::Timeout=30",
b.loopHeader(),
// Backoff between attempts, but not after the final one.
fmt.Sprintf("if [ \"$attempt\" -lt %d ]; then", b.updateAttempts),
fmt.Sprintf("sleep %d", b.backoff),
// Exhausting the retries proceeds best-effort against the
// cached index; the warning is what keeps that visible in the
// job log instead of a silent fall-through.
fmt.Sprintf("echo \"::warning::apt-get update failed after %d attempts; proceeding with cached index\"", b.updateAttempts),
fmt.Sprintf("sudo timeout %d apt-get install -y %s", b.installTimeout, b.installPkgs),
} {
if !strings.Contains(step.Run, required) {
t.Errorf("%s command does not contain %q:\n%s", b.step, required, step.Run)
}
}
})
}
}
// TestAptGetStepTimeoutsExceedTheirOwnBudget is the relation the first round of
// this hardening got backwards: the step-level cap was set *tighter* than the
// inner `timeout` calls it was meant to backstop.
func TestAptGetStepTimeoutsExceedTheirOwnBudget(t *testing.T) {
for _, b := range aptStepBudgets() {
t.Run(b.workflow+"/"+b.step, func(t *testing.T) {
workflow := readCIWorkflow(t, b.workflow)
job := workflow.job(t, b.job)
step := job.step(t, b.step)
if step.TimeoutMinutes != b.capMinutes {
t.Errorf("%s timeout-minutes = %d, want %d", b.step, step.TimeoutMinutes, b.capMinutes)
}
// Read the relations off the workflow itself, never off capMinutes:
// checking a declared constant against the budget it was derived
// from is self-satisfying, and would stay green while the YAML
// drifted underneath it.
capSeconds := step.TimeoutMinutes * 60
if capSeconds <= b.worstCaseSeconds() {
t.Errorf("%s timeout-minutes = %d (%ds) does not exceed its own worst case of %ds "+
"(%d update attempts x %ds + %d backoffs x %ds + %ds install); "+
"raise the cap or shrink the inner budgets",
b.step, step.TimeoutMinutes, capSeconds, b.worstCaseSeconds(),
b.updateAttempts, b.updateTimeout, b.updateAttempts-1, b.backoff, b.installTimeout)
}
// Where the job carries its own cap, the step must fail first --
// otherwise the whole job is cancelled and the log gives no hint
// that apt was the thing that stalled.
if job.TimeoutMinutes > 0 && step.TimeoutMinutes >= job.TimeoutMinutes {
t.Errorf("%s timeout-minutes = %d is not below job %q timeout-minutes = %d",
b.step, step.TimeoutMinutes, b.job, job.TimeoutMinutes)
}
})
}
}