* ci(F4): advisory Linux-cross-built Windows test binaries, macOS cache restore Side-by-side rollout slice of the PR CI -> 5 min plan (F4): - Add pr.yml `windows-test-binaries`: a Linux job that cross-compiles bd.exe and two go-test-c binaries (cgo and nocgo) for windows/amd64 via mingw (CC/CXX=x86_64-w64-mingw32-*, matching release.yml's goreleaser setup), driven by a committed manifest (scripts/ci/windows-test-binaries.txt) and build script (scripts/ci/build-windows-test-binaries.sh). Artifacts are uploaded and consumed by two new advisory "-prebuilt" twins (test-windows-liveness-prebuilt, worktree-remove-windows-prebuilt) that run the identical test selection/flags/env (including BEADS_TEST_REPO_ROOT) against the prebuilt binaries via scripts/ci/run-go-test-binary.sh and scripts/test.sh's new BEADS_TEST_PREBUILT_TEST_BINARY short-circuit. - The original native Windows jobs are untouched and remain required; the new jobs are advisory only (not in CI_GATE_REQUIRED) for this rollout phase. Flipping to the cross-built path later is a one-line CI_GATE_REQUIRED change. - Add main.yml `windows-test-binaries-cache`: seeds the Go build cache for the cross-compile toolchain/packages on a schedule, mirroring the existing module/build cache warmers; reorder build-artifacts' warm-cache step to precede the module-cache save for a cleaner build -> warm -> save grouping. - Restore macOS's native build-cache save step in main.yml (lost during an earlier job consolidation). - Remove the non-required "Test (macos-latest)" full-suite job from PRs; the native macOS gate jobs still run, so required coverage is unaffected. This does drop a non-required pre-merge macOS race signal; see docs/notes for the accepted tradeoff. - Add scripts/windows_test_binaries_manifest_test.go: policy tests pinning the manifest's three advisory-phase entries, pinning the mingw toolchain to release.yml's, checking consumer jobs only reference manifest-listed binaries, checking upload/download artifact names and `needs` match, checking the prebuilt jobs select exactly the same tests (-run literals) as their native counterparts, and checking none of the new jobs reference secrets (fork safety). All mutation-tested. - Add two scripts/ci_workflow_apt_get_timeout_test.go budget entries for the new "Install mingw cross-compiler" steps. - Fix two latent Bazel-only scripts_test failures surfaced while wiring the new test file into scripts/BUILD.bazel: materialize cmd/bd's sources via the existing //cmd/bd:go_srcs filegroup so sourceRepoRoot(t)-based package-directory checks work under Bazel's runfiles root, and strip Bazel's own TEST_TIMEOUT env var before exercising scripts/test.sh's hardcoded default in test_script_test.go (name collision with test.sh's own override knob). Verified: go test ./scripts/...; bazel --nohome_rc test //scripts:scripts_test; actionlint; shellcheck; make ci-pr-policy; make bazel-sync-check; a from-scratch /tmp clone cross-build using the exact job commands (cold 130.2s, warm/cache-hit 30.3s, PE32+ x86-64 validity via file/objdump, byte-identical SHA-256 across cold/warm/traced runs); wine -test.list confirms TestPrimeBinaryPortfolio/TestIsServerProbablyRunning* present in the cgo binary and the full TestWorktreeRemove* family (including Windows-specific variants) present in the nocgo binary. Scoping note: only 2 of the spec's 4 consumer jobs (liveness, worktree-remove) get prebuilt twins, and only 3 of 8 spec-listed manifest binaries are built this slice; preflight, doc-freshness and credential-fixture consumers are deferred as documented follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(ci): address F4 review findings (SF-1..SF-6 + nits) Addresses the F4 review (~/beads-bazel-plan/f4/review.md): no blockers, six substantive findings plus several nits. - SF-1: main.yml's windows-test-binaries-cache job now seeds the Windows cross-compile cache from both runner selections pr.yml's consumer job can land on (Blacksmith for same-repo PRs/merge_group, ubuntu-latest for forks/Dependabot) via a 2-leg matrix with identical keys/paths, instead of a single ubuntu-latest leg Blacksmith runners can never see. - SF-2: removed the two consumer-less "Warm non-race Go build cache (test-only packages)" steps from main.yml (build-artifacts, test macOS leg); they had no real consumer and cost ~270s on main's macOS leg for nothing. - SF-3: scripts/ci/run-go-test-binary.sh now injects the same -test.timeout=10m default (when -timeout is omitted) and unconditional -test.paniconexit0 flag `go test` itself injects; added timeout-minutes to test-windows-liveness-prebuilt; added a direct regression test (TestRunGoTestBinaryDefaultsTimeoutWhenOmitted) for the omitted-timeout branch, which no path through scripts/test.sh can reach (test.sh always supplies its own -timeout default first). - SF-4: scripts/test.sh's BEADS_TEST_PREBUILT_TEST_BINARY branch now runs run-go-test-binary.sh as a child process (capturing and re-exiting its status) instead of `exec`-ing it, so the EXIT trap (beads_test_env_cleanup) still runs and the per-run BEADS_TEST_ENV_ROOT temp dir (and, with BEADS_TEST_SHARED_SERVER=1, the shared dolt sql-server) is cleaned up instead of leaked/orphaned. - SF-5: closed the five surviving mutations (cgo/nocgo swap between the Windows prebuilt twins, dropped -count=1, an unguarded twin in ci-gate.needs, a goreleaser compiler change, a PKGDIR change) with dedicated policy tests, and replaced the three-edits-per-job required/ advisory toggle with a single committed WINDOWS_PREBUILT_REQUIRED flag (pr.yml top-level env) in the style of the existing BAZEL_RETIRES_LEGACY_* flags: both Windows pairs are now unconditionally in ci-gate.needs and always evaluated, and only which pair lands in CI_GATE_REQUIRED vs. CI_GATE_SKIPPED_OK is decided by the flag, in one place. - SF-6: dropped the //cmd/bd:go_srcs data dep from scripts_test; the prebuilt-binary contract test now exercises ./scripts/ci (already a data dep of this target) instead of ./cmd/bd, since it never builds or runs the real package. - Nits: removed dangling ~/beads-bazel-plan references and a nonexistent test name from committed comments; log the resolved mingw-w64 version when cross-compiling; removed a no-op shellcheck disable comment; documented why run-go-test-binary.sh's BEADS_TEST_REPO_ROOT re-export doesn't race beads_test_env_enter's unset; rewrote the pr.yml tombstone comment for the removed macOS PR job to state the actual, narrower tradeoff instead of an overclaiming "main.yml covers this" statement. Verified: bazel --nohome_rc test //scripts:scripts_test passes; actionlint clean on pr.yml/main.yml; shellcheck clean on all edited scripts (only pre-existing, unrelated warnings remain elsewhere); make bazel-sync-check and make ci-pr-policy both pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(scripts): isolate prebuilt-binary contract test from ambient hermetic env TestTestScriptPrebuiltTestBinaryContract's "cleans up BEADS_TEST_ENV_ROOT after the binary exits" subtest (added for F4 review finding SF-4) passed locally but failed in CI's scripts-go-checks/scripts-test leg. scripts/ci/scripts-go-test.sh (the CI driver for this package's `go test ./scripts/...` run) calls beads_test_env_enter in its own shell before invoking go test, so BEADS_TEST_ENV_ACTIVE=1 and an outer BEADS_TEST_ENV_ROOT are already present in this test binary's own os.Environ(). runTestScriptPrebuiltBinary built the scripts/test.sh subprocess's env from os.Environ() (minus TEST_TIMEOUT), so that ambient state leaked through. Inside the subprocess, scripts/test.sh's own beads_test_env_enter call (test.sh:17) saw the already-active guard and no-opped: no new root, no new EXIT trap armed in that process - so the fake binary observed the outer, still-alive root, and the subtest's "directory is gone after test.sh exits" assertion failed, even though the SF-4 fix (run the prebuilt binary as a child instead of exec'ing it) is correct. Every real production caller of scripts/test.sh's prebuilt path (pr.yml's Windows liveness/worktree-remove prebuilt steps) runs it as the first command of a fresh GitHub Actions `run:` step, never nested under an already-entered hermetic env, so this nesting is purely an artifact of this test binary's own ambient environment under scripts-go-test.sh, not a real CI behavior gap. Strip BEADS_TEST_ENV_ACTIVE/_ROOT/_DISABLE/_KEEP (alongside the existing TEST_TIMEOUT filter) from the child env in runTestScriptPrebuiltBinary so the subtest deterministically exercises scripts/test.sh's real top-level, non-nested invocation regardless of what ambient hermetic-env state this test binary itself happens to be running under. Verified: reproduced the CI failure locally by exporting BEADS_TEST_ENV_ACTIVE=1/BEADS_TEST_ENV_ROOT before `go test ./scripts/...`; confirmed the fix resolves it; re-ran bazel test //scripts:scripts_test and scripts/ci/scripts-go-test.sh directly, both pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
165 lines
5.9 KiB
Go
165 lines
5.9 KiB
Go
package scripts_test
|
|
|
|
import (
|
|
"fmt"
|
|
"strconv"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// aptStepBudget describes one bounded `apt-get` step: how long its own inner
|
|
// `timeout` calls may run, and the step-level `timeout-minutes` that backstops
|
|
// them.
|
|
//
|
|
// The components are declared here rather than only pinning the final minute
|
|
// count so that the two halves are checked against each other. A cap that
|
|
// lands below the step's own worst case reproduces exactly the undiagnosable
|
|
// mid-flight kill this hardening exists to remove, just at a smaller number --
|
|
// and that is invisible if the test only asserts the cap's literal value.
|
|
type aptStepBudget struct {
|
|
workflow string
|
|
job string
|
|
step string
|
|
|
|
updateAttempts int // iterations of the `for attempt in ...` loop
|
|
updateTimeout int // seconds allowed per `apt-get update` attempt
|
|
backoff int // seconds slept between attempts (never after the last)
|
|
installTimeout int // seconds allowed for `apt-get install`
|
|
installPkgs string
|
|
|
|
capMinutes int // the step's `timeout-minutes`
|
|
}
|
|
|
|
// worstCaseSeconds is the longest the step's own bounds permit it to run.
|
|
func (b aptStepBudget) worstCaseSeconds() int {
|
|
return b.updateAttempts*b.updateTimeout +
|
|
(b.updateAttempts-1)*b.backoff +
|
|
b.installTimeout
|
|
}
|
|
|
|
// loopHeader is the `for` header the declared attempt count implies.
|
|
func (b aptStepBudget) loopHeader() string {
|
|
attempts := make([]string, 0, b.updateAttempts)
|
|
for i := 1; i <= b.updateAttempts; i++ {
|
|
attempts = append(attempts, strconv.Itoa(i))
|
|
}
|
|
return "for attempt in " + strings.Join(attempts, " ")
|
|
}
|
|
|
|
func aptStepBudgets() []aptStepBudget {
|
|
return []aptStepBudget{
|
|
{
|
|
workflow: "migration-test.yml",
|
|
job: "historical-upgrades",
|
|
step: "Install test dependencies",
|
|
updateAttempts: 3,
|
|
updateTimeout: 120,
|
|
backoff: 10,
|
|
installTimeout: 180,
|
|
installPkgs: "jq libicu74",
|
|
capMinutes: 11,
|
|
},
|
|
{
|
|
workflow: "release.yml",
|
|
job: "goreleaser",
|
|
step: "Install cross-compilation toolchains and signing tools",
|
|
updateAttempts: 3,
|
|
updateTimeout: 120,
|
|
backoff: 10,
|
|
installTimeout: 300,
|
|
installPkgs: "gcc-mingw-w64-x86-64 gcc-aarch64-linux-gnu osslsigncode",
|
|
capMinutes: 13,
|
|
},
|
|
{
|
|
workflow: "pr.yml",
|
|
job: "windows-test-binaries",
|
|
step: "Install mingw cross-compiler",
|
|
updateAttempts: 3,
|
|
updateTimeout: 120,
|
|
backoff: 10,
|
|
installTimeout: 300,
|
|
installPkgs: "gcc-mingw-w64-x86-64",
|
|
capMinutes: 12,
|
|
},
|
|
{
|
|
workflow: "main.yml",
|
|
job: "windows-test-binaries-cache",
|
|
step: "Install mingw cross-compiler",
|
|
updateAttempts: 3,
|
|
updateTimeout: 120,
|
|
backoff: 10,
|
|
installTimeout: 300,
|
|
installPkgs: "gcc-mingw-w64-x86-64",
|
|
capMinutes: 12,
|
|
},
|
|
}
|
|
}
|
|
|
|
// TestAptGetStepsAreBoundedAndRetried pins the bounded/retried invocations, so
|
|
// a revert to a bare `apt-get update` -- the unbounded shape that burned whole
|
|
// job budgets on a stalled mirror -- goes red.
|
|
func TestAptGetStepsAreBoundedAndRetried(t *testing.T) {
|
|
for _, b := range aptStepBudgets() {
|
|
t.Run(b.workflow+"/"+b.step, func(t *testing.T) {
|
|
step := readCIWorkflow(t, b.workflow).job(t, b.job).step(t, b.step)
|
|
|
|
for _, required := range []string{
|
|
fmt.Sprintf("sudo timeout %d apt-get", b.updateTimeout),
|
|
"-o Acquire::Retries=3",
|
|
"-o Acquire::http::Timeout=30",
|
|
"-o Acquire::https::Timeout=30",
|
|
b.loopHeader(),
|
|
// Backoff between attempts, but not after the final one.
|
|
fmt.Sprintf("if [ \"$attempt\" -lt %d ]; then", b.updateAttempts),
|
|
fmt.Sprintf("sleep %d", b.backoff),
|
|
// Exhausting the retries proceeds best-effort against the
|
|
// cached index; the warning is what keeps that visible in the
|
|
// job log instead of a silent fall-through.
|
|
fmt.Sprintf("echo \"::warning::apt-get update failed after %d attempts; proceeding with cached index\"", b.updateAttempts),
|
|
fmt.Sprintf("sudo timeout %d apt-get install -y %s", b.installTimeout, b.installPkgs),
|
|
} {
|
|
if !strings.Contains(step.Run, required) {
|
|
t.Errorf("%s command does not contain %q:\n%s", b.step, required, step.Run)
|
|
}
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestAptGetStepTimeoutsExceedTheirOwnBudget is the relation the first round of
|
|
// this hardening got backwards: the step-level cap was set *tighter* than the
|
|
// inner `timeout` calls it was meant to backstop.
|
|
func TestAptGetStepTimeoutsExceedTheirOwnBudget(t *testing.T) {
|
|
for _, b := range aptStepBudgets() {
|
|
t.Run(b.workflow+"/"+b.step, func(t *testing.T) {
|
|
workflow := readCIWorkflow(t, b.workflow)
|
|
job := workflow.job(t, b.job)
|
|
step := job.step(t, b.step)
|
|
|
|
if step.TimeoutMinutes != b.capMinutes {
|
|
t.Errorf("%s timeout-minutes = %d, want %d", b.step, step.TimeoutMinutes, b.capMinutes)
|
|
}
|
|
|
|
// Read the relations off the workflow itself, never off capMinutes:
|
|
// checking a declared constant against the budget it was derived
|
|
// from is self-satisfying, and would stay green while the YAML
|
|
// drifted underneath it.
|
|
capSeconds := step.TimeoutMinutes * 60
|
|
if capSeconds <= b.worstCaseSeconds() {
|
|
t.Errorf("%s timeout-minutes = %d (%ds) does not exceed its own worst case of %ds "+
|
|
"(%d update attempts x %ds + %d backoffs x %ds + %ds install); "+
|
|
"raise the cap or shrink the inner budgets",
|
|
b.step, step.TimeoutMinutes, capSeconds, b.worstCaseSeconds(),
|
|
b.updateAttempts, b.updateTimeout, b.updateAttempts-1, b.backoff, b.installTimeout)
|
|
}
|
|
|
|
// Where the job carries its own cap, the step must fail first --
|
|
// otherwise the whole job is cancelled and the log gives no hint
|
|
// that apt was the thing that stalled.
|
|
if job.TimeoutMinutes > 0 && step.TimeoutMinutes >= job.TimeoutMinutes {
|
|
t.Errorf("%s timeout-minutes = %d is not below job %q timeout-minutes = %d",
|
|
b.step, step.TimeoutMinutes, b.job, job.TimeoutMinutes)
|
|
}
|
|
})
|
|
}
|
|
}
|