beads/scripts/check_release_cross_compile_test.go
Julian Knutsen fac9a1e32c
ci: same-repo Linux PR jobs on Blacksmith; fold tiny checks and cross-compile legs (F7a) (#7139)
* ci: F7a same-repo Blacksmith moves and job folds (pr.yml, pr-risk.yml)

Move cache-free same-repo pr.yml/pr-risk.yml jobs onto the org's Blacksmith
runner pool (forks/Dependabot stay ubuntu-latest), and fold several
independent single-purpose jobs into fewer multi-step jobs, to cut same-repo
PR queue time without changing either required gate's name or its token
contract:

- fast-checks: folds check-build-tags, check-version-consistency,
  check-migration-hygiene, check-no-beads-changes and fmt-check into one
  Blacksmith job with five independent steps; ci-gate reads each step's
  outcome via job outputs into the same CI_GATE_REQUIRED token the standalone
  job used, plus a FAST_CHECKS backstop token for a lost job.
- check-release-target-cross-compilation: 8 per-target matrix legs collapse
  to 2 (unix/desktop), driven by the new scripts/ci/release-targets.txt
  manifest (kept in sync with .goreleaser.yml) and
  scripts/ci/check-release-cross-compile.sh.
- advisory-reports: folds build-examples and complexity-report (both
  already advisory) into one job, keeping their pre-fold
  step-level continue-on-error/timeout design.
- windows-make-shell: 3-way host matrix folds into one job with an
  aggregator step reproducing the old native/MSYS2/Cygwin pass/fail rules;
  test-windows-doltversion and test-windows-dbproxy-server fold into
  test-windows-small (neither was gated).
- Runner moves: fast-checks, advisory-reports,
  check-release-target-cross-compilation, check-doc-flags,
  pr-policy-wrapper, test-dolt-server-fingerprint (pr.yml) and test-nix
  (pr-risk.yml) move to the same-repo Blacksmith expression at their
  measured vCPU size.

Policy tests: extend TestSameRepoBlacksmithRunners for the moved/added jobs,
add TestSameRepoBlacksmithExpressionSemantics, TestBlacksmithJobsReadNoSecrets
and TestPRCIGateFastChecksTokens, retarget the complexity/build-examples and
dolt-server-fingerprint tests, rewrite
TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser against the new
manifest, and teach the ci-gate simulation helper to resolve
needs.fast-checks.outputs.*. Document the moves/folds and the Blacksmith-off
revert set in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): windows-make-shell/test-windows-small fail natively, raise advisory-reports headroom (review SF-1, SF-2, N-2, N-3)

- windows-make-shell: drop continue-on-error from install/exercise steps
  (kept only on the Cygwin setup step), delete the hand-written "Fail if a
  required Windows Make shell leg failed" aggregator, and delete the unused
  native/msys2/cygwin job outputs. A failing leg now reds the job via native
  GitHub Actions propagation instead of relying on an aggregator step that a
  mutation showed could silently undershoot (SF-1, N-3).
- test-windows-small: same fix - drop continue-on-error from doltversion/
  dbproxy-server steps and delete their aggregator step.
- Add timeout-minutes: 10 to the MSYS2 and Cygwin "Set up ... GNU Make"
  steps, so a hung installer download can no longer take out the whole job
  (SF-2).
- advisory-reports: raise timeout-minutes from 45 to 60 for more headroom
  above the sum of its step timeouts (N-2).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci): real GH-expression evaluator shared with F7c, behavioral xcompile test, secret/run-command coverage (review SF-3, SF-5, SF-6, N-1, N-4, N-6)

- New scripts/ci_blacksmith_runner_test.go: a real GitHub Actions expression
  evaluator (tokenizer + recursive-descent parser for ==, !=, &&, ||, !,
  parens, string literals, dotted lookups, with GitHub's short-circuit-
  returns-operand && / || semantics) that evaluates the REAL pinned
  sameRepoBlacksmith{2,4,8}vcpu expression strings from pr.yml/pr-risk.yml,
  replacing the old TestSameRepoBlacksmithExpressionSemantics' tautological
  hand-written Go mirror (SF-3). This file is the shared helper F7c's own
  TestSameRepoBlacksmithExpressionSemantics should rebase onto - see its
  doc comment for the "Public surface" (the three consts, evalGHExpr, and
  mustEvalGHRunsOn) (SF-5).
- ci_workflow_test.go: drop the old consts/test now living in the shared
  file; extend TestBlacksmithJobsReadNoSecrets to also scan job.If and
  step.If for secret references, not just env/with/run (N-4; container/
  services credentials and matrix-marker resolution stay deferred to F7b);
  pin fast-checks' build-tags step invoking all 3 scripts and the
  migration-hygiene BASE_SHA env, and add
  TestPRWindowsSmallDbproxyServerIsCGODisabled (N-6); add
  TestPRWindowsMakeShellLegsFailTheJob and
  TestPRWindowsMakeShellSetupStepsHaveTimeouts pinning the SF-1/SF-2 fixes.
- New scripts/check_release_cross_compile_test.go: behavioral test for
  check-release-cross-compile.sh using a fake `go` on PATH - asserts a
  failing target propagates as non-zero, every target in the group is still
  attempted, every invocation passes CGO_ENABLED=0 and the gms_pure_go tag,
  and an unknown group fails loudly with zero invocations (SF-6).
- check-release-cross-compile.sh: document the deliberate sequential-build
  (vs. the spec's 2-wide suggestion) choice and its revisit trigger (N-1).
- scripts/BUILD.bazel: add the two new test files to scripts_test's explicit
  srcs list (not a glob); verified against `make bazel-sync-check`.

All new/changed tests mutation-tested: each production-code mutation was
caught by the relevant test, then reverted via `cp` from a /tmp backup and
diff-verified byte-identical (never git stash / git checkout --).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(ci): fix stale check names and aggregator description in CI_REQUIRED_CHECK_TOPOLOGY.md (review N-7)

- Replace the stale "Check build-tag policy", "Check version consistency",
  "Check for .beads changes" and "Check formatting" entries under "Do not
  require these existing check names" with the actual post-fold name,
  "Fast checks (build tags, versions, migrations, beads diff, fmt)", and add
  the cross-compilation and windows-make-shell fold's actual check names.
- Rewrite the windows-make-shell fold description: it no longer has a
  hand-written aggregator step (deleted per SF-1 in the prior commit), and
  "the separate cygwin job" is now described correctly as a pre-fold matrix
  leg, not a job.
- Update the advisory-reports timeout mention from 45 to 60 minutes (N-2).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(ci): drop local design-doc path from topology doc

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 13:13:33 -07:00

163 lines
6.6 KiB
Go

package scripts_test
import (
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"testing"
)
// Review SF-6 (2026-10-03): check-release-cross-compile.sh (the F7a fold of
// pr.yml's check-release-target-cross-compilation matrix into two per-group
// legs) had no behavioural test. A planted "always exit 0" mutation passed
// every other policy test, which would make the required
// CHECK_RELEASE_TARGET_CROSS_COMPILATION token green even with every target
// broken. These tests put a fake `go` on PATH that records every invocation
// and can be told to fail for one target, then assert the script's actual
// failure-propagation, env/flag, and group-coverage behaviour against it.
// fakeGoRecordingScript is a stand-in for `go build` that appends one line
// per invocation to $GO_CALL_LOG (goos, goarch, CGO_ENABLED, and the
// remaining args) and exits non-zero only when GOOS/GOARCH matches
// $GO_FAIL_TARGET.
const fakeGoRecordingScript = `#!/usr/bin/env bash
set -euo pipefail
printf '%s\t%s\t%s\t%s\n' "${GOOS:-}" "${GOARCH:-}" "${CGO_ENABLED:-}" "$*" >> "$GO_CALL_LOG"
if [ "${GOOS:-}/${GOARCH:-}" = "${GO_FAIL_TARGET:-}" ]; then
echo "simulated build failure for ${GOOS:-}/${GOARCH:-}" >&2
exit 1
fi
exit 0
`
// runCheckReleaseCrossCompile copies the real check-release-cross-compile.sh
// into a throwaway directory alongside a caller-supplied manifest (so the
// test does not depend on the real scripts/ci/release-targets.txt growing or
// shrinking), puts a fake `go` on PATH ahead of everything else, and runs the
// script for the given group. failTarget (e.g. "linux/amd64") is the single
// GOOS/GOARCH the fake go fails for; empty means every target succeeds.
func runCheckReleaseCrossCompile(t *testing.T, group, manifest, failTarget string) (out string, err error, log string) {
t.Helper()
if runtime.GOOS == "windows" {
t.Skip("script is a Bash boundary")
}
script, readErr := os.ReadFile(filepath.Join(sourceRepoRoot(t), "scripts", "ci", "check-release-cross-compile.sh"))
if readErr != nil {
t.Fatalf("read check-release-cross-compile.sh: %v", readErr)
}
dir := t.TempDir()
scriptsCI := filepath.Join(dir, "scripts", "ci")
if mkErr := os.MkdirAll(scriptsCI, 0o700); mkErr != nil {
t.Fatal(mkErr)
}
scriptPath := filepath.Join(scriptsCI, "check-release-cross-compile.sh")
if writeErr := os.WriteFile(scriptPath, script, 0o700); writeErr != nil {
t.Fatal(writeErr)
}
if writeErr := os.WriteFile(filepath.Join(scriptsCI, "release-targets.txt"), []byte(manifest), 0o600); writeErr != nil {
t.Fatal(writeErr)
}
fakeBin := filepath.Join(dir, "fakebin")
if mkErr := os.MkdirAll(fakeBin, 0o700); mkErr != nil {
t.Fatal(mkErr)
}
if writeErr := os.WriteFile(filepath.Join(fakeBin, "go"), []byte(fakeGoRecordingScript), 0o700); writeErr != nil {
t.Fatal(writeErr)
}
logPath := filepath.Join(dir, "go-calls.log")
if writeErr := os.WriteFile(logPath, nil, 0o600); writeErr != nil {
t.Fatal(writeErr)
}
cmd := exec.Command("bash", scriptPath, group)
cmd.Env = append(os.Environ(),
"PATH="+fakeBin+":"+os.Getenv("PATH"),
"GO_CALL_LOG="+logPath,
"GO_FAIL_TARGET="+failTarget,
)
outBytes, runErr := cmd.CombinedOutput()
logBytes, _ := os.ReadFile(logPath)
return string(outBytes), runErr, string(logBytes)
}
const testManifest = "linux amd64 unix\n" +
"darwin arm64 unix\n" +
"windows amd64 desktop\n"
// A failure in one target must propagate as a non-zero exit (the gate's
// whole point), must still attempt every other target in the group (so a PR
// touching two platforms sees both failures in one run), must name the
// failing target in an ::error:: line, and must pass CGO_ENABLED=0 and the
// gms_pure_go tag through to every invocation, not just the failing one.
func TestCheckReleaseCrossCompilePropagatesFailureAndCoversGroup(t *testing.T) {
out, err, log := runCheckReleaseCrossCompile(t, "unix", testManifest, "linux/amd64")
if err == nil {
t.Fatalf("script exited 0 with a failing target; want non-zero\noutput:\n%s", out)
}
if !strings.Contains(out, "::error::cross-compilation failed for linux/amd64") {
t.Errorf("output does not name the failing target:\n%s", out)
}
if !strings.Contains(out, "::error::cross-compilation failed for: linux/amd64") {
t.Errorf("output does not contain the final failed-targets summary line:\n%s", out)
}
// Both unix-group targets must have been attempted, in spite of the first
// one failing.
for _, want := range []string{"linux\tamd64", "darwin\tarm64"} {
if !strings.Contains(log, want) {
t.Errorf("fake go call log is missing an attempt for %s; every group target must run even after an earlier one fails:\n%s", want, log)
}
}
// The desktop-group target must not have been touched by a unix-group run.
if strings.Contains(log, "windows\tamd64") {
t.Errorf("fake go call log recorded a desktop-group target during a unix-group run:\n%s", log)
}
for _, line := range strings.Split(strings.TrimRight(log, "\n"), "\n") {
if line == "" {
continue
}
fields := strings.Split(line, "\t")
if len(fields) != 4 {
t.Fatalf("malformed fake go call log line %q", line)
}
if fields[2] != "0" {
t.Errorf("fake go call %q: CGO_ENABLED = %q, want \"0\"", line, fields[2])
}
if !strings.Contains(fields[3], "-tags gms_pure_go ./...") {
t.Errorf("fake go call %q: args = %q, want them to contain \"-tags gms_pure_go ./...\"", line, fields[3])
}
}
}
// Every target in a group still passes without a failTarget.
func TestCheckReleaseCrossCompileAllTargetsPass(t *testing.T) {
out, err, log := runCheckReleaseCrossCompile(t, "unix", testManifest, "")
if err != nil {
t.Fatalf("script failed with no simulated failures: %v\noutput:\n%s", err, out)
}
if !strings.Contains(out, "cross-compilation passed for group 'unix'") {
t.Errorf("output does not report success for the unix group:\n%s", out)
}
for _, want := range []string{"linux\tamd64", "darwin\tarm64"} {
if !strings.Contains(log, want) {
t.Errorf("fake go call log is missing an attempt for %s:\n%s", want, log)
}
}
}
// A group name absent from the manifest must fail loudly, not silently
// succeed with zero targets built.
func TestCheckReleaseCrossCompileUnknownGroupFails(t *testing.T) {
out, err, log := runCheckReleaseCrossCompile(t, "bogus-group", testManifest, "")
if err == nil {
t.Fatalf("script exited 0 for an unknown group; want non-zero\noutput:\n%s", out)
}
if !strings.Contains(out, "::error::no release targets found for group 'bogus-group'") {
t.Errorf("output does not report the unknown group:\n%s", out)
}
if log != "" {
t.Errorf("fake go was invoked for an unknown group; want zero invocations, got log:\n%s", log)
}
}