Compare commits

...
Sign in to create a new pull request.

1 commit

Author SHA1 Message Date
Julian Knutsen
18eda16bcb ci(bazel): cross-compile every release target with Bazel, retire the go build job
pr.yml's check-release-target-cross-compilation ran `CGO_ENABLED=0
GOOS/GOARCH go build -tags gms_pure_go ./...` for the eight release targets
on two GitHub/Blacksmith runners. It is now a step of bazel.yml's pure-Go
lane (BAZEL_PURE, already required by ci-gate):
scripts/ci/bazel-release-cross-compile.sh builds //tools/bazel:release_cross
once, remotely, with --//tools/bazel:release_platforms set from
scripts/ci/release-targets.txt (still the single source of truth, pinned to
.goreleaser.yml). release_cross's split transition configures every
go_library/go_binary for each rules_go platform with cgo off, so all eight
platforms build in parallel in one invocation.

- tools/bazel/release_cross.bzl: the split-transition rule and the
  string-list flag.
- tools/bazel/go_srcs.py (make bazel-sync, checked by bazel-sync-check and
  TestBazelGoSrcsBlocksCurrent) keeps the release_cross target list; the
  script checks with bazel query that every Bazel package holding Go
  targets is reached before it builds.
- //tools/bazel:pure_bd_has_no_cgo_only_deps: Bazel compiles gozstd's pure
  stubs where `go build` rejects them, so the pure bd of every platform
  must not link gozstd.
- //internal/storage/embeddeddolt/cmd:cmd (cgo-only main, skipped by
  `go build ./...` under CGO_ENABLED=0) is tagged cgo-only and incompatible
  with pure builds.
- A failed build is re-run per platform (cache hits) to name each failing
  GOOS/GOARCH.

Same cgo-off coverage boundary as the retired job. Refs ga-96smfk.20.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 02:50:08 +00:00
16 changed files with 703 additions and 333 deletions

View file

@ -1049,6 +1049,16 @@ jobs:
//internal/storage/embeddeddolt:embeddeddolt_test
//internal/tracker:tracker_test
# Every release target (scripts/ci/release-targets.txt) on every PR, so
# a platform-gating mistake is caught here instead of at tag time
# (#5662): the whole tree, cgo off, one remote `bazel build
# --platforms=...` per target (the script's header has the coverage
# boundary: the cgo release builds are not compiled here).
- name: Cross-compile every release target (--config=release-cross)
run: |
set -euo pipefail
./scripts/ci/bazel-release-cross-compile.sh
# `go build` with CGO_ENABLED=0 rejects a pure binary whose imports reach
# dolt's cgo-only store/nbs (gozstd). Bazel compiles gozstd's pure stubs
# instead, whose init panics (third_party/patches/gozstd_nocgo.patch), so

View file

@ -233,8 +233,8 @@ jobs:
#
# D2 step 3: stands down where bazel-coverage says pr_lanes; Bazel's
# `pure-Go and js/wasm` lane (bazel-pure) runs the same boundaries, and
# check-release-target-cross-compilation still `go build`s ./... with
# CGO_ENABLED=0 on every PR.
# also builds the whole tree with cgo off for every release target
# (scripts/ci/bazel-release-cross-compile.sh).
check-cmd-bd-puregeo-tests:
name: Check pure-Go and js/wasm boundaries (CGO_ENABLED=0)
runs-on: ubuntu-latest
@ -318,61 +318,6 @@ jobs:
run: |
go test -tags gms_pure_go -count=1 -short -run '^Test(Help|CheckRemoteSafety|FormatDestroyToken|ShouldWireInitRemote|ExtractPrefix|IsNumericID|GetWorktreeGitDir|DriftItemStatuses|RunDriftChecks|IsServerProbablyRunning|CheckHooksDriftNotGitRepo|CheckServerDriftNoBeadsDir|CheckRemoteDriftNoBeadsDir|NocgoFactoriesHonorSharedServer|NocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|NocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag)' ./cmd/bd
# Compile every shipped release target on every PR, so a platform-gating
# mistake is caught here instead of at tag time (#5662).
#
# Coverage boundary, recorded so the green is not over-read: every leg below
# builds CGO_ENABLED=0, but .goreleaser.yml builds bd-linux-amd64,
# bd-linux-arm64 and bd-windows-amd64 with CGO_ENABLED=1, and the darwin pair
# is built natively at CGO_ENABLED=1 by release.yml's goreleaser-macos job.
# Five of these eight legs are therefore a weaker proxy than the binary they
# stand for: the cgo half of the tree (beads_cgo.go ->
# internal/storage/embeddeddolt) is never compiled here, so a break confined
# to it still passes this check. Only android/arm64, windows/arm64 and
# freebsd/amd64 replicate their release configuration exactly.
#
# The build scope is deliberately ./... and not the ./cmd/bd that every
# .goreleaser.yml build id declares as main:. Compiling the whole tree per
# platform is intentional extra coverage, not an oversight.
#
# F7a: folded from 8 per-target jobs into 2 per-group jobs. The target list
# moved out of this matrix and into scripts/ci/release-targets.txt, which
# scripts/ci/check-release-cross-compile.sh reads and
# TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser pins against
# .goreleaser.yml, so the manifest (not this YAML) is now the single
# source of truth for the target set. Moved to Blacksmith for same-repo
# PRs/merge_group: this job restores no GitHub-saved build cache (only a
# best-effort module-cache restore, whose misses are cheap network
# fetches, not lost compute). Runner choice must match the
# TestSameRepoBlacksmithRunners pin.
check-release-target-cross-compilation:
name: Check release target cross-compilation (${{ matrix.group }})
runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
group: [unix, desktop]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6
- name: Set up Go
id: setup-go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: 'go.mod'
cache: false
- name: Restore Go module cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/go/pkg/mod
key: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
restore-keys: beads-go-mod-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
- name: Build complete release target tree for group ${{ matrix.group }}
run: ./scripts/ci/check-release-cross-compile.sh ${{ matrix.group }}
# The lingering-handle liveness regression test is //go:build windows: the
# Ubuntu job above selects it by name but compiles it out, and main.yml's
# Windows job runs only on pushes to main (build/smoke, no go test). Run
@ -2290,7 +2235,6 @@ jobs:
- build-artifacts
- fast-checks
- check-cmd-bd-puregeo-tests
- check-release-target-cross-compilation
- test-windows-liveness
- worktree-remove-windows
- test-windows-liveness-prebuilt
@ -2319,7 +2263,6 @@ jobs:
CHECK_MIGRATION_HYGIENE
CHECK_NO_BEADS_CHANGES
CHECK_CMD_BD_PUREGEO_TESTS
CHECK_RELEASE_TARGET_CROSS_COMPILATION
CHECK_DOC_FLAGS
CHECK_DOC_FRESHNESS_PLATFORMS
PR_PREFLIGHT_PLATFORMS
@ -2359,7 +2302,6 @@ jobs:
# job.
FAST_CHECKS: ${{ needs.fast-checks.result }}
CHECK_CMD_BD_PUREGEO_TESTS: ${{ needs.check-cmd-bd-puregeo-tests.result }}
CHECK_RELEASE_TARGET_CROSS_COMPILATION: ${{ needs.check-release-target-cross-compilation.result }}
TEST_WINDOWS_LIVENESS: ${{ needs.test-windows-liveness.result }}
WORKTREE_REMOVE_WINDOWS: ${{ needs.worktree-remove-windows.result }}
# F4 side-by-side rollout: always evaluated, like the native pair

View file

@ -143,9 +143,9 @@ Before starting a release:
- [ ] **Upgrade smoke tests pass** (`make test-upgrade`) — see [Release Stability Gate](engdocs/RELEASE-STABILITY-GATE.md)
- [ ] **Regression tests pass** (`make test-regression`)
- [ ] **Every release target cross-compiles** — see
[Cross-compile before tagging](#cross-compile-before-tagging). PR CI does
not build them ([#5662](https://github.com/gastownhall/beads/issues/5662)),
and a target that fails at tag time burns the tag.
[Cross-compile before tagging](#cross-compile-before-tagging). PR CI
builds them only with cgo off, and a target that fails at tag time
burns the tag.
- [ ] **CHANGELOG.md updated with release notes** (see format below)
- [ ] **CHANGELOG rollup checked** — nothing left under `[Unreleased]` that
belongs in this release, and nothing filed under a *previous* release's
@ -163,12 +163,15 @@ Before starting a release:
### Cross-compile before tagging
`.goreleaser.yml` builds darwin, linux, windows **and freebsd**, but PR CI
builds none of the cross targets — the v1.2.0 tag burned on a freebsd
compilation failure that no pre-tag gate could have caught
([#5661](https://github.com/gastownhall/beads/pull/5661) fixed the break,
[#5662](https://github.com/gastownhall/beads/issues/5662) tracks the CI gap).
Until that gap is closed, build them by hand before tagging:
`.goreleaser.yml` builds darwin, linux, windows **and freebsd**; the v1.2.0
tag burned on a freebsd compilation failure no pre-tag gate caught
([#5661](https://github.com/gastownhall/beads/pull/5661),
[#5662](https://github.com/gastownhall/beads/issues/5662)). PR CI now builds
the whole tree for every target in `scripts/ci/release-targets.txt` with Bazel
(`scripts/ci/bazel-release-cross-compile.sh`, in bazel.yml's pure-Go lane),
but with cgo off, while linux/amd64, linux/arm64, windows/amd64 and the darwin
pair ship with `CGO_ENABLED=1`. Build the real release configuration by hand
before tagging:
```bash
goreleaser build --snapshot --clean

View file

@ -636,8 +636,6 @@ Do not require these existing check names directly:
[Repository Guards as Bazel Tests](#repository-guards-as-bazel-tests))
- `Check pure-Go and js/wasm boundaries (CGO_ENABLED=0)`
- `Check doc flags freshness`
- `Check release target cross-compilation (unix)` and `(desktop)` (F7a fold
of the former eight-target `check-release-target-cross-compilation` matrix)
- `Windows Make shell (native, msys2, cygwin)` (F7a fold of the former
`host: [native, msys2, cygwin]` three-job matrix)
- `Test (Windows) small packages (doltversion, dbproxy server)` (F7a fold of
@ -926,8 +924,16 @@ manifests. On those PRs they are the tiers' only pre-merge run, and
(`Test (Dolt server fingerprint)`, `TEST_DOLT_SERVER_FINGERPRINT`)
pulled a docker image the workers do not have, so it is retired; the
image tag stays pinned to the dolt release by
`TestPinnedDoltCLIMatchesContainerImage`. `check-release-target-cross-compilation`
still `go build`s `./...` with `CGO_ENABLED=0` on every PR.
`TestPinnedDoltCLIMatchesContainerImage`. The release-target
cross-compilation (formerly pr.yml's
`check-release-target-cross-compilation`, `go build ./...` with
`CGO_ENABLED=0` per target) is a step of bazel.yml's `bazel-pure` lane
(`BAZEL_PURE`): `scripts/ci/bazel-release-cross-compile.sh` runs one
remote `bazel build //tools/bazel:release_cross`, every `go_library` and
`go_binary` for each row of `scripts/ci/release-targets.txt`, plus
`//tools/bazel:pure_bd_has_no_cgo_only_deps` (a pure bd must not link
gozstd, which `go build` rejects with cgo off). Same cgo-off boundary as
the job it replaces.
- Also kept on every PR, in the required job `scripts-go-checks`
(`SCRIPTS_GO_CHECKS`, one leg, `Go checks (vet)`): `go test`'s own vet
checks (cmd/go's `defaultVetFlags`, policy-tested equal to the
@ -1220,7 +1226,8 @@ scope, not this slice's.
`scripts/ci/check-release-cross-compile.sh <group>`, which builds every
target in its group sequentially and reports every failure before exiting
non-zero, so a PR touching two platforms at once sees both failures in one
log instead of needing a per-target re-run.
log instead of needing a per-target re-run. (Since retired: bazel.yml's `bazel-pure`
lane builds the same manifest with Bazel, `--platforms` per target.)
- **`advisory-reports` fold.** `build-examples` and `complexity-report` (both
already advisory: neither was in ci-gate's `needs`/`CI_GATE_REQUIRED`)
became one job's steps. Each keeps its pre-fold step-level
@ -1531,7 +1538,6 @@ PR timings, 2026-10): `fast-checks` (~40 s), `pr-policy-wrapper`
(native/darwin/windows, up to ~4 min), `check-doc-flags` (~1.7 min),
`check-doc-freshness-platforms` and `pr-preflight-platforms` (Linux,
Windows and macOS legs, up to ~5 min on Windows),
`check-release-target-cross-compilation` (~5 min),
`windows-make-shell` (~2.7 min),
`windows-test-binaries` plus the prebuilt Windows pair (~7 min end to
end), the advisory native Windows pair (~5 min; the gate waits for them),

View file

@ -8,9 +8,17 @@ go_library(
deps = ["//internal/storage/embeddeddolt"],
)
# main.go is //go:build cgo: without cgo the package is empty, which
# `go build ./...` skips and a go_binary cannot link. cgo-only: left out of
# //tools/bazel:release_cross (tools/bazel/go_srcs.py).
go_binary(
name = "cmd",
embed = [":cmd_lib"],
tags = ["cgo-only"],
target_compatible_with = select({
"//tools/bazel:pure": ["@platforms//:incompatible"],
"//conditions:default": [],
}),
visibility = ["//:__subpackages__"],
)

View file

@ -852,6 +852,12 @@ var allowedBazelTestTags = map[string]string{
// integration tag, as releases are, and running it again under the
// integration build would only repeat it against a test-shaped bd.
"pr-core-only": "runs only in --config=prcore/ci; excluded from --config=integration",
// For a go_binary whose every source file is `//go:build cgo`: it also
// sets target_compatible_with incompatible under //tools/bazel:pure, and
// tools/bazel/go_srcs.py and scripts/ci/bazel-release-cross-compile.sh
// leave it out of //tools/bazel:release_cross by this tag (`go build
// ./...` skips the package with CGO_ENABLED=0).
"cgo-only": "a go_binary with only cgo sources: incompatible with pure builds and left out of //tools/bazel:release_cross",
}
// bazelPRCoreExcludedTags are the tags whose targets never run in the PR-core

View file

@ -9,155 +9,219 @@ import (
"testing"
)
// Review SF-6 (2026-10-03): check-release-cross-compile.sh (the F7a fold of
// pr.yml's check-release-target-cross-compilation matrix into two per-group
// legs) had no behavioural test. A planted "always exit 0" mutation passed
// every other policy test, which would make the required
// CHECK_RELEASE_TARGET_CROSS_COMPILATION token green even with every target
// broken. These tests put a fake `go` on PATH that records every invocation
// and can be told to fail for one target, then assert the script's actual
// failure-propagation, env/flag, and group-coverage behaviour against it.
// bazel-release-cross-compile.sh is the release-target cross-compilation
// gate: bazel.yml's pure-Go lane runs it, and it builds
// //tools/bazel:release_cross for every row of scripts/ci/release-targets.txt
// in one invocation. These tests put a fake `bazel` on PATH that records
// every invocation and can be told to fail for one platform, then assert the
// script's platform list, coverage check, flags, failure propagation and
// per-platform attribution against it, so an "always exit 0" mutation cannot
// keep the gate green with every target broken.
// fakeGoRecordingScript is a stand-in for `go build` that appends one line
// per invocation to $GO_CALL_LOG (goos, goarch, CGO_ENABLED, and the
// remaining args) and exits non-zero only when GOOS/GOARCH matches
// $GO_FAIL_TARGET.
const fakeGoRecordingScript = `#!/usr/bin/env bash
// fakeBazelRecordingScript stands in for bazel. `query` prints
// $FAKE_BUILT_PKGS for the deps of //tools/bazel:release_cross and
// $FAKE_WANT_PKGS for anything else; every command appends one line
// ("<command>\t<args>") to $BAZEL_CALL_LOG; `build` exits non-zero when its
// --//tools/bazel:release_platforms list names $BAZEL_FAIL_PLATFORM.
const fakeBazelRecordingScript = `#!/usr/bin/env bash
set -euo pipefail
printf '%s\t%s\t%s\t%s\n' "${GOOS:-}" "${GOARCH:-}" "${CGO_ENABLED:-}" "$*" >> "$GO_CALL_LOG"
if [ "${GOOS:-}/${GOARCH:-}" = "${GO_FAIL_TARGET:-}" ]; then
echo "simulated build failure for ${GOOS:-}/${GOARCH:-}" >&2
exit 1
cmd="$1"; shift
printf '%s\t%s\n' "$cmd" "$*" >> "$BAZEL_CALL_LOG"
if [ "$cmd" = query ]; then
case "$*" in
*"deps(//tools/bazel:release_cross)"*) printf '%s\n' $FAKE_BUILT_PKGS ;;
*) printf '%s\n' $FAKE_WANT_PKGS ;;
esac
exit 0
fi
for a in "$@"; do
case "$a" in
--//tools/bazel:release_platforms=*)
if [ -n "${BAZEL_FAIL_PLATFORM:-}" ] && [[ ",${a#*=}," == *",$BAZEL_FAIL_PLATFORM,"* ]]; then
echo "simulated build failure for $a" >&2
exit 1
fi ;;
esac
done
exit 0
`
// runCheckReleaseCrossCompile copies the real check-release-cross-compile.sh
// into a throwaway directory alongside a caller-supplied manifest (so the
// test does not depend on the real scripts/ci/release-targets.txt growing or
// shrinking), puts a fake `go` on PATH ahead of everything else, and runs the
// script for the given group. failTarget (e.g. "linux/amd64") is the single
// GOOS/GOARCH the fake go fails for; empty means every target succeeds.
func runCheckReleaseCrossCompile(t *testing.T, group, manifest, failTarget string) (out string, err error, log string) {
const testPkgs = "cmd/bd internal/types"
// runBazelReleaseCrossCompile copies the real script next to a
// caller-supplied manifest (so the test does not track the real
// release-targets.txt), puts the fake bazel first on PATH and runs it.
func runBazelReleaseCrossCompile(t *testing.T, manifest, failPlatform, builtPkgs string, args ...string) (out string, err error, calls []string) {
t.Helper()
if runtime.GOOS == "windows" {
t.Skip("script is a Bash boundary")
}
script, readErr := os.ReadFile(filepath.Join(sourceRepoRoot(t), "scripts", "ci", "check-release-cross-compile.sh"))
script, readErr := os.ReadFile(filepath.Join(sourceRepoRoot(t), "scripts", "ci", "bazel-release-cross-compile.sh"))
if readErr != nil {
t.Fatalf("read check-release-cross-compile.sh: %v", readErr)
t.Fatalf("read bazel-release-cross-compile.sh: %v", readErr)
}
dir := t.TempDir()
scriptsCI := filepath.Join(dir, "scripts", "ci")
if mkErr := os.MkdirAll(scriptsCI, 0o700); mkErr != nil {
t.Fatal(mkErr)
}
scriptPath := filepath.Join(scriptsCI, "check-release-cross-compile.sh")
if writeErr := os.WriteFile(scriptPath, script, 0o700); writeErr != nil {
t.Fatal(writeErr)
}
if writeErr := os.WriteFile(filepath.Join(scriptsCI, "release-targets.txt"), []byte(manifest), 0o600); writeErr != nil {
t.Fatal(writeErr)
}
fakeBin := filepath.Join(dir, "fakebin")
if mkErr := os.MkdirAll(fakeBin, 0o700); mkErr != nil {
t.Fatal(mkErr)
for _, d := range []string{scriptsCI, fakeBin} {
if mkErr := os.MkdirAll(d, 0o700); mkErr != nil {
t.Fatal(mkErr)
}
}
if writeErr := os.WriteFile(filepath.Join(fakeBin, "go"), []byte(fakeGoRecordingScript), 0o700); writeErr != nil {
t.Fatal(writeErr)
scriptPath := filepath.Join(scriptsCI, "bazel-release-cross-compile.sh")
for path, content := range map[string]string{
scriptPath: string(script),
filepath.Join(scriptsCI, "release-targets.txt"): manifest,
filepath.Join(fakeBin, "bazel"): fakeBazelRecordingScript,
} {
if writeErr := os.WriteFile(path, []byte(content), 0o700); writeErr != nil {
t.Fatal(writeErr)
}
}
logPath := filepath.Join(dir, "go-calls.log")
logPath := filepath.Join(dir, "bazel-calls.log")
if writeErr := os.WriteFile(logPath, nil, 0o600); writeErr != nil {
t.Fatal(writeErr)
}
cmd := exec.Command("bash", scriptPath, group)
cmd := exec.Command("bash", append([]string{scriptPath}, args...)...)
cmd.Dir = dir
cmd.Env = append(os.Environ(),
"PATH="+fakeBin+":"+os.Getenv("PATH"),
"GO_CALL_LOG="+logPath,
"GO_FAIL_TARGET="+failTarget,
"BAZEL_CALL_LOG="+logPath,
"BAZEL_FAIL_PLATFORM="+failPlatform,
"FAKE_WANT_PKGS="+testPkgs,
"FAKE_BUILT_PKGS="+builtPkgs,
)
outBytes, runErr := cmd.CombinedOutput()
logBytes, _ := os.ReadFile(logPath)
return string(outBytes), runErr, string(logBytes)
for _, line := range strings.Split(strings.TrimRight(string(logBytes), "\n"), "\n") {
if line != "" {
calls = append(calls, line)
}
}
return string(outBytes), runErr, calls
}
const testManifest = "linux amd64 unix\n" +
"darwin arm64 unix\n" +
"windows amd64 desktop\n"
const testReleaseManifest = "# comment\n" +
"linux amd64\n" +
"\n" +
"windows arm64\n" +
"darwin arm64\n"
// A failure in one target must propagate as a non-zero exit (the gate's
// whole point), must still attempt every other target in the group (so a PR
// touching two platforms sees both failures in one run), must name the
// failing target in an ::error:: line, and must pass CGO_ENABLED=0 and the
// gms_pure_go tag through to every invocation, not just the failing one.
func TestCheckReleaseCrossCompilePropagatesFailureAndCoversGroup(t *testing.T) {
out, err, log := runCheckReleaseCrossCompile(t, "unix", testManifest, "linux/amd64")
if err == nil {
t.Fatalf("script exited 0 with a failing target; want non-zero\noutput:\n%s", out)
}
if !strings.Contains(out, "::error::cross-compilation failed for linux/amd64") {
t.Errorf("output does not name the failing target:\n%s", out)
}
if !strings.Contains(out, "::error::cross-compilation failed for: linux/amd64") {
t.Errorf("output does not contain the final failed-targets summary line:\n%s", out)
}
// Both unix-group targets must have been attempted, in spite of the first
// one failing.
for _, want := range []string{"linux\tamd64", "darwin\tarm64"} {
if !strings.Contains(log, want) {
t.Errorf("fake go call log is missing an attempt for %s; every group target must run even after an earlier one fails:\n%s", want, log)
}
}
// The desktop-group target must not have been touched by a unix-group run.
if strings.Contains(log, "windows\tamd64") {
t.Errorf("fake go call log recorded a desktop-group target during a unix-group run:\n%s", log)
}
for _, line := range strings.Split(strings.TrimRight(log, "\n"), "\n") {
if line == "" {
continue
}
fields := strings.Split(line, "\t")
if len(fields) != 4 {
t.Fatalf("malformed fake go call log line %q", line)
}
if fields[2] != "0" {
t.Errorf("fake go call %q: CGO_ENABLED = %q, want \"0\"", line, fields[2])
}
if !strings.Contains(fields[3], "-tags gms_pure_go ./...") {
t.Errorf("fake go call %q: args = %q, want them to contain \"-tags gms_pure_go ./...\"", line, fields[3])
func buildCalls(calls []string) []string {
var out []string
for _, c := range calls {
if strings.HasPrefix(c, "build\t") {
out = append(out, c)
}
}
return out
}
// Every target in a group still passes without a failTarget.
func TestCheckReleaseCrossCompileAllTargetsPass(t *testing.T) {
out, err, log := runCheckReleaseCrossCompile(t, "unix", testManifest, "")
const releaseFlag = "--//tools/bazel:release_platforms="
// One `bazel build --keep_going` of //tools/bazel:release_cross with every
// manifest row as a rules_go platform name, the caller's extra flags passed
// through, after a coverage check that queries the Go packages bazel knows
// and the ones release_cross reaches.
func TestBazelReleaseCrossCompileBuildsEveryManifestPlatform(t *testing.T) {
out, err, calls := runBazelReleaseCrossCompile(t, testReleaseManifest, "", testPkgs, "--config=remote-exec")
if err != nil {
t.Fatalf("script failed with no simulated failures: %v\noutput:\n%s", err, out)
}
if !strings.Contains(out, "cross-compilation passed for group 'unix'") {
t.Errorf("output does not report success for the unix group:\n%s", out)
if len(calls) != 3 {
t.Fatalf("bazel calls = %q, want two queries and one build", calls)
}
for _, want := range []string{"linux\tamd64", "darwin\tarm64"} {
if !strings.Contains(log, want) {
t.Errorf("fake go call log is missing an attempt for %s:\n%s", want, log)
for i, want := range []string{
`kind("go_library|go_binary", //...) except siblings(attr(tags, "\bcgo-only\b", `,
"deps(//tools/bazel:release_cross)",
} {
if !strings.HasPrefix(calls[i], "query\t") || !strings.Contains(calls[i], want) || !strings.Contains(calls[i], "--output=package") {
t.Errorf("bazel call %d = %q, want a package query containing %q", i, calls[i], want)
}
}
build := calls[2]
for _, required := range []string{
"build\t--keep_going",
releaseFlag + "linux_amd64,windows_arm64,darwin_arm64",
"--config=remote-exec -- //tools/bazel:release_cross",
} {
if !strings.Contains(build, required) {
t.Errorf("build call %q does not contain %q", build, required)
}
}
if !strings.Contains(out, "release cross-compilation passed: linux/amd64 windows/arm64 darwin/arm64") {
t.Errorf("output does not report success for every target:\n%s", out)
}
}
// A Go package release_cross does not reach fails the script before any
// build, naming the package.
func TestBazelReleaseCrossCompileFailsOnUncoveredPackage(t *testing.T) {
out, err, calls := runBazelReleaseCrossCompile(t, testReleaseManifest, "", "cmd/bd")
if err == nil {
t.Fatalf("script exited 0 with an uncovered package; want non-zero\noutput:\n%s", out)
}
if !strings.Contains(out, "::error::Go packages //tools/bazel:release_cross does not build") || !strings.Contains(out, "internal/types") {
t.Errorf("output does not name the uncovered package:\n%s", out)
}
if b := buildCalls(calls); len(b) != 0 {
t.Errorf("bazel build ran despite an uncovered package: %q", b)
}
}
// A failing platform fails the script; each platform is then rebuilt alone,
// so the output names exactly the failing ones.
func TestBazelReleaseCrossCompilePropagatesFailureAndNamesPlatform(t *testing.T) {
out, err, calls := runBazelReleaseCrossCompile(t, testReleaseManifest, "windows_arm64", testPkgs)
if err == nil {
t.Fatalf("script exited 0 with a failing platform; want non-zero\noutput:\n%s", out)
}
builds := buildCalls(calls)
if len(builds) != 4 {
t.Fatalf("bazel build calls = %q, want the full build plus one per platform", builds)
}
for i, platform := range []string{"linux_amd64", "windows_arm64", "darwin_arm64"} {
if !strings.Contains(builds[i+1], releaseFlag+platform+" ") {
t.Errorf("attribution build %d = %q, want %s alone", i, builds[i+1], platform)
}
}
for _, want := range []string{
"::error::release cross-compilation failed for windows/arm64",
"::error::release cross-compilation failed for: windows/arm64",
} {
if !strings.Contains(out, want) {
t.Errorf("output does not contain %q:\n%s", want, out)
}
}
for _, unwanted := range []string{"failed for linux/amd64", "failed for darwin/arm64"} {
if strings.Contains(out, unwanted) {
t.Errorf("output blames a passing platform (%q):\n%s", unwanted, out)
}
}
}
// A group name absent from the manifest must fail loudly, not silently
// succeed with zero targets built.
func TestCheckReleaseCrossCompileUnknownGroupFails(t *testing.T) {
out, err, log := runCheckReleaseCrossCompile(t, "bogus-group", testManifest, "")
// An empty manifest must fail loudly rather than pass with nothing built.
func TestBazelReleaseCrossCompileEmptyManifestFails(t *testing.T) {
out, err, calls := runBazelReleaseCrossCompile(t, "# nothing\n\n", "", testPkgs)
if err == nil {
t.Fatalf("script exited 0 for an unknown group; want non-zero\noutput:\n%s", out)
t.Fatalf("script exited 0 for an empty manifest; want non-zero\noutput:\n%s", out)
}
if !strings.Contains(out, "::error::no release targets found for group 'bogus-group'") {
t.Errorf("output does not report the unknown group:\n%s", out)
if !strings.Contains(out, "::error::no release targets in") {
t.Errorf("output does not report the empty manifest:\n%s", out)
}
if log != "" {
t.Errorf("fake go was invoked for an unknown group; want zero invocations, got log:\n%s", log)
if len(calls) != 0 {
t.Errorf("bazel ran for an empty manifest: %q", calls)
}
}
// A malformed row (not exactly GOOS GOARCH) must fail, not be skipped.
func TestBazelReleaseCrossCompileMalformedRowFails(t *testing.T) {
out, err, _ := runBazelReleaseCrossCompile(t, "linux amd64 unix\n", "", testPkgs)
if err == nil {
t.Fatalf("script exited 0 for a malformed row; want non-zero\noutput:\n%s", out)
}
if !strings.Contains(out, "::error::malformed row") {
t.Errorf("output does not report the malformed row:\n%s", out)
}
}

View file

@ -0,0 +1,91 @@
#!/usr/bin/env bash
# Release-target cross-compilation gate (bazel.yml's pure-Go lane): build
# every non-test Go target of the module for every GOOS/GOARCH row of
# scripts/ci/release-targets.txt, the Bazel form of
# `CGO_ENABLED=0 GOOS=<os> GOARCH=<arch> go build -tags gms_pure_go ./...`.
#
# One `bazel build` of //tools/bazel:release_cross with
# --//tools/bazel:release_platforms set to every manifest row: its split
# transition builds every listed target once per platform with cgo off (and
# the gms_pure_go tag every build carries), all platforms in parallel on the
# remote workers. --keep_going reports every broken target of every
# platform in one run, and a failed run names each failing platform. The target list is every go_library and go_binary
# (the packages `go build ./...` compiles; go_test targets are `go test`'s),
# kept by tools/bazel/go_srcs.py (`make bazel-sync`) minus the ones tagged
# cgo-only (incompatible with //tools/bazel:pure), which `go build ./...` skips
# under CGO_ENABLED=0. Before building, the list is checked against `bazel
# query`, so a target the generator missed fails here instead of going
# uncompiled. The list also holds //tools/bazel:pure_bd_has_no_cgo_only_deps:
# Bazel compiles gozstd's pure stubs (third_party/patches/
# gozstd_nocgo.patch) where `go build` rejects a CGO_ENABLED=0 import of it,
# so that check fails a platform whose pure bd links gozstd.
#
# Coverage boundary, as in the `go build` job this replaces: every target
# builds with cgo off, but .goreleaser.yml builds linux/amd64, linux/arm64
# and windows/amd64 with CGO_ENABLED=1, and release.yml builds the darwin
# pair natively with CGO_ENABLED=1, so the cgo half of the tree (beads_cgo.go
# -> internal/storage/embeddeddolt) is compiled for none of them here.
#
# Usage: bazel-release-cross-compile.sh [extra bazel build flags...]
set -euo pipefail
manifest="$(dirname "${BASH_SOURCE[0]}")/release-targets.txt"
targets=()
while read -r goos goarch extra; do
if [[ -z "$goarch" || -n "$extra" ]]; then
echo "::error::malformed row in $manifest: '$goos $goarch $extra', want 'GOOS GOARCH'"
exit 1
fi
targets+=("$goos/$goarch")
done < <(grep -v '^[[:space:]]*#' "$manifest" | grep -v '^[[:space:]]*$' || true)
if [[ "${#targets[@]}" -eq 0 ]]; then
echo "::error::no release targets in $manifest"
exit 1
fi
platforms=()
for target in "${targets[@]}"; do
platforms+=("${target%/*}_${target#*/}")
done
# Every Bazel package holding a Go target (less testonly targets and the
# packages of cgo-only ones) must be reached by //tools/bazel:release_cross;
# a listed label that no longer exists fails the build itself.
universe='kind("go_library|go_binary", //...)'
wanted="$universe except siblings(attr(tags, \"\\bcgo-only\\b\", $universe)) except attr(testonly, 1, $universe)"
# One query at a time: they share the Bazel server.
want_pkgs="$(bazel query "$wanted" --output=package | LC_ALL=C sort -u)"
built_pkgs="$(bazel query 'deps(//tools/bazel:release_cross)' --output=package | LC_ALL=C sort -u)"
missing="$(LC_ALL=C comm -23 <(echo "$want_pkgs") <(echo "$built_pkgs"))"
if [[ -n "$missing" ]]; then
echo "::error::Go packages //tools/bazel:release_cross does not build (run \`make bazel-sync\`; a package whose Go targets are all invisible to //tools/bazel needs one visible to it):"
echo "$missing"
exit 1
fi
platform_list="$(IFS=,; echo "${platforms[*]}")"
echo "::group::bazel build //tools/bazel:release_cross for ${targets[*]}"
status=0
bazel build --keep_going "--//tools/bazel:release_platforms=$platform_list" \
"$@" -- //tools/bazel:release_cross || status=$?
echo "::endgroup::"
if [[ "$status" -ne 0 ]]; then
# Bazel's errors above name the target, not the platform (each split
# configuration's output directory is a hash). Rebuild one platform at a
# time to name them: the same configurations, so everything that built
# is a cache hit and only the failures run again.
failed=()
for i in "${!platforms[@]}"; do
if ! bazel build --keep_going "--//tools/bazel:release_platforms=${platforms[$i]}" \
"$@" -- //tools/bazel:release_cross >/dev/null 2>&1; then
echo "::error::release cross-compilation failed for ${targets[$i]}"
failed+=("${targets[$i]}")
fi
done
echo "::error::release cross-compilation failed for: ${failed[*]:-(bazel exit $status; no single platform reproduced it, see the log above)}"
exit 1
fi
echo "release cross-compilation passed: ${targets[*]}"

View file

@ -1,64 +0,0 @@
#!/usr/bin/env bash
# Build the whole tree (not just ./cmd/bd) for every release target in the
# given group of scripts/ci/release-targets.txt, with CGO disabled.
#
# This is the F7a fold of pr.yml's check-release-target-cross-compilation job:
# one call per matrix leg (group), each building its targets sequentially in a
# single job/runner instead of one runner per target. Every target in the
# group runs even if an earlier one fails, so a PR touching two platforms at
# once sees both failures in one log instead of needing a re-run per leg.
#
# Usage: check-release-cross-compile.sh <group>
#
# Review N-1 (2026-10-03): the spec sketched a 2-wide parallel build per leg
# with a per-target GOCACHE. This builds sequentially with one shared GOCACHE
# instead - simpler, and safe because the targets in a group never touch the
# same GOCACHE entries destructively. Measured per-target time on an 8 vCPU
# Blacksmith runner is a couple of minutes, off the gate's critical path; if a
# future group's wall time regresses past ~4 minutes, revisit 2-wide.
#
# Coverage boundary, carried over from the pre-fold job: every target here
# builds CGO_ENABLED=0, but .goreleaser.yml builds bd-linux-amd64,
# bd-linux-arm64 and bd-windows-amd64 with CGO_ENABLED=1, and the darwin pair
# is built natively at CGO_ENABLED=1 by release.yml's goreleaser-macos job.
# Five of these eight targets are therefore a weaker proxy than the binary
# they stand for: the cgo half of the tree (beads_cgo.go ->
# internal/storage/embeddeddolt) is never compiled here, so a break confined
# to it still passes this check. Only android/arm64, windows/arm64 and
# freebsd/amd64 replicate their release configuration exactly.
set -euo pipefail
group="${1:?usage: check-release-cross-compile.sh <group>}"
manifest="$(dirname "${BASH_SOURCE[0]}")/release-targets.txt"
targets=()
while read -r goos goarch row_group; do
[[ "$row_group" == "$group" ]] || continue
targets+=("$goos/$goarch")
done < <(grep -v '^#' "$manifest" | grep -v '^[[:space:]]*$')
if [[ "${#targets[@]}" -eq 0 ]]; then
echo "::error::no release targets found for group '$group' in $manifest"
exit 1
fi
failed=()
for target in "${targets[@]}"; do
goos="${target%/*}"
goarch="${target#*/}"
echo "::group::Build $goos/$goarch"
if CGO_ENABLED=0 GOOS="$goos" GOARCH="$goarch" go build -tags gms_pure_go ./...; then
echo "ok: $goos/$goarch"
else
echo "::error::cross-compilation failed for $goos/$goarch"
failed+=("$goos/$goarch")
fi
echo "::endgroup::"
done
if [[ "${#failed[@]}" -gt 0 ]]; then
echo "::error::cross-compilation failed for: ${failed[*]}"
exit 1
fi
echo "cross-compilation passed for group '$group': ${targets[*]}"

View file

@ -1,7 +1,6 @@
# Release target manifest for pr.yml's check-release-target-cross-compilation
# job (F7a). One "GOOS GOARCH GROUP" row per target; GROUP assigns the target
# to one of the job's two matrix legs (see
# scripts/ci/check-release-cross-compile.sh).
# Release target manifest: one "GOOS GOARCH" row per shipped release target.
# scripts/ci/bazel-release-cross-compile.sh (bazel.yml's pure-Go lane) builds
# the whole tree for each row with Bazel (--platforms, cgo off).
#
# Kept in sync with .goreleaser.yml's builds: list, PLUS the darwin/amd64 and
# darwin/arm64 pair that .goreleaser.yml deliberately omits (built instead by
@ -11,15 +10,11 @@
# TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser enforces this: it
# fails if this file and .goreleaser.yml (plus the darwin exception) ever
# disagree on the target set.
#
# Groups are an even split for balanced job runtime, not a semantic
# boundary - cross-compilation never touches the target OS, so both groups
# build and run entirely on the Linux runner.
linux amd64 unix
linux arm64 unix
android arm64 unix
freebsd amd64 unix
darwin amd64 desktop
darwin arm64 desktop
windows amd64 desktop
windows arm64 desktop
linux amd64
linux arm64
android arm64
freebsd amd64
darwin amd64
darwin arm64
windows amd64
windows arm64

View file

@ -35,8 +35,7 @@ const sameRepoBlacksmith2vcpu = "${{ (github.event_name == 'merge_group' || (git
// F7a: the same same-repo expression at 4 vCPU and 8 vCPU, for jobs sized
// larger than the 2 vCPU default (check-doc-flags,
// pr-risk.yml's test-nix at 4 vCPU; check-release-target-cross-compilation at
// 8 vCPU). F7c's advisory workflows also use the 4 vCPU size.
// pr-risk.yml's test-nix at 4 vCPU; windows-test-binaries at 8 vCPU). F7c's advisory workflows also use the 4 vCPU size.
const sameRepoBlacksmith4vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}"
const sameRepoBlacksmith8vcpu = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}"

View file

@ -393,42 +393,63 @@ esac
}
}
// TestPRCIGateRequiresReleaseTargetCrossCompilation pins the cross-compilation
// check into the gate. Wiring a job into ci-gate takes three separate edits --
// needs:, the CI_GATE_REQUIRED token list, and the CHECK_* env mapping -- and
// the gate silently ignores a token that is missing any one of them. Every
// other load-bearing check in this file is pinned by name for that reason.
func TestPRCIGateRequiresReleaseTargetCrossCompilation(t *testing.T) {
// TestReleaseCrossCompileRunsInBazelPureLane pins the release-target
// cross-compilation gate into bazel.yml's pure-Go lane, whose job.status
// ci-gate requires (BAZEL_PURE), and keeps the retired pr.yml job from
// coming back beside it: a second, `go build` copy of the gate would run
// every target twice.
func TestReleaseCrossCompileRunsInBazelPureLane(t *testing.T) {
const (
jobName = "check-release-target-cross-compilation"
token = "CHECK_RELEASE_TARGET_CROSS_COMPILATION"
retiredJob = "check-release-target-cross-compilation"
retiredToken = "CHECK_RELEASE_TARGET_CROSS_COMPILATION"
stepName = "Cross-compile every release target (--config=release-cross)"
)
gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate")
pr := readCIWorkflow(t, "pr.yml")
if _, ok := pr.Jobs[retiredJob]; ok {
t.Errorf("pr.yml still defines %s; bazel.yml's %s lane runs the release cross-compilation", retiredJob, bazelPureJobName)
}
gate := pr.job(t, "ci-gate")
gateEnv := gate.step(t, "Evaluate CI gate").Env
if contains(gate.Needs, retiredJob) || gateEnv[retiredToken] != "" || contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), retiredToken) {
t.Errorf("ci-gate still wires the retired %s job", retiredJob)
}
if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), "BAZEL_PURE") {
t.Errorf("ci-gate CI_GATE_REQUIRED does not include BAZEL_PURE, the lane that cross-compiles the release targets")
}
if !contains(gate.Needs, jobName) {
t.Errorf("ci-gate needs %q: %v", jobName, gate.Needs)
run := readCIWorkflow(t, bazelWorkflowName).job(t, bazelPureJobName).step(t, stepName).Run
for _, required := range []string{
"set -euo pipefail",
"./scripts/ci/bazel-release-cross-compile.sh",
} {
if !strings.Contains(run, required) {
t.Errorf("%s step %q does not contain %q:\n%s", bazelPureJobName, stepName, required, run)
}
}
if got, want := gateEnv[token], "${{ needs."+jobName+".result }}"; got != want {
t.Errorf("ci-gate env %s = %q, want %q", token, got, want)
}
if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), token) {
t.Errorf("ci-gate CI_GATE_REQUIRED does not include %q", token)
// The transition is what makes every platform's build pure Go (cgo off,
// as CGO_ENABLED=0 go build) for the platform the script names.
bzl := readPolicyFile(t, bazelPolicyRoot(t), "tools/bazel/release_cross.bzl")
for _, required := range []string{
`"//command_line_option:platforms": ["@rules_go//go/toolchain:" + platform],`,
`"@rules_go//go/config:pure": True,`,
`inputs = ["//tools/bazel:release_platforms"],`,
} {
if !strings.Contains(bzl, required) {
t.Errorf("tools/bazel/release_cross.bzl does not contain %q", required)
}
}
}
// TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser keeps
// scripts/ci/release-targets.txt (F7a: the single source of truth for the
// pr.yml cross-compilation job's two matrix legs, read by
// scripts/ci/check-release-cross-compile.sh) and the set of shipped release
// targets in lockstep. Without a guard a newly added release target -- the
// way freebsd/amd64 once was -- is silently uncovered while a green "release
// target cross-compilation" check still stands. That is worse than having no
// check at all, because the check's existence implies the coverage it has
// quietly lost.
// scripts/ci/release-targets.txt (the single source of truth for the
// platforms scripts/ci/bazel-release-cross-compile.sh builds in bazel.yml's
// pure-Go lane) and the set of shipped release targets in lockstep. Without
// a guard a newly added release target -- the way freebsd/amd64 once was --
// is silently uncovered while a green cross-compilation check still stands.
// That is worse than having no check at all, because the check's existence
// implies the coverage it has quietly lost.
func TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser(t *testing.T) {
const jobName = "check-release-target-cross-compilation"
const manifestPath = "scripts/ci/release-targets.txt"
// darwin/amd64 and darwin/arm64 are shipped release targets that are
@ -447,51 +468,21 @@ func TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser(t *testing.T) {
}
}
// The job itself must still read the manifest, so a future rewrite of the
// job that stops threading matrix.group through to the script cannot pass
// silently.
job := readCIWorkflow(t, "pr.yml").job(t, jobName)
groups := append([]string(nil), job.Strategy.Matrix.Group...)
sort.Strings(groups)
if !reflect.DeepEqual(groups, []string{"desktop", "unix"}) {
t.Errorf("%s matrix groups = %v, want [desktop unix]", jobName, groups)
}
var ranScript bool
for _, step := range job.Steps {
if strings.Contains(step.Run, "check-release-cross-compile.sh ${{ matrix.group }}") {
ranScript = true
}
}
if !ranScript {
t.Errorf("%s does not run check-release-cross-compile.sh with matrix.group", jobName)
}
raw, err := os.ReadFile(filepath.Join(sourceRepoRoot(t), manifestPath))
if err != nil {
t.Fatalf("read %s: %v", manifestPath, err)
}
got := make(map[string]bool)
seenGroups := make(map[string]bool)
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
fields := strings.Fields(line)
if len(fields) != 3 {
t.Fatalf("%s: malformed row %q, want 'GOOS GOARCH GROUP'", manifestPath, line)
}
goos, goarch, group := fields[0], fields[1], fields[2]
if group != "unix" && group != "desktop" {
t.Fatalf("%s: row %q has unknown group %q", manifestPath, line, group)
}
seenGroups[group] = true
got[goos+"/"+goarch] = true
}
for _, g := range []string{"unix", "desktop"} {
if !seenGroups[g] {
t.Errorf("%s: no targets in group %q", manifestPath, g)
if len(fields) != 2 {
t.Fatalf("%s: malformed row %q, want 'GOOS GOARCH'", manifestPath, line)
}
got[fields[0]+"/"+fields[1]] = true
}
for target, source := range want {
@ -1421,11 +1412,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) {
assertGoCacheInventory(t, workflows["pr.yml"].job(t, "test-windows-liveness"), []goCacheStep{
restoreModuleCache(), restoreBuildCache("non-race"),
})
for _, jobName := range []string{
"advisory-reports", "check-release-target-cross-compilation",
} {
assertGoCacheInventory(t, workflows["pr.yml"].job(t, jobName), []goCacheStep{restoreModuleCache()})
}
assertGoCacheInventory(t, workflows["pr.yml"].job(t, "advisory-reports"), []goCacheStep{restoreModuleCache()})
// F7b review (S2): check-doc-freshness-platforms additionally restores a
// restore-only non-race Go build cache, seeded by the same push-to-main
// savers pr-preflight-platforms' leg restores below (see that job's own
@ -1462,7 +1449,7 @@ func TestGoCacheOwnershipTopology(t *testing.T) {
"pr.yml": {
"build-artifacts": true, "pr-core-wrapper": true, "scripts-go-checks": true, "worktree-remove-windows": true,
"check-doc-freshness-platforms": true, "pr-preflight-platforms": true, "advisory-reports": true,
"check-release-target-cross-compilation": true, "pr-lint-wrapper": true, "windows-test-binaries": true,
"pr-lint-wrapper": true, "windows-test-binaries": true,
"test-windows-liveness": true,
},
"pr-risk.yml": {"build-embedded": true},
@ -2988,15 +2975,13 @@ func TestSameRepoBlacksmithRunners(t *testing.T) {
// size (module-cache misses on the Blacksmith pool are acceptable
// per spec; jobs that need a Blacksmith-side build-cache saver are
// F7b's scope, not this one). windows-test-binaries is F4's Linux
// mingw cross-compile job, sized at 8vcpu like the cross-compilation
// matrix fold below.
// mingw cross-compile job, sized at 8vcpu.
"pr.yml": {
"bazel-coverage": sameRepoBlacksmith2vcpu, "ci-gate": sameRepoBlacksmith2vcpu,
"fast-checks": sameRepoBlacksmith2vcpu,
"advisory-reports": sameRepoBlacksmith2vcpu,
"check-doc-flags": sameRepoBlacksmith4vcpu,
"check-release-target-cross-compilation": sameRepoBlacksmith8vcpu,
"windows-test-binaries": sameRepoBlacksmith8vcpu,
"fast-checks": sameRepoBlacksmith2vcpu,
"advisory-reports": sameRepoBlacksmith2vcpu,
"check-doc-flags": sameRepoBlacksmith4vcpu,
"windows-test-binaries": sameRepoBlacksmith8vcpu,
// F7b: scripts-go-checks/pr-lint-wrapper moved to same-repo
// Blacksmith now that main.yml's blacksmith-go-build-cache and
// pr-lint-wrapper `venue` matrix seed their Blacksmith-selection

View file

@ -8,7 +8,12 @@ go_library(
"resolver_windows.go",
],
importpath = "github.com/steveyegge/beads/scripts/internal/testbash",
visibility = ["//scripts:__subpackages__"],
# //tools/bazel:release_cross cross-compiles it like every package
# `go build ./...` compiles.
visibility = [
"//scripts:__subpackages__",
"//tools/bazel:__pkg__",
],
)
go_test(

View file

@ -1,6 +1,7 @@
load("@rules_shell//shell:sh_binary.bzl", "sh_binary")
load("@rules_shell//shell:sh_test.bzl", "sh_test")
load(":dolt.bzl", "hermetic_bin")
load(":release_cross.bzl", "release_cross_build", "string_list_flag")
# Platforms with a pinned dolt release; elsewhere (e.g. windows) the hermetic
# test wrapper is skipped by `bazel build //...` instead of failing analysis.
@ -99,6 +100,207 @@ filegroup(
for cpu in ("amd64", "arm64")
]
# On when rules_go builds pure Go (--@rules_go//go/config:pure, as
# --config=pure and :release_cross's transition set it): cgo is off.
config_setting(
name = "pure",
flag_values = {"@rules_go//go/config:pure": "true"},
visibility = ["//:__subpackages__"],
)
# Bazel compiles gozstd's pure stubs (third_party/patches/gozstd_nocgo.patch)
# where `go build` with CGO_ENABLED=0 rejects any import of it, so a pure bd
# that links gozstd still builds. Go binaries keep their function names
# (pclntab) even when stripped, so this fails the build of a pure bd that
# links gozstd for whatever platform it targets, without running it:
# :release_cross builds it for every release platform. "main.main" proves the name table
# was readable. Pure builds only: a cgo bd links gozstd by design.
genrule(
name = "pure_bd_has_no_cgo_only_deps",
srcs = ["//cmd/bd"],
outs = ["pure_bd_has_no_cgo_only_deps.txt"],
cmd = """
if grep -qaF 'github.com/dolthub/gozstd.' $<; then
echo "$< links github.com/dolthub/gozstd (cgo-only) in a pure build; CGO_ENABLED=0 go build would reject it" >&2
exit 1
fi
grep -qaF 'main.main' $< || { echo "$<: no Go function names found" >&2; exit 1; }
echo "no cgo-only deps: $<" > $@
""",
target_compatible_with = select({
":pure": [],
"//conditions:default": ["@platforms//:incompatible"],
}),
)
# The rules_go platforms (<os>_<arch>) :release_cross builds for.
# scripts/ci/bazel-release-cross-compile.sh sets it from
# scripts/ci/release-targets.txt, the single source of truth; empty fails
# :release_cross's analysis.
string_list_flag(
name = "release_platforms",
build_setting_default = [],
)
# :release_cross (the managed block below) is the release-target
# cross-compilation: every go_library and go_binary, cgo off, once per
# platform in :release_platforms, all in one invocation. It is the Bazel
# form of `CGO_ENABLED=0 GOOS=<os> GOARCH=<arch> go build -tags gms_pure_go
# ./...` per release target, plus :pure_bd_has_no_cgo_only_deps. Manual: it
# needs :release_platforms, which `bazel build //...` does not set.
# --- begin release_cross (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---
# manual: needs --//tools/bazel:release_platforms (see above).
release_cross_build(
name = "release_cross",
tags = ["manual"],
targets = [
":pure_bd_has_no_cgo_only_deps",
"//:beads",
"//backend/conformance:conformance",
"//backend:backend",
"//beadserrors:beadserrors",
"//cmd/bd/doctor/fix:fix",
"//cmd/bd/doctor:doctor",
"//cmd/bd/protocol:protocol",
"//cmd/bd/setup:setup",
"//cmd/bd:bd",
"//format:format",
"//internal/ado:ado",
"//internal/atomicfile:atomicfile",
"//internal/audit:audit",
"//internal/beads:beads",
"//internal/ceiling:ceiling",
"//internal/compact:compact",
"//internal/config:config",
"//internal/configfile:configfile",
"//internal/creds:creds",
"//internal/debug:debug",
"//internal/doltremote:doltremote",
"//internal/doltserver:doltserver",
"//internal/doltversion:doltversion",
"//internal/eventsjournal:eventsjournal",
"//internal/execenv:execenv",
"//internal/fdhygiene:fdhygiene",
"//internal/formula/cmd/schemagen:schemagen",
"//internal/formula/schemagen:schemagen",
"//internal/formula:formula",
"//internal/git:git",
"//internal/gitenv:gitenv",
"//internal/githooksenv:githooksenv",
"//internal/github:github",
"//internal/gitignore:gitignore",
"//internal/gitlab:gitlab",
"//internal/gittraceenv:gittraceenv",
"//internal/hooks:hooks",
"//internal/httpapi/apigen:apigen",
"//internal/httpapi/spec:spec",
"//internal/httpapi/wireshape/cmd/gendigest:gendigest",
"//internal/httpapi/wireshape:wireshape",
"//internal/httpapi:httpapi",
"//internal/httpclient/encode:encode",
"//internal/httpclient/wire:wire",
"//internal/httpclient:httpclient",
"//internal/idgen:idgen",
"//internal/jira:jira",
"//internal/linear:linear",
"//internal/lockfile:lockfile",
"//internal/memoryapi:memoryapi",
"//internal/metrics:metrics",
"//internal/migration/legacysqlite:legacysqlite",
"//internal/migration:migration",
"//internal/molecules:molecules",
"//internal/notion:notion",
"//internal/procid:procid",
"//internal/query:query",
"//internal/recipes:recipes",
"//internal/remotecache:remotecache",
"//internal/routing:routing",
"//internal/storage/backendnames:backendnames",
"//internal/storage/backends:backends",
"//internal/storage/batchbench:batchbench",
"//internal/storage/batchfixtures:batchfixtures",
"//internal/storage/contextinfo:contextinfo",
"//internal/storage/createbatchequiv:createbatchequiv",
"//internal/storage/dberrors:dberrors",
"//internal/storage/dbproxy/identity:identity",
"//internal/storage/dbproxy/pidfile:pidfile",
"//internal/storage/dbproxy/proxy:proxy",
"//internal/storage/dbproxy/server:server",
"//internal/storage/dbproxy/util:util",
"//internal/storage/depid:depid",
"//internal/storage/dolt:dolt",
"//internal/storage/doltutil:doltutil",
"//internal/storage/domain/db:db",
"//internal/storage/domain/fs:fs",
"//internal/storage/domain/git:git",
"//internal/storage/domain:domain",
"//internal/storage/embeddeddolt:embeddeddolt",
"//internal/storage/externaldeps:externaldeps",
"//internal/storage/fs:fs",
"//internal/storage/git:git",
"//internal/storage/issueops:issueops",
"//internal/storage/journalscan:journalscan",
"//internal/storage/kvkeys:kvkeys",
"//internal/storage/memoryops:memoryops",
"//internal/storage/rowid:rowid",
"//internal/storage/schema:schema",
"//internal/storage/sqlbuild:sqlbuild",
"//internal/storage/sqlclass:sqlclass",
"//internal/storage/sqlcount:sqlcount",
"//internal/storage/storagecontract:storagecontract",
"//internal/storage/uow:uow",
"//internal/storage/versioncontrolops:versioncontrolops",
"//internal/storage:storage",
"//internal/telemetry:telemetry",
"//internal/templates/agents:agents",
"//internal/templates:templates",
"//internal/testutil/bazeltest:bazeltest",
"//internal/testutil/credentialcmd:credentialcmd",
"//internal/testutil/fixtures:fixtures",
"//internal/testutil/integration:integration",
"//internal/testutil:testutil",
"//internal/timeparsing:timeparsing",
"//internal/tracker/conformance:conformance",
"//internal/tracker:tracker",
"//internal/types:types",
"//internal/ui:ui",
"//internal/uimd:uimd",
"//internal/utils:utils",
"//internal/validation:validation",
"//internal/versioncheck:versioncheck",
"//internal/workapi/storecounter:storecounter",
"//internal/workapi/storequerier:storequerier",
"//internal/workapi/storereader:storereader",
"//internal/workapi/storereadycounter:storereadycounter",
"//internal/workapi/storestats:storestats",
"//internal/workapi/storeversionreconciler:storeversionreconciler",
"//internal/workapi/storeworkspaceconfig:storeworkspaceconfig",
"//internal/workapi:workapi",
"//internal/workspacegate:workspacegate",
"//internal/worktreeremove:worktreeremove",
"//issueops:issueops",
"//journalops:journalops",
"//memoryops:memoryops",
"//plugins/beads:beads",
"//schema:schema",
"//scripts/bench-ready-indexes:bench-ready-indexes",
"//scripts/check-versions:check-versions",
"//scripts/checkworkflowtags:checkworkflowtags",
"//scripts/internal/testbash:testbash",
"//scripts/migration-test/catalog:catalog",
"//scripts/pr-lint:pr-lint",
"//scripts/repro-dolt-hang:repro-dolt-hang",
"//scripts/repro-dolt-prod-timeouts:repro-dolt-prod-timeouts",
"//scripts/workflowtags:workflowtags",
"//test/conformance:conformance",
"//tools/docsmint:docsmint",
],
)
# --- end release_cross ---
# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---
filegroup(

View file

@ -32,6 +32,17 @@ sync step, on every PR) fails on a missing or stale block. Trees in
.bazelignore (.beads, website, the nested example modules, agent worktrees,
node_modules) are outside Bazel and so outside //:repo_files.
tools/bazel/BUILD.bazel also gets a managed `release_cross` block: the
release_cross_build (tools/bazel/release_cross.bzl) that
scripts/ci/bazel-release-cross-compile.sh builds for every release platform,
listing every go_library and go_binary it can see (the packages `go build
./...` compiles; a private library is compiled by the go_binary that embeds
it) except the cgo-only ones a pure build cannot link (tagged "cgo-only" and
incompatible with //tools/bazel:pure) and testonly fixtures. The script checks
against `bazel query` before it builds that every Bazel package holding Go
targets is reached, so a package this parser misses fails CI instead of
going uncompiled.
With --check nothing is written: stale blocks are printed as a diff and the
exit status is 1 (see `make bazel-sync-check`).
"""
@ -65,11 +76,24 @@ TREES = (
BEGIN = "# --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---"
END = "# --- end go_srcs ---"
RELEASE_BEGIN = "# --- begin release_cross (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---"
RELEASE_END = "# --- end release_cross ---"
RELEASE_PKG = "tools/bazel"
# Built in every release configuration besides the Go targets.
RELEASE_EXTRA_TARGETS = (":pure_bd_has_no_cgo_only_deps",)
# Visibilities that let //tools/bazel:release_cross depend on a target.
RELEASE_VISIBLE = ('"//visibility:public"', '"//:__subpackages__"', '"//tools/bazel:__pkg__"')
# A buildifier-formatted go_library/go_binary call and its name.
GO_RULE_RE = re.compile(r'^(go_library|go_binary)\(\n name = "([^"]+)",\n(.*?)^\)', re.DOTALL | re.MULTILINE)
# How a cgo-only target opts out of pure builds (see
# internal/storage/embeddeddolt/cmd/BUILD.bazel and the "cgo-only" tag in
# scripts/bazel_policy_test.go).
CGO_ONLY_TAG = '"cgo-only"'
REPO_BEGIN = "# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---"
REPO_END = "# --- end repo_files ---"
BLOCK_RES = tuple(
re.compile(r"\n*" + re.escape(begin) + r".*?" + re.escape(end) + r"\n*", re.DOTALL)
for begin, end in ((BEGIN, END), (REPO_BEGIN, REPO_END))
for begin, end in ((BEGIN, END), (RELEASE_BEGIN, RELEASE_END), (REPO_BEGIN, REPO_END))
)
SKIP_DIRS = {"testdata", "node_modules"}
@ -206,6 +230,43 @@ def repo_files_block(pkg: str, packages: list[str]) -> str:
return "\n".join(lines) + "\n"
def go_targets(packages: list[str]) -> list[str]:
"""Every go_library and go_binary that a pure build can compile."""
labels = []
for pkg in packages:
with open(os.path.join(pkg, "BUILD.bazel")) as f:
src = f.read()
for kind, name, body in GO_RULE_RE.findall(src):
# cgo-only: `go build ./...` skips it with CGO_ENABLED=0.
# testonly: a fixture under testdata/, which `./...` excludes.
# Not visible here: a main package's private embedded library
# or a package-restricted helper; its package is covered by a
# visible target (or the script's package check fails).
if CGO_ONLY_TAG in body or "testonly = True" in body:
continue
vis = re.search(r"^ visibility = \[(.*?)\]", body, re.DOTALL | re.MULTILINE)
if not vis or not any(v in vis.group(1) for v in RELEASE_VISIBLE):
continue
path = "" if pkg == "." else pkg
labels.append(f"//{path}:{name}")
return sorted(labels)
def release_cross_block(targets: list[str]) -> str:
lines = [
RELEASE_BEGIN,
"",
"# manual: needs --//tools/bazel:release_platforms (see above).",
"release_cross_build(",
' name = "release_cross",',
' tags = ["manual"],',
" targets = [",
]
lines += [f' "{t}",' for t in sorted(RELEASE_EXTRA_TARGETS) + targets]
lines += [" ],", ")", "", RELEASE_END]
return "\n".join(lines) + "\n"
def rewrite(path: str, new_blocks: list[str], check: bool) -> bool:
"""Bring path's managed blocks up to date; return True if any was stale.
@ -266,6 +327,8 @@ def main(argv: list[str]) -> int:
for pkg in packages:
path = os.path.join(pkg, "BUILD.bazel")
blocks = [block(pkg, wanted[pkg])] if pkg in wanted else []
if pkg == RELEASE_PKG:
blocks.append(release_cross_block(go_targets(packages)))
blocks.append(repo_files_block(pkg, packages))
changed = rewrite(path, blocks, check)
if changed:

View file

@ -0,0 +1,55 @@
"""Build Go targets for every release platform in one Bazel invocation.
scripts/ci/bazel-release-cross-compile.sh passes the platforms of
scripts/ci/release-targets.txt as --//tools/bazel:release_platforms; the
split transition below then configures every target once per platform, with
pure Go (cgo off), so all platforms build in parallel on the remote workers
instead of one `bazel build --platforms=...` after another.
"""
def _release_platforms_transition_impl(settings, _attr):
platforms = settings["//tools/bazel:release_platforms"]
if not platforms:
fail("--//tools/bazel:release_platforms is empty; scripts/ci/bazel-release-cross-compile.sh sets it from scripts/ci/release-targets.txt")
return {
platform: {
"//command_line_option:platforms": ["@rules_go//go/toolchain:" + platform],
"@rules_go//go/config:pure": True,
}
for platform in platforms
}
_release_platforms_transition = transition(
implementation = _release_platforms_transition_impl,
inputs = ["//tools/bazel:release_platforms"],
outputs = [
"//command_line_option:platforms",
"@rules_go//go/config:pure",
],
)
def _release_cross_build_impl(ctx):
return [DefaultInfo(files = depset(transitive = [
dep[DefaultInfo].files
for deps in ctx.split_attr.targets.values()
for dep in deps
]))]
release_cross_build = rule(
implementation = _release_cross_build_impl,
doc = "Builds `targets` for every rules_go platform (<os>_<arch>) in --//tools/bazel:release_platforms, with cgo off.",
attrs = {
"targets": attr.label_list(
cfg = _release_platforms_transition,
mandatory = True,
),
},
)
def _string_list_flag_impl(_ctx):
return []
string_list_flag = rule(
implementation = _string_list_flag_impl,
build_setting = config.string_list(flag = True),
)