Commit graph

6 commits

Author SHA1 Message Date
Julian Knutsen
5e73670866
ci(bazel): cross-compile every release target with Bazel, retire the go build job (#7332)
pr.yml's check-release-target-cross-compilation ran `CGO_ENABLED=0
GOOS/GOARCH go build -tags gms_pure_go ./...` for the eight release targets
on two GitHub/Blacksmith runners. It is now a step of bazel.yml's pure-Go
lane (BAZEL_PURE, already required by ci-gate):
scripts/ci/bazel-release-cross-compile.sh builds //tools/bazel:release_cross
once, remotely, with --//tools/bazel:release_platforms set from
scripts/ci/release-targets.txt (still the single source of truth, pinned to
.goreleaser.yml). release_cross's split transition configures every
go_library/go_binary for each rules_go platform with cgo off, so all eight
platforms build in parallel in one invocation.

- tools/bazel/release_cross.bzl: the split-transition rule and the
  string-list flag.
- tools/bazel/go_srcs.py (make bazel-sync, checked by bazel-sync-check and
  TestBazelGoSrcsBlocksCurrent) keeps the release_cross target list; the
  script checks with bazel query that every Bazel package holding Go
  targets is reached before it builds.
- //tools/bazel:pure_bd_has_no_cgo_only_deps: Bazel compiles gozstd's pure
  stubs where `go build` rejects them, so the pure bd of every platform
  must not link gozstd.
- //internal/storage/embeddeddolt/cmd:cmd (cgo-only main, skipped by
  `go build ./...` under CGO_ENABLED=0) is tagged cgo-only and incompatible
  with pure builds.
- A failed build is re-run per platform (cache hits) to name each failing
  GOOS/GOARCH.

Same cgo-off coverage boundary as the retired job. Refs ga-96smfk.20.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 03:18:37 +00:00
Julian Knutsen
b1fe68e63a
ci(bazel): run pr.yml's repository guards as Bazel tests (#7329)
fast-checks, PR Policy, and the Linux leg of check-doc-freshness-platforms
ran repository guards on GitHub runners. They now run as Bazel tests in
bazel.yml's `test` lane (remote, cached), each over only the files it reads:

- tools/bazel/repo_subset.bzl picks a narrow, path-selected slice of
  //:repo_files at analysis time. scripts/repochecks/run_check.sh copies that
  slice into a throwaway git checkout and runs the unchanged guard there.
- //scripts/repochecks: build_tags, workflow_tags, go_install_guidance,
  winget_portable_alias, migration_hygiene (checks A and B),
  fmt (registered SDK gofmt), doc_freshness, workapi_frontend_boundary,
  types_gen_drift (`make api-check` drift, oapi-codegen from go.mod).
- //scripts:doc_freshness_required_test (integration lane, -required-suite)
  and //scripts/gitattributespolicy:gitattributespolicy_required_host_test
  cover the Linux doc-freshness leg.
- check-versions stays covered by //internal/versioncheck:versioncheck_test.
  check-testing-short stays covered by //scripts:scripts_test.

pr-policy-wrapper is deleted. fast-checks keeps only the two checks that
need the PR base: migration hygiene's delta half (MIGRATION_HYGIENE_SCOPE=
delta) and the .beads diff. check-doc-flags stays, because it builds bd from
the pinned release tag. The example Go modules leave .bazelignore so gofmt
and the grep guards see them. check_testcases.py now accepts --not-go
//pkg:all.

Refs: ga-96smfk.20

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 00:59:18 +00:00
Julian Knutsen
e72cd8b315
ci: merge-queue readiness for the required gates; cache test results (fresh nightly) (#7264)
* ci: make the required gates merge-queue ready

The default-branch ruleset requires "CI Gate / Required" and "PR Risk
Gate / Required". Prepare both for a GitHub merge queue (ALLGREEN,
squash), so enabling the ruleset's merge_queue rule is the only step left.

- pr.yml, pr-risk.yml: merge_group pinned to types [checks_requested].
- D2: bazel-coverage covers every merge_group run (same repository, CI
  secrets, actor github-merge-queue[bot], bazel.yml mode remote), so the
  retired legacy tiers stay retired in the queue and ci-gate requires
  their Bazel lanes to have run remotely and passed; a missing secret or
  the farm switch off is red, as on a same-repo PR.
- Diff bases on merge_group: the package gates diff
  merge_group.base_sha..head_sha (not HEAD^, which misses all but the
  last commit of a rebase-method entry), the .beads/issues.jsonl guard
  now runs on merge_group against base_sha (ci-gate no longer excuses
  its skip), and check-doc-flags passes BD_DOCS_DIFF_BASE so CLI-docs
  drift already on main cannot eject every queued PR.
- Test result reuse: on a merge_group run and on attempt 2+ of a PR run,
  the six always-execute Bazel lanes append --config=reuse-test-results
  (--cache_test_results=auto), so tests whose action key is unchanged
  reuse the farm's recorded PASS and only what the merge changed runs.
  Failures are never reused, eviction retries stay 0, and each PR's first
  attempt, push to main and nightly still execute every test.
- scripts/ci_merge_queue_test.go simulates merge_group through the shared
  evalGHExpr evaluator (now with true/false/null literals, hyphenated
  identifiers and case-insensitive ==) and the workflows' own scripts:
  triggers and gate names, concurrency groups, every pull_request field
  read, the coverage and rbe decisions, result reuse, the package-gate,
  .beads and doc-drift diff bases.
- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md: new "Merge Queue" section (how
  to queue, settings, what runs, result reuse rationale, ejection and
  flakes, rollback).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci: cache Bazel test results except nightly; queue trust and sizing

Review follow-ups to the merge-queue readiness change.

- Test results (S1): --nocache_test_results also stops Bazel writing
  results, so a PR's first run never fed the cache and the merge-queue
  reuse config could only hit on later reuse runs. Drop it from
  test:embedded, test:doltserver-proxied, test:doltserver-integration and
  test:sole-run (which keeps --experimental_remote_cache_eviction_retries=0),
  and drop test:reuse-test-results and BAZEL_REUSE_RESULTS. Results are now
  cached (read and write) on PRs, merge groups and pushes to main.
  nightly.yml passes the new fresh-test-results input, so every `bazel
  test` appends --config=fresh (--nocache_test_results) and the suite
  re-executes nightly. The docker lane keeps --nocache_test_results.
  Measured locally with a disk cache: a default run writes and the next
  reads ("(cached) PASSED"); a fresh run neither reads nor writes.
- Security headers (S2): bazel.yml now states that a merge_group run
  executes the queued commit's workflows, so a queued fork or Dependabot
  PR's CI edits run with every CI secret, the pre-warm app key included
  (merging-level trust), and records the maintainer rule: review fork
  changes to .github/, .bazelrc, tools/bazel, MODULE.bazel, BUILD files
  and CI scripts before queueing. Same note in the topology doc.
- Queue sizing (S3): recommend max_entries_to_build 3 and
  check_response_timeout_minutes 60 (max_entries_to_merge 5, ALLGREEN,
  SQUASH unchanged).
- Nits: the .beads guard fails closed on a missing diff base, and its
  test runs under GitHub's default `bash -e`; evalGHExpr follows GitHub's
  loose equality (true == 'true' is false, false == '' is true, missing
  context keys are null) with a typed-context variant; the
  detect-package-gates comment notes the SQUASH/HEAD^ equivalence and
  path-filter scope; the doc covers re-run reuse, flake re-sampling via
  dispatch, stale queue runs and how to confirm reuse on the farm.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 13:54:15 -07:00
Julian Knutsen
4493eb8d23
ci(pr): run same-repo macOS preflight/doc-freshness legs on Blacksmith macOS (#7231)
pr-preflight-platforms and check-doc-freshness-platforms now resolve their
same-repo-macos marker like the Linux/Windows legs: blacksmith-6vcpu-macos-26
for same-repo PRs and merge_group, GitHub-hosted macos-latest for forks,
Dependabot and every other event. The label is pinned to macos-26 because
GitHub's macos-latest is macos-26-arm64 today, so both paths run the same
OS/arch; 6 vCPU is already 2x macos-latest's 3 vCPU.

Blacksmith cannot see GitHub-saved caches, so main.yml gains
blacksmith-macos-go-build-cache: a push-to-main-only saver on the same label
(same job guard as the other Blacksmith savers) that builds ./cmd/bd and runs
the shared warm-non-race-cache.sh, saving the module cache and a non-race
GOCACHE keyed by go.sum + UTC day. The PR legs stay restore-only. main.yml's
GitHub-hosted macOS full-suite leg is unchanged and keeps seeding the fork
path.

Policy tests: the macOS marker arm is evaluated for same-repo/merge_group/
fork/deleted-head/dependabot/push/schedule and must match the flat macOS
const; the saver joins the PR-guard and per-day-key sweeps and the cache
ownership inventory; TestBlacksmithMacOSSaverMatchesPRLegs pins the
label/path/key/warm-up contract between saver and PR legs.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 13:19:49 -07:00
Julian Knutsen
84ce244857
ci: same-repo Linux/Windows PR jobs on Blacksmith with main-seeded warm build caches (F7b) (#7155)
* ci: move same-repo PR Linux/Windows/macOS legs to Blacksmith, shard preflight (F7b)

PR-path jobs for same-repo PRs/merge_group now run on Blacksmith instead
of GitHub-hosted runners, following F4/F7a's ternary-expression and
restore-only-cache patterns; forks/Dependabot are unaffected:

- scripts-go-checks, pr-lint-wrapper: same-repo Blacksmith 4vcpu Linux.
- check-doc-freshness-platforms, pr-preflight-platforms: restructured
  from a top-level matrix.os list to an include:-only "matrix marker"
  shape (runner: same-repo-linux/-windows/-macos) so each host can
  resolve an independent same-repo Blacksmith selection via one chained
  ternary runs-on expression.
- Six standalone Windows jobs (test-windows-liveness,
  worktree-remove-windows, test-windows-small,
  test-windows-liveness-prebuilt, worktree-remove-windows-prebuilt,
  windows-make-shell) move to same-repo Blacksmith Windows
  (blacksmith-{2,4}vcpu-windows-2025, public beta). CGO/mingw-w64 and
  Chocolatey preinstall are unconfirmed on that image; both risks are
  flagged with one-line rollback instructions at the call sites.
- pr-preflight-platforms' Windows leg is split into two shards (common
  steps on shard 1, the three Windows-only heavy steps on shard 2) to
  shorten its critical path; shard 2 downloads Go modules directly
  instead of sharing shard 1's runner/cache.

main.yml gains cache savers for the newly same-repo-Blacksmith PR
consumers, since Blacksmith and GitHub-hosted runners cannot see each
other's actions/cache entries even for identical keys:

- blacksmith-go-build-cache warms the Blacksmith-selection race/non-race
  GOCACHEs scripts-go-checks and pr-preflight-platforms/check-doc-
  freshness-platforms' ubuntu legs need, push-to-main only. Its non-race
  leg shares scripts/ci/warm-non-race-cache.sh with build-artifacts, and
  compiles tools/bazel/run_allowlisted_go_tests.py's allowlisted
  packages via its new --compile-only flag, so the two paths and the
  warm-up can never drift from what they actually run.
- pr-lint-wrapper, go-vet-cache, test-windows gain a venue: [blacksmith,
  github] matrix leg to seed both runner selections pr.yml's
  same-repo-Blacksmith-or-fork-hosted consumers can land on.

Updates the policy-test suite (ci_workflow_test.go,
ci_blacksmith_runner_test.go, pr_lanes_bazel_coverage_test.go) to cover
the new runner selections, matrix-marker shape, shard gating, and cache
topology, including a dedicated inventory/writer check for
blacksmith-go-build-cache.

Deferred follow-up (not done here): sharding check-doc-freshness-
platforms and the standalone Windows jobs, and moving/sharding macOS
legs beyond what check-doc-freshness-platforms'/pr-preflight-platforms'
matrix-marker form already covers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): F7b review response -- msys2/Blacksmith-saver guards, macOS stays hosted (B1/B2/S3)

Addresses ~/beads-bazel-plan/f7b/review.md's two cache-safety blockers and
one follow-up gap found after F7b moved same-repo PR Linux/Windows/macOS
legs onto Blacksmith and added push-to-main Blacksmith cache seeders:

- B1: windows-make-shell's msys2/setup-msys2 step caches its installed
  packages by default, which implicitly SAVED a GitHub Actions cache entry
  from a same-repo PR the moment it ran on a Blacksmith runner (confirmed
  via a real cache-list entry from a refs/pull/.../merge ref). Disabled with
  a fail-closed `cache: ${{ runner.environment == 'github-hosted' }}`, and
  generalized the sweep (TestBlacksmithReachablePRJobsDisableDefaultCachingActions)
  to cover any default-caching setup-* action on a Blacksmith-reachable job,
  not just this one.

- B2: blacksmith-go-build-cache, pr-lint-wrapper, go-vet-cache and
  test-windows (main.yml) had no (or an incomplete) same-repo push-to-main
  guard, so a same-repo fork with push access to its own main could run --
  and save a cache entry from -- a paid Blacksmith job. Added the exact
  push-to-main/same-repo `if:` F7c's own blacksmith-setup-go-cache already
  uses, pinned by TestBlacksmithSaverJobsGuardedAgainstPullRequest.

- S3: the macOS leg of pr-preflight-platforms/check-doc-freshness-platforms
  resolved to Blacksmith for same-repo PRs; Blacksmith bills macOS at
  roughly 20x its Linux/Windows rate and no push-to-main Blacksmith-macOS
  saver exists, so that leg would pay the premium for a guaranteed cold
  cache. macOS now stays unconditionally on GitHub-hosted macos-latest.

Also pins S7 (blacksmith-go-build-cache/go-vet-cache/test-windows's cache
keys are bounded by go.sum+UTC-day, not github.sha) and S6/S8 (their
20-minute timeouts), both already in place but previously unpinned, and
replaces run_allowlisted_go_tests.py's fragile GO_TEST_FLAGS[-2:] slice with
an explicit BUILD_FLAGS constant (N1).

Re-ran the reviewer's 15 mutations (~/beads-bazel-plan/f7b/review/mutate.py):
all now killed or structurally moot (the Windows preflight shard split this
review asked to revert-unless-helpful was reverted, in an earlier commit on
this branch, removing the vulnerability surface 6 of them targeted).

scripts/BUILD.bazel: add the new ci_f7b_review_test.go to scripts_test's
srcs -- bazel test caught it missing (file existed on disk but was not
compiled into the Bazel-built test binary).

Verification: go vet/test/build, gofmt, bazel --nohome_rc test
//scripts:scripts_test, actionlint -shellcheck, shellcheck, make
ci-pr-policy, make bazel-sync-check all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci): drop local review-doc path from comment

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 17:58:31 -07:00
Julian Knutsen
fac9a1e32c
ci: same-repo Linux PR jobs on Blacksmith; fold tiny checks and cross-compile legs (F7a) (#7139)
* ci: F7a same-repo Blacksmith moves and job folds (pr.yml, pr-risk.yml)

Move cache-free same-repo pr.yml/pr-risk.yml jobs onto the org's Blacksmith
runner pool (forks/Dependabot stay ubuntu-latest), and fold several
independent single-purpose jobs into fewer multi-step jobs, to cut same-repo
PR queue time without changing either required gate's name or its token
contract:

- fast-checks: folds check-build-tags, check-version-consistency,
  check-migration-hygiene, check-no-beads-changes and fmt-check into one
  Blacksmith job with five independent steps; ci-gate reads each step's
  outcome via job outputs into the same CI_GATE_REQUIRED token the standalone
  job used, plus a FAST_CHECKS backstop token for a lost job.
- check-release-target-cross-compilation: 8 per-target matrix legs collapse
  to 2 (unix/desktop), driven by the new scripts/ci/release-targets.txt
  manifest (kept in sync with .goreleaser.yml) and
  scripts/ci/check-release-cross-compile.sh.
- advisory-reports: folds build-examples and complexity-report (both
  already advisory) into one job, keeping their pre-fold
  step-level continue-on-error/timeout design.
- windows-make-shell: 3-way host matrix folds into one job with an
  aggregator step reproducing the old native/MSYS2/Cygwin pass/fail rules;
  test-windows-doltversion and test-windows-dbproxy-server fold into
  test-windows-small (neither was gated).
- Runner moves: fast-checks, advisory-reports,
  check-release-target-cross-compilation, check-doc-flags,
  pr-policy-wrapper, test-dolt-server-fingerprint (pr.yml) and test-nix
  (pr-risk.yml) move to the same-repo Blacksmith expression at their
  measured vCPU size.

Policy tests: extend TestSameRepoBlacksmithRunners for the moved/added jobs,
add TestSameRepoBlacksmithExpressionSemantics, TestBlacksmithJobsReadNoSecrets
and TestPRCIGateFastChecksTokens, retarget the complexity/build-examples and
dolt-server-fingerprint tests, rewrite
TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser against the new
manifest, and teach the ci-gate simulation helper to resolve
needs.fast-checks.outputs.*. Document the moves/folds and the Blacksmith-off
revert set in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): windows-make-shell/test-windows-small fail natively, raise advisory-reports headroom (review SF-1, SF-2, N-2, N-3)

- windows-make-shell: drop continue-on-error from install/exercise steps
  (kept only on the Cygwin setup step), delete the hand-written "Fail if a
  required Windows Make shell leg failed" aggregator, and delete the unused
  native/msys2/cygwin job outputs. A failing leg now reds the job via native
  GitHub Actions propagation instead of relying on an aggregator step that a
  mutation showed could silently undershoot (SF-1, N-3).
- test-windows-small: same fix - drop continue-on-error from doltversion/
  dbproxy-server steps and delete their aggregator step.
- Add timeout-minutes: 10 to the MSYS2 and Cygwin "Set up ... GNU Make"
  steps, so a hung installer download can no longer take out the whole job
  (SF-2).
- advisory-reports: raise timeout-minutes from 45 to 60 for more headroom
  above the sum of its step timeouts (N-2).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci): real GH-expression evaluator shared with F7c, behavioral xcompile test, secret/run-command coverage (review SF-3, SF-5, SF-6, N-1, N-4, N-6)

- New scripts/ci_blacksmith_runner_test.go: a real GitHub Actions expression
  evaluator (tokenizer + recursive-descent parser for ==, !=, &&, ||, !,
  parens, string literals, dotted lookups, with GitHub's short-circuit-
  returns-operand && / || semantics) that evaluates the REAL pinned
  sameRepoBlacksmith{2,4,8}vcpu expression strings from pr.yml/pr-risk.yml,
  replacing the old TestSameRepoBlacksmithExpressionSemantics' tautological
  hand-written Go mirror (SF-3). This file is the shared helper F7c's own
  TestSameRepoBlacksmithExpressionSemantics should rebase onto - see its
  doc comment for the "Public surface" (the three consts, evalGHExpr, and
  mustEvalGHRunsOn) (SF-5).
- ci_workflow_test.go: drop the old consts/test now living in the shared
  file; extend TestBlacksmithJobsReadNoSecrets to also scan job.If and
  step.If for secret references, not just env/with/run (N-4; container/
  services credentials and matrix-marker resolution stay deferred to F7b);
  pin fast-checks' build-tags step invoking all 3 scripts and the
  migration-hygiene BASE_SHA env, and add
  TestPRWindowsSmallDbproxyServerIsCGODisabled (N-6); add
  TestPRWindowsMakeShellLegsFailTheJob and
  TestPRWindowsMakeShellSetupStepsHaveTimeouts pinning the SF-1/SF-2 fixes.
- New scripts/check_release_cross_compile_test.go: behavioral test for
  check-release-cross-compile.sh using a fake `go` on PATH - asserts a
  failing target propagates as non-zero, every target in the group is still
  attempted, every invocation passes CGO_ENABLED=0 and the gms_pure_go tag,
  and an unknown group fails loudly with zero invocations (SF-6).
- check-release-cross-compile.sh: document the deliberate sequential-build
  (vs. the spec's 2-wide suggestion) choice and its revisit trigger (N-1).
- scripts/BUILD.bazel: add the two new test files to scripts_test's explicit
  srcs list (not a glob); verified against `make bazel-sync-check`.

All new/changed tests mutation-tested: each production-code mutation was
caught by the relevant test, then reverted via `cp` from a /tmp backup and
diff-verified byte-identical (never git stash / git checkout --).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(ci): fix stale check names and aggregator description in CI_REQUIRED_CHECK_TOPOLOGY.md (review N-7)

- Replace the stale "Check build-tag policy", "Check version consistency",
  "Check for .beads changes" and "Check formatting" entries under "Do not
  require these existing check names" with the actual post-fold name,
  "Fast checks (build tags, versions, migrations, beads diff, fmt)", and add
  the cross-compilation and windows-make-shell fold's actual check names.
- Rewrite the windows-make-shell fold description: it no longer has a
  hand-written aggregator step (deleted per SF-1 in the prior commit), and
  "the separate cygwin job" is now described correctly as a pre-fold matrix
  leg, not a job.
- Update the advisory-reports timeout mention from 45 to 60 minutes (N-2).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(ci): drop local design-doc path from topology doc

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 13:13:33 -07:00