From 30f30ca95b9d177b84451ba16cf53e208aab4d5b Mon Sep 17 00:00:00 2001 From: Jim Wordelman Date: Wed, 19 Aug 2026 20:39:38 -0700 Subject: [PATCH] ci: PR-CI lane runs the full internal/storage/dolt suite against a live Dolt server (#5836) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(ci): red — gastownhall/beads: no PR-CI lane runs TestBenchDBPurgeDoesNotLeak against a live Dolt server (checkDolt SKIP) (refs be-aiy5) * feat(ci): green — gastownhall/beads: no PR-CI lane runs TestBenchDBPurgeDoesNotLeak against a live Dolt server (checkDolt SKIP) (refs be-aiy5) Add test-server-storage-full as a sibling job to test-server-storage in pr-risk.yml. build-embedded already compiles every top-level test in internal/storage/dolt into one binary (embedded-test-binaries); the existing job only ever runs -test.run '^TestConformance$' against it. The new job reuses that same artifact and live-server setup with no new build step, and runs everything else via -test.skip '^TestConformance$' so previously-stranded tests (TestCreateGuard_*, federation/concurrent suites, and diff-owned tests such as TestBenchDBPurgeDoesNotLeak on PR #5792) get a real PASS/FAIL instead of a silent SKIP. Sets BEADS_TEST_ENV_RUN_DOLT=1 so federation_test.go's own live-server check fails loudly rather than skipping if this job's setup ever regresses. Wired into ci-gate: added to needs, CI_GATE_REQUIRED, and the tier-gated skipped_ok fallback (mirrors test-server-storage's existing treatment in each). Also updates two pre-existing meta-tests over pr-risk.yml that pinned exact snapshots of the file this job necessarily changes: scripts/ci_capability_selector_test.go's ci-gate authority snapshot, and scripts/pull_dolt_image_test.go's retry-helper call count (pr-risk.yml: 2 -> 3, the new job's own Dolt-image-pull step). * chore: release gate PASS for be-pp7e (PR-CI lane for TestBenchDBPurgeDoesNotLeak) Deploy gate record for be-pp7e, gating be-aiy5's PR-CI lane addition (runs TestBenchDBPurgeDoesNotLeak against a live Dolt server instead of letting checkDolt SKIP it). Review: be-w90n, verdict pass. All 7 gate criteria PASS; two pre-existing/unrelated failures encountered during evaluation (be-jy56, be-vf95) were attributed and excluded from the verdict. * test(ci): red — shard test-server-storage-full across 16 matrix jobs (refs be-yy25) TestPRRiskGateReachesFullServerDoltStorageSuite now asserts pr-risk.yml's test-server-storage-full job is a fail-fast:false matrix over shard 1..16, with the Test step invoking server-storage-test-shard.sh, instead of pinning the old single-job unsharded command. Fails against the current workflow, which still runs all 1126 tests unsharded in one 15m job (be-extn). * feat(ci): green — shard test-server-storage-full across 16 matrix jobs (refs be-yy25) Adds .github/scripts/server-storage-test-shard.sh (mirrors embedded-storage-test-shard.sh: discovers internal/storage/dolt/*_test.go, excludes TestConformance, manifest-plus-hash-fallback shard assignment) and its shard manifest .github/scripts/server-storage-test-shards.txt, pinning TestCloudAuthCLIRouting (~9.5m) alone on shard 1 and spreading the other known 10-20s-heavy tests across the rest. pr-risk.yml's test-server-storage-full job becomes a fail-fast:false matrix over shard 1..16, Test step now runs server-storage-test-shard.sh instead of the old unsharded single-job command that died at 256/1126 tests after 15m28s with zero failures (job 95616282508). Documents the new lane in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md alongside the existing embedded matrix description. TestPRRiskGateReachesFullServerDoltStorageSuite now passes. Offline shard partition verified disjoint and complete across all 1125 discovered tests (BEADS_TEST_SHARD_LIST_ONLY=1 for i=1..16). (be-extn) * test(feat): red — Fix: sharded lane runs prebuilt test binary from repo root, breaking ../schema/migrations and ../issueops (6 tests) (refs be-gepv) * feat: green — Fix: sharded lane runs prebuilt test binary from repo root, breaking ../schema/migrations and ../issueops (6 tests) (refs be-gepv) * chore: release gate PASS for be-23f3 Extends deploy/be-pp7e-gate (PR gastownhall/beads#5836) with be-gepv's fix rather than cutting a fresh isolated branch: be-gepv's own bead text and be-extn's precedent both establish this fix belongs on the already-open PR. All 7 gate criteria PASS; two repo-wide lint/policy failures attributed to already-tracked pre-existing issues (be-vf95, be-jy56), confirmed untouched by this PR lineage. be-gepv, be-6yo1, be-23f3 * fix(dolt): federation test isolation + panic (be-3c78s) TestFederationDatabaseIsolation and TestFederationVersionControlAPIs are federation-prototype tests that predate the suite's shared-server, branch-per-test isolation architecture (setupTestStore/StartTestBranch) and were never updated to account for it. - setupFederationStore hardcoded Database: "beads" for both towns. New() always connects in server mode against one shared Dolt server for the whole test binary (TestMain), so Path alone doesn't isolate two stores — Database does. Both towns silently shared one database, so Alpha could read Beta's issues and vice versa. Give each town its own database (beads_test_federation_, matching the beads_test AD-01 cleanup-firewall prefix). - TestFederationDatabaseIsolation's not-found assertions assumed GetIssue returns (nil, nil) on a miss. It never does — issueops.GetIssueInTx always returns a wrapped storage.ErrNotFound. Switched to errors.Is(err, storage.ErrNotFound). - TestFederationVersionControlAPIs hardcoded Checkout(ctx, "main") to return to its "starting" branch, but setupTestStore already checks out a private branch (test--) — "main" is a branch shared by every parallel test in the binary and never has vc-001. That mismatch is what produced the nil-pointer panic (GetIssue's error was discarded via `_`, and the resulting nil issue was dereferenced). Capture the real starting branch via CurrentBranch() right after setupTestStore() and check both previously-discarded GetIssue errors before dereferencing. Neither bug is a production security issue: New(), Checkout(), and GetIssue() all behave exactly as specified. Both bugs are confined to this prototype test file's own setup/assertions, which hadn't caught up to the isolation model the rest of the suite already uses correctly. Verified via `go build`/`go vet`/`go test -c` (all clean) and running the compiled binary, which now reaches and cleanly SKIPs both tests (no Docker locally) instead of the previous isolation failure / panic surfacing at compile-adjacent runtime. Full GREEN confirmation is via CI, which has a working Dolt/Docker environment. Refs be-3c78s * chore: release gate PASS for be-42tlj --- .github/scripts/server-storage-test-shard.sh | 160 +++++++++++++++++ .../scripts/server-storage-test-shards.txt | 41 +++++ .github/workflows/pr-risk.yml | 68 ++++++- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md | 24 +++ internal/storage/dolt/federation_test.go | 67 ++++--- release-gates/be-23f3-gate.md | 110 ++++++++++++ release-gates/be-42tlj-gate.md | 95 ++++++++++ release-gates/be-pp7e-gate.md | 99 +++++++++++ scripts/ci_capability_selector_test.go | 8 +- scripts/ci_workflow_test.go | 166 +++++++++++++++++- scripts/pull_dolt_image_test.go | 2 +- 11 files changed, 809 insertions(+), 31 deletions(-) create mode 100755 .github/scripts/server-storage-test-shard.sh create mode 100644 .github/scripts/server-storage-test-shards.txt create mode 100644 release-gates/be-23f3-gate.md create mode 100644 release-gates/be-42tlj-gate.md create mode 100644 release-gates/be-pp7e-gate.md diff --git a/.github/scripts/server-storage-test-shard.sh b/.github/scripts/server-storage-test-shard.sh new file mode 100755 index 000000000..c5ff42a61 --- /dev/null +++ b/.github/scripts/server-storage-test-shard.sh @@ -0,0 +1,160 @@ +#!/usr/bin/env bash +# server-storage-test-shard.sh — run a shard of server dolt storage tests. +# +# Usage: server-storage-test-shard.sh +# +# Discovers all top-level Test* functions from internal/storage/dolt/*_test.go, +# excluding TestConformance. That suite runs in the dedicated +# test-server-storage job instead, sharded by sub-test path (-test.run +# '^TestConformance$'), not by top-level function name; this manifest's +# exact-name matching does not cover it. See +# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md for why the two use different +# sharding mechanisms. +# +# Tests listed in server-storage-test-shards.txt for the requested shard count +# use that committed assignment; newly-added tests fall back to +# hash(name) % total. Runs the matching subset using the pre-built test +# binary at BEADS_TEST_SERVER_TEST_BINARY (or /tmp/dolt-conformance-test). +# +# This package hands 1126 top-level tests to a single job (be-extn): the +# sibling embedded lane's test-embedded-storage shards 324 tests over 5 jobs. +# Server mode is also slower per-test (real socket round-trips, per-test +# CREATE/DROP DATABASE) than embedded. TestCloudAuthCLIRouting alone took 566s +# (63% of the old 15m budget) across 16 subtests, each paying a full +# CREATE DATABASE + migration chain + dolt init + dolt remote add + DROP +# DATABASE (~35s) to test one routing predicate; it uses t.Setenv, so it can +# never be parallel. The manifest pins it (and other known-heavy tests) to +# distinct shards so they do not cluster. +# +# Environment: +# BEADS_TEST_ENV_RUN_DOLT=1 required (tests skip without it) +# BEADS_TEST_SERVER_TEST_BINARY= pre-built storage test binary +# (default: /tmp/dolt-conformance-test) +# BEADS_TEST_SHARD_LIST_ONLY=1 print selected tests without running them + +set -euo pipefail + +SHARD_NUMBER="${1:?usage: $0 }" +TOTAL_SHARDS="${2:?usage: $0 }" +shift 2 + +if ! [[ "$SHARD_NUMBER" =~ ^[0-9]+$ ]] || ! [[ "$TOTAL_SHARDS" =~ ^[0-9]+$ ]]; then + echo "Shard number and total shards must be positive integers" >&2 + exit 1 +fi +if (( TOTAL_SHARDS < 1 || SHARD_NUMBER < 1 || SHARD_NUMBER > TOTAL_SHARDS )); then + echo "Invalid shard ${SHARD_NUMBER}/${TOTAL_SHARDS}" >&2 + exit 1 +fi + +SHARD_INDEX=$(( SHARD_NUMBER - 1 )) +MANIFEST="${BEADS_TEST_SHARD_MANIFEST:-.github/scripts/server-storage-test-shards.txt}" + +# Discover all top-level Test* functions except TestConformance, which is +# sharded separately (test-server-storage job, -test.run '^TestConformance$') +# because it is a single top-level function with sub-tests reached via +# -test.run path regex, not distinct top-level function names. +ALL_TESTS=$(grep -rh '^func Test' internal/storage/dolt/*_test.go \ + | sed 's/func \(Test[A-Za-z0-9_]*\).*/\1/' \ + | grep -v '^TestConformance$' \ + | sort -u) + +if [ -z "$ALL_TESTS" ]; then + echo "No Test* functions found" >&2 + exit 1 +fi + +declare -A ALL_TEST_SET=() +while IFS= read -r name; do + ALL_TEST_SET["$name"]=1 +done <<< "$ALL_TESTS" + +declare -A MANIFEST_SHARDS=() +if [ -f "$MANIFEST" ]; then + while IFS= read -r line || [ -n "$line" ]; do + line="${line%%#*}" + read -r manifest_total manifest_shard test_name extra <<< "$line" + if [ -z "${manifest_total:-}" ]; then + continue + fi + if [ -n "${extra:-}" ] || ! [[ "$manifest_total" =~ ^[0-9]+$ ]] || ! [[ "$manifest_shard" =~ ^[0-9]+$ ]] || ! [[ "$test_name" =~ ^Test[A-Za-z0-9_]+$ ]] || [ "$test_name" = "TestConformance" ]; then + echo "Invalid shard manifest line: $line" >&2 + exit 1 + fi + if (( manifest_total != TOTAL_SHARDS )); then + continue + fi + if (( manifest_shard < 1 || manifest_shard > TOTAL_SHARDS )); then + echo "Invalid shard ${manifest_shard}/${manifest_total} for $test_name in $MANIFEST" >&2 + exit 1 + fi + if [ -n "${MANIFEST_SHARDS[$test_name]:-}" ]; then + echo "Duplicate shard manifest entry for $test_name in $MANIFEST" >&2 + exit 1 + fi + MANIFEST_SHARDS["$test_name"]="$manifest_shard" + done < "$MANIFEST" +fi + +for name in "${!MANIFEST_SHARDS[@]}"; do + if [ -z "${ALL_TEST_SET[$name]:-}" ]; then + echo "Shard manifest entry does not match a discovered test: $name" >&2 + exit 1 + fi +done + +SHARD_TESTS=() +MANIFEST_COUNT=0 +FALLBACK_COUNT=0 +while IFS= read -r name; do + assigned_shard="${MANIFEST_SHARDS[$name]:-}" + if [ -n "$assigned_shard" ]; then + if (( assigned_shard == SHARD_NUMBER )); then + SHARD_TESTS+=("$name") + MANIFEST_COUNT=$(( MANIFEST_COUNT + 1 )) + fi + continue + fi + + # Use cksum for a portable numeric hash fallback for newly-added tests. + hash=$(printf %s "$name" | cksum | cut -d ' ' -f 1) + if (( hash % TOTAL_SHARDS == SHARD_INDEX )); then + SHARD_TESTS+=("$name") + FALLBACK_COUNT=$(( FALLBACK_COUNT + 1 )) + fi +done <<< "$ALL_TESTS" + +if [ ${#SHARD_TESTS[@]} -eq 0 ]; then + echo "Shard ${SHARD_NUMBER}/${TOTAL_SHARDS}: no tests assigned (all hashed to other shards)" + exit 0 +fi + +# Build the -run regex: "^(TestA|TestB|TestC)$" +RUN_REGEX="^($(IFS='|'; echo "${SHARD_TESTS[*]}"))$" + +echo "Shard ${SHARD_NUMBER}/${TOTAL_SHARDS}: running ${#SHARD_TESTS[@]} test(s)" +echo " manifest: ${MANIFEST_COUNT}, fallback: ${FALLBACK_COUNT}" +printf " %s\n" "${SHARD_TESTS[@]}" +echo "" + +# Use pre-built test binary if available, otherwise fall back to go test. +STORAGE_BINARY="${BEADS_TEST_SERVER_TEST_BINARY:-/tmp/dolt-conformance-test}" +if [ "${BEADS_TEST_SHARD_LIST_ONLY:-}" = "1" ]; then + exit 0 +fi + +if [ -x "$STORAGE_BINARY" ]; then + # The tests use paths relative to their own package dir (e.g. + # ../schema/migrations, ../issueops). `go test` runs them with cwd = the + # package dir; a prebuilt binary inherits OUR cwd, so set it explicitly. + cd internal/storage/dolt + exec "$STORAGE_BINARY" -test.v -test.count=1 -test.timeout=15m \ + -test.run "$RUN_REGEX" \ + "$@" +else + echo "Warning: pre-built test binary not found at $STORAGE_BINARY, falling back to go test" + exec go test -tags=integration,gms_pure_go -v -count=1 -timeout 15m \ + -run "$RUN_REGEX" \ + "$@" \ + ./internal/storage/dolt/ +fi diff --git a/.github/scripts/server-storage-test-shards.txt b/.github/scripts/server-storage-test-shards.txt new file mode 100644 index 000000000..e053f0d82 --- /dev/null +++ b/.github/scripts/server-storage-test-shards.txt @@ -0,0 +1,41 @@ +# Server storage test shard manifest. +# +# Format: +# +# Unlike the embedded manifest, per-test timing is known for the tests +# pinned below (see be-extn's investigation of CI job 95616282508): +# TestCloudAuthCLIRouting alone costs ~9.5 minutes -- its 16 subtests each +# call openCloudAuthTestStore, which does CREATE DATABASE + the full +# migration chain + dolt init + dolt remote add + DROP DATABASE, and the +# test uses t.Setenv so it can never run t.Parallel. It is pinned alone on +# shard 1 so no other test shares its runtime. The remaining pins below +# (TestCloudAuthCLIRoutingStructural, TestCreateGuard_* (9 tests), +# TestFederationPeerCredentialLifecycleLazyKeyInit, +# TestConcurrentWorkQueueDrain, TestHighContentionStress, +# TestSerializationConflictRetry, +# TestConcurrentCommentAndCloseWithoutCallerRetry, TestReadWriteMix, and +# TestConcurrentIssueCreationWithoutCallerRetry) are each roughly 10-20s and +# are spread across the remaining shards so they don't cluster on one. +# Every other discovered test (the bulk of the 1126) is not listed here and +# falls through to server-storage-test-shard.sh's hash(name) % total +# fallback, the same mechanism the embedded manifest uses for its unlisted +# tests. TestConformance is deliberately excluded -- it runs in the separate +# test-server-storage job (sharded by sub-test path, not by this manifest). +16 1 TestCloudAuthCLIRouting +16 2 TestCloudAuthCLIRoutingStructural +16 2 TestReadWriteMix +16 3 TestCreateGuard_ErrorMessage +16 3 TestConcurrentIssueCreationWithoutCallerRetry +16 4 TestCreateGuard_ExistingDB_DoesNotCreateMissingBeadsDir +16 5 TestCreateGuard_ExistingDB_NoFlag +16 6 TestCreateGuard_ExistingDB_WithData +16 7 TestCreateGuard_MissingDB_CreateIfMissing +16 8 TestCreateGuard_MissingDB_DefaultConfig +16 9 TestCreateGuard_ReadOnly_MissingDB +16 10 TestCreateGuard_UnderscoreBothExist +16 11 TestCreateGuard_UnderscoreInName +16 12 TestFederationPeerCredentialLifecycleLazyKeyInit +16 13 TestConcurrentWorkQueueDrain +16 14 TestHighContentionStress +16 15 TestSerializationConflictRetry +16 16 TestConcurrentCommentAndCloseWithoutCallerRetry diff --git a/.github/workflows/pr-risk.yml b/.github/workflows/pr-risk.yml index 6a8fc979d..7748cba71 100644 --- a/.github/workflows/pr-risk.yml +++ b/.github/workflows/pr-risk.yml @@ -324,6 +324,69 @@ jobs: # suite runs instead of self-skipping. run: /tmp/dolt-conformance-test -test.v -test.count=1 -test.timeout=15m -test.run '^TestConformance$' + test-server-storage-full: + name: Test (Server Dolt Full Suite ${{ matrix.shard }}/${{ strategy.job-total }}) + needs: [detect-ci-tier, build-embedded] + if: needs.detect-ci-tier.outputs.full_embedded == 'true' + runs-on: ubuntu-latest + timeout-minutes: 20 + strategy: + fail-fast: false + matrix: + shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16] + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 + + - name: Download binaries + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + name: embedded-test-binaries + path: /tmp/ + + - name: Fix permissions + run: chmod +x /tmp/dolt-conformance-test + + - name: Configure Git + run: | + git config --global user.name "CI Bot" + git config --global user.email "ci@beads.test" + + - name: Install Dolt + run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash + + - name: Configure Dolt + run: | + dolt config --global --add user.name "CI Bot" + dolt config --global --add user.email "ci@beads.test" + dolt version + + - name: Pull Dolt sql-server image + # Presence of this image flips isDoltImageCached() true, so the suite + # exercises server Dolt instead of self-skipping. Keep the tag in sync + # with internal/testutil/testdoltcommon.go:DoltDockerImage. + run: ./scripts/ci/pull-dolt-image.sh + + - name: Test + # build-embedded's dolt-conformance-test binary compiles every + # top-level test in internal/storage/dolt, not just conformance -- + # test-server-storage above only ever runs the ^TestConformance$ + # subset. Everything else in that binary (TestCreateGuard_*, + # federation/create-guard/concurrent suites, and diff-owned tests such + # as TestBenchDBPurgeDoesNotLeak on PR #5792) was reached by no + # PR-triggered lane and silently SKIPped for lack of a live server. + # This sibling job reuses the same artifact and server setup and runs + # everything else instead, so a SKIP there gets a real PASS/FAIL. + # + # Manifest-driven shard of the full suite (mirrors test-embedded-storage + # / server-storage-test-shard.sh): 1126 tests in one 15m-Go-timeout job + # never finished (job 95616282508 died at 256/1126 tests, 15m28s in, + # zero failures -- it just ran out of time). server-storage-test-shard.sh + # excludes TestConformance from discovery, so it never lands in one of + # these shards. + env: + BEADS_TEST_ENV_RUN_DOLT: "1" + run: bash .github/scripts/server-storage-test-shard.sh ${{ matrix.shard }} 16 + test-embedded-cmd: name: Test (Embedded Dolt Cmd ${{ matrix.shard }}/${{ strategy.job-total }}) needs: [detect-ci-tier, build-embedded] @@ -389,6 +452,7 @@ jobs: - test-embedded-storage - test-embedded-conformance - test-server-storage + - test-server-storage-full - test-embedded-cmd - test-proxied-cmd - test-nix @@ -405,6 +469,7 @@ jobs: TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE + TEST_SERVER_STORAGE_FULL TEST_EMBEDDED_CMD TEST_PROXIED_CMD TEST_NIX @@ -414,13 +479,14 @@ jobs: TEST_EMBEDDED_STORAGE: ${{ needs.test-embedded-storage.result }} TEST_EMBEDDED_CONFORMANCE: ${{ needs.test-embedded-conformance.result }} TEST_SERVER_STORAGE: ${{ needs.test-server-storage.result }} + TEST_SERVER_STORAGE_FULL: ${{ needs.test-server-storage-full.result }} TEST_EMBEDDED_CMD: ${{ needs.test-embedded-cmd.result }} TEST_PROXIED_CMD: ${{ needs.test-proxied-cmd.result }} TEST_NIX: ${{ needs.test-nix.result }} run: | skipped_ok="" if [[ "$FULL_EMBEDDED" != "true" ]]; then - skipped_ok="BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_EMBEDDED_CMD TEST_PROXIED_CMD" + skipped_ok="BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_SERVER_STORAGE_FULL TEST_EMBEDDED_CMD TEST_PROXIED_CMD" fi export CI_GATE_SKIPPED_OK="$skipped_ok" bash .github/scripts/ci-gate.sh diff --git a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md index 2c06b53f4..3de6f93c5 100644 --- a/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md +++ b/engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md @@ -277,6 +277,30 @@ The current embedded Dolt topology already fits the required-check model: - Docs-only PRs can skip the embedded matrix without leaving the required gate pending, because the aggregate job still runs. +### Server Dolt Storage Matrix + +`test-server-storage-full` mirrors `test-embedded-storage`'s sharding, one +tier down in the same workflow: + +- Job-level `if` uses the same `detect-ci-tier` gate as the embedded matrix. +- `.github/scripts/server-storage-test-shard.sh` discovers top-level + `Test*` functions from `internal/storage/dolt/*_test.go` (excluding + `TestConformance`, which keeps its own `test-server-storage` job), assigns + known-heavy tests via the committed + `.github/scripts/server-storage-test-shards.txt` manifest, and + hash-distributes everything else — the same manifest-plus-fallback + mechanism `embedded-storage-test-shard.sh` uses. +- 16 shards (vs. embedded's 5): server-mode tests are real socket + round-trips against a containerized Dolt server with a per-test + CREATE/DROP DATABASE, and the package has 3.5x as many top-level tests + (1126 vs. 324) as the embedded suite. An earlier unsharded single job + (15m Go timeout, `timeout-minutes: 20`) never finished — it died at + 256/1126 tests with zero failures, just out of time. One test, + `TestCloudAuthCLIRouting`, alone costs ~9.5 minutes and is pinned alone + on shard 1 in the manifest so it cannot delay any other shard. +- `fail-fast: false`, matching every other matrix job in this workflow: one + slow or flaky shard should not cancel its siblings. + ### Regression Tests `Regression Tests` can stay visible as a non-required workflow. If regression diff --git a/internal/storage/dolt/federation_test.go b/internal/storage/dolt/federation_test.go index 6e14624b0..e8bc4e61e 100644 --- a/internal/storage/dolt/federation_test.go +++ b/internal/storage/dolt/federation_test.go @@ -105,23 +105,20 @@ func TestFederationDatabaseIsolation(t *testing.T) { t.Fatalf("failed to commit in beta: %v", err) } - // Verify isolation: Alpha should NOT see Beta's issue + // Verify isolation: Alpha should NOT see Beta's issue. GetIssue never + // returns (nil, nil) on a miss — issueops.GetIssueInTx always returns a + // wrapped storage.ErrNotFound — so isolation is proven by that error, + // not by a nil issue with no error. issueFromAlpha, err := alphaStore.GetIssue(ctx, "beta-001") - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if issueFromAlpha != nil { - t.Fatalf("isolation violated: alpha found beta-001") + if !errors.Is(err, storage.ErrNotFound) { + t.Fatalf("isolation violated: alpha found beta-001 (issue=%v, err=%v)", issueFromAlpha, err) } t.Log("✓ Alpha cannot see beta-001") // Verify isolation: Beta should NOT see Alpha's issue issueFromBeta, err := betaStore.GetIssue(ctx, "alpha-001") - if err != nil { - t.Fatalf("unexpected error: %v", err) - } - if issueFromBeta != nil { - t.Fatalf("isolation violated: beta found alpha-001") + if !errors.Is(err, storage.ErrNotFound) { + t.Fatalf("isolation violated: beta found alpha-001 (issue=%v, err=%v)", issueFromBeta, err) } t.Log("✓ Beta cannot see alpha-001") @@ -150,6 +147,17 @@ func TestFederationVersionControlAPIs(t *testing.T) { store, cleanup := setupTestStore(t) defer cleanup() + // setupTestStore already checked this store out to its own isolated + // branch (testutil.StartTestBranch), not the shared database's literal + // "main" — under this suite's branch-per-test isolation, "main" names a + // branch shared by every parallel test in the binary. Capture the actual + // starting branch so this test returns to ITS OWN state later instead of + // jumping onto that shared branch, which never has vc-001 (be-3c78s). + startBranch, err := store.CurrentBranch(ctx) + if err != nil { + t.Fatalf("failed to get starting branch: %v", err) + } + // Create initial issue issue := &types.Issue{ ID: "vc-001", @@ -199,17 +207,20 @@ func TestFederationVersionControlAPIs(t *testing.T) { t.Fatalf("failed to commit: %v", err) } - // Switch back to main - if err := store.Checkout(ctx, "main"); err != nil { - t.Fatalf("failed to checkout main: %v", err) + // Switch back to this test's own starting branch + if err := store.Checkout(ctx, startBranch); err != nil { + t.Fatalf("failed to checkout %s: %v", startBranch, err) } - // Verify main still has original title - mainIssue, _ := store.GetIssue(ctx, "vc-001") - if mainIssue.Title != "Version control test" { - t.Errorf("main should have original title, got %q", mainIssue.Title) + // Verify original branch still has original title + mainIssue, err := store.GetIssue(ctx, "vc-001") + if err != nil { + t.Fatalf("failed to get issue after checkout: %v", err) } - t.Log("✓ Main branch unchanged") + if mainIssue.Title != "Version control test" { + t.Errorf("original branch should have original title, got %q", mainIssue.Title) + } + t.Log("✓ Original branch unchanged") // Merge feature branch conflicts, err := store.Merge(ctx, "feature-branch") @@ -222,7 +233,10 @@ func TestFederationVersionControlAPIs(t *testing.T) { t.Log("✓ Merged feature-branch into main") // Verify merge result - mergedIssue, _ := store.GetIssue(ctx, "vc-001") + mergedIssue, err := store.GetIssue(ctx, "vc-001") + if err != nil { + t.Fatalf("failed to get issue after merge: %v", err) + } if mergedIssue.Title != "Updated on feature branch" { t.Errorf("expected merged title, got %q", mergedIssue.Title) } @@ -1489,10 +1503,15 @@ func setupFederationStore(t *testing.T, ctx context.Context, path, prefix string t.Helper() cfg := &Config{ - Path: path, - CommitterName: "town-" + prefix, - CommitterEmail: prefix + "@federation.test", - Database: "beads", + Path: path, + CommitterName: "town-" + prefix, + CommitterEmail: prefix + "@federation.test", + // Each town needs its own database, not just its own Path: New() + // always connects in server mode, and TestMain runs one shared Dolt + // server for the whole test binary, so Path alone does not isolate + // two stores — Database does. A shared literal "beads" here let every + // town silently attach to the same database (be-3c78s). + Database: "beads_test_federation_" + prefix, CreateIfMissing: true, // test creates fresh database } diff --git a/release-gates/be-23f3-gate.md b/release-gates/be-23f3-gate.md new file mode 100644 index 000000000..ae824bfbb --- /dev/null +++ b/release-gates/be-23f3-gate.md @@ -0,0 +1,110 @@ +# Release gate — shard test-server-storage-full so prebuilt-binary runs land in the right package dir (be-gepv) + +- **Builder bead:** be-gepv — fix `.github/scripts/server-storage-test-shard.sh` + so the prebuilt-binary branch `cd`s into `internal/storage/dolt` before + exec, matching the go-test fallback branch's working directory. +- **Deploy bead:** be-23f3 +- **Review bead:** be-6yo1 — verdict **PASS**, recorded on commit + `15c6d0a7d08b8fa03e4f2a827adf23f81e55de4d` +- **Commits:** `2cb3d8f2845b023741d65571827267a0a5323ce4` (tdd_red) then + `15c6d0a7d08b8fa03e4f2a827adf23f81e55de4d` (tdd_green), 2 commits over the + prior deploy tip `9caf412234fa71dc1564e8f25df6fc3be52d9e4b` +- **Branch:** `builder/be-gepv` (provenance only, not a push target) +- **Evaluated:** 2026-08-18 by beads/deployer + +## Scope deviation: extends PR #5836 instead of a fresh isolated branch + +be-gepv's own bead text is explicit that this fix targets an already-open +PR: *"This lands on PR #5836, branch `deploy/be-pp7e-gate`. ... So this fix +belongs ON that PR, not on main."* This matches established precedent — +be-extn's prior fix (sharding `test-server-storage-full`'s 1126-test job to +stop timing out) was already pushed directly onto the same +`deploy/be-pp7e-gate` branch rather than spawning a new isolated +branch+PR. be-23f3's boilerplate description text ("cut a fresh ISOLATED +deploy branch... open the PR") is the generic single-bead template; it is +superseded here by the bead's own first-party provenance note and the +directly-applicable precedent. + +Accordingly, this gate evaluates be-gepv's own incremental diff against the +existing `deploy/be-pp7e-gate` tip, not a full rebuild of PR #5836's +7-commit history (already separately gated: be-pp7e/be-aiy5/be-extn covered +the prior commits). Ancestry scope formally validated via +`assert_deploy_ancestry_scope`: + +``` +assert_deploy_ancestry_scope 9caf412234fa71dc1564e8f25df6fc3be52d9e4b \ + 15c6d0a7d08b8fa03e4f2a827adf23f81e55de4d be-gepv be-6yo1 # rc=0 +assert_deploy_ancestry_scope origin/main \ + 15c6d0a7d08b8fa03e4f2a827adf23f81e55de4d \ + be-gepv be-yy25 be-pp7e be-aiy5 be-extn be-6yo1 be-w90n # rc=0 +assert_safe_push_target "deploy/be-pp7e-gate" # rc=0 +``` + +## Scope + +be-gepv's own increment, confirmed via +`git diff --stat 9caf412234f..15c6d0a7d` (2 files, 86 insertions, 0 +deletions, no `.claude/**` paths): + +- `.github/scripts/server-storage-test-shard.sh` — 4-line fix: adds a + comment and `cd internal/storage/dolt` immediately before the + `exec "$STORAGE_BINARY"` call, on the prebuilt-binary branch only. +- `scripts/ci_workflow_test.go` — new test + `TestServerStorageShardScriptRunsPrebuiltBinaryFromPackageDir` (82 + lines) asserting the fix's exact line-ordering invariants: discovery-grep + before `cd`, `cd` between the prebuilt-binary guard and its exec, `cd` + absent from the go-test fallback branch. + +## Gate criteria + +| # | Criterion | Verdict | Evidence | +|---|-----------|---------|----------| +| 1 | Review PASS present | **PASS** | be-6yo1 records verdict PASS on commit `15c6d0a7d08b8fa03e4f2a827adf23f81e55de4d` — exact match to this deploy's commit. | +| 2 | Acceptance criteria met | **PASS** | be-6yo1's `spec_findings.tests_green: true`, confirmed via `TEST_COVER=1 ./scripts/test.sh`; be-gepv's Done-when checklist (4 items) independently re-confirmed below. | +| 3 | Tests pass | **PASS** | Diff-owned package re-run by deployer, real PASS (see below), matching reviewer's own record. | +| 3a | Pre-existing-failure attribution | **N/A — attributed** | `make ci-pr-lint` repo-wide fails solely on 3x gosec G602 in `backend/conformance/{cycle_detector_contract.go:494,importer_contract.go:381,relations_contract.go:672}` — tracked by already-open **be-vf95**, confirmed byte-identical to `origin/main` (`git log origin/main..HEAD -- backend/conformance/` empty — this PR lineage never touches that directory). `make ci-pr-policy` fails solely on the `.githooks/commit-msg` version-marker check — tracked by already-open **be-jy56**, likewise confirmed untouched by this lineage (`git log origin/main..HEAD -- .githooks/commit-msg` empty). | +| 3b | Policy/lint lane | **PASS** | Diff-scoped `golangci-lint run ./scripts/...` reports **0 issues**. `gofmt` clean (via `make ci-pr-lint`'s own Go-scoped check — "All Go files are properly formatted"; a stray manual `gofmt -l` invocation against the shell script itself is a self-inflicted false alarm, gofmt cannot parse non-Go source). `go vet ./scripts/...` clean. `bash -n` on the shell script clean. | +| 4 | No unresolved HIGH findings | **PASS** | be-6yo1: `style_findings: none`, `security_findings: none` — full OWASP Top-10 walk, minimal diff surface, no injection/auth/data-exposure concerns. | +| 5 | Clean working tree | **PASS** | `git status --porcelain` on the evaluated commit shows no staged/unstaged changes — only the pre-existing, unrelated untracked scratch files already present in this worktree (`release-gates/be-hi97-*.md`, `release-gates/be-k9js-*.md`, `release-gates/be-uoat-*.md`, `scripts/rebase-resolve-lib.sh`), never staged. | +| 6 | Clean divergence from `origin/main` | **PASS** | `git merge-base --is-ancestor origin/main HEAD` succeeds — clean fast-forward relationship. `assert_deploy_ancestry_scope` over the full 7-commit lineage returns rc=0 against the accepted bead-id set (see scope-deviation section above). | +| 7 | Single feature theme | **PASS** | Both files serve the one fix: the shell-script correction plus the one test file whose new case pins the corrected behavior. No unrelated changes riding along. | + +## Tests run on release branch (independent re-verification) + +Diff-owned package, run via the canonical test runner: + +``` +./scripts/test.sh ./scripts/... +``` + +Result: package `github.com/steveyegge/beads/scripts` — **PASS** in 2.439s, +no skips. Includes `TestServerStorageShardScriptRunsPrebuiltBinaryFromPackageDir` +passing for real (not skipped), confirming the line-ordering invariants the +fix establishes. + +Static checks, independently re-run: + +| Check | Result | +|---|---| +| `go build ./...` | clean | +| `go vet ./scripts/...` | clean | +| `bash -n .github/scripts/server-storage-test-shard.sh` | clean | +| `golangci-lint run ./scripts/...` (diff-scoped) | clean, 0 issues | +| `make ci-pr-lint` (repo-wide) | FAIL — solely be-vf95 (pre-existing; see criterion 3a) | +| `make ci-pr-policy` (repo-wide) | FAIL — solely be-jy56 (pre-existing; see criterion 3a) | + +## Findings from reviews (no action required) + +From be-6yo1: no style or security findings. gofmt/`bash -n`/go vet all +clean; minimal diff surface (4-line shell fix + 1 new test); no +injection/auth/data-exposure/etc. concerns identified across the full +OWASP Top-10 walk. + +## Verdict + +**PASS** — all 7 criteria (plus 3a/3b) clear. This fix extends the already +gated-and-open `deploy/be-pp7e-gate` branch / PR #5836 rather than cutting a +new isolated branch (see scope-deviation section). Per this repo's +contributor-only status, the job ends at the updated PR — no merge-request +will be routed to mayor and no deploy-clearance status will be posted; +merge authority belongs to the upstream maintainers. diff --git a/release-gates/be-42tlj-gate.md b/release-gates/be-42tlj-gate.md new file mode 100644 index 000000000..0df87b702 --- /dev/null +++ b/release-gates/be-42tlj-gate.md @@ -0,0 +1,95 @@ +# Release gate — Fix federation cross-tenant isolation violation + panic + +- **Builder bead:** be-3c78s — fix `TestFederationDatabaseIsolation` (cross- + tenant isolation violation) and `TestFederationVersionControlAPIs` + (unrecovered panic) in `internal/storage/dolt/federation_test.go`. +- **Deploy bead:** be-42tlj +- **Review bead:** be-0vbbs — verdict **PASS**, recorded on commit + `78e32879084ff412923cef5f6bb49f13ee69488c` +- **Commits:** `7d82a9e46c35b2a9f2ef761ad06644e6719bc28b` (tdd_red, = the + prior deploy tip, "chore: release gate PASS for be-23f3") then + `78e32879084ff412923cef5f6bb49f13ee69488c` (tdd_green) — exactly 1 commit + over the prior deploy tip, confirmed via `git rev-parse HEAD~1`. +- **Branch:** `deploy/be-pp7e-gate` (reused; lands on existing open PR + #5836, pushed via `prhead` — origin is fetch-only for this rig) +- **Evaluated:** 2026-08-19 by beads/deployer + +## Scope deviation: extends PR #5836 instead of a fresh isolated branch + +be-3c78s's own bead text ("Landing" section) is explicit: PR #5836 (branch +`deploy/be-pp7e-gate`) is the still-open home for this test suite; push +directly onto that branch per this rig's established deploy-branch-reuse +precedent (be-pp7e → be-23f3 → be-gepv → be-3c78s) rather than opening a new +PR. This matches the identical override already applied and documented in +this same branch's prior gate (`release-gates/be-23f3-gate.md`). + +Accordingly, this gate evaluates be-3c78s's own incremental diff against the +existing `deploy/be-pp7e-gate` tip, not a full rebuild of PR #5836's entire +history (already separately gated by the prior commits' own gate records). + +## Scope + +be-3c78s's own increment, confirmed via `git diff --stat +7d82a9e46c..78e328790` — 1 file, +42/-23, no `.claude/**` paths: + +- `internal/storage/dolt/federation_test.go` — `setupFederationStore` gave + every federated "town" the same literal `Database: "beads"`; since `New()` + always connects in server mode against the one shared `TestMain`-started + Dolt server, `Path` alone never isolated the two towns (only `Database` + does) — root cause of the cross-tenant isolation violation. Also fixed the + tests' own assumption that `GetIssue` returns `(nil, nil)` on a miss (it + always returns a wrapped `storage.ErrNotFound`), and + `TestFederationVersionControlAPIs` hardcoding `Checkout(ctx, "main")` + instead of capturing its actual `setupTestStore`-assigned starting branch + — that mismatch, compounded by a discarded `GetIssue` error, was the + panic's root cause. Zero production code touched. + +## Gate criteria + +| # | Criterion | Verdict | Evidence | +|---|-----------|---------|----------| +| 1 | Review PASS present | **PASS** | be-0vbbs closed, reason "pass", recorded on commit `78e32879084ff412923cef5f6bb49f13ee69488c` — exact match to this deploy's commit. | +| 2 | Acceptance criteria met | **PASS** | be-3c78s's 3 acceptance criteria (isolation test passes, panic test passes with no hidden post-panic failures, security verdict stated) all independently re-confirmed below — not accepted on the builder's or reviewer's word alone. | +| 3 | Tests pass | **PASS** | Both diff-owned tests independently re-verified PASS by direct CI job-log inspection (see below). Other shard failures in the same run are pre-existing and unrelated (3a). | +| 3a | Pre-existing-failure attribution | **N/A — attributed** | 10/16 "Server Dolt Full Suite" shards fail in run `32312105789`, none touching `federation_test.go`. Root catalog: **be-7ch8l** (P1, closed — decomposed into 7 tracked children after finding 14/16 shards failing *before any fix landed*, i.e. proven pre-existing at base ref). Failing-test names pulled directly from shard logs and matched to open siblings: `TestGitRemotePushSkipsUserPrePushHook`/`TestCreateIssueAfterPull` → **be-hsa9t** (open, git-remote-harness cluster); `TestCLIBundleMatchesRuntimeCommittedSchema` → **be-vmzni**/**be-y7ddy** (open, schema-migration cluster); `TestRigIssueIsPersistentButHiddenFromReady` → **be-11ck7** (open, rig issue-type cluster); `TestReclaimRevertsExpiredOnly` → the specific flake be-0vbbs's own review notes already named (lease_test.go, unrelated). All 4 attribution clauses satisfied: not diff-owned (no path overlap with `federation_test.go`), tracked bead-id per cluster, proven pre-existing at base ref (be-7ch8l's original catalog), no path overlap. | +| 3b | Policy/lint lane | **N/A — attributed** | `make ci-pr-policy` fails solely at "check version consistency": `.githooks/commit-msg` missing BEADS INTEGRATION markers. Root cause already tracked and closed as a false positive (**be-jygq**, **be-2q84**, **be-jy56**): the file is a git-untracked, `.git/info/exclude`d local gc-rig session shim (`worktree-setup.sh`), not a real repo file — independently re-confirmed here via `git ls-files` (empty) and `git check-ignore -v` (matches `.git/info/exclude:40`). Reversible repro: moved the shim aside (sha256 backed up), re-ran `make ci-pr-policy` → **exit 0, full PASS**, then restored the shim byte-identical (checksum-verified). Real upstream CI checks out a clean tree without this shim and is unaffected. | +| 4 | No unresolved HIGH findings | **PASS** | be-0vbbs: `style_findings: none`; `security_findings`: full OWASP Top-10 walk plus independent re-derivation of the builder's "not a production issue" claim (4 independent reasons, citing `issueops/get_issue.go`, `federation.go`, and 3 named runtime firewalls in `store.go`). `bd` search across bead notes for be-0vbbs/be-3c78s/be-42tlj surfaces no separate finding beads. | +| 5 | Clean working tree | **PASS** | `git status --short` on the evaluated commit shows only the pre-existing, unrelated untracked scratch files already present in this worktree (`release-gates/be-hi97-*.md`, `release-gates/be-k9js-*.md`, `release-gates/be-p7dzx-*.md`, `release-gates/be-uoat-*.md`, `scripts/rebase-resolve-lib.sh`), never staged, unaffected by the shim move/restore. | +| 6 | Clean divergence from `origin/main` | **PASS** | HEAD is 9 ahead / 15 behind `origin/main` (stale but not a fast-forward). `gh pr view 5836 --json mergeable,mergeStateStatus` reports `mergeable: MERGEABLE`. `git merge-tree $(git merge-base HEAD origin/main) HEAD origin/main` contains **zero** real conflict markers (`grep -c '^<<<<<<<\|^CONFLICT'` = 0). `mergeStateStatus: UNSTABLE` reflects the already-attributed pre-existing shard/policy failures in 3a/3b, not a textual conflict. | +| 7 | Single feature theme | **PASS** | One file, one cohesive fix: federation-test isolation (unique `Database` per town) and the panic it was masking (stale `Checkout` target + discarded error). No unrelated changes riding along. | + +## Tests run on release branch (independent re-verification) + +Diff-owned tests, confirmed by direct CI job-log inspection (not the +reviewer's summary) via the GitHub Jobs API against run `32312105789` +("PR Risk" workflow, the exact commit `78e32879084ff412923cef5f6bb49f13ee69488c`): + +| Test | Shard | Job conclusion | Result | +|---|---|---|---| +| `TestFederationDatabaseIsolation` | 8/16 | success | `--- PASS: TestFederationDatabaseIsolation (7.58s)` — no isolation violation, no other FAIL/panic in this shard's log | +| `TestFederationVersionControlAPIs` | 5/16 | success | `--- PASS: TestFederationVersionControlAPIs (0.33s)` — no panic, no other FAIL/panic in this shard's log (post-panic tail runs to completion) | + +Static checks, independently re-run at `78e32879084ff412923cef5f6bb49f13ee69488c`: + +| Check | Result | +|---|---| +| `go build ./...` | clean | +| `go vet ./...` | clean | +| `make ci-pr-policy` (repo-wide) | FAIL — solely the untracked `.githooks/commit-msg` shim artifact (pre-existing, see 3b); **PASS** when the shim is moved aside | + +## Findings from reviews (no action required) + +From be-0vbbs: no style or security findings against the diff. One narrow, +explicitly out-of-scope secondary observation noted (not a fix target): +`verifyProjectIdentity`'s GH#4637 safety net fails open when `beadsDir` is +empty — real `bd` CLI flows always populate it via `bd init`, so this is a +backward-compat gap, not an active production risk. + +## Verdict + +**PASS** — all 7 criteria (plus 3a/3b) clear. This fix extends the already +gated-and-open `deploy/be-pp7e-gate` branch / PR #5836 rather than cutting a +new isolated branch (see scope-deviation section). Per this repo's +contributor-only status, the job ends at the updated PR — no merge-request +will be routed to mayor and no deploy-clearance status will be posted; +merge authority belongs to the upstream maintainers. diff --git a/release-gates/be-pp7e-gate.md b/release-gates/be-pp7e-gate.md new file mode 100644 index 000000000..2d2973312 --- /dev/null +++ b/release-gates/be-pp7e-gate.md @@ -0,0 +1,99 @@ +# Release gate — PR-CI lane for `TestBenchDBPurgeDoesNotLeak` against a live Dolt server (be-aiy5) + +- **Builder bead:** be-aiy5 — add a PR-CI lane that runs + `TestBenchDBPurgeDoesNotLeak` against a live Dolt server instead of letting + `checkDolt` SKIP it. +- **Deploy bead:** be-pp7e +- **Review bead:** be-w90n — verdict **PASS**, recorded on commit + `7b7732178f4c5c8ef7dd6c787e139d8c869dc8d1` +- **Commits:** `6e0674c9c2808d53d1f1bac700688a06deff5c01` (tdd_red) then + `7b7732178f4c5c8ef7dd6c787e139d8c869dc8d1` (tdd_green), 2 commits over + `origin/main`, both citing `be-aiy5` +- **Branch:** `builder/be-aiy5` (provenance only, not a push target — pushed + to both `fork` and `headfork`, i.e. `quad341/beads-sec003-contrib`) +- **Evaluated:** 2026-08-18 by beads/deployer + +## Scope + +Adds a new PR-CI job to `.github/workflows/pr-risk.yml` that runs +`TestBenchDBPurgeDoesNotLeak` against a real, live Dolt server instead of +relying on the existing `checkDolt`-gated lane, which self-skips when no +live server is reachable. Diff scope, confirmed via +`git diff --stat origin/main...7b7732178f4c5c8ef7dd6c787e139d8c869dc8d1` +(4 files, 124 insertions, 6 deletions, no `.claude/**` paths): + +- `.github/workflows/pr-risk.yml` — new job, primary diff content +- `scripts/ci_workflow_test.go` — new meta-test asserting the new job's shape +- `scripts/ci_capability_selector_test.go` — modified expectations +- `scripts/pull_dolt_image_test.go` — modified expectations + +## Gate criteria + +| # | Criterion | Verdict | Evidence | +|---|-----------|---------|----------| +| 1 | Review PASS present | **PASS** | be-w90n records `verdict: pass`, `deploy_bead: be-pp7e`, `deploy_commit: 7b7732178f4c5c8ef7dd6c787e139d8c869dc8d1` — exact match. | +| 2 | Acceptance criteria met | **PASS** | be-w90n's `spec_findings` map all 4 of be-aiy5's exit-contract items (real PASS/FAIL not SKIP; gate not loosened; stretch fail-loud scoped via single `BEADS_TEST_ENV_RUN_DOLT` occurrence; scope contained to gastownhall/beads) with `uncovered_criteria: none`. | +| 3 | Tests pass | **PASS** | All 3 diff-owned tests independently re-run by deployer, real PASS (see below), matching reviewer's own record. | +| 3a | Pre-existing-failure attribution | **N/A — attributed** | Two unrelated failures hit while gating (see below); both confirmed pre-existing on `origin/main` and tracked (be-jy56, be-vf95 — the latter filed by this gate). Neither touches a diff file. | +| 3b | Policy/lint lane | **PASS** | `make ci-pr-policy`: build-tag policy and go-install-guidance sub-checks clean; the one failing sub-check (version-marker) is be-jy56, pre-existing on `origin/main`, unrelated. `make ci-pr-lint`: `gofmt` clean; repo-wide `golangci-lint` noise is be-vf95 (cross-worktree cache contamination + unrelated `backend/conformance` findings); diff-scoped `golangci-lint run ./scripts/...` reports **0 issues**. | +| 4 | No unresolved HIGH findings | **PASS** | be-w90n: zero HIGH/MEDIUM findings across style and security passes — "No blockers or majors found" / "No style issues found." All items INFO-level, none blocking. | +| 5 | Clean working tree | **PASS** | `git status` on the evaluated commit shows no staged/unstaged changes — only the pre-existing, unrelated untracked scratch files already present in this worktree (`release-gates/be-hi97-*.md`, `release-gates/be-k9js-*.md`, `release-gates/be-uoat-*.md`, `scripts/rebase-resolve-lib.sh`), never staged. | +| 6 | Clean divergence from `origin/main` | **PASS** | `origin/main` is already an ancestor of `7b7732178f4c5c8` — clean fast-forward relationship, zero rebase work required. `assert_deploy_ancestry_scope origin/main 7b7732178f4c5c8ef7dd6c787e139d8c869dc8d1 be-pp7e be-aiy5` → rc=0 (both commits cite `be-aiy5`, the accepted sibling builder bead). | +| 7 | Single feature theme | **PASS** | All 4 files serve the one CI-lane feature: the workflow job itself plus the 3 test files whose expectations that job's existence changes. No unrelated changes riding along. | + +## Tests run on release branch (independent re-verification) + +Environment: `DOCKER_HOST=unix:///run/user/1000/podman/podman.sock`, +`TESTCONTAINERS_RYUK_DISABLED=true`, per the test-evidence-integrity +protocol. + +Diff-owned tests, run by exact name (`go test ./scripts/... -run '' -v`): + +| Test | Result | +|---|---| +| `TestCICapabilitySelectorWorkflowPreservesExistingAuthority` | PASS | +| `TestPRRiskGateReachesFullServerDoltStorageSuite` | PASS | +| `TestDoltImagePullWorkflowsUseRetryHelper` | PASS | + +All 3/3 pass, matching the reviewer's own `diff_tests_executed` record. +Reviewer's own evidence (trusted, not re-run here given cost): focused +`go test ./scripts/... -v` — 60 PASS, 0 FAIL, 1 SKIP (non-diff-owned, +self-skipping-by-design `TestTestScriptPrebuiltBinaryLaunchProbe`, justified +in be-w90n's notes); full `TEST_COVER=1 make test` — 93 packages ok, 0 FAIL, +`internal/storage/dolt` itself ran and passed for real (confirming live +podman/Dolt infra, not silently bypassed). + +Static checks, independently re-run: + +| Check | Result | +|---|---| +| `make ci-pr-policy` | FAIL — but solely on be-jy56 (pre-existing, unrelated; see criterion 3a) | +| `make ci-pr-lint` | FAIL — but solely on be-vf95 (pre-existing/cross-worktree noise, unrelated; see criterion 3a) | +| `golangci-lint run ./scripts/...` (diff-scoped) | clean, 0 issues | +| `gofmt` (repo-wide, via `make ci-pr-lint`) | clean | + +## Findings from reviews (no action required) + +From be-w90n: no HIGH or MEDIUM findings, style or security. All items +INFO-level: gofmt/go vet/actionlint clean; workflow trigger is +`pull_request` (not `pull_request_target`), no secrets usage, no +PR-controlled context interpolated into new steps, pinned action SHAs +consistent with existing jobs, the new job's Dolt-install curl-pipe-sudo +step is a pre-existing idiom already used at 2 other call sites (not a new +risk), no new dependencies, touched Go test files only change expected +literals (no new exec/file-write surface). + +## Verdict + +**PASS** — all 7 criteria (plus 3a/3b) clear. Proceeding to cut the isolated +`deploy/be-pp7e-gate` branch from `7b7732178f4c5c8ef7dd6c787e139d8c869dc8d1` +and open the PR against `gastownhall/beads`. Per this repo's contributor-only +status, the job ends at the opened PR — no merge-request will be routed to +mayor and no deploy-clearance status will be posted; merge authority belongs +to the upstream maintainers. + +**Side note for the record (no action taken here):** be-pp7e's underlying +CI-lane fix is very likely the piece that be-vc1m's criterion 3 has been +waiting on (both concern `TestBenchDBPurgeDoesNotLeak`/live-Dolt-server +PR-CI coverage). be-vc1m remains parked on `hold:mayor`, blocked on +`be-w90n`, and is intentionally left untouched by this gate. diff --git a/scripts/ci_capability_selector_test.go b/scripts/ci_capability_selector_test.go index fb8f71fd0..d794ecf9a 100644 --- a/scripts/ci_capability_selector_test.go +++ b/scripts/ci_capability_selector_test.go @@ -96,7 +96,7 @@ func TestCICapabilitySelectorWorkflowPreservesExistingAuthority(t *testing.T) { if err := yaml.Unmarshal(data, &workflow); err != nil { t.Fatalf("parse workflow: %v", err) } - wantJobs := map[string]bool{"detect-ci-tier": true, "build-embedded": true, "test-embedded-storage": true, "test-embedded-conformance": true, "test-server-storage": true, "test-embedded-cmd": true, "test-proxied-cmd": true, "test-nix": true, "ci-gate": true} + wantJobs := map[string]bool{"detect-ci-tier": true, "build-embedded": true, "test-embedded-storage": true, "test-embedded-conformance": true, "test-server-storage": true, "test-server-storage-full": true, "test-embedded-cmd": true, "test-proxied-cmd": true, "test-nix": true, "ci-gate": true} if len(workflow.Jobs) != len(wantJobs) { t.Errorf("job count = %d, want %d", len(workflow.Jobs), len(wantJobs)) } @@ -118,7 +118,7 @@ func TestCICapabilitySelectorWorkflowPreservesExistingAuthority(t *testing.T) { if tier.ID != "tier" || shadow.ID != "shadow-selector" || !shadow.ContinueOnError || detector.stepIndex(t, shadow.Name) != detector.stepIndex(t, tier.Name)+1 { t.Errorf("shadow step is not immediately after tier: tier=%+v shadow=%+v", tier, shadow) } - for _, name := range []string{"build-embedded", "test-embedded-storage", "test-embedded-conformance", "test-server-storage", "test-embedded-cmd", "test-proxied-cmd"} { + for _, name := range []string{"build-embedded", "test-embedded-storage", "test-embedded-conformance", "test-server-storage", "test-server-storage-full", "test-embedded-cmd", "test-proxied-cmd"} { if got := workflow.Jobs[name].If; got != "needs.detect-ci-tier.outputs.full_embedded == 'true'" { t.Errorf("%s if = %q", name, got) } @@ -127,9 +127,9 @@ func TestCICapabilitySelectorWorkflowPreservesExistingAuthority(t *testing.T) { t.Errorf("test-nix if = %q, want ungated", workflow.Jobs["test-nix"].If) } gate := workflow.Jobs["ci-gate"] - wantNeeds := []string{"detect-ci-tier", "build-embedded", "test-embedded-storage", "test-embedded-conformance", "test-server-storage", "test-embedded-cmd", "test-proxied-cmd", "test-nix"} + wantNeeds := []string{"detect-ci-tier", "build-embedded", "test-embedded-storage", "test-embedded-conformance", "test-server-storage", "test-server-storage-full", "test-embedded-cmd", "test-proxied-cmd", "test-nix"} gateStep := gate.step(t, "Evaluate CI gate") - if gate.If != "${{ always() }}" || strings.Join(gate.Needs, ",") != strings.Join(wantNeeds, ",") || gateStep.Env["FULL_EMBEDDED"] != "${{ needs.detect-ci-tier.outputs.full_embedded }}" || gateStep.Env["CI_GATE_REQUIRED"] != "DETECT_CI_TIER BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_EMBEDDED_CMD TEST_PROXIED_CMD TEST_NIX" || !strings.Contains(gateStep.Run, "skipped_ok=\"BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_EMBEDDED_CMD TEST_PROXIED_CMD\"") { + if gate.If != "${{ always() }}" || strings.Join(gate.Needs, ",") != strings.Join(wantNeeds, ",") || gateStep.Env["FULL_EMBEDDED"] != "${{ needs.detect-ci-tier.outputs.full_embedded }}" || gateStep.Env["CI_GATE_REQUIRED"] != "DETECT_CI_TIER BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_SERVER_STORAGE_FULL TEST_EMBEDDED_CMD TEST_PROXIED_CMD TEST_NIX" || !strings.Contains(gateStep.Run, "skipped_ok=\"BUILD_EMBEDDED TEST_EMBEDDED_STORAGE TEST_EMBEDDED_CONFORMANCE TEST_SERVER_STORAGE TEST_SERVER_STORAGE_FULL TEST_EMBEDDED_CMD TEST_PROXIED_CMD\"") { t.Errorf("ci-gate authority changed: needs=%v env=%v", gate.Needs, gateStep.Env) } for name, job := range workflow.Jobs { diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 3c22c5f74..51e9f324e 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -4,6 +4,7 @@ import ( "fmt" "os" "path/filepath" + "reflect" "regexp" "strings" "testing" @@ -600,6 +601,167 @@ func TestGoCacheOwnershipTopology(t *testing.T) { } } +// TestPRRiskGateReachesFullServerDoltStorageSuite is the regression test for +// be-aiy5: test-server-storage already keeps a live Dolt server up (via +// build-embedded's /tmp/dolt-conformance-test binary, which compiles every +// top-level test in package internal/storage/dolt, not just conformance), +// but restricts execution to -test.run '^TestConformance$'. Every other +// server-gated test in that same binary -- TestCreateGuard_*, +// TestFederationPeerCredentialLifecycleLazyKeyInit, and diff-owned tests +// such as TestBenchDBPurgeDoesNotLeak on PR #5792 -- is reached by no +// PR-triggered lane, so it silently SKIPs instead of producing a real +// PASS/FAIL. A sibling job must run everything else in the same binary +// against the same live server, without a new build step, and must be wired +// into ci-gate as required so a SKIP there can no longer hide behind green. +func TestPRRiskGateReachesFullServerDoltStorageSuite(t *testing.T) { + const jobName = "test-server-storage-full" + + workflow := readCIWorkflow(t, "pr-risk.yml") + job := workflow.job(t, jobName) + + if job.RunsOn != "ubuntu-latest" { + t.Errorf("%s runs-on = %q, want ubuntu-latest", jobName, job.RunsOn) + } + if job.TimeoutMinutes != 20 { + t.Errorf("%s timeout = %d minutes, want 20 (matches test-server-storage)", jobName, job.TimeoutMinutes) + } + if !contains(job.Needs, "detect-ci-tier") || !contains(job.Needs, "build-embedded") { + t.Errorf("%s needs = %v, want detect-ci-tier and build-embedded (reuse the existing artifact, no new build)", jobName, job.Needs) + } + if job.If != "needs.detect-ci-tier.outputs.full_embedded == 'true'" { + t.Errorf("%s if = %q, want the same tier gate as test-server-storage", jobName, job.If) + } + + download := job.step(t, "Download binaries") + if download.With["name"] != "embedded-test-binaries" { + t.Errorf("%s does not download the existing embedded-test-binaries artifact (would require a new build step): %v", jobName, download.With) + } + + // Pro-rata against the embedded lane (75 shard-minutes / 324 tests): + // 1126 tests at the same per-test cost is ~260 shard-minutes; 16 shards + // x 15m = 240 shard-minutes, with the ~9.5-minute TestCloudAuthCLIRouting + // outlier isolated on its own shard via the shard manifest. + const totalShards = 16 + wantShards := make([]int, totalShards) + for i := range wantShards { + wantShards[i] = i + 1 + } + if !reflect.DeepEqual(job.Strategy.Matrix.Shard, wantShards) { + t.Errorf("%s strategy.matrix.shard = %v, want %v", jobName, job.Strategy.Matrix.Shard, wantShards) + } + if job.Strategy.FailFast { + t.Errorf("%s strategy.fail-fast = true, want false (one slow/flaky shard should not cancel the others)", jobName) + } + + wantTestCommand := fmt.Sprintf("bash .github/scripts/server-storage-test-shard.sh ${{ matrix.shard }} %d", totalShards) + assertStepRunsExactly(t, job, "Test", wantTestCommand) + // federation_test.go Fatals instead of silently skipping when this is set + // and the server it expects isn't reachable -- this job's whole point is + // a real PASS/FAIL, not a hidden self-skip if its own setup regresses. + assertStepEnvValue(t, job, "Test", "BEADS_TEST_ENV_RUN_DOLT", "1") + + // test-server-storage itself is untouched: conformance keeps its own + // dedicated job and timeout budget; this is an additive sibling, not a + // widened filter on the existing job. + existing := workflow.job(t, "test-server-storage") + assertStepRunsExactly(t, existing, "Test", `/tmp/dolt-conformance-test -test.v -test.count=1 -test.timeout=15m -test.run '^TestConformance$'`) + + gate := workflow.job(t, "ci-gate") + gateEnv := gate.step(t, "Evaluate CI gate").Env + const gateKey = "TEST_SERVER_STORAGE_FULL" + if !contains(gate.Needs, jobName) { + t.Errorf("ci-gate does not need %q: %v", jobName, gate.Needs) + } + if got, want := gateEnv[gateKey], fmt.Sprintf("${{ needs.%s.result }}", jobName); got != want { + t.Errorf("ci-gate env %s = %q, want %q", gateKey, got, want) + } + if !contains(strings.Fields(gateEnv["CI_GATE_REQUIRED"]), gateKey) { + t.Errorf("ci-gate CI_GATE_REQUIRED does not include %q", gateKey) + } +} + +func TestServerStorageShardScriptRunsPrebuiltBinaryFromPackageDir(t *testing.T) { + // pr4107_corruption_test.go and journal_scope_completeness_test.go use + // paths relative to internal/storage/dolt (e.g. ../schema/migrations, + // ../issueops). `go test` runs a package's tests with cwd = the package + // dir, so those resolve; a prebuilt test binary inherits the invoking + // shell's cwd instead. server-storage-test-shard.sh is invoked from the + // repo root (see TestPRRiskGateReachesFullServerDoltStorageSuite above), + // so the prebuilt-binary branch must cd into the package dir itself, + // immediately before exec -- any earlier and it breaks the repo-root- + // relative manifest/discovery above it; any later, or absent, and the 6 + // relative-path tests fail with "open ../issueops: no such file or + // directory". + path := filepath.Join(sourceRepoRoot(t), ".github", "scripts", "server-storage-test-shard.sh") + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + lines := strings.Split(string(data), "\n") + + indexOfContains := func(want string) int { + for i, line := range lines { + if strings.Contains(line, want) { + return i + } + } + return -1 + } + indexOfExact := func(want string) int { + for i, line := range lines { + if strings.TrimSpace(line) == want { + return i + } + } + return -1 + } + + discoveryIndex := indexOfContains(`grep -rh '^func Test' internal/storage/dolt/*_test.go`) + if discoveryIndex < 0 { + t.Fatal("could not find repo-root-relative test-discovery grep line") + } + prebuiltBranchIndex := indexOfContains(`if [ -x "$STORAGE_BINARY" ]; then`) + if prebuiltBranchIndex < 0 { + t.Fatal("could not find prebuilt-binary branch") + } + execIndex := indexOfContains(`exec "$STORAGE_BINARY"`) + if execIndex < 0 { + t.Fatal("could not find prebuilt-binary exec line") + } + fallbackExecIndex := indexOfContains(`exec go test`) + if fallbackExecIndex < 0 { + t.Fatal("could not find go-test fallback exec line") + } + if fallbackExecIndex < execIndex { + t.Fatal("go-test fallback exec line appears before the prebuilt-binary exec line -- branch order assumption violated") + } + + cdIndex := indexOfExact("cd internal/storage/dolt") + if cdIndex < 0 { + t.Fatal(`script does not "cd internal/storage/dolt" before running the prebuilt binary -- ` + + `relative-path tests (../schema/migrations, ../issueops) will fail when this script ` + + `is invoked from the repo root, as pr-risk.yml's test-server-storage-full job does`) + } + if cdIndex <= discoveryIndex { + t.Fatalf("cd internal/storage/dolt at line %d is at or before the repo-root-relative test "+ + "discovery grep at line %d -- that discovery must still run from the repo root", + cdIndex+1, discoveryIndex+1) + } + if cdIndex <= prebuiltBranchIndex || cdIndex >= execIndex { + t.Fatalf("cd internal/storage/dolt at line %d must sit strictly between the prebuilt-binary "+ + "branch at line %d and its exec at line %d", cdIndex+1, prebuiltBranchIndex+1, execIndex+1) + } + + // The go-test fallback already scopes via the ./internal/storage/dolt/ + // argument (cwd = repo root is fine for `go test`); it must not also cd. + for i := execIndex + 1; i <= fallbackExecIndex; i++ { + if strings.TrimSpace(lines[i]) == "cd internal/storage/dolt" { + t.Fatalf("unexpected cd internal/storage/dolt at line %d in the go-test fallback branch -- "+ + "it already scopes via the ./internal/storage/dolt/ argument", i+1) + } + } +} + func assertNoUnmanagedGoCacheSteps(t *testing.T, workflows map[string]ciWorkflow, managed map[string]map[string]bool) { t.Helper() @@ -926,11 +1088,13 @@ type ciWorkflowJob struct { } type ciWorkflowStrategy struct { - Matrix ciWorkflowMatrix `yaml:"matrix"` + FailFast bool `yaml:"fail-fast"` + Matrix ciWorkflowMatrix `yaml:"matrix"` } type ciWorkflowMatrix struct { OS []string `yaml:"os"` + Shard []int `yaml:"shard"` Include []ciWorkflowMatrixInclude `yaml:"include"` } diff --git a/scripts/pull_dolt_image_test.go b/scripts/pull_dolt_image_test.go index 651ebf3ab..eb17ce8d1 100644 --- a/scripts/pull_dolt_image_test.go +++ b/scripts/pull_dolt_image_test.go @@ -56,7 +56,7 @@ func TestDoltImagePullWorkflowsUseRetryHelper(t *testing.T) { wantCalls := map[string]int{ "main.yml": 2, "pr.yml": 2, - "pr-risk.yml": 2, + "pr-risk.yml": 3, "regression.yml": 1, }