beads/.github/scripts/server-storage-test-shard.sh
Jim Wordelman 30f30ca95b
ci: PR-CI lane runs the full internal/storage/dolt suite against a live Dolt server (#5836)
* test(ci): red — gastownhall/beads: no PR-CI lane runs TestBenchDBPurgeDoesNotLeak against a live Dolt server (checkDolt SKIP) (refs be-aiy5)

* feat(ci): green — gastownhall/beads: no PR-CI lane runs TestBenchDBPurgeDoesNotLeak against a live Dolt server (checkDolt SKIP) (refs be-aiy5)

Add test-server-storage-full as a sibling job to test-server-storage in
pr-risk.yml. build-embedded already compiles every top-level test in
internal/storage/dolt into one binary (embedded-test-binaries); the
existing job only ever runs -test.run '^TestConformance$' against it.
The new job reuses that same artifact and live-server setup with no new
build step, and runs everything else via -test.skip '^TestConformance$'
so previously-stranded tests (TestCreateGuard_*, federation/concurrent
suites, and diff-owned tests such as TestBenchDBPurgeDoesNotLeak on PR
#5792) get a real PASS/FAIL instead of a silent SKIP. Sets
BEADS_TEST_ENV_RUN_DOLT=1 so federation_test.go's own live-server check
fails loudly rather than skipping if this job's setup ever regresses.
Wired into ci-gate: added to needs, CI_GATE_REQUIRED, and the
tier-gated skipped_ok fallback (mirrors test-server-storage's existing
treatment in each).

Also updates two pre-existing meta-tests over pr-risk.yml that pinned
exact snapshots of the file this job necessarily changes:
scripts/ci_capability_selector_test.go's ci-gate authority snapshot,
and scripts/pull_dolt_image_test.go's retry-helper call count
(pr-risk.yml: 2 -> 3, the new job's own Dolt-image-pull step).

* chore: release gate PASS for be-pp7e (PR-CI lane for TestBenchDBPurgeDoesNotLeak)

Deploy gate record for be-pp7e, gating be-aiy5's PR-CI lane addition
(runs TestBenchDBPurgeDoesNotLeak against a live Dolt server instead of
letting checkDolt SKIP it). Review: be-w90n, verdict pass. All 7 gate
criteria PASS; two pre-existing/unrelated failures encountered during
evaluation (be-jy56, be-vf95) were attributed and excluded from the
verdict.

* test(ci): red — shard test-server-storage-full across 16 matrix jobs (refs be-yy25)

TestPRRiskGateReachesFullServerDoltStorageSuite now asserts
pr-risk.yml's test-server-storage-full job is a fail-fast:false matrix
over shard 1..16, with the Test step invoking
server-storage-test-shard.sh, instead of pinning the old single-job
unsharded command. Fails against the current workflow, which still
runs all 1126 tests unsharded in one 15m job (be-extn).

* feat(ci): green — shard test-server-storage-full across 16 matrix jobs (refs be-yy25)

Adds .github/scripts/server-storage-test-shard.sh (mirrors
embedded-storage-test-shard.sh: discovers internal/storage/dolt/*_test.go,
excludes TestConformance, manifest-plus-hash-fallback shard assignment) and
its shard manifest .github/scripts/server-storage-test-shards.txt, pinning
TestCloudAuthCLIRouting (~9.5m) alone on shard 1 and spreading the other
known 10-20s-heavy tests across the rest.

pr-risk.yml's test-server-storage-full job becomes a fail-fast:false matrix
over shard 1..16, Test step now runs server-storage-test-shard.sh instead of
the old unsharded single-job command that died at 256/1126 tests after
15m28s with zero failures (job 95616282508).

Documents the new lane in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md alongside the
existing embedded matrix description.

TestPRRiskGateReachesFullServerDoltStorageSuite now passes. Offline shard
partition verified disjoint and complete across all 1125 discovered tests
(BEADS_TEST_SHARD_LIST_ONLY=1 for i=1..16). (be-extn)

* test(feat): red — Fix: sharded lane runs prebuilt test binary from repo root, breaking ../schema/migrations and ../issueops (6 tests) (refs be-gepv)

* feat: green — Fix: sharded lane runs prebuilt test binary from repo root, breaking ../schema/migrations and ../issueops (6 tests) (refs be-gepv)

* chore: release gate PASS for be-23f3

Extends deploy/be-pp7e-gate (PR gastownhall/beads#5836) with be-gepv's
fix rather than cutting a fresh isolated branch: be-gepv's own bead
text and be-extn's precedent both establish this fix belongs on the
already-open PR. All 7 gate criteria PASS; two repo-wide lint/policy
failures attributed to already-tracked pre-existing issues (be-vf95,
be-jy56), confirmed untouched by this PR lineage.

be-gepv, be-6yo1, be-23f3

* fix(dolt): federation test isolation + panic (be-3c78s)

TestFederationDatabaseIsolation and TestFederationVersionControlAPIs are
federation-prototype tests that predate the suite's shared-server,
branch-per-test isolation architecture (setupTestStore/StartTestBranch)
and were never updated to account for it.

- setupFederationStore hardcoded Database: "beads" for both towns. New()
  always connects in server mode against one shared Dolt server for the
  whole test binary (TestMain), so Path alone doesn't isolate two stores
  — Database does. Both towns silently shared one database, so Alpha
  could read Beta's issues and vice versa. Give each town its own
  database (beads_test_federation_<prefix>, matching the beads_test AD-01
  cleanup-firewall prefix).

- TestFederationDatabaseIsolation's not-found assertions assumed GetIssue
  returns (nil, nil) on a miss. It never does — issueops.GetIssueInTx
  always returns a wrapped storage.ErrNotFound. Switched to
  errors.Is(err, storage.ErrNotFound).

- TestFederationVersionControlAPIs hardcoded Checkout(ctx, "main") to
  return to its "starting" branch, but setupTestStore already checks out
  a private branch (test-<name>-<hex>) — "main" is a branch shared by
  every parallel test in the binary and never has vc-001. That mismatch
  is what produced the nil-pointer panic (GetIssue's error was discarded
  via `_`, and the resulting nil issue was dereferenced). Capture the
  real starting branch via CurrentBranch() right after setupTestStore()
  and check both previously-discarded GetIssue errors before
  dereferencing.

Neither bug is a production security issue: New(), Checkout(), and
GetIssue() all behave exactly as specified. Both bugs are confined to
this prototype test file's own setup/assertions, which hadn't caught up
to the isolation model the rest of the suite already uses correctly.

Verified via `go build`/`go vet`/`go test -c` (all clean) and running the
compiled binary, which now reaches and cleanly SKIPs both tests (no
Docker locally) instead of the previous isolation failure / panic
surfacing at compile-adjacent runtime. Full GREEN confirmation is via
CI, which has a working Dolt/Docker environment.

Refs be-3c78s

* chore: release gate PASS for be-42tlj
2026-08-19 20:39:38 -07:00

160 lines
6 KiB
Bash
Executable file

#!/usr/bin/env bash
# server-storage-test-shard.sh — run a shard of server dolt storage tests.
#
# Usage: server-storage-test-shard.sh <shard_number> <total_shards>
#
# Discovers all top-level Test* functions from internal/storage/dolt/*_test.go,
# excluding TestConformance. That suite runs in the dedicated
# test-server-storage job instead, sharded by sub-test path (-test.run
# '^TestConformance$'), not by top-level function name; this manifest's
# exact-name matching does not cover it. See
# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md for why the two use different
# sharding mechanisms.
#
# Tests listed in server-storage-test-shards.txt for the requested shard count
# use that committed assignment; newly-added tests fall back to
# hash(name) % total. Runs the matching subset using the pre-built test
# binary at BEADS_TEST_SERVER_TEST_BINARY (or /tmp/dolt-conformance-test).
#
# This package hands 1126 top-level tests to a single job (be-extn): the
# sibling embedded lane's test-embedded-storage shards 324 tests over 5 jobs.
# Server mode is also slower per-test (real socket round-trips, per-test
# CREATE/DROP DATABASE) than embedded. TestCloudAuthCLIRouting alone took 566s
# (63% of the old 15m budget) across 16 subtests, each paying a full
# CREATE DATABASE + migration chain + dolt init + dolt remote add + DROP
# DATABASE (~35s) to test one routing predicate; it uses t.Setenv, so it can
# never be parallel. The manifest pins it (and other known-heavy tests) to
# distinct shards so they do not cluster.
#
# Environment:
# BEADS_TEST_ENV_RUN_DOLT=1 required (tests skip without it)
# BEADS_TEST_SERVER_TEST_BINARY=<path> pre-built storage test binary
# (default: /tmp/dolt-conformance-test)
# BEADS_TEST_SHARD_LIST_ONLY=1 print selected tests without running them
set -euo pipefail
SHARD_NUMBER="${1:?usage: $0 <shard_number> <total_shards>}"
TOTAL_SHARDS="${2:?usage: $0 <shard_number> <total_shards>}"
shift 2
if ! [[ "$SHARD_NUMBER" =~ ^[0-9]+$ ]] || ! [[ "$TOTAL_SHARDS" =~ ^[0-9]+$ ]]; then
echo "Shard number and total shards must be positive integers" >&2
exit 1
fi
if (( TOTAL_SHARDS < 1 || SHARD_NUMBER < 1 || SHARD_NUMBER > TOTAL_SHARDS )); then
echo "Invalid shard ${SHARD_NUMBER}/${TOTAL_SHARDS}" >&2
exit 1
fi
SHARD_INDEX=$(( SHARD_NUMBER - 1 ))
MANIFEST="${BEADS_TEST_SHARD_MANIFEST:-.github/scripts/server-storage-test-shards.txt}"
# Discover all top-level Test* functions except TestConformance, which is
# sharded separately (test-server-storage job, -test.run '^TestConformance$')
# because it is a single top-level function with sub-tests reached via
# -test.run path regex, not distinct top-level function names.
ALL_TESTS=$(grep -rh '^func Test' internal/storage/dolt/*_test.go \
| sed 's/func \(Test[A-Za-z0-9_]*\).*/\1/' \
| grep -v '^TestConformance$' \
| sort -u)
if [ -z "$ALL_TESTS" ]; then
echo "No Test* functions found" >&2
exit 1
fi
declare -A ALL_TEST_SET=()
while IFS= read -r name; do
ALL_TEST_SET["$name"]=1
done <<< "$ALL_TESTS"
declare -A MANIFEST_SHARDS=()
if [ -f "$MANIFEST" ]; then
while IFS= read -r line || [ -n "$line" ]; do
line="${line%%#*}"
read -r manifest_total manifest_shard test_name extra <<< "$line"
if [ -z "${manifest_total:-}" ]; then
continue
fi
if [ -n "${extra:-}" ] || ! [[ "$manifest_total" =~ ^[0-9]+$ ]] || ! [[ "$manifest_shard" =~ ^[0-9]+$ ]] || ! [[ "$test_name" =~ ^Test[A-Za-z0-9_]+$ ]] || [ "$test_name" = "TestConformance" ]; then
echo "Invalid shard manifest line: $line" >&2
exit 1
fi
if (( manifest_total != TOTAL_SHARDS )); then
continue
fi
if (( manifest_shard < 1 || manifest_shard > TOTAL_SHARDS )); then
echo "Invalid shard ${manifest_shard}/${manifest_total} for $test_name in $MANIFEST" >&2
exit 1
fi
if [ -n "${MANIFEST_SHARDS[$test_name]:-}" ]; then
echo "Duplicate shard manifest entry for $test_name in $MANIFEST" >&2
exit 1
fi
MANIFEST_SHARDS["$test_name"]="$manifest_shard"
done < "$MANIFEST"
fi
for name in "${!MANIFEST_SHARDS[@]}"; do
if [ -z "${ALL_TEST_SET[$name]:-}" ]; then
echo "Shard manifest entry does not match a discovered test: $name" >&2
exit 1
fi
done
SHARD_TESTS=()
MANIFEST_COUNT=0
FALLBACK_COUNT=0
while IFS= read -r name; do
assigned_shard="${MANIFEST_SHARDS[$name]:-}"
if [ -n "$assigned_shard" ]; then
if (( assigned_shard == SHARD_NUMBER )); then
SHARD_TESTS+=("$name")
MANIFEST_COUNT=$(( MANIFEST_COUNT + 1 ))
fi
continue
fi
# Use cksum for a portable numeric hash fallback for newly-added tests.
hash=$(printf %s "$name" | cksum | cut -d ' ' -f 1)
if (( hash % TOTAL_SHARDS == SHARD_INDEX )); then
SHARD_TESTS+=("$name")
FALLBACK_COUNT=$(( FALLBACK_COUNT + 1 ))
fi
done <<< "$ALL_TESTS"
if [ ${#SHARD_TESTS[@]} -eq 0 ]; then
echo "Shard ${SHARD_NUMBER}/${TOTAL_SHARDS}: no tests assigned (all hashed to other shards)"
exit 0
fi
# Build the -run regex: "^(TestA|TestB|TestC)$"
RUN_REGEX="^($(IFS='|'; echo "${SHARD_TESTS[*]}"))$"
echo "Shard ${SHARD_NUMBER}/${TOTAL_SHARDS}: running ${#SHARD_TESTS[@]} test(s)"
echo " manifest: ${MANIFEST_COUNT}, fallback: ${FALLBACK_COUNT}"
printf " %s\n" "${SHARD_TESTS[@]}"
echo ""
# Use pre-built test binary if available, otherwise fall back to go test.
STORAGE_BINARY="${BEADS_TEST_SERVER_TEST_BINARY:-/tmp/dolt-conformance-test}"
if [ "${BEADS_TEST_SHARD_LIST_ONLY:-}" = "1" ]; then
exit 0
fi
if [ -x "$STORAGE_BINARY" ]; then
# The tests use paths relative to their own package dir (e.g.
# ../schema/migrations, ../issueops). `go test` runs them with cwd = the
# package dir; a prebuilt binary inherits OUR cwd, so set it explicitly.
cd internal/storage/dolt
exec "$STORAGE_BINARY" -test.v -test.count=1 -test.timeout=15m \
-test.run "$RUN_REGEX" \
"$@"
else
echo "Warning: pre-built test binary not found at $STORAGE_BINARY, falling back to go test"
exec go test -tags=integration,gms_pure_go -v -count=1 -timeout 15m \
-run "$RUN_REGEX" \
"$@" \
./internal/storage/dolt/
fi