* test(ci): red — gastownhall/beads: no PR-CI lane runs TestBenchDBPurgeDoesNotLeak against a live Dolt server (checkDolt SKIP) (refs be-aiy5) * feat(ci): green — gastownhall/beads: no PR-CI lane runs TestBenchDBPurgeDoesNotLeak against a live Dolt server (checkDolt SKIP) (refs be-aiy5) Add test-server-storage-full as a sibling job to test-server-storage in pr-risk.yml. build-embedded already compiles every top-level test in internal/storage/dolt into one binary (embedded-test-binaries); the existing job only ever runs -test.run '^TestConformance$' against it. The new job reuses that same artifact and live-server setup with no new build step, and runs everything else via -test.skip '^TestConformance$' so previously-stranded tests (TestCreateGuard_*, federation/concurrent suites, and diff-owned tests such as TestBenchDBPurgeDoesNotLeak on PR #5792) get a real PASS/FAIL instead of a silent SKIP. Sets BEADS_TEST_ENV_RUN_DOLT=1 so federation_test.go's own live-server check fails loudly rather than skipping if this job's setup ever regresses. Wired into ci-gate: added to needs, CI_GATE_REQUIRED, and the tier-gated skipped_ok fallback (mirrors test-server-storage's existing treatment in each). Also updates two pre-existing meta-tests over pr-risk.yml that pinned exact snapshots of the file this job necessarily changes: scripts/ci_capability_selector_test.go's ci-gate authority snapshot, and scripts/pull_dolt_image_test.go's retry-helper call count (pr-risk.yml: 2 -> 3, the new job's own Dolt-image-pull step). * chore: release gate PASS for be-pp7e (PR-CI lane for TestBenchDBPurgeDoesNotLeak) Deploy gate record for be-pp7e, gating be-aiy5's PR-CI lane addition (runs TestBenchDBPurgeDoesNotLeak against a live Dolt server instead of letting checkDolt SKIP it). Review: be-w90n, verdict pass. All 7 gate criteria PASS; two pre-existing/unrelated failures encountered during evaluation (be-jy56, be-vf95) were attributed and excluded from the verdict. * test(ci): red — shard test-server-storage-full across 16 matrix jobs (refs be-yy25) TestPRRiskGateReachesFullServerDoltStorageSuite now asserts pr-risk.yml's test-server-storage-full job is a fail-fast:false matrix over shard 1..16, with the Test step invoking server-storage-test-shard.sh, instead of pinning the old single-job unsharded command. Fails against the current workflow, which still runs all 1126 tests unsharded in one 15m job (be-extn). * feat(ci): green — shard test-server-storage-full across 16 matrix jobs (refs be-yy25) Adds .github/scripts/server-storage-test-shard.sh (mirrors embedded-storage-test-shard.sh: discovers internal/storage/dolt/*_test.go, excludes TestConformance, manifest-plus-hash-fallback shard assignment) and its shard manifest .github/scripts/server-storage-test-shards.txt, pinning TestCloudAuthCLIRouting (~9.5m) alone on shard 1 and spreading the other known 10-20s-heavy tests across the rest. pr-risk.yml's test-server-storage-full job becomes a fail-fast:false matrix over shard 1..16, Test step now runs server-storage-test-shard.sh instead of the old unsharded single-job command that died at 256/1126 tests after 15m28s with zero failures (job 95616282508). Documents the new lane in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md alongside the existing embedded matrix description. TestPRRiskGateReachesFullServerDoltStorageSuite now passes. Offline shard partition verified disjoint and complete across all 1125 discovered tests (BEADS_TEST_SHARD_LIST_ONLY=1 for i=1..16). (be-extn) * test(feat): red — Fix: sharded lane runs prebuilt test binary from repo root, breaking ../schema/migrations and ../issueops (6 tests) (refs be-gepv) * feat: green — Fix: sharded lane runs prebuilt test binary from repo root, breaking ../schema/migrations and ../issueops (6 tests) (refs be-gepv) * chore: release gate PASS for be-23f3 Extends deploy/be-pp7e-gate (PR gastownhall/beads#5836) with be-gepv's fix rather than cutting a fresh isolated branch: be-gepv's own bead text and be-extn's precedent both establish this fix belongs on the already-open PR. All 7 gate criteria PASS; two repo-wide lint/policy failures attributed to already-tracked pre-existing issues (be-vf95, be-jy56), confirmed untouched by this PR lineage. be-gepv, be-6yo1, be-23f3 * fix(dolt): federation test isolation + panic (be-3c78s) TestFederationDatabaseIsolation and TestFederationVersionControlAPIs are federation-prototype tests that predate the suite's shared-server, branch-per-test isolation architecture (setupTestStore/StartTestBranch) and were never updated to account for it. - setupFederationStore hardcoded Database: "beads" for both towns. New() always connects in server mode against one shared Dolt server for the whole test binary (TestMain), so Path alone doesn't isolate two stores — Database does. Both towns silently shared one database, so Alpha could read Beta's issues and vice versa. Give each town its own database (beads_test_federation_<prefix>, matching the beads_test AD-01 cleanup-firewall prefix). - TestFederationDatabaseIsolation's not-found assertions assumed GetIssue returns (nil, nil) on a miss. It never does — issueops.GetIssueInTx always returns a wrapped storage.ErrNotFound. Switched to errors.Is(err, storage.ErrNotFound). - TestFederationVersionControlAPIs hardcoded Checkout(ctx, "main") to return to its "starting" branch, but setupTestStore already checks out a private branch (test-<name>-<hex>) — "main" is a branch shared by every parallel test in the binary and never has vc-001. That mismatch is what produced the nil-pointer panic (GetIssue's error was discarded via `_`, and the resulting nil issue was dereferenced). Capture the real starting branch via CurrentBranch() right after setupTestStore() and check both previously-discarded GetIssue errors before dereferencing. Neither bug is a production security issue: New(), Checkout(), and GetIssue() all behave exactly as specified. Both bugs are confined to this prototype test file's own setup/assertions, which hadn't caught up to the isolation model the rest of the suite already uses correctly. Verified via `go build`/`go vet`/`go test -c` (all clean) and running the compiled binary, which now reaches and cleanly SKIPs both tests (no Docker locally) instead of the previous isolation failure / panic surfacing at compile-adjacent runtime. Full GREEN confirmation is via CI, which has a working Dolt/Docker environment. Refs be-3c78s * chore: release gate PASS for be-42tlj
160 lines
6 KiB
Bash
Executable file
160 lines
6 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# server-storage-test-shard.sh — run a shard of server dolt storage tests.
|
|
#
|
|
# Usage: server-storage-test-shard.sh <shard_number> <total_shards>
|
|
#
|
|
# Discovers all top-level Test* functions from internal/storage/dolt/*_test.go,
|
|
# excluding TestConformance. That suite runs in the dedicated
|
|
# test-server-storage job instead, sharded by sub-test path (-test.run
|
|
# '^TestConformance$'), not by top-level function name; this manifest's
|
|
# exact-name matching does not cover it. See
|
|
# engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md for why the two use different
|
|
# sharding mechanisms.
|
|
#
|
|
# Tests listed in server-storage-test-shards.txt for the requested shard count
|
|
# use that committed assignment; newly-added tests fall back to
|
|
# hash(name) % total. Runs the matching subset using the pre-built test
|
|
# binary at BEADS_TEST_SERVER_TEST_BINARY (or /tmp/dolt-conformance-test).
|
|
#
|
|
# This package hands 1126 top-level tests to a single job (be-extn): the
|
|
# sibling embedded lane's test-embedded-storage shards 324 tests over 5 jobs.
|
|
# Server mode is also slower per-test (real socket round-trips, per-test
|
|
# CREATE/DROP DATABASE) than embedded. TestCloudAuthCLIRouting alone took 566s
|
|
# (63% of the old 15m budget) across 16 subtests, each paying a full
|
|
# CREATE DATABASE + migration chain + dolt init + dolt remote add + DROP
|
|
# DATABASE (~35s) to test one routing predicate; it uses t.Setenv, so it can
|
|
# never be parallel. The manifest pins it (and other known-heavy tests) to
|
|
# distinct shards so they do not cluster.
|
|
#
|
|
# Environment:
|
|
# BEADS_TEST_ENV_RUN_DOLT=1 required (tests skip without it)
|
|
# BEADS_TEST_SERVER_TEST_BINARY=<path> pre-built storage test binary
|
|
# (default: /tmp/dolt-conformance-test)
|
|
# BEADS_TEST_SHARD_LIST_ONLY=1 print selected tests without running them
|
|
|
|
set -euo pipefail
|
|
|
|
SHARD_NUMBER="${1:?usage: $0 <shard_number> <total_shards>}"
|
|
TOTAL_SHARDS="${2:?usage: $0 <shard_number> <total_shards>}"
|
|
shift 2
|
|
|
|
if ! [[ "$SHARD_NUMBER" =~ ^[0-9]+$ ]] || ! [[ "$TOTAL_SHARDS" =~ ^[0-9]+$ ]]; then
|
|
echo "Shard number and total shards must be positive integers" >&2
|
|
exit 1
|
|
fi
|
|
if (( TOTAL_SHARDS < 1 || SHARD_NUMBER < 1 || SHARD_NUMBER > TOTAL_SHARDS )); then
|
|
echo "Invalid shard ${SHARD_NUMBER}/${TOTAL_SHARDS}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
SHARD_INDEX=$(( SHARD_NUMBER - 1 ))
|
|
MANIFEST="${BEADS_TEST_SHARD_MANIFEST:-.github/scripts/server-storage-test-shards.txt}"
|
|
|
|
# Discover all top-level Test* functions except TestConformance, which is
|
|
# sharded separately (test-server-storage job, -test.run '^TestConformance$')
|
|
# because it is a single top-level function with sub-tests reached via
|
|
# -test.run path regex, not distinct top-level function names.
|
|
ALL_TESTS=$(grep -rh '^func Test' internal/storage/dolt/*_test.go \
|
|
| sed 's/func \(Test[A-Za-z0-9_]*\).*/\1/' \
|
|
| grep -v '^TestConformance$' \
|
|
| sort -u)
|
|
|
|
if [ -z "$ALL_TESTS" ]; then
|
|
echo "No Test* functions found" >&2
|
|
exit 1
|
|
fi
|
|
|
|
declare -A ALL_TEST_SET=()
|
|
while IFS= read -r name; do
|
|
ALL_TEST_SET["$name"]=1
|
|
done <<< "$ALL_TESTS"
|
|
|
|
declare -A MANIFEST_SHARDS=()
|
|
if [ -f "$MANIFEST" ]; then
|
|
while IFS= read -r line || [ -n "$line" ]; do
|
|
line="${line%%#*}"
|
|
read -r manifest_total manifest_shard test_name extra <<< "$line"
|
|
if [ -z "${manifest_total:-}" ]; then
|
|
continue
|
|
fi
|
|
if [ -n "${extra:-}" ] || ! [[ "$manifest_total" =~ ^[0-9]+$ ]] || ! [[ "$manifest_shard" =~ ^[0-9]+$ ]] || ! [[ "$test_name" =~ ^Test[A-Za-z0-9_]+$ ]] || [ "$test_name" = "TestConformance" ]; then
|
|
echo "Invalid shard manifest line: $line" >&2
|
|
exit 1
|
|
fi
|
|
if (( manifest_total != TOTAL_SHARDS )); then
|
|
continue
|
|
fi
|
|
if (( manifest_shard < 1 || manifest_shard > TOTAL_SHARDS )); then
|
|
echo "Invalid shard ${manifest_shard}/${manifest_total} for $test_name in $MANIFEST" >&2
|
|
exit 1
|
|
fi
|
|
if [ -n "${MANIFEST_SHARDS[$test_name]:-}" ]; then
|
|
echo "Duplicate shard manifest entry for $test_name in $MANIFEST" >&2
|
|
exit 1
|
|
fi
|
|
MANIFEST_SHARDS["$test_name"]="$manifest_shard"
|
|
done < "$MANIFEST"
|
|
fi
|
|
|
|
for name in "${!MANIFEST_SHARDS[@]}"; do
|
|
if [ -z "${ALL_TEST_SET[$name]:-}" ]; then
|
|
echo "Shard manifest entry does not match a discovered test: $name" >&2
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
SHARD_TESTS=()
|
|
MANIFEST_COUNT=0
|
|
FALLBACK_COUNT=0
|
|
while IFS= read -r name; do
|
|
assigned_shard="${MANIFEST_SHARDS[$name]:-}"
|
|
if [ -n "$assigned_shard" ]; then
|
|
if (( assigned_shard == SHARD_NUMBER )); then
|
|
SHARD_TESTS+=("$name")
|
|
MANIFEST_COUNT=$(( MANIFEST_COUNT + 1 ))
|
|
fi
|
|
continue
|
|
fi
|
|
|
|
# Use cksum for a portable numeric hash fallback for newly-added tests.
|
|
hash=$(printf %s "$name" | cksum | cut -d ' ' -f 1)
|
|
if (( hash % TOTAL_SHARDS == SHARD_INDEX )); then
|
|
SHARD_TESTS+=("$name")
|
|
FALLBACK_COUNT=$(( FALLBACK_COUNT + 1 ))
|
|
fi
|
|
done <<< "$ALL_TESTS"
|
|
|
|
if [ ${#SHARD_TESTS[@]} -eq 0 ]; then
|
|
echo "Shard ${SHARD_NUMBER}/${TOTAL_SHARDS}: no tests assigned (all hashed to other shards)"
|
|
exit 0
|
|
fi
|
|
|
|
# Build the -run regex: "^(TestA|TestB|TestC)$"
|
|
RUN_REGEX="^($(IFS='|'; echo "${SHARD_TESTS[*]}"))$"
|
|
|
|
echo "Shard ${SHARD_NUMBER}/${TOTAL_SHARDS}: running ${#SHARD_TESTS[@]} test(s)"
|
|
echo " manifest: ${MANIFEST_COUNT}, fallback: ${FALLBACK_COUNT}"
|
|
printf " %s\n" "${SHARD_TESTS[@]}"
|
|
echo ""
|
|
|
|
# Use pre-built test binary if available, otherwise fall back to go test.
|
|
STORAGE_BINARY="${BEADS_TEST_SERVER_TEST_BINARY:-/tmp/dolt-conformance-test}"
|
|
if [ "${BEADS_TEST_SHARD_LIST_ONLY:-}" = "1" ]; then
|
|
exit 0
|
|
fi
|
|
|
|
if [ -x "$STORAGE_BINARY" ]; then
|
|
# The tests use paths relative to their own package dir (e.g.
|
|
# ../schema/migrations, ../issueops). `go test` runs them with cwd = the
|
|
# package dir; a prebuilt binary inherits OUR cwd, so set it explicitly.
|
|
cd internal/storage/dolt
|
|
exec "$STORAGE_BINARY" -test.v -test.count=1 -test.timeout=15m \
|
|
-test.run "$RUN_REGEX" \
|
|
"$@"
|
|
else
|
|
echo "Warning: pre-built test binary not found at $STORAGE_BINARY, falling back to go test"
|
|
exec go test -tags=integration,gms_pure_go -v -count=1 -timeout 15m \
|
|
-run "$RUN_REGEX" \
|
|
"$@" \
|
|
./internal/storage/dolt/
|
|
fi
|