* test(dolt): red — testdb_* leak onto production Dolt port when Docker absent (refs be-h2h25) * feat: green — Test suites create testdb_* on the PRODUCTION dolt server: harness fails open on the ambient port when Docker is absent (refs be-h2h25) * docs(release-gate): add be-napos gate — ambient Dolt port fail-open to fail-closed Deploy bead be-napos (build be-h2h25, review be-7fym1, verdict pass round 2). 7/7 gate criteria PASS; full independent evidence in the gate doc. refs be-napos, be-h2h25, be-7fym1 * docs(release-gate): record PR link for be-napos gate refs be-napos * test(testutil): make the fail-closed gate deterministic, not Docker-conditional The regression test skipped itself on every host that has Docker, by calling the real EnsureDoltContainerForTestMain and skipping when it succeeded. That asserted nothing on CI -- the only lane where it matters -- and on a Docker-having host it also started the shared singleton container purely to decide to skip, with no TestMain in internal/testutil to terminate it. Add checkDoltFn, a package-level seam over checkDolt, so the test can force a not-ready state without a Docker daemon and without starting a container. Only EnsureDoltContainerForTestMain routes through it; the neighbouring t.Skip call sites gate a skip rather than the fail-closed behaviour, so they keep calling checkDolt directly. The gate now covers all four not-ready states rather than the one the old test happened to hit, which also pins the intent that an explicit BEADS_TEST_SKIP=dolt opt-out must fail closed exactly as a Docker-less host does. Split into a platform-independent half (the helper's own contract, both variables cleared, idempotent) and a !windows half (the wiring, via the seam). Red/green: with the neutralizeAmbientDoltPort call removed from the not-ready branch, 4 of 4 subtests fail and 0 pass; restored, 4 of 4 pass in 0.06s with no Docker and no container. (refs be-napos) * fix(testutil): clear the ambient Dolt port on the Windows stub too The Windows EnsureDoltContainerForTestMain returned its error without touching BEADS_DOLT_SERVER_PORT / BEADS_DOLT_PORT, so the fail-open this branch fixes still shipped there -- unconditionally, since that stub never succeeds. Windows is a real lane: .github/workflows/pr.yml runs windows-latest on four jobs plus the cross-platform matrix. Gating it needed one pre-existing repair. testdoltserver_env_test.go has no build tag but calls checkDolt, which is !windows-only, so the internal/testutil test binary has never compiled for Windows -- reproduced identically on origin/main. Tagging that file !windows fixes the whole package's Windows test lane; without it a new Windows gate would compile nowhere and assert nothing, which is the same defect this branch is already correcting. Verified GOOS=windows go vet ./internal/testutil/ clean (it fails on origin/main), and GOOS=windows go build clean before and after. The Windows gate itself is compile-verified only -- no Windows host here to execute it. (refs be-napos) * docs(testing): state that ambient BEADS_DOLT_*_PORT is never honored The guard fires on every not-ready state, not only "Docker unavailable" -- a missing or wrong-version image and an explicit BEADS_TEST_SKIP=dolt opt-out clear the ambient port too. That is deliberate: the invariant is the absence of a test container, not the reason for it. Say so where the next person will look before investigating, and name the supported way to target a specific server. (refs be-napos) * chore: remove leaked internal release-gate doc from PR branch release-gates/be-napos-ambient-dolt-port-failclosed-gate.md Tracked our internal gc-management/beads-rig release-gate checklist and should never have been committed to this upstream branch -- it's process residue from our deploy tooling, not part of the change itself. Per maintainer bee-ghosttrack's request during review of PR #6571. The root cause (our deploy tooling auto-committing this file) is already fixed as of gm-qatxl1; this branch predates that fix, so this commit is cleanup only -- no other content on this branch is touched. Ref: be-lk7o3 * chore(bazel): enumerate the PR's new test files in BUILD.bazel Base added BUILD.bazel to internal/testutil and internal/storage/dolt after this PR was opened (#6881 made the Bazel lane real, and fork PRs always run it). The four new test files this PR adds were therefore absent from the go_test srcs lists, which the bazel lane's equivalence check reports as a missing target. Generated by 'make bazel-sync' (gazelle), not hand-edited, because the CI drift step diffs against exactly what gazelle emits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(testutil): reconcile the shared-server harness with the fail-closed clear Review iterations 1-2 (adopt-pr, PR #5885): every gating finding from both review syntheses, plus their minors and nits. Iteration 1, Major 1 (Change Impact / Blast Radius). scripts/test.sh's BEADS_TEST_SHARED_SERVER=1 flow starts a dockerless `dolt sql-server` and exports BEADS_DOLT_PORT for every package to reuse. checkDolt() has no notion of an already-provisioned shared server, so on that flow's primary audience -- a host without Docker -- all 12 EnsureDoltContainerForTestMain callers erased the harness's own sanctioned port before m.Run(), turning green dockerless runs red, while engdocs/CI_TEST_SURFACE_AUDIT.md and the new TESTING.md paragraph advertised the two mechanisms unreconciled. Taken: a dedicated marker, BEADS_TEST_SHARED_DOLT_SERVER, exported by scripts/test.sh beside the port it allocated, which the helper's failure paths honor. Both docs state the reconciliation, including the precedence: a container, where one can be started, still overwrites both variables. Not taken: "skip the clear when BEADS_TEST_SHARED_SERVER=1 and a port var is already set". That flag is overloaded (cmd/bd/shared_server_integration_test.go reads it too), and the script starts no server when a port is already set, so the flag beside a port is exactly the case where the port is ambient. Iteration 2, blocker (Behavioral Correctness). Iteration 1 shipped that marker as a boolean ("1") which spared BOTH port variables, while the script vouched only for the BEADS_DOLT_PORT it exported and its entry guard never looked at BEADS_DOLT_SERVER_PORT -- the variable applyConfigDefaults resolves first. The runner's test environment (beads_test_env_enter) unsets both variables before the flow runs, so the hole needed that isolation skipped, e.g. with the documented BEADS_TEST_ENV_DISABLE=1; then an inherited BEADS_DOLT_SERVER_PORT reached `go test` beside the marker and outranked the shared server -- gm-2g3g5r, re-opened by its own fix. Measured with a fake `go` that prints its environment: SERVER_PORT=59999 PORT=<shared> MARKER=1 under the iteration-1 script. Now: - the marker's value is the port it vouches for, and the helper decides per variable (neutralizeUnvouchedDoltPorts): only a variable holding exactly that port survives, and says so on stderr; every other one is cleared, marker or not; - the flow starts only when neither port variable is set, and says on stderr when BEADS_TEST_SHARED_SERVER=1 is ignored for that reason. Not taken: also exporting BEADS_DOLT_SERVER_PORT from the flow. The widened guard means it is always unset there, so the export adds no safety, and five sites read that variable alone (internal/doltserver/doltserver.go:796, cmd/bd/init.go:773 and :1351, cmd/bd/bootstrap.go:857, internal/testutil/testdoltbranch.go:164) -- consumers the flow has never fed. Iteration 2, minor (Test Evidence Quality). The exception's second arm, a container that would not start, was reached by no test with its guard live. Both failure paths now share one clear, and the container start is a seam (startSharedContainerFn, pinned to startSharedContainer by identity) so tests reach that path without consuming doltServerOnce. TestEnsureDoltContainerForTestMain_KeepsHarnessProvisionedPort runs its whole table, marked_but_divergent_var included, through both paths. Iteration 2, minor (Contract & Interface Fidelity). The marker prose in TESTING.md, CI_TEST_SURFACE_AUDIT.md, test.sh and testdoltserver.go describes the per-port behavior the code now implements. Iteration 1, Major 2 (Contract & Interface Fidelity). neutralizeAmbientDoltPort's doc comment claimed "With no port resolvable, applyConfigDefaults' BEADS_TEST_MODE guard forces port 1". Clearing the env vars is not "no port resolvable": resolution falls through to the file chain and derives a directory from filepath.Dir(cfg.Path) when BeadsDir is empty (store.go:1661-1664), so a reachable .beads/dolt-server.port still yields a non-zero port and the ServerPort == 0 arm (store.go:1683-1687) never fires; production-port detection is strictly narrower than resolution. Narrowed to what the change actually guarantees -- no environment-named port is honored -- at all three sites carrying the claim: the helper doc, the TESTING.md paragraph, and TestApplyConfigDefaults_NoResolvablePortFailsClosed's own comment, whose fixture (an empty beadsDir) is what excludes the surviving path. The residual stays tracked in be-rl6tm. That test's stale cross-reference to a renamed test is corrected in the same pass. Iteration 1, minor (Test Evidence Quality). The Windows gate's comment claimed a CI lane it does not have; it now states the coverage exactly -- compile-checked, not CI-executed -- and lists the four windows-latest jobs' named selectors. Iteration 1, nits. doltServerErr is read under doltServerMu via sharedContainerErr(); TestCheckDoltFn_DefaultsToRealProbe pins function identity with reflect instead of asserting non-nil, which any leaked stub satisfied. Mutation ledger (scoped `go test ./internal/testutil/`, scrubbed env; every mutant restored from a copy and verified byte-identical): a set marker spares every marked_but_divergent_var, divergent_legacy_var, variable (the i2 blocker) marker_names_neither_var and boolean_marker red on BOTH failure paths the marker covers marked_but_divergent_var, marker_names_neither_var BEADS_DOLT_PORT only and boolean_marker red on both paths presence instead of value the same red set as the first row no clear after a failed the 5 container_start_failed cases that expect container start a clear red; every not_ready case stays green start attempted before the all four incumbent fail-closed subtests and probe every not_ready case red leak a wrapper into StartSharedContainerFn_DefaultsToRealStart red startSharedContainerFn delete the hermetic ClearsAmbientPortWhenNotReady red t.Setenv, marker ambient at the fixture's port Full package green with Docker, including the real-container EnsureDoltContainerForTestMain path (43 test and subtest passes, no skips); `go test ./scripts/`, which drives test.sh, green; `GOOS=windows go vet ./internal/testutil/` clean; `tools/bazel/go_srcs.py --check` clean, and no BUILD change is due (no new files, no new non-stdlib imports). Deferred deliberately: the iteration-1 synthesis's optional windows-latest step for ./internal/testutil/. It is newly feasible -- this branch is what made the package's Windows test binary compile -- but it would also newly execute four pre-existing test functions in this package that have never run on Windows, so it belongs in its own change. RequireDoltContainer's unsynchronized doltServerErr read is base-owned and left alone for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Julian Knutsen <julian@gastownhall.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
229 lines
8.8 KiB
Bash
Executable file
229 lines
8.8 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Test runner that automatically skips known broken tests
|
|
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
SKIP_FILE="$REPO_ROOT/.test-skip"
|
|
|
|
# Canonical build flags (GOFLAGS=-tags=gms_pure_go, CGO_ENABLED=1).
|
|
# Opt-in ICU-path coverage remains available via scripts/test-icu-path.sh.
|
|
# shellcheck source=../.buildflags
|
|
source "$REPO_ROOT/.buildflags"
|
|
# shellcheck source=ci/lib/test-env.sh
|
|
source "$REPO_ROOT/scripts/ci/lib/test-env.sh"
|
|
|
|
beads_test_env_enter
|
|
|
|
# Build skip pattern from .test-skip file
|
|
build_skip_pattern() {
|
|
if [[ ! -f "$SKIP_FILE" ]]; then
|
|
echo ""
|
|
return
|
|
fi
|
|
|
|
# Read non-comment, non-empty lines and join with |
|
|
local pattern=$(grep -v '^#' "$SKIP_FILE" | grep -v '^[[:space:]]*$' | paste -sd '|' -)
|
|
echo "$pattern"
|
|
}
|
|
|
|
# Default values.
|
|
#
|
|
# TIMEOUT is go test's PER-PACKAGE deadline — a hang backstop, not a
|
|
# performance budget: it costs nothing while packages pass. The floor is set
|
|
# by cmd/bd, the slowest package: measured 2026-07-26 on a busy darwin fleet
|
|
# box (wy-4mtr0), its default suite is ~1090s of test time across ~1490 tests
|
|
# (mostly serial — subprocess tests that spawn bd + embedded Dolt per
|
|
# invocation, largely t.Setenv-bound so they cannot t.Parallel), giving
|
|
# ~18min wall WITH the prebuilt-binary fast path below already removing the
|
|
# in-test `go build` steps. 3m could never fit that, which made every
|
|
# full-suite run FAIL with a package deadline panic naming no failing test.
|
|
# 25m holds the measurement plus fleet-load headroom (the passing full-suite
|
|
# acceptance run clocked cmd/bd at 870s under -p 4 package concurrency).
|
|
# Raise via TEST_TIMEOUT; don't lower it below cmd/bd's measured runtime.
|
|
TIMEOUT="${TEST_TIMEOUT:-25m}"
|
|
GO_TEST_PKG_PARALLEL="${GO_TEST_PKG_PARALLEL:-4}"
|
|
GO_TEST_PARALLEL="${GO_TEST_PARALLEL:-4}"
|
|
SKIP_PATTERN=$(build_skip_pattern)
|
|
VERBOSE="${TEST_VERBOSE:-}"
|
|
RUN_PATTERN="${TEST_RUN:-}"
|
|
COVERAGE="${TEST_COVER:-}"
|
|
COVERPROFILE="${TEST_COVERPROFILE:-/tmp/beads.coverage.out}"
|
|
COVERPKG="${TEST_COVERPKG:-}"
|
|
|
|
# Parse arguments
|
|
PACKAGES=()
|
|
while [[ $# -gt 0 ]]; do
|
|
case $1 in
|
|
-v|--verbose)
|
|
VERBOSE="-v"
|
|
shift
|
|
;;
|
|
-timeout)
|
|
TIMEOUT="$2"
|
|
shift 2
|
|
;;
|
|
-run)
|
|
RUN_PATTERN="$2"
|
|
shift 2
|
|
;;
|
|
-skip)
|
|
# Allow additional skip patterns
|
|
if [[ -n "$SKIP_PATTERN" ]]; then
|
|
SKIP_PATTERN="$SKIP_PATTERN|$2"
|
|
else
|
|
SKIP_PATTERN="$2"
|
|
fi
|
|
shift 2
|
|
;;
|
|
*)
|
|
PACKAGES+=("$1")
|
|
shift
|
|
;;
|
|
esac
|
|
done
|
|
|
|
# Default to all packages if none specified
|
|
if [[ ${#PACKAGES[@]} -eq 0 ]]; then
|
|
PACKAGES=("./...")
|
|
fi
|
|
|
|
# Prebuild bd once for subprocess-style tests (wy-4mtr0). cmd/bd has a dozen
|
|
# test helpers that otherwise each `go build` the full bd binary inside the
|
|
# test run — on a busy machine those link steps alone can blow the package
|
|
# deadline, and one helper used to silently fall back to a stale repo-root
|
|
# ./bd. CI already exports BEADS_TEST_BD_BINARY from a prebuilt artifact
|
|
# (.github/workflows/main.yml); this gives the local runner the same fast
|
|
# path. A caller-supplied BEADS_TEST_BD_BINARY always wins; skipped when the
|
|
# requested packages cannot include cmd/bd.
|
|
if [[ -z "${BEADS_TEST_BD_BINARY:-}" ]]; then
|
|
case " ${PACKAGES[*]} " in
|
|
# Any recursive pattern (./..., ./cmd/...) can expand to cmd/bd.
|
|
*"..."* | *"cmd/bd"*)
|
|
if [[ -n "${BEADS_TEST_ENV_ROOT:-}" ]]; then
|
|
PREBUILT_BD_DIR="$BEADS_TEST_ENV_ROOT/prebuilt-bd"
|
|
else
|
|
PREBUILT_BD_DIR=$(mktemp -d "${TMPDIR:-/tmp}/beads-prebuilt-bd-XXXXXX")
|
|
fi
|
|
mkdir -p "$PREBUILT_BD_DIR"
|
|
echo "Prebuilding bd for subprocess tests..." >&2
|
|
PREBUILT_BD_BIN="$PREBUILT_BD_DIR/bd$(go env GOEXE)"
|
|
if go build -o "$PREBUILT_BD_BIN" "$REPO_ROOT/cmd/bd"; then
|
|
export BEADS_TEST_BD_BINARY="$PREBUILT_BD_BIN"
|
|
echo "Prebuilt bd: $BEADS_TEST_BD_BINARY" >&2
|
|
else
|
|
echo "WARN: bd prebuild failed; tests will build their own binaries" >&2
|
|
fi
|
|
;;
|
|
esac
|
|
fi
|
|
|
|
# Optional: start a single shared Dolt test server for all packages.
|
|
# When BEADS_TEST_SHARED_SERVER=1, we start one dolt sql-server and export
|
|
# BEADS_DOLT_PORT (plus BEADS_TEST_SHARED_DOLT_SERVER set to that same port,
|
|
# which marks it as ours rather than inherited) so every test package reuses it
|
|
# instead of spawning its own. This reduces 8-16+ concurrent dolt processes down
|
|
# to 1. Packages whose TestMain can start a Dolt container still prefer the
|
|
# container: it overwrites both port variables on success. Nothing starts when
|
|
# either port variable is already set: a port someone else named is not ours to
|
|
# vouch for. beads_test_env_enter cleared both above, so one can only still be
|
|
# set here when that isolation was skipped (e.g. BEADS_TEST_ENV_DISABLE=1).
|
|
if [[ "${BEADS_TEST_SHARED_SERVER:-}" == "1" && -z "${BEADS_DOLT_PORT:-}" && -z "${BEADS_DOLT_SERVER_PORT:-}" ]]; then
|
|
if command -v dolt &>/dev/null; then
|
|
SHARED_DOLT_DIR=$(mktemp -d /tmp/beads-shared-test-dolt-XXXXXX)
|
|
DOLT_ROOT_PATH="$SHARED_DOLT_DIR"
|
|
export DOLT_ROOT_PATH
|
|
|
|
dolt config --global --add user.name "beads-test" 2>/dev/null
|
|
dolt config --global --add user.email "test@beads.local" 2>/dev/null
|
|
|
|
SHARED_DB_DIR="$SHARED_DOLT_DIR/data"
|
|
mkdir -p "$SHARED_DB_DIR"
|
|
(cd "$SHARED_DB_DIR" && dolt init) >/dev/null 2>&1
|
|
|
|
# Find a free port
|
|
SHARED_PORT=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()')
|
|
|
|
dolt sql-server -H 127.0.0.1 -P "$SHARED_PORT" --no-auto-commit \
|
|
--data-dir "$SHARED_DB_DIR" &>/dev/null &
|
|
SHARED_DOLT_PID=$!
|
|
|
|
# Wait for server to accept connections (up to 30s)
|
|
for i in $(seq 1 60); do
|
|
if nc -z 127.0.0.1 "$SHARED_PORT" 2>/dev/null; then
|
|
break
|
|
fi
|
|
sleep 0.5
|
|
done
|
|
|
|
if nc -z 127.0.0.1 "$SHARED_PORT" 2>/dev/null; then
|
|
export BEADS_DOLT_PORT="$SHARED_PORT"
|
|
# Mark the port as harness-provisioned. testutil's
|
|
# EnsureDoltContainerForTestMain clears an ambient BEADS_DOLT_PORT
|
|
# whenever it is left without a container (gm-2g3g5r: an inherited
|
|
# port can be a production server), which would erase the server we
|
|
# just started on any host without Docker -- this flow's primary
|
|
# audience. The marker is what distinguishes "a test server we
|
|
# allocated" from "whatever the environment happened to name". Its
|
|
# value is the port it vouches for, so it covers the BEADS_DOLT_PORT
|
|
# exported above and no other variable; it is exported only here,
|
|
# only after the server answered, and only for a port this script
|
|
# allocated itself (the guard above requires both port variables to
|
|
# have been unset).
|
|
export BEADS_TEST_SHARED_DOLT_SERVER="$SHARED_PORT"
|
|
export BEADS_TEST_MODE=1
|
|
echo "Shared test Dolt server started on port $SHARED_PORT (PID $SHARED_DOLT_PID)" >&2
|
|
cleanup_shared_server() {
|
|
kill "$SHARED_DOLT_PID" 2>/dev/null || true
|
|
wait "$SHARED_DOLT_PID" 2>/dev/null || true
|
|
rm -rf "$SHARED_DOLT_DIR"
|
|
}
|
|
trap 'cleanup_shared_server; beads_test_env_cleanup' EXIT
|
|
else
|
|
echo "WARN: shared Dolt server failed to start, falling back to per-package servers" >&2
|
|
kill "$SHARED_DOLT_PID" 2>/dev/null || true
|
|
rm -rf "$SHARED_DOLT_DIR"
|
|
fi
|
|
fi
|
|
elif [[ "${BEADS_TEST_SHARED_SERVER:-}" == "1" ]]; then
|
|
echo "WARN: BEADS_TEST_SHARED_SERVER=1 ignored: BEADS_DOLT_PORT or BEADS_DOLT_SERVER_PORT is already set, so no shared server was started" >&2
|
|
fi
|
|
|
|
# Build go test command
|
|
CMD=(go test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -timeout "$TIMEOUT")
|
|
|
|
if [[ -n "$VERBOSE" ]]; then
|
|
CMD+=(-v)
|
|
fi
|
|
|
|
if [[ -n "$SKIP_PATTERN" ]]; then
|
|
CMD+=(-skip "$SKIP_PATTERN")
|
|
fi
|
|
|
|
if [[ -n "$RUN_PATTERN" ]]; then
|
|
CMD+=(-run "$RUN_PATTERN")
|
|
fi
|
|
|
|
if [[ -n "$COVERAGE" ]]; then
|
|
CMD+=(-covermode=atomic -coverprofile "$COVERPROFILE")
|
|
if [[ -n "$COVERPKG" ]]; then
|
|
CMD+=(-coverpkg "$COVERPKG")
|
|
fi
|
|
fi
|
|
|
|
CMD+=("${PACKAGES[@]}")
|
|
|
|
echo "Running: ${CMD[*]}" >&2
|
|
echo "Skipping: $SKIP_PATTERN" >&2
|
|
echo "" >&2
|
|
|
|
"${CMD[@]}"
|
|
status=$?
|
|
|
|
if [[ -n "$COVERAGE" ]]; then
|
|
total=$(go tool cover -func="$COVERPROFILE" | awk '/^total:/ {print $NF}')
|
|
echo "Total coverage: ${total} (profile: ${COVERPROFILE})" >&2
|
|
fi
|
|
|
|
exit $status
|