beads/scripts/test.sh
Jim Wordelman a4bf13f042
Fix: test suites fail closed instead of open on the ambient Dolt port when no test container is started (be-napos) (#5885)
* test(dolt): red — testdb_* leak onto production Dolt port when Docker absent (refs be-h2h25)

* feat: green — Test suites create testdb_* on the PRODUCTION dolt server: harness fails open on the ambient port when Docker is absent (refs be-h2h25)

* docs(release-gate): add be-napos gate — ambient Dolt port fail-open to fail-closed

Deploy bead be-napos (build be-h2h25, review be-7fym1, verdict pass round 2).
7/7 gate criteria PASS; full independent evidence in the gate doc.

refs be-napos, be-h2h25, be-7fym1

* docs(release-gate): record PR link for be-napos gate

refs be-napos

* test(testutil): make the fail-closed gate deterministic, not Docker-conditional

The regression test skipped itself on every host that has Docker, by calling
the real EnsureDoltContainerForTestMain and skipping when it succeeded. That
asserted nothing on CI -- the only lane where it matters -- and on a
Docker-having host it also started the shared singleton container purely to
decide to skip, with no TestMain in internal/testutil to terminate it.

Add checkDoltFn, a package-level seam over checkDolt, so the test can force a
not-ready state without a Docker daemon and without starting a container. Only
EnsureDoltContainerForTestMain routes through it; the neighbouring t.Skip call
sites gate a skip rather than the fail-closed behaviour, so they keep calling
checkDolt directly.

The gate now covers all four not-ready states rather than the one the old test
happened to hit, which also pins the intent that an explicit
BEADS_TEST_SKIP=dolt opt-out must fail closed exactly as a Docker-less host
does. Split into a platform-independent half (the helper's own contract, both
variables cleared, idempotent) and a !windows half (the wiring, via the seam).

Red/green: with the neutralizeAmbientDoltPort call removed from the not-ready
branch, 4 of 4 subtests fail and 0 pass; restored, 4 of 4 pass in 0.06s with
no Docker and no container. (refs be-napos)

* fix(testutil): clear the ambient Dolt port on the Windows stub too

The Windows EnsureDoltContainerForTestMain returned its error without touching
BEADS_DOLT_SERVER_PORT / BEADS_DOLT_PORT, so the fail-open this branch fixes
still shipped there -- unconditionally, since that stub never succeeds. Windows
is a real lane: .github/workflows/pr.yml runs windows-latest on four jobs plus
the cross-platform matrix.

Gating it needed one pre-existing repair. testdoltserver_env_test.go has no
build tag but calls checkDolt, which is !windows-only, so the internal/testutil
test binary has never compiled for Windows -- reproduced identically on
origin/main. Tagging that file !windows fixes the whole package's Windows test
lane; without it a new Windows gate would compile nowhere and assert nothing,
which is the same defect this branch is already correcting.

Verified GOOS=windows go vet ./internal/testutil/ clean (it fails on
origin/main), and GOOS=windows go build clean before and after. The Windows
gate itself is compile-verified only -- no Windows host here to execute it.
(refs be-napos)

* docs(testing): state that ambient BEADS_DOLT_*_PORT is never honored

The guard fires on every not-ready state, not only "Docker unavailable" -- a
missing or wrong-version image and an explicit BEADS_TEST_SKIP=dolt opt-out
clear the ambient port too. That is deliberate: the invariant is the absence of
a test container, not the reason for it. Say so where the next person will look
before investigating, and name the supported way to target a specific server.
(refs be-napos)

* chore: remove leaked internal release-gate doc from PR branch

  release-gates/be-napos-ambient-dolt-port-failclosed-gate.md

Tracked our internal gc-management/beads-rig release-gate checklist and
should never have been committed to this upstream branch -- it's process
residue from our deploy tooling, not part of the change itself.

Per maintainer bee-ghosttrack's request during review of PR #6571. The
root cause (our deploy tooling auto-committing this file) is already fixed
as of gm-qatxl1; this branch predates that fix, so this commit is cleanup
only -- no other content on this branch is touched.

Ref: be-lk7o3

* chore(bazel): enumerate the PR's new test files in BUILD.bazel

Base added BUILD.bazel to internal/testutil and internal/storage/dolt after
this PR was opened (#6881 made the Bazel lane real, and fork PRs always run
it). The four new test files this PR adds were therefore absent from the
go_test srcs lists, which the bazel lane's equivalence check reports as a
missing target.

Generated by 'make bazel-sync' (gazelle), not hand-edited, because the CI
drift step diffs against exactly what gazelle emits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(testutil): reconcile the shared-server harness with the fail-closed clear

Review iterations 1-2 (adopt-pr, PR #5885): every gating finding from both
review syntheses, plus their minors and nits.

Iteration 1, Major 1 (Change Impact / Blast Radius). scripts/test.sh's
BEADS_TEST_SHARED_SERVER=1 flow starts a dockerless `dolt sql-server` and
exports BEADS_DOLT_PORT for every package to reuse. checkDolt() has no notion
of an already-provisioned shared server, so on that flow's primary audience --
a host without Docker -- all 12 EnsureDoltContainerForTestMain callers erased
the harness's own sanctioned port before m.Run(), turning green dockerless runs
red, while engdocs/CI_TEST_SURFACE_AUDIT.md and the new TESTING.md paragraph
advertised the two mechanisms unreconciled.

Taken: a dedicated marker, BEADS_TEST_SHARED_DOLT_SERVER, exported by
scripts/test.sh beside the port it allocated, which the helper's failure paths
honor. Both docs state the reconciliation, including the precedence: a
container, where one can be started, still overwrites both variables. Not
taken: "skip the clear when BEADS_TEST_SHARED_SERVER=1 and a port var is
already set". That flag is overloaded (cmd/bd/shared_server_integration_test.go
reads it too), and the script starts no server when a port is already set, so
the flag beside a port is exactly the case where the port is ambient.

Iteration 2, blocker (Behavioral Correctness). Iteration 1 shipped that marker
as a boolean ("1") which spared BOTH port variables, while the script vouched
only for the BEADS_DOLT_PORT it exported and its entry guard never looked at
BEADS_DOLT_SERVER_PORT -- the variable applyConfigDefaults resolves first. The
runner's test environment (beads_test_env_enter) unsets both variables before
the flow runs, so the hole needed that isolation skipped, e.g. with the
documented BEADS_TEST_ENV_DISABLE=1; then an inherited BEADS_DOLT_SERVER_PORT
reached `go test` beside the marker and outranked the shared server --
gm-2g3g5r, re-opened by its own fix. Measured with a fake `go` that prints its
environment: SERVER_PORT=59999 PORT=<shared> MARKER=1 under the iteration-1
script. Now:

  - the marker's value is the port it vouches for, and the helper decides per
    variable (neutralizeUnvouchedDoltPorts): only a variable holding exactly
    that port survives, and says so on stderr; every other one is cleared,
    marker or not;
  - the flow starts only when neither port variable is set, and says on stderr
    when BEADS_TEST_SHARED_SERVER=1 is ignored for that reason.

Not taken: also exporting BEADS_DOLT_SERVER_PORT from the flow. The widened
guard means it is always unset there, so the export adds no safety, and five
sites read that variable alone (internal/doltserver/doltserver.go:796,
cmd/bd/init.go:773 and :1351, cmd/bd/bootstrap.go:857,
internal/testutil/testdoltbranch.go:164) -- consumers the flow has never fed.

Iteration 2, minor (Test Evidence Quality). The exception's second arm, a
container that would not start, was reached by no test with its guard live.
Both failure paths now share one clear, and the container start is a seam
(startSharedContainerFn, pinned to startSharedContainer by identity) so tests
reach that path without consuming doltServerOnce.
TestEnsureDoltContainerForTestMain_KeepsHarnessProvisionedPort runs its whole
table, marked_but_divergent_var included, through both paths.

Iteration 2, minor (Contract & Interface Fidelity). The marker prose in
TESTING.md, CI_TEST_SURFACE_AUDIT.md, test.sh and testdoltserver.go describes
the per-port behavior the code now implements.

Iteration 1, Major 2 (Contract & Interface Fidelity). neutralizeAmbientDoltPort's
doc comment claimed "With no port resolvable, applyConfigDefaults'
BEADS_TEST_MODE guard forces port 1". Clearing the env vars is not "no port
resolvable": resolution falls through to the file chain and derives a
directory from filepath.Dir(cfg.Path) when BeadsDir is empty
(store.go:1661-1664), so a reachable .beads/dolt-server.port still yields a
non-zero port and the ServerPort == 0 arm (store.go:1683-1687) never fires;
production-port detection is strictly narrower than resolution. Narrowed to
what the change actually guarantees -- no environment-named port is honored --
at all three sites carrying the claim: the helper doc, the TESTING.md
paragraph, and TestApplyConfigDefaults_NoResolvablePortFailsClosed's own
comment, whose fixture (an empty beadsDir) is what excludes the surviving
path. The residual stays tracked in be-rl6tm. That test's stale
cross-reference to a renamed test is corrected in the same pass.

Iteration 1, minor (Test Evidence Quality). The Windows gate's comment claimed
a CI lane it does not have; it now states the coverage exactly --
compile-checked, not CI-executed -- and lists the four windows-latest jobs'
named selectors.

Iteration 1, nits. doltServerErr is read under doltServerMu via
sharedContainerErr(); TestCheckDoltFn_DefaultsToRealProbe pins function
identity with reflect instead of asserting non-nil, which any leaked stub
satisfied.

Mutation ledger (scoped `go test ./internal/testutil/`, scrubbed env; every
mutant restored from a copy and verified byte-identical):

  a set marker spares every    marked_but_divergent_var, divergent_legacy_var,
  variable (the i2 blocker)    marker_names_neither_var and boolean_marker red
                               on BOTH failure paths
  the marker covers            marked_but_divergent_var, marker_names_neither_var
  BEADS_DOLT_PORT only         and boolean_marker red on both paths
  presence instead of value    the same red set as the first row
  no clear after a failed      the 5 container_start_failed cases that expect
  container start              a clear red; every not_ready case stays green
  start attempted before the   all four incumbent fail-closed subtests and
  probe                        every not_ready case red
  leak a wrapper into          StartSharedContainerFn_DefaultsToRealStart red
  startSharedContainerFn
  delete the hermetic          ClearsAmbientPortWhenNotReady red
  t.Setenv, marker ambient
  at the fixture's port

Full package green with Docker, including the real-container
EnsureDoltContainerForTestMain path (43 test and subtest passes, no skips);
`go test ./scripts/`, which drives test.sh, green; `GOOS=windows go vet
./internal/testutil/` clean; `tools/bazel/go_srcs.py --check` clean, and no
BUILD change is due (no new files, no new non-stdlib imports).

Deferred deliberately: the iteration-1 synthesis's optional windows-latest step
for ./internal/testutil/. It is newly feasible -- this branch is what made the
package's Windows test binary compile -- but it would also newly execute four
pre-existing test functions in this package that have never run on Windows, so
it belongs in its own change. RequireDoltContainer's unsynchronized
doltServerErr read is base-owned and left alone for the same reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julian@gastownhall.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
2026-10-01 05:21:49 -07:00

229 lines
8.8 KiB
Bash
Executable file

#!/usr/bin/env bash
# Test runner that automatically skips known broken tests
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
SKIP_FILE="$REPO_ROOT/.test-skip"
# Canonical build flags (GOFLAGS=-tags=gms_pure_go, CGO_ENABLED=1).
# Opt-in ICU-path coverage remains available via scripts/test-icu-path.sh.
# shellcheck source=../.buildflags
source "$REPO_ROOT/.buildflags"
# shellcheck source=ci/lib/test-env.sh
source "$REPO_ROOT/scripts/ci/lib/test-env.sh"
beads_test_env_enter
# Build skip pattern from .test-skip file
build_skip_pattern() {
if [[ ! -f "$SKIP_FILE" ]]; then
echo ""
return
fi
# Read non-comment, non-empty lines and join with |
local pattern=$(grep -v '^#' "$SKIP_FILE" | grep -v '^[[:space:]]*$' | paste -sd '|' -)
echo "$pattern"
}
# Default values.
#
# TIMEOUT is go test's PER-PACKAGE deadline — a hang backstop, not a
# performance budget: it costs nothing while packages pass. The floor is set
# by cmd/bd, the slowest package: measured 2026-07-26 on a busy darwin fleet
# box (wy-4mtr0), its default suite is ~1090s of test time across ~1490 tests
# (mostly serial — subprocess tests that spawn bd + embedded Dolt per
# invocation, largely t.Setenv-bound so they cannot t.Parallel), giving
# ~18min wall WITH the prebuilt-binary fast path below already removing the
# in-test `go build` steps. 3m could never fit that, which made every
# full-suite run FAIL with a package deadline panic naming no failing test.
# 25m holds the measurement plus fleet-load headroom (the passing full-suite
# acceptance run clocked cmd/bd at 870s under -p 4 package concurrency).
# Raise via TEST_TIMEOUT; don't lower it below cmd/bd's measured runtime.
TIMEOUT="${TEST_TIMEOUT:-25m}"
GO_TEST_PKG_PARALLEL="${GO_TEST_PKG_PARALLEL:-4}"
GO_TEST_PARALLEL="${GO_TEST_PARALLEL:-4}"
SKIP_PATTERN=$(build_skip_pattern)
VERBOSE="${TEST_VERBOSE:-}"
RUN_PATTERN="${TEST_RUN:-}"
COVERAGE="${TEST_COVER:-}"
COVERPROFILE="${TEST_COVERPROFILE:-/tmp/beads.coverage.out}"
COVERPKG="${TEST_COVERPKG:-}"
# Parse arguments
PACKAGES=()
while [[ $# -gt 0 ]]; do
case $1 in
-v|--verbose)
VERBOSE="-v"
shift
;;
-timeout)
TIMEOUT="$2"
shift 2
;;
-run)
RUN_PATTERN="$2"
shift 2
;;
-skip)
# Allow additional skip patterns
if [[ -n "$SKIP_PATTERN" ]]; then
SKIP_PATTERN="$SKIP_PATTERN|$2"
else
SKIP_PATTERN="$2"
fi
shift 2
;;
*)
PACKAGES+=("$1")
shift
;;
esac
done
# Default to all packages if none specified
if [[ ${#PACKAGES[@]} -eq 0 ]]; then
PACKAGES=("./...")
fi
# Prebuild bd once for subprocess-style tests (wy-4mtr0). cmd/bd has a dozen
# test helpers that otherwise each `go build` the full bd binary inside the
# test run — on a busy machine those link steps alone can blow the package
# deadline, and one helper used to silently fall back to a stale repo-root
# ./bd. CI already exports BEADS_TEST_BD_BINARY from a prebuilt artifact
# (.github/workflows/main.yml); this gives the local runner the same fast
# path. A caller-supplied BEADS_TEST_BD_BINARY always wins; skipped when the
# requested packages cannot include cmd/bd.
if [[ -z "${BEADS_TEST_BD_BINARY:-}" ]]; then
case " ${PACKAGES[*]} " in
# Any recursive pattern (./..., ./cmd/...) can expand to cmd/bd.
*"..."* | *"cmd/bd"*)
if [[ -n "${BEADS_TEST_ENV_ROOT:-}" ]]; then
PREBUILT_BD_DIR="$BEADS_TEST_ENV_ROOT/prebuilt-bd"
else
PREBUILT_BD_DIR=$(mktemp -d "${TMPDIR:-/tmp}/beads-prebuilt-bd-XXXXXX")
fi
mkdir -p "$PREBUILT_BD_DIR"
echo "Prebuilding bd for subprocess tests..." >&2
PREBUILT_BD_BIN="$PREBUILT_BD_DIR/bd$(go env GOEXE)"
if go build -o "$PREBUILT_BD_BIN" "$REPO_ROOT/cmd/bd"; then
export BEADS_TEST_BD_BINARY="$PREBUILT_BD_BIN"
echo "Prebuilt bd: $BEADS_TEST_BD_BINARY" >&2
else
echo "WARN: bd prebuild failed; tests will build their own binaries" >&2
fi
;;
esac
fi
# Optional: start a single shared Dolt test server for all packages.
# When BEADS_TEST_SHARED_SERVER=1, we start one dolt sql-server and export
# BEADS_DOLT_PORT (plus BEADS_TEST_SHARED_DOLT_SERVER set to that same port,
# which marks it as ours rather than inherited) so every test package reuses it
# instead of spawning its own. This reduces 8-16+ concurrent dolt processes down
# to 1. Packages whose TestMain can start a Dolt container still prefer the
# container: it overwrites both port variables on success. Nothing starts when
# either port variable is already set: a port someone else named is not ours to
# vouch for. beads_test_env_enter cleared both above, so one can only still be
# set here when that isolation was skipped (e.g. BEADS_TEST_ENV_DISABLE=1).
if [[ "${BEADS_TEST_SHARED_SERVER:-}" == "1" && -z "${BEADS_DOLT_PORT:-}" && -z "${BEADS_DOLT_SERVER_PORT:-}" ]]; then
if command -v dolt &>/dev/null; then
SHARED_DOLT_DIR=$(mktemp -d /tmp/beads-shared-test-dolt-XXXXXX)
DOLT_ROOT_PATH="$SHARED_DOLT_DIR"
export DOLT_ROOT_PATH
dolt config --global --add user.name "beads-test" 2>/dev/null
dolt config --global --add user.email "test@beads.local" 2>/dev/null
SHARED_DB_DIR="$SHARED_DOLT_DIR/data"
mkdir -p "$SHARED_DB_DIR"
(cd "$SHARED_DB_DIR" && dolt init) >/dev/null 2>&1
# Find a free port
SHARED_PORT=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()')
dolt sql-server -H 127.0.0.1 -P "$SHARED_PORT" --no-auto-commit \
--data-dir "$SHARED_DB_DIR" &>/dev/null &
SHARED_DOLT_PID=$!
# Wait for server to accept connections (up to 30s)
for i in $(seq 1 60); do
if nc -z 127.0.0.1 "$SHARED_PORT" 2>/dev/null; then
break
fi
sleep 0.5
done
if nc -z 127.0.0.1 "$SHARED_PORT" 2>/dev/null; then
export BEADS_DOLT_PORT="$SHARED_PORT"
# Mark the port as harness-provisioned. testutil's
# EnsureDoltContainerForTestMain clears an ambient BEADS_DOLT_PORT
# whenever it is left without a container (gm-2g3g5r: an inherited
# port can be a production server), which would erase the server we
# just started on any host without Docker -- this flow's primary
# audience. The marker is what distinguishes "a test server we
# allocated" from "whatever the environment happened to name". Its
# value is the port it vouches for, so it covers the BEADS_DOLT_PORT
# exported above and no other variable; it is exported only here,
# only after the server answered, and only for a port this script
# allocated itself (the guard above requires both port variables to
# have been unset).
export BEADS_TEST_SHARED_DOLT_SERVER="$SHARED_PORT"
export BEADS_TEST_MODE=1
echo "Shared test Dolt server started on port $SHARED_PORT (PID $SHARED_DOLT_PID)" >&2
cleanup_shared_server() {
kill "$SHARED_DOLT_PID" 2>/dev/null || true
wait "$SHARED_DOLT_PID" 2>/dev/null || true
rm -rf "$SHARED_DOLT_DIR"
}
trap 'cleanup_shared_server; beads_test_env_cleanup' EXIT
else
echo "WARN: shared Dolt server failed to start, falling back to per-package servers" >&2
kill "$SHARED_DOLT_PID" 2>/dev/null || true
rm -rf "$SHARED_DOLT_DIR"
fi
fi
elif [[ "${BEADS_TEST_SHARED_SERVER:-}" == "1" ]]; then
echo "WARN: BEADS_TEST_SHARED_SERVER=1 ignored: BEADS_DOLT_PORT or BEADS_DOLT_SERVER_PORT is already set, so no shared server was started" >&2
fi
# Build go test command
CMD=(go test -p "$GO_TEST_PKG_PARALLEL" -parallel "$GO_TEST_PARALLEL" -timeout "$TIMEOUT")
if [[ -n "$VERBOSE" ]]; then
CMD+=(-v)
fi
if [[ -n "$SKIP_PATTERN" ]]; then
CMD+=(-skip "$SKIP_PATTERN")
fi
if [[ -n "$RUN_PATTERN" ]]; then
CMD+=(-run "$RUN_PATTERN")
fi
if [[ -n "$COVERAGE" ]]; then
CMD+=(-covermode=atomic -coverprofile "$COVERPROFILE")
if [[ -n "$COVERPKG" ]]; then
CMD+=(-coverpkg "$COVERPKG")
fi
fi
CMD+=("${PACKAGES[@]}")
echo "Running: ${CMD[*]}" >&2
echo "Skipping: $SKIP_PATTERN" >&2
echo "" >&2
"${CMD[@]}"
status=$?
if [[ -n "$COVERAGE" ]]; then
total=$(go tool cover -func="$COVERPROFILE" | awk '/^total:/ {print $NF}')
echo "Total coverage: ${total} (profile: ${COVERPROFILE})" >&2
fi
exit $status