fix(bench): stop dolt benchmarks from leaking test databases onto shared servers (#5029)

* test(feat): red — Fix: Dolt benchmark leaks beads_test_bench_* DBs onto shared prod server (firewall only guards port 3307) (refs be-cfm3z)

Pure-logic regression tests for benchDoltServerPort (not yet implemented,
intentionally RED via compile failure): benchmarks must never resolve
their server port from ambient BEADS_DOLT_SERVER_PORT/BEADS_DOLT_PORT,
since gc agent shells set those to point at a shared production Dolt
server. Tests are network-free by design so RED verification cannot
itself touch the real server.

* feat: green — Fix: Dolt benchmark leaks beads_test_bench_* DBs onto shared prod server (firewall only guards port 3307) (refs be-cfm3z)

setupBenchStore and BenchmarkBootstrapEmbedded no longer resolve their
server port from ambient BEADS_DOLT_SERVER_PORT/BEADS_DOLT_PORT (gc agent
shells set these to point at a shared production Dolt server). Both now:

- scrub the inherited env vars via b.Setenv
- require explicit opt-in via BEADS_BENCH_DOLT_PORT, skipping otherwise
- set BEADS_TEST_MODE=1 and BEADS_TEST_SERVER=1 as backstops
- drop their benchmark database in cleanup via dropBenchDatabase

Verified: full benchmark suite (49 benchmarks/sub-cases) skips cleanly
under default env with zero network I/O; go test/vet green with no new
failures vs preflight baseline.

* docs(release-gates): add gate record for be-cfm3z dolt-bench leak fix

Gate PASS (7/7) for be-k4hyk, deploying be-cfm3z's dolt benchmark
prod-DB-leak fix (reviewed PASS). Independently re-verified all
criteria including a from-scratch test run that surfaced and resolved
a test-environment methodology question (raw go test vs the repo's
hermetic scripts/test.sh wrapper).
This commit is contained in:
Jim Wordelman 2026-07-25 15:40:54 -07:00 • committed by GitHub
commit f7e3fdad3f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 196 additions and 2 deletions

View file

@ -0,0 +1,61 @@
// Regression tests for be-cfm3z: the Dolt benchmark suite used to resolve
// its server port from ambient BEADS_DOLT_SERVER_PORT/BEADS_DOLT_PORT, which
// gc agent shells set to point at a shared production Dolt server. These
// tests exercise the pure decision helper (benchDoltServerPort) directly —
// no server, no network — so they can never themselves reach a real Dolt
// server, production or otherwise, regardless of ambient env state.
package dolt
import "testing"
func TestBenchDoltServerPort_SkipsWithoutOptIn(t *testing.T) {
t.Setenv("BEADS_BENCH_DOLT_PORT", "")
// Simulate the stock gc agent shell, which exports this pointing at the
// shared production server.
t.Setenv("BEADS_DOLT_SERVER_PORT", "28231")
t.Setenv("BEADS_DOLT_PORT", "")
port, skip, reason := benchDoltServerPort()
if !skip {
t.Fatalf("expected skip=true when BEADS_BENCH_DOLT_PORT is unset, got skip=false (port=%d)", port)
}
if reason == "" {
t.Fatal("expected a non-empty skip reason")
}
}
func TestBenchDoltServerPort_UsesExplicitOptIn(t *testing.T) {
t.Setenv("BEADS_BENCH_DOLT_PORT", "19999")
port, skip, reason := benchDoltServerPort()
if skip {
t.Fatalf("expected skip=false when BEADS_BENCH_DOLT_PORT is set, got reason %q", reason)
}
if port != 19999 {
t.Fatalf("expected port 19999, got %d", port)
}
}
func TestBenchDoltServerPort_RejectsInvalidOptIn(t *testing.T) {
t.Setenv("BEADS_BENCH_DOLT_PORT", "not-a-port")
_, skip, reason := benchDoltServerPort()
if !skip {
t.Fatal("expected skip=true for a non-numeric BEADS_BENCH_DOLT_PORT value")
}
if reason == "" {
t.Fatal("expected a non-empty skip reason for an invalid port")
}
}
func TestBenchDoltServerPort_RejectsZeroOrNegative(t *testing.T) {
t.Setenv("BEADS_BENCH_DOLT_PORT", "0")
_, skip, reason := benchDoltServerPort()
if !skip {
t.Fatal("expected skip=true for port 0")
}
if reason == "" {
t.Fatal("expected a non-empty skip reason for port 0")
}
}

View file

@ -15,6 +15,7 @@ import (
"encoding/json"
"fmt"
"os"
"strconv"
"strings"
"sync"
"testing"
@ -29,6 +30,41 @@ func benchDatabaseName() string {
return fmt.Sprintf("beads_test_bench_%d", time.Now().UnixNano())
}
// benchDoltServerPort decides whether benchmarks may connect to a real Dolt
// server. Benchmarks must never resolve their server port from ambient
// BEADS_DOLT_SERVER_PORT/BEADS_DOLT_PORT: gc agent shells set those to point
// at a shared production Dolt server, and there is no way to distinguish "a
// dedicated throwaway benchmark port" from "the shared prod port" other than
// requiring an explicit, benchmark-only opt-in (be-cfm3z).
//
// Returns skip=true with a reason when BEADS_BENCH_DOLT_PORT is unset or
// invalid; the caller should b.Skip(reason) rather than connect.
func benchDoltServerPort() (port int, skip bool, reason string) {
v := os.Getenv("BEADS_BENCH_DOLT_PORT")
if v == "" {
return 0, true, "set BEADS_BENCH_DOLT_PORT to a dedicated throwaway dolt server to run benchmarks"
}
p, err := strconv.Atoi(v)
if err != nil || p <= 0 {
return 0, true, fmt.Sprintf("BEADS_BENCH_DOLT_PORT=%q is not a valid port", v)
}
return p, false, ""
}
// dropBenchDatabase best-effort drops a benchmark-created database from the
// server described by cfg. Opens its own short-lived admin connection rather
// than reusing a *DoltStore's pool, so it works regardless of that store's
// close state. Errors are ignored: this is cleanup, not the test assertion.
func dropBenchDatabase(cfg *Config, name string) {
dsn := buildServerDSN(cfg, "")
db, err := sql.Open("mysql", dsn)
if err != nil {
return
}
defer db.Close()
_, _ = db.Exec(fmt.Sprintf("DROP DATABASE IF EXISTS `%s`", name))
}
// setupBenchStore creates a store for benchmarks
func setupBenchStore(b *testing.B) (*DoltStore, func()) {
b.Helper()
@ -42,17 +78,32 @@ func setupBenchStore(b *testing.B) (*DoltStore, func()) {
}
}
// Never resolve the benchmark's server port from ambient env — gc agent
// shells set BEADS_DOLT_SERVER_PORT to point at a shared production
// server (be-cfm3z).
b.Setenv("BEADS_DOLT_SERVER_PORT", "")
b.Setenv("BEADS_DOLT_PORT", "")
b.Setenv("BEADS_TEST_MODE", "1")
b.Setenv("BEADS_TEST_SERVER", "1") // forward-compat opt-in for PR #3632/AD-01's firewall
port, skip, reason := benchDoltServerPort()
if skip {
b.Skip(reason)
}
ctx := context.Background()
tmpDir, err := os.MkdirTemp("", "dolt-bench-*")
if err != nil {
b.Fatalf("failed to create temp dir: %v", err)
}
dbName := benchDatabaseName()
cfg := &Config{
Path: tmpDir,
CommitterName: "bench",
CommitterEmail: "bench@example.com",
Database: benchDatabaseName(),
Database: dbName,
ServerPort: port,
CreateIfMissing: true,
}
@ -70,6 +121,7 @@ func setupBenchStore(b *testing.B) (*DoltStore, func()) {
cleanup := func() {
store.Close()
dropBenchDatabase(cfg, dbName)
os.RemoveAll(tmpDir)
}
@ -91,6 +143,19 @@ func BenchmarkBootstrapEmbedded(b *testing.B) {
}
}
// Never resolve the benchmark's server port from ambient env — gc agent
// shells set BEADS_DOLT_SERVER_PORT to point at a shared production
// server (be-cfm3z).
b.Setenv("BEADS_DOLT_SERVER_PORT", "")
b.Setenv("BEADS_DOLT_PORT", "")
b.Setenv("BEADS_TEST_MODE", "1")
b.Setenv("BEADS_TEST_SERVER", "1") // forward-compat opt-in for PR #3632/AD-01's firewall
port, skip, reason := benchDoltServerPort()
if skip {
b.Skip(reason)
}
tmpDir, err := os.MkdirTemp("", "dolt-bootstrap-bench-*")
if err != nil {
b.Fatalf("failed to create temp dir: %v", err)
@ -100,11 +165,13 @@ func BenchmarkBootstrapEmbedded(b *testing.B) {
ctx := context.Background()
// Create initial store to set up schema
dbName := benchDatabaseName()
cfg := &Config{
Path: tmpDir,
CommitterName: "bench",
CommitterEmail: "bench@example.com",
Database: benchDatabaseName(),
Database: dbName,
ServerPort: port,
CreateIfMissing: true,
}
@ -113,6 +180,7 @@ func BenchmarkBootstrapEmbedded(b *testing.B) {
b.Fatalf("failed to create initial store: %v", err)
}
initStore.Close()
defer dropBenchDatabase(cfg, dbName)
b.ResetTimer()
for i := 0; i < b.N; i++ {

View file

@ -0,0 +1,65 @@
# Release gate — be-k4hyk (be-cfm3z dolt-benchmark prod-DB-leak fix)
**Date:** 2026-07-25
**Deployer:** beads/deployer
**Bead (deploy):** be-k4hyk — Deploy: be-cfm3z dolt-benchmark prod-DB-leak fix (reviewed PASS)
**Source bead:** be-cfm3z — closed, review verdict PASS
**Source commit:** `2d20764f4d08ab9855ee860b6e796a0d546182f5` (provenance branch `gc-builder-d7bc3eb5bd82`, review worktree `/var/tmp/be-cfm3z-review`)
**Branch:** `deploy/be-k4hyk-gate` (isolated, cut fresh at the reviewed SHA — never a shared `gc-builder-*` branch)
**Base:** `origin/main` @ `c2796999d` ("fix(reclaim): repair a mojibake em dash in the replica help text")
**Merge-base:** `b7e25f091` ("Merge pull request #4997 from gastownhall/db/gate-check")
**Merge-tree simulation:** `git merge-tree --write-tree origin/main 2d20764f4` → tree `4c500b895`, exit 0, **zero conflicts**
## Verdict: PASS
## Criteria walk
| # | Criterion | Result | Evidence |
|---|-----------|--------|----------|
| 1 | Review PASS present | PASS | beads/reviewer PASS verdict in be-cfm3z notes: 0 blockers, build/vet/lint clean, 173/173 package tests, both headline claims (clean-skip under live ambient danger; opt-in path against a real throwaway dolt server) empirically verified end-to-end. One non-blocking count nitpick (48 vs claimed 49 benchmark sub-cases), fully reconciled, not a blocker. |
| 2 | Acceptance criteria met | PASS | All 4 "Done-when" items from be-cfm3z independently verified against the actual diff (see Acceptance check below). |
| 3 | Tests pass | PASS | `./scripts/test.sh -v ./internal/storage/dolt/...` on `deploy/be-k4hyk-gate` (commit `2d20764f4`) → **exit 0, 173 PASS / 0 FAIL**, matching the reviewer's report exactly. `go build ./...` and `go vet ./...` both exit 0. See Test-environment note for a methodology correction. |
| 4 | No HIGH-severity findings open | PASS | Reviewer notes record zero HIGH/blocker findings. |
| 5 | Final branch is clean | PASS | `git status` on `deploy/be-k4hyk-gate` at `2d20764f4`: "nothing to commit, working tree clean." |
| 6 | Branch diverges cleanly from main | PASS | `git merge-tree --write-tree origin/main 2d20764f4` succeeds with a single merged tree and no conflict markers. |
| 7 | Single feature theme | PASS | `git diff --stat b7e25f091 2d20764f4` — exactly 2 files (`dolt_benchmark_test.go` modified, `dolt_benchmark_safety_test.go` new), 131 insertions(+), 2 deletions(-). One theme: stop the dolt benchmark suite from leaking test databases onto the shared production Dolt server. |
## Acceptance check (be-cfm3z "Done-when")
1. **`setupBenchStore`/`BenchmarkBootstrapEmbedded` skip when `BEADS_BENCH_DOLT_PORT` is unset; never resolve server port from ambient `BEADS_DOLT_SERVER_PORT`.**
- `dolt_benchmark_test.go`: both call sites scrub `BEADS_DOLT_SERVER_PORT`/`BEADS_DOLT_PORT` to `""` via `b.Setenv` *before* calling the new `benchDoltServerPort()` helper, then `b.Skip(reason)` when it reports `skip=true`.
- Directly re-ran the 4 new pure-logic tests in `dolt_benchmark_safety_test.go` (`TestBenchDoltServerPort_{SkipsWithoutOptIn,UsesExplicitOptIn,RejectsInvalidOptIn,RejectsZeroOrNegative}`) under this rig's real live ambient value (`BEADS_DOLT_SERVER_PORT=28231`) — all 4 PASS, confirming the skip decision is correct under the exact real danger condition, not just a simulated one.
- **PASS.**
2. **`cleanup()` drops the bench database.**
- `dropBenchDatabase()` opens a short-lived admin connection and issues `DROP DATABASE IF EXISTS` for the exact database name used by setup; wired into both benchmark cleanup paths.
- **PASS.**
3. **Regression test (pure logic, no server) proving the skip behavior; plus a test that `setupBenchStore` skips under the default agent env.**
- `dolt_benchmark_safety_test.go` is a new file of exactly this shape: 4 table tests against the pure decision helper `benchDoltServerPort()`, no network, no server dependency — cannot itself reach a real Dolt server regardless of ambient env.
- **PASS.**
4. **`CGO_ENABLED=0 go test -tags gms_pure_go -count=1 ./internal/storage/dolt/...` green.**
- Ran the canonical repo test entrypoint (`./scripts/test.sh`, which wraps this exact invocation with the repo's standard hermetic env setup) — 173/173 green. See Test-environment note.
- **PASS.**
## Test-environment note (methodology correction, non-blocking)
Independently re-running tests surfaced a discrepancy worth recording. A raw
```
CGO_ENABLED=0 go test -tags gms_pure_go -count=1 ./internal/storage/dolt/...
```
invocation (i.e., *without* going through `scripts/test.sh`) on `deploy/be-k4hyk-gate` at `2d20764f4` fails 5 tests: `TestApplyConfigDefaults_{TestModeUseSentinelPort,TestModeWithPort,TestModeBlocksProdPort,EnvOverridesConfig,ProductionFallback}` in `store_unit_test.go`. This is **not** a regression from be-cfm3z's diff (which never touches `store.go` or `store_unit_test.go`); it is the exact same baseline pre-existing failure the builder's own preflight note already documented, root-caused to `store_unit_test.go` clearing legacy `BEADS_DOLT_PORT` but not the newer primary `BEADS_DOLT_SERVER_PORT=28231` this rig's shell exports.
Verified two ways this is not a real gate blocker:
- **Correct test entrypoint.** `scripts/test.sh` sources `scripts/ci/lib/test-env.sh`, whose `beads_test_env_enter()` explicitly `unset`s `BEADS_DOLT_SERVER_PORT`/`BEADS_DOLT_PORT` (and other ambient vars) before running Go tests, for a hermetic environment. Running the canonical `./scripts/test.sh -v ./internal/storage/dolt/...` on the same commit produces **173 PASS / 0 FAIL** — matching the reviewer's reported result exactly.
- **Merge-result cross-check.** Before finding the above, I also built a local, unpushed merge-simulation commit (`git commit-tree <merge-tree> -p origin/main -p 2d20764f4`) and ran the raw (non-hermetic) suite there: 181 PASS / 0 FAIL, 0 skipped-as-fail. `store.go`/`store_unit_test.go` changed substantially on `origin/main` since this branch's merge-base (`internal/storage/dolt/store.go` +213/−?, `store_unit_test.go` +66, via PR #3632/AD-01's production-port firewall generalization), and since be-cfm3z's diff doesn't touch either file, merging cleanly inherits main's fix regardless of test entrypoint.
One correction to the reviewer's stated rationale: the reviewer attributed their clean 173/173 run to "this branch's rebase onto a post-#3632 main," but `git merge-base --is-ancestor <PR-3632-merge-commit> 2d20764f4` returns false — PR #3632 is **not** actually an ancestor of the reviewed commit. The reviewer's bottom-line result (173/173, PASS) is fully correct and independently reproduced; the actual reason is that `scripts/test.sh`'s hermetic env wrapper scrubs the ambient port vars, making the branch's relationship to PR #3632 irrelevant to this specific test outcome. Recorded here so the next deployer doesn't have to re-derive it. Not a blocker for this deploy.
## Hand-off
- Push: `deploy/be-k4hyk-gate` → `sec003` (`quad341/beads-sec003-contrib`) — re-verified current precedent via `gh pr view 5028 --json headRepository` before pushing (origin is push-disabled upstream `gastownhall/beads`; `fork` and `sec003` both resolve to `quad341`-owned remotes in this rig, `sec003` is the active convention per the immediately-prior deploy be-q5hh6).
- PR: cross-repo `quad341:deploy/be-k4hyk-gate` → `gastownhall:main`.
- Merge decision routed to mayor (deployer does not merge).
- No follow-ups required: be-cfm3z's Part 2 (store.go firewall generalization) was deliberately deferred and is superseded by already-merged PR #3632/AD-01.