beads/internal/testutil/testdoltserver.go
Jim Wordelman a4bf13f042
Fix: test suites fail closed instead of open on the ambient Dolt port when no test container is started (be-napos) (#5885)
* test(dolt): red — testdb_* leak onto production Dolt port when Docker absent (refs be-h2h25)

* feat: green — Test suites create testdb_* on the PRODUCTION dolt server: harness fails open on the ambient port when Docker is absent (refs be-h2h25)

* docs(release-gate): add be-napos gate — ambient Dolt port fail-open to fail-closed

Deploy bead be-napos (build be-h2h25, review be-7fym1, verdict pass round 2).
7/7 gate criteria PASS; full independent evidence in the gate doc.

refs be-napos, be-h2h25, be-7fym1

* docs(release-gate): record PR link for be-napos gate

refs be-napos

* test(testutil): make the fail-closed gate deterministic, not Docker-conditional

The regression test skipped itself on every host that has Docker, by calling
the real EnsureDoltContainerForTestMain and skipping when it succeeded. That
asserted nothing on CI -- the only lane where it matters -- and on a
Docker-having host it also started the shared singleton container purely to
decide to skip, with no TestMain in internal/testutil to terminate it.

Add checkDoltFn, a package-level seam over checkDolt, so the test can force a
not-ready state without a Docker daemon and without starting a container. Only
EnsureDoltContainerForTestMain routes through it; the neighbouring t.Skip call
sites gate a skip rather than the fail-closed behaviour, so they keep calling
checkDolt directly.

The gate now covers all four not-ready states rather than the one the old test
happened to hit, which also pins the intent that an explicit
BEADS_TEST_SKIP=dolt opt-out must fail closed exactly as a Docker-less host
does. Split into a platform-independent half (the helper's own contract, both
variables cleared, idempotent) and a !windows half (the wiring, via the seam).

Red/green: with the neutralizeAmbientDoltPort call removed from the not-ready
branch, 4 of 4 subtests fail and 0 pass; restored, 4 of 4 pass in 0.06s with
no Docker and no container. (refs be-napos)

* fix(testutil): clear the ambient Dolt port on the Windows stub too

The Windows EnsureDoltContainerForTestMain returned its error without touching
BEADS_DOLT_SERVER_PORT / BEADS_DOLT_PORT, so the fail-open this branch fixes
still shipped there -- unconditionally, since that stub never succeeds. Windows
is a real lane: .github/workflows/pr.yml runs windows-latest on four jobs plus
the cross-platform matrix.

Gating it needed one pre-existing repair. testdoltserver_env_test.go has no
build tag but calls checkDolt, which is !windows-only, so the internal/testutil
test binary has never compiled for Windows -- reproduced identically on
origin/main. Tagging that file !windows fixes the whole package's Windows test
lane; without it a new Windows gate would compile nowhere and assert nothing,
which is the same defect this branch is already correcting.

Verified GOOS=windows go vet ./internal/testutil/ clean (it fails on
origin/main), and GOOS=windows go build clean before and after. The Windows
gate itself is compile-verified only -- no Windows host here to execute it.
(refs be-napos)

* docs(testing): state that ambient BEADS_DOLT_*_PORT is never honored

The guard fires on every not-ready state, not only "Docker unavailable" -- a
missing or wrong-version image and an explicit BEADS_TEST_SKIP=dolt opt-out
clear the ambient port too. That is deliberate: the invariant is the absence of
a test container, not the reason for it. Say so where the next person will look
before investigating, and name the supported way to target a specific server.
(refs be-napos)

* chore: remove leaked internal release-gate doc from PR branch

  release-gates/be-napos-ambient-dolt-port-failclosed-gate.md

Tracked our internal gc-management/beads-rig release-gate checklist and
should never have been committed to this upstream branch -- it's process
residue from our deploy tooling, not part of the change itself.

Per maintainer bee-ghosttrack's request during review of PR #6571. The
root cause (our deploy tooling auto-committing this file) is already fixed
as of gm-qatxl1; this branch predates that fix, so this commit is cleanup
only -- no other content on this branch is touched.

Ref: be-lk7o3

* chore(bazel): enumerate the PR's new test files in BUILD.bazel

Base added BUILD.bazel to internal/testutil and internal/storage/dolt after
this PR was opened (#6881 made the Bazel lane real, and fork PRs always run
it). The four new test files this PR adds were therefore absent from the
go_test srcs lists, which the bazel lane's equivalence check reports as a
missing target.

Generated by 'make bazel-sync' (gazelle), not hand-edited, because the CI
drift step diffs against exactly what gazelle emits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(testutil): reconcile the shared-server harness with the fail-closed clear

Review iterations 1-2 (adopt-pr, PR #5885): every gating finding from both
review syntheses, plus their minors and nits.

Iteration 1, Major 1 (Change Impact / Blast Radius). scripts/test.sh's
BEADS_TEST_SHARED_SERVER=1 flow starts a dockerless `dolt sql-server` and
exports BEADS_DOLT_PORT for every package to reuse. checkDolt() has no notion
of an already-provisioned shared server, so on that flow's primary audience --
a host without Docker -- all 12 EnsureDoltContainerForTestMain callers erased
the harness's own sanctioned port before m.Run(), turning green dockerless runs
red, while engdocs/CI_TEST_SURFACE_AUDIT.md and the new TESTING.md paragraph
advertised the two mechanisms unreconciled.

Taken: a dedicated marker, BEADS_TEST_SHARED_DOLT_SERVER, exported by
scripts/test.sh beside the port it allocated, which the helper's failure paths
honor. Both docs state the reconciliation, including the precedence: a
container, where one can be started, still overwrites both variables. Not
taken: "skip the clear when BEADS_TEST_SHARED_SERVER=1 and a port var is
already set". That flag is overloaded (cmd/bd/shared_server_integration_test.go
reads it too), and the script starts no server when a port is already set, so
the flag beside a port is exactly the case where the port is ambient.

Iteration 2, blocker (Behavioral Correctness). Iteration 1 shipped that marker
as a boolean ("1") which spared BOTH port variables, while the script vouched
only for the BEADS_DOLT_PORT it exported and its entry guard never looked at
BEADS_DOLT_SERVER_PORT -- the variable applyConfigDefaults resolves first. The
runner's test environment (beads_test_env_enter) unsets both variables before
the flow runs, so the hole needed that isolation skipped, e.g. with the
documented BEADS_TEST_ENV_DISABLE=1; then an inherited BEADS_DOLT_SERVER_PORT
reached `go test` beside the marker and outranked the shared server --
gm-2g3g5r, re-opened by its own fix. Measured with a fake `go` that prints its
environment: SERVER_PORT=59999 PORT=<shared> MARKER=1 under the iteration-1
script. Now:

  - the marker's value is the port it vouches for, and the helper decides per
    variable (neutralizeUnvouchedDoltPorts): only a variable holding exactly
    that port survives, and says so on stderr; every other one is cleared,
    marker or not;
  - the flow starts only when neither port variable is set, and says on stderr
    when BEADS_TEST_SHARED_SERVER=1 is ignored for that reason.

Not taken: also exporting BEADS_DOLT_SERVER_PORT from the flow. The widened
guard means it is always unset there, so the export adds no safety, and five
sites read that variable alone (internal/doltserver/doltserver.go:796,
cmd/bd/init.go:773 and :1351, cmd/bd/bootstrap.go:857,
internal/testutil/testdoltbranch.go:164) -- consumers the flow has never fed.

Iteration 2, minor (Test Evidence Quality). The exception's second arm, a
container that would not start, was reached by no test with its guard live.
Both failure paths now share one clear, and the container start is a seam
(startSharedContainerFn, pinned to startSharedContainer by identity) so tests
reach that path without consuming doltServerOnce.
TestEnsureDoltContainerForTestMain_KeepsHarnessProvisionedPort runs its whole
table, marked_but_divergent_var included, through both paths.

Iteration 2, minor (Contract & Interface Fidelity). The marker prose in
TESTING.md, CI_TEST_SURFACE_AUDIT.md, test.sh and testdoltserver.go describes
the per-port behavior the code now implements.

Iteration 1, Major 2 (Contract & Interface Fidelity). neutralizeAmbientDoltPort's
doc comment claimed "With no port resolvable, applyConfigDefaults'
BEADS_TEST_MODE guard forces port 1". Clearing the env vars is not "no port
resolvable": resolution falls through to the file chain and derives a
directory from filepath.Dir(cfg.Path) when BeadsDir is empty
(store.go:1661-1664), so a reachable .beads/dolt-server.port still yields a
non-zero port and the ServerPort == 0 arm (store.go:1683-1687) never fires;
production-port detection is strictly narrower than resolution. Narrowed to
what the change actually guarantees -- no environment-named port is honored --
at all three sites carrying the claim: the helper doc, the TESTING.md
paragraph, and TestApplyConfigDefaults_NoResolvablePortFailsClosed's own
comment, whose fixture (an empty beadsDir) is what excludes the surviving
path. The residual stays tracked in be-rl6tm. That test's stale
cross-reference to a renamed test is corrected in the same pass.

Iteration 1, minor (Test Evidence Quality). The Windows gate's comment claimed
a CI lane it does not have; it now states the coverage exactly --
compile-checked, not CI-executed -- and lists the four windows-latest jobs'
named selectors.

Iteration 1, nits. doltServerErr is read under doltServerMu via
sharedContainerErr(); TestCheckDoltFn_DefaultsToRealProbe pins function
identity with reflect instead of asserting non-nil, which any leaked stub
satisfied.

Mutation ledger (scoped `go test ./internal/testutil/`, scrubbed env; every
mutant restored from a copy and verified byte-identical):

  a set marker spares every    marked_but_divergent_var, divergent_legacy_var,
  variable (the i2 blocker)    marker_names_neither_var and boolean_marker red
                               on BOTH failure paths
  the marker covers            marked_but_divergent_var, marker_names_neither_var
  BEADS_DOLT_PORT only         and boolean_marker red on both paths
  presence instead of value    the same red set as the first row
  no clear after a failed      the 5 container_start_failed cases that expect
  container start              a clear red; every not_ready case stays green
  start attempted before the   all four incumbent fail-closed subtests and
  probe                        every not_ready case red
  leak a wrapper into          StartSharedContainerFn_DefaultsToRealStart red
  startSharedContainerFn
  delete the hermetic          ClearsAmbientPortWhenNotReady red
  t.Setenv, marker ambient
  at the fixture's port

Full package green with Docker, including the real-container
EnsureDoltContainerForTestMain path (43 test and subtest passes, no skips);
`go test ./scripts/`, which drives test.sh, green; `GOOS=windows go vet
./internal/testutil/` clean; `tools/bazel/go_srcs.py --check` clean, and no
BUILD change is due (no new files, no new non-stdlib imports).

Deferred deliberately: the iteration-1 synthesis's optional windows-latest step
for ./internal/testutil/. It is newly feasible -- this branch is what made the
package's Windows test binary compile -- but it would also newly execute four
pre-existing test functions in this package that have never run on Windows, so
it belongs in its own change. RequireDoltContainer's unsynchronized
doltServerErr read is base-owned and left alone for the same reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julian@gastownhall.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
2026-10-01 05:21:49 -07:00

720 lines
26 KiB
Go

//go:build !windows
package testutil
import (
"context"
"database/sql"
"fmt"
"io"
"os"
"os/exec"
"strconv"
"strings"
"sync"
"testing"
"time"
_ "github.com/go-sql-driver/mysql" // required by testcontainers Dolt module
"github.com/testcontainers/testcontainers-go"
tcexec "github.com/testcontainers/testcontainers-go/exec"
"github.com/testcontainers/testcontainers-go/modules/dolt"
)
// doltServer represents a running test Dolt container instance.
type doltServer struct {
container *dolt.DoltContainer
local *localDoltServer // set instead of container by the local backend
}
// serverStartTimeout is the max time to wait for the test Dolt server to accept connections.
const serverStartTimeout = 60 * time.Second
// Module-level singleton state.
var (
doltServerOnce sync.Once
doltServerMu sync.Mutex
doltServerErr error
doltTestPort string
doltSingletonSrv *doltServer
doltTerminateOnce sync.Once
dockerOnce sync.Once
dockerAvail bool
doltCheckOnce sync.Once
doltCached doltReadiness
)
// doltReadiness describes why Dolt integration tests can or cannot run.
type doltReadiness int
// doltDockerRepo is the repository portion of DoltDockerImage (without the tag).
var doltDockerRepo, _, _ = strings.Cut(DoltDockerImage, ":")
const (
doltNoDocker doltReadiness = iota // Docker daemon not reachable
doltNoImage // no Dolt image at all
doltWrongVersion // image exists but wrong tag
doltSkipped // explicit opt-out via BEADS_TEST_SKIP
doltReady // ready to start containers
doltNoLocalCLI // local backend: pinned dolt CLI missing or wrong version, or a bad backend name
)
// doltLocalCLIErr explains doltNoLocalCLI.
var doltLocalCLIErr error
func (d doltReadiness) String() string {
switch d {
case doltNoDocker:
return "Docker not available"
case doltNoImage:
return fmt.Sprintf("Docker image %s not cached locally (run 'docker pull %s')", DoltDockerImage, DoltDockerImage)
case doltWrongVersion:
return fmt.Sprintf("Docker image %s cached but wrong version (run 'docker pull %s')", doltDockerRepo, DoltDockerImage)
case doltSkipped:
return "Dolt tests skipped (BEADS_TEST_SKIP=dolt)"
case doltReady:
return "Dolt ready"
case doltNoLocalCLI:
return fmt.Sprintf("local Dolt test server backend unavailable: %v", doltLocalCLIErr)
default:
return fmt.Sprintf("unknown dolt readiness state: %d", int(d))
}
}
// isDockerAvailable returns true if the Docker daemon is reachable.
// The result is cached after the first call.
func isDockerAvailable() bool {
dockerOnce.Do(func() {
dockerAvail = exec.Command("docker", "info").Run() == nil
})
return dockerAvail
}
// hasTestSkip returns true if the given service appears in the BEADS_TEST_SKIP
// env var (comma-separated list). Example: BEADS_TEST_SKIP=dolt,slow
func hasTestSkip(service string) bool {
val := os.Getenv("BEADS_TEST_SKIP")
if val == "" {
return false
}
for _, s := range strings.Split(val, ",") {
if strings.TrimSpace(s) == service {
return true
}
}
return false
}
func skipOrFailDoltUnavailable(t *testing.T, state doltReadiness) {
t.Helper()
if os.Getenv(EnvRequireDoltContainer) == "1" {
t.Fatalf("Dolt test container unavailable (%s) but %s=1; this lane must not skip", state, EnvRequireDoltContainer)
}
t.Skipf("skipping test: %s", state)
}
// checkDolt returns the readiness state for Dolt integration tests.
// It composes hasTestSkip, isDockerAvailable, isDoltImageCached, and
// isDoltRepoImageCached, caching the result.
func checkDolt() doltReadiness {
doltCheckOnce.Do(func() {
// Explicit skip checked first to avoid ~1s docker info cost.
if hasTestSkip("dolt") {
doltCached = doltSkipped
return
}
if err := doltBackendErr(); err != nil {
doltLocalCLIErr = err
doltCached = doltNoLocalCLI
return
}
if useLocalDoltServer() {
if _, err := resolveLocalDoltBinary(); err != nil {
doltLocalCLIErr = err
doltCached = doltNoLocalCLI
return
}
doltCached = doltReady
return
}
if !isDockerAvailable() {
return // doltCached zero value is doltNoDocker
}
if isDoltImageCached() {
doltCached = doltReady
return
}
if isDoltRepoImageCached() {
doltCached = doltWrongVersion
return
}
doltCached = doltNoImage
})
return doltCached
}
// isDoltImageCached returns true if the exact Dolt Docker image (repo:tag)
// is available locally, avoiding unnecessary network calls to Docker Hub.
func isDoltImageCached() bool {
return exec.Command("docker", "image", "inspect", DoltDockerImage).Run() == nil
}
// isDoltRepoImageCached returns true if ANY version of the Dolt image repo
// exists locally (e.g. dolthub/dolt-sql-server with a different tag).
func isDoltRepoImageCached() bool {
out, err := exec.Command("docker", "images", doltDockerRepo, "-q").Output()
return err == nil && len(strings.TrimSpace(string(out))) > 0
}
// startDoltContainer starts the singleton Dolt container.
func startDoltContainer() error {
if useLocalDoltServer() {
s, err := startLocalDoltServer()
if err != nil {
return fmt.Errorf("starting local Dolt server: %w", err)
}
doltTestPort = strconv.Itoa(s.Port())
doltSingletonSrv = &doltServer{local: s}
return nil
}
ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout)
defer cancel()
ctr, err := dolt.Run(ctx, DoltDockerImage,
dolt.WithDatabase("beads_test"),
// Docker port-forwarding makes connections appear as non-localhost
// (e.g., 172.17.0.1). The entrypoint defaults DOLT_ROOT_HOST to
// "localhost", so root@localhost won't match external connections.
// Set to "%" so root can connect from any host.
testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}),
)
if err != nil {
return fmt.Errorf("starting Dolt container: %w", err)
}
p, err := ctr.MappedPort(ctx, "3306/tcp")
if err != nil {
_ = testcontainers.TerminateContainer(ctr)
return fmt.Errorf("getting mapped port: %w", err)
}
if _, err := strconv.Atoi(p.Port()); err != nil {
_ = testcontainers.TerminateContainer(ctr)
return fmt.Errorf("parsing port %q: %w", p.Port(), err)
}
doltTestPort = p.Port()
if err := waitForDoltReady(doltTestPort); err != nil {
_ = testcontainers.TerminateContainer(ctr)
return fmt.Errorf("waiting for Dolt server to be query-ready: %w", err)
}
doltSingletonSrv = &doltServer{
container: ctr,
}
return nil
}
// doltReadyProbeTimeout bounds how long waitForDoltReady polls before giving up.
const doltReadyProbeTimeout = 30 * time.Second
// waitForDoltReady polls the server with a trivial query until it responds
// or doltReadyProbeTimeout elapses.
//
// The testcontainers wait strategy above only matches a log line ("Server
// ready. Accepting connections."), which confirms the TCP listener is up but
// not that Dolt's SQL engine can actually serve a query yet. In that narrow
// startup window, a query issued with an unbounded context (as several test
// helpers do) can block indefinitely instead of erroring, because nothing
// ever cancels it to unstick the read — confirmed via goroutine dump during
// be-fgd round-2 triage: the connection sat in mysqlConn.readWithTimeout /
// net.Read for 2+ minutes after the container had already logged ready. Each
// probe attempt here uses its own short bounded context, so a still-warming
// server fails an attempt fast and gets retried, rather than wedging.
func waitForDoltReady(port string) error {
dsn := fmt.Sprintf("root@tcp(127.0.0.1:%s)/", port)
deadline := time.Now().Add(doltReadyProbeTimeout)
var lastErr error
for {
lastErr = pingDoltOnce(dsn)
if lastErr == nil {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("dolt server at port %s did not become query-ready within %s: %w", port, doltReadyProbeTimeout, lastErr)
}
time.Sleep(200 * time.Millisecond)
}
}
// pingDoltOnce opens a short-lived connection and pings it with a bounded
// context so a non-responsive server fails this attempt instead of hanging.
func pingDoltOnce(dsn string) error {
db, err := sql.Open("mysql", dsn)
if err != nil {
return err
}
defer db.Close()
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
return db.PingContext(ctx)
}
// terminateSharedContainer stops and removes the shared Dolt container.
// Safe to call concurrently or multiple times (sync.Once).
func terminateSharedContainer() {
doltTerminateOnce.Do(func() {
doltServerMu.Lock()
defer doltServerMu.Unlock()
stopSharedContainerLocked()
})
}
// stopSharedContainerLocked drops the current singleton container. The caller
// holds doltServerMu. It does not consume doltTerminateOnce, so a later
// replacement can still be removed by TerminateDoltContainer.
func stopSharedContainerLocked() {
if doltSingletonSrv != nil && doltSingletonSrv.local != nil {
doltSingletonSrv.local.terminate()
doltSingletonSrv.local = nil
}
if doltSingletonSrv != nil && doltSingletonSrv.container != nil {
_ = testcontainers.TerminateContainer(doltSingletonSrv.container)
doltSingletonSrv.container = nil
}
doltSingletonSrv = nil
}
// RestartSharedDoltContainer replaces the shared test container with a new
// one and returns its host port. A package that keeps one container for the
// whole TestMain (tracker, and the same shape elsewhere) otherwise turns a
// single server death into a failure for every later test: each dial reports
// "Dolt server unreachable" and the job goes red dozens of times. The
// replacement is empty; the caller re-creates its shared database.
func RestartSharedDoltContainer() (int, error) {
if state := checkDolt(); state != doltReady {
return 0, fmt.Errorf("%s", state)
}
doltServerMu.Lock()
defer doltServerMu.Unlock()
stopSharedContainerLocked()
if err := startDoltContainer(); err != nil {
doltServerErr = err
return 0, err
}
if err := os.Setenv("BEADS_DOLT_PORT", doltTestPort); err != nil {
doltServerErr = fmt.Errorf("set BEADS_DOLT_PORT: %w", err)
return 0, doltServerErr
}
if err := os.Setenv("BEADS_DOLT_SERVER_PORT", doltTestPort); err != nil {
doltServerErr = fmt.Errorf("set BEADS_DOLT_SERVER_PORT: %w", err)
return 0, doltServerErr
}
doltServerErr = nil
return DoltContainerPortInt(), nil
}
// ServerUnreachable reports whether err is a shared Dolt server that is gone,
// as opposed to a query the server refused. Connection refused and the
// "Dolt server unreachable" dial error are the shapes left behind when the
// singleton container exits; unexpected EOF is the MySQL driver watching that
// exit mid-handshake.
func ServerUnreachable(err error) bool {
if err == nil {
return false
}
if DoltContainerCrashed() {
return true
}
msg := strings.ToLower(err.Error())
return strings.Contains(msg, "connection refused") ||
strings.Contains(msg, "dolt server unreachable") ||
strings.Contains(msg, "unexpected eof")
}
// IsolatedDoltContainer is a per-test Dolt container together with the
// accessors a test needs to inspect it from the inside.
type IsolatedDoltContainer struct {
// Port is the mapped host port of the container's Dolt server.
Port string
ctr *dolt.DoltContainer
local *localDoltServer // local backend instead of ctr
}
// Stop pauses the container while retaining its data volume.
func (c *IsolatedDoltContainer) Stop(ctx context.Context) error {
if c != nil && c.local != nil {
return c.local.stop()
}
if c == nil || c.ctr == nil {
return fmt.Errorf("no Dolt container running")
}
return c.ctr.Stop(ctx, nil)
}
// Start resumes a container previously stopped with Stop.
func (c *IsolatedDoltContainer) Start(ctx context.Context) error {
if c != nil && c.local != nil {
return c.local.restart()
}
if c == nil || c.ctr == nil {
return fmt.Errorf("no Dolt container running")
}
return c.ctr.Start(ctx)
}
// CurrentPort returns the container's currently mapped SQL port. Container
// runtimes may assign a new host port after Stop/Start.
func (c *IsolatedDoltContainer) CurrentPort(ctx context.Context) (string, error) {
if c != nil && c.local != nil {
return strconv.Itoa(c.local.Port()), nil
}
if c == nil || c.ctr == nil {
return "", fmt.Errorf("no Dolt container running")
}
p, err := c.ctr.MappedPort(ctx, "3306/tcp")
if err != nil {
return "", err
}
return p.Port(), nil
}
// Exec runs cmd inside the container and returns its exit code and combined
// output, demultiplexed (see containerExec). Its working directory is the
// server's data directory (the image's WORKDIR, /var/lib/dolt); with the local
// backend cmd runs on the host in the local server's data directory, so
// commands must address the data through relative paths.
func (c *IsolatedDoltContainer) Exec(ctx context.Context, cmd []string) (int, string, error) {
if c != nil && c.local != nil {
return c.local.execInDataDir(ctx, cmd)
}
if c == nil || c.ctr == nil {
return 0, "", fmt.Errorf("no Dolt container running")
}
return containerExec(ctx, c.ctr, cmd)
}
// StartIsolatedDoltContainerHandle starts a per-test Dolt container and
// returns a handle to it. The container is terminated automatically when the
// test finishes.
//
// Unlike StartIsolatedDoltContainer this does NOT touch BEADS_DOLT_PORT or
// BEADS_DOLT_SERVER_PORT: those are process-wide, so a test that only wants
// its own server should not perturb sibling tests sharing the process.
func StartIsolatedDoltContainerHandle(t *testing.T) *IsolatedDoltContainer {
t.Helper()
if state := checkDolt(); state != doltReady {
skipOrFailDoltUnavailable(t, state)
}
if useLocalDoltServer() {
return startIsolatedLocalDoltServer(t)
}
return startIsolatedDoltContainer(t)
}
// startIsolatedLocalDoltServer is StartIsolatedDoltContainerHandle for the
// local backend, without the readiness gate.
func startIsolatedLocalDoltServer(t *testing.T) *IsolatedDoltContainer {
t.Helper()
s, err := startLocalDoltServer()
if err != nil {
t.Fatalf("starting local Dolt server: %v", err)
}
t.Cleanup(s.terminate)
return &IsolatedDoltContainer{Port: strconv.Itoa(s.Port()), local: s}
}
// startIsolatedDoltContainer is StartIsolatedDoltContainerHandle for the
// container backend, without the readiness gate.
func startIsolatedDoltContainer(t *testing.T) *IsolatedDoltContainer {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout)
defer cancel()
ctr, err := dolt.Run(ctx, DoltDockerImage,
dolt.WithDatabase("beads_test"),
testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}),
)
if err != nil {
t.Fatalf("starting Dolt container: %v", err)
}
t.Cleanup(func() {
if err := testcontainers.TerminateContainer(ctr); err != nil {
t.Logf("terminating Dolt container: %v", err)
}
})
port, err := ctr.MappedPort(ctx, "3306/tcp")
if err != nil {
t.Fatalf("getting mapped port: %v", err)
}
return &IsolatedDoltContainer{Port: port.Port(), ctr: ctr}
}
// StartIsolatedDoltContainer starts a per-test Dolt container and returns the
// mapped host port, additionally pointing BEADS_DOLT_PORT and
// BEADS_DOLT_SERVER_PORT at it for the duration of the test.
func StartIsolatedDoltContainer(t *testing.T) string {
t.Helper()
c := StartIsolatedDoltContainerHandle(t)
t.Setenv("BEADS_DOLT_PORT", c.Port)
t.Setenv("BEADS_DOLT_SERVER_PORT", c.Port)
return c.Port
}
// ensureSharedContainer starts the singleton container and sets
// BEADS_DOLT_PORT and BEADS_DOLT_SERVER_PORT.
func ensureSharedContainer() {
doltServerOnce.Do(func() {
doltServerMu.Lock()
defer doltServerMu.Unlock()
doltServerErr = startDoltContainer()
if doltServerErr == nil && doltTestPort != "" {
if err := os.Setenv("BEADS_DOLT_PORT", doltTestPort); err != nil {
doltServerErr = fmt.Errorf("set BEADS_DOLT_PORT: %w", err)
} else if err := os.Setenv("BEADS_DOLT_SERVER_PORT", doltTestPort); err != nil {
doltServerErr = fmt.Errorf("set BEADS_DOLT_SERVER_PORT: %w", err)
}
}
})
}
// checkDoltFn is the readiness probe used by EnsureDoltContainerForTestMain.
// It is a variable purely so the fail-closed regression test can force a
// not-ready state deterministically -- without a Docker daemon, and without
// starting the shared singleton container just to observe the failure path.
// Production code never reassigns it. The direct checkDolt() calls in the
// t.Skip paths above and below are deliberately left alone: they gate a skip,
// not the fail-closed behavior this seam exists to test.
var checkDoltFn = checkDolt
// startSharedContainerFn starts the shared container once checkDoltFn reports
// ready. It is a variable for the same reason checkDoltFn is: the fail-closed
// tests reach the container-would-not-start path through it without consuming
// doltServerOnce, which would leave this package's real-container test running
// against a singleton that has already failed. Production code never
// reassigns it.
var startSharedContainerFn = startSharedContainer
// EnvSharedDoltServer names the one Dolt port that a test harness started a
// server on for this process tree, as opposed to a port inherited from the
// surrounding environment. Its value is that port number: scripts/test.sh sets
// it to the same value as the BEADS_DOLT_PORT it exports for the dockerless
// `dolt sql-server` of its BEADS_TEST_SHARED_SERVER=1 flow. It vouches for that
// value only, never for the process: a port variable holding anything else is
// still ambient, however it got there.
//
// It is deliberately a distinct name from BEADS_TEST_SHARED_SERVER. That flag
// is also read by cmd/bd's shared-server integration test for an unrelated
// purpose, and it only says a shared server was asked for, not that one was
// started: the flow starts none when a port variable is already set, so the
// flag beside a port cannot tell the script's own port from an inherited one.
const EnvSharedDoltServer = "BEADS_TEST_SHARED_DOLT_SERVER"
// sharedContainerErr reads the shared-container error under doltServerMu.
// Every write to doltServerErr takes that mutex (ensureSharedContainer,
// RestartSharedDoltContainer), so reads belong under it too; callers must not
// already hold the mutex.
func sharedContainerErr() error {
doltServerMu.Lock()
defer doltServerMu.Unlock()
return doltServerErr
}
// startSharedContainer is ensureSharedContainer plus the error it recorded.
func startSharedContainer() error {
ensureSharedContainer()
return sharedContainerErr()
}
// EnsureDoltContainerForTestMain starts a shared Dolt container for use in
// TestMain functions. Call TerminateDoltContainer() after m.Run() to clean up.
// On success it sets BEADS_DOLT_PORT and BEADS_DOLT_SERVER_PORT process-wide,
// which also means a container, when one can be started, takes precedence over
// any port the environment already named.
//
// On ANY failure -- Docker missing, image missing or wrong version, an
// explicit BEADS_TEST_SKIP=dolt opt-out, or a container that would not start
// -- it instead clears both variables (see neutralizeAmbientDoltPort) and
// returns an error. Callers that warn and run the suite anyway therefore
// cannot resolve a store onto an ambient server.
//
// The single exception is a variable holding exactly the port that
// EnvSharedDoltServer names, i.e. one a harness provisioned for this process
// tree: that is a dedicated test server and survives the failure paths (see
// neutralizeUnvouchedDoltPorts). Without that exception this function would
// erase the port scripts/test.sh exports for its own shared `dolt sql-server`,
// which on a Docker-less host is every one of these TestMains.
func EnsureDoltContainerForTestMain() error {
var err error
if state := checkDoltFn(); state != doltReady {
err = fmt.Errorf("%s", state)
} else {
err = startSharedContainerFn()
}
// One clear serves both failure paths, so "no usable container" and "a
// container that would not start" cannot drift apart.
if err != nil {
neutralizeUnvouchedDoltPorts()
}
return err
}
// neutralizeUnvouchedDoltPorts is the clear EnsureDoltContainerForTestMain
// applies when it has no container: neutralizeAmbientDoltPort, except that a
// port variable holding exactly the port EnvSharedDoltServer names survives,
// and says so on stderr, because the suite is about to talk to that server.
//
// The decision is made per variable, not per process. scripts/test.sh vouches
// for the BEADS_DOLT_PORT it exports; an unrelated BEADS_DOLT_SERVER_PORT
// beside it -- which applyConfigDefaults in internal/storage/dolt resolves
// FIRST, falling back to BEADS_DOLT_PORT only when it is empty -- is cleared
// even though the marker is set, or it would outrank the harness's server and
// carry the suite onto whatever server it names (gm-2g3g5r).
func neutralizeUnvouchedDoltPorts() {
vouched := os.Getenv(EnvSharedDoltServer)
if vouched == "" {
neutralizeAmbientDoltPort()
return
}
for _, name := range []string{"BEADS_DOLT_SERVER_PORT", "BEADS_DOLT_PORT"} {
if os.Getenv(name) == vouched {
fmt.Fprintf(os.Stderr, "testutil: keeping %s=%s, the harness-provisioned test server named by %s\n",
name, vouched, EnvSharedDoltServer)
continue
}
_ = os.Unsetenv(name)
}
}
// neutralizeAmbientDoltPort clears the inherited connection-port variables so
// that a TestMain which warns-and-continues without a container cannot resolve
// a store onto whatever server the ambient environment names -- in a gc-managed
// city, the production one (gm-2g3g5r).
//
// What this guarantees is exactly that: no ENVIRONMENT-NAMED port is honored.
// It is not the wider claim that no port is resolvable afterwards. Port
// resolution continues past the env vars into the file chain, and
// applyConfigDefaults derives a directory from filepath.Dir(cfg.Path) when
// cfg.BeadsDir is empty (internal/storage/dolt/store.go:1661-1664), so a
// reachable .beads/dolt-server.port can still supply a non-zero port -- the
// ServerPort == 0 arm of the BEADS_TEST_MODE guard (store.go:1683-1687) then
// never fires. Port 1 is forced only via isProductionPort, and
// production-port DETECTION is narrower than resolution: Rule 1 recognizes
// only DefaultSQLPort, and Rule 3 -- the only rule that sees a dynamic port --
// needs an explicit cfg.BeadsDir and is suppressed by the BEADS_TEST_SERVER=1
// that beads TestMains set. That residual is tracked in be-rl6tm; clearing the
// env vars removes the channel this leak actually traveled on, not every
// channel.
//
// This fires on EVERY not-ready state, not only "Docker unavailable": a
// missing or wrong-version image and an explicit BEADS_TEST_SKIP=dolt opt-out
// clear the ambient port too. That is deliberate. The invariant is about the
// absence of a test container, not about the reason for it -- an opted-out run
// has no more business reaching a shared server than a Docker-less one does.
func neutralizeAmbientDoltPort() {
_ = os.Unsetenv("BEADS_DOLT_SERVER_PORT")
_ = os.Unsetenv("BEADS_DOLT_PORT")
}
// RequireDoltContainer ensures a shared Dolt container is running. Skips the
// test if Docker is not available.
func RequireDoltContainer(t *testing.T) {
t.Helper()
if state := checkDolt(); state != doltReady {
skipOrFailDoltUnavailable(t, state)
}
ensureSharedContainer()
if doltServerErr != nil {
t.Fatalf("Dolt container setup failed: %v", doltServerErr)
}
}
// DoltContainerAddr returns the address (host:port) of the Dolt container.
func DoltContainerAddr() string {
return "127.0.0.1:" + doltTestPort
}
// DoltContainerPort returns the mapped host port of the Dolt container.
func DoltContainerPort() string {
return doltTestPort
}
// DoltContainerPortInt returns the mapped host port as an int.
func DoltContainerPortInt() int {
p, _ := strconv.Atoi(doltTestPort)
return p
}
// TerminateDoltContainer stops and removes the shared Dolt container.
// Called from TestMain after m.Run().
func TerminateDoltContainer() {
terminateSharedContainer()
}
// DoltContainerCrashed returns true if the shared container has exited unexpectedly.
// Returns false if no container was started.
func DoltContainerCrashed() bool {
if doltSingletonSrv != nil && doltSingletonSrv.local != nil {
gone, _ := doltSingletonSrv.local.exited()
return gone
}
if doltSingletonSrv == nil || doltSingletonSrv.container == nil {
return false
}
state, err := doltSingletonSrv.container.State(context.Background())
if err != nil {
return true // can't check state — assume crashed
}
return !state.Running
}
// DoltContainerCrashError returns an error if the shared container has exited
// unexpectedly, nil otherwise.
func DoltContainerCrashError() error {
if doltSingletonSrv != nil && doltSingletonSrv.local != nil {
if gone, err := doltSingletonSrv.local.exited(); gone {
return fmt.Errorf("local Dolt server exited (%v); log: %s", err, doltSingletonSrv.local.logSince(0))
}
return nil
}
if doltSingletonSrv == nil || doltSingletonSrv.container == nil {
return nil
}
state, err := doltSingletonSrv.container.State(context.Background())
if err != nil {
return fmt.Errorf("failed to check container state: %w", err)
}
if !state.Running {
return fmt.Errorf("Dolt container exited (status=%s, exit=%d)", state.Status, state.ExitCode)
}
return nil
}
// containerExec runs cmd inside ctr and returns its exit code and combined
// output. Used by tests that need to inspect a container's filesystem (e.g.
// the .dolt_dropped_databases/ directory), which has no host-visible path
// since the containers are started without a bind-mounted data dir.
//
// tcexec.Multiplexed() is load-bearing, not decorative. Without it Exec hands
// back the raw hijacked Docker stream and io.ReadAll glues the 8-byte stdcopy
// frame header onto the payload: `echo hello` returns
// "\x01\x00\x00\x00\x00\x00\x00\x06hello\n". A caller that feeds such
// a string back into the container — a path read out of find(1), say — then
// fails with "invalid argument" on a prefix it cannot see.
func containerExec(ctx context.Context, ctr *dolt.DoltContainer, cmd []string) (int, string, error) {
code, reader, err := ctr.Exec(ctx, cmd, tcexec.Multiplexed())
if err != nil {
return 0, "", fmt.Errorf("exec in Dolt container: %w", err)
}
out, err := io.ReadAll(reader)
if err != nil {
return code, "", fmt.Errorf("reading exec output: %w", err)
}
return code, string(out), nil
}