* test(dolt): red — testdb_* leak onto production Dolt port when Docker absent (refs be-h2h25) * feat: green — Test suites create testdb_* on the PRODUCTION dolt server: harness fails open on the ambient port when Docker is absent (refs be-h2h25) * docs(release-gate): add be-napos gate — ambient Dolt port fail-open to fail-closed Deploy bead be-napos (build be-h2h25, review be-7fym1, verdict pass round 2). 7/7 gate criteria PASS; full independent evidence in the gate doc. refs be-napos, be-h2h25, be-7fym1 * docs(release-gate): record PR link for be-napos gate refs be-napos * test(testutil): make the fail-closed gate deterministic, not Docker-conditional The regression test skipped itself on every host that has Docker, by calling the real EnsureDoltContainerForTestMain and skipping when it succeeded. That asserted nothing on CI -- the only lane where it matters -- and on a Docker-having host it also started the shared singleton container purely to decide to skip, with no TestMain in internal/testutil to terminate it. Add checkDoltFn, a package-level seam over checkDolt, so the test can force a not-ready state without a Docker daemon and without starting a container. Only EnsureDoltContainerForTestMain routes through it; the neighbouring t.Skip call sites gate a skip rather than the fail-closed behaviour, so they keep calling checkDolt directly. The gate now covers all four not-ready states rather than the one the old test happened to hit, which also pins the intent that an explicit BEADS_TEST_SKIP=dolt opt-out must fail closed exactly as a Docker-less host does. Split into a platform-independent half (the helper's own contract, both variables cleared, idempotent) and a !windows half (the wiring, via the seam). Red/green: with the neutralizeAmbientDoltPort call removed from the not-ready branch, 4 of 4 subtests fail and 0 pass; restored, 4 of 4 pass in 0.06s with no Docker and no container. (refs be-napos) * fix(testutil): clear the ambient Dolt port on the Windows stub too The Windows EnsureDoltContainerForTestMain returned its error without touching BEADS_DOLT_SERVER_PORT / BEADS_DOLT_PORT, so the fail-open this branch fixes still shipped there -- unconditionally, since that stub never succeeds. Windows is a real lane: .github/workflows/pr.yml runs windows-latest on four jobs plus the cross-platform matrix. Gating it needed one pre-existing repair. testdoltserver_env_test.go has no build tag but calls checkDolt, which is !windows-only, so the internal/testutil test binary has never compiled for Windows -- reproduced identically on origin/main. Tagging that file !windows fixes the whole package's Windows test lane; without it a new Windows gate would compile nowhere and assert nothing, which is the same defect this branch is already correcting. Verified GOOS=windows go vet ./internal/testutil/ clean (it fails on origin/main), and GOOS=windows go build clean before and after. The Windows gate itself is compile-verified only -- no Windows host here to execute it. (refs be-napos) * docs(testing): state that ambient BEADS_DOLT_*_PORT is never honored The guard fires on every not-ready state, not only "Docker unavailable" -- a missing or wrong-version image and an explicit BEADS_TEST_SKIP=dolt opt-out clear the ambient port too. That is deliberate: the invariant is the absence of a test container, not the reason for it. Say so where the next person will look before investigating, and name the supported way to target a specific server. (refs be-napos) * chore: remove leaked internal release-gate doc from PR branch release-gates/be-napos-ambient-dolt-port-failclosed-gate.md Tracked our internal gc-management/beads-rig release-gate checklist and should never have been committed to this upstream branch -- it's process residue from our deploy tooling, not part of the change itself. Per maintainer bee-ghosttrack's request during review of PR #6571. The root cause (our deploy tooling auto-committing this file) is already fixed as of gm-qatxl1; this branch predates that fix, so this commit is cleanup only -- no other content on this branch is touched. Ref: be-lk7o3 * chore(bazel): enumerate the PR's new test files in BUILD.bazel Base added BUILD.bazel to internal/testutil and internal/storage/dolt after this PR was opened (#6881 made the Bazel lane real, and fork PRs always run it). The four new test files this PR adds were therefore absent from the go_test srcs lists, which the bazel lane's equivalence check reports as a missing target. Generated by 'make bazel-sync' (gazelle), not hand-edited, because the CI drift step diffs against exactly what gazelle emits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(testutil): reconcile the shared-server harness with the fail-closed clear Review iterations 1-2 (adopt-pr, PR #5885): every gating finding from both review syntheses, plus their minors and nits. Iteration 1, Major 1 (Change Impact / Blast Radius). scripts/test.sh's BEADS_TEST_SHARED_SERVER=1 flow starts a dockerless `dolt sql-server` and exports BEADS_DOLT_PORT for every package to reuse. checkDolt() has no notion of an already-provisioned shared server, so on that flow's primary audience -- a host without Docker -- all 12 EnsureDoltContainerForTestMain callers erased the harness's own sanctioned port before m.Run(), turning green dockerless runs red, while engdocs/CI_TEST_SURFACE_AUDIT.md and the new TESTING.md paragraph advertised the two mechanisms unreconciled. Taken: a dedicated marker, BEADS_TEST_SHARED_DOLT_SERVER, exported by scripts/test.sh beside the port it allocated, which the helper's failure paths honor. Both docs state the reconciliation, including the precedence: a container, where one can be started, still overwrites both variables. Not taken: "skip the clear when BEADS_TEST_SHARED_SERVER=1 and a port var is already set". That flag is overloaded (cmd/bd/shared_server_integration_test.go reads it too), and the script starts no server when a port is already set, so the flag beside a port is exactly the case where the port is ambient. Iteration 2, blocker (Behavioral Correctness). Iteration 1 shipped that marker as a boolean ("1") which spared BOTH port variables, while the script vouched only for the BEADS_DOLT_PORT it exported and its entry guard never looked at BEADS_DOLT_SERVER_PORT -- the variable applyConfigDefaults resolves first. The runner's test environment (beads_test_env_enter) unsets both variables before the flow runs, so the hole needed that isolation skipped, e.g. with the documented BEADS_TEST_ENV_DISABLE=1; then an inherited BEADS_DOLT_SERVER_PORT reached `go test` beside the marker and outranked the shared server -- gm-2g3g5r, re-opened by its own fix. Measured with a fake `go` that prints its environment: SERVER_PORT=59999 PORT=<shared> MARKER=1 under the iteration-1 script. Now: - the marker's value is the port it vouches for, and the helper decides per variable (neutralizeUnvouchedDoltPorts): only a variable holding exactly that port survives, and says so on stderr; every other one is cleared, marker or not; - the flow starts only when neither port variable is set, and says on stderr when BEADS_TEST_SHARED_SERVER=1 is ignored for that reason. Not taken: also exporting BEADS_DOLT_SERVER_PORT from the flow. The widened guard means it is always unset there, so the export adds no safety, and five sites read that variable alone (internal/doltserver/doltserver.go:796, cmd/bd/init.go:773 and :1351, cmd/bd/bootstrap.go:857, internal/testutil/testdoltbranch.go:164) -- consumers the flow has never fed. Iteration 2, minor (Test Evidence Quality). The exception's second arm, a container that would not start, was reached by no test with its guard live. Both failure paths now share one clear, and the container start is a seam (startSharedContainerFn, pinned to startSharedContainer by identity) so tests reach that path without consuming doltServerOnce. TestEnsureDoltContainerForTestMain_KeepsHarnessProvisionedPort runs its whole table, marked_but_divergent_var included, through both paths. Iteration 2, minor (Contract & Interface Fidelity). The marker prose in TESTING.md, CI_TEST_SURFACE_AUDIT.md, test.sh and testdoltserver.go describes the per-port behavior the code now implements. Iteration 1, Major 2 (Contract & Interface Fidelity). neutralizeAmbientDoltPort's doc comment claimed "With no port resolvable, applyConfigDefaults' BEADS_TEST_MODE guard forces port 1". Clearing the env vars is not "no port resolvable": resolution falls through to the file chain and derives a directory from filepath.Dir(cfg.Path) when BeadsDir is empty (store.go:1661-1664), so a reachable .beads/dolt-server.port still yields a non-zero port and the ServerPort == 0 arm (store.go:1683-1687) never fires; production-port detection is strictly narrower than resolution. Narrowed to what the change actually guarantees -- no environment-named port is honored -- at all three sites carrying the claim: the helper doc, the TESTING.md paragraph, and TestApplyConfigDefaults_NoResolvablePortFailsClosed's own comment, whose fixture (an empty beadsDir) is what excludes the surviving path. The residual stays tracked in be-rl6tm. That test's stale cross-reference to a renamed test is corrected in the same pass. Iteration 1, minor (Test Evidence Quality). The Windows gate's comment claimed a CI lane it does not have; it now states the coverage exactly -- compile-checked, not CI-executed -- and lists the four windows-latest jobs' named selectors. Iteration 1, nits. doltServerErr is read under doltServerMu via sharedContainerErr(); TestCheckDoltFn_DefaultsToRealProbe pins function identity with reflect instead of asserting non-nil, which any leaked stub satisfied. Mutation ledger (scoped `go test ./internal/testutil/`, scrubbed env; every mutant restored from a copy and verified byte-identical): a set marker spares every marked_but_divergent_var, divergent_legacy_var, variable (the i2 blocker) marker_names_neither_var and boolean_marker red on BOTH failure paths the marker covers marked_but_divergent_var, marker_names_neither_var BEADS_DOLT_PORT only and boolean_marker red on both paths presence instead of value the same red set as the first row no clear after a failed the 5 container_start_failed cases that expect container start a clear red; every not_ready case stays green start attempted before the all four incumbent fail-closed subtests and probe every not_ready case red leak a wrapper into StartSharedContainerFn_DefaultsToRealStart red startSharedContainerFn delete the hermetic ClearsAmbientPortWhenNotReady red t.Setenv, marker ambient at the fixture's port Full package green with Docker, including the real-container EnsureDoltContainerForTestMain path (43 test and subtest passes, no skips); `go test ./scripts/`, which drives test.sh, green; `GOOS=windows go vet ./internal/testutil/` clean; `tools/bazel/go_srcs.py --check` clean, and no BUILD change is due (no new files, no new non-stdlib imports). Deferred deliberately: the iteration-1 synthesis's optional windows-latest step for ./internal/testutil/. It is newly feasible -- this branch is what made the package's Windows test binary compile -- but it would also newly execute four pre-existing test functions in this package that have never run on Windows, so it belongs in its own change. RequireDoltContainer's unsynchronized doltServerErr read is base-owned and left alone for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Julian Knutsen <julian@gastownhall.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
720 lines
26 KiB
Go
720 lines
26 KiB
Go
//go:build !windows
|
|
|
|
package testutil
|
|
|
|
import (
|
|
"context"
|
|
"database/sql"
|
|
"fmt"
|
|
"io"
|
|
"os"
|
|
"os/exec"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
_ "github.com/go-sql-driver/mysql" // required by testcontainers Dolt module
|
|
"github.com/testcontainers/testcontainers-go"
|
|
tcexec "github.com/testcontainers/testcontainers-go/exec"
|
|
"github.com/testcontainers/testcontainers-go/modules/dolt"
|
|
)
|
|
|
|
// doltServer represents a running test Dolt container instance.
|
|
type doltServer struct {
|
|
container *dolt.DoltContainer
|
|
local *localDoltServer // set instead of container by the local backend
|
|
}
|
|
|
|
// serverStartTimeout is the max time to wait for the test Dolt server to accept connections.
|
|
const serverStartTimeout = 60 * time.Second
|
|
|
|
// Module-level singleton state.
|
|
var (
|
|
doltServerOnce sync.Once
|
|
doltServerMu sync.Mutex
|
|
doltServerErr error
|
|
doltTestPort string
|
|
doltSingletonSrv *doltServer
|
|
doltTerminateOnce sync.Once
|
|
dockerOnce sync.Once
|
|
dockerAvail bool
|
|
doltCheckOnce sync.Once
|
|
doltCached doltReadiness
|
|
)
|
|
|
|
// doltReadiness describes why Dolt integration tests can or cannot run.
|
|
type doltReadiness int
|
|
|
|
// doltDockerRepo is the repository portion of DoltDockerImage (without the tag).
|
|
var doltDockerRepo, _, _ = strings.Cut(DoltDockerImage, ":")
|
|
|
|
const (
|
|
doltNoDocker doltReadiness = iota // Docker daemon not reachable
|
|
doltNoImage // no Dolt image at all
|
|
doltWrongVersion // image exists but wrong tag
|
|
doltSkipped // explicit opt-out via BEADS_TEST_SKIP
|
|
doltReady // ready to start containers
|
|
doltNoLocalCLI // local backend: pinned dolt CLI missing or wrong version, or a bad backend name
|
|
)
|
|
|
|
// doltLocalCLIErr explains doltNoLocalCLI.
|
|
var doltLocalCLIErr error
|
|
|
|
func (d doltReadiness) String() string {
|
|
switch d {
|
|
case doltNoDocker:
|
|
return "Docker not available"
|
|
case doltNoImage:
|
|
return fmt.Sprintf("Docker image %s not cached locally (run 'docker pull %s')", DoltDockerImage, DoltDockerImage)
|
|
case doltWrongVersion:
|
|
return fmt.Sprintf("Docker image %s cached but wrong version (run 'docker pull %s')", doltDockerRepo, DoltDockerImage)
|
|
case doltSkipped:
|
|
return "Dolt tests skipped (BEADS_TEST_SKIP=dolt)"
|
|
case doltReady:
|
|
return "Dolt ready"
|
|
case doltNoLocalCLI:
|
|
return fmt.Sprintf("local Dolt test server backend unavailable: %v", doltLocalCLIErr)
|
|
default:
|
|
return fmt.Sprintf("unknown dolt readiness state: %d", int(d))
|
|
}
|
|
}
|
|
|
|
// isDockerAvailable returns true if the Docker daemon is reachable.
|
|
// The result is cached after the first call.
|
|
func isDockerAvailable() bool {
|
|
dockerOnce.Do(func() {
|
|
dockerAvail = exec.Command("docker", "info").Run() == nil
|
|
})
|
|
return dockerAvail
|
|
}
|
|
|
|
// hasTestSkip returns true if the given service appears in the BEADS_TEST_SKIP
|
|
// env var (comma-separated list). Example: BEADS_TEST_SKIP=dolt,slow
|
|
func hasTestSkip(service string) bool {
|
|
val := os.Getenv("BEADS_TEST_SKIP")
|
|
if val == "" {
|
|
return false
|
|
}
|
|
for _, s := range strings.Split(val, ",") {
|
|
if strings.TrimSpace(s) == service {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
func skipOrFailDoltUnavailable(t *testing.T, state doltReadiness) {
|
|
t.Helper()
|
|
if os.Getenv(EnvRequireDoltContainer) == "1" {
|
|
t.Fatalf("Dolt test container unavailable (%s) but %s=1; this lane must not skip", state, EnvRequireDoltContainer)
|
|
}
|
|
t.Skipf("skipping test: %s", state)
|
|
}
|
|
|
|
// checkDolt returns the readiness state for Dolt integration tests.
|
|
// It composes hasTestSkip, isDockerAvailable, isDoltImageCached, and
|
|
// isDoltRepoImageCached, caching the result.
|
|
func checkDolt() doltReadiness {
|
|
doltCheckOnce.Do(func() {
|
|
// Explicit skip checked first to avoid ~1s docker info cost.
|
|
if hasTestSkip("dolt") {
|
|
doltCached = doltSkipped
|
|
return
|
|
}
|
|
if err := doltBackendErr(); err != nil {
|
|
doltLocalCLIErr = err
|
|
doltCached = doltNoLocalCLI
|
|
return
|
|
}
|
|
if useLocalDoltServer() {
|
|
if _, err := resolveLocalDoltBinary(); err != nil {
|
|
doltLocalCLIErr = err
|
|
doltCached = doltNoLocalCLI
|
|
return
|
|
}
|
|
doltCached = doltReady
|
|
return
|
|
}
|
|
if !isDockerAvailable() {
|
|
return // doltCached zero value is doltNoDocker
|
|
}
|
|
if isDoltImageCached() {
|
|
doltCached = doltReady
|
|
return
|
|
}
|
|
if isDoltRepoImageCached() {
|
|
doltCached = doltWrongVersion
|
|
return
|
|
}
|
|
doltCached = doltNoImage
|
|
})
|
|
return doltCached
|
|
}
|
|
|
|
// isDoltImageCached returns true if the exact Dolt Docker image (repo:tag)
|
|
// is available locally, avoiding unnecessary network calls to Docker Hub.
|
|
func isDoltImageCached() bool {
|
|
return exec.Command("docker", "image", "inspect", DoltDockerImage).Run() == nil
|
|
}
|
|
|
|
// isDoltRepoImageCached returns true if ANY version of the Dolt image repo
|
|
// exists locally (e.g. dolthub/dolt-sql-server with a different tag).
|
|
func isDoltRepoImageCached() bool {
|
|
out, err := exec.Command("docker", "images", doltDockerRepo, "-q").Output()
|
|
return err == nil && len(strings.TrimSpace(string(out))) > 0
|
|
}
|
|
|
|
// startDoltContainer starts the singleton Dolt container.
|
|
func startDoltContainer() error {
|
|
if useLocalDoltServer() {
|
|
s, err := startLocalDoltServer()
|
|
if err != nil {
|
|
return fmt.Errorf("starting local Dolt server: %w", err)
|
|
}
|
|
doltTestPort = strconv.Itoa(s.Port())
|
|
doltSingletonSrv = &doltServer{local: s}
|
|
return nil
|
|
}
|
|
ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout)
|
|
defer cancel()
|
|
|
|
ctr, err := dolt.Run(ctx, DoltDockerImage,
|
|
dolt.WithDatabase("beads_test"),
|
|
// Docker port-forwarding makes connections appear as non-localhost
|
|
// (e.g., 172.17.0.1). The entrypoint defaults DOLT_ROOT_HOST to
|
|
// "localhost", so root@localhost won't match external connections.
|
|
// Set to "%" so root can connect from any host.
|
|
testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}),
|
|
)
|
|
if err != nil {
|
|
return fmt.Errorf("starting Dolt container: %w", err)
|
|
}
|
|
|
|
p, err := ctr.MappedPort(ctx, "3306/tcp")
|
|
if err != nil {
|
|
_ = testcontainers.TerminateContainer(ctr)
|
|
return fmt.Errorf("getting mapped port: %w", err)
|
|
}
|
|
|
|
if _, err := strconv.Atoi(p.Port()); err != nil {
|
|
_ = testcontainers.TerminateContainer(ctr)
|
|
return fmt.Errorf("parsing port %q: %w", p.Port(), err)
|
|
}
|
|
|
|
doltTestPort = p.Port()
|
|
|
|
if err := waitForDoltReady(doltTestPort); err != nil {
|
|
_ = testcontainers.TerminateContainer(ctr)
|
|
return fmt.Errorf("waiting for Dolt server to be query-ready: %w", err)
|
|
}
|
|
|
|
doltSingletonSrv = &doltServer{
|
|
container: ctr,
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// doltReadyProbeTimeout bounds how long waitForDoltReady polls before giving up.
|
|
const doltReadyProbeTimeout = 30 * time.Second
|
|
|
|
// waitForDoltReady polls the server with a trivial query until it responds
|
|
// or doltReadyProbeTimeout elapses.
|
|
//
|
|
// The testcontainers wait strategy above only matches a log line ("Server
|
|
// ready. Accepting connections."), which confirms the TCP listener is up but
|
|
// not that Dolt's SQL engine can actually serve a query yet. In that narrow
|
|
// startup window, a query issued with an unbounded context (as several test
|
|
// helpers do) can block indefinitely instead of erroring, because nothing
|
|
// ever cancels it to unstick the read — confirmed via goroutine dump during
|
|
// be-fgd round-2 triage: the connection sat in mysqlConn.readWithTimeout /
|
|
// net.Read for 2+ minutes after the container had already logged ready. Each
|
|
// probe attempt here uses its own short bounded context, so a still-warming
|
|
// server fails an attempt fast and gets retried, rather than wedging.
|
|
func waitForDoltReady(port string) error {
|
|
dsn := fmt.Sprintf("root@tcp(127.0.0.1:%s)/", port)
|
|
deadline := time.Now().Add(doltReadyProbeTimeout)
|
|
var lastErr error
|
|
for {
|
|
lastErr = pingDoltOnce(dsn)
|
|
if lastErr == nil {
|
|
return nil
|
|
}
|
|
if time.Now().After(deadline) {
|
|
return fmt.Errorf("dolt server at port %s did not become query-ready within %s: %w", port, doltReadyProbeTimeout, lastErr)
|
|
}
|
|
time.Sleep(200 * time.Millisecond)
|
|
}
|
|
}
|
|
|
|
// pingDoltOnce opens a short-lived connection and pings it with a bounded
|
|
// context so a non-responsive server fails this attempt instead of hanging.
|
|
func pingDoltOnce(dsn string) error {
|
|
db, err := sql.Open("mysql", dsn)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer db.Close()
|
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
|
defer cancel()
|
|
return db.PingContext(ctx)
|
|
}
|
|
|
|
// terminateSharedContainer stops and removes the shared Dolt container.
|
|
// Safe to call concurrently or multiple times (sync.Once).
|
|
func terminateSharedContainer() {
|
|
doltTerminateOnce.Do(func() {
|
|
doltServerMu.Lock()
|
|
defer doltServerMu.Unlock()
|
|
stopSharedContainerLocked()
|
|
})
|
|
}
|
|
|
|
// stopSharedContainerLocked drops the current singleton container. The caller
|
|
// holds doltServerMu. It does not consume doltTerminateOnce, so a later
|
|
// replacement can still be removed by TerminateDoltContainer.
|
|
func stopSharedContainerLocked() {
|
|
if doltSingletonSrv != nil && doltSingletonSrv.local != nil {
|
|
doltSingletonSrv.local.terminate()
|
|
doltSingletonSrv.local = nil
|
|
}
|
|
if doltSingletonSrv != nil && doltSingletonSrv.container != nil {
|
|
_ = testcontainers.TerminateContainer(doltSingletonSrv.container)
|
|
doltSingletonSrv.container = nil
|
|
}
|
|
doltSingletonSrv = nil
|
|
}
|
|
|
|
// RestartSharedDoltContainer replaces the shared test container with a new
|
|
// one and returns its host port. A package that keeps one container for the
|
|
// whole TestMain (tracker, and the same shape elsewhere) otherwise turns a
|
|
// single server death into a failure for every later test: each dial reports
|
|
// "Dolt server unreachable" and the job goes red dozens of times. The
|
|
// replacement is empty; the caller re-creates its shared database.
|
|
func RestartSharedDoltContainer() (int, error) {
|
|
if state := checkDolt(); state != doltReady {
|
|
return 0, fmt.Errorf("%s", state)
|
|
}
|
|
doltServerMu.Lock()
|
|
defer doltServerMu.Unlock()
|
|
stopSharedContainerLocked()
|
|
if err := startDoltContainer(); err != nil {
|
|
doltServerErr = err
|
|
return 0, err
|
|
}
|
|
if err := os.Setenv("BEADS_DOLT_PORT", doltTestPort); err != nil {
|
|
doltServerErr = fmt.Errorf("set BEADS_DOLT_PORT: %w", err)
|
|
return 0, doltServerErr
|
|
}
|
|
if err := os.Setenv("BEADS_DOLT_SERVER_PORT", doltTestPort); err != nil {
|
|
doltServerErr = fmt.Errorf("set BEADS_DOLT_SERVER_PORT: %w", err)
|
|
return 0, doltServerErr
|
|
}
|
|
doltServerErr = nil
|
|
return DoltContainerPortInt(), nil
|
|
}
|
|
|
|
// ServerUnreachable reports whether err is a shared Dolt server that is gone,
|
|
// as opposed to a query the server refused. Connection refused and the
|
|
// "Dolt server unreachable" dial error are the shapes left behind when the
|
|
// singleton container exits; unexpected EOF is the MySQL driver watching that
|
|
// exit mid-handshake.
|
|
func ServerUnreachable(err error) bool {
|
|
if err == nil {
|
|
return false
|
|
}
|
|
if DoltContainerCrashed() {
|
|
return true
|
|
}
|
|
msg := strings.ToLower(err.Error())
|
|
return strings.Contains(msg, "connection refused") ||
|
|
strings.Contains(msg, "dolt server unreachable") ||
|
|
strings.Contains(msg, "unexpected eof")
|
|
}
|
|
|
|
// IsolatedDoltContainer is a per-test Dolt container together with the
|
|
// accessors a test needs to inspect it from the inside.
|
|
type IsolatedDoltContainer struct {
|
|
// Port is the mapped host port of the container's Dolt server.
|
|
Port string
|
|
|
|
ctr *dolt.DoltContainer
|
|
local *localDoltServer // local backend instead of ctr
|
|
}
|
|
|
|
// Stop pauses the container while retaining its data volume.
|
|
func (c *IsolatedDoltContainer) Stop(ctx context.Context) error {
|
|
if c != nil && c.local != nil {
|
|
return c.local.stop()
|
|
}
|
|
if c == nil || c.ctr == nil {
|
|
return fmt.Errorf("no Dolt container running")
|
|
}
|
|
return c.ctr.Stop(ctx, nil)
|
|
}
|
|
|
|
// Start resumes a container previously stopped with Stop.
|
|
func (c *IsolatedDoltContainer) Start(ctx context.Context) error {
|
|
if c != nil && c.local != nil {
|
|
return c.local.restart()
|
|
}
|
|
if c == nil || c.ctr == nil {
|
|
return fmt.Errorf("no Dolt container running")
|
|
}
|
|
return c.ctr.Start(ctx)
|
|
}
|
|
|
|
// CurrentPort returns the container's currently mapped SQL port. Container
|
|
// runtimes may assign a new host port after Stop/Start.
|
|
func (c *IsolatedDoltContainer) CurrentPort(ctx context.Context) (string, error) {
|
|
if c != nil && c.local != nil {
|
|
return strconv.Itoa(c.local.Port()), nil
|
|
}
|
|
if c == nil || c.ctr == nil {
|
|
return "", fmt.Errorf("no Dolt container running")
|
|
}
|
|
p, err := c.ctr.MappedPort(ctx, "3306/tcp")
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
return p.Port(), nil
|
|
}
|
|
|
|
// Exec runs cmd inside the container and returns its exit code and combined
|
|
// output, demultiplexed (see containerExec). Its working directory is the
|
|
// server's data directory (the image's WORKDIR, /var/lib/dolt); with the local
|
|
// backend cmd runs on the host in the local server's data directory, so
|
|
// commands must address the data through relative paths.
|
|
func (c *IsolatedDoltContainer) Exec(ctx context.Context, cmd []string) (int, string, error) {
|
|
if c != nil && c.local != nil {
|
|
return c.local.execInDataDir(ctx, cmd)
|
|
}
|
|
if c == nil || c.ctr == nil {
|
|
return 0, "", fmt.Errorf("no Dolt container running")
|
|
}
|
|
return containerExec(ctx, c.ctr, cmd)
|
|
}
|
|
|
|
// StartIsolatedDoltContainerHandle starts a per-test Dolt container and
|
|
// returns a handle to it. The container is terminated automatically when the
|
|
// test finishes.
|
|
//
|
|
// Unlike StartIsolatedDoltContainer this does NOT touch BEADS_DOLT_PORT or
|
|
// BEADS_DOLT_SERVER_PORT: those are process-wide, so a test that only wants
|
|
// its own server should not perturb sibling tests sharing the process.
|
|
func StartIsolatedDoltContainerHandle(t *testing.T) *IsolatedDoltContainer {
|
|
t.Helper()
|
|
if state := checkDolt(); state != doltReady {
|
|
skipOrFailDoltUnavailable(t, state)
|
|
}
|
|
|
|
if useLocalDoltServer() {
|
|
return startIsolatedLocalDoltServer(t)
|
|
}
|
|
return startIsolatedDoltContainer(t)
|
|
}
|
|
|
|
// startIsolatedLocalDoltServer is StartIsolatedDoltContainerHandle for the
|
|
// local backend, without the readiness gate.
|
|
func startIsolatedLocalDoltServer(t *testing.T) *IsolatedDoltContainer {
|
|
t.Helper()
|
|
s, err := startLocalDoltServer()
|
|
if err != nil {
|
|
t.Fatalf("starting local Dolt server: %v", err)
|
|
}
|
|
t.Cleanup(s.terminate)
|
|
return &IsolatedDoltContainer{Port: strconv.Itoa(s.Port()), local: s}
|
|
}
|
|
|
|
// startIsolatedDoltContainer is StartIsolatedDoltContainerHandle for the
|
|
// container backend, without the readiness gate.
|
|
func startIsolatedDoltContainer(t *testing.T) *IsolatedDoltContainer {
|
|
t.Helper()
|
|
ctx, cancel := context.WithTimeout(context.Background(), serverStartTimeout)
|
|
defer cancel()
|
|
ctr, err := dolt.Run(ctx, DoltDockerImage,
|
|
dolt.WithDatabase("beads_test"),
|
|
testcontainers.WithEnv(map[string]string{"DOLT_ROOT_HOST": "%"}),
|
|
)
|
|
if err != nil {
|
|
t.Fatalf("starting Dolt container: %v", err)
|
|
}
|
|
t.Cleanup(func() {
|
|
if err := testcontainers.TerminateContainer(ctr); err != nil {
|
|
t.Logf("terminating Dolt container: %v", err)
|
|
}
|
|
})
|
|
|
|
port, err := ctr.MappedPort(ctx, "3306/tcp")
|
|
if err != nil {
|
|
t.Fatalf("getting mapped port: %v", err)
|
|
}
|
|
|
|
return &IsolatedDoltContainer{Port: port.Port(), ctr: ctr}
|
|
}
|
|
|
|
// StartIsolatedDoltContainer starts a per-test Dolt container and returns the
|
|
// mapped host port, additionally pointing BEADS_DOLT_PORT and
|
|
// BEADS_DOLT_SERVER_PORT at it for the duration of the test.
|
|
func StartIsolatedDoltContainer(t *testing.T) string {
|
|
t.Helper()
|
|
c := StartIsolatedDoltContainerHandle(t)
|
|
t.Setenv("BEADS_DOLT_PORT", c.Port)
|
|
t.Setenv("BEADS_DOLT_SERVER_PORT", c.Port)
|
|
return c.Port
|
|
}
|
|
|
|
// ensureSharedContainer starts the singleton container and sets
|
|
// BEADS_DOLT_PORT and BEADS_DOLT_SERVER_PORT.
|
|
func ensureSharedContainer() {
|
|
doltServerOnce.Do(func() {
|
|
doltServerMu.Lock()
|
|
defer doltServerMu.Unlock()
|
|
doltServerErr = startDoltContainer()
|
|
if doltServerErr == nil && doltTestPort != "" {
|
|
if err := os.Setenv("BEADS_DOLT_PORT", doltTestPort); err != nil {
|
|
doltServerErr = fmt.Errorf("set BEADS_DOLT_PORT: %w", err)
|
|
} else if err := os.Setenv("BEADS_DOLT_SERVER_PORT", doltTestPort); err != nil {
|
|
doltServerErr = fmt.Errorf("set BEADS_DOLT_SERVER_PORT: %w", err)
|
|
}
|
|
}
|
|
})
|
|
}
|
|
|
|
// checkDoltFn is the readiness probe used by EnsureDoltContainerForTestMain.
|
|
// It is a variable purely so the fail-closed regression test can force a
|
|
// not-ready state deterministically -- without a Docker daemon, and without
|
|
// starting the shared singleton container just to observe the failure path.
|
|
// Production code never reassigns it. The direct checkDolt() calls in the
|
|
// t.Skip paths above and below are deliberately left alone: they gate a skip,
|
|
// not the fail-closed behavior this seam exists to test.
|
|
var checkDoltFn = checkDolt
|
|
|
|
// startSharedContainerFn starts the shared container once checkDoltFn reports
|
|
// ready. It is a variable for the same reason checkDoltFn is: the fail-closed
|
|
// tests reach the container-would-not-start path through it without consuming
|
|
// doltServerOnce, which would leave this package's real-container test running
|
|
// against a singleton that has already failed. Production code never
|
|
// reassigns it.
|
|
var startSharedContainerFn = startSharedContainer
|
|
|
|
// EnvSharedDoltServer names the one Dolt port that a test harness started a
|
|
// server on for this process tree, as opposed to a port inherited from the
|
|
// surrounding environment. Its value is that port number: scripts/test.sh sets
|
|
// it to the same value as the BEADS_DOLT_PORT it exports for the dockerless
|
|
// `dolt sql-server` of its BEADS_TEST_SHARED_SERVER=1 flow. It vouches for that
|
|
// value only, never for the process: a port variable holding anything else is
|
|
// still ambient, however it got there.
|
|
//
|
|
// It is deliberately a distinct name from BEADS_TEST_SHARED_SERVER. That flag
|
|
// is also read by cmd/bd's shared-server integration test for an unrelated
|
|
// purpose, and it only says a shared server was asked for, not that one was
|
|
// started: the flow starts none when a port variable is already set, so the
|
|
// flag beside a port cannot tell the script's own port from an inherited one.
|
|
const EnvSharedDoltServer = "BEADS_TEST_SHARED_DOLT_SERVER"
|
|
|
|
// sharedContainerErr reads the shared-container error under doltServerMu.
|
|
// Every write to doltServerErr takes that mutex (ensureSharedContainer,
|
|
// RestartSharedDoltContainer), so reads belong under it too; callers must not
|
|
// already hold the mutex.
|
|
func sharedContainerErr() error {
|
|
doltServerMu.Lock()
|
|
defer doltServerMu.Unlock()
|
|
return doltServerErr
|
|
}
|
|
|
|
// startSharedContainer is ensureSharedContainer plus the error it recorded.
|
|
func startSharedContainer() error {
|
|
ensureSharedContainer()
|
|
return sharedContainerErr()
|
|
}
|
|
|
|
// EnsureDoltContainerForTestMain starts a shared Dolt container for use in
|
|
// TestMain functions. Call TerminateDoltContainer() after m.Run() to clean up.
|
|
// On success it sets BEADS_DOLT_PORT and BEADS_DOLT_SERVER_PORT process-wide,
|
|
// which also means a container, when one can be started, takes precedence over
|
|
// any port the environment already named.
|
|
//
|
|
// On ANY failure -- Docker missing, image missing or wrong version, an
|
|
// explicit BEADS_TEST_SKIP=dolt opt-out, or a container that would not start
|
|
// -- it instead clears both variables (see neutralizeAmbientDoltPort) and
|
|
// returns an error. Callers that warn and run the suite anyway therefore
|
|
// cannot resolve a store onto an ambient server.
|
|
//
|
|
// The single exception is a variable holding exactly the port that
|
|
// EnvSharedDoltServer names, i.e. one a harness provisioned for this process
|
|
// tree: that is a dedicated test server and survives the failure paths (see
|
|
// neutralizeUnvouchedDoltPorts). Without that exception this function would
|
|
// erase the port scripts/test.sh exports for its own shared `dolt sql-server`,
|
|
// which on a Docker-less host is every one of these TestMains.
|
|
func EnsureDoltContainerForTestMain() error {
|
|
var err error
|
|
if state := checkDoltFn(); state != doltReady {
|
|
err = fmt.Errorf("%s", state)
|
|
} else {
|
|
err = startSharedContainerFn()
|
|
}
|
|
// One clear serves both failure paths, so "no usable container" and "a
|
|
// container that would not start" cannot drift apart.
|
|
if err != nil {
|
|
neutralizeUnvouchedDoltPorts()
|
|
}
|
|
return err
|
|
}
|
|
|
|
// neutralizeUnvouchedDoltPorts is the clear EnsureDoltContainerForTestMain
|
|
// applies when it has no container: neutralizeAmbientDoltPort, except that a
|
|
// port variable holding exactly the port EnvSharedDoltServer names survives,
|
|
// and says so on stderr, because the suite is about to talk to that server.
|
|
//
|
|
// The decision is made per variable, not per process. scripts/test.sh vouches
|
|
// for the BEADS_DOLT_PORT it exports; an unrelated BEADS_DOLT_SERVER_PORT
|
|
// beside it -- which applyConfigDefaults in internal/storage/dolt resolves
|
|
// FIRST, falling back to BEADS_DOLT_PORT only when it is empty -- is cleared
|
|
// even though the marker is set, or it would outrank the harness's server and
|
|
// carry the suite onto whatever server it names (gm-2g3g5r).
|
|
func neutralizeUnvouchedDoltPorts() {
|
|
vouched := os.Getenv(EnvSharedDoltServer)
|
|
if vouched == "" {
|
|
neutralizeAmbientDoltPort()
|
|
return
|
|
}
|
|
for _, name := range []string{"BEADS_DOLT_SERVER_PORT", "BEADS_DOLT_PORT"} {
|
|
if os.Getenv(name) == vouched {
|
|
fmt.Fprintf(os.Stderr, "testutil: keeping %s=%s, the harness-provisioned test server named by %s\n",
|
|
name, vouched, EnvSharedDoltServer)
|
|
continue
|
|
}
|
|
_ = os.Unsetenv(name)
|
|
}
|
|
}
|
|
|
|
// neutralizeAmbientDoltPort clears the inherited connection-port variables so
|
|
// that a TestMain which warns-and-continues without a container cannot resolve
|
|
// a store onto whatever server the ambient environment names -- in a gc-managed
|
|
// city, the production one (gm-2g3g5r).
|
|
//
|
|
// What this guarantees is exactly that: no ENVIRONMENT-NAMED port is honored.
|
|
// It is not the wider claim that no port is resolvable afterwards. Port
|
|
// resolution continues past the env vars into the file chain, and
|
|
// applyConfigDefaults derives a directory from filepath.Dir(cfg.Path) when
|
|
// cfg.BeadsDir is empty (internal/storage/dolt/store.go:1661-1664), so a
|
|
// reachable .beads/dolt-server.port can still supply a non-zero port -- the
|
|
// ServerPort == 0 arm of the BEADS_TEST_MODE guard (store.go:1683-1687) then
|
|
// never fires. Port 1 is forced only via isProductionPort, and
|
|
// production-port DETECTION is narrower than resolution: Rule 1 recognizes
|
|
// only DefaultSQLPort, and Rule 3 -- the only rule that sees a dynamic port --
|
|
// needs an explicit cfg.BeadsDir and is suppressed by the BEADS_TEST_SERVER=1
|
|
// that beads TestMains set. That residual is tracked in be-rl6tm; clearing the
|
|
// env vars removes the channel this leak actually traveled on, not every
|
|
// channel.
|
|
//
|
|
// This fires on EVERY not-ready state, not only "Docker unavailable": a
|
|
// missing or wrong-version image and an explicit BEADS_TEST_SKIP=dolt opt-out
|
|
// clear the ambient port too. That is deliberate. The invariant is about the
|
|
// absence of a test container, not about the reason for it -- an opted-out run
|
|
// has no more business reaching a shared server than a Docker-less one does.
|
|
func neutralizeAmbientDoltPort() {
|
|
_ = os.Unsetenv("BEADS_DOLT_SERVER_PORT")
|
|
_ = os.Unsetenv("BEADS_DOLT_PORT")
|
|
}
|
|
|
|
// RequireDoltContainer ensures a shared Dolt container is running. Skips the
|
|
// test if Docker is not available.
|
|
func RequireDoltContainer(t *testing.T) {
|
|
t.Helper()
|
|
if state := checkDolt(); state != doltReady {
|
|
skipOrFailDoltUnavailable(t, state)
|
|
}
|
|
|
|
ensureSharedContainer()
|
|
if doltServerErr != nil {
|
|
t.Fatalf("Dolt container setup failed: %v", doltServerErr)
|
|
}
|
|
}
|
|
|
|
// DoltContainerAddr returns the address (host:port) of the Dolt container.
|
|
func DoltContainerAddr() string {
|
|
return "127.0.0.1:" + doltTestPort
|
|
}
|
|
|
|
// DoltContainerPort returns the mapped host port of the Dolt container.
|
|
func DoltContainerPort() string {
|
|
return doltTestPort
|
|
}
|
|
|
|
// DoltContainerPortInt returns the mapped host port as an int.
|
|
func DoltContainerPortInt() int {
|
|
p, _ := strconv.Atoi(doltTestPort)
|
|
return p
|
|
}
|
|
|
|
// TerminateDoltContainer stops and removes the shared Dolt container.
|
|
// Called from TestMain after m.Run().
|
|
func TerminateDoltContainer() {
|
|
terminateSharedContainer()
|
|
}
|
|
|
|
// DoltContainerCrashed returns true if the shared container has exited unexpectedly.
|
|
// Returns false if no container was started.
|
|
func DoltContainerCrashed() bool {
|
|
if doltSingletonSrv != nil && doltSingletonSrv.local != nil {
|
|
gone, _ := doltSingletonSrv.local.exited()
|
|
return gone
|
|
}
|
|
if doltSingletonSrv == nil || doltSingletonSrv.container == nil {
|
|
return false
|
|
}
|
|
state, err := doltSingletonSrv.container.State(context.Background())
|
|
if err != nil {
|
|
return true // can't check state — assume crashed
|
|
}
|
|
return !state.Running
|
|
}
|
|
|
|
// DoltContainerCrashError returns an error if the shared container has exited
|
|
// unexpectedly, nil otherwise.
|
|
func DoltContainerCrashError() error {
|
|
if doltSingletonSrv != nil && doltSingletonSrv.local != nil {
|
|
if gone, err := doltSingletonSrv.local.exited(); gone {
|
|
return fmt.Errorf("local Dolt server exited (%v); log: %s", err, doltSingletonSrv.local.logSince(0))
|
|
}
|
|
return nil
|
|
}
|
|
if doltSingletonSrv == nil || doltSingletonSrv.container == nil {
|
|
return nil
|
|
}
|
|
state, err := doltSingletonSrv.container.State(context.Background())
|
|
if err != nil {
|
|
return fmt.Errorf("failed to check container state: %w", err)
|
|
}
|
|
if !state.Running {
|
|
return fmt.Errorf("Dolt container exited (status=%s, exit=%d)", state.Status, state.ExitCode)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// containerExec runs cmd inside ctr and returns its exit code and combined
|
|
// output. Used by tests that need to inspect a container's filesystem (e.g.
|
|
// the .dolt_dropped_databases/ directory), which has no host-visible path
|
|
// since the containers are started without a bind-mounted data dir.
|
|
//
|
|
// tcexec.Multiplexed() is load-bearing, not decorative. Without it Exec hands
|
|
// back the raw hijacked Docker stream and io.ReadAll glues the 8-byte stdcopy
|
|
// frame header onto the payload: `echo hello` returns
|
|
// "\x01\x00\x00\x00\x00\x00\x00\x06hello\n". A caller that feeds such
|
|
// a string back into the container — a path read out of find(1), say — then
|
|
// fails with "invalid argument" on a prefix it cannot see.
|
|
func containerExec(ctx context.Context, ctr *dolt.DoltContainer, cmd []string) (int, string, error) {
|
|
code, reader, err := ctr.Exec(ctx, cmd, tcexec.Multiplexed())
|
|
if err != nil {
|
|
return 0, "", fmt.Errorf("exec in Dolt container: %w", err)
|
|
}
|
|
out, err := io.ReadAll(reader)
|
|
if err != nil {
|
|
return code, "", fmt.Errorf("reading exec output: %w", err)
|
|
}
|
|
return code, string(out), nil
|
|
}
|