beads/internal/testutil
Bee 1cf193bc9f
fix(testutil): sweep Dolt test containers whose owner process exited (#7028)
* fix(testutil): sweep Dolt test containers whose owner process exited (wy-4hknjt)

Every Dolt container the test harness starts now carries two labels naming
its owner: the test binary's pid and the host it runs on. Before a process
starts its first container, the harness lists the containers labelled for
this host and removes the ones whose owner pid no longer exists.

In-process cleanup (t.Cleanup, TerminateDoltContainer after m.Run) never
runs when a test binary is killed or panics on `go test -timeout`, and a box
running with TESTCONTAINERS_RYUK_DISABLED=true has no reaper either. Each
such run stranded one idle dolt-sql-server holding 100-470 MiB; 23 were
standing on one loaded box after a single evening of killed runs. The sweep
bounds the pile at "runs since the last run".

Safety under concurrent test processes on one host: a container is removed
only when kill(pid, 0) reports ESRCH. A live owner, a pid reused by another
live process, or a pid this user may not signal (EPERM) all keep the
container. Containers without the owner labels (older harnesses, other
tools) are never touched, and the host label keeps a remote DOCKER_HOST's
containers, whose pids mean nothing here, out of the sweep. The local
(BEADS_TEST_DOLT_SERVER=local) backend is unaffected.

Tests: the decision table (dead owners only; missing, non-numeric and
non-positive pids skipped without probing), pidAlive on self and on an
impossible pid, the label option stamping this process, and a Docker test
that starts a dead-owned and a live-owned container, sweeps, and checks
only the dead-owned one is gone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(bazel): sync BUILD files for testutil dolt sweep test

wy-4hknjt: apply the gazelle bazel-sync patch from run 36720164405
so internal/testutil/BUILD.bazel carries the testcontainers deps.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(testutil): key the Dolt container sweep on the owner's PID namespace

Maintainer fixup for #7028, review iterations 1 and 2 and pre-approval
CI.

A pid names a process only inside its PID namespace. The sweep removed a
labelled container whenever kill(owner-pid, 0) returned ESRCH, so a
sweeper in another namespace that shares the hostname and the Docker
daemon reads every live owner as dead. That is the default shape of the
maintainers' agent fleet: each session runs under bwrap --unshare-pid
without --unshare-uts. Every session therefore reports one hostname, sees
only its own handful of pids, and talks to one /var/run/docker.sock, with
the Docker backend as the test default. On one such box on 2026-10-06,
three live dolt-sql-server test containers from three test runs shared the
daemon. With the original sweep merged, the first container start of a
test binary in any other session would have removed them mid-test.

Reproduced and fixed by running the sweep from a container (its own pid
namespace, same hostname, same daemon) against a container labelled with a
pid that was alive in the labelling session:
- at 5978cedaa2: "removed stale Dolt test container ... (owner process
  exited)" while the owner was alive;
- with this change: the row is listed and kept.

Changes:
- Stamp a third owner label, owner-pidns: the /proc/self/ns/pid link on
  Linux and "" elsewhere. staleContainers considers only rows from the
  sweeper's own namespace. On Linux the sweep is skipped when that link
  cannot be read. A start-time label alone would not fix this: across
  namespaces it would turn a pid collision into "dead" as well. An owner
  whose hostname or namespace cannot be read is stamped "", which no
  sweep on the host matches, so its container leaks instead.
- docker rm -f -v: the image declares VOLUME /var/lib/dolt, and plain
  rm -f left that anonymous volume behind for every swept container, unlike
  testcontainers' own RemoveVolumes teardown.
- Bound the sweep with a 30s context, plus WaitDelay on each docker
  command. Without WaitDelay, killing the docker process on timeout does not
  unblock Output() while a child it started still holds the pipe. With a
  forking docker shim on PATH, the call returns after 3s with this change
  and is still blocked after 15s without either part.
- Extract parseOwnedRows and table-test it: empty fields, CRLF, trailing
  newline.
- TestSweepStaleDoltContainers: the dead-owned container's owner is a
  child process that lives until both containers are up and the volumes
  are read, and is then killed and reaped (a zombie still reads as
  alive). A sibling test binary's sweep (go test ./... runs them in
  parallel) removes a container as soon as its owner reads as dead, and a
  never-live owner pid exposed the container for its whole start. The
  test checks only the end state for that container and its volume,
  polled for up to sweepTimeout, because the sibling may get there first
  or still be removing it when our rm fails.

Reproduced with a second test process in the same PID namespace running
the real sweep every 100ms alongside TestSweepStaleDoltContainers:
- with a never-live owner pid: "starting Dolt container: ... No such
  container", the container removed inside dolt.Run;
- with the child owner but no poll: 1 of 4 runs failed with "dead-owned
  container ... still exists after the sweep", after the sibling's
  rm -f -v won and ours exited 1;
- with both: 6 of 6 runs pass, 2 of them through that same path.
Without the reap, or without -v, the test fails.

A container leaked from another namespace is now left to a sweep from that
namespace, or to Ryuk. That is the conservative side of the trade.

Pre-approval CI: main's //internal/testutil:testutil_dolt_test (#7328)
runs this package with BEADS_TEST_DOLT_SERVER=local and
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 in the required dolt-server lane,
whose remote workers have no docker. Under the local backend checkDolt
reports ready once the pinned dolt CLI resolves, so
TestSweepStaleDoltContainers went on to dolt.Run and failed ("failed to
create Docker provider"). It now gates like TestDoltServerFingerprint's
container subtest: under the local backend it runs only when docker and
the image are available, and skips otherwise. On the merge with main
b1fe68e63a, with docker unreachable, that package run (race,
-parallel 4) failed only this test; with this change it passes with the
test skipped, and with docker available the test still runs and passes
under either backend.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <julianknutsen@users.noreply.github.com>
2026-10-10 02:39:59 +00:00
..
bazeltest ci(bazel): split //scripts:scripts_test by what it reads; partition //:repo_files (#7414) 2026-10-09 05:13:07 +00:00
credentialcmd ci(bazel): split //scripts:scripts_test by what it reads; partition //:repo_files (#7414) 2026-10-09 05:13:07 +00:00
fixtures ci(bazel): split //scripts:scripts_test by what it reads; partition //:repo_files (#7414) 2026-10-09 05:13:07 +00:00
integration ci(bazel): split //scripts:scripts_test by what it reads; partition //:repo_files (#7414) 2026-10-09 05:13:07 +00:00
testdata testutil: migrate test Dolt server from binary-spawning to testcontainers 2026-02-28 21:12:41 -08:00
BUILD.bazel fix(testutil): sweep Dolt test containers whose owner process exited (#7028) 2026-10-10 02:39:59 +00:00
container_provider.go fix(testutil): sweep Dolt test containers whose owner process exited (#7028) 2026-10-10 02:39:59 +00:00
container_provider_windows.go /{cmd,internal}: some cleanup 2026-04-08 14:45:18 -07:00
dockercontext.go fix(test): pin DOCKER_HOST from the active docker context before TestMain swaps HOME (bd-84kos) (#5391) 2026-08-07 00:32:40 -07:00
dockercontext_test.go fix(test): pin DOCKER_HOST from the active docker context before TestMain swaps HOME (bd-84kos) (#5391) 2026-08-07 00:32:40 -07:00
git_test_hooks.go Test isolation follow-up: consolidate and guard hooksPath leakage (#2234) (#2237) 2026-03-01 14:41:08 -08:00
gm2g3g5r_failopen_test.go Fix: test suites fail closed instead of open on the ambient Dolt port when no test container is started (be-napos) (#5885) 2026-10-01 05:21:49 -07:00
gm2g3g5r_failopen_unix_test.go Fix: test suites fail closed instead of open on the ambient Dolt port when no test container is started (be-napos) (#5885) 2026-10-01 05:21:49 -07:00
gm2g3g5r_failopen_windows_test.go Fix: test suites fail closed instead of open on the ambient Dolt port when no test container is started (be-napos) (#5885) 2026-10-01 05:21:49 -07:00
localdoltserver.go test(testutil): opt-in local dolt sql-server backend for Dolt-backed tests (#6891) 2026-09-28 04:06:49 -07:00
localdoltserver_linux.go test(testutil): opt-in local dolt sql-server backend for Dolt-backed tests (#6891) 2026-09-28 04:06:49 -07:00
localdoltserver_other.go test(testutil): opt-in local dolt sql-server backend for Dolt-backed tests (#6891) 2026-09-28 04:06:49 -07:00
localdoltserver_test.go test(testutil): opt-in local dolt sql-server backend for Dolt-backed tests (#6891) 2026-09-28 04:06:49 -07:00
suite_temp.go fix(doltserver): recognize GOTMPDIR as a temp-dir sweep root (Go 1.26+) (#6733) 2026-09-26 04:30:59 -07:00
suite_temp_test.go fix(doctor): reap leaked test dolt sql-servers from CGO_ENABLED=0 suites (#5917) 2026-09-02 18:11:33 -07:00
testdoltbranch.go fix(testutil): wait for database visibility in SetupSharedTestDB (#5788) 2026-09-29 17:22:06 -07:00
testdoltbranch_test.go fix(testutil): wait for database visibility in SetupSharedTestDB (#5788) 2026-09-29 17:22:06 -07:00
testdoltcommon.go test(testutil): opt-in local dolt sql-server backend for Dolt-backed tests (#6891) 2026-09-28 04:06:49 -07:00
testdoltreap.go fix(testutil): sweep Dolt test containers whose owner process exited (#7028) 2026-10-10 02:39:59 +00:00
testdoltreap_test.go fix(testutil): sweep Dolt test containers whose owner process exited (#7028) 2026-10-10 02:39:59 +00:00
testdoltserver.go fix(testutil): sweep Dolt test containers whose owner process exited (#7028) 2026-10-10 02:39:59 +00:00
testdoltserver_env_test.go test(testutil): opt-in local dolt sql-server backend for Dolt-backed tests (#6891) 2026-09-28 04:06:49 -07:00
testdoltserver_unreachable_test.go fix(test): recover when a Dolt test server dies mid-suite (#6756) 2026-09-26 14:27:18 -07:00
testdoltserver_windows.go Fix: test suites fail closed instead of open on the ambient Dolt port when no test container is started (be-napos) (#5885) 2026-10-01 05:21:49 -07:00
tmpfs.go style: run gofmt across entire codebase (bd-nmk12) 2026-02-07 19:35:20 -08:00
tmpfs_test.go style: run gofmt across entire codebase (bd-nmk12) 2026-02-07 19:35:20 -08:00