* fix(testutil): sweep Dolt test containers whose owner process exited (wy-4hknjt)
Every Dolt container the test harness starts now carries two labels naming
its owner: the test binary's pid and the host it runs on. Before a process
starts its first container, the harness lists the containers labelled for
this host and removes the ones whose owner pid no longer exists.
In-process cleanup (t.Cleanup, TerminateDoltContainer after m.Run) never
runs when a test binary is killed or panics on `go test -timeout`, and a box
running with TESTCONTAINERS_RYUK_DISABLED=true has no reaper either. Each
such run stranded one idle dolt-sql-server holding 100-470 MiB; 23 were
standing on one loaded box after a single evening of killed runs. The sweep
bounds the pile at "runs since the last run".
Safety under concurrent test processes on one host: a container is removed
only when kill(pid, 0) reports ESRCH. A live owner, a pid reused by another
live process, or a pid this user may not signal (EPERM) all keep the
container. Containers without the owner labels (older harnesses, other
tools) are never touched, and the host label keeps a remote DOCKER_HOST's
containers, whose pids mean nothing here, out of the sweep. The local
(BEADS_TEST_DOLT_SERVER=local) backend is unaffected.
Tests: the decision table (dead owners only; missing, non-numeric and
non-positive pids skipped without probing), pidAlive on self and on an
impossible pid, the label option stamping this process, and a Docker test
that starts a dead-owned and a live-owned container, sweeps, and checks
only the dead-owned one is gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* build(bazel): sync BUILD files for testutil dolt sweep test
wy-4hknjt: apply the gazelle bazel-sync patch from run 36720164405
so internal/testutil/BUILD.bazel carries the testcontainers deps.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(testutil): key the Dolt container sweep on the owner's PID namespace
Maintainer fixup for #7028, review iterations 1 and 2 and pre-approval
CI.
A pid names a process only inside its PID namespace. The sweep removed a
labelled container whenever kill(owner-pid, 0) returned ESRCH, so a
sweeper in another namespace that shares the hostname and the Docker
daemon reads every live owner as dead. That is the default shape of the
maintainers' agent fleet: each session runs under bwrap --unshare-pid
without --unshare-uts. Every session therefore reports one hostname, sees
only its own handful of pids, and talks to one /var/run/docker.sock, with
the Docker backend as the test default. On one such box on 2026-10-06,
three live dolt-sql-server test containers from three test runs shared the
daemon. With the original sweep merged, the first container start of a
test binary in any other session would have removed them mid-test.
Reproduced and fixed by running the sweep from a container (its own pid
namespace, same hostname, same daemon) against a container labelled with a
pid that was alive in the labelling session:
- at 5978cedaa2: "removed stale Dolt test container ... (owner process
exited)" while the owner was alive;
- with this change: the row is listed and kept.
Changes:
- Stamp a third owner label, owner-pidns: the /proc/self/ns/pid link on
Linux and "" elsewhere. staleContainers considers only rows from the
sweeper's own namespace. On Linux the sweep is skipped when that link
cannot be read. A start-time label alone would not fix this: across
namespaces it would turn a pid collision into "dead" as well. An owner
whose hostname or namespace cannot be read is stamped "", which no
sweep on the host matches, so its container leaks instead.
- docker rm -f -v: the image declares VOLUME /var/lib/dolt, and plain
rm -f left that anonymous volume behind for every swept container, unlike
testcontainers' own RemoveVolumes teardown.
- Bound the sweep with a 30s context, plus WaitDelay on each docker
command. Without WaitDelay, killing the docker process on timeout does not
unblock Output() while a child it started still holds the pipe. With a
forking docker shim on PATH, the call returns after 3s with this change
and is still blocked after 15s without either part.
- Extract parseOwnedRows and table-test it: empty fields, CRLF, trailing
newline.
- TestSweepStaleDoltContainers: the dead-owned container's owner is a
child process that lives until both containers are up and the volumes
are read, and is then killed and reaped (a zombie still reads as
alive). A sibling test binary's sweep (go test ./... runs them in
parallel) removes a container as soon as its owner reads as dead, and a
never-live owner pid exposed the container for its whole start. The
test checks only the end state for that container and its volume,
polled for up to sweepTimeout, because the sibling may get there first
or still be removing it when our rm fails.
Reproduced with a second test process in the same PID namespace running
the real sweep every 100ms alongside TestSweepStaleDoltContainers:
- with a never-live owner pid: "starting Dolt container: ... No such
container", the container removed inside dolt.Run;
- with the child owner but no poll: 1 of 4 runs failed with "dead-owned
container ... still exists after the sweep", after the sibling's
rm -f -v won and ours exited 1;
- with both: 6 of 6 runs pass, 2 of them through that same path.
Without the reap, or without -v, the test fails.
A container leaked from another namespace is now left to a sweep from that
namespace, or to Ryuk. That is the conservative side of the trade.
Pre-approval CI: main's //internal/testutil:testutil_dolt_test (#7328)
runs this package with BEADS_TEST_DOLT_SERVER=local and
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 in the required dolt-server lane,
whose remote workers have no docker. Under the local backend checkDolt
reports ready once the pinned dolt CLI resolves, so
TestSweepStaleDoltContainers went on to dolt.Run and failed ("failed to
create Docker provider"). It now gates like TestDoltServerFingerprint's
container subtest: under the local backend it runs only when docker and
the image are available, and skips otherwise. On the merge with main
b1fe68e63a, with docker unreachable, that package run (race,
-parallel 4) failed only this test; with this change it passes with the
test skipped, and with docker available the test still runs and passes
under either backend.
Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <julianknutsen@users.noreply.github.com>