fix(metrics): stop failing the reap test on a transient zombie (#6262)

* fix(metrics): stop failing the reap test on a transient zombie

startDetached (spawn.go) reaps asynchronously via `go func() { cmd.Wait() }()`,
which is correct: GH#5900's fix already reaps every child, just not
synchronously. Between a child's exit and that goroutine's waitpid, `ps`
briefly reports it as STAT=Z -- the normal kernel process-table lag, not
a leak.

The test's in-loop check faulted the instant it saw "Z" in ps output, so it
was failing on the transient window itself rather than on a child that
actually stays a zombie. Measured on clean main under 32 busy loops (40
probe runs): zombie observed 12/40, never-reaped 0/40, window 5.0-23.4ms --
wider than the loop's own 20ms poll spacing, so the false positive was
guaranteed to land sometimes. Reproduced at ~14% (13/90 failures) building
`go test -c ./internal/metrics` and running under nproc busy loops, matching
the exact failure text seen in CI.

Remove the in-loop check and stop binding its now-unused `st`. The
post-loop 3s-deadline assertion already covers the real regression: a
child that STAYS a zombie keeps `alive` true, so the loop spins to the
deadline and fails there instead. Verified that path still works by
temporarily swapping startDetached's Wait goroutine for the old
`cmd.Process.Release()` (pre-GH#5900) reap -- the test fails 3/3 with
"still present after 3s" -- then reverting.

Before: 13/90 failures under load. After: 0/90.

Agent-Signature: claude-code-claude-sonnet-5-unknown-reasoning on behalf of Jim Wordelman

* docs(release-gates): PASS 7/7 — be-3vzut (transient zombie flake in TestStartDetachedReapsExitedChild)

Independently re-verified review bead be-5gizx's evidence for the fix
built in be-kh65q (root-caused by investigator bead be-heifp's PR #6147
triage): full-suite make test re-run at the deploy SHA (6348 PASS/0
FAIL/2104 SKIP, all SKIPs pre-existing Dolt/embedded/proxied opt-in
gates), diff-owned test independently isolated and confirmed PASS,
regression-catch re-verified (3/3 fail on reintroduced bug, 0/90 under
load after fix vs 13/90 before), gofmt/vet clean, 0 divergence from
origin/main. Criterion 7 (ancestry scope) carries a bounded mayor
waiver (gm-wisp-9mtr84) — see the gate file for full reasoning.

Agent-Signature: claude-code-claude-sonnet-5-unknown-reasoning on behalf of Jim Wordelman

* docs(release-gates): record opened PR #6262 for be-3vzut

Agent-Signature: claude-code-claude-sonnet-5-unknown-reasoning on behalf of Jim Wordelman

* docs(metrics): reword reap-test doc comment to persistence framing

The function doc comment still asserted the absolute invariant this PR
repudiates -- "startDetached must reap it so `ps` never reports STAT Z
under this process" -- which contradicted the new in-loop comment and
invited a future maintainer to re-add the removed in-loop Z fatal,
reintroducing the flake. Reword to the persistence framing already used
by startDetached's own doc comment (spawn.go:210-211, "never left as Z
under this process") and state explicitly that a transient Z is the
normal reaped-child path.

Comment-only; no behavior change. Review fix for the sole actionable
finding in the #6262 review synthesis (minor, claude+gemini).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Eddie the Engineer <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Jim Wordelman 2026-09-08 18:16:35 -07:00 • committed by GitHub
commit a690b0a8c4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 83 additions and 5 deletions

View file

@ -13,7 +13,9 @@ import (
// TestStartDetachedReapsExitedChild is the GH#5900 regression: a child that
// exits while the parent stays alive must be waitpid'd, not left as a zombie.
// The helper re-execs this test binary and exits immediately; startDetached
// must reap it so `ps` never reports STAT Z under this process.
// must reap it so the child is never left as STAT Z under this process. A
// transient Z between the child's exit and the Wait goroutine's waitpid is
// the normal reaped-child path, not a failure -- see the poll loop below.
func TestStartDetachedReapsExitedChild(t *testing.T) {
if os.Getenv("BD_TEST_FLUSHER_HELPER") == "1" {
// Stay alive long enough for the parent to observe a live pid.
@ -47,10 +49,17 @@ func TestStartDetachedReapsExitedChild(t *testing.T) {
seenLive := false
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
st, alive := unixProcStat(pid)
if strings.Contains(st, "Z") {
t.Fatalf("child pid %d became a zombie (stat=%q); startDetached did not reap", pid, st)
}
// A transient Z is the NORMAL post-exit state of a correctly
// reaped child: the kernel keeps the process-table entry until
// the Wait goroutine's waitpid runs. Measured on clean main
// under load, that window is 5.0-23.4ms -- wider than this
// loop's own 20ms poll spacing -- so failing on sight of Z is
// a false positive, not a detection (this flaked 7/50 that
// way). What the GH#5900 regression actually requires is that
// the child does not STAY a zombie: the post-loop check below
// asserts exactly that, and still catches the old
// cmd.Process.Release() path (verified: 3/3 fail).
_, alive := unixProcStat(pid)
if alive {
seenLive = true
time.Sleep(20 * time.Millisecond)

View file

@ -0,0 +1,69 @@
# Release gate — be-3vzut (Fix: TestStartDetachedReapsExitedChild fails on a transient zombie, flaking ~14% and reddening unrelated PRs)
**Date:** 2026-09-04
**Deployer:** beads/deployer
**Bead (deploy):** be-3vzut
**Source bead:** be-5gizx — review verdict PASS (style/security/spec findings: none/none/none); build bead `be-kh65q`; root-caused by investigator bead `be-heifp` (PR #6147 triage)
**Source commit:** `7d53c890623e75e4c2e2748e3eb1167ba39db9f2` (parent `c0d8da42de5fd15c95adac85e342ba4a121da0fb` = origin/main's own tip — zero divergence, no rebase needed)
**Branch:** `deploy/be-3vzut-gate`, cut directly at the reviewed SHA via `resolve_deploy_branch_target be-3vzut 7d53c890623e75e4c2e2748e3eb1167ba39db9f2`
**Push target:** `headfork` (`quad341/beads-sec003-contrib`) — `origin` push-disabled by design (`DISABLED-upstream-is-fetch-only-push-to-fork-and-PR`). Pushed and independently verified: `git ls-remote headfork refs/heads/deploy/be-3vzut-gate` → `7d53c890623e75e4c2e2748e3eb1167ba39db9f2`, matches local HEAD exactly.
**PR:** https://github.com/gastownhall/beads/pull/6262 — OPEN, `gastownhall/beads:main` ← `quad341/beads-sec003-contrib:deploy/be-3vzut-gate`, MERGEABLE
## Verdict: 7/7 — criterion 7 (ancestry scope) via bounded mayor waiver, all others raw PASS
## Criteria walk
| # | Criterion | Result | Evidence |
|---|-----------|--------|----------|
| 1 | Review PASS present | PASS | be-5gizx: VERDICT PASS. style_findings: none (gofmt/vet independently re-run clean). security_findings: none (diff is test-only, `internal/metrics/spawn_reap_test.go`; `spawn.go` diff empty — confirmed zero production code touched). spec_findings: root cause and fix independently re-verified by reviewer, not just trusted from builder/investigator reports. |
| 2 | Acceptance criteria met | PASS | Diff matches the bead's stated fix theme exactly: removes the in-loop early-fail check and its now-unused `st` binding, adds a comment documenting the kernel zombie-retention window vs. poll-spacing rationale. Matches investigator's (be-heifp) verbatim patch per be-5gizx's review. |
| 3 | Tests pass | PASS | Full-suite `make test` re-run at the exact deploy SHA this round (see "Criterion 3" below for counts). Diff-owned regression-catch independently re-verified by both reviewer and deployer: reverting `startDetached`'s reap to the pre-GH#5900 path fails 3/3 with "still present after 3s"; reverting back passes clean. Before/after under load: 13/90 → 0/90 failures. |
| 4 | No HIGH-severity findings open | PASS | be-5gizx security_findings: none. Test-only diff, no external input/auth/secrets/PII/injection surface, no OWASP-relevant change. |
| 5 | Final branch is clean | PASS | `git status --short` clean on `deploy/be-3vzut-gate` post-cut. |
| 6 | Branch diverges cleanly from main | PASS | Cut directly from the reviewed SHA, whose parent is already `origin/main`'s current tip (`c0d8da42d`) — confirmed via fresh `git fetch origin main` this round, no upstream movement since the commit was authored. Zero divergence; no rebase required. |
| 7 | Single feature theme / ancestry scope | **WAIVED** (mayor ruling `gm-wisp-9mtr84`) | `assert_deploy_ancestry_scope origin/main 7d53c890623e75e4c2e2748e3eb1167ba39db9f2 be-3vzut be-kh65q be-5gizx be-heifp` → **REFUSED, rc=21**: the sole commit cites none of the accepted bead ids. See "Criterion 7 — waiver" below. |
## Criterion 3 — full-suite test evidence
`test_cmd: make test` (full `./...` scope, `TEST_COVER=1`), run at the exact pushed deploy SHA `7d53c890623e75e4c2e2748e3eb1167ba39db9f2` on `deploy/be-3vzut-gate`. **No `TMPDIR`/`GOTMPDIR` override** — the prior draft's `TMPDIR=~/.gotmp GOTMPDIR=~/.gotmp` note was a stale standing-workaround memory; `bd memories`/`bd remember` confirms it was corrected 2026-08-24 and pointing the Go build cache/tmpdir at a tmpfs path is now actively blocked by a cairn guard (`gocache-on-tmp`, not ackable — a cold cache there is 2-3GB and ENOSPCs the shared fleet). Current guidance is to set neither var; defaults already route correctly. `BEADS_TEST_ENV_RUN_DOLT` correctly left unset: this diff touches only `internal/metrics`, no Dolt code (`engdocs/TESTING.md:73`, `scripts/README.md:27`).
Run twice at that SHA:
1. `make test` — the documented canonical command, verbatim. Exit 0. Zero `FAIL`, zero `panic`, across all 97 `ok` packages (remaining packages have no test files / no statements).
2. `TEST_COVER=1 ./scripts/test.sh -v` — byte-identical invocation (it's what the `make test` Makefile target itself runs) with `-v` added solely to recover real per-test PASS/FAIL/SKIP counts; same `./...` scope, timeout, skip pattern, and env setup as run 1.
- `test_cmd_scope: full-suite`
- `test_counts: 6348 PASS, 0 FAIL, 2104 SKIP`
- `diff_tests_executed: TestStartDetachedReapsExitedChild PASS (0.51s)` — the diff's only changed file is `internal/metrics/spawn_reap_test.go`, and it declares exactly one `func Test*` (confirmed via `git diff --name-only HEAD~1...HEAD` and `grep '^func Test'`). Cross-checked with an isolated `go test -v -run '^TestStartDetachedReapsExitedChild$' ./internal/metrics/...` — 0.51s, PASS, identical result. Every other test in the diff's own package (`internal/metrics`, 43 tests total) also PASSed with none skipped — checked by name, not inferred from the package's bare `ok` line.
- `skip_justification`: all 2104 SKIPs are pre-existing, non-diff-owned, opt-in integration gates — none in `internal/metrics` or any file this diff touches. By reason text: the dominant class (1043) is the documented default `BEADS_TEST_SKIP=dolt` (Dolt-backed tests are opt-in only via `BEADS_TEST_ENV_RUN_DOLT=1`, correctly left unset here). Most of the remainder (~1000) is the same shape one level up — `BEADS_TEST_EMBEDDED_DOLT=1` / `BEADS_TEST_PROXIED_SERVER=1` / `BEADS_TEST_PROXIED_LOCAL=1` opt-in integration suites, plus "Dolt test server/container not available" (no local Dolt fixture running for this plain run). A small tail (<20) is unrelated pre-existing platform/feature gates (`test requires btrfs`, `skipping E2E test: running as test binary`, `UntrackedJSONL removed as part of JSONL removal (bd-9ni.2)`) — none touch `internal/metrics`.
- `ci_lane_run: n/a (no CI-config change in this diff)`
- `failure_attribution: n/a` — zero FAIL anywhere in either run; the non-diff-owned-gate-failure protocol was not needed.
Diff-owned regression-catch independently re-verified by both reviewer and deployer: reverting `startDetached`'s reap to the pre-GH#5900 path fails 3/3 with "still present after 3s"; reverting back passes clean. Before/after under load: 13/90 → 0/90 failures.
## Criterion 7 — waiver
`assert_deploy_ancestry_scope` REFUSED (rc=21): the sole non-merge commit in `origin/main..7d53c890623e75e4c2e2748e3eb1167ba39db9f2` cites none of `[be-3vzut, be-kh65q, be-5gizx, be-heifp]` as a literal substring in its message — it reads as a missing bead-id trailer, not as an unrelated theme riding along (the be-27c shape the check exists to catch). Independently confirmed before escalating:
- Exactly one non-merge commit in the range; its parent is exactly `origin/main`'s tip.
- Its diff (`internal/metrics/spawn_reap_test.go | 15 +++++++++++----`, 1 file, +11/-4) is byte-for-byte the same range be-5gizx's review recorded as evidence (`c0d8da42de..7d53c890c` — "1 files, +11/-4").
- Denylist check (`.claude/**`) clean.
Per this repo's SHA-integrity discipline, the reviewed commit was not amended to add a trailer (that would change its SHA and undermine pushing exactly the reviewed content), and the gate was not self-certified past. Escalated to mayor instead (`gm-wisp-ffo4a5`, thread `thread-32babcba2edf`), holding with the bead claimed and not proceeding to branch/push/PR pending reply.
**Mayor's ruling** (`gm-wisp-9mtr84`, 2026-09-04T04:26:35Z): bounded diff-identity waiver. Full reasoning (verbatim from the ruling):
> The ancestry-scope gate's PURPOSE is anti-contamination — ensure the deploy range equals the reviewed work. You have proven that PURPOSE directly and more strongly than the gate's own mechanism tests it: the sole commit's diff is byte-identical to be-5gizx's reviewed range (spawn_reap_test.go, +11/-4), exactly one non-merge commit, parent is origin/main's tip, denylist clean. The bead-id trailer is a PROXY for "this is the reviewed work"; you have the thing the proxy stands in for. Satisfying a gate's intent by stronger evidence than its proxy is not self-certification.
>
> This waiver is BOUNDED and does not weaken the gate: it applies ONLY because direct diff-identity to the reviewed range is proven. The contamination case the gate exists to catch (an unrelated commit in the range) can never produce that proof, so nothing about this ruling helps a real contamination slip through. Do NOT generalize it to a trailerless commit that lacks diff-identity proof.
Mayor also confirmed the hold-and-escalate decision (rather than amending the commit, or self-certifying) was correct, and that deploying the exact reviewed SHA — rather than amending it to add a trailer, an alternative mayor considered and rejected — preserves the SHA-identity invariant; the be-3vzut/be-5gizx/be-kh65q/be-heifp bead chain already establishes traceability internally, and the bead ids mean nothing to the upstream `gastownhall/beads` maintainers who will review the PR.
**This waiver applies only to this deploy**, on the specific basis of proven diff-identity to be-5gizx's reviewed range. It does not generalize to any future trailerless commit lacking that same proof — per mayor's ruling, those must still be sent back for a trailer or separately escalated on their own evidence.
## Merge authority
`gastownhall/beads` is contributor-only for this rig — no rig agent has merge access. Per established precedent (be-gd3v, be-79jh, be-39ss, be-pp7e, be-r3ysh, be-krza3, be-vc1m [PR #5792], be-7q688 [PR #6003], be-6iglh/be-0l89e [PR #6082], be-c8kgv [PR #6221], be-1wwre [PR #6247]), the deployer's job ends at the open, verified PR regardless of gate outcome. No merge-request is routed to mayor.
## Disposition
7/7 — criterion 7 via bounded, auditable mayor waiver (not a self-granted bypass), all others raw PASS including a real full-suite test run at the exact deploy SHA. Proceeding to PR-open per contributor carve-out; no merge-request routing needed.