beads/scripts/mutation-equivalence.sh
Julian Knutsen 869020c221
Fix a SIGPIPE in the mutation harness's worktree guard (#5434)
Reported by the first caller to use scripts/mutation-equivalence.sh at scale: it
killed 5 of 9 mutation runs, presenting as "not a worktree of $REPO" for a path
that plainly was one.

grep -q exits on the first match, which SIGPIPEs the git worktree list feeding
it. Under set -o pipefail the pipeline is then nonzero and the negation takes
the failure branch, rejecting a legitimate worktree. It only bites once the
listing is large enough that git is still writing when grep exits — this repo
has ~370 worktrees and a 53 KB listing — so it reads as intermittent and
machine-specific.

Captured into a variable instead of piped. Verified both directions: a real
worktree is now recognised and falls through to the branch-checked-out guard as
intended.

The guard this broke was added two commits earlier, in response to a review that
found the tool would hard-reset a live checkout. Tightening it introduced a
failure mode that made it refuse legitimate ones. Worth recording, because the
reflex when a guard misfires is to loosen the guard rather than look for the bug
in it — and the caller who hit this worked around the symptom rather than
reporting it, which is how a tool acquires a reputation for flakiness.
2026-08-08 08:28:08 -07:00

223 lines
11 KiB
Bash
Executable file

#!/usr/bin/env bash
#
# Mutation equivalence: does test B catch what test A catches?
#
# ./scripts/mutation-equivalence.sh <prod-file> <mutation.py> <pkgA> <runA> <pkgB> <runB>
#
# The question this answers is NOT "do these two tests look similar". That
# reading is how you delete real coverage and keep every gate green. It is:
# when the production body breaks, do BOTH suites notice?
#
# A red, B red the contract catches it too — A is redundant
# A red, B green only A caught it — A is UNIQUE; port it, then delete
# A green, B red A does not cover this break — wrong mutation for this pairing
# both green neither covers it — proves nothing about either
#
# WHAT THIS TOOL CANNOT TELL YOU, and what a caller must supply: a verdict is
# only true of the BODY THE MUTATION WAS IN. The uow provider runs its own
# bodies under internal/storage/domain/ where the two store backends share
# others, so "A red, B red" against a shared body says nothing about a wrapper
# or a per-backend composition that the same test also observes. Three separate
# coverage losses in this repo came from accepting a red/red verdict for the
# wrong body. Name the body the assertion observes, then mutate THAT.
#
# <mutation.py> receives the file path as argv[1] and rewrites it in place.
#
# The guards below all exist because their absence produced a wrong answer at
# least once:
#
# - All mutation happens in a DISPOSABLE worktree. The tool is routinely
# interrupted mid-run, and a trap does not survive SIGKILL; mutating the
# checkout you are working in strands broken production code that compiles.
# - A `-run` regex matching nothing is fatal. `go test -run NoSuchTest` EXITS
# 0, so a typo reads as PASS forever.
# - A mutation that changes no bytes is fatal. It reports agreement.
# - A baseline that is not green is fatal. The verdict would be noise.
# - A nonzero exit is not a catch. A build failure or timeout is reported as
# UNUSABLE, never as FAIL, because either would turn an assertion-free test
# into a false REDUNDANT.
# - The worktree path must be a worktree of $REPO. It is hard-reset and
# cleaned, so pointing it at a live checkout would destroy that checkout.
# - A failed checkout/reset is fatal: it would measure a DIFFERENT COMMIT and
# still exit 0.
# - The mutation must change only the file it names, and A and B must differ.
# - It measures $REPO's HEAD COMMIT, not its working tree; uncommitted work is
# invisible and the run warns rather than silently measuring the wrong code.
# - Result greps do not anchor leading whitespace: `go test -v` indents
# subtests by 4 and NESTED subtests by 8, so an over-anchored pattern finds
# nothing and reads as "did not fail". scripts/conformance.sh documents the
# mirror image of this hazard for its own column-0 case.
#
set -uo pipefail
usage() {
sed -n '2,51p' "$0" | sed 's|^#||'
exit 2
}
[ $# -eq 6 ] || usage
REPO="${REPO:-$(cd "$(dirname "$0")/.." && pwd)}"
WT="${MUTEQ_WT:-${TMPDIR:-/tmp}/beads-mutation-worktree}"
TIMEOUT="${MUTEQ_TIMEOUT:-1200}"
PROD="$1"; MUT="$2"; APKG="$3"; ARUN="$4"; BPKG="$5"; BRUN="$6"
# Comparing a suite with itself yields REDUNDANT mechanically, because B IS A.
if [ "$APKG" = "$BPKG" ] && [ "$ARUN" = "$BRUN" ]; then
echo "FATAL: A and B are the same selector; the verdict would be tautological" >&2; exit 2
fi
[ -f "$REPO/$PROD" ] || { echo "FATAL: no such file in $REPO: $PROD" >&2; exit 2; }
[ -f "$MUT" ] || { echo "FATAL: no such mutation script: $MUT" >&2; exit 2; }
HEAD_SHA="$(git -C "$REPO" rev-parse HEAD)"
# THIS TOOL MEASURES $REPO's HEAD COMMIT, NOT ITS WORKING TREE. The scratch
# worktree is built from HEAD_SHA, so uncommitted work — including the very test
# you are trying to verify — is invisible to the run. That failure is silent and
# reads as INCONCLUSIVE, which is indistinguishable from "the test is weak"
# unless you know to look. Commit first, or accept that the verdict is about
# what is committed.
if [ -n "$(git -C "$REPO" status --porcelain)" ]; then
echo "WARNING: $REPO has uncommitted changes. This run measures HEAD" >&2
echo " ($(git -C "$REPO" rev-parse --short HEAD)); those changes are NOT in it." >&2
git -C "$REPO" status --short >&2 | head -10
echo >&2
fi
# The worktree path is hard-reset and cleaned below, so pointing it at a live
# checkout DESTROYS that checkout — including $REPO itself, which is the exact
# accident the disposable-worktree design exists to prevent. Only a path this
# script created, or one git already lists as a worktree of $REPO, is eligible.
if [ -e "$WT" ]; then
WT_ABS="$(cd "$WT" 2>/dev/null && pwd)" || WT_ABS=""
REPO_ABS="$(cd "$REPO" && pwd)"
if [ -z "$WT_ABS" ] || [ "$WT_ABS" = "$REPO_ABS" ]; then
echo "FATAL: MUTEQ_WT resolves to \$REPO itself. This script hard-resets and" >&2
echo " cleans that path; running would destroy the checkout it is measuring." >&2
exit 2
fi
# Captured into a variable, NOT piped. `grep -q` exits on the first match,
# which SIGPIPEs git; with `pipefail` that makes the whole pipeline nonzero and
# the negation below then rejects a perfectly good worktree. It only bites once
# the list is large enough for git to still be writing — this repo has ~370
# worktrees / 53 KB — so it looks intermittent and machine-specific. It killed
# 5 of 9 runs for the first caller who hit it.
WT_LIST="$(git -C "$REPO" worktree list --porcelain)"
if ! grep -qx "worktree $WT_ABS" <<<"$WT_LIST"; then
echo "FATAL: $WT exists but is not a worktree of $REPO." >&2
echo " This script hard-resets and cleans that path; refusing to touch it." >&2
exit 2
fi
# A worktree with a branch checked out is somebody's working copy, not a
# scratch one this tool created. Only a DETACHED head is disposable.
if git -C "$WT" symbolic-ref -q HEAD >/dev/null 2>&1; then
echo "FATAL: $WT has a branch checked out ($(git -C "$WT" rev-parse --abbrev-ref HEAD))." >&2
echo " Only a detached scratch worktree is safe to hard-reset. Point" >&2
echo " MUTEQ_WT at a path this tool created, or at nothing." >&2
exit 2
fi
fi
if [ ! -e "$WT/.git" ]; then
echo "== creating mutation worktree at $WT =="
git -C "$REPO" worktree add --detach "$WT" "$HEAD_SHA" >/dev/null 2>&1 \
|| { echo "FATAL: could not create worktree at $WT" >&2; exit 2; }
else
# A failure here would silently measure a DIFFERENT COMMIT and still exit 0,
# so it is fatal rather than suppressed.
git -C "$WT" checkout -q --detach "$HEAD_SHA" \
|| { echo "FATAL: could not check out $HEAD_SHA in $WT" >&2; exit 2; }
fi
git -C "$WT" reset -q --hard "$HEAD_SHA" \
|| { echo "FATAL: could not reset $WT to $HEAD_SHA" >&2; exit 2; }
git -C "$WT" clean -qfd
if [ "$(git -C "$WT" rev-parse HEAD)" != "$HEAD_SHA" ]; then
echo "FATAL: $WT is at $(git -C "$WT" rev-parse --short HEAD), not $REPO's HEAD" >&2
exit 2
fi
# A previous run that was killed can leave the worktree dirty, which would make
# this run's "baseline" someone else's mutation.
if [ -n "$(git -C "$WT" status --porcelain)" ]; then
echo "FATAL: mutation worktree is dirty after reset; refusing to run" >&2
git -C "$WT" status --short >&2
exit 2
fi
cd "$WT" || exit 2
# shellcheck disable=SC1091
[ -f .buildflags ] && source .buildflags
export BEADS_TEST_EMBEDDED_DOLT="${BEADS_TEST_EMBEDDED_DOLT:-1}"
restore() { git -C "$WT" checkout -q -- . 2>/dev/null; }
trap restore EXIT INT TERM
# Count executed tests. Leading whitespace deliberately unanchored.
count() {
timeout "$TIMEOUT" go test "$1" -run "$2" -count=1 -v 2>/dev/null | grep -cE '^ *=== RUN'
}
# A nonzero `go test` exit is NOT the same as "this suite caught the mutation".
# A build failure and a timeout both exit nonzero, and both would otherwise read
# as a catch — turning an assertion-free test into a false REDUNDANT, which is
# the one wrong answer that costs coverage.
result() {
local out rc
out="$(timeout "$TIMEOUT" go test "$1" -run "$2" -count=1 2>&1)"; rc=$?
if [ $rc -eq 124 ]; then echo TIMEOUT; return; fi
if grep -qE '^(# |.*\[build failed\]|.*\[setup failed\])' <<<"$out"; then echo BUILDFAIL; return; fi
[ $rc -eq 0 ] && echo PASS || echo FAIL
}
echo "== baseline ($WT @ $(git rev-parse --short HEAD)) =="
AN=$(count "$APKG" "$ARUN")
BN=$(count "$BPKG" "$BRUN")
[ "${AN:-0}" -gt 0 ] || { echo "FATAL: -run '$ARUN' matches no test in $APKG" >&2; exit 3; }
[ "${BN:-0}" -gt 0 ] || { echo "FATAL: -run '$BRUN' matches no test in $BPKG" >&2; exit 3; }
A0=$(result "$APKG" "$ARUN"); B0=$(result "$BPKG" "$BRUN")
printf " A %-4s %3s executed %s %s\n" "$A0" "$AN" "$APKG" "$ARUN"
printf " B %-4s %3s executed %s %s\n" "$B0" "$BN" "$BPKG" "$BRUN"
if [ "$A0" != PASS ] || [ "$B0" != PASS ]; then
echo "FATAL: baseline is not green; a mutation result would mean nothing" >&2; exit 4
fi
echo "== mutating $PROD =="
BEFORE="$(mktemp)"; cp "$PROD" "$BEFORE"
python3 "$MUT" "$PROD" || { rm -f "$BEFORE"; echo "FATAL: mutation script failed" >&2; exit 5; }
if cmp -s "$BEFORE" "$PROD"; then
rm -f "$BEFORE"; echo "FATAL: mutation changed nothing — it would report false agreement" >&2; exit 6
fi
if ! go build ./... >/dev/null 2>&1; then
rm -f "$BEFORE"; echo "FATAL: mutated tree does not compile; use a semantic mutation" >&2; exit 7
fi
# The displayed diff covers only $PROD, so a script that also edited something
# else would show a harmless-looking change while the verdict came from a break
# the operator never saw — the wrong-body hazard, manufactured by this tool.
STRAY="$(git -C "$WT" status --porcelain | awk '{print $2}' | grep -vx "$PROD" || true)"
if [ -n "$STRAY" ]; then
rm -f "$BEFORE"
echo "FATAL: the mutation also changed files it did not name:" >&2
printf ' %s\n' $STRAY >&2
exit 8
fi
diff "$BEFORE" "$PROD" | head -20; rm -f "$BEFORE"
A1=$(result "$APKG" "$ARUN"); B1=$(result "$BPKG" "$BRUN")
printf "== under mutation ==\n A %s\n B %s\n" "$A1" "$B1"
echo "== verdict =="
case "$A1/$B1" in
FAIL/FAIL)
echo " REDUNDANT B catches this too — for THIS body, and for THIS promise."
echo " A retires only when every promise it pins has its own verdict."
;;
FAIL/PASS) echo " UNIQUE Only A caught it. Port the case into B, THEN delete A." ;;
PASS/FAIL) echo " MISDIRECTED A does not cover this mutation. Wrong pairing." ;;
PASS/PASS) echo " INCONCLUSIVE Neither caught it. Proves nothing about either." ;;
*TIMEOUT*|*BUILDFAIL*)
echo " UNUSABLE A suite did not fail on an ASSERTION: A=$A1 B=$B1."
echo " A build failure or a timeout exits nonzero like a real"
echo " catch does. Treating it as one is how an assertion-free"
echo " test reads as REDUNDANT. Fix the mutation and re-run."
exit 9
;;
esac