beads/scripts/pr-preflight.sh
matt wilkie c3e600c940
fix(pr-preflight): warn when a PR-gate workflow is red for every PR while the base shows green (#5251)
* fix(pr-preflight): sample PR-gate health, not just base-branch runs (bd-msll)

The base-branch health check reads gh run list --branch <base>, which only
sees workflows that run on the base branch itself. A job that exists only in
the PR workflow (pr.yml) never runs there, so preflight could report "base is
green" while every fresh PR was red in that job, and pr-babysit mirrors
preflight's verdict. Add a supplemental sample of recent completed
pull_request-event runs across PRs: warn (or block under
PR_PREFLIGHT_BLOCK_RED_BASE=1) only when at least 6 decisive runs span 3+
distinct branches and every one of them failed; stay silent otherwise
(some success, too few samples, or an empty/unreadable sample).

Agent-Signature: claude-sonnet-5-medium on behalf of matt wilkie

* fix(pr-preflight): per-workflow PR-gate detection, warn-only, skip on red base

Review found the lumped-sample design could never fire for a single broken
pr.yml-only job: mixing all pull_request-event runs together diluted one red
workflow against every other green PR workflow. Rework to group_by(workflowName)
and flag any single workflow whose decisive runs (>=5, >=3 distinct
headBranch values) are ALL failure-class, naming that workflow in the message.

Also: drop the PR_PREFLIGHT_BLOCK_RED_BASE coupling entirely (a [block] line
here matched neither pr-babysit's red-base nor transient-block patterns, which
would have parked every merge lane and revoked the red-base merge exception —
wiring this into the patrol is a separate change); skip the sample when the
base branch is already red (that already explains PR redness and avoids the
contradictory "base branch shows green" wording); add startup_failure to the
failure-class regex to mirror pr-babysit's probe; and guard against gh exiting
0 with non-JSON output so a malformed sample stays silent instead of aborting
under set -euo pipefail.

Tests reworked for the per-workflow design: fake gh keys the PR-gate dispatch
on the presence of --event pull_request; new cases cover a mixed-workflow
sample (only the all-red workflow fires and is named), red-base-skips-sample,
an unreadable/non-JSON sample, positive coverage of the base-branch dispatch
via BASE_RUN_LIST, and the newest-run selection. Removed the
PR_PREFLIGHT_BLOCK_RED_BASE=1 block-mode test for this detector.

Agent-Signature: claude-sonnet-5-medium on behalf of matt wilkie

* test(pr-preflight): pin the PR-gate detector's warn-only contract; harden jq guard; document

Review round: a regression test asserts the detector never emits [block]
even under PR_PREFLIGHT_BLOCK_RED_BASE=1 (downstream automation classifies
unrecognized block lines as genuine merge blockers), the group_by jq gets an
abort guard for valid-JSON-but-not-objects input, and scripts/README.md
describes the new sample.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie

* fix(pr-preflight): only diagnose the PR gate against a known-green base

Codex review round: with zero decisive base runs preflight has just warned
that base health is undetermined, and the broken-gate message's 'while the
base branch shows green' clause would contradict it. Gate the sample on
base_green_count > 0; fixture cases now supply a green base sample and a new
case pins the undetermined-base silence.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie

* fix(pr-preflight): group the PR-gate sample by workflowDatabaseId, not display name

Codex review round 2: workflow display names are not unique and org/ruleset
runs can lack one entirely, so name-grouping could merge distinct gates.
Group by the stable id, keep the name for display. The repo-wide sample's
base-branch conflation is documented as an accepted limitation of a
warn-only supplemental detector.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
2026-08-01 22:46:44 -07:00

430 lines
17 KiB
Bash
Executable file
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

#!/usr/bin/env bash
# Agent PR preflight for beads.
#
# This script is read-only. It turns the contributor-protection policy into a
# concrete checklist before an agent reviews, closes, merges, or supersedes a PR.
set -euo pipefail
usage() {
cat <<'EOF'
Usage:
scripts/pr-preflight.sh <pr-number-or-url> [--repo owner/name]
scripts/pr-preflight.sh --search "<topic keywords>" [--repo owner/name]
Before changing or merging an existing PR:
scripts/pr-preflight.sh 123 --repo gastownhall/beads
Before implementing a feature/fix or opening a replacement PR:
scripts/pr-preflight.sh --search "dependency cycle detection" --repo gastownhall/beads
What this checks:
- whether an existing PR is external/cross-repository contributor work
- draft, review, mergeability, and check status
- CI health of the base branch (red base makes "failures are pre-existing"
reasoning unsafe; set PR_PREFLIGHT_BLOCK_RED_BASE=1 to block instead of warn)
- a supplemental, warn-only sample for a PR-gate workflow (e.g. pr.yml) that
is red for every PR while the base branch itself looks green
- risky diff signals: .beads data, missing tests for code changes, large diffs
- contributor-protection next steps and attribution reminders
The script does not replace code review or local validation.
EOF
}
die() {
printf 'error: %s\n' "$*" >&2
exit 2
}
need() {
command -v "$1" >/dev/null 2>&1 || die "missing required command: $1"
}
repo_from_url() {
sed -E \
-e 's#^git@github.com:##' \
-e 's#^https://github.com/##' \
-e 's#^ssh://git@github.com/##' \
-e 's#\.git$##' \
<<<"$1"
}
default_repo() {
local url
if url=$(git remote get-url upstream 2>/dev/null); then
repo_from_url "$url"
return
fi
if url=$(git remote get-url origin 2>/dev/null); then
repo_from_url "$url"
return
fi
gh repo view --json nameWithOwner --jq .nameWithOwner
}
closing_issue_count() {
local pr_url="$1"
local pr_number="$2"
local repo_host repo_owner repo_name response count
if [[ "$pr_url" =~ ^https://([^/]+)/([^/]+)/([^/]+)/pull/([0-9]+)$ ]]; then
repo_host="${BASH_REMATCH[1]}"
repo_owner="${BASH_REMATCH[2]}"
repo_name="${BASH_REMATCH[3]}"
if [[ "${BASH_REMATCH[4]}" != "$pr_number" ]]; then
die "PR URL number does not match returned PR number: ${pr_url}"
fi
else
die "invalid canonical GitHub PR URL: ${pr_url}"
fi
# GraphQL variables are intentionally literal and are bound by gh -f/-F.
# shellcheck disable=SC2016
if ! response=$(gh api graphql \
--hostname "$repo_host" \
-f query='query($owner: String!, $name: String!, $number: Int!) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
closingIssuesReferences(first: 1) { nodes { id } }
}
}
}' \
-f owner="$repo_owner" \
-f name="$repo_name" \
-F number="$pr_number"); then
die "could not query closing issue references for ${repo_owner}/${repo_name}#${pr_number}"
fi
if ! count=$(jq -ser '
select(length == 1)
| .[0]
| select(has("errors") | not)
| .data.repository.pullRequest.closingIssuesReferences.nodes
| if type != "array" then empty
elif length == 0 then "0"
elif length == 1
and (.[0] | type == "object")
and (.[0] | keys == ["id"])
and (.[0].id | type == "string" and length > 0)
then "1"
else empty
end
' <<<"$response"); then
die "invalid closing issue response for ${repo_owner}/${repo_name}#${pr_number}"
fi
printf '%s\n' "$count"
}
repo=""
pr=""
search=""
while (($#)); do
case "$1" in
--repo)
[[ $# -ge 2 ]] || die "--repo requires owner/name"
repo="$2"
shift 2
;;
--search)
[[ $# -ge 2 ]] || die "--search requires keywords"
search="$2"
shift 2
;;
-h|--help)
usage
exit 0
;;
-*)
die "unknown option: $1"
;;
*)
[[ -z "$pr" ]] || die "only one PR may be provided"
pr="$1"
shift
;;
esac
done
need gh
need git
need jq
repo="${repo:-$(default_repo)}"
[[ -n "$repo" ]] || die "could not determine GitHub repo; pass --repo owner/name"
if [[ -n "$search" ]]; then
printf 'PR topic preflight for %s\n' "$repo"
printf 'Search: %s\n\n' "$search"
results=$(gh pr list \
--repo "$repo" \
--state open \
--search "$search" \
--json number,title,author,headRefName,isCrossRepository,url,updatedAt)
count=$(jq 'length' <<<"$results")
if [[ "$count" -eq 0 ]]; then
printf '[pass] No open PRs matched this search.\n'
printf '[next] Use broader keywords if the topic is ambiguous before opening a new PR.\n'
exit 0
fi
printf '[block] Found %s open PR(s). Review relevant contributor work before building separately.\n\n' "$count"
jq -r '.[] |
"- PR #\(.number): \(.title)\n author: \(.author.login) external: \(.isCrossRepository) branch: \(.headRefName)\n url: \(.url)"' \
<<<"$results"
printf '\n[next] Run scripts/pr-preflight.sh <number> --repo %s for each relevant PR.\n' "$repo"
exit 1
fi
[[ -n "$pr" ]] || { usage; exit 2; }
json=$(gh pr view "$pr" \
--repo "$repo" \
--json number,title,author,url,baseRefName,headRefName,headRepositoryOwner,isCrossRepository,isDraft,maintainerCanModify,mergeStateStatus,mergeable,reviewDecision,changedFiles,additions,deletions,files,statusCheckRollup,latestReviews)
number=$(jq -r .number <<<"$json")
title=$(jq -r .title <<<"$json")
author=$(jq -r .author.login <<<"$json")
url=$(jq -r .url <<<"$json")
external=$(jq -r .isCrossRepository <<<"$json")
draft=$(jq -r .isDraft <<<"$json")
review=$(jq -r 'if (.reviewDecision // "") == "" then "REVIEW_REQUIRED" else .reviewDecision end' <<<"$json")
merge_state=$(jq -r '.mergeStateStatus // "UNKNOWN"' <<<"$json")
mergeable=$(jq -r '.mergeable // "UNKNOWN"' <<<"$json")
files=$(jq -r .changedFiles <<<"$json")
additions=$(jq -r .additions <<<"$json")
deletions=$(jq -r .deletions <<<"$json")
maintainer_can_modify=$(jq -r '.maintainerCanModify // false' <<<"$json")
status=0
block() {
printf '[block] %s\n' "$*"
status=1
}
warn() {
printf '[warn] %s\n' "$*"
}
pass() {
printf '[pass] %s\n' "$*"
}
printf 'PR preflight for %s#%s\n' "$repo" "$number"
printf 'Title: %s\n' "$title"
printf 'Author: %s\n' "$author"
printf 'URL: %s\n' "$url"
printf 'Base/head: %s <- %s\n' "$(jq -r .baseRefName <<<"$json")" "$(jq -r .headRefName <<<"$json")"
printf 'Diff: %s files, +%s/-%s\n\n' "$files" "$additions" "$deletions"
if [[ "$external" == "true" ]]; then
warn "External contributor PR. Do not rewrite, close, or supersede silently."
if [[ "$maintainer_can_modify" == "true" ]]; then
pass "Maintainers can push fixes to this PR branch."
else
warn "Maintainers cannot push to this PR branch; coordinate in comments or use a credited follow-up branch."
fi
else
pass "PR is not cross-repository contributor work."
fi
if [[ "$draft" == "true" ]]; then
block "PR is draft."
fi
case "$review" in
APPROVED)
pass "Review decision is APPROVED."
;;
CHANGES_REQUESTED)
block "Review decision is CHANGES_REQUESTED."
;;
*)
warn "Review decision is ${review}."
;;
esac
case "$merge_state" in
CLEAN|HAS_HOOKS)
pass "Merge state is ${merge_state}."
;;
BLOCKED|DIRTY|UNKNOWN|UNSTABLE)
block "Merge state is ${merge_state}."
;;
*)
warn "Merge state is ${merge_state}; verify in GitHub before merging."
;;
esac
if [[ "$mergeable" == "CONFLICTING" ]]; then
block "GitHub reports merge conflicts."
fi
# Base-branch CI health. Merging onto a red base is how breakage stacks:
# every open PR inherits the red checks, "failures are pre-existing" becomes
# the default reasoning, and new failures ride in under that cover. Cancelled
# runs (superseded by a newer push) carry no signal, so judge by the newest
# completed run that finished with a decisive green or red conclusion.
base_ref=$(jq -r .baseRefName <<<"$json")
base_runs=$(gh run list --repo "$repo" --branch "$base_ref" --status completed \
--limit 30 --json conclusion,workflowName,createdAt,url 2>/dev/null) || base_runs=""
# Latest decisive run per workflow, so a green unrelated workflow that happened
# to run last cannot mask a red test workflow (and vice versa).
base_latest_per_wf=$(jq '[group_by(.workflowName)[]
| [.[] | select((.conclusion // "") == "success" or ((.conclusion // "") | test("^(failure|timed_out|action_required)$")))][0]
| select(. != null)]' <<<"${base_runs:-[]}")
base_red=$(jq '[.[] | select((.conclusion // "") | test("^(failure|timed_out|action_required)$"))]' <<<"$base_latest_per_wf")
base_red_count=$(jq 'length' <<<"$base_red")
base_green_count=$(jq '[.[] | select(.conclusion == "success")] | length' <<<"$base_latest_per_wf")
if [[ "$base_red_count" -gt 0 ]]; then
red_base_msg="Base branch ${base_ref} CI is RED: $(jq -r 'map("\(.workflowName) failed at \(.createdAt) (\(.url))") | join("; ")' <<<"$base_red"). Check failures on this PR may be pre-existing base breakage, and merging onto a red base hides new failures. Merge only the fix for ${base_ref} while it is red; for everything else wait, then update the branch and re-run checks."
if [[ "${PR_PREFLIGHT_BLOCK_RED_BASE:-0}" == "1" ]]; then
block "$red_base_msg"
else
warn "$red_base_msg"
fi
elif [[ "$base_green_count" -gt 0 ]]; then
pass "Base branch ${base_ref} CI is green (latest completed run of each of ${base_green_count} workflow(s) succeeded)."
else
warn "Could not determine CI health of base branch ${base_ref}; check it manually before merging."
fi
# PR-gate health sample. The base-branch check above reads runs on the base
# branch itself, so it cannot see a job that exists only in the PR workflow
# (pr.yml) - such a job never runs on the base branch, and one broken job
# there is diluted to invisibility by every other green PR workflow if runs
# are judged in aggregate. Sample recent completed pull_request-event runs
# across PRs, group by workflow, and flag any single workflow whose decisive
# runs are uniformly failing across several distinct branches.
#
# This is a supplemental, warn-only detector, deliberately never coupled to
# PR_PREFLIGHT_BLOCK_RED_BASE: a [block] line here matches neither the
# red-base nor the transient-block patterns pr-babysit's patrol looks for,
# so blocking would park every merge lane and silently revoke the red-base
# merge exception. (Wiring this signal into the patrol is a separate,
# coordinated change.) Skip the sample entirely when the base branch is
# already red - that already explains per-PR redness, and avoids reporting
# "the base branch shows green" right after a red-base line above - and stay
# silent on an empty, unreadable, or mixed-health sample.
if [[ "$base_red_count" -eq 0 && "$base_green_count" -gt 0 ]]; then
# Only diagnose the PR gate against a base branch known to be green: on a red
# base the red-base message already explains PR redness, and on an
# undetermined base the "while the base branch shows green" explanation
# would contradict the warn printed just above.
pr_gate_runs=$(gh run list --repo "$repo" --event pull_request --status completed \
--limit 60 --json conclusion,headBranch,workflowName,workflowDatabaseId,createdAt,url 2>/dev/null) || pr_gate_runs=""
# gh can exit 0 with non-JSON (or empty) output; treat anything that is not
# a JSON array the same as an empty sample rather than let jq abort under -e.
jq -e 'type == "array"' >/dev/null 2>&1 <<<"${pr_gate_runs:-}" || pr_gate_runs='[]'
# Grouped by workflowDatabaseId (stable identity), not display name: two
# workflow files may share a legal name:, and org/ruleset runs can have no
# name at all. Known limitation, accepted for a warn-only supplemental
# detector: the sample is repo-wide, so runs from PRs targeting other base
# branches are mixed in — resolving each run's PR to filter by base would
# cost one API call per run.
pr_gate_broken=$(jq '
[group_by(.workflowDatabaseId)[]
| { workflow: (.[0].workflowName // "unknown"),
decisive: [.[] | select(
(.conclusion // "") == "success" or
((.conclusion // "") | test("^(failure|timed_out|action_required|startup_failure)$"))
)] }
| select((.decisive | length) >= 5)
| select((.decisive | map(.headBranch) | unique | length) >= 3)
| select(all(.decisive[]; (.conclusion // "") | test("^(failure|timed_out|action_required|startup_failure)$")))
] | .[0] // empty' <<<"$pr_gate_runs") || pr_gate_broken=""
if [[ -n "$pr_gate_broken" ]]; then
pr_gate_workflow=$(jq -r '.workflow' <<<"$pr_gate_broken")
pr_gate_count=$(jq -r '.decisive | length' <<<"$pr_gate_broken")
pr_gate_heads=$(jq -r '.decisive | map(.headBranch) | unique | length' <<<"$pr_gate_broken")
pr_gate_newest=$(jq -r '.decisive | sort_by(.createdAt) | last | "\(.createdAt) \(.url)"' <<<"$pr_gate_broken")
warn "PR gate workflow '${pr_gate_workflow}' appears broken: all ${pr_gate_count} of its recent completed pull_request runs across ${pr_gate_heads} branches failed (newest: ${pr_gate_newest}). A job that exists only in that workflow may be red for every PR while the base branch shows green. Investigate before trusting per-PR check verdicts."
fi
fi
# statusCheckRollup keeps every check run on the head SHA, including runs
# from cancelled or superseded check suites. A head whose earlier suite was
# cancelled mid-flight carries stale CANCELLED/FAILURE entries forever, next
# to the fresh green re-runs of the same checks — GitHub's own mergeability
# uses latest-per-name, so counting raw entries permanently blocks PRs that
# GitHub reports CLEAN (observed on gastownhall/beads#5073–5076: 20 stale
# CANCELLED runs + 1 stale gate FAILURE alongside 80 green checks). Collapse
# to the most recent entry per check name/context before classifying.
# Group key includes __typename and workflowName: a commit status and a
# check run sharing a name are independent gates (GitHub requires both),
# and identically named jobs in different workflows must not collapse into
# one entry where a green run could mask the other workflow's red one.
# gh exports absent CheckRun times as the zero time (0001-01-01...), so
# normalize before comparing or the // fallback never fires; "latest" is
# the max of completed/started/created so a fresh in-progress rerun beats
# an old completed failure (and is then reported as pending, not failed).
rollup_latest=$(jq '
def nz: (. // "") | if startswith("0001-") then "" else . end;
[.statusCheckRollup[]?]
| group_by([(.__typename // ""), (.workflowName // ""), (.name // .context // "")])
| map(max_by([(.completedAt | nz), (.startedAt | nz), (.createdAt | nz)] | max))' <<<"$json")
failed_checks=$(jq '[.[] | select(
((.conclusion // "") | test("FAILURE|CANCELLED|TIMED_OUT|ACTION_REQUIRED")) or
((.state // "") | test("ERROR|FAILURE"))
)] | length' <<<"$rollup_latest")
pending_checks=$(jq '[.[] | select(
if (.status? // null) != null then
.status != "COMPLETED"
elif (.state? // null) != null then
(.state | test("^(EXPECTED|PENDING)$"))
else
true
end
)] | length' <<<"$rollup_latest")
if [[ "$failed_checks" -gt 0 ]]; then
block "$failed_checks status check(s) failed or require action."
elif [[ "$pending_checks" -gt 0 ]]; then
block "$pending_checks status check(s) are still pending."
else
pass "No failed or pending status checks reported."
fi
beads_files=$(jq '[.files[]?.path | select(startswith(".beads/"))] | length' <<<"$json")
if [[ "$beads_files" -gt 0 ]]; then
block "PR changes .beads/ data; contributor PRs must not include planning database changes."
fi
code_files=$(jq '[.files[]?.path | select(test("\\.(go|py|sh)$"))] | length' <<<"$json")
test_files=$(jq '[.files[]?.path | select(test("(^|/)(test|tests)/|_test\\.go$|\\.bats$"))] | length' <<<"$json")
if [[ "$code_files" -gt 0 && "$test_files" -eq 0 ]]; then
warn "Code changed but no obvious test files changed."
fi
if [[ "$files" -gt 30 || "$additions" -gt 1000 ]]; then
warn "Large PR; verify scope is one issue and one PR."
fi
issue_count=$(closing_issue_count "$url" "$number")
if [[ "$issue_count" == "0" ]]; then
warn "No closing issue reference found."
else
pass "PR references at least one closing issue."
fi
printf '\nChanged files:\n'
jq -r '.files[]?.path | " - \(.)"' <<<"$json"
printf '\nRequired agent behavior:\n'
printf ' - Review contributor work first; build on it when relevant.\n'
printf ' - Preserve contributor tests unless they are wrong.\n'
printf ' - Preserve attribution with existing commits or Co-authored-by trailers.\n'
printf ' - If a rewrite is unavoidable, explain why on PR #%s and credit design/tests.\n' "$number"
printf ' - Run local validation before merge, normally: make test\n'
if [[ "$status" -ne 0 ]]; then
printf '\nResult: BLOCKED for autonomous merge/close/replacement until addressed.\n'
else
printf '\nResult: preflight passed. Continue with code review and local validation.\n'
fi
exit "$status"