ci: retry the module fetch in pr-policy-wrapper, not the build (#5901)

* ci: retry the module fetch in pr-policy-wrapper, not the build

PR Policy (wrapper timing) is a required check, and it is the one job in
pr.yml that can be failed by someone else's network. setup-go runs with
cache: false, so every run pulls the entire module graph from
proxy.golang.org, and nothing retries. A single proxy blip fails the check
outright.

Observed on a docs-only PR (one markdown file, +18/-0): the job died inside
a ~12s proxy.golang.org outage with

  stream error: stream ID NNN; INTERNAL_ERROR; received from peer

while 40 other jobs on the same commit, triggered at the same instant, built
the same module graph and passed. The split was by clock, not by content --
nothing about the change could have caused it.

Fetch the modules in their own retried step before the wrapper runs. With
the cache warm, the go build inside scripts/ci/pr-policy.sh compiles without
touching the network and stays unwrapped: a compile error still fails on the
first attempt rather than taking three, and an exhausted retry reports the
fetch as the cause instead of presenting a module outage as a build failure.

The loop mirrors the bounded apt-get steps in migration-test.yml and
release.yml: three attempts, a per-attempt timeout, a fixed backoff with none
after the last, and a step-level timeout-minutes above the resulting worst
case.

* ci: retry the module fetch in main.yml's pr-policy-wrapper too

pr.yml and main.yml carry byte-identical `pr-policy-wrapper` jobs. The
retry added in the previous commit covered only the pr.yml copy, so the
same unretried cold fetch stayed on the main-branch trigger — the job
this PR names is fixed in one workflow and not the other.

Mirrors the pr.yml step exactly: retries the fetch, never the build, and
fails load-bearing while naming the fetch so a module outage is not
reported as a build failure.

This is not a cache step, so the enumerated cache topology pinned by
TestGoCacheOwnershipTopology (scripts/ci_workflow_test.go) is untouched;
`go test ./scripts/` and actionlint are clean.

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jim Wordelman 2026-09-30 07:11:35 -07:00 • committed by GitHub
commit 9c8be5e85f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 72 additions and 0 deletions

View file

@ -367,6 +367,41 @@ jobs:
go-version-file: 'go.mod'
cache: false
# Same cold-fetch exposure as pr.yml's pr-policy-wrapper: setup-go above
# runs with cache: false, so every run pulls the whole module graph from
# the proxy. Retrying it here means a proxy blip costs seconds instead of
# failing a required check.
#
# Deliberately NOT wrapped around the build: `make ci-pr-policy` runs a
# `go build ./cmd/bd` (scripts/ci/pr-policy.sh, build_docs_binary), and a
# compile error is not transient. See pr.yml's copy for the full rationale.
#
# 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s
# of headroom, matching the bounded apt-get steps in migration-test.yml.
- name: Download modules
timeout-minutes: 8
run: |
downloaded=""
for attempt in 1 2 3; do
if timeout 120 go mod download; then
downloaded=yes
break
fi
echo "::warning::go mod download attempt $attempt failed or timed out"
# No backoff after the final attempt - there is nothing left to wait for.
if [ "$attempt" -lt 3 ]; then
sleep 10
fi
done
# Unlike the best-effort apt-get retries elsewhere in this repo, this
# one is load-bearing: nothing downstream can build without the
# modules. Fail here, naming the fetch, so a module outage is not
# reported as a build failure.
if [ -z "$downloaded" ]; then
echo "::error::go mod download failed after 3 attempts - module fetch failure, not a build failure"
exit 1
fi
- name: Run PR policy wrapper
run: make ci-pr-policy

View file

@ -690,6 +690,43 @@ jobs:
go-version-file: 'go.mod'
cache: false
# The module fetch is the only part of this job that talks to the network,
# and setup-go above runs with cache: false, so every run pulls the whole
# module graph from the proxy. Doing it here, retried, means a proxy blip
# costs seconds instead of failing a required check.
#
# Deliberately NOT wrapped around the build: `make ci-pr-policy` runs a
# `go build ./cmd/bd` (scripts/ci/pr-policy.sh, build_docs_binary), and a
# compile error is not transient. Retrying it would make a genuinely broken
# build take three attempts to report and look like a slow one. The fetch
# has a legitimate transient failure mode; the build does not.
#
# 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s
# of headroom, matching the bounded apt-get steps in migration-test.yml.
- name: Download modules
timeout-minutes: 8
run: |
downloaded=""
for attempt in 1 2 3; do
if timeout 120 go mod download; then
downloaded=yes
break
fi
echo "::warning::go mod download attempt $attempt failed or timed out"
# No backoff after the final attempt - there is nothing left to wait for.
if [ "$attempt" -lt 3 ]; then
sleep 10
fi
done
# Unlike the best-effort apt-get retries elsewhere in this repo, this
# one is load-bearing: nothing downstream can build without the
# modules. Fail here, naming the fetch, so a module outage is not
# reported as a build failure.
if [ -z "$downloaded" ]; then
echo "::error::go mod download failed after 3 attempts - module fetch failure, not a build failure"
exit 1
fi
- name: Run PR policy wrapper
run: make ci-pr-policy