From 9c8be5e85f704dfd7be8cd72ebd7a9b216592964 Mon Sep 17 00:00:00 2001 From: Jim Wordelman Date: Wed, 30 Sep 2026 07:11:35 -0700 Subject: [PATCH] ci: retry the module fetch in pr-policy-wrapper, not the build (#5901) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * ci: retry the module fetch in pr-policy-wrapper, not the build PR Policy (wrapper timing) is a required check, and it is the one job in pr.yml that can be failed by someone else's network. setup-go runs with cache: false, so every run pulls the entire module graph from proxy.golang.org, and nothing retries. A single proxy blip fails the check outright. Observed on a docs-only PR (one markdown file, +18/-0): the job died inside a ~12s proxy.golang.org outage with stream error: stream ID NNN; INTERNAL_ERROR; received from peer while 40 other jobs on the same commit, triggered at the same instant, built the same module graph and passed. The split was by clock, not by content -- nothing about the change could have caused it. Fetch the modules in their own retried step before the wrapper runs. With the cache warm, the go build inside scripts/ci/pr-policy.sh compiles without touching the network and stays unwrapped: a compile error still fails on the first attempt rather than taking three, and an exhausted retry reports the fetch as the cause instead of presenting a module outage as a build failure. The loop mirrors the bounded apt-get steps in migration-test.yml and release.yml: three attempts, a per-attempt timeout, a fixed backoff with none after the last, and a step-level timeout-minutes above the resulting worst case. * ci: retry the module fetch in main.yml's pr-policy-wrapper too pr.yml and main.yml carry byte-identical `pr-policy-wrapper` jobs. The retry added in the previous commit covered only the pr.yml copy, so the same unretried cold fetch stayed on the main-branch trigger — the job this PR names is fixed in one workflow and not the other. Mirrors the pr.yml step exactly: retries the fetch, never the build, and fails load-bearing while naming the fetch so a module outage is not reported as a build failure. This is not a cache step, so the enumerated cache topology pinned by TestGoCacheOwnershipTopology (scripts/ci_workflow_test.go) is untouched; `go test ./scripts/` and actionlint are clean. --------- Co-authored-by: Gas City Adopt-PR Co-authored-by: Claude Opus 5.5 --- .github/workflows/main.yml | 35 +++++++++++++++++++++++++++++++++++ .github/workflows/pr.yml | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e8af10a9b..b8c91d791 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -367,6 +367,41 @@ jobs: go-version-file: 'go.mod' cache: false + # Same cold-fetch exposure as pr.yml's pr-policy-wrapper: setup-go above + # runs with cache: false, so every run pulls the whole module graph from + # the proxy. Retrying it here means a proxy blip costs seconds instead of + # failing a required check. + # + # Deliberately NOT wrapped around the build: `make ci-pr-policy` runs a + # `go build ./cmd/bd` (scripts/ci/pr-policy.sh, build_docs_binary), and a + # compile error is not transient. See pr.yml's copy for the full rationale. + # + # 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s + # of headroom, matching the bounded apt-get steps in migration-test.yml. + - name: Download modules + timeout-minutes: 8 + run: | + downloaded="" + for attempt in 1 2 3; do + if timeout 120 go mod download; then + downloaded=yes + break + fi + echo "::warning::go mod download attempt $attempt failed or timed out" + # No backoff after the final attempt - there is nothing left to wait for. + if [ "$attempt" -lt 3 ]; then + sleep 10 + fi + done + # Unlike the best-effort apt-get retries elsewhere in this repo, this + # one is load-bearing: nothing downstream can build without the + # modules. Fail here, naming the fetch, so a module outage is not + # reported as a build failure. + if [ -z "$downloaded" ]; then + echo "::error::go mod download failed after 3 attempts - module fetch failure, not a build failure" + exit 1 + fi + - name: Run PR policy wrapper run: make ci-pr-policy diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 10408ca3a..8c7692222 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -690,6 +690,43 @@ jobs: go-version-file: 'go.mod' cache: false + # The module fetch is the only part of this job that talks to the network, + # and setup-go above runs with cache: false, so every run pulls the whole + # module graph from the proxy. Doing it here, retried, means a proxy blip + # costs seconds instead of failing a required check. + # + # Deliberately NOT wrapped around the build: `make ci-pr-policy` runs a + # `go build ./cmd/bd` (scripts/ci/pr-policy.sh, build_docs_binary), and a + # compile error is not transient. Retrying it would make a genuinely broken + # build take three attempts to report and look like a slow one. The fetch + # has a legitimate transient failure mode; the build does not. + # + # 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s + # of headroom, matching the bounded apt-get steps in migration-test.yml. + - name: Download modules + timeout-minutes: 8 + run: | + downloaded="" + for attempt in 1 2 3; do + if timeout 120 go mod download; then + downloaded=yes + break + fi + echo "::warning::go mod download attempt $attempt failed or timed out" + # No backoff after the final attempt - there is nothing left to wait for. + if [ "$attempt" -lt 3 ]; then + sleep 10 + fi + done + # Unlike the best-effort apt-get retries elsewhere in this repo, this + # one is load-bearing: nothing downstream can build without the + # modules. Fail here, naming the fetch, so a module outage is not + # reported as a build failure. + if [ -z "$downloaded" ]; then + echo "::error::go mod download failed after 3 attempts - module fetch failure, not a build failure" + exit 1 + fi + - name: Run PR policy wrapper run: make ci-pr-policy