Commit graph

1 commit

Author SHA1 Message Date
Jim Wordelman
6ec78f3a2d
fix: bound and retry apt-get calls in CI to avoid unbounded 15-minute job timeout (#5810)
* test(feat): red — Fix: unbounded apt-get in CI burns the whole 15m job timeout (migration-test.yml, release.yml) (refs be-na2q)

* fix(feat): green — Fix: unbounded apt-get in CI burns the whole 15m job timeout (migration-test.yml, release.yml) (refs be-na2q)

* fix(ci): make apt-get update retry exhaustion deliberate, not a silent fall-through

After three failed `apt-get update` attempts the loop simply ended and the
step went on to `apt-get install` against a stale index, with nothing in the
log saying the index was never refreshed. That contradicted the PR body's
claim that the step "fails fast with a clear error".

Proceeding best-effort is the right behavior -- the runner image's cached
index installs these packages far more often than not, and failing here would
turn a transient mirror outage into a red build -- so make it an explicit,
logged decision instead of a fall-through:

  - track whether any attempt succeeded and emit
    `:⚠️:apt-get update failed after 3 attempts; proceeding with
    cached index` when none did
  - drop "; retrying" from the per-attempt warning, which was wrong on the
    final attempt
  - skip the 10s backoff after the last attempt, where there is nothing left
    to wait for

The workflow tests pin both new strings, so a revert to the silent
fall-through goes red.

* fix(ci): raise apt step timeout-minutes above the retry budget they backstop

The step-level caps were tighter than the inner `timeout` calls they were
meant to backstop, so they reproduced the mid-flight kill this hardening
exists to remove -- just at 6 and 8 minutes instead of 15:

  migration-test.yml  worst case 3x120s + 2x10s + 180s = 560s vs cap 360s
  release.yml         worst case 3x120s + 2x10s + 300s = 680s vs cap 480s

Raise them to 11m and 13m, each ~100s above its own worst case, and record
the arithmetic next to the value so the next edit re-derives it. 11m also
stays under the historical-upgrades job's own 15m budget, so a stuck step
fails as a step rather than cancelling the job with no apt-specific signal.
The goreleaser job sets no job-level timeout-minutes at all, which makes its
step cap the only bound on a stuck apt there.

The workflow tests now assert the relations instead of only pinning the
minute counts:

  - step timeout-minutes must exceed the step's own worst-case budget,
    computed from the attempt count, per-attempt timeout, backoff and
    install timeout that the same table pins against the YAML
  - it must stay below the job's timeout-minutes where the job sets one

Both read the value off the workflow rather than off the table's own
constant, so they stay honest if the YAML drifts.
2026-08-19 11:47:22 -07:00