beads/tools/bazel/shard_budget.py
Julian Knutsen 873162dafa
ci(bazel): dolt-int gates server-storage + cmd-dolt in mode remote (item 5 A2) (#7535)
* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2)

In mode remote, folds bazel-server-storage (--config=doltserver-integration)
and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job
(--config=dolt-int), following the dolt-race pattern from #7514. The
members have different gating categories (server-storage plain-gated,
cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true
and a bespoke --label-scoped Shard balance step. All other modes keep the
two lanes running unchanged. Action keys are byte-identical between the
member configs and the union (17 + 46 = 63 TestRunner actions), so cached
results carry over either direction and a revert is a safe rollback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review)

Coordinator review of #7535 found three must-fixes: integration:"off" was
ignored in remote mode (the union's members already honor it); job-level
continue-on-error made a server-storage-only failure read success on
push/nightly/dispatch, where main stays red; and the PR's key-parity claim
overstated byte-identical actionKeys when only the remote action digest
(not the local actionKey) is unaffected by moving --test_env/--test_arg
onto the targets' own env/args.

Fixes:
- bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching
  the members (#7491's review); bazelLaneRunModes and
  TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case.
- Job-level continue-on-error removed; only the `bazel test` step carries
  it, and a new always() "Classify tier results" step
  (tools/bazel/classify_tier_results.py, with its own unit tests and a Go
  wrapper) computes server_storage_result/cmd_dolt_result from the BEP and
  decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on
  synthetic empty-output inputs that a dead job still fails the gate.
- Corrected the local-actionKey-vs-remote-digest wording in .bazelrc,
  bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md.
- Should-fix: test step 23->25 / job 58 minutes with reasoning; --label
  coverage added to shard_budget_test.py; stale lane comment fixed in
  tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's
  docstring corrected (members bake sole-run's line in directly, never via
  --config=sole-run).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review)

Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack,
julianknutsen):

- classify_tier_results.py now fails every tier shut (exit 1) on a
  non-PASSED label outside every --tier list, so a newly tagged target
  added without updating the step's --tier args cannot silently read as
  success (must-fix 1). Its --tier/--check label sets are policy-tied to
  the BUILD files' own tag sets by a new
  TestBazelDoltIntTierLabelsMatchTheTags.
- A missing or non-integer --rc (a timed-out or cancelled test step)
  takes the same fail-everything-shut branch instead of crashing argparse
  with no outputs written (bee-ghosttrack's should-fix 2).
- check_testcases.py gained a repeatable --label flag, and bazel.yml's
  shared "Every target and shard ran tests" step is split into check-ss
  and check-cd, each continue-on-error and folded into its own tier via
  classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard
  no longer reddens the job on push/nightly (must-fix 2).
- classify's run: argv is now pinned by exact equality
  (bazelDoltIntClassifyRun), and every flag except --tier/--check is
  --StoreOnce; a repeated --tier/--check name or an unrecognized flag is
  now a hard parser error (must-fix 3).
- "Upload test logs" now also fires on steps.test.outcome == 'failure',
  not just failure() (must-fix 4).
- shard_budget.py --label now fails closed (::error, exit 1) on a label
  with no executed shard in the BEP at all, instead of silently reporting
  "no sharded test target executed two or more shards" (should-fix 3,
  carried across both review rounds).
- Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and
  scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the
  nits: stale "Advisory (continue-on-error: true)" wording, the
  bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int,
  "byte-identical" without specifying remote action digest vs. local
  actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch
  for an empty union output.

Verified: python3 -I on all four tools/bazel/*_test.py (classify 29,
check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test
./scripts/... green; bazel --nohome_rc test //scripts:scripts_test
--config=ci green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): shard_budget --label fail-closed only on a truly unknown label

The previous commit's fail-closed --label check compared against
read_shard_durations, which only records executed (non-cached) shards, so
a target whose every shard was a cache hit this run had zero entries and
was wrongly treated the same as a --label typo. This was not caught by
the new tests (none exercised an all-cached target) and broke this PR's
own dolt-int CI run: bazel-dolt-int's Shard balance step went red on
//internal/storage/dolt:dolt_server_full_test with "no executed shard for
it", even though that target was correctly, fully cache-hit.

Fix: a new read_tested_labels() reads every label with any testResult
event (cached or executed), and --label's fail-closed check now tests
against that set instead of read_shard_durations. A fully-cached target
still legitimately reports "nothing to judge" through evaluate(); only a
label absent from the BEP altogether is now an error. Added the missing
test row (every shard cached) that would have caught this before push.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(bazel): pin bazel-dolt-int's Upload test logs if-condition

A full 42-mutation re-run for PR #7535's second review round found that
nothing pinned this job's own "Upload test logs" if-condition (must-fix 4):
the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's
(unlike dolt-race, a brand-new job) had no equivalent assertion, so a future
accidental revert to the old failure()-only condition would go uncaught.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-13 09:17:54 +00:00

232 lines
8.9 KiB
Python

#!/usr/bin/env python3
"""Warn (or fail) when a sharded test's slowest shard is far above its median.
Usage: shard_budget.py --bep <build_event_json_file> [--fail-on-imbalance]
Rationale (rbe-ci-cost-latency-study.md, recommendation 4): an unbalanced
shard split quietly grows the critical path of whichever lane runs it. gascity
#7233/#7241 showed the gain of re-splitting a hot shard; this script keeps
that gain from regressing and catches the tier this repository has today:
//internal/storage/dolt:dolt_server_full_test's last shard has trailed the
rest by 4.3-4.5 min in 2 of 8 runs (study Section 3.4).
Reads one `bazel test` invocation's --build_event_json_file (the same file
tools/bazel/check_testcases.py and check_shard_coverage.py already read) and,
for every sharded test target (shard_count > 1) that this invocation actually
executed at least two shards of, computes:
median = the median wall time of the shards that executed (not a cache
hit; a cached shard's duration says nothing about balance)
slowest = the maximum of those
A shard with no executed result (every attempt was a cache hit) is left out
of both: it ran nowhere this invocation. A target with fewer than two shards
left is skipped (not enough evidence this run).
Thresholds (study Section 6.4, picked from its shard data):
warn when slowest > max(150s, 2 x median)
fail when slowest > max(300s, 3 x median) -- only with --fail-on-imbalance
Day one: this prints `::warning` and always exits 0 (CI must not block on
it yet). Pass --fail-on-imbalance (a workflow step can gate this behind a
repository variable) to exit 1 and print `::error` instead, once the budget
has run clean for a while.
--label (repeatable) restricts which sharded targets this invocation judges,
by exact label. A union job's BEP has every member's shards in it; pass each
label whose owning lane's job this script runs in (item 5 A2:
bazel-dolt-int's step passes only server-storage's
//internal/storage/dolt:dolt_server_full_test, so it keeps judging that tier
alone and not cmd-dolt's bd_dolt_server_test / regression_test shards, which
no job budgeted before the union and which this script has never judged).
Omitted (bazel-server-storage's own step, unchanged): every sharded target in
the BEP, as before.
A --label naming a target with no testResult event in the BEP at all (cached
or executed) is a workflow authoring error (a typo, or a target renamed out
from under a pinned step), not a budget outcome: the step prints ::error and
exits 1 rather than silently reporting "no sharded test target executed two
or more shards" (PR #7535 review, should-fix 3, carried across both rounds).
A label that is in the BEP but had every shard served from cache this run --
so it has zero executed shards, same as a label with only one -- is not an
error: a fully cache-hit invocation is a legitimate outcome evaluate()
already reports as "nothing to judge" for that target, same as before.
"""
import argparse
import json
import statistics
import sys
WARN_FLOOR_S = 150.0
WARN_FACTOR = 2.0
FAIL_FLOOR_S = 300.0
FAIL_FACTOR = 3.0
def read_shard_durations(path):
"""Return {label: {shard_index: executed_duration_ms}} from a BEP file.
Only testResult events are read (as equivalence.py's read_bep does); a
shard's duration is the longest executed (non-cached) attempt reported
for it. shard defaults to 1 (an unsharded test), matching Bazel's BEP,
which omits the field for shard_count == 1.
"""
shards = {}
with open(path, encoding="utf-8") as f:
for line in f:
if not line.strip():
continue
try:
ev = json.loads(line)
except ValueError:
continue
tr_id = ev.get("id", {}).get("testResult")
tr = ev.get("testResult")
if not tr_id or not tr:
continue
cached = bool(tr.get("cachedLocally")) or bool(
(tr.get("executionInfo") or {}).get("cachedRemotely")
)
if cached:
continue
label = tr_id["label"]
shard = int(tr_id.get("shard") or 1)
ms = int(tr.get("testAttemptDurationMillis") or 0)
per_label = shards.setdefault(label, {})
per_label[shard] = max(per_label.get(shard, 0), ms)
return shards
def read_tested_labels(path):
"""Return the set of every label with at least one testResult event in
the BEP, executed or cached.
Separate from read_shard_durations, which only counts executed (non-
cached) attempts: a target whose every shard was a cache hit this
invocation has no entry in read_shard_durations at all, which is a
legitimate "nothing to judge" outcome (evaluate() already skips it),
not a --label typo. This set is what --label's fail-closed check
(main()) tests against instead, so a fully-cached run does not falsely
report an unknown label.
"""
labels = set()
with open(path, encoding="utf-8") as f:
for line in f:
if not line.strip():
continue
try:
ev = json.loads(line)
except ValueError:
continue
tr_id = ev.get("id", {}).get("testResult")
tr = ev.get("testResult")
if not tr_id or not tr:
continue
labels.add(tr_id["label"])
return labels
class Imbalance:
"""One sharded test target's balance verdict."""
def __init__(self, label, shards, median_ms, slowest_ms, slowest_shard):
self.label = label
self.shards = shards
self.median_ms = median_ms
self.slowest_ms = slowest_ms
self.slowest_shard = slowest_shard
@property
def warn_threshold_ms(self):
return max(WARN_FLOOR_S * 1000, WARN_FACTOR * self.median_ms)
@property
def fail_threshold_ms(self):
return max(FAIL_FLOOR_S * 1000, FAIL_FACTOR * self.median_ms)
@property
def warn(self):
return self.slowest_ms > self.warn_threshold_ms
@property
def fail(self):
return self.slowest_ms > self.fail_threshold_ms
def message(self):
return (
f"{self.label}: shard {self.slowest_shard} of {self.shards} took "
f"{self.slowest_ms / 1000:.0f}s against a median of {self.median_ms / 1000:.0f}s "
f"(warn over {self.warn_threshold_ms / 1000:.0f}s, fail over {self.fail_threshold_ms / 1000:.0f}s)"
)
def evaluate(shard_durations, labels=None):
"""Return a sorted list of Imbalance for every target with >= 2 executed
shards this invocation, whether or not it crosses the warn threshold.
labels, if given (a set), restricts evaluation to those exact labels: a
union job's BEP covers every member's shards, and a caller that only
owns one member's shard budget passes its labels so the other member's
shards are never judged here.
"""
out = []
for label, per_shard in sorted(shard_durations.items()):
if labels is not None and label not in labels:
continue
if len(per_shard) < 2:
continue
durations = list(per_shard.values())
median_ms = statistics.median(durations)
slowest_ms = max(durations)
slowest_shard = max(per_shard, key=per_shard.get)
out.append(Imbalance(label, len(per_shard), median_ms, slowest_ms, slowest_shard))
out.sort(key=lambda i: i.slowest_ms, reverse=True)
return out
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--bep", required=True, help="--build_event_json_file of the bazel test run")
ap.add_argument(
"--fail-on-imbalance",
action="store_true",
help="exit 1 (and print ::error) past the fail threshold; default is warn-only",
)
ap.add_argument(
"--label",
action="append",
default=None,
help="restrict to this exact target label (repeatable); default: every sharded target in the BEP",
)
args = ap.parse_args(argv)
shard_durations = read_shard_durations(args.bep)
failed = False
if args.label:
tested = read_tested_labels(args.bep)
for label in sorted(set(args.label) - tested):
failed = True
print(f"::error title=shard budget::{label}: --label, but the BEP has no result for it")
results = evaluate(shard_durations, labels=set(args.label) if args.label else None)
for r in results:
if r.fail and args.fail_on_imbalance:
failed = True
print(f"::error title=shard budget::{r.message()}")
elif r.warn:
print(f"::warning title=shard budget::{r.message()}")
else:
print(f"ok: {r.message()}")
if not results:
print("shard budget: no sharded test target executed two or more shards in this invocation")
return 1 if failed else 0
if __name__ == "__main__":
sys.exit(main())