* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2) In mode remote, folds bazel-server-storage (--config=doltserver-integration) and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job (--config=dolt-int), following the dolt-race pattern from #7514. The members have different gating categories (server-storage plain-gated, cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true and a bespoke --label-scoped Shard balance step. All other modes keep the two lanes running unchanged. Action keys are byte-identical between the member configs and the union (17 + 46 = 63 TestRunner actions), so cached results carry over either direction and a revert is a safe rollback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review) Coordinator review of #7535 found three must-fixes: integration:"off" was ignored in remote mode (the union's members already honor it); job-level continue-on-error made a server-storage-only failure read success on push/nightly/dispatch, where main stays red; and the PR's key-parity claim overstated byte-identical actionKeys when only the remote action digest (not the local actionKey) is unaffected by moving --test_env/--test_arg onto the targets' own env/args. Fixes: - bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching the members (#7491's review); bazelLaneRunModes and TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case. - Job-level continue-on-error removed; only the `bazel test` step carries it, and a new always() "Classify tier results" step (tools/bazel/classify_tier_results.py, with its own unit tests and a Go wrapper) computes server_storage_result/cmd_dolt_result from the BEP and decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on synthetic empty-output inputs that a dead job still fails the gate. - Corrected the local-actionKey-vs-remote-digest wording in .bazelrc, bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md. - Should-fix: test step 23->25 / job 58 minutes with reasoning; --label coverage added to shard_budget_test.py; stale lane comment fixed in tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's docstring corrected (members bake sole-run's line in directly, never via --config=sole-run). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review) Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack, julianknutsen): - classify_tier_results.py now fails every tier shut (exit 1) on a non-PASSED label outside every --tier list, so a newly tagged target added without updating the step's --tier args cannot silently read as success (must-fix 1). Its --tier/--check label sets are policy-tied to the BUILD files' own tag sets by a new TestBazelDoltIntTierLabelsMatchTheTags. - A missing or non-integer --rc (a timed-out or cancelled test step) takes the same fail-everything-shut branch instead of crashing argparse with no outputs written (bee-ghosttrack's should-fix 2). - check_testcases.py gained a repeatable --label flag, and bazel.yml's shared "Every target and shard ran tests" step is split into check-ss and check-cd, each continue-on-error and folded into its own tier via classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard no longer reddens the job on push/nightly (must-fix 2). - classify's run: argv is now pinned by exact equality (bazelDoltIntClassifyRun), and every flag except --tier/--check is --StoreOnce; a repeated --tier/--check name or an unrecognized flag is now a hard parser error (must-fix 3). - "Upload test logs" now also fires on steps.test.outcome == 'failure', not just failure() (must-fix 4). - shard_budget.py --label now fails closed (::error, exit 1) on a label with no executed shard in the BEP at all, instead of silently reporting "no sharded test target executed two or more shards" (should-fix 3, carried across both review rounds). - Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the nits: stale "Advisory (continue-on-error: true)" wording, the bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int, "byte-identical" without specifying remote action digest vs. local actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch for an empty union output. Verified: python3 -I on all four tools/bazel/*_test.py (classify 29, check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test ./scripts/... green; bazel --nohome_rc test //scripts:scripts_test --config=ci green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): shard_budget --label fail-closed only on a truly unknown label The previous commit's fail-closed --label check compared against read_shard_durations, which only records executed (non-cached) shards, so a target whose every shard was a cache hit this run had zero entries and was wrongly treated the same as a --label typo. This was not caught by the new tests (none exercised an all-cached target) and broke this PR's own dolt-int CI run: bazel-dolt-int's Shard balance step went red on //internal/storage/dolt:dolt_server_full_test with "no executed shard for it", even though that target was correctly, fully cache-hit. Fix: a new read_tested_labels() reads every label with any testResult event (cached or executed), and --label's fail-closed check now tests against that set instead of read_shard_durations. A fully-cached target still legitimately reports "nothing to judge" through evaluate(); only a label absent from the BEP altogether is now an error. Added the missing test row (every shard cached) that would have caught this before push. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * test(bazel): pin bazel-dolt-int's Upload test logs if-condition A full 42-mutation re-run for PR #7535's second review round found that nothing pinned this job's own "Upload test logs" if-condition (must-fix 4): the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's (unlike dolt-race, a brand-new job) had no equivalent assertion, so a future accidental revert to the old failure()-only condition would go uncaught. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
230 lines
10 KiB
Python
230 lines
10 KiB
Python
#!/usr/bin/env python3
|
|
"""Classify a union `bazel test` run's result by tier, from its BEP.
|
|
|
|
Usage: classify_tier_results.py --bep <build_event_json_file> --rc <bazel exit code>
|
|
--tier NAME=LABEL[,LABEL...] [--tier NAME=LABEL...]
|
|
--gate-tier NAME --github-output <path>
|
|
[--check NAME=success|failure|cancelled ...]
|
|
|
|
item 5 A2's bazel-dolt-int job runs two tiers (server-storage, cmd-dolt) in
|
|
one `bazel test --config=dolt-int` invocation. The job carries no job-level
|
|
continue-on-error: on push, nightly and dispatch, nothing reads
|
|
BAZEL_SERVER_STORAGE off a `workflow_call` output, so a job-level
|
|
continue-on-error (reading job.status as success regardless of the `bazel
|
|
test` step's own outcome) would let a server-storage failure end the run
|
|
green there, where it must stay red exactly as it would on main today. The
|
|
`bazel test` step instead carries continue-on-error, and this script -- an
|
|
always() step of its own -- computes each tier's result independently from
|
|
the BEP and decides whether the job itself must fail.
|
|
|
|
Bazel's own exit code governs first. 0 (ran, all passed) and 3 (ran, some
|
|
test failed) are the only codes this script trusts the BEP for: a build
|
|
failure, an infra error or a crash (any other code) cannot be safely
|
|
attributed to one tier's tests, so both tiers are marked "failure" and the
|
|
script always exits 1 for it (no silent cmd-dolt-only downgrade of a
|
|
not-actually-a-test-failure). A missing or non-integer --rc (bee-ghosttrack's
|
|
review, PR #7535 should-fix 2: the `bazel test` step's 25-minute timeout or a
|
|
cancellation before it can `echo "rc=$rc"` leaves bazel.yml's $TEST_RC empty)
|
|
takes this same branch rather than crashing argparse with no outputs
|
|
written: it is treated exactly like an exit code outside {0, 3}.
|
|
|
|
Otherwise each --tier is "failure" iff a BEP testResult for one of its
|
|
labels has a non-PASSED status (FAILED, TIMEOUT, REMOTE_FAILURE, INCOMPLETE,
|
|
FLAKY all count: these lanes run flaky=False, so a flaky-looking status is
|
|
still a failure), or if the BEP has no result at all for one of its labels
|
|
(an attempt that never ran cannot be presumed to have passed; a genuine
|
|
coverage gap already fails the job through check_testcases.py's own step).
|
|
|
|
Fail-closed on an unknown label (coordinator review, PR #7535 must-fix 1):
|
|
if the BEP has a non-PASSED result for any label that is not one of the
|
|
--tier lists (a newly `dolt-server-integration`/`dolt-server-cmd`-tagged
|
|
target added without updating the step's --tier args, i.e. bazel.yml and
|
|
this script's lists drifting from the BUILD files' tag sets -- see
|
|
TestBazelDoltIntTierLabelsMatchTheTags), every tier is marked "failure" and
|
|
the script exits 1: an un-enumerated failing target must not read as success
|
|
just because no --tier list claims it.
|
|
|
|
--check NAME=OUTCOME (repeatable) folds a per-tier shell check's own GitHub
|
|
Actions step outcome into that tier's result: "success" leaves the BEP-only
|
|
result alone, anything else (bazel.yml passes the step's own
|
|
`steps.<id>.outcome`: "failure" or "cancelled") forces that tier to
|
|
"failure" regardless of the BEP. This is how bazel.yml's two
|
|
"Every target and shard ran tests (<tier>)" steps (the shared
|
|
check_testcases.py split per tier, PR #7535 should-fix 2) report a zero-
|
|
testcase shard without going through job.status: cmd-dolt's check runs with
|
|
its own continue-on-error so it cannot fail the job directly, but its result
|
|
must still reach cmd_dolt_result.
|
|
|
|
Exit status is 1 iff --gate-tier's tier is "failure" (or the exit-code or
|
|
unknown-label branches above applied); a failure confined to another tier
|
|
leaves this script's own exit 0, so the job step -- and so job.status --
|
|
stays green, mirroring that tier's own advisory, continue-on-error nature
|
|
outside mode remote. Every tier's result, win or lose, is appended to
|
|
--github-output as "<name>_result=success|failure" (NAME lowercased, "-"
|
|
turned into "_"), so the job can expose one output per tier.
|
|
|
|
Every flag may be given at most once, except --tier and --check (one per
|
|
distinct NAME: a repeated NAME, or an unrecognized flag, is a hard error --
|
|
coordinator review, PR #7535 must-fix 3 -- so a future edit cannot silently
|
|
widen or narrow the gate by appending an extra --tier/--gate-tier/--check
|
|
that argparse would otherwise just let the last value win).
|
|
"""
|
|
|
|
import argparse
|
|
import json
|
|
import sys
|
|
|
|
PASS_STATUS = "PASSED"
|
|
|
|
|
|
class _StoreOnce(argparse.Action):
|
|
"""Like the default "store" action, but errors if given more than once."""
|
|
|
|
def __call__(self, parser, namespace, values, option_string=None):
|
|
if getattr(namespace, self.dest, None) is not None:
|
|
parser.error(f"{option_string} may only be given once")
|
|
setattr(namespace, self.dest, values)
|
|
|
|
|
|
def read_test_statuses(bep_path):
|
|
"""Return {label: set(status seen across every testResult event)}."""
|
|
statuses = {}
|
|
with open(bep_path, encoding="utf-8") as f:
|
|
for line in f:
|
|
if not line.strip():
|
|
continue
|
|
try:
|
|
ev = json.loads(line)
|
|
except ValueError:
|
|
continue
|
|
tr_id = ev.get("id", {}).get("testResult")
|
|
tr = ev.get("testResult")
|
|
if not tr_id or not tr:
|
|
continue
|
|
label = tr_id["label"]
|
|
statuses.setdefault(label, set()).add(tr.get("status", "NO_STATUS"))
|
|
return statuses
|
|
|
|
|
|
def tier_result(statuses, labels):
|
|
"""Return "success" or "failure" for one tier's labels."""
|
|
for label in labels:
|
|
seen = statuses.get(label)
|
|
if not seen or seen - {PASS_STATUS}:
|
|
return "failure"
|
|
return "success"
|
|
|
|
|
|
def output_key(name):
|
|
return name.lower().replace("-", "_") + "_result"
|
|
|
|
|
|
def classify(bep_path, rc, tiers, gate_tier, checks=None):
|
|
"""Return ({tier: "success"|"failure"}, exit_status, reason|None).
|
|
|
|
checks, if given, is {tier name: GitHub Actions step outcome}: any value
|
|
other than "success" forces that tier to "failure" (see the module
|
|
docstring's --check section). reason is None iff exit_status is 0.
|
|
"""
|
|
checks = checks or {}
|
|
try:
|
|
rc = int(rc)
|
|
except (TypeError, ValueError):
|
|
reason = (
|
|
f"bazel exit code {rc!r} is missing or not an integer: the test "
|
|
"step likely timed out or was cancelled before it could record one"
|
|
)
|
|
return {name: "failure" for name in tiers}, 1, reason
|
|
if rc not in (0, 3):
|
|
reason = f"bazel exit {rc} (not 0 or 3): a build or infra failure, not attributable to one tier"
|
|
return {name: "failure" for name in tiers}, 1, reason
|
|
|
|
statuses = read_test_statuses(bep_path)
|
|
|
|
known_labels = set()
|
|
for labels in tiers.values():
|
|
known_labels.update(labels)
|
|
unknown_failed = sorted(
|
|
label for label, seen in statuses.items()
|
|
if label not in known_labels and (seen - {PASS_STATUS})
|
|
)
|
|
if unknown_failed:
|
|
# A real test failure Bazel reported (rc 3) that no --tier list
|
|
# claims: the tier/tag lists have drifted from what actually ran.
|
|
# Fail everything shut rather than silently reading it as success.
|
|
reason = f"non-PASSED result for a label outside every --tier: {', '.join(unknown_failed)}"
|
|
return {name: "failure" for name in tiers}, 1, reason
|
|
|
|
results = {}
|
|
for name, labels in tiers.items():
|
|
result = tier_result(statuses, labels)
|
|
if checks.get(name, "success") != "success":
|
|
result = "failure"
|
|
results[name] = result
|
|
if results.get(gate_tier) == "failure":
|
|
return results, 1, f"{gate_tier} tier failed"
|
|
return results, 0, None
|
|
|
|
|
|
def parse_tier_arg(spec):
|
|
name, _, labels = spec.partition("=")
|
|
if not name or not labels:
|
|
raise argparse.ArgumentTypeError(f"--tier {spec!r}: want NAME=LABEL[,LABEL...]")
|
|
return name, [l for l in labels.split(",") if l]
|
|
|
|
|
|
def parse_check_arg(spec):
|
|
name, sep, outcome = spec.partition("=")
|
|
if not name or not sep or not outcome:
|
|
raise argparse.ArgumentTypeError(f"--check {spec!r}: want NAME=OUTCOME")
|
|
return name, outcome
|
|
|
|
|
|
def _dict_no_duplicates(parser, pairs, flag):
|
|
out = {}
|
|
for name, value in pairs:
|
|
if name in out:
|
|
parser.error(f"{flag} {name!r} given more than once")
|
|
out[name] = value
|
|
return out
|
|
|
|
|
|
def main(argv=None):
|
|
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
p.add_argument("--bep", required=True, action=_StoreOnce)
|
|
# Not type=int: an empty or non-integer --rc (the test step's own
|
|
# $TEST_RC left unset by a timeout or cancellation) must reach
|
|
# classify()'s own handling below, not crash argparse with no outputs
|
|
# written (should-fix 2).
|
|
p.add_argument("--rc", required=True, action=_StoreOnce)
|
|
p.add_argument("--tier", action="append", type=parse_tier_arg, default=[], dest="tiers", metavar="NAME=LABEL[,LABEL...]")
|
|
p.add_argument("--gate-tier", required=True, action=_StoreOnce)
|
|
p.add_argument("--github-output", required=True, action=_StoreOnce)
|
|
p.add_argument("--check", action="append", type=parse_check_arg, default=[], dest="checks", metavar="NAME=OUTCOME")
|
|
args = p.parse_args(argv)
|
|
|
|
tiers = _dict_no_duplicates(p, args.tiers, "--tier")
|
|
if not tiers:
|
|
p.error("at least one --tier is required")
|
|
if args.gate_tier not in tiers:
|
|
p.error(f"--gate-tier {args.gate_tier!r} is not one of --tier's names {sorted(tiers)}")
|
|
checks = _dict_no_duplicates(p, args.checks, "--check")
|
|
for name in checks:
|
|
if name not in tiers:
|
|
p.error(f"--check {name!r} is not one of --tier's names {sorted(tiers)}")
|
|
|
|
results, status, reason = classify(args.bep, args.rc, tiers, args.gate_tier, checks)
|
|
|
|
with open(args.github_output, "a", encoding="utf-8") as f:
|
|
for name, result in results.items():
|
|
line = f"{output_key(name)}={result}"
|
|
print(line)
|
|
f.write(line + "\n")
|
|
|
|
if status:
|
|
print(f"::error::classify_tier_results: {reason}", file=sys.stderr)
|
|
return status
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|