* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2) In mode remote, folds bazel-server-storage (--config=doltserver-integration) and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job (--config=dolt-int), following the dolt-race pattern from #7514. The members have different gating categories (server-storage plain-gated, cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true and a bespoke --label-scoped Shard balance step. All other modes keep the two lanes running unchanged. Action keys are byte-identical between the member configs and the union (17 + 46 = 63 TestRunner actions), so cached results carry over either direction and a revert is a safe rollback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review) Coordinator review of #7535 found three must-fixes: integration:"off" was ignored in remote mode (the union's members already honor it); job-level continue-on-error made a server-storage-only failure read success on push/nightly/dispatch, where main stays red; and the PR's key-parity claim overstated byte-identical actionKeys when only the remote action digest (not the local actionKey) is unaffected by moving --test_env/--test_arg onto the targets' own env/args. Fixes: - bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching the members (#7491's review); bazelLaneRunModes and TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case. - Job-level continue-on-error removed; only the `bazel test` step carries it, and a new always() "Classify tier results" step (tools/bazel/classify_tier_results.py, with its own unit tests and a Go wrapper) computes server_storage_result/cmd_dolt_result from the BEP and decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on synthetic empty-output inputs that a dead job still fails the gate. - Corrected the local-actionKey-vs-remote-digest wording in .bazelrc, bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md. - Should-fix: test step 23->25 / job 58 minutes with reasoning; --label coverage added to shard_budget_test.py; stale lane comment fixed in tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's docstring corrected (members bake sole-run's line in directly, never via --config=sole-run). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review) Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack, julianknutsen): - classify_tier_results.py now fails every tier shut (exit 1) on a non-PASSED label outside every --tier list, so a newly tagged target added without updating the step's --tier args cannot silently read as success (must-fix 1). Its --tier/--check label sets are policy-tied to the BUILD files' own tag sets by a new TestBazelDoltIntTierLabelsMatchTheTags. - A missing or non-integer --rc (a timed-out or cancelled test step) takes the same fail-everything-shut branch instead of crashing argparse with no outputs written (bee-ghosttrack's should-fix 2). - check_testcases.py gained a repeatable --label flag, and bazel.yml's shared "Every target and shard ran tests" step is split into check-ss and check-cd, each continue-on-error and folded into its own tier via classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard no longer reddens the job on push/nightly (must-fix 2). - classify's run: argv is now pinned by exact equality (bazelDoltIntClassifyRun), and every flag except --tier/--check is --StoreOnce; a repeated --tier/--check name or an unrecognized flag is now a hard parser error (must-fix 3). - "Upload test logs" now also fires on steps.test.outcome == 'failure', not just failure() (must-fix 4). - shard_budget.py --label now fails closed (::error, exit 1) on a label with no executed shard in the BEP at all, instead of silently reporting "no sharded test target executed two or more shards" (should-fix 3, carried across both review rounds). - Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the nits: stale "Advisory (continue-on-error: true)" wording, the bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int, "byte-identical" without specifying remote action digest vs. local actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch for an empty union output. Verified: python3 -I on all four tools/bazel/*_test.py (classify 29, check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test ./scripts/... green; bazel --nohome_rc test //scripts:scripts_test --config=ci green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): shard_budget --label fail-closed only on a truly unknown label The previous commit's fail-closed --label check compared against read_shard_durations, which only records executed (non-cached) shards, so a target whose every shard was a cache hit this run had zero entries and was wrongly treated the same as a --label typo. This was not caught by the new tests (none exercised an all-cached target) and broke this PR's own dolt-int CI run: bazel-dolt-int's Shard balance step went red on //internal/storage/dolt:dolt_server_full_test with "no executed shard for it", even though that target was correctly, fully cache-hit. Fix: a new read_tested_labels() reads every label with any testResult event (cached or executed), and --label's fail-closed check now tests against that set instead of read_shard_durations. A fully-cached target still legitimately reports "nothing to judge" through evaluate(); only a label absent from the BEP altogether is now an error. Added the missing test row (every shard cached) that would have caught this before push. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * test(bazel): pin bazel-dolt-int's Upload test logs if-condition A full 42-mutation re-run for PR #7535's second review round found that nothing pinned this job's own "Upload test logs" if-condition (must-fix 4): the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's (unlike dolt-race, a brand-new job) had no equivalent assertion, so a future accidental revert to the old failure()-only condition would go uncaught. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
123 lines
5.4 KiB
Python
Executable file
123 lines
5.4 KiB
Python
Executable file
#!/usr/bin/env python3
|
|
"""Fail if a Bazel test (or any shard of it) ran no Go tests.
|
|
|
|
Usage: check_testcases.py --bep <build_event_json_file> [--testlogs DIR]
|
|
[--not-go LABEL ...] [--label LABEL ...]
|
|
|
|
A test target whose Go binary matches no test still exits 0: a shard script
|
|
whose manifest assigns a shard nothing ("no tests assigned"), a -test.run
|
|
selector that matches nothing, a manifest missing from runfiles. This reads
|
|
the tests a `bazel test` invocation ran from its BEP and requires every
|
|
target's test.xml, and every shard's, to list at least one top-level
|
|
<testcase> (a Test* name without "/"). It needs the test.xml files locally
|
|
(--remote_download_regex=.*/test\\.xml$) and -test.v in them
|
|
(GO_TEST_WRAP_TESTV=1), as --config=ci, --config=embedded and
|
|
--config=integration set.
|
|
|
|
--not-go LABEL exempts a test that is not a Go binary (a plain sh_test such
|
|
as //tools/bazel:dolt_version_test, whose test.xml Bazel writes with no Go
|
|
testcases) from the count. The label must still be in the BEP: an
|
|
exemption for a target the run did not test is an error, so a stale one
|
|
cannot linger. //pkg:all exempts every test in that package (one that holds
|
|
only non-Go tests, such as //scripts/repochecks), and must match at least one
|
|
tested target.
|
|
|
|
--label LABEL (repeatable) restricts which tested targets this invocation
|
|
judges, by exact label; a requested label absent from the BEP is itself a
|
|
failure (a typo or a stale label must not silently narrow the check to
|
|
nothing). item 5 A2: bazel-dolt-int's union BEP has both tiers' targets, and
|
|
its two "Every target and shard ran tests (<tier>)" steps each pass only
|
|
their own tier's labels, so a cmd-dolt zero-testcase shard reaches only
|
|
cmd_dolt_result (classify_tier_results.py's --check), not server-storage's,
|
|
and vice versa. Omitted (bazel-test and every other lane's own step,
|
|
unchanged): every tested target in the BEP, as before.
|
|
|
|
Exit status: 0 if every test.xml lists a test, 1 otherwise (including a BEP
|
|
with no test results at all, or a --label naming a target the BEP has no
|
|
result for).
|
|
"""
|
|
|
|
import argparse
|
|
import os
|
|
import sys
|
|
import xml.etree.ElementTree as ET
|
|
|
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|
from equivalence import read_bep, testlog_xmls # noqa: E402
|
|
|
|
|
|
def count_testcases(xml_path):
|
|
"""Return the number of top-level Go tests in a test.xml."""
|
|
n = 0
|
|
for tc in ET.parse(xml_path).getroot().iter("testcase"):
|
|
name = tc.get("name", "")
|
|
if name.startswith("Test") and "/" not in name:
|
|
n += 1
|
|
return n
|
|
|
|
|
|
def check(tested, testlogs, not_go=(), labels=None):
|
|
"""Return ([(xml path, count)], [problems]) for {label: shard_count}.
|
|
|
|
labels, if given (a set), restricts the check to those exact labels; a
|
|
label with no entry in tested is itself a problem (see the module
|
|
docstring's --label section).
|
|
"""
|
|
counts, problems = [], []
|
|
if labels is not None:
|
|
for label in sorted(set(labels) - set(tested)):
|
|
problems.append(f"{label}: --label, but the BEP has no result for it")
|
|
tested = {l: n for l, n in tested.items() if l in labels}
|
|
if not tested:
|
|
problems.append("the BEP lists no test results")
|
|
def exempt(label, entry):
|
|
if entry.endswith(":all"):
|
|
return label.split(":", 1)[0] == entry[: -len(":all")]
|
|
return label == entry
|
|
|
|
for entry in sorted(set(not_go)):
|
|
if not any(exempt(label, entry) for label in tested):
|
|
problems.append(f"{entry}: --not-go, but the BEP has no result for it")
|
|
for label, shards in sorted(tested.items()):
|
|
if any(exempt(label, entry) for entry in not_go):
|
|
continue
|
|
for xml_path in testlog_xmls(testlogs, label, shards):
|
|
rel = os.path.relpath(xml_path, testlogs)
|
|
if not os.path.exists(xml_path):
|
|
problems.append(f"{label}: {rel} missing (was test.xml downloaded?)")
|
|
continue
|
|
try:
|
|
n = count_testcases(xml_path)
|
|
except ET.ParseError as e:
|
|
problems.append(f"{label}: cannot parse {rel}: {e}")
|
|
continue
|
|
counts.append((rel, n))
|
|
if n == 0:
|
|
problems.append(f"{label}: {rel} lists no tests")
|
|
return counts, problems
|
|
|
|
|
|
def main(argv=None):
|
|
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
|
ap.add_argument("--bep", required=True, help="--build_event_json_file of the bazel test run")
|
|
ap.add_argument("--testlogs", default=None, help="default: from the BEP, else ./bazel-testlogs")
|
|
ap.add_argument("--not-go", action="append", default=[], metavar="LABEL",
|
|
help="a tested non-Go target to leave out of the count (repeatable)")
|
|
ap.add_argument("--label", action="append", default=None, metavar="LABEL",
|
|
help="restrict to this exact target label (repeatable); default: every tested target in the BEP")
|
|
args = ap.parse_args(argv)
|
|
|
|
tested, _, bep_testlogs = read_bep(args.bep)
|
|
testlogs = args.testlogs or bep_testlogs or "bazel-testlogs"
|
|
counts, problems = check(tested, testlogs, set(args.not_go), labels=set(args.label) if args.label else None)
|
|
for rel, n in counts:
|
|
print(f"{n:5d} {rel}")
|
|
total = sum(n for _, n in counts)
|
|
print(f"{len(tested)} targets, {len(counts)} test.xml, {total} top-level tests")
|
|
for p in problems:
|
|
print(f"FAIL: {p}", file=sys.stderr)
|
|
return 1 if problems else 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|