beads/tools/bazel/check_testcases_test.py
Julian Knutsen 873162dafa
ci(bazel): dolt-int gates server-storage + cmd-dolt in mode remote (item 5 A2) (#7535)
* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2)

In mode remote, folds bazel-server-storage (--config=doltserver-integration)
and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job
(--config=dolt-int), following the dolt-race pattern from #7514. The
members have different gating categories (server-storage plain-gated,
cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true
and a bespoke --label-scoped Shard balance step. All other modes keep the
two lanes running unchanged. Action keys are byte-identical between the
member configs and the union (17 + 46 = 63 TestRunner actions), so cached
results carry over either direction and a revert is a safe rollback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review)

Coordinator review of #7535 found three must-fixes: integration:"off" was
ignored in remote mode (the union's members already honor it); job-level
continue-on-error made a server-storage-only failure read success on
push/nightly/dispatch, where main stays red; and the PR's key-parity claim
overstated byte-identical actionKeys when only the remote action digest
(not the local actionKey) is unaffected by moving --test_env/--test_arg
onto the targets' own env/args.

Fixes:
- bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching
  the members (#7491's review); bazelLaneRunModes and
  TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case.
- Job-level continue-on-error removed; only the `bazel test` step carries
  it, and a new always() "Classify tier results" step
  (tools/bazel/classify_tier_results.py, with its own unit tests and a Go
  wrapper) computes server_storage_result/cmd_dolt_result from the BEP and
  decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on
  synthetic empty-output inputs that a dead job still fails the gate.
- Corrected the local-actionKey-vs-remote-digest wording in .bazelrc,
  bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md.
- Should-fix: test step 23->25 / job 58 minutes with reasoning; --label
  coverage added to shard_budget_test.py; stale lane comment fixed in
  tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's
  docstring corrected (members bake sole-run's line in directly, never via
  --config=sole-run).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review)

Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack,
julianknutsen):

- classify_tier_results.py now fails every tier shut (exit 1) on a
  non-PASSED label outside every --tier list, so a newly tagged target
  added without updating the step's --tier args cannot silently read as
  success (must-fix 1). Its --tier/--check label sets are policy-tied to
  the BUILD files' own tag sets by a new
  TestBazelDoltIntTierLabelsMatchTheTags.
- A missing or non-integer --rc (a timed-out or cancelled test step)
  takes the same fail-everything-shut branch instead of crashing argparse
  with no outputs written (bee-ghosttrack's should-fix 2).
- check_testcases.py gained a repeatable --label flag, and bazel.yml's
  shared "Every target and shard ran tests" step is split into check-ss
  and check-cd, each continue-on-error and folded into its own tier via
  classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard
  no longer reddens the job on push/nightly (must-fix 2).
- classify's run: argv is now pinned by exact equality
  (bazelDoltIntClassifyRun), and every flag except --tier/--check is
  --StoreOnce; a repeated --tier/--check name or an unrecognized flag is
  now a hard parser error (must-fix 3).
- "Upload test logs" now also fires on steps.test.outcome == 'failure',
  not just failure() (must-fix 4).
- shard_budget.py --label now fails closed (::error, exit 1) on a label
  with no executed shard in the BEP at all, instead of silently reporting
  "no sharded test target executed two or more shards" (should-fix 3,
  carried across both review rounds).
- Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and
  scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the
  nits: stale "Advisory (continue-on-error: true)" wording, the
  bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int,
  "byte-identical" without specifying remote action digest vs. local
  actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch
  for an empty union output.

Verified: python3 -I on all four tools/bazel/*_test.py (classify 29,
check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test
./scripts/... green; bazel --nohome_rc test //scripts:scripts_test
--config=ci green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): shard_budget --label fail-closed only on a truly unknown label

The previous commit's fail-closed --label check compared against
read_shard_durations, which only records executed (non-cached) shards, so
a target whose every shard was a cache hit this run had zero entries and
was wrongly treated the same as a --label typo. This was not caught by
the new tests (none exercised an all-cached target) and broke this PR's
own dolt-int CI run: bazel-dolt-int's Shard balance step went red on
//internal/storage/dolt:dolt_server_full_test with "no executed shard for
it", even though that target was correctly, fully cache-hit.

Fix: a new read_tested_labels() reads every label with any testResult
event (cached or executed), and --label's fail-closed check now tests
against that set instead of read_shard_durations. A fully-cached target
still legitimately reports "nothing to judge" through evaluate(); only a
label absent from the BEP altogether is now an error. Added the missing
test row (every shard cached) that would have caught this before push.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(bazel): pin bazel-dolt-int's Upload test logs if-condition

A full 42-mutation re-run for PR #7535's second review round found that
nothing pinned this job's own "Upload test logs" if-condition (must-fix 4):
the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's
(unlike dolt-race, a brand-new job) had no equivalent assertion, so a future
accidental revert to the old failure()-only condition would go uncaught.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-13 09:17:54 +00:00

106 lines
3.8 KiB
Python

#!/usr/bin/env python3
"""Unit tests for check_testcases.py, in particular its --label flag.
item 5 A2: bazel-dolt-int's union BEP has both tiers' targets, so its two
"Every target and shard ran tests (<tier>)" steps each need --label to scope
the check to their own tier's labels only (should-fix 2, PR #7535 review).
This covers check()'s --label behaviour directly; main()'s argument wiring is
exercised indirectly by bazel_dolt_int_test.go's step-content assertions and
by the manual BEP runs in that PR's review thread.
"""
import os
import sys
import tempfile
import unittest
import xml.etree.ElementTree as ET
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from check_testcases import check # noqa: E402
def write_testxml(path, testcase_names):
root = ET.Element("testsuite")
for name in testcase_names:
ET.SubElement(root, "testcase", name=name)
ET.ElementTree(root).write(path)
class CheckTestcasesTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.testlogs = self.tmp.name
def tearDown(self):
self.tmp.cleanup()
def write(self, label, names, shard=None):
rel_dir = label.lstrip("/").replace(":", "/")
d = os.path.join(self.testlogs, rel_dir)
if shard is not None:
d = os.path.join(d, "shard_%d_of_1" % shard)
os.makedirs(d, exist_ok=True)
write_testxml(os.path.join(d, "test.xml"), names)
def test_no_label_checks_every_tested_target(self):
self.write("//a:x", ["TestFoo"])
self.write("//b:y", ["TestBar"])
counts, problems = check({"//a:x": 1, "//b:y": 1}, self.testlogs)
self.assertEqual(problems, [])
self.assertEqual(len(counts), 2)
def test_label_restricts_to_named_targets(self):
self.write("//a:x", ["TestFoo"])
self.write("//b:y", []) # would fail if checked
counts, problems = check(
{"//a:x": 1, "//b:y": 1}, self.testlogs, labels={"//a:x"}
)
self.assertEqual(problems, [])
self.assertEqual(len(counts), 1)
def test_label_not_in_bep_is_a_failure(self):
self.write("//a:x", ["TestFoo"])
counts, problems = check(
{"//a:x": 1}, self.testlogs, labels={"//a:x", "//missing:z"}
)
self.assertTrue(
any("//missing:z" in p and "--label" in p for p in problems),
problems,
)
def test_label_does_not_mask_a_zero_testcase_shard_in_its_own_tier(self):
self.write("//a:x", [])
counts, problems = check({"//a:x": 1}, self.testlogs, labels={"//a:x"})
self.assertTrue(any("//a:x" in p and "lists no tests" in p for p in problems))
def test_label_scoping_is_independent_of_other_tiers_zero_testcase_shard(self):
# The exact bazel-dolt-int scenario: cmd-dolt's zero-testcase shard
# must not surface when --label scopes the check to server-storage.
self.write("//server:ss", ["TestServer"])
self.write("//cmd:cd", [])
counts, problems = check(
{"//server:ss": 1, "//cmd:cd": 1},
self.testlogs,
labels={"//server:ss"},
)
self.assertEqual(problems, [])
self.assertEqual(len(counts), 1)
def test_empty_label_set_with_no_tested_targets_is_a_failure(self):
counts, problems = check({}, self.testlogs, labels={"//a:x"})
self.assertTrue(any("//a:x" in p for p in problems))
def test_not_go_still_applies_within_a_labeled_subset(self):
self.write("//a:x", ["TestFoo"])
self.write("//a:shtest", []) # exempt via --not-go
counts, problems = check(
{"//a:x": 1, "//a:shtest": 1},
self.testlogs,
not_go=("//a:shtest",),
labels={"//a:x", "//a:shtest"},
)
self.assertEqual(problems, [])
if __name__ == "__main__":
unittest.main()