* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2) In mode remote, folds bazel-server-storage (--config=doltserver-integration) and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job (--config=dolt-int), following the dolt-race pattern from #7514. The members have different gating categories (server-storage plain-gated, cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true and a bespoke --label-scoped Shard balance step. All other modes keep the two lanes running unchanged. Action keys are byte-identical between the member configs and the union (17 + 46 = 63 TestRunner actions), so cached results carry over either direction and a revert is a safe rollback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review) Coordinator review of #7535 found three must-fixes: integration:"off" was ignored in remote mode (the union's members already honor it); job-level continue-on-error made a server-storage-only failure read success on push/nightly/dispatch, where main stays red; and the PR's key-parity claim overstated byte-identical actionKeys when only the remote action digest (not the local actionKey) is unaffected by moving --test_env/--test_arg onto the targets' own env/args. Fixes: - bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching the members (#7491's review); bazelLaneRunModes and TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case. - Job-level continue-on-error removed; only the `bazel test` step carries it, and a new always() "Classify tier results" step (tools/bazel/classify_tier_results.py, with its own unit tests and a Go wrapper) computes server_storage_result/cmd_dolt_result from the BEP and decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on synthetic empty-output inputs that a dead job still fails the gate. - Corrected the local-actionKey-vs-remote-digest wording in .bazelrc, bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md. - Should-fix: test step 23->25 / job 58 minutes with reasoning; --label coverage added to shard_budget_test.py; stale lane comment fixed in tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's docstring corrected (members bake sole-run's line in directly, never via --config=sole-run). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review) Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack, julianknutsen): - classify_tier_results.py now fails every tier shut (exit 1) on a non-PASSED label outside every --tier list, so a newly tagged target added without updating the step's --tier args cannot silently read as success (must-fix 1). Its --tier/--check label sets are policy-tied to the BUILD files' own tag sets by a new TestBazelDoltIntTierLabelsMatchTheTags. - A missing or non-integer --rc (a timed-out or cancelled test step) takes the same fail-everything-shut branch instead of crashing argparse with no outputs written (bee-ghosttrack's should-fix 2). - check_testcases.py gained a repeatable --label flag, and bazel.yml's shared "Every target and shard ran tests" step is split into check-ss and check-cd, each continue-on-error and folded into its own tier via classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard no longer reddens the job on push/nightly (must-fix 2). - classify's run: argv is now pinned by exact equality (bazelDoltIntClassifyRun), and every flag except --tier/--check is --StoreOnce; a repeated --tier/--check name or an unrecognized flag is now a hard parser error (must-fix 3). - "Upload test logs" now also fires on steps.test.outcome == 'failure', not just failure() (must-fix 4). - shard_budget.py --label now fails closed (::error, exit 1) on a label with no executed shard in the BEP at all, instead of silently reporting "no sharded test target executed two or more shards" (should-fix 3, carried across both review rounds). - Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the nits: stale "Advisory (continue-on-error: true)" wording, the bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int, "byte-identical" without specifying remote action digest vs. local actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch for an empty union output. Verified: python3 -I on all four tools/bazel/*_test.py (classify 29, check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test ./scripts/... green; bazel --nohome_rc test //scripts:scripts_test --config=ci green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): shard_budget --label fail-closed only on a truly unknown label The previous commit's fail-closed --label check compared against read_shard_durations, which only records executed (non-cached) shards, so a target whose every shard was a cache hit this run had zero entries and was wrongly treated the same as a --label typo. This was not caught by the new tests (none exercised an all-cached target) and broke this PR's own dolt-int CI run: bazel-dolt-int's Shard balance step went red on //internal/storage/dolt:dolt_server_full_test with "no executed shard for it", even though that target was correctly, fully cache-hit. Fix: a new read_tested_labels() reads every label with any testResult event (cached or executed), and --label's fail-closed check now tests against that set instead of read_shard_durations. A fully-cached target still legitimately reports "nothing to judge" through evaluate(); only a label absent from the BEP altogether is now an error. Added the missing test row (every shard cached) that would have caught this before push. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * test(bazel): pin bazel-dolt-int's Upload test logs if-condition A full 42-mutation re-run for PR #7535's second review round found that nothing pinned this job's own "Upload test logs" if-condition (must-fix 4): the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's (unlike dolt-race, a brand-new job) had no equivalent assertion, so a future accidental revert to the old failure()-only condition would go uncaught. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
139 lines
4.6 KiB
Python
139 lines
4.6 KiB
Python
load("@rules_go//go:def.bzl", "go_test")
|
|
load("@rules_shell//shell:sh_test.bzl", "sh_test")
|
|
load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64")
|
|
|
|
# The differential regression suite (formerly regression.yml, which ran
|
|
# ./tests/regression with the regression build tag): every scenario runs against the
|
|
# catalog-pinned baseline release and against the candidate, and their
|
|
# normalized state is diffed. The baseline label must name the release in
|
|
# BASELINE_VERSION (TestMain checks the binary's own version against it). The
|
|
# candidate is the non-race bd_for_tests, built like the job's
|
|
# `go build -tags gms_pure_go ./cmd/bd`. Both bd's talk to a hermetic dolt
|
|
# sql-server TestMain starts from the pinned dolt (BEADS_TEST_DOLT_SERVER=local)
|
|
# where the job pulled the dolt-sql-server image; a server that cannot start
|
|
# fails the target (BEADS_TEST_REQUIRE_DOLT_CONTAINER=1).
|
|
#
|
|
# The go_test is only the binary (manual: never run on its own);
|
|
# :regression_test runs it in 14 shards. Shards 1-10 each run the scenarios
|
|
# pinned_shards.txt pins to them (duration-balanced by
|
|
# tools/bazel/pin_shards.py); shards 11-14 are the Go binary's own round-robin
|
|
# shards over every other top-level test, a test added since the manifest was
|
|
# written included (tools/bazel/go_test_pinned_shard.sh), so each runs in
|
|
# exactly one shard. Round-robin alone over 8 shards left the slowest at twice
|
|
# the median (TestListAllOverridesLimitWithLabel, TestDepTypeExportFidelity).
|
|
go_test(
|
|
name = "regression_suite",
|
|
srcs = [
|
|
"regression_test.go",
|
|
"scenarios_test.go",
|
|
],
|
|
tags = ["manual"],
|
|
target_compatible_with = LINUX_AMD64,
|
|
deps = [
|
|
"//internal/testutil",
|
|
"//internal/testutil/bazeltest",
|
|
],
|
|
)
|
|
|
|
# Tags:
|
|
# dolt-server-cmd: starts a hermetic dolt sql-server and needs the
|
|
# integration build tag; selected by --config=doltserver-cmd (bazel.yml's
|
|
# bazel-cmd-dolt lane) and, in mode remote, by --config=dolt-int
|
|
# (bazel-dolt-int, item 5 A2's union of that lane and bazel-server-storage)
|
|
# and excluded from every other lane, whose configurations either drop
|
|
# both test files or set BEADS_TEST_SKIP=dolt.
|
|
sh_test(
|
|
name = "regression_test",
|
|
size = "large",
|
|
# --config=doltserver-cmd's -test.timeout=19m fires (with a goroutine
|
|
# dump) before Bazel's 1200s action limit.
|
|
timeout = "eternal",
|
|
srcs = ["//tools/bazel:go_test_pinned_shard.sh"],
|
|
args = [
|
|
"$(rootpath :pinned_shards.txt)",
|
|
"$(rootpath :regression_suite)",
|
|
# Item 5 A2: moved from --config=doltserver-cmd's --test_arg (see
|
|
# cmd/bd:bd_dolt_server_test's comment); keeps --config=dolt-int's
|
|
# other member (server-storage, a 15m Go timeout) unaffected.
|
|
"-test.timeout=19m",
|
|
],
|
|
data = [
|
|
"BASELINE_VERSION",
|
|
"pinned_shards.txt",
|
|
":regression_suite",
|
|
"//cmd/bd:bd_for_tests",
|
|
"@bd_releases//:v0.49.6",
|
|
],
|
|
env = {
|
|
"BD_REGRESSION_BASELINE_BIN": "$(rlocationpath @bd_releases//:v0.49.6)",
|
|
"BEADS_TEST_BD_BINARY": "$(rlocationpath //cmd/bd:bd_for_tests)",
|
|
"BEADS_TEST_DOLT_SERVER": "local",
|
|
# Item 5 A2: moved from --config=doltserver-cmd's --test_env.
|
|
"BEADS_TEST_REQUIRE_EXCLUDE_PERMISSION": "1",
|
|
"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1",
|
|
},
|
|
shard_count = 14,
|
|
tags = ["dolt-server-cmd"],
|
|
target_compatible_with = LINUX_AMD64,
|
|
)
|
|
|
|
# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---
|
|
|
|
filegroup(
|
|
name = "repo_doc_files",
|
|
srcs = glob(
|
|
[
|
|
"**/*.md",
|
|
"**/*.mdx",
|
|
],
|
|
allow_empty = True,
|
|
exclude = ["**/node_modules/**"],
|
|
),
|
|
visibility = ["//:__subpackages__"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "repo_go_srcs",
|
|
srcs = glob(
|
|
["**/*.go"],
|
|
allow_empty = True,
|
|
exclude = [
|
|
"**/*_test.go",
|
|
"**/node_modules/**",
|
|
],
|
|
),
|
|
visibility = ["//:__subpackages__"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "repo_go_test_srcs",
|
|
srcs = glob(
|
|
["**/*_test.go"],
|
|
allow_empty = True,
|
|
exclude = ["**/node_modules/**"],
|
|
),
|
|
visibility = ["//:__subpackages__"],
|
|
)
|
|
|
|
filegroup(
|
|
name = "repo_other_files",
|
|
srcs = glob(
|
|
["**"],
|
|
exclude = [
|
|
"**/*.db",
|
|
"**/*.exe",
|
|
"**/*.go",
|
|
"**/*.md",
|
|
"**/*.mdx",
|
|
"**/*.out",
|
|
"**/*.prof",
|
|
"**/*.pyc",
|
|
"**/*.test",
|
|
"**/__pycache__/**",
|
|
"**/node_modules/**",
|
|
],
|
|
),
|
|
visibility = ["//:__subpackages__"],
|
|
)
|
|
|
|
# --- end repo_files ---
|