beads/scripts/BUILD.bazel
Julian Knutsen 873162dafa
ci(bazel): dolt-int gates server-storage + cmd-dolt in mode remote (item 5 A2) (#7535)
* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2)

In mode remote, folds bazel-server-storage (--config=doltserver-integration)
and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job
(--config=dolt-int), following the dolt-race pattern from #7514. The
members have different gating categories (server-storage plain-gated,
cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true
and a bespoke --label-scoped Shard balance step. All other modes keep the
two lanes running unchanged. Action keys are byte-identical between the
member configs and the union (17 + 46 = 63 TestRunner actions), so cached
results carry over either direction and a revert is a safe rollback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review)

Coordinator review of #7535 found three must-fixes: integration:"off" was
ignored in remote mode (the union's members already honor it); job-level
continue-on-error made a server-storage-only failure read success on
push/nightly/dispatch, where main stays red; and the PR's key-parity claim
overstated byte-identical actionKeys when only the remote action digest
(not the local actionKey) is unaffected by moving --test_env/--test_arg
onto the targets' own env/args.

Fixes:
- bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching
  the members (#7491's review); bazelLaneRunModes and
  TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case.
- Job-level continue-on-error removed; only the `bazel test` step carries
  it, and a new always() "Classify tier results" step
  (tools/bazel/classify_tier_results.py, with its own unit tests and a Go
  wrapper) computes server_storage_result/cmd_dolt_result from the BEP and
  decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on
  synthetic empty-output inputs that a dead job still fails the gate.
- Corrected the local-actionKey-vs-remote-digest wording in .bazelrc,
  bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md.
- Should-fix: test step 23->25 / job 58 minutes with reasoning; --label
  coverage added to shard_budget_test.py; stale lane comment fixed in
  tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's
  docstring corrected (members bake sole-run's line in directly, never via
  --config=sole-run).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review)

Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack,
julianknutsen):

- classify_tier_results.py now fails every tier shut (exit 1) on a
  non-PASSED label outside every --tier list, so a newly tagged target
  added without updating the step's --tier args cannot silently read as
  success (must-fix 1). Its --tier/--check label sets are policy-tied to
  the BUILD files' own tag sets by a new
  TestBazelDoltIntTierLabelsMatchTheTags.
- A missing or non-integer --rc (a timed-out or cancelled test step)
  takes the same fail-everything-shut branch instead of crashing argparse
  with no outputs written (bee-ghosttrack's should-fix 2).
- check_testcases.py gained a repeatable --label flag, and bazel.yml's
  shared "Every target and shard ran tests" step is split into check-ss
  and check-cd, each continue-on-error and folded into its own tier via
  classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard
  no longer reddens the job on push/nightly (must-fix 2).
- classify's run: argv is now pinned by exact equality
  (bazelDoltIntClassifyRun), and every flag except --tier/--check is
  --StoreOnce; a repeated --tier/--check name or an unrecognized flag is
  now a hard parser error (must-fix 3).
- "Upload test logs" now also fires on steps.test.outcome == 'failure',
  not just failure() (must-fix 4).
- shard_budget.py --label now fails closed (::error, exit 1) on a label
  with no executed shard in the BEP at all, instead of silently reporting
  "no sharded test target executed two or more shards" (should-fix 3,
  carried across both review rounds).
- Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and
  scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the
  nits: stale "Advisory (continue-on-error: true)" wording, the
  bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int,
  "byte-identical" without specifying remote action digest vs. local
  actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch
  for an empty union output.

Verified: python3 -I on all four tools/bazel/*_test.py (classify 29,
check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test
./scripts/... green; bazel --nohome_rc test //scripts:scripts_test
--config=ci green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): shard_budget --label fail-closed only on a truly unknown label

The previous commit's fail-closed --label check compared against
read_shard_durations, which only records executed (non-cached) shards, so
a target whose every shard was a cache hit this run had zero entries and
was wrongly treated the same as a --label typo. This was not caught by
the new tests (none exercised an all-cached target) and broke this PR's
own dolt-int CI run: bazel-dolt-int's Shard balance step went red on
//internal/storage/dolt:dolt_server_full_test with "no executed shard for
it", even though that target was correctly, fully cache-hit.

Fix: a new read_tested_labels() reads every label with any testResult
event (cached or executed), and --label's fail-closed check now tests
against that set instead of read_shard_durations. A fully-cached target
still legitimately reports "nothing to judge" through evaluate(); only a
label absent from the BEP altogether is now an error. Added the missing
test row (every shard cached) that would have caught this before push.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(bazel): pin bazel-dolt-int's Upload test logs if-condition

A full 42-mutation re-run for PR #7535's second review round found that
nothing pinned this job's own "Upload test logs" if-condition (must-fix 4):
the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's
(unlike dolt-race, a brand-new job) had no equivalent assertion, so a future
accidental revert to the old failure()-only condition would go uncaught.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-13 09:17:54 +00:00

443 lines
16 KiB
Python

load("@rules_go//go:def.bzl", "go_test")
# CI and repo-policy tests: they parse the workflows, run the CI shell scripts
# against fixtures, and check repository-wide rules (BUILD tag taxonomy, skip
# allowlist, managed BUILD blocks, shard manifests, tracked-file scans).
#
# They are one Go package but several go_test targets, each compiling only
# the files whose tests it runs and declaring only what those tests read, so
# that a change re-runs only the targets that read it: a Go-only change runs
# no workflow policy test, a docs-only or workflow-only change no Go-source
# scan. The data are the partitions of the checkout that tools/bazel/go_srcs.py
# generates (//:repo_go_srcs, //:repo_go_test_srcs, //:repo_doc_files,
# //:repo_other_files, and per package) or single files; only
# :tracked_files_test declares //:repo_files, every tracked file.
#
# Every *_test.go here is in exactly one of these targets, except the
# helpers-only files (no Test functions) that several compile; srcs are kept
# by hand (`# keep` stops gazelle from putting every file back into
# scripts_test), and TestScriptsTestTargetsPartitionTheTests
# (scripts_targets_test.go) fails on a file in no target or a test in two.
#
# No tags: they run remotely like any other target. The scripts and fixtures
# run bash, git, python3, jq, yq, curl, openssl, unzip and coreutils from the
# executor (the rbe-west worker image, or the runner in fork-cache mode); a
# missing tool fails the test under Bazel instead of skipping it
# (host_tools_test.go), so a cached pass never stands for an executor that
# could not run it. Go and the release formula come in hermetically.
# The CI and Bazel policy tests: workflows, the setup-bazel action, .bazelrc,
# MODULE.bazel, every BUILD and .bzl file, tools/bazel and the CI scripts,
# all in //:repo_other_files (every tracked file but Go and Markdown source),
# plus the one Go file whose dolt image pin they compare.
go_test(
name = "scripts_test",
srcs = [
"bazel_autofix_push_test.go",
"bazel_cache_mode_test.go",
"bazel_dolt_int_test.go",
"bazel_dolt_race_test.go",
"bazel_farm_workflow_test.go",
"bazel_fork_mode_test.go",
"bazel_go_repository_tools_test.go",
"bazel_hermetic_cc_test.go",
"bazel_invocations_test.go",
"bazel_policy_test.go",
"bazel_rbe_prewarm_test.go",
"bazel_rrc_test.go",
"ci_actions_policy_test.go",
"ci_analytics_workflow_test.go",
"ci_blacksmith_runner_test.go",
"ci_f7b_review_test.go",
"ci_f7c_advisory_test.go",
"ci_merge_queue_test.go",
"ci_workflow_apt_get_timeout_test.go",
"ci_workflow_core_test.go",
"ci_workflow_test.go",
"embedded_shard_count_test.go",
"host_tools_test.go",
"makefile_bazel_primary_test.go",
"migration_corpus_bazel_test.go",
"nogo_integration_step_script_test.go",
"nogo_lint_policy_test.go",
"pinned_shards_helpers_test.go",
"pinned_shards_test.go",
"pr_lanes_bazel_coverage_test.go",
"pr_risk_bazel_coverage_test.go",
"rbe_worker_env_test.go",
"repo_helpers_test.go",
"worker_env_autosync_test.go",
], # keep
data = [
"//:repo_other_files",
"//internal/testutil:testdoltcommon.go",
"@go_sdk//:bin/go",
"@go_sdk//:files",
],
# The Go SDK is go.mod's toolchain, whatever go the executor has on PATH.
# No //:repo_go_srcs here (F5 S2, cost fix, independent review of #7486):
# T4 (TestNogoIntegrationStepCoversItsConfiguration) and the
# unvalidated_integration_tests.txt exactness check moved to their own
# :nogo_integration_scan_test target below, which alone depends on
# //:repo_go_srcs and //:repo_go_test_srcs, so a Go-only change no longer
# re-runs the rest of this suite too.
env = {"BEADS_TEST_GO": "$(rlocationpath @go_sdk//:bin/go)"},
deps = [
"//internal/formula",
"//internal/testutil/bazeltest",
"//scripts/internal/testbash",
"@in_gopkg_yaml_v3//:yaml_v3",
],
)
# T4 (TestNogoIntegrationStepCoversItsConfiguration) and T6
# (TestUnvalidatedIntegrationTestsFileIsExact, independent review of #7486
# must-fix 6): both source-scan every tracked .go file (test and non-test)
# for the "integration" build tag, so they alone depend on //:repo_go_srcs
# and //:repo_go_test_srcs. Carved out of scripts_test (cost fix, same
# review): with this data in scripts_test, any Go-only change re-ran the
# whole nogo-policy suite, raising scripts_test's rerun rate from 73% to 91%
# of main commits. bazel_invocations_test.go and ci_workflow_core_test.go are
# helper-only files (no Test functions) shared with scripts_test, so neither
# copy duplicates the other's parsing logic.
go_test(
name = "nogo_integration_scan_test",
srcs = [
"bazel_invocations_test.go",
"ci_workflow_core_test.go",
"nogo_integration_scan_test.go",
"repo_helpers_test.go",
], # keep
data = [
"//:repo_go_srcs",
"//:repo_go_test_srcs",
"//:repo_other_files",
],
deps = [
"//internal/testutil/bazeltest",
"@in_gopkg_yaml_v3//:yaml_v3",
],
)
# The shell scripts' own behavior, run against fixtures: this package's
# scripts and their testdata, the few root files they read, and the release
# formula.
go_test(
name = "shell_scripts_test",
srcs = [
"check_build_tags_test.go",
"check_doc_freshness_test.go",
"check_release_cross_compile_test.go",
"check_testing_short_test.go",
"check_winget_portable_alias_test.go",
"ci_analytics_extract_test.go",
"complexity_script_test.go",
"conformance_script_test.go",
"host_tools_test.go",
"install_dolt_test.go",
"pr_preflight_test.go",
"pre_push_suite_test.go",
"pull_dolt_image_test.go",
"release_script_test.go",
"repo_helpers_test.go",
"required_suite_test.go",
"test_script_test.go",
"upgrade_smoke_loop_test.go",
],
data = [
":repo_other_files",
"//:.buildflags",
"//:.githooks/pre-push",
"//:.test-skip",
"//internal/httpclient:repo_other_files",
"//test/conformance:repo_go_test_srcs",
"//test/conformance:repo_other_files",
"//tools/bazel:repo_other_files",
"@beads_formulas//:beads-release.formula.toml",
],
# The release formula lives in .beads/ (in .bazelignore: it also holds
# live bd data), so it comes in through its own local repository.
env = {"BEADS_TEST_RELEASE_FORMULA": "$(rlocationpath @beads_formulas//:beads-release.formula.toml)"},
deps = [
"//internal/formula",
"//internal/testutil/bazeltest",
"//scripts/internal/testbash",
],
)
# Scans of the Go test files: the Bazel-skip allowlist, the managed local
# lane, and the shard manifests and scripts against the tests they shard.
# Every _test.go file (//:repo_go_test_srcs) but no non-test Go source, so a
# non-test Go change leaves it cached.
go_test(
name = "go_test_sources_test",
srcs = [
"bazel_skip_allowlist_test.go",
"embedded_shard_count_test.go",
"host_tools_test.go",
"managed_local_lane_test.go",
"pinned_shards_helpers_test.go",
"pinned_shards_sources_test.go",
"proxied_lane_coverage_test.go",
"repo_helpers_test.go",
"scripts_targets_test.go",
"shard_manifest_test.go",
],
data = [
"BUILD.bazel",
"ci/_embedded_shard_manifest_lib.py",
"ci/embedded_cmd_test_durations.json",
"ci/embedded_storage_test_durations.json",
"ci/gen_embedded_cmd_shard_manifest.py",
"ci/gen_embedded_storage_shard_manifest.py",
"ci/gen_proxied_shard_manifest.py",
"ci/proxied_test_durations.json",
"//:.github/scripts/embedded-cmd-test-shards.txt",
"//:.github/scripts/embedded-storage-test-shard.sh",
"//:.github/scripts/embedded-storage-test-shards.txt",
"//:.github/scripts/embedded-test-shard.sh",
"//:.github/scripts/proxied-cmd-test-shards.txt",
"//:.github/scripts/proxied-test-shard.sh",
"//:.github/scripts/server-storage-test-shard.sh",
"//:.github/scripts/server-storage-test-shards.txt",
"//:nested_go_modules",
"//:repo_go_test_srcs",
"//cmd/bd:repo_other_files",
"//internal/httpclient:repo_other_files",
"//internal/storage/dolt:repo_other_files",
"//internal/storage/embeddeddolt:repo_other_files",
"//tests/regression:repo_other_files",
"//tools/bazel:check_shard_coverage.py",
"//tools/bazel:equivalence_allowlist.txt",
],
deps = ["//internal/testutil/bazeltest"],
)
# check-testing-short.sh over every Go file, test and non-test, plus the
# Go-exec half of the host-go-under-Bazel policy (host_tool_exec_policy_test.go
# scans //:repo_go_srcs and //:repo_go_test_srcs for a literal go/gofmt exec).
# The shell-script half (host_tool_shell_policy_test.go) is its own
# :shell_host_tool_policy_test below: it reads a fixed, narrow list of shell
# scripts, not every Go source file, so it would otherwise pay this target's
# full re-run cost for files it never looks at.
go_test(
name = "go_sources_test",
srcs = [
"host_tool_exec_policy_test.go",
"repo_helpers_test.go",
"testing_short_tree_test.go",
],
data = [
"check-testing-short.sh",
"//:repo_go_srcs",
"//:repo_go_test_srcs",
],
deps = ["//internal/testutil/bazeltest"],
)
# The shell-script half of the host-go-under-Bazel policy
# (host_tool_shell_policy_test.go: see its package comment for shellPolicyScope
# and the detection tradeoffs). Split out of go_sources_test (which otherwise
# re-ran on every Go-source change to serve only this file) with data limited
# to exactly the shell scripts shellPolicyScope names -- no BUILD.bazel/.bzl
# walk, and so no //:repo_other_files: TestShellPolicyScopeCoversBazelShellData
# (dropped; see that test's old doc comment in version control) could not
# parse a `//pkg:file.sh` label or a glob() either, so paying 758 extra input
# files and ~4 points of cache-rerun rate across the whole repo for a walk
# that demonstrably still missed real in-scope scripts was not worth it.
go_test(
name = "shell_host_tool_policy_test",
srcs = [
"host_tool_shell_policy_test.go",
"repo_helpers_test.go",
],
data = [
"check-build-tags.sh",
"check-doc-freshness.sh",
"check-go-install-guidance.sh",
"check-migration-hygiene.sh",
"check-testing-short.sh",
"check-versions.sh",
"check-winget-portable-alias.sh",
"check-workapi-frontend-boundary.sh",
"ci/fmt-check.sh",
"ci/gofmt-bin.sh",
"ci/pr-lint.sh",
"conformance.sh",
"migrate-legacy-to-current.sh",
"migration-test/historical-dolt-upgrade-test.sh",
"migration-test/legacy-bridge-test.sh",
"migration-test/lib/binary.sh",
"release.sh",
"test.sh",
"upgrade-smoke-test.sh",
"//:.github/scripts/embedded-storage-test-shard.sh",
"//:.github/scripts/embedded-test-shard.sh",
"//:.github/scripts/proxied-test-shard.sh",
"//:.github/scripts/server-storage-test-shard.sh",
"//scripts/repochecks:run_check.sh",
"//scripts/repochecks:types_gen_drift_test.sh",
],
deps = ["//internal/testutil/bazeltest"],
)
# Scans of every tracked file (//:repo_files): any change re-runs them, so
# keep them few and fast.
go_test(
name = "tracked_files_test",
srcs = [
"repo_helpers_test.go",
"tracked_files_test.go",
],
data = ["//:repo_files"],
deps = ["//internal/testutil/bazeltest"],
)
# pr.yml's check-doc-freshness-platforms Linux leg: `go test
# -tags integration ./scripts -count=1 -run '^(TestDocFreshness.*|
# TestRequiredSuiteContract)$' -args -required-suite=doc-freshness`, whose
# TestMain fails unless exactly the doc-freshness inventory is compiled and
# selected. shell_scripts_test runs those tests too (in the integration lane);
# this target adds the inventory and selection contract, over only the files
# the suite reads (its fixtures copy check-doc-freshness.sh). The macOS and
# Windows legs stay in pr.yml.
# Tags:
# integration-only: every TestDocFreshness test is //go:build integration,
# so --config=integration's lane runs it.
go_test(
name = "doc_freshness_required_test",
srcs = [
"check_doc_freshness_test.go",
"repo_helpers_test.go",
"required_suite_test.go",
],
args = [
"-test.count=1",
"-test.run=^(TestDocFreshness.*|TestRequiredSuiteContract)$$",
"-required-suite=doc-freshness",
],
data = ["check-doc-freshness.sh"],
tags = ["integration-only"],
deps = [
"//internal/testutil/bazeltest",
"//scripts/internal/testbash",
],
)
exports_files(
["migration-test/release-catalog.json"],
visibility = ["//scripts/migration-test/catalog:__pkg__"],
)
# The historical-upgrade harness (scripts/migration-test) and the public
# SQLite bridge it drives, for //tests/migration's per-release targets.
filegroup(
name = "migration_harness",
srcs = [
"migrate-legacy-to-current.sh",
"migration-test/historical-dolt-upgrade-test.sh",
"migration-test/legacy-bridge-test.sh",
"migration-test/lib/binary.sh",
"migration-test/lib/versions.sh",
],
visibility = ["//tests/migration:__pkg__"],
)
exports_files(
["upgrade-smoke-test.sh"],
visibility = ["//tests/upgrade_smoke:__pkg__"],
)
exports_files(
[
"ci/fmt-check.sh",
"ci/lib/timing.sh",
"ci/pr-lint.sh",
],
visibility = ["//scripts/prlintmake:__pkg__"],
)
exports_files(
["ci/gofmt-bin.sh"],
visibility = ["//scripts/prlintmake:__pkg__"],
)
# The compatibility entrypoint whose exit-status contract
# //scripts/check-versions:check-versions_test drives.
exports_files(
["check-versions.sh"],
visibility = ["//scripts/check-versions:__pkg__"],
)
# Scripts the docs and agent instructions link to by path, which
# //test/docsync checks resolve.
exports_files(
[
"bump-version.sh",
"check-cli-docs-drift.sh",
"generate-cli-docs.sh",
"test.sh",
],
visibility = ["//test/docsync:__pkg__"],
)
# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---
filegroup(
name = "repo_doc_files",
srcs = glob(
[
"**/*.md",
"**/*.mdx",
],
allow_empty = True,
exclude = ["**/node_modules/**"],
),
visibility = ["//:__subpackages__"],
)
filegroup(
name = "repo_go_srcs",
srcs = glob(
["**/*.go"],
allow_empty = True,
exclude = [
"**/*_test.go",
"**/node_modules/**",
],
),
visibility = ["//:__subpackages__"],
)
filegroup(
name = "repo_go_test_srcs",
srcs = glob(
["**/*_test.go"],
allow_empty = True,
exclude = ["**/node_modules/**"],
),
visibility = ["//:__subpackages__"],
)
filegroup(
name = "repo_other_files",
srcs = glob(
["**"],
exclude = [
"**/*.db",
"**/*.exe",
"**/*.go",
"**/*.md",
"**/*.mdx",
"**/*.out",
"**/*.prof",
"**/*.pyc",
"**/*.test",
"**/__pycache__/**",
"**/node_modules/**",
],
),
visibility = ["//:__subpackages__"],
)
# --- end repo_files ---