fix(bazel): remove host-go dep from compact GC test; drop frozen shard blocks (#7478)
* fix(bazel): remove host-go dep from compact GC test; drop frozen shard blocks Replace the ungated `go build` in compact_gc_target_test.go's dolt-stand-in fixture with a Bazel-built go_binary (cmd/bd:compact_gc_fixture) wired via rlocationpath env + data, mirroring the existing preflight_go_fixture pattern. Falls back to `go build` only when not running under Bazel (bazeltest.IsBazel()). Also gate test_repo_beads_guard_test.go's `go env GOCACHE`/`GOMODCACHE` probes on !bazeltest.IsBazel(), since they previously ran unconditionally in every cmd/bd TestMain. Separately, delete the legacy 20-shard embedded-cmd and 15-shard proxied-cmd "frozen" manifest blocks in .github/scripts/*-test-shards.txt: their only consumers were pr-risk.yml fork/push jobs that have since migrated to bazel.yml. Update gen_embedded_cmd_shard_manifest.py and gen_proxied_shard_manifest.py to drop the frozen-block rendering/guard logic and default to the live Bazel-only shard counts (100 and 34), and refresh stale comments/docs (cmd/bd/BUILD.bazel, scripts/shard_manifest_test.go, engdocs/TESTING.md, engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md) that still described the deleted blocks. Verified no other BUILD/.github/scripts references read them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * build(bazel): auto-sync BUILD files Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/38069936930 (`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy). Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see scripts/bazel-autofix-push.sh. * fix(bazel): gate remaining host-go calls; fix shard-manifest --weights footgun Address PR #7478 review findings (bee-ghosttrack): - scripts/ci/lib/test-env.sh: skip the GOCACHE/GOMODCACHE `go env` probes when TEST_SRCDIR is set, so scripts/test.sh's beads_test_env_enter never shells out to a bare `go` when reached from a Bazel-run go_test (e.g. //scripts:shell_scripts_test's TestTestScriptPrebuiltTestBinaryContract), where a farm masks host go/dolt to exit 127. Nothing on that path needs a cache directory. - scripts/ci/fmt-check.sh: describe_gofmt() no longer calls a PATH `go` under Bazel. It now resolves BEADS_TEST_GO (the hermetic SDK go already wired into scripts/prlintmake:prlintmake_test's env) via TEST_SRCDIR, and skips the version-reporting arm entirely when that isn't set (e.g. scripts/repochecks:fmt_test), rather than ever touching a possibly-masked PATH `go`. - Verified with a PATH stub (go/dolt scripts that log and exit 127) via `bazel --nohome_rc test --config=prcore //scripts/... //cmd/bd:bd_test --test_env=PATH=<stubdir>:/usr/bin:/bin:/usr/local/bin`: 26/26 targets pass, zero stub invocations logged. - gen_proxied_shard_manifest.py / gen_embedded_cmd_shard_manifest.py: default --weights to duration now that the manifests' only remaining blocks are duration-packed (the legacy inits-packed blocks were deleted alongside the PR Risk fork/push jobs that read them). Previously the bare `--write` default (inits) silently clobbered the live block's duration header and repacked with the wrong cost model; --check couldn't see it. Added TestProxiedShardManifestBareWriteIsIdempotent and TestEmbeddedCmdShardManifestBareWriteIsIdempotent to pin a bare --write as a no-op against the committed files. - cmd/bd/compact_gc_target_test.go: resolve the non-Bazel fallback fixture path from this file's own directory (runtime.Caller) instead of the test binary's working directory. - Refresh stale comments in .github/workflows/bazel.yml, scripts/shard_manifest_test.go, scripts/pr_risk_bazel_coverage_test.go, and scripts/embedded_shard_count_test.go that still described the deleted legacy 20/15-shard blocks and their retired fork/push jobs in the present tense. - Confirmed cmd/bd/BUILD.bazel's compact_gc_fixture go_binary (implicit pure/race defaults) still passes TestRunCompactDoltTargetsOnlyAuthorizedActiveDatabase under `bazel test //cmd/bd:bd_test`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This commit is contained in:
parent
c4e667cb14
commit
3e7705a2cd
16 changed files with 382 additions and 634 deletions
158
.github/scripts/embedded-cmd-test-shards.txt
vendored
158
.github/scripts/embedded-cmd-test-shards.txt
vendored
|
|
@ -1,161 +1,3 @@
|
|||
# Embedded cmd test shard manifest.
|
||||
#
|
||||
# Format: <total_shards> <shard_number> <top_level_test_name>
|
||||
#
|
||||
# There is no committed recent duration artifact for these tests. This manifest
|
||||
# pins a conservative even split for the 20-shard CI matrix while
|
||||
# embedded-test-shard.sh continues to hash-distribute newly-added tests that are
|
||||
# not listed here.
|
||||
20 1 TestEmbeddedBackup
|
||||
20 2 TestEmbeddedBackupConcurrent
|
||||
20 3 TestEmbeddedBlocked
|
||||
20 4 TestEmbeddedBlockedConcurrent
|
||||
20 6 TestEmbeddedBootstrapConcurrent
|
||||
20 7 TestEmbeddedBranch
|
||||
20 8 TestEmbeddedBranchConcurrent
|
||||
20 10 TestEmbeddedChildrenConcurrent
|
||||
20 11 TestEmbeddedCloseGuardsAndEpics
|
||||
20 12 TestEmbeddedCloseConcurrent
|
||||
20 13 TestEmbeddedComments
|
||||
20 14 TestEmbeddedCommentsCLI
|
||||
20 15 TestEmbeddedCommentsCLIConcurrent
|
||||
20 16 TestEmbeddedCompact
|
||||
20 17 TestEmbeddedCompactConcurrent
|
||||
20 18 TestEmbeddedConfig
|
||||
20 19 TestEmbeddedConfigConcurrent
|
||||
20 20 TestEmbeddedConfigNoWorkspaceStoreFreeCommands
|
||||
20 1 TestEmbeddedConfigValidateJSONNoWorkspaceWritesStdout
|
||||
20 2 TestEmbeddedContext
|
||||
20 3 TestEmbeddedContextConcurrent
|
||||
20 5 TestEmbeddedCount
|
||||
20 6 TestEmbeddedCountConcurrent
|
||||
20 7 TestEmbeddedCreateScalarAndRelationships
|
||||
20 7 TestEmbeddedCreateGraphAndBulk
|
||||
20 8 TestEmbeddedCreateCommitPending
|
||||
20 9 TestEmbeddedCreateConcurrent
|
||||
20 10 TestEmbeddedCreateCrossRepo
|
||||
20 11 TestEmbeddedCreateCrossRepoUninit
|
||||
20 12 TestEmbeddedCreateCrossRepoWithParent
|
||||
20 13 TestEmbeddedCreateWithGitRemote
|
||||
20 14 TestEmbeddedDefer
|
||||
20 15 TestEmbeddedDeferConcurrent
|
||||
20 18 TestEmbeddedDepConcurrent
|
||||
20 19 TestEmbeddedDiff
|
||||
20 20 TestEmbeddedDiffConcurrent
|
||||
20 1 TestEmbeddedDoltBlockedAndCommit
|
||||
20 1 TestEmbeddedDoltRemoteAndPush
|
||||
20 2 TestEmbeddedDoltConcurrent
|
||||
20 3 TestEmbeddedDuplicate
|
||||
20 4 TestEmbeddedDuplicateConcurrent
|
||||
20 6 TestEmbeddedDuplicatesConcurrent
|
||||
20 7 TestEmbeddedEpicStatus
|
||||
20 7 TestEmbeddedEpicCloseEligibleReasonFlag
|
||||
20 7 TestEmbeddedEpicCloseEligibleReasonOutput
|
||||
20 8 TestEmbeddedEpicConcurrent
|
||||
20 9 TestEmbeddedExportFormats
|
||||
20 9 TestEmbeddedExportFlags
|
||||
20 10 TestEmbeddedExportConcurrent
|
||||
20 11 TestEmbeddedFederationPeers
|
||||
20 11 TestEmbeddedFederationStatus
|
||||
20 12 TestEmbeddedFederationConcurrent
|
||||
20 13 TestEmbeddedFindDuplicates
|
||||
20 14 TestEmbeddedFindDuplicatesConcurrent
|
||||
20 15 TestEmbeddedFlatten
|
||||
20 16 TestEmbeddedFlattenConcurrent
|
||||
20 18 TestEmbeddedGCConcurrent
|
||||
20 20 TestEmbeddedGateConcurrent
|
||||
20 2 TestEmbeddedGetDependencies
|
||||
20 3 TestEmbeddedGraph
|
||||
20 4 TestEmbeddedGraphConcurrent
|
||||
20 5 TestEmbeddedHistory
|
||||
20 6 TestEmbeddedHistoryConcurrent
|
||||
20 7 TestEmbeddedHooks
|
||||
20 8 TestEmbeddedHooksConcurrent
|
||||
20 10 TestEmbeddedHumanConcurrent
|
||||
20 11 TestEmbeddedImportFromSources
|
||||
20 11 TestEmbeddedImportUpsertAndPrefixSync
|
||||
20 12 TestEmbeddedImportConcurrent
|
||||
20 13 TestEmbeddedInfo
|
||||
20 14 TestEmbeddedInfoConcurrent
|
||||
20 15 TestEmbeddedInitC
|
||||
20 16 TestEmbeddedInitConcurrent
|
||||
20 17 TestEmbeddedKV
|
||||
20 18 TestEmbeddedKVConcurrent
|
||||
20 19 TestEmbeddedLabelPropagateRename
|
||||
20 20 TestEmbeddedLabelConcurrent
|
||||
20 1 TestEmbeddedLint
|
||||
20 2 TestEmbeddedLintConcurrent
|
||||
20 4 TestEmbeddedListConcurrent
|
||||
20 13 TestEmbeddedMaxRowsListBasic
|
||||
20 13 TestEmbeddedMaxRowsListLimitInteraction
|
||||
20 11 TestEmbeddedMaxRowsDepTreeAndDuplicates
|
||||
20 12 TestEmbeddedMaxRowsDoctorAndConfig
|
||||
20 14 TestEmbeddedMaxRowsGraph
|
||||
20 15 TestEmbeddedMaxRowsReady
|
||||
20 16 TestEmbeddedMaxRowsReadyMerge
|
||||
20 5 TestEmbeddedMemory
|
||||
20 6 TestEmbeddedMemoryConcurrent
|
||||
20 7 TestEmbeddedMigrate
|
||||
20 8 TestEmbeddedMigrateConcurrent
|
||||
20 9 TestEmbeddedOnboard
|
||||
20 10 TestEmbeddedOnboardConcurrent
|
||||
20 11 TestEmbeddedOrphans
|
||||
20 12 TestEmbeddedOrphansConcurrent
|
||||
20 13 TestEmbeddedPrime
|
||||
20 14 TestEmbeddedPrimeConcurrent
|
||||
20 15 TestEmbeddedPromote
|
||||
20 16 TestEmbeddedPromoteCLI
|
||||
20 17 TestEmbeddedPromoteCLIConcurrent
|
||||
20 18 TestEmbeddedPruneSafetyAndScope
|
||||
20 18 TestEmbeddedPruneDryRunAndFilters
|
||||
20 19 TestEmbeddedPurgeBasics
|
||||
20 19 TestEmbeddedPurgeFiltered
|
||||
20 20 TestEmbeddedPurgeConcurrent
|
||||
20 1 TestEmbeddedQuery
|
||||
20 2 TestEmbeddedQueryConcurrent
|
||||
20 3 TestEmbeddedQuick
|
||||
20 4 TestEmbeddedQuickConcurrent
|
||||
20 5 TestEmbeddedQuickstart
|
||||
20 6 TestEmbeddedQuickstartConcurrent
|
||||
20 8 TestEmbeddedReadyConcurrent
|
||||
20 9 TestEmbeddedRenamePrefixBasic
|
||||
20 9 TestEmbeddedRenamePrefixEdgeCases
|
||||
20 10 TestEmbeddedRenamePrefixConcurrent
|
||||
20 11 TestEmbeddedReopen
|
||||
20 12 TestEmbeddedReopenConcurrent
|
||||
20 13 TestEmbeddedRestore
|
||||
20 14 TestEmbeddedRestoreConcurrent
|
||||
20 16 TestEmbeddedSearchConcurrent
|
||||
20 17 TestEmbeddedSetup
|
||||
20 18 TestEmbeddedSetupConcurrent
|
||||
20 19 TestEmbeddedShip
|
||||
20 20 TestEmbeddedShipConcurrent
|
||||
20 1 TestEmbeddedShowBasicsAndJSON
|
||||
20 2 TestEmbeddedShowConcurrent
|
||||
20 3 TestEmbeddedStale
|
||||
20 4 TestEmbeddedStaleConcurrent
|
||||
20 5 TestEmbeddedState
|
||||
20 6 TestEmbeddedStateConcurrent
|
||||
20 7 TestEmbeddedStatus
|
||||
20 8 TestEmbeddedStatusConcurrent
|
||||
20 9 TestEmbeddedSupersede
|
||||
20 10 TestEmbeddedSupersedeConcurrent
|
||||
20 12 TestEmbeddedSwarmConcurrent
|
||||
20 13 TestEmbeddedTodo
|
||||
20 14 TestEmbeddedTodoConcurrent
|
||||
20 15 TestEmbeddedTypes
|
||||
20 16 TestEmbeddedTypesConcurrent
|
||||
20 17 TestEmbeddedUndefer
|
||||
20 18 TestEmbeddedUndeferConcurrent
|
||||
20 20 TestEmbeddedUpdateConcurrent
|
||||
20 1 TestEmbeddedVCStatusAndCommit
|
||||
20 1 TestEmbeddedVCCommitVariantsAndMerge
|
||||
20 2 TestEmbeddedVCConcurrent
|
||||
20 3 TestEmbeddedVersion
|
||||
20 4 TestEmbeddedVersionConcurrent
|
||||
20 5 TestEmbeddedWhere
|
||||
20 6 TestEmbeddedWhereConcurrent
|
||||
|
||||
# Embedded-Dolt cmd/bd test shard manifest.
|
||||
#
|
||||
# Format: <total_shards> <shard_number> <top_level_test_name>
|
||||
|
|
|
|||
232
.github/scripts/proxied-cmd-test-shards.txt
vendored
232
.github/scripts/proxied-cmd-test-shards.txt
vendored
|
|
@ -1,235 +1,3 @@
|
|||
# Proxied-server cmd test shard manifest.
|
||||
#
|
||||
# Format: <total_shards> <shard_number> <top_level_test_name>
|
||||
#
|
||||
# 15-shard split, bin-packed longest-processing-time-first by
|
||||
# estimated cost (bd-init count; per-init migration chains dominate), as of
|
||||
# the test set that existed when this block was last generated.
|
||||
# Regenerating it (scripts/ci/gen_proxied_shard_manifest.py 15 --write) is
|
||||
# deliberately NOT part of normal maintenance: this block is frozen for
|
||||
# pr-risk.yml's/main.yml's legacy fork/push jobs (see engdocs/TESTING.md),
|
||||
# so every TestProxiedServer*/TestServerMode* function added since it was
|
||||
# last generated hash-distributes via proxied-test-shard.sh (cksum(name) %
|
||||
# total) instead of being explicitly listed here. Only reweight it on
|
||||
# purpose, and expect the shard assignments below to change substantially
|
||||
# if you do (a fresh run packs every currently-discovered test, not just
|
||||
# the ones already listed here).
|
||||
#
|
||||
# Deliberate exception (Oct 2026): four tests that had been hash-placed
|
||||
# onto the two shards closest to the 15m -test.timeout are pinned here
|
||||
# by hand, onto shards whose measured CI job time was lowest; nothing
|
||||
# that was already listed moved. TestProxiedServerStorageClass and
|
||||
# TestProxiedServerServeRefusesAStaleCheckpointWithGone (discovered
|
||||
# from #7129 on) had hashed onto shard 5, which then hit the timeout on
|
||||
# most Main runs; pinning them returns shard 5 to its earlier test set.
|
||||
# TestProxiedServerListComments and
|
||||
# TestProxiedServerServeRefusesAStreamOnADisabledJournal had hashed
|
||||
# onto shard 3, which runs within about a minute of the timeout. An
|
||||
# incremental --write keeps pinned lines where they are; a pin is a
|
||||
# reviewed choice, not a regeneration of this block.
|
||||
#
|
||||
# TestProxiedServerOutageReconnectAcceptanceMatrix is pinned to shard 1
|
||||
# (the lightest in recent Main runs, ~6.5 min) instead of hashing onto
|
||||
# shard 14, which already runs ~13.5 min of the 15m timeout. These jobs
|
||||
# have no socat, so today it skips here; the pin keeps shard 14 safe if
|
||||
# socat is ever installed.
|
||||
#
|
||||
# TestProxiedServerClose3 (shard 13) and TestProxiedServerClose4 (shard 2)
|
||||
# are the two parents gastownhall/beads#7173 split off TestProxiedServerClose
|
||||
# to pull shard 5 (15m+) back under the timeout, pinned onto the two
|
||||
# lightest shards of that PR's own run other than shard 5 itself (6m51s
|
||||
# and 7m57s; shard 5 ran 5m50s only because their 12 subtests had just
|
||||
# left it, and putting Close3 back would recreate most of the set that
|
||||
# timed out) rather than hashed; the lines are listed under their shards
|
||||
# like every other pin.
|
||||
#
|
||||
# This file is generated: notes added here are erased by the next
|
||||
# regeneration. Add them to the generator instead. Known blind spots
|
||||
# of the cost proxy (it counts inits, not the work done per init):
|
||||
#
|
||||
# TestProxiedServerListSortRetiresTheWalk is the most expensive
|
||||
# entry and one of the cheapest-looking: off a single init it seeds
|
||||
# 1400 rows and walks the listing three ways. It is the measurement
|
||||
# behind the claim that sort=priority retires the client-side walk,
|
||||
# so it has to run on PRs.
|
||||
#
|
||||
# TestProxiedServerServeReadParity is the CLI/HTTP read parity
|
||||
# oracle, the behavioural half of the anti-drift claim. It being in
|
||||
# this lane, and this lane being required by the PR Risk CI gate,
|
||||
# is what makes that claim enforced on PRs rather than asserted in
|
||||
# a doc comment.
|
||||
|
||||
15 1 TestProxiedServerBlocked
|
||||
15 1 TestProxiedServerCompactDolt
|
||||
15 1 TestProxiedServerCreate
|
||||
15 1 TestProxiedServerDep2
|
||||
15 1 TestProxiedServerDepConcurrent
|
||||
15 1 TestProxiedServerListSortRetiresTheWalk
|
||||
15 1 TestProxiedServerMolLastActivity
|
||||
15 1 TestProxiedServerOutageReconnectAcceptanceMatrix
|
||||
15 1 TestProxiedServerReopenConcurrent
|
||||
15 1 TestProxiedServerServeReadParity
|
||||
15 1 TestProxiedServerServeRefusesAStreamOnADisabledJournal
|
||||
15 1 TestProxiedServerUpdateConcurrentClaim
|
||||
|
||||
15 2 TestProxiedServerClose4
|
||||
15 2 TestProxiedServerCompactHistory
|
||||
15 2 TestProxiedServerCreate2
|
||||
15 2 TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge
|
||||
15 2 TestProxiedServerFindDuplicates
|
||||
15 2 TestProxiedServerListWatchCountsTheOffsetIntoMaxRows
|
||||
15 2 TestProxiedServerReady2
|
||||
15 2 TestProxiedServerSQL
|
||||
15 2 TestProxiedServerServeReadRevision
|
||||
15 2 TestProxiedServerUpdateConcurrentMetadata
|
||||
15 2 TestProxiedServerUpdateIfGuards
|
||||
|
||||
15 3 TestProxiedServerCascadeParity
|
||||
15 3 TestProxiedServerDoltLifecycle
|
||||
15 3 TestProxiedServerMolBurn
|
||||
15 3 TestProxiedServerMolStale
|
||||
15 3 TestProxiedServerMoleculeFrontDoorDirectServerProxyParity
|
||||
15 3 TestProxiedServerSQLDatabaseFlag
|
||||
15 3 TestProxiedServerServeRelatedIssues
|
||||
15 3 TestProxiedServerTag
|
||||
15 3 TestProxiedServerUpdate
|
||||
15 3 TestProxiedServerWispCreateRejectsEnumViolatingVar
|
||||
|
||||
15 4 TestProxiedServerCreateInfraTypeRoutesToWisps
|
||||
15 4 TestProxiedServerEdit
|
||||
15 4 TestProxiedServerListComments
|
||||
15 4 TestProxiedServerMoleculeFrontDoorWriteParity
|
||||
15 4 TestProxiedServerPour
|
||||
15 4 TestProxiedServerReassignFence
|
||||
15 4 TestProxiedServerServeAddComment
|
||||
15 4 TestProxiedServerServeRelease
|
||||
15 4 TestProxiedServerUpdate2
|
||||
15 4 TestProxiedServerUpdateHooks
|
||||
|
||||
15 5 TestProxiedServerClose
|
||||
15 5 TestProxiedServerConfigConcurrent
|
||||
15 5 TestProxiedServerExport
|
||||
15 5 TestProxiedServerGateLifecycle
|
||||
15 5 TestProxiedServerMolReadyGated
|
||||
15 5 TestProxiedServerMultiStatementSQL
|
||||
15 5 TestProxiedServerServeAddDependencies
|
||||
15 5 TestProxiedServerServeRemoveDependency
|
||||
15 5 TestProxiedServerWispCreate
|
||||
|
||||
15 6 TestProxiedServerClose2
|
||||
15 6 TestProxiedServerContext
|
||||
15 6 TestProxiedServerExternalCreate
|
||||
15 6 TestProxiedServerLink
|
||||
15 6 TestProxiedServerNote
|
||||
15 6 TestProxiedServerRelate
|
||||
15 6 TestProxiedServerServeBatchClose
|
||||
15 6 TestProxiedServerServeSettingsWrites
|
||||
15 6 TestProxiedServerWispList
|
||||
|
||||
15 7 TestProxiedServerAssign
|
||||
15 7 TestProxiedServerBackupRefusedOnExternalTopology
|
||||
15 7 TestProxiedServerCount
|
||||
15 7 TestProxiedServerGC
|
||||
15 7 TestProxiedServerList
|
||||
15 7 TestProxiedServerOrphans
|
||||
15 7 TestProxiedServerReady
|
||||
15 7 TestProxiedServerServeClaim
|
||||
15 7 TestProxiedServerServeRefusesAStaleCheckpointWithGone
|
||||
15 7 TestProxiedServerServeUpdate
|
||||
15 7 TestProxiedServerState
|
||||
|
||||
15 8 TestProxiedServerBatchParity
|
||||
15 8 TestProxiedServerCountIncludeInfra
|
||||
15 8 TestProxiedServerDoltRemoteRemove
|
||||
15 8 TestProxiedServerGateCreateInheritsRepoMetadata
|
||||
15 8 TestProxiedServerLabel
|
||||
15 8 TestProxiedServerMolBond
|
||||
15 8 TestProxiedServerPathHelpers
|
||||
15 8 TestProxiedServerRecomputeBlocked
|
||||
15 8 TestProxiedServerServeClaimNext
|
||||
15 8 TestProxiedServerSetState
|
||||
|
||||
15 9 TestProxiedServerChildren
|
||||
15 9 TestProxiedServerComment
|
||||
15 9 TestProxiedServerDeferAutoWake
|
||||
15 9 TestProxiedServerGateFrontDoorDirectServerProxyParity
|
||||
15 9 TestProxiedServerMemory
|
||||
15 9 TestProxiedServerPing
|
||||
15 9 TestProxiedServerPrune
|
||||
15 9 TestProxiedServerReopen
|
||||
15 9 TestProxiedServerServeClose
|
||||
15 9 TestProxiedServerStale
|
||||
|
||||
15 10 TestProxiedServerCleanDatabases
|
||||
15 10 TestProxiedServerDeferUndefer
|
||||
15 10 TestProxiedServerDuplicates
|
||||
15 10 TestProxiedServerGraph
|
||||
15 10 TestProxiedServerPourRejectsEnumViolatingVar
|
||||
15 10 TestProxiedServerSearch
|
||||
15 10 TestProxiedServerServeCount
|
||||
15 10 TestProxiedServerShow
|
||||
15 10 TestProxiedServerStatuses
|
||||
15 10 TestProxiedServerUpdateWisp
|
||||
|
||||
15 11 TestProxiedServerDelete
|
||||
15 11 TestProxiedServerDep
|
||||
15 11 TestProxiedServerGateCheck
|
||||
15 11 TestProxiedServerGateList
|
||||
15 11 TestProxiedServerHeartbeat
|
||||
15 11 TestProxiedServerMolProgress
|
||||
15 11 TestProxiedServerPriority
|
||||
15 11 TestProxiedServerPromote
|
||||
15 11 TestProxiedServerServeCreate
|
||||
15 11 TestProxiedServerSwarm
|
||||
|
||||
15 12 TestProxiedServerDeleteBlockedJSONSingleDoc
|
||||
15 12 TestProxiedServerHistory
|
||||
15 12 TestProxiedServerKV
|
||||
15 12 TestProxiedServerMolShow
|
||||
15 12 TestProxiedServerQuery
|
||||
15 12 TestProxiedServerQuick
|
||||
15 12 TestProxiedServerServeDelete
|
||||
15 12 TestProxiedServerShow2
|
||||
15 12 TestProxiedServerTypes
|
||||
15 12 TestProxiedServerUpdateClosePolicy
|
||||
|
||||
15 13 TestProxiedServerClose3
|
||||
15 13 TestProxiedServerDeleteConcurrent
|
||||
15 13 TestProxiedServerInfo
|
||||
15 13 TestProxiedServerMolSquash
|
||||
15 13 TestProxiedServerReadyClaimConcurrent
|
||||
15 13 TestProxiedServerReadyWisp
|
||||
15 13 TestProxiedServerServeEdgeCounts
|
||||
15 13 TestProxiedServerShow3
|
||||
15 13 TestProxiedServerStatus
|
||||
15 13 TestProxiedServerUnclaim
|
||||
15 13 TestProxiedServerWispGC
|
||||
|
||||
15 14 TestProxiedServerCleanDatabasesPurgeDropped
|
||||
15 14 TestProxiedServerConfig
|
||||
15 14 TestProxiedServerConfigSetMany
|
||||
15 14 TestProxiedServerDeleteForceOrphans
|
||||
15 14 TestProxiedServerImport
|
||||
15 14 TestProxiedServerInitIfMissing
|
||||
15 14 TestProxiedServerMolDistill
|
||||
15 14 TestProxiedServerPrime
|
||||
15 14 TestProxiedServerReadyConcurrent
|
||||
15 14 TestProxiedServerServeLifecycle
|
||||
15 14 TestProxiedServerStorageClass
|
||||
15 14 TestProxiedServerUnclaimIfAssigneeParity
|
||||
|
||||
15 15 TestProxiedServerCloseConcurrent
|
||||
15 15 TestProxiedServerDeleteWisp
|
||||
15 15 TestProxiedServerEpic
|
||||
15 15 TestProxiedServerHuman
|
||||
15 15 TestProxiedServerLint
|
||||
15 15 TestProxiedServerMolCurrent
|
||||
15 15 TestProxiedServerReclaim
|
||||
15 15 TestProxiedServerServeModeGate
|
||||
15 15 TestProxiedServerTodo
|
||||
15 15 TestProxiedServerUpdate3
|
||||
15 15 TestProxiedServerUpdateConcurrentAppendNotes
|
||||
|
||||
# Proxied-server cmd test shard manifest.
|
||||
#
|
||||
# Format: <total_shards> <shard_number> <top_level_test_name>
|
||||
|
|
|
|||
22
.github/workflows/bazel.yml
vendored
22
.github/workflows/bazel.yml
vendored
|
|
@ -1337,9 +1337,11 @@ jobs:
|
|||
JOB_STATUS: ${{ job.status }}
|
||||
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# pr-risk.yml's embedded-Dolt tier ("Test (Embedded Dolt Cmd N/20)",
|
||||
# pr-risk.yml's retired embedded-Dolt tier ("Test (Embedded Dolt Cmd N/20)",
|
||||
# "Test (Embedded Dolt Storage N/5)", "Test (Embedded Dolt Conformance -
|
||||
# core/audit)") as --config=embedded: the embedded-tagged variants of
|
||||
# core/audit)") as --config=embedded: those jobs, and the frozen 20-shard
|
||||
# cmd block they read, no longer exist (see engdocs/TESTING.md) — this runs
|
||||
# the embedded-tagged variants of
|
||||
# //cmd/bd:bd_test and //internal/storage/embeddeddolt:embeddeddolt_test.
|
||||
# The shard variants run the jobs' own shard scripts and manifests, so Bazel
|
||||
# shard k runs exactly the tests of job k. A job of its own rather than a
|
||||
|
|
@ -1830,13 +1832,15 @@ jobs:
|
|||
JOB_STATUS: ${{ job.status }}
|
||||
run: echo "result=$JOB_STATUS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Like pr-risk.yml's proxied-server tier, "Test (Proxied Dolt Cmd N/15)",
|
||||
# as --config=doltserver-proxied: //cmd/bd:bd_proxied_test runs the shard
|
||||
# script, so shard k runs exactly that manifest entry's tests. This lane
|
||||
# runs 34 shards of its own (.github/scripts/proxied-cmd-test-shards.txt's
|
||||
# duration-balanced 34-block; see scripts/ci/proxied_test_durations.json),
|
||||
# not the legacy jobs' 15 — shard k here is not the same split as their
|
||||
# shard k. Same build as --config=doltserver (race bd_test, non-race
|
||||
# Like pr-risk.yml's retired proxied-server tier, "Test (Proxied Dolt Cmd
|
||||
# N/15)" (that job, and the frozen 15-shard block it read, no longer exist;
|
||||
# see engdocs/TESTING.md), as --config=doltserver-proxied:
|
||||
# //cmd/bd:bd_proxied_test runs the shard script, so shard k runs exactly
|
||||
# that manifest entry's tests. This lane runs 34 shards of its own
|
||||
# (.github/scripts/proxied-cmd-test-shards.txt's duration-balanced 34-block;
|
||||
# see scripts/ci/proxied_test_durations.json) — shard k here was never the
|
||||
# same split as the retired job's shard k. Same build as --config=doltserver
|
||||
# (race bd_test, non-race
|
||||
# bd_for_tests), shared with bazel-doltserver through the remote cache.
|
||||
# The test binary is race, as PR Risk's retired jobs built it. It also runs
|
||||
# //cmd/bd:bd_managed_local_test, the managed-local proxied lifecycle
|
||||
|
|
|
|||
|
|
@ -12,6 +12,16 @@ go_binary(
|
|||
srcs = ["testdata/preflight-go.go"],
|
||||
)
|
||||
|
||||
# The external `dolt` stand-in for TestRunCompactDoltTargetsOnlyAuthorizedActiveDatabase
|
||||
# (compact_gc_target_test.go), entirely env-driven at run time, so one
|
||||
# prebuilt copy covers every test case and mode: the test never needs a host
|
||||
# Go toolchain under Bazel (bazeltest.RunfileEnv, BEADS_TEST_COMPACT_GC_FIXTURE).
|
||||
go_binary(
|
||||
name = "compact_gc_fixture",
|
||||
testonly = True,
|
||||
srcs = ["testdata/compact-gc-fixture.go"],
|
||||
)
|
||||
|
||||
go_library(
|
||||
name = "bd_lib",
|
||||
srcs = [
|
||||
|
|
@ -1269,6 +1279,7 @@ go_test(
|
|||
],
|
||||
data = glob(["testdata/**"]) + [
|
||||
":bd_for_tests",
|
||||
":compact_gc_fixture",
|
||||
":go_srcs",
|
||||
":preflight_go_fixture",
|
||||
"//:bd_test_files",
|
||||
|
|
@ -1281,6 +1292,7 @@ go_test(
|
|||
env = {
|
||||
"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)",
|
||||
"BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)",
|
||||
"BEADS_TEST_COMPACT_GC_FIXTURE": "$(rlocationpath :compact_gc_fixture)",
|
||||
"BEADS_TEST_PREFLIGHT_GO": "$(rlocationpath :preflight_go_fixture)",
|
||||
},
|
||||
shard_count = 16,
|
||||
|
|
@ -1413,13 +1425,12 @@ go_test(
|
|||
# check_shard_coverage.py fails unless the two ranges tile the block exactly
|
||||
# once). Both run this configuration's bd_test with
|
||||
# BEADS_TEST_EMBEDDED_DOLT=1 and the race bd (//cmd/bd:bd under
|
||||
# --config=embedded) as the subprocess binary. This is a *different* shard
|
||||
# count from PR Risk's legacy fork/push jobs, "Test (Embedded Dolt Cmd
|
||||
# N/20)" (pr-risk.yml): those keep running 20 shards of their own, frozen
|
||||
# hand-assigned block in the same manifest file — shard k here is not the
|
||||
# same split as their shard k. Each repeats bd_test's `env` (the wrapper
|
||||
# cannot inherit it). The two shard_counts must sum to --shard-total, the
|
||||
# manifest block's shard total.
|
||||
# --config=embedded) as the subprocess binary. PR Risk's legacy fork/push
|
||||
# jobs, "Test (Embedded Dolt Cmd N/20)" (pr-risk.yml), which used to run 20
|
||||
# shards of their own hand-assigned block in the same manifest file, no
|
||||
# longer exist; the manifest now holds only this 100-shard block. Each
|
||||
# repeats bd_test's `env` (the wrapper cannot inherit it). The two
|
||||
# shard_counts must sum to --shard-total, the manifest block's shard total.
|
||||
#
|
||||
# Tags:
|
||||
# embedded: the embedded-Dolt tier; excluded from --config=prcore/ci,
|
||||
|
|
@ -1511,15 +1522,14 @@ sh_test(
|
|||
# functions, assigned by the manifest's 34-shard block (duration-balanced;
|
||||
# see scripts/ci/proxied_test_durations.json), against this configuration's
|
||||
# (race) bd_test with BEADS_TEST_PROXIED_SERVER=1 and the non-race
|
||||
# bd_for_tests as the subprocess binary. This is a *different* shard count
|
||||
# from PR Risk's legacy fork/push jobs, "Test (Proxied Dolt Cmd N/15)"
|
||||
# (pr-risk.yml): those keep running 15 shards of their own,
|
||||
# frozen bd-init-cost-proxy block in the same manifest file — shard k here
|
||||
# is not the same split as their shard k. PR Risk builds its bd-cmd-test
|
||||
# with -race, as here. The shared proxied harness gets its dolt sql-server from
|
||||
# internal/testutil's local backend (the pinned dolt in the test
|
||||
# wrapper's runfiles) instead of the jobs' container, and fails instead
|
||||
# of skipping when it cannot start one. It repeats bd_test's `env`.
|
||||
# bd_for_tests as the subprocess binary. PR Risk's legacy fork/push jobs,
|
||||
# "Test (Proxied Dolt Cmd N/15)" (pr-risk.yml), which used to run 15 shards
|
||||
# of their own frozen bd-init-cost-proxy block in the same manifest file, no
|
||||
# longer exist; the manifest now holds only this 34-shard block. The shared
|
||||
# proxied harness gets its dolt sql-server from internal/testutil's local
|
||||
# backend (the pinned dolt in the test wrapper's runfiles) instead of a
|
||||
# container, and fails instead of skipping when it cannot start one. It
|
||||
# repeats bd_test's `env`.
|
||||
# shard_count must stay equal to the manifest block's shard total.
|
||||
#
|
||||
# Tags:
|
||||
|
|
@ -1604,6 +1614,7 @@ sh_test(
|
|||
data = [
|
||||
":bd_for_tests",
|
||||
":bd_test",
|
||||
":compact_gc_fixture",
|
||||
":go_srcs",
|
||||
":preflight_go_fixture",
|
||||
"@go_sdk//:bin/gofmt",
|
||||
|
|
@ -1612,6 +1623,7 @@ sh_test(
|
|||
"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)",
|
||||
"BEADS_TEST_DOLT_SERVER": "local",
|
||||
"BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)",
|
||||
"BEADS_TEST_COMPACT_GC_FIXTURE": "$(rlocationpath :compact_gc_fixture)",
|
||||
"BEADS_TEST_PREFLIGHT_GO": "$(rlocationpath :preflight_go_fixture)",
|
||||
"BEADS_TEST_PROXIED_LOCAL": "1",
|
||||
"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1",
|
||||
|
|
@ -1660,6 +1672,7 @@ sh_test(
|
|||
data = [
|
||||
":bd_for_tests",
|
||||
":bd_test",
|
||||
":compact_gc_fixture",
|
||||
":dolt_server_pinned_shards.txt",
|
||||
":preflight_go_fixture",
|
||||
"@go_sdk//:bin/gofmt",
|
||||
|
|
@ -1670,6 +1683,7 @@ sh_test(
|
|||
# The integration jobs' global git identity (tools/bazel/test_env.sh).
|
||||
"BEADS_TEST_GIT_IDENTITY": "1",
|
||||
"BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)",
|
||||
"BEADS_TEST_COMPACT_GC_FIXTURE": "$(rlocationpath :compact_gc_fixture)",
|
||||
"BEADS_TEST_PREFLIGHT_GO": "$(rlocationpath :preflight_go_fixture)",
|
||||
"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1",
|
||||
# TestSharedServerConcurrent (skipped everywhere else) as a bounded
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import (
|
|||
"github.com/steveyegge/beads/internal/beads"
|
||||
"github.com/steveyegge/beads/internal/hooks"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/testutil/bazeltest"
|
||||
)
|
||||
|
||||
type compactGCStoreStub struct {
|
||||
|
|
@ -292,31 +293,59 @@ func TestRunCompactDoltTargetsOnlyAuthorizedActiveDatabase(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// compactGCFixtureEnv names the environment variable a go_test sets from its
|
||||
// BUILD file to the prebuilt compact-gc-fixture.go binary (built once by
|
||||
// Bazel as a testonly go_binary), the same pattern preflight_test.go uses for
|
||||
// BEADS_TEST_PREFLIGHT_GO:
|
||||
//
|
||||
// data = [":compact_gc_fixture"],
|
||||
// env = {"BEADS_TEST_COMPACT_GC_FIXTURE": "$(rlocationpath :compact_gc_fixture)"},
|
||||
//
|
||||
// so this test never needs a host Go toolchain under Bazel.
|
||||
const compactGCFixtureEnv = "BEADS_TEST_COMPACT_GC_FIXTURE"
|
||||
|
||||
// buildCompactGCFixture produces a directory holding a `dolt` executable that
|
||||
// stands in for the external Dolt CLI. Under Bazel it copies the prebuilt
|
||||
// fixture from runfiles (bazeltest.RunfileEnv); outside Bazel (plain
|
||||
// `go test`) it falls back to compiling testdata/compact-gc-fixture.go with
|
||||
// the host Go toolchain, as before.
|
||||
func buildCompactGCFixture(t *testing.T) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
const source = `package main
|
||||
import ("encoding/json"; "fmt"; "os")
|
||||
func main() {
|
||||
dir, err := os.Getwd(); if err != nil { panic(err) }
|
||||
log, err := os.OpenFile(os.Getenv("BEADS_COMPACT_GC_FIXTURE_LOG"), os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600); if err != nil { panic(err) }
|
||||
if err := json.NewEncoder(log).Encode(struct { Dir string; Args []string }{dir, os.Args[1:]}); err != nil { panic(err) }; if err := log.Close(); err != nil { panic(err) }
|
||||
mode := os.Getenv("BEADS_COMPACT_GC_FIXTURE_MODE")
|
||||
if mode == "fallback" && len(os.Args) == 4 { fmt.Fprintln(os.Stderr, "unknown flag: --archive-level"); os.Exit(23) }
|
||||
if mode == "failure" { fmt.Fprintln(os.Stderr, "genuine GC failure"); os.Exit(23) }
|
||||
if err := os.WriteFile("gc-ran", []byte("collected"), 0600); err != nil { panic(err) }
|
||||
}
|
||||
`
|
||||
path := filepath.Join(dir, "fixture.go")
|
||||
if err := os.WriteFile(path, []byte(source), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
name := "dolt"
|
||||
if runtime.GOOS == "windows" {
|
||||
name += ".exe"
|
||||
}
|
||||
cmd := exec.Command("go", "build", "-o", filepath.Join(dir, name), path)
|
||||
cmd.Dir = dir
|
||||
target := filepath.Join(dir, name)
|
||||
if bazeltest.IsBazel() {
|
||||
fixture, err := bazeltest.RunfileEnv(compactGCFixtureEnv)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := os.ReadFile(fixture)
|
||||
if err != nil {
|
||||
t.Fatalf("read prebuilt GC fixture: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(target, data, 0o755); err != nil {
|
||||
t.Fatalf("install prebuilt GC fixture: %v", err)
|
||||
}
|
||||
return dir
|
||||
}
|
||||
realGo, err := exec.LookPath("go")
|
||||
if err != nil {
|
||||
t.Fatalf("locate Go toolchain for subprocess fixture: %v", err)
|
||||
}
|
||||
// Absolute, resolved from this file's own directory rather than the test
|
||||
// binary's working directory: `go test` happens to run from the package
|
||||
// directory, but nothing here should depend on that (and t.Chdir, a
|
||||
// -run subset launched from elsewhere, etc. would otherwise break this
|
||||
// silently with a confusing "no such file" from `go build`).
|
||||
_, thisFile, _, ok := runtime.Caller(0)
|
||||
if !ok {
|
||||
t.Fatal("runtime.Caller failed")
|
||||
}
|
||||
path := filepath.Join(filepath.Dir(thisFile), "testdata", "compact-gc-fixture.go")
|
||||
cmd := exec.Command(realGo, "build", "-o", target, path)
|
||||
cmd.Env = append(os.Environ(), "CGO_ENABLED=0", "GOWORK=off")
|
||||
if output, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("build owned GC fixture: %v\n%s", err, output)
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ import (
|
|||
"github.com/steveyegge/beads/internal/metrics"
|
||||
"github.com/steveyegge/beads/internal/migration"
|
||||
"github.com/steveyegge/beads/internal/testutil"
|
||||
"github.com/steveyegge/beads/internal/testutil/bazeltest"
|
||||
"github.com/steveyegge/beads/internal/testutil/credentialcmd"
|
||||
"github.com/steveyegge/beads/internal/workspacegate"
|
||||
)
|
||||
|
|
@ -176,23 +177,28 @@ func testMainInner(m *testing.M) int {
|
|||
// /tmp and exhausted tmpfs over time (bd-3q2u).
|
||||
testTempRoot = tmp
|
||||
|
||||
// Preserve Go build cache before changing HOME.
|
||||
// On macOS, GOCACHE defaults to $HOME/Library/Caches/go-build.
|
||||
// Changing HOME would cause tests that run `go build` (e.g., TestShow)
|
||||
// to miss the cache and do a full CGO rebuild (~80s each).
|
||||
if os.Getenv("GOCACHE") == "" {
|
||||
if out, err := exec.Command("go", "env", "GOCACHE").Output(); err == nil {
|
||||
_ = os.Setenv("GOCACHE", strings.TrimSpace(string(out)))
|
||||
// Preserve Go build cache before changing HOME. On macOS, GOCACHE
|
||||
// defaults to $HOME/Library/Caches/go-build. Changing HOME would cause
|
||||
// tests that run `go build` (e.g., TestShow) to miss the cache and do a
|
||||
// full CGO rebuild (~80s each). Under Bazel there is no host Go
|
||||
// toolchain on the hermetic test PATH (tools/bazel/test_env.sh) and no
|
||||
// in-test `go build` to preserve a cache for, so skip the host `go env`
|
||||
// exec entirely rather than rely on its error being ignored.
|
||||
if !bazeltest.IsBazel() {
|
||||
if os.Getenv("GOCACHE") == "" {
|
||||
if out, err := exec.Command("go", "env", "GOCACHE").Output(); err == nil {
|
||||
_ = os.Setenv("GOCACHE", strings.TrimSpace(string(out)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Same for the module cache: GOMODCACHE defaults to $HOME/go/pkg/mod,
|
||||
// so without this the in-test `go build` (buildEmbeddedBD) re-downloads
|
||||
// every dependency into the temp HOME on each run — slow, and a hard
|
||||
// failure when the network is unavailable.
|
||||
if os.Getenv("GOMODCACHE") == "" {
|
||||
if out, err := exec.Command("go", "env", "GOMODCACHE").Output(); err == nil {
|
||||
_ = os.Setenv("GOMODCACHE", strings.TrimSpace(string(out)))
|
||||
// Same for the module cache: GOMODCACHE defaults to $HOME/go/pkg/mod,
|
||||
// so without this the in-test `go build` (buildEmbeddedBD) re-downloads
|
||||
// every dependency into the temp HOME on each run — slow, and a hard
|
||||
// failure when the network is unavailable.
|
||||
if os.Getenv("GOMODCACHE") == "" {
|
||||
if out, err := exec.Command("go", "env", "GOMODCACHE").Output(); err == nil {
|
||||
_ = os.Setenv("GOMODCACHE", strings.TrimSpace(string(out)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
45
cmd/bd/testdata/compact-gc-fixture.go
vendored
Normal file
45
cmd/bd/testdata/compact-gc-fixture.go
vendored
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
// Command compact-gc-fixture stands in for an external `dolt` binary in
|
||||
// TestRunCompactDoltTargetsOnlyAuthorizedActiveDatabase
|
||||
// (cmd/bd/compact_gc_target_test.go). It is entirely env-driven at run time
|
||||
// (BEADS_COMPACT_GC_FIXTURE_LOG, BEADS_COMPACT_GC_FIXTURE_MODE), so one
|
||||
// prebuilt copy serves every test case and every mode: the test never needs
|
||||
// to invoke a Go toolchain to produce it.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
)
|
||||
|
||||
func main() {
|
||||
dir, err := os.Getwd()
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
log, err := os.OpenFile(os.Getenv("BEADS_COMPACT_GC_FIXTURE_LOG"), os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := json.NewEncoder(log).Encode(struct {
|
||||
Dir string
|
||||
Args []string
|
||||
}{dir, os.Args[1:]}); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
if err := log.Close(); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
mode := os.Getenv("BEADS_COMPACT_GC_FIXTURE_MODE")
|
||||
if mode == "fallback" && len(os.Args) == 4 {
|
||||
fmt.Fprintln(os.Stderr, "unknown flag: --archive-level")
|
||||
os.Exit(23)
|
||||
}
|
||||
if mode == "failure" {
|
||||
fmt.Fprintln(os.Stderr, "genuine GC failure")
|
||||
os.Exit(23)
|
||||
}
|
||||
if err := os.WriteFile("gc-ran", []byte("collected"), 0600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
|
|
@ -1159,11 +1159,12 @@ Required` requires them to have run remotely and passed.
|
|||
- every Bazel shard of `//cmd/bd:bd_embedded_test` and
|
||||
`//cmd/bd:bd_embedded_part2_test` (50 each: shards 1-50 and 51-100 of
|
||||
the manifest's 100-shard block, which their ranges must tile exactly
|
||||
once; the manifest's frozen 20-shard block was the retired
|
||||
`test-embedded-cmd` jobs' split, F1), `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (40;
|
||||
once; the manifest's frozen 20-shard block, once the retired
|
||||
`test-embedded-cmd` jobs' split, has been deleted, F1),
|
||||
`//internal/storage/embeddeddolt:embeddeddolt_embedded_test` (40;
|
||||
the frozen 5-shard block was `test-embedded-storage`'s, F1),
|
||||
`//cmd/bd:bd_proxied_test` (34; the frozen 15-shard block was
|
||||
`test-proxied-cmd`'s, F2) and
|
||||
`//cmd/bd:bd_proxied_test` (34; the manifest's frozen 15-shard block,
|
||||
once `test-proxied-cmd`'s split, has been deleted, F2) and
|
||||
`//internal/storage/dolt:dolt_server_full_test` (16) to have run
|
||||
exactly the tests its shard script lists (list-only mode, minus
|
||||
`TestMain`, which `grep '^func Test'` lists but which is never a
|
||||
|
|
|
|||
|
|
@ -184,9 +184,9 @@ builds with the integration tag like `--config=integration`. Each shard runs
|
|||
its CI job's shard script, so for `--config=doltserver-integration` Bazel
|
||||
shard k runs the tests of job k+1 (both split the manifest's 16-shard block
|
||||
the same way). `--config=doltserver-proxied`'s `bd_proxied_test` instead
|
||||
runs the manifest's own 34-shard block — bin-packed by measured duration,
|
||||
not the legacy jobs' 15-shard, bd-init-cost-proxy block — so shard k there
|
||||
is not job k+1's tests; it is a different split of the same tests.
|
||||
runs the manifest's own 34-shard block, bin-packed by measured duration (the
|
||||
manifest's legacy 15-shard, bd-init-cost-proxy block, once read by PR Risk's
|
||||
retired fork/push jobs, has been deleted; those jobs no longer exist).
|
||||
|
||||
`--config=doltserver-cmd` (`//cmd/bd:bd_dolt_server_test`, bazel.yml's
|
||||
advisory `bazel-cmd-dolt` job) runs the whole integration-tagged cmd/bd
|
||||
|
|
|
|||
|
|
@ -15,9 +15,29 @@ cd "$REPO_ROOT"
|
|||
GOFMT_BIN="$("$SCRIPT_DIR/gofmt-bin.sh")"
|
||||
|
||||
describe_gofmt() {
|
||||
local bin="$1" version=""
|
||||
if command -v go >/dev/null 2>&1; then
|
||||
version="$(go version "$bin" 2>/dev/null | awk '{ print $NF }')"
|
||||
local bin="$1" version="" go_bin=""
|
||||
if [[ -n "${TEST_SRCDIR:-}" ]]; then
|
||||
# Under `bazel test` a bare `go` on PATH can be the farm's exit-127
|
||||
# mask for the host toolchain, so only call a go binary the test
|
||||
# explicitly declared as data: BEADS_TEST_GO, set the same way
|
||||
# scripts/prlintmake:prlintmake_test wires in the registered SDK's go
|
||||
# ($(rlocationpath @go_sdk//:bin/go)). A caller that did not wire it
|
||||
# up (e.g. scripts/repochecks:fmt_test) just gets the version-less
|
||||
# fallback below, never a PATH `go`.
|
||||
if [[ -n "${BEADS_TEST_GO:-}" ]]; then
|
||||
local candidate="$BEADS_TEST_GO"
|
||||
if [[ "$candidate" != /* ]]; then
|
||||
candidate="$TEST_SRCDIR/$candidate"
|
||||
fi
|
||||
if [[ -x "$candidate" ]]; then
|
||||
go_bin="$candidate"
|
||||
fi
|
||||
fi
|
||||
elif command -v go >/dev/null 2>&1; then
|
||||
go_bin="go"
|
||||
fi
|
||||
if [[ -n "$go_bin" ]]; then
|
||||
version="$("$go_bin" version "$bin" 2>/dev/null | awk '{ print $NF }')"
|
||||
fi
|
||||
if [[ -n "$version" ]]; then
|
||||
printf '%s (%s)' "$version" "$bin"
|
||||
|
|
|
|||
|
|
@ -4,32 +4,35 @@
|
|||
Discovery must stay in step with embedded-test-shard.sh: both select every
|
||||
top-level TestEmbedded* function from cmd/bd/*_embedded_test.go.
|
||||
|
||||
Two cost models, selected by --weights (default: inits, matching
|
||||
Two cost models, selected by --weights (default: duration, matching
|
||||
gen_proxied_shard_manifest.py's convention — see that script's module
|
||||
docstring for the full rationale this one shares):
|
||||
|
||||
--weights=inits (default): a cheap static proxy for wall-time — the
|
||||
function's subtest count (t.Run( occurrences), floored at 1. Unlike the
|
||||
proxied generator's bd-init count, embedded cmd tests do not share one
|
||||
dominant per-call cost center, so subtest count is the simplest proxy
|
||||
that is still monotonic in "more scenarios this function exercises."
|
||||
|
||||
--weights=duration: measured wall-time from
|
||||
--weights=duration (default): measured wall-time from
|
||||
scripts/ci/embedded_cmd_test_durations.json (see that file's header for
|
||||
provenance). A function missing from that file — a test added since it
|
||||
was last captured — falls back to its inits cost times the file's
|
||||
"seconds_per_init_fallback" ratio. A from-scratch pack models the shard
|
||||
process (pack_concurrent): functions without a top-level t.Parallel()
|
||||
run one after another, the rest share -test.parallel=4 slots.
|
||||
run one after another, the rest share -test.parallel=4 slots. The
|
||||
manifest's only remaining block is duration-packed (see below), so this
|
||||
is also the only --weights whose render() header matches what is
|
||||
committed — a bare `--write` with no flags reproduces the committed file
|
||||
instead of silently clobbering its header with the inits one.
|
||||
|
||||
This file's existing 20-shard block (hand-assigned, round-robin; see its own
|
||||
header) is the FROZEN block PR Risk's and main.yml's legacy fork/push jobs
|
||||
read — see engdocs/TESTING.md. Do NOT regenerate it. This generator's
|
||||
--weights=duration path targets a *separate*, Bazel-only shard total (the
|
||||
bazel-embedded job in .github/workflows/bazel.yml): a different total_shards
|
||||
value, written as its own block in the same manifest file (see
|
||||
split_blocks()/write_block() in _embedded_shard_manifest_lib.py for how one
|
||||
file holds more than one independent block).
|
||||
--weights=inits: a cheap static proxy for wall-time — the function's
|
||||
subtest count (t.Run( occurrences), floored at 1. Unlike the proxied
|
||||
generator's bd-init count, embedded cmd tests do not share one dominant
|
||||
per-call cost center, so subtest count is the simplest proxy that is
|
||||
still monotonic in "more scenarios this function exercises." Pass this
|
||||
explicitly if you want it; --write still honors it, it is just no longer
|
||||
the default.
|
||||
|
||||
This generator's --weights=duration path targets the Bazel-only shard total
|
||||
(the bazel-embedded job in .github/workflows/bazel.yml). The legacy,
|
||||
hand-assigned 20-shard block that PR Risk's and main.yml's retired fork/push
|
||||
jobs used to read has been deleted (those jobs no longer exist; see
|
||||
engdocs/TESTING.md): the manifest file now holds exactly the one live block.
|
||||
|
||||
--write is INCREMENTAL by default (see --repack to force a full rebalance);
|
||||
--check verifies exact-once NAME coverage only, not shard assignments. Both
|
||||
|
|
@ -54,7 +57,9 @@ from _embedded_shard_manifest_lib import ( # noqa: E402
|
|||
|
||||
default_manifest_path = '.github/scripts/embedded-cmd-test-shards.txt'
|
||||
durations_filename = 'embedded_cmd_test_durations.json'
|
||||
default_total_shards = 20
|
||||
# The Bazel-only block's total (bd_embedded_test + bd_embedded_part2_test's
|
||||
# shard_counts in cmd/bd/BUILD.bazel must sum to this).
|
||||
default_total_shards = 100
|
||||
# .bazelrc's test:embedded --test_arg=-test.parallel=4: how many t.Parallel
|
||||
# top-level tests one shard's process runs at once (--weights=duration).
|
||||
test_parallelism = 4
|
||||
|
|
@ -131,35 +136,27 @@ def render(total, shards, weights):
|
|||
out.append('#')
|
||||
out.append('# Format: <total_shards> <shard_number> <top_level_test_name>')
|
||||
out.append('#')
|
||||
if total == default_total_shards and weights == 'inits':
|
||||
out.append(f'# {total}-shard split. This block is hand-assigned (round-robin,')
|
||||
out.append('# not cost-balanced) and FROZEN for pr-risk.yml/main.yml\'s legacy')
|
||||
out.append('# fork/push jobs (see engdocs/TESTING.md): do not regenerate it. Newly-')
|
||||
out.append('# added tests not listed here hash-distribute via embedded-test-shard.sh:')
|
||||
out.append('# cksum(name) % total. --check does not cover this block; only the')
|
||||
out.append('# Bazel-only --weights=duration block is checked in CI.')
|
||||
else:
|
||||
out.append(f'# {total}-shard split for the Bazel-only embedded-Dolt cmd tier')
|
||||
out.append('# (bazel-embedded in .github/workflows/bazel.yml), bin-packed')
|
||||
out.append('# longest-first by measured wall-time from')
|
||||
out.append(f'# scripts/ci/{durations_filename} (see that file for provenance')
|
||||
out.append('# and its "relative weight, not absolute SLA" caveat) onto the shard')
|
||||
out.append('# whose estimated wall time grows least: serial tests (no top-level')
|
||||
out.append(f'# t.Parallel) add up, parallel ones share -test.parallel={test_parallelism}.')
|
||||
out.append(f'# Regenerate with scripts/ci/gen_embedded_cmd_shard_manifest.py {total}')
|
||||
out.append('# --weights=duration --write after adding, splitting or removing')
|
||||
out.append('# TestEmbedded* functions in cmd/bd/*_embedded_test.go. --write is')
|
||||
out.append('# incremental: it keeps every already-assigned test on its current shard')
|
||||
out.append('# and only places newly-discovered names, onto whichever shard is')
|
||||
out.append('# currently lightest, so two unrelated PRs that each add one test do not')
|
||||
out.append('# textually conflict or silently invalidate each other\'s packing. Pass')
|
||||
out.append('# --repack to force a full from-scratch rebalance instead (a deliberate,')
|
||||
out.append('# separate change — expect a large diff). --check only verifies every')
|
||||
out.append('# discovered name is listed here exactly once (no stale or duplicate')
|
||||
out.append('# entries); it does NOT verify the packing is still well-balanced.')
|
||||
out.append('#')
|
||||
out.append('# This file is generated: notes added here are erased by the next')
|
||||
out.append('# regeneration. Add them to the generator or the durations file instead.')
|
||||
out.append(f'# {total}-shard split for the Bazel-only embedded-Dolt cmd tier')
|
||||
out.append('# (bazel-embedded in .github/workflows/bazel.yml), bin-packed')
|
||||
out.append('# longest-first by measured wall-time from')
|
||||
out.append(f'# scripts/ci/{durations_filename} (see that file for provenance')
|
||||
out.append('# and its "relative weight, not absolute SLA" caveat) onto the shard')
|
||||
out.append('# whose estimated wall time grows least: serial tests (no top-level')
|
||||
out.append(f'# t.Parallel) add up, parallel ones share -test.parallel={test_parallelism}.')
|
||||
out.append(f'# Regenerate with scripts/ci/gen_embedded_cmd_shard_manifest.py {total}')
|
||||
out.append('# --weights=duration --write after adding, splitting or removing')
|
||||
out.append('# TestEmbedded* functions in cmd/bd/*_embedded_test.go. --write is')
|
||||
out.append('# incremental: it keeps every already-assigned test on its current shard')
|
||||
out.append('# and only places newly-discovered names, onto whichever shard is')
|
||||
out.append('# currently lightest, so two unrelated PRs that each add one test do not')
|
||||
out.append('# textually conflict or silently invalidate each other\'s packing. Pass')
|
||||
out.append('# --repack to force a full from-scratch rebalance instead (a deliberate,')
|
||||
out.append('# separate change — expect a large diff). --check only verifies every')
|
||||
out.append('# discovered name is listed here exactly once (no stale or duplicate')
|
||||
out.append('# entries); it does NOT verify the packing is still well-balanced.')
|
||||
out.append('#')
|
||||
out.append('# This file is generated: notes added here are erased by the next')
|
||||
out.append('# regeneration. Add them to the generator or the durations file instead.')
|
||||
out.append('')
|
||||
for i in range(total):
|
||||
for name in sorted(shards[i]):
|
||||
|
|
@ -171,7 +168,7 @@ def render(total, shards, weights):
|
|||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])
|
||||
ap.add_argument('total_shards', nargs='?', type=int, default=None)
|
||||
ap.add_argument('--weights', choices=['inits', 'duration'], default='inits')
|
||||
ap.add_argument('--weights', choices=['inits', 'duration'], default='duration')
|
||||
ap.add_argument('--manifest', default=default_manifest_path,
|
||||
help=f'manifest file to read/write (default: {default_manifest_path})')
|
||||
mode = ap.add_mutually_exclusive_group()
|
||||
|
|
@ -186,20 +183,6 @@ def main():
|
|||
'from-scratch LPT pack')
|
||||
args = ap.parse_args()
|
||||
if args.total_shards is None:
|
||||
# N3 (F1 review): omitting total_shards used to default to 20, the
|
||||
# FROZEN legacy block's own total, with --weights=inits also
|
||||
# defaulting on -- so a bare `--write` silently rewrote the frozen
|
||||
# legacy 20-shard block instead of erroring or targeting the
|
||||
# Bazel-only block. Require the caller to say which total they mean
|
||||
# whenever they're about to write; --check/dry-run keep the old
|
||||
# default_total_shards=20 fallback since reading the legacy block is
|
||||
# harmless and already how engdocs/TESTING.md documents inspecting it.
|
||||
if args.write:
|
||||
ap.error(f'total_shards is required with --write (the default, {default_total_shards}, '
|
||||
'is the FROZEN legacy block -- see its header in embedded-cmd-test-shards.txt; '
|
||||
'it must not be regenerated). Pass the Bazel-only total explicitly instead, e.g. '
|
||||
'"100 --weights=duration" for bazel-embedded\'s block (see cmd/bd/BUILD.bazel\'s '
|
||||
'bd_embedded_test/bd_embedded_part2_test --shard-total for the current value)')
|
||||
args.total_shards = default_total_shards
|
||||
|
||||
inits = discover_inits_cost()
|
||||
|
|
|
|||
|
|
@ -5,24 +5,24 @@ Discovery must stay in step with proxied-test-shard.sh: both select the
|
|||
TestProxiedServer* and TestServerMode* entry points, which share the
|
||||
requireSharedProxiedServer gate and the shared dolt sql-server.
|
||||
|
||||
Two cost models, selected by --weights (default: inits, unchanged from
|
||||
before this flag existed — the committed 15-shard block for the legacy
|
||||
pr-risk.yml/main.yml jobs was generated this way and is NOT regenerated by
|
||||
this script's --weights=duration path; it stays frozen so those jobs' shard
|
||||
assignment keeps its exact current behavior):
|
||||
Two cost models, selected by --weights (default: duration):
|
||||
|
||||
--weights=inits (default): bd-init count (newSharedProxiedProject
|
||||
occurrences) per top-level function — a cheap static proxy for
|
||||
wall-time that undercounts a test whose cost is dominated by what it
|
||||
does per init rather than how many inits it runs (see the two
|
||||
known-expensive entries in the committed 15-shard manifest's header).
|
||||
|
||||
--weights=duration: measured wall-time from
|
||||
--weights=duration (default): measured wall-time from
|
||||
scripts/ci/proxied_test_durations.json (that file's header explains its
|
||||
provenance and limits). A function missing from that file — a test
|
||||
added since it was last captured — falls back to its inits cost times
|
||||
the file's "seconds_per_init_fallback" ratio, so a brand-new test still
|
||||
gets a plausible seconds-shaped weight instead of a unit-mismatched one.
|
||||
The manifest's only remaining block is duration-packed (see below), so
|
||||
this is also the only --weights that renders a header matching what is
|
||||
committed — a bare `--write` with no flags reproduces the committed file
|
||||
instead of silently clobbering its header with the inits one.
|
||||
|
||||
--weights=inits: bd-init count (newSharedProxiedProject occurrences) per
|
||||
top-level function — a cheap static proxy for wall-time that undercounts
|
||||
a test whose cost is dominated by what it does per init rather than how
|
||||
many inits it runs. Pass this explicitly if you want it; --write still
|
||||
honors it, it is just no longer the default.
|
||||
|
||||
Either way, a from-scratch pack puts TOTAL shards' worth of tests together
|
||||
longest-processing-time-first (heaviest first onto the currently lightest
|
||||
|
|
@ -54,16 +54,15 @@ and read the stderr shard-load report, or pass --check anyway: it still
|
|||
reports the block's current loads to stderr even though it only gates on
|
||||
name coverage.
|
||||
|
||||
The manifest file holds more than one block (one per total_shards value in
|
||||
use): the legacy, frozen 15-shard block pr-risk.yml/main.yml's jobs read, and
|
||||
the Bazel-only lane's own block (34 shards since 2026-10-09). By default this prints
|
||||
only the requested total's block to stdout, which is NOT safe to redirect
|
||||
straight into the manifest file (`gen... 30 > file` deletes every other
|
||||
block). Use --write to update the file in place instead: it replaces only
|
||||
the block for the requested total_shards and leaves every other block
|
||||
untouched. Use --check to verify the committed block for a total_shards
|
||||
has exact-once name coverage, without writing anything — this is the form a
|
||||
CI policy check should call.
|
||||
The manifest file holds the Bazel-only lane's one block (34 shards since
|
||||
2026-10-09; the legacy, frozen 15-shard block pr-risk.yml/main.yml's retired
|
||||
fork/push jobs used to read has been deleted — those jobs no longer exist,
|
||||
see engdocs/TESTING.md). By default this prints the requested total's block
|
||||
to stdout, which is NOT safe to redirect straight into the manifest file
|
||||
(`gen... 30 > file` loses the header if total does not match what is
|
||||
committed). Use --write to update the file in place instead. Use --check to
|
||||
verify the committed block for a total_shards has exact-once name coverage,
|
||||
without writing anything — this is the form a CI policy check should call.
|
||||
|
||||
Usage: gen_proxied_shard_manifest.py [total_shards] [--weights=inits|duration]
|
||||
[--write [--repack] | --check] [--manifest PATH]
|
||||
|
|
@ -187,44 +186,12 @@ def render(total, shards, weights):
|
|||
if weights == 'inits':
|
||||
out.append(f'# {total}-shard split, bin-packed longest-processing-time-first by')
|
||||
out.append('# estimated cost (bd-init count; per-init migration chains dominate).')
|
||||
out.append(f'# Regenerating it (scripts/ci/gen_proxied_shard_manifest.py {total} --write)')
|
||||
out.append('# is deliberately NOT part of normal maintenance; see the frozen note')
|
||||
out.append('# below. Newly-added tests not listed here hash-distribute via')
|
||||
out.append(f'# Regenerate with scripts/ci/gen_proxied_shard_manifest.py {total} --write.')
|
||||
out.append('# Newly-added tests not listed here hash-distribute via')
|
||||
out.append('# proxied-test-shard.sh: cksum(name) % total.')
|
||||
out.append('#')
|
||||
out.append('# This file is generated: notes added here are erased by the next')
|
||||
out.append('# regeneration. Add them to the generator instead. Known blind spots')
|
||||
out.append('# of the cost proxy (it counts inits, not the work done per init):')
|
||||
out.append('#')
|
||||
out.append('# TestProxiedServerListSortRetiresTheWalk is the most expensive')
|
||||
out.append('# entry and one of the cheapest-looking: off a single init it seeds')
|
||||
out.append('# 1400 rows and walks the listing three ways. It is the measurement')
|
||||
out.append('# behind the claim that sort=priority retires the client-side walk,')
|
||||
out.append('# so it has to run on PRs.')
|
||||
out.append('#')
|
||||
out.append('# TestProxiedServerServeReadParity is the CLI/HTTP read parity')
|
||||
out.append('# oracle, the behavioural half of the anti-drift claim. It being in')
|
||||
out.append('# this lane, and this lane being required by the PR Risk CI gate,')
|
||||
out.append('# is what makes that claim enforced on PRs rather than asserted in')
|
||||
out.append('# a doc comment.')
|
||||
out.append('#')
|
||||
out.append(f'# This {total}-shard block is frozen for pr-risk.yml/main.yml\'s legacy')
|
||||
out.append('# fork/push jobs (see engdocs/TESTING.md): do not regenerate it even')
|
||||
out.append('# after adding tests (they hash-distribute until a maintainer chooses')
|
||||
out.append('# to reweight on purpose). --check does not cover this block; only the')
|
||||
out.append('# Bazel-only --weights=duration block is checked in CI.')
|
||||
out.append('#')
|
||||
out.append('# Hand-pinned lines are allowed: when a hash-placed test overloads a')
|
||||
out.append('# shard toward the 15m -test.timeout, a reviewed change may list it')
|
||||
out.append('# on a lighter shard without moving anything already listed (done in')
|
||||
out.append('# Oct 2026 for TestProxiedServerStorageClass,')
|
||||
out.append('# TestProxiedServerServeRefusesAStaleCheckpointWithGone,')
|
||||
out.append('# TestProxiedServerListComments and')
|
||||
out.append('# TestProxiedServerServeRefusesAStreamOnADisabledJournal; and for')
|
||||
out.append('# TestProxiedServerOutageReconnectAcceptanceMatrix, pinned to shard 1')
|
||||
out.append('# instead of hashing onto shard 14; and for TestProxiedServerClose3')
|
||||
out.append('# and TestProxiedServerClose4, the parents #7173 split off')
|
||||
out.append('# TestProxiedServerClose, pinned to shards 13 and 2).')
|
||||
out.append('# regeneration. Add them to the generator instead.')
|
||||
else:
|
||||
out.append(f'# {total}-shard split for the Bazel-only proxied-server tier')
|
||||
out.append('# (bazel-proxied in .github/workflows/bazel.yml), bin-packed')
|
||||
|
|
@ -361,8 +328,8 @@ def check_coverage(path, total, universe):
|
|||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])
|
||||
ap.add_argument('total_shards', nargs='?', type=int, default=15)
|
||||
ap.add_argument('--weights', choices=['inits', 'duration'], default='inits')
|
||||
ap.add_argument('total_shards', nargs='?', type=int, default=34)
|
||||
ap.add_argument('--weights', choices=['inits', 'duration'], default='duration')
|
||||
ap.add_argument('--manifest', default=default_manifest_path,
|
||||
help=f'manifest file to read/write (default: {default_manifest_path})')
|
||||
mode = ap.add_mutually_exclusive_group()
|
||||
|
|
|
|||
|
|
@ -19,18 +19,25 @@ beads_test_env_enter() {
|
|||
export BEADS_TEST_ENV_ROOT="$root"
|
||||
export BEADS_TEST_ENV_ACTIVE=1
|
||||
|
||||
if [[ -z "${GOCACHE:-}" ]]; then
|
||||
local go_cache
|
||||
go_cache="$(go env GOCACHE 2>/dev/null || true)"
|
||||
if [[ -n "$go_cache" ]]; then
|
||||
export GOCACHE="$go_cache"
|
||||
# Under `bazel test` (TEST_SRCDIR set) a bare `go` on PATH can be the
|
||||
# farm's exit-127 mask for the host toolchain, and nothing downstream of
|
||||
# this wrapper in that mode builds Go code or needs a cache directory
|
||||
# (scripts/test.sh's prebuilt-test-binary path runs a binary Bazel already
|
||||
# built), so skip the probe entirely rather than risk invoking it.
|
||||
if [[ -z "${TEST_SRCDIR:-}" ]]; then
|
||||
if [[ -z "${GOCACHE:-}" ]]; then
|
||||
local go_cache
|
||||
go_cache="$(go env GOCACHE 2>/dev/null || true)"
|
||||
if [[ -n "$go_cache" ]]; then
|
||||
export GOCACHE="$go_cache"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
if [[ -z "${GOMODCACHE:-}" ]]; then
|
||||
local go_mod_cache
|
||||
go_mod_cache="$(go env GOMODCACHE 2>/dev/null || true)"
|
||||
if [[ -n "$go_mod_cache" ]]; then
|
||||
export GOMODCACHE="$go_mod_cache"
|
||||
if [[ -z "${GOMODCACHE:-}" ]]; then
|
||||
local go_mod_cache
|
||||
go_mod_cache="$(go env GOMODCACHE 2>/dev/null || true)"
|
||||
if [[ -n "$go_mod_cache" ]]; then
|
||||
export GOMODCACHE="$go_mod_cache"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
|
|
|
|||
|
|
@ -82,10 +82,12 @@ func bazelEmbeddedCmdParts(t *testing.T) []embeddedCmdPart {
|
|||
// bazelEmbeddedCmdShardCount returns the shard total of the bazel-embedded
|
||||
// lane's cmd/bd manifest block, which embeddedCmdTargets split between them
|
||||
// (bazelEmbeddedCmdParts): the single source of truth for the Bazel-only
|
||||
// lane's cmd/bd shard split (slice F1), which no longer has to equal PR
|
||||
// lane's cmd/bd shard split (slice F1). It no longer has to equal PR
|
||||
// Risk's/main.yml's legacy "Test (Embedded Dolt Cmd N/20)" fork/push jobs'
|
||||
// matrix size — mirrors bazelProxiedShardCount (F2; see that function's doc
|
||||
// comment below for the shared rationale, not repeated here).
|
||||
// matrix size — those jobs, and the frozen 20-shard block they read, have
|
||||
// since been deleted (see engdocs/TESTING.md) — mirrors bazelProxiedShardCount
|
||||
// (F2; see that function's doc comment below for the shared rationale, not
|
||||
// repeated here).
|
||||
func bazelEmbeddedCmdShardCount(t *testing.T) int {
|
||||
t.Helper()
|
||||
return bazelEmbeddedCmdParts(t)[0].total
|
||||
|
|
|
|||
|
|
@ -333,9 +333,12 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) {
|
|||
{bazelEmbedJobName, "embedded", []suite{
|
||||
// bazel-embedded runs its own duration-balanced manifest blocks
|
||||
// (scripts/ci/embedded_{cmd,storage}_test_durations.json), not
|
||||
// PR Risk's frozen 20- and 5-shard blocks (slice F1): neither is
|
||||
// a drop-in retirement of its legacy job's shard count, just its
|
||||
// tests. bazelEmbeddedCmdShardCount/bazelEmbeddedStorageShardCount
|
||||
// PR Risk's 20-shard cmd block, which was frozen and has since
|
||||
// been deleted (those jobs no longer exist; see
|
||||
// engdocs/TESTING.md), or the still-frozen 5-shard storage block
|
||||
// (slice F1): neither is a drop-in retirement of its legacy
|
||||
// job's shard count, just its tests.
|
||||
// bazelEmbeddedCmdShardCount/bazelEmbeddedStorageShardCount
|
||||
// read the real counts from BUILD.bazel, making these suites'
|
||||
// `want` (below) genuine cross-file pins against bazel.yml's own
|
||||
// check_shard_coverage.py arguments, not independently
|
||||
|
|
@ -347,9 +350,11 @@ func TestBazelRetiredLanesCheckListedTestsRan(t *testing.T) {
|
|||
}, []string{"//internal/storage/embeddeddolt:embeddeddolt_conformance_core_test", "//internal/storage/embeddeddolt:embeddeddolt_conformance_core_slow_test", "//internal/storage/embeddeddolt:embeddeddolt_conformance_audit_test"}},
|
||||
{bazelProxiedJobName, "doltserver-proxied", []suite{
|
||||
// bazel-proxied runs its own duration-balanced manifest block
|
||||
// (scripts/ci/proxied_test_durations.json), not PR Risk's frozen
|
||||
// 15-shard bd-init-cost-proxy block: it is not a drop-in
|
||||
// retirement of test-proxied-cmd's shard count, just its tests.
|
||||
// (scripts/ci/proxied_test_durations.json), not PR Risk's
|
||||
// 15-shard bd-init-cost-proxy block, which was frozen and has
|
||||
// since been deleted (that job no longer exists; see
|
||||
// engdocs/TESTING.md): it was never a drop-in retirement of
|
||||
// test-proxied-cmd's shard count, just its tests.
|
||||
// bazelProxiedShardCount reads the real count from
|
||||
// cmd/bd/BUILD.bazel, making this suite's `want` (below) a
|
||||
// genuine cross-file pin against bazel.yml's own
|
||||
|
|
@ -913,8 +918,9 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) {
|
|||
"//cmd/bd:bd_managed_local_test": {
|
||||
[]string{"$(rootpath :bd_test)", "-test.run=^TestManagedLocalProxied", "-test.timeout=15m"},
|
||||
map[string]string{
|
||||
"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)", "BEADS_TEST_DOLT_SERVER": "local",
|
||||
"BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", "BEADS_TEST_PREFLIGHT_GO": "$(rlocationpath :preflight_go_fixture)",
|
||||
"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)", "BEADS_TEST_COMPACT_GC_FIXTURE": "$(rlocationpath :compact_gc_fixture)",
|
||||
"BEADS_TEST_DOLT_SERVER": "local",
|
||||
"BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", "BEADS_TEST_PREFLIGHT_GO": "$(rlocationpath :preflight_go_fixture)",
|
||||
"BEADS_TEST_PROXIED_LOCAL": "1", "BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", "BEADS_TEST_SKIP": "dolt",
|
||||
},
|
||||
},
|
||||
|
|
|
|||
|
|
@ -19,19 +19,20 @@ import (
|
|||
|
||||
// S3 (F1, mirroring F2's TestProxiedShardManifestGeneratorNotStale below —
|
||||
// see that test's doc comment for the full --check rationale, not repeated
|
||||
// here): the Bazel-only 100-shard cmd block and 40-shard storage block are
|
||||
// not frozen like their files' legacy 20- and 5-shard blocks.
|
||||
// here): the Bazel-only 100-shard cmd block and 40-shard storage block.
|
||||
// gen_embedded_{cmd,storage}_shard_manifest.py --check verifies only that
|
||||
// the committed block names every discovered test exactly once, failing with
|
||||
// the exact command to fix it when a name is missing, stale, or duplicated —
|
||||
// run here so a drifted block fails `go test ./scripts/...`
|
||||
// (//scripts:go_test_sources_test under Bazel) instead of only surfacing as
|
||||
// a test silently never running in any shard. The legacy blocks are
|
||||
// deliberately excluded: both files document that their 20- and 5-shard
|
||||
// blocks are frozen (see .github/scripts/embedded-{cmd,storage}-
|
||||
// test-shards.txt and engdocs/TESTING.md), so a --check against them is
|
||||
// expected to report "missing" entries by design (see those generators'
|
||||
// module docstrings) and is not what this test runs.
|
||||
// a test silently never running in any shard. The cmd file's legacy,
|
||||
// hand-assigned 20-shard block (frozen for PR Risk's and main.yml's retired
|
||||
// fork/push jobs) has been deleted; those jobs no longer exist (see
|
||||
// engdocs/TESTING.md). The storage file's legacy 5-shard block is still
|
||||
// frozen and deliberately excluded here: its header documents that (see
|
||||
// .github/scripts/embedded-storage-test-shards.txt and engdocs/TESTING.md),
|
||||
// so a --check against it is expected to report "missing" entries by design
|
||||
// (see that generator's module docstring) and is not what this test runs.
|
||||
func TestCmdEmbeddedShardManifestGeneratorNotStale(t *testing.T) {
|
||||
python := requireHostTool(t, "python3")
|
||||
root := sourceRepoRoot(t)
|
||||
|
|
@ -158,9 +159,13 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) {
|
|||
case ".github/scripts/proxied-test-shard.sh":
|
||||
totalsSet[bazelProxiedShardCount(t)] = true
|
||||
case ".github/scripts/embedded-test-shard.sh":
|
||||
// F1: the Bazel-only bazel-embedded lane reads a 50-shard cmd
|
||||
// block that no PR-Risk-matrix-only (20-shard) check exercises;
|
||||
// without this, "BUILD.bazel's shard_count and bazel.yml's
|
||||
// F1: the Bazel-only bazel-embedded lane reads a 100-shard cmd
|
||||
// block (split over embeddedCmdTargets); the legacy, frozen
|
||||
// 20-shard block PR Risk's/main.yml's retired fork/push jobs
|
||||
// used to read has been deleted (those jobs no longer exist; see
|
||||
// engdocs/TESTING.md), so there is no longer a second check that
|
||||
// would catch this total drifting on its own. Without this,
|
||||
// "BUILD.bazel's shard_count and bazel.yml's
|
||||
// check_shard_coverage.py arg both drift to a new total with no
|
||||
// manifest block" passes every policy test (see review S3) --
|
||||
// the lane then silently goes 100% hash fallback and loses its
|
||||
|
|
@ -249,26 +254,24 @@ func TestShardScriptsListOnlyRealTests(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// S3: the Bazel-only 34-shard block is not frozen like the legacy 15-shard
|
||||
// block (TestShardScriptsListOnlyRealTests's B1 fix catches outright
|
||||
// corruption, but not a committed block that has drifted from the currently
|
||||
// discovered TestProxiedServer*/TestServerMode* test set, e.g. a test added,
|
||||
// renamed, or removed without anyone running --write). gen_proxied_shard_
|
||||
// manifest.py --check verifies only that the committed block names every
|
||||
// discovered test exactly once -- not that its shard *assignments* match a
|
||||
// fresh LPT pack -- and fails with the exact command to fix it when a name
|
||||
// is missing, stale, or duplicated. It deliberately does NOT fail merely
|
||||
// because proxied_test_durations.json's weights changed and the existing
|
||||
// packing is now suboptimal: two PRs each adding one proxied test would
|
||||
// otherwise force a full repack and conflict on unrelated shard lines (see
|
||||
// --repack below for the explicit opt-in to that). Run --check here so a
|
||||
// block with missing/stale/duplicate names fails go test ./scripts/...
|
||||
// (//scripts:go_test_sources_test under Bazel) instead of only
|
||||
// surfacing as a test silently never running in any shard. The legacy
|
||||
// 15-shard block is deliberately excluded: its header documents that it is
|
||||
// frozen and must not be regenerated (see
|
||||
// .github/scripts/proxied-cmd-test-shards.txt and engdocs/TESTING.md), so a
|
||||
// --check against it would always fail by design.
|
||||
// S3: the manifest's Bazel-only 34-shard block, the file's only remaining
|
||||
// block now that its legacy, frozen 15-shard block (read by PR Risk's and
|
||||
// main.yml's retired fork/push jobs) has been deleted — those jobs no
|
||||
// longer exist (see engdocs/TESTING.md). TestShardScriptsListOnlyRealTests's
|
||||
// B1 fix catches outright corruption, but not a committed block that has
|
||||
// drifted from the currently discovered TestProxiedServer*/TestServerMode*
|
||||
// test set, e.g. a test added, renamed, or removed without anyone running
|
||||
// --write. gen_proxied_shard_manifest.py --check verifies only that the
|
||||
// committed block names every discovered test exactly once -- not that its
|
||||
// shard *assignments* match a fresh LPT pack -- and fails with the exact
|
||||
// command to fix it when a name is missing, stale, or duplicated. It
|
||||
// deliberately does NOT fail merely because proxied_test_durations.json's
|
||||
// weights changed and the existing packing is now suboptimal: two PRs each
|
||||
// adding one proxied test would otherwise force a full repack and conflict
|
||||
// on unrelated shard lines (see --repack below for the explicit opt-in to
|
||||
// that). Run --check here so a block with missing/stale/duplicate names
|
||||
// fails go test ./scripts/... (//scripts:go_test_sources_test under Bazel)
|
||||
// instead of only surfacing as a test silently never running in any shard.
|
||||
func TestProxiedShardManifestGeneratorNotStale(t *testing.T) {
|
||||
python := requireHostTool(t, "python3")
|
||||
root := sourceRepoRoot(t)
|
||||
|
|
@ -279,3 +282,54 @@ func TestProxiedShardManifestGeneratorNotStale(t *testing.T) {
|
|||
t.Errorf("gen_proxied_shard_manifest.py %s --weights=duration --check: %v\n%s", shards, err, out)
|
||||
}
|
||||
}
|
||||
|
||||
// bareWriteReproducesManifest runs `generator --write --manifest COPY` with
|
||||
// no other flags (in particular no --weights) against a scratch copy of
|
||||
// manifest, and asserts the result is byte-identical to the committed file.
|
||||
//
|
||||
// This pins the --weights default against the footgun a reviewer caught on
|
||||
// this PR: the manifest's only remaining block is duration-packed (the
|
||||
// legacy, frozen inits-packed blocks were deleted alongside the PR Risk
|
||||
// fork/push jobs that read them — see engdocs/TESTING.md), so a --weights
|
||||
// default of "inits" made a bare `--write` silently rewrite the live
|
||||
// block's header (and, since costs then came from the wrong model, its
|
||||
// incremental placement of any newly-discovered test) to the inits form
|
||||
// without --check ever catching it. Defaulting --weights to "duration"
|
||||
// makes a bare `--write` idempotent on an up-to-date, duration-packed file.
|
||||
func bareWriteReproducesManifest(t *testing.T, root, generator, manifest string) {
|
||||
t.Helper()
|
||||
python := requireHostTool(t, "python3")
|
||||
want, err := os.ReadFile(filepath.Join(root, manifest))
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", manifest, err)
|
||||
}
|
||||
scratch := filepath.Join(t.TempDir(), filepath.Base(manifest))
|
||||
if err := os.WriteFile(scratch, want, 0o644); err != nil {
|
||||
t.Fatalf("seed scratch copy of %s: %v", manifest, err)
|
||||
}
|
||||
cmd := exec.Command(python, generator, "--write", "--manifest", scratch)
|
||||
cmd.Dir = root
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatalf("%s --write --manifest %s (no --weights): %v\n%s", generator, scratch, err, out)
|
||||
}
|
||||
got, err := os.ReadFile(scratch)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s after --write: %v", scratch, err)
|
||||
}
|
||||
if string(got) != string(want) {
|
||||
t.Errorf("%s --write with no --weights changed %s; a bare --write should reproduce the "+
|
||||
"committed, duration-packed file exactly (--weights defaulting to anything but "+
|
||||
"\"duration\" clobbers its header and repacks with the wrong cost model):\ngot:\n%s\nwant:\n%s",
|
||||
generator, manifest, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProxiedShardManifestBareWriteIsIdempotent(t *testing.T) {
|
||||
bareWriteReproducesManifest(t, sourceRepoRoot(t),
|
||||
"scripts/ci/gen_proxied_shard_manifest.py", ".github/scripts/proxied-cmd-test-shards.txt")
|
||||
}
|
||||
|
||||
func TestEmbeddedCmdShardManifestBareWriteIsIdempotent(t *testing.T) {
|
||||
bareWriteReproducesManifest(t, sourceRepoRoot(t),
|
||||
"scripts/ci/gen_embedded_cmd_shard_manifest.py", ".github/scripts/embedded-cmd-test-shards.txt")
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue