Compare commits

...
Sign in to create a new pull request.

3 commits

Author SHA1 Message Date
Julian Knutsen
c36014de42 ci(bazel): run the historical-upgrade targets in the PR-core lane only
The integration lane runs every untagged target too, and its
check_testcases.py requires Go testcases, which a shell test's test.xml does
not list. A new pr-core-only tag keeps //tests/migration out of
--config=integration (bazelIntegrationExcludedTags adds it), so the corpus
runs once, in bazel-test, against the release-shaped bd_for_tests (no
integration tag). build_source_tag.sh carries the build-tags opt-out: it
builds a historical release with that release's own go.mod, which predates
gms_pure_go.

Refs: ga-96smfk.19

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 01:51:14 +00:00
Julian Knutsen
a3e84c20be ci(bazel): run the historical-upgrade corpus under Bazel; retire migration-test.yml
migration-test.yml ran scripts/migration-test/run.sh --version <release> for
14 reviewed releases in 3 shards on GitHub runners: it downloaded each
release archive, built v0.9.1 from source with Go 1.25 through the module
proxy, installed libicu74 and the dolt 2.1.8 runtime, all over the network.

//tests/migration now has one target per corpus release
(historical_upgrade_<release>_test) plus legacy_bridge_test, all hermetic:
- Published releases come from @bd_releases (the release catalog's
  sha256-verified assets) through a per-target directory
  (HISTORICAL_RELEASE_DIR); lib/binary.sh uses it instead of downloading and
  still checks each binary's self-reported version.
- v0.9.1 is //tests/migration:bd_source_tag, the same `go mod download` +
  `go build -trimpath` with CGO and Go 1.25.0, run offline as a build
  action: the module files come from tests/migration/source_tag_modules.json
  through a file:// GOPROXY tree (tools/bazel/go_module_proxy.bzl), the
  release's own zip is the catalog's source_zip, and go checks every
  dependency against the release's go.sum (SOURCE_TAG_SQLITE_BIN).
- The external Dolt runtime is @dolt_test_runtime_linux_amd64 (dolt.bzl's
  dolt_release gains a version attribute; 2.1.8 is pinned to versions.sh's
  digest), passed as DOLT_BIN to the server-Dolt and wisp-plane releases.
- The workers already provide jq, python3, sqlite3 and libicu74.

The targets are untagged, so bazel-test runs them remotely on every PR,
merge group and push; result caching replaces the path filter. Every
release also passed inside a loopback-only network namespace.
scripts/migration_corpus_bazel_test.go keeps the BUILD lists equal to
versions.sh's corpus, the catalog digests equal to STRICT_RELEASE_SHA256,
the source-tag lock equal to the catalog's source_zip, and the Dolt runtime
pin equal to versions.sh's. migration-test.yml and the policy tests that
pinned its shards, caches, apt step and runner are removed.

Refs: ga-96smfk.19

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 01:51:13 +00:00
Julian Knutsen
bb7e18057a chore(migration-test): extend the release catalog through v1.3.1
The catalog stopped at v1.1.2 (the v1.2 upgrade census). The migration
corpus already upgrades from v1.2.2, and the Bazel targets that replace
migration-test.yml and cross-version-smoke.yml take their pinned release
binaries from the catalog (@bd_releases), so it has to cover the releases
since. Regenerated with `catalog generate` after raising maximumVersion to
v1.3.1: five new stable versions (v1.2.0, v1.2.1, v1.2.2, v1.3.0, v1.3.1;
three with a linux/amd64 asset) and five new proxy prereleases; no existing
entry, drift record or repository-only exclusion changed. The count
constants and the identity digest are updated to match, and the README now
says how to add a release.

Refs: ga-96smfk.19

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 01:51:03 +00:00
29 changed files with 1012 additions and 715 deletions

View file

@ -260,7 +260,9 @@ test:integration --test_env=BEADS_TEST_SKIP=dolt
# keeps every container- or server-backed test skipped, which is what the
# untagged base targets already do. scripts/bazel_policy_test.go derives the
# required exclusions from the PR-core list, so a new lane tag lands here too.
test:integration --test_tag_filters=-requires-docker,-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual
# pr-core-only targets (the historical-release suites: shell tests against
# the release-shaped bd, which the PR-core lane already runs) stay out too.
test:integration --test_tag_filters=-requires-docker,-dolt-server,-dolt-server-proxied,-dolt-server-integration,-dolt-server-cmd,-embedded,-manual,-pr-core-only
test:integration --build_tests_only
test:integration --keep_going
test:integration --test_summary=terse

View file

@ -427,7 +427,7 @@ jobs:
# compile error is not transient. See pr.yml's copy for the full rationale.
#
# 3 attempts x 120s + 2 backoffs x 10s = 380s worst case. 8m leaves ~100s
# of headroom, matching the bounded apt-get steps in migration-test.yml.
# of headroom, matching the bounded apt-get step in release.yml.
- name: Download modules
timeout-minutes: 8
run: |
@ -1082,7 +1082,7 @@ jobs:
# F7c: Blacksmith cannot see GitHub-hosted cache entries, so the advisory
# workflows that moved onto a same-repo-PR Blacksmith runner (Regression's
# PR run, Migration Test Harness, Cross-Version Smoke, Proxied Local) need a
# PR run, Cross-Version Smoke, Proxied Local) need a
# Blacksmith-side writer for their Go build
# cache. This job is push-to-main only (always trusted), so it is the ONLY
# writer: it always restores whatever is available, unconditionally re-runs

View file

@ -1,267 +0,0 @@
name: Migration Test Harness
on:
pull_request:
# "Upgrade-relevant code" allowlist (spec-f7.md §2.4): kept identical
# (modulo each workflow's own file/script entries) across conformance.yml,
# migration-test.yml and cross-version-smoke.yml, and pinned by
# TestAdvisoryWorkflowPathFilters.
paths:
- '**.go'
- '!**_test.go'
- 'go.mod'
- 'go.sum'
- 'Makefile'
- '.buildflags'
# Embedded into the binary (go:embed in internal/storage/schema), so a
# migration-only change is an upgrade-relevant code change.
- 'internal/storage/schema/migrations/**'
- '.github/workflows/migration-test.yml'
- 'scripts/migration-test/**'
- 'scripts/migrate-legacy-to-current.sh'
push:
tags: ['v*']
workflow_dispatch:
permissions:
contents: read
env:
BD_DISABLE_METRICS: "1"
BD_DISABLE_EVENT_FLUSH: "1"
DOLT_TEST_RUNTIME_VERSION: v2.1.8
DOLT_TEST_RUNTIME_SHA256: f66318f08ed66e409fc39363ae0fff8ce6fbf6dba9f5bac632b91527b9632a74
jobs:
historical-upgrades:
name: Historical ${{ matrix.shard }} shard → candidate
# v0.55.4 is dynamically linked against the ICU version shipped by Ubuntu
# 24.04's libicu74 package. blacksmith-*-ubuntu-2404 IS 24.04, so the pin
# holds there. The non-Blacksmith fallback below is pinned to the literal
# `ubuntu-24.04` label (NOT `ubuntu-latest`) for the same reason:
# actions/runner-images#14748 schedules `ubuntu-latest` to move from 24.04
# to 26.04 starting 2026-10-19, and 26.04 drops libicu74 entirely, which
# would break the "Install test dependencies" step below and the v0.55.4
# fixture's ICU ABI even if it didn't. Do not change this fallback to
# `ubuntu-latest`; see sameRepoBlacksmith4vcpuNoble in ci_workflow_test.go.
# F7c: 14 matrix legs (one per historical version) folded into 3 shards.
# Each shard builds the candidate once, installs its own dependencies
# once, then loops over `run.sh --version` for every version it owns,
# collecting failures so one bad version does not hide the rest (see
# "Verify explicit historical upgrades" below).
runs-on: ${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
# src: the one version built from source against Go 1.25, instead
# of installed as a pinned binary.
- shard: src
versions: '["v0.9.1"]'
cache-hash: 453aa2e25bbad328
# pre-dolt: historical versions that predate the separate Dolt
# runtime oracle (no "Install Dolt test runtime" step needed).
- shard: pre-dolt
versions: '["v0.17.0","v0.49.6","v0.50.3","v0.63.3","v1.0.0"]'
cache-hash: 03cccab45291ead3
# dolt-runtime: versions that need the pinned Dolt runtime binary
# as a read-only oracle (the old per-version `contains(...)` list).
- shard: dolt-runtime
versions: '["v0.55.4","v0.56.1","v0.57.0","v0.62.0","v1.0.1","v1.1.0","v1.1.2","v1.2.2"]'
cache-hash: a8579bd64e9a0926
# The v1.1.2 tree re-tagged by the recovery release, so it adds no
# new storage regime — but it is the tag field upgrades actually
# start from, and it was never in this corpus.
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Set up Go
id: setup-go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: 'go.mod'
cache: ${{ runner.environment == 'github-hosted' }}
# Blacksmith runners cannot see GitHub-hosted setup-go cache entries, so
# setup-go's own `cache: true` (used above on ubuntu-24.04/ubuntu-latest)
# is disabled here and this job restores from the Blacksmith-side cache
# main.yml's blacksmith-setup-go-cache job seeds on every push to main.
# Restore-only by design (see B2 in the F7c implementation report): a
# same-repo PR can read this cache but never write to it, so a PR cannot
# poison what main's seeder or other PRs read.
# F7c review fix S5: this job's own key copy must match main.yml's
# seeder key byte-for-byte (TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers),
# which now keys by UTC calendar day instead of github.sha so main
# pushes don't churn a multi-GB cache entry on every commit; this step
# computes that same day value locally so the key expression below
# resolves (steps.* references only resolve within the same job).
- name: Compute cache date
id: cache-date
if: runner.environment == 'self-hosted'
run: echo "today=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT"
- name: Restore Blacksmith setup-go cache
if: runner.environment == 'self-hosted'
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/go/pkg/mod
~/.cache/go-build
key: blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-${{ steps.cache-date.outputs.today }}
restore-keys: |
blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.sum') }}-
blacksmith-sg-v1-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-
- name: Install test dependencies
# Backstop for a hang that escapes the inner `timeout` calls, so it has
# to sit ABOVE what those calls can legitimately consume: 3 update
# attempts x 120s + 2 backoffs x 10s + 180s install = 560s worst case.
# 11m leaves ~100s of headroom and stays under the job's 20m budget, so
# a stuck step fails as a step instead of cancelling the whole job.
timeout-minutes: 11
run: |
updated=""
for attempt in 1 2 3; do
if sudo timeout 120 apt-get \
-o Acquire::Retries=3 \
-o Acquire::http::Timeout=30 \
-o Acquire::https::Timeout=30 \
update; then
updated=yes
break
fi
echo "::warning::apt-get update attempt $attempt failed or timed out"
# No backoff after the final attempt - there is nothing left to wait for.
if [ "$attempt" -lt 3 ]; then
sleep 10
fi
done
# Deliberate best-effort: a stale index still installs these packages
# from the runner image cache far more often than not, and failing the
# job here would turn a transient mirror outage into a red build. The
# warning is what makes the degraded path visible in the job log.
if [ -z "$updated" ]; then
echo "::warning::apt-get update failed after 3 attempts; proceeding with cached index"
fi
sudo timeout 180 apt-get install -y jq libicu74
- name: Install Dolt test runtime
# v0.55.4-v0.62.0 host their fixture on this server; v1.0.1-v1.1.2 use
# the same pinned binary as a read-only oracle over the wisp plane,
# which has no `bd dolt sql` equivalent. This is exactly the
# dolt-runtime shard's version list.
if: matrix.shard == 'dolt-runtime'
run: |
archive="$RUNNER_TEMP/dolt-linux-amd64.tar.gz"
runtime="$RUNNER_TEMP/dolt-runtime"
install_root="$GITHUB_WORKSPACE/.tools/dolt"
curl -fsSL --retry 3 --retry-all-errors --retry-delay 2 \
"https://github.com/dolthub/dolt/releases/download/${DOLT_TEST_RUNTIME_VERSION}/dolt-linux-amd64.tar.gz" \
-o "$archive"
echo "${DOLT_TEST_RUNTIME_SHA256} $archive" | sha256sum -c -
mkdir -p "$runtime" "$install_root/bin"
tar -xzf "$archive" -C "$runtime"
install -m 0755 "$runtime/dolt-linux-amd64/bin/dolt" "$install_root/bin/dolt"
"$install_root/bin/dolt" version
- name: Configure Git
run: |
git config --global user.name "CI Bot"
git config --global user.email "ci@beads.test"
# F7c review fix X1: a bare actions/cache here would auto-save on any
# key miss. On Blacksmith, same-repo PRs are the only ones that ever
# run this job (tag push and workflow_dispatch land on ubuntu-24.04,
# not Blacksmith), so a bare save would make every historical-dolt-*
# entry a same-repo-PR-only write - exactly what B2 forbids elsewhere.
# The checksum verification lib/binary.sh does on every extracted
# archive is not a sufficient mitigation: `tar -P` extraction plus
# `cp -f` following a planted `verified/<ver>/bd` symlink can still
# write outside ~/.cache/beads-regression (e.g. over the checked-out
# workspace or the candidate binary itself) before the checksum check
# ever runs. Restore-only here, same split as smoke-binaries-*/
# regression-baseline-* below: only push/dispatch (always GitHub-hosted
# for this job) may save.
- name: Restore pinned historical release cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/beads-regression
key: historical-dolt-${{ matrix.shard }}-${{ matrix.cache-hash }}-${{ runner.os }}-${{ runner.arch }}
- name: Build candidate
run: make build
- name: Set up Go 1.25 for v0.9.1 source build
if: matrix.shard == 'src'
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version: '1.25.0'
# No Blacksmith restore step for this narrow one-off (src shard
# only, a single version): accepting a cold build here rather than
# adding a second seeded cache keyspace for one matrix leg.
cache: ${{ runner.environment == 'github-hosted' }}
# Each version runs as its own `run.sh` subprocess: the underlying
# historical-dolt-upgrade-test.sh uses `set -euo pipefail` and aborts at
# its first failure within one process, so a single multi-version
# invocation could silently hide every version queued after the first
# failure. Looping subprocess calls here, guarded by `if !`, is what
# lets "all versions run, and the job fails if any failed" (spec-f7.md
# §2.3.7) hold for shards with more than one version.
- name: Verify explicit historical upgrades
env:
CANDIDATE_BIN: ./bd
GIT_CONFIG_NOSYSTEM: "1"
DOLT_BIN: ${{ github.workspace }}/.tools/dolt/bin/dolt
SHARD_VERSIONS: ${{ matrix.versions }}
SHARD: ${{ matrix.shard }}
run: |
set -euo pipefail
mapfile -t versions < <(jq -r '.[]' <<<"$SHARD_VERSIONS")
# Fail closed: `jq -r '.[]'` over an empty array, a non-array
# scalar, or unparsable JSON all produce zero lines rather than a
# non-zero exit under process substitution, so mapfile would
# otherwise silently succeed with an empty $versions and this step
# would report green having verified nothing.
if [ "${#versions[@]}" -eq 0 ]; then
echo "::error::shard $SHARD resolved to zero versions from SHARD_VERSIONS=$SHARD_VERSIONS; failing closed instead of reporting a false pass"
exit 1
fi
failed=0
failed_versions=()
for v in "${versions[@]}"; do
echo "::group::Historical upgrade: $v"
# 8m per version plus a 30s SIGKILL grace: observed real runs take
# well under a minute, so this only trips on a genuine hang, and
# bounds the damage to one version's budget instead of a stuck
# subprocess silently consuming the rest of the job's 20m timeout
# and hiding every version queued after it.
if ! timeout --kill-after=30s 8m ./scripts/migration-test/run.sh --version "$v"; then
echo "::error::historical upgrade failed for $v"
failed=1
failed_versions+=("$v")
fi
echo "::endgroup::"
done
if [ "${#failed_versions[@]}" -gt 0 ]; then
{
echo "### Failed historical upgrades (shard $SHARD)"
printf -- '- %s\n' "${failed_versions[@]}"
} >> "$GITHUB_STEP_SUMMARY"
fi
exit "$failed"
- name: Save pinned historical release cache
# Allow-list push/workflow_dispatch rather than deny-list
# pull_request: a deny-list also admits a hypothetical future
# merge_group event, which this job's runs-on expression already
# resolves to Blacksmith for - exactly the same same-repo-writer
# risk X1 closed for pull_request. Pinned by
# TestBlacksmithReachableAdvisoryJobsNeverSaveACache (checks both the
# pull_request and merge_group trust contexts independently).
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/beads-regression
key: historical-dolt-${{ matrix.shard }}-${{ matrix.cache-hash }}-${{ runner.os }}-${{ runner.arch }}

View file

@ -374,6 +374,7 @@ filegroup(
"//test/conformance:repo_files",
"//test/docsync:repo_files",
"//test/testmainconvention:repo_files",
"//tests/migration:repo_files",
"//tests/regression:repo_files",
"//third_party/cc_toolchain:repo_files",
"//third_party/patches:repo_files",

View file

@ -220,6 +220,15 @@ dolt_release(
platform = "darwin-arm64",
)
# The migration corpus's external Dolt runtime (scripts/migration-test/lib/
# versions.sh DOLT_TEST_RUNTIME_VERSION): an older release than DOLT_VERSION,
# deliberately, as the historical server-Dolt fixtures' host.
dolt_release(
name = "dolt_test_runtime_linux_amd64",
platform = "linux-amd64",
version = "2.1.8",
)
# The tracked release formula, for //scripts:scripts_test and the
# whole-checkout guard //scripts/repochecks:go_install_guidance_test. .beads/
# also holds live bd data and is in .bazelignore, so no main-repo target can
@ -246,3 +255,33 @@ new_local_repository(
bd_releases = use_extension("//tools/bazel:bd_releases.bzl", "bd_releases")
bd_releases.catalog(path = "//scripts:migration-test/release-catalog.json")
use_repo(bd_releases, "bd_releases")
# The source-tag SQLite release (v0.9.1) has no release asset: the migration
# corpus builds it from its module source with the Go release it was
# qualified on, offline (//tests/migration:bd_source_tag). The proxy files
# are pinned in tests/migration/source_tag_modules.json, the toolchain here.
go_module_proxy = use_repo_rule("//tools/bazel:go_module_proxy.bzl", "go_module_proxy")
go_module_proxy(
name = "bd_source_tag_goproxy",
lock = "//tests/migration:source_tag_modules.json",
)
http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "go_1_25_0_linux_amd64",
build_file_content = """exports_files(
["go/bin/go"],
visibility = ["//visibility:public"],
)
filegroup(
name = "sdk",
srcs = glob(["go/**"]),
visibility = ["//visibility:public"],
)
""",
sha256 = "2852af0cb20a13139b3448992e69b868e50ed0f8a1e5940ee1de9e19a123b613",
urls = ["https://go.dev/dl/go1.25.0.linux-amd64.tar.gz"],
)

View file

@ -495,6 +495,7 @@ go_binary(
"//internal/beads:__pkg__",
"//internal/storage/uow:__pkg__",
"//test/conformance:__pkg__",
"//tests/migration:__pkg__",
"//tests/regression:__pkg__",
],
)

View file

@ -201,7 +201,7 @@ Key jobs preserved by display name:
|---|---|---|
| `bazel.yml` `bazel-cmd-dolt` (formerly `regression.yml`) | PR, merge group, push to `main` | `//tests/regression:regression_test` (8 shards, cached until its inputs change) against the catalog-pinned v0.49.6 baseline; local entrypoint stays `make test-regression`. |
| `cross-version-smoke.yml` | Tags, PRs, manual | PRs test latest 5 releases, tags test latest 30, via `scripts/upgrade-smoke-test.sh`. |
| `migration-test.yml` | Tags, PRs touching upgrade-relevant code (advisory, path-filtered), manual | Builds candidate once per shard (3 shards, folded from 14 per-version legs, F7c) and loops `scripts/migration-test/run.sh --version` over each shard's versions; not a required PR/main gate. |
| `bazel.yml` `bazel-test` (formerly `migration-test.yml`) | PR, merge group, push to `main` | `//tests/migration:historical_upgrade_<release>_test`, one target per reviewed release (14, v0.9.1 source-built offline) plus `legacy_bridge_test`, against catalog-pinned release binaries; cached until their inputs change. Local entrypoint stays `scripts/migration-test/run.sh`. |
| `nightly.yml` | Daily schedule, manual | `go test -v -race -tags=integration,gms_pure_go -coverprofile=coverage.out -timeout=30m ./...` with `BEADS_TEST_SKIP=dolt`; checks coverage >= 30%. |
| `nix-build.yml` | Push to `main` paths for Nix or Go module files, manual | `nix build .#default --print-build-logs`; no longer runs on `pull_request` (F7c, spec-f7.md §2.4) since PR Risk's required `test-nix` job (`nix run .#default` plus `nix flake check -L`) is a superset. |
| `deploy-docs.yml` | Push to `main` paths `website/**` or `scripts/generate-llms-full.sh`, manual | `npm ci`, generate `llms-full.txt`, `npm run build`, internal link check, non-blocking external link check, deploy Pages. |

View file

@ -55,7 +55,7 @@ Every build path that produces a binary for users must include `-tags gms_pure_g
| PR CI | `.github/workflows/pr.yml`, `.github/workflows/pr-risk.yml` |
| Main CI test matrix | `.github/workflows/main.yml` (Linux, macOS, Windows) |
| macOS release | `.github/workflows/release.yml` |
| Migration tests | `.github/workflows/migration-test.yml` |
| Migration tests | `//tests/migration/...` (Bazel, `bazel.yml` `bazel-test`) |
| Nightly tests | `.github/workflows/nightly.yml` |
| Cross-version smoke | `.github/workflows/cross-version-smoke.yml` |
| Regression tests | `//tests/regression:regression_test` (Bazel, `bazel.yml` `bazel-cmd-dolt`) |

View file

@ -41,6 +41,7 @@ go_test(
"host_tools_test.go",
"install_dolt_test.go",
"managed_local_lane_test.go",
"migration_corpus_bazel_test.go",
"pr_lanes_bazel_coverage_test.go",
"pr_preflight_test.go",
"pr_risk_bazel_coverage_test.go",
@ -110,6 +111,20 @@ exports_files(
visibility = ["//scripts/migration-test/catalog:__pkg__"],
)
# The historical-upgrade harness (scripts/migration-test) and the public
# SQLite bridge it drives, for //tests/migration's per-release targets.
filegroup(
name = "migration_harness",
srcs = [
"migrate-legacy-to-current.sh",
"migration-test/historical-dolt-upgrade-test.sh",
"migration-test/legacy-bridge-test.sh",
"migration-test/lib/binary.sh",
"migration-test/lib/versions.sh",
],
visibility = ["//tests/migration:__pkg__"],
)
exports_files(
[
"ci/fmt-check.sh",

View file

@ -846,6 +846,12 @@ var allowedBazelTestTags = map[string]string{
// zero tests, which check_testcases.py rejects and equivalence.py can
// only note. gazelle keeps the hand-written tags attribute.
"integration-only": "holds tests only under the integration build tag; excluded from --config=prcore/ci, run by --config=integration",
// For a non-Go sh_test (no Go testcases in its test.xml, which the
// integration lane's check_testcases.py would reject) that runs the
// release-shaped bd_for_tests: the PR-core lane builds bd without the
// integration tag, as releases are, and running it again under the
// integration build would only repeat it against a test-shaped bd.
"pr-core-only": "runs only in --config=prcore/ci; excluded from --config=integration",
}
// bazelPRCoreExcludedTags are the tags whose targets never run in the PR-core
@ -862,10 +868,14 @@ var bazelPRCoreExcludedTags = []string{"requires-docker", "dolt-server", "dolt-s
// other lane's variant stays out of it.
var bazelIntegrationRunsTags = map[string]bool{"integration-only": true}
// bazelPRCoreOnlyTags are the tags the PR-core lane runs and the integration
// lane does not.
var bazelPRCoreOnlyTags = []string{"pr-core-only"}
// bazelIntegrationExcludedTags is bazelPRCoreExcludedTags less the tags the
// integration lane runs.
// integration lane runs, plus bazelPRCoreOnlyTags.
func bazelIntegrationExcludedTags() []string {
var tags []string
tags := append([]string(nil), bazelPRCoreOnlyTags...)
for _, tag := range bazelPRCoreExcludedTags {
if !bazelIntegrationRunsTags[tag] {
tags = append(tags, tag)

View file

@ -1,19 +1,13 @@
package scripts_test
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"testing"
"time"
"gopkg.in/yaml.v3"
)
@ -29,11 +23,11 @@ import (
// advisoryPathFilteredWorkflows are the workflows that share the
// "upgrade-relevant code" allowlist verbatim, save for each one's own
// workflow-file and script entries (spec-f7.md §2.4). conformance.yml was
// the third until its tiers moved to Bazel (embeddeddolt_conformance_*,
// //test/conformance:conformance_test).
// workflow-file and script entries (spec-f7.md §2.4). conformance.yml and
// migration-test.yml were the others until their tiers moved to Bazel
// (embeddeddolt_conformance_*, //test/conformance:conformance_test,
// //tests/migration).
var advisoryPathFilteredWorkflows = []string{
"migration-test.yml",
"cross-version-smoke.yml",
}
@ -57,14 +51,6 @@ var advisoryPathFilterBase = []string{
// advisoryPathFilterOwnEntries is each workflow's own file/script additions,
// appended after advisoryPathFilterBase.
var advisoryPathFilterOwnEntries = map[string][]string{
"migration-test.yml": {
".github/workflows/migration-test.yml",
"scripts/migration-test/**",
// F7c review fix (S3): the migration harness invokes this script
// directly, but it lives at scripts/ root, not under
// scripts/migration-test/, so it was not covered by the filter.
"scripts/migrate-legacy-to-current.sh",
},
"cross-version-smoke.yml": {
".github/workflows/cross-version-smoke.yml",
"scripts/upgrade-smoke-test.sh",
@ -130,7 +116,7 @@ func TestAdvisoryWorkflowPathFiltersAreIdentical(t *testing.T) {
// TestAdvisoryWorkflowPathFiltersCoverOwnInputs pins that each workflow also
// allowlists its own workflow file and the scripts/fixtures it actually
// exercises, so an edit to e.g. scripts/migration-test/** is never silently
// exercises, so an edit to e.g. scripts/upgrade-smoke-test.sh is never silently
// skipped by the filter that was added to cut unrelated-PR load.
func TestAdvisoryWorkflowPathFiltersCoverOwnInputs(t *testing.T) {
for file, want := range advisoryPathFilterOwnEntries {
@ -199,357 +185,6 @@ func TestNixBuildDropsPullRequestTriggerNotPushOrDispatch(t *testing.T) {
}
}
// --- Migration Test Harness: 14 -> 3 shards, no version dropped -----------
// migrationHarnessOriginalVersions is the pre-F7c 14-leg matrix's version
// list, captured verbatim so a shard rebalance can be checked against it
// without re-deriving it from the (now folded) workflow file.
var migrationHarnessOriginalVersions = []string{
"v0.9.1", "v0.17.0", "v0.49.6", "v0.50.3", "v0.55.4", "v0.56.1",
"v0.57.0", "v0.62.0", "v0.63.3", "v1.0.0", "v1.0.1", "v1.1.0",
"v1.1.2", "v1.2.2",
}
// migrationHarnessDoltRuntimeVersions is the old per-version
// `contains(fromJSON('[...]'), matrix.version)` list that gated the "Install
// Dolt test runtime" step. It must become exactly the dolt-runtime shard.
var migrationHarnessDoltRuntimeVersions = []string{
"v0.55.4", "v0.56.1", "v0.57.0", "v0.62.0", "v1.0.1", "v1.1.0", "v1.1.2", "v1.2.2",
}
func migrationHarnessShards(t *testing.T) map[string][]string {
t.Helper()
workflow := readCIWorkflow(t, "migration-test.yml")
job := workflow.job(t, "historical-upgrades")
shards := map[string][]string{}
for _, leg := range job.Strategy.Matrix.Include {
shardAny, ok := leg.Extra["shard"]
if !ok {
t.Fatalf("migration-test.yml matrix leg %+v has no shard field", leg.Extra)
}
shard, ok := shardAny.(string)
if !ok {
t.Fatalf("migration-test.yml matrix leg shard = %#v, not a string", shardAny)
}
versionsAny, ok := leg.Extra["versions"]
if !ok {
t.Fatalf("migration-test.yml shard %q has no versions field", shard)
}
versionsJSON, ok := versionsAny.(string)
if !ok {
t.Fatalf("migration-test.yml shard %q versions = %#v, not a string", shard, versionsAny)
}
var versions []string
if err := json.Unmarshal([]byte(versionsJSON), &versions); err != nil {
t.Fatalf("migration-test.yml shard %q versions %q does not parse as a JSON string array: %v", shard, versionsJSON, err)
}
if _, dup := shards[shard]; dup {
t.Fatalf("migration-test.yml declares shard %q more than once", shard)
}
shards[shard] = versions
}
return shards
}
// TestMigrationHarnessShardsCoverAllHistoricalVersions is the equivalence
// check the fold requires: the union of the 3 shards' version lists must be
// exactly the old 14-version set, with no duplicates and nothing dropped.
func TestMigrationHarnessShardsCoverAllHistoricalVersions(t *testing.T) {
shards := migrationHarnessShards(t)
wantShardNames := []string{"src", "pre-dolt", "dolt-runtime"}
for _, name := range wantShardNames {
if _, ok := shards[name]; !ok {
t.Errorf("migration-test.yml is missing shard %q", name)
}
}
if len(shards) != len(wantShardNames) {
t.Errorf("migration-test.yml has shards %v, want exactly %v", mapKeys(shards), wantShardNames)
}
seen := map[string]string{} // version -> owning shard
var union []string
for shard, versions := range shards {
for _, v := range versions {
if owner, dup := seen[v]; dup {
t.Errorf("version %s is claimed by both shard %q and shard %q", v, owner, shard)
continue
}
seen[v] = shard
union = append(union, v)
}
}
sort.Strings(union)
want := append([]string(nil), migrationHarnessOriginalVersions...)
sort.Strings(want)
if !equalStrings(union, want) {
t.Fatalf("shard union = %v, want exactly the old 14-version set %v", union, want)
}
if !equalStrings(sortedCopy(shards["dolt-runtime"]), sortedCopy(migrationHarnessDoltRuntimeVersions)) {
t.Errorf("dolt-runtime shard = %v, want exactly the old Dolt-runtime contains() list %v", shards["dolt-runtime"], migrationHarnessDoltRuntimeVersions)
}
if !equalStrings(shards["src"], []string{"v0.9.1"}) {
t.Errorf("src shard = %v, want exactly [v0.9.1]", shards["src"])
}
}
// TestMigrationHarnessLoopExecutesBehaviorally runs the actual "Verify
// explicit historical upgrades" bash against a stub run.sh, proving the
// loop's real behavior instead of its source text (F7c review fix S1/S4/S5,
// closes mutations M1 `exit 0`, M2 `break`, M14 `| head -1`, and pins the
// fail-closed and per-version-timeout fixes the reviewer required):
// - every version in the shard is attempted even after an earlier one
// fails (a short-circuiting `exit 0`/`break` would hide the rest);
// - the step's own exit code reflects any failure;
// - failures are both ::error:: annotated and written to
// $GITHUB_STEP_SUMMARY;
// - an empty or unparsable SHARD_VERSIONS fails closed instead of
// reporting a false pass, without ever invoking run.sh;
// - a hung version is bounded by `timeout`, and the loop still reaches the
// version queued after it.
//
// A bare `./scripts/migration-test/run.sh --version "$HISTORICAL_VERSION"`
// (the pre-fold single-version invocation) must also be gone: if it came
// back, the shard would silently only test one version again.
func TestMigrationHarnessLoopExecutesBehaviorally(t *testing.T) {
requireHostTool(t, "bash")
requireHostTool(t, "jq")
requireHostTool(t, "timeout")
job := readCIWorkflow(t, "migration-test.yml").job(t, "historical-upgrades")
step := job.step(t, "Verify explicit historical upgrades")
if strings.Contains(step.Run, "$HISTORICAL_VERSION") {
t.Errorf("migration-test.yml's Verify step still references the old single-version $HISTORICAL_VERSION env var")
}
writeHarness := func(t *testing.T, runSh string) (dir, summaryFile string) {
t.Helper()
dir = t.TempDir()
scriptDir := filepath.Join(dir, "scripts", "migration-test")
if err := os.MkdirAll(scriptDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(scriptDir, "run.sh"), []byte(runSh), 0o755); err != nil {
t.Fatal(err)
}
summaryFile = filepath.Join(dir, "summary.md")
if err := os.WriteFile(summaryFile, nil, 0o644); err != nil {
t.Fatal(err)
}
return dir, summaryFile
}
baseEnv := func(shardVersions, summaryFile string) []string {
return append(os.Environ(),
"CANDIDATE_BIN=./bd",
"GIT_CONFIG_NOSYSTEM=1",
"DOLT_BIN=/nonexistent/dolt",
"SHARD_VERSIONS="+shardVersions,
"SHARD=test-shard",
"GITHUB_STEP_SUMMARY="+summaryFile,
)
}
run := func(t *testing.T, script, shardVersions, runSh string) (exitCode int, out, summary string) {
t.Helper()
dir, summaryFile := writeHarness(t, runSh)
cmd := exec.Command("bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", script)
cmd.Dir = dir
cmd.Env = baseEnv(shardVersions, summaryFile)
outBytes, err := cmd.CombinedOutput()
code := 0
if err != nil {
exitErr, ok := err.(*exec.ExitError)
if !ok {
t.Fatalf("run loop: %v\noutput:\n%s", err, outBytes)
}
code = exitErr.ExitCode()
}
summaryBytes, readErr := os.ReadFile(summaryFile)
if readErr != nil {
t.Fatal(readErr)
}
return code, string(outBytes), string(summaryBytes)
}
t.Run("every version runs even after an early failure, and the step fails", func(t *testing.T) {
const stub = `#!/usr/bin/env bash
set -euo pipefail
v=""
while [ $# -gt 0 ]; do
if [ "$1" = "--version" ]; then v="$2"; fi
shift
done
echo "stub-ran:$v"
[ "$v" = "v2" ] && exit 1
exit 0
`
code, out, summary := run(t, step.Run, `["v1","v2","v3"]`, stub)
for _, v := range []string{"v1", "v2", "v3"} {
if !strings.Contains(out, "stub-ran:"+v) {
t.Errorf("version %s never ran; output:\n%s", v, out)
}
}
if code != 1 {
t.Errorf("exit code = %d, want 1 (v2 failed)", code)
}
if !strings.Contains(out, "::error::historical upgrade failed for v2") {
t.Errorf("missing ::error:: for v2; output:\n%s", out)
}
if strings.Contains(out, "::error::historical upgrade failed for v1") || strings.Contains(out, "::error::historical upgrade failed for v3") {
t.Errorf("v1/v3 must not be reported as failed; output:\n%s", out)
}
if !strings.Contains(summary, "v2") {
t.Errorf("GITHUB_STEP_SUMMARY does not mention the failed version v2:\n%s", summary)
}
if strings.Contains(summary, "v1") || strings.Contains(summary, "v3") {
t.Errorf("GITHUB_STEP_SUMMARY must only list failed versions:\n%s", summary)
}
})
t.Run("all versions pass, step exits 0", func(t *testing.T) {
const stub = "#!/usr/bin/env bash\nexit 0\n"
code, out, summary := run(t, step.Run, `["v1","v2"]`, stub)
if code != 0 {
t.Errorf("exit code = %d, want 0, output:\n%s", code, out)
}
if strings.TrimSpace(summary) != "" {
t.Errorf("GITHUB_STEP_SUMMARY should be untouched on an all-pass run, got:\n%s", summary)
}
})
for _, badInput := range []string{`[]`, `null`, `not-json`, `"a string, not an array"`} {
t.Run("fails closed on "+badInput, func(t *testing.T) {
const stub = "#!/usr/bin/env bash\necho ran >&2\nexit 1\n"
code, out, _ := run(t, step.Run, badInput, stub)
if code != 1 {
t.Errorf("SHARD_VERSIONS=%s: exit code = %d, want 1 (fail closed)", badInput, code)
}
if !strings.Contains(out, "::error::shard test-shard resolved to zero versions") {
t.Errorf("SHARD_VERSIONS=%s: missing the fail-closed ::error::; output:\n%s", badInput, out)
}
if strings.Contains(out, "ran") {
t.Errorf("SHARD_VERSIONS=%s: run.sh must never be invoked when the version list fails closed; output:\n%s", badInput, out)
}
})
}
t.Run("a hung version is bounded by timeout and does not hide the next version", func(t *testing.T) {
// Exercise the REAL timeout/kill-after wiring, just at durations a
// unit test can afford: substitute the pinned 8m/30s for 2s/1s. If a
// mutation removed the `timeout` wrapper entirely, this substitution
// is a no-op and the stub's 10s sleep would make this subtest's own
// assertions fail well before a developer-visible hang, bounded by
// the 20s context below.
fastStep := strings.NewReplacer("8m", "2s", "30s", "1s").Replace(step.Run)
const stub = `#!/usr/bin/env bash
v=""
while [ $# -gt 0 ]; do
if [ "$1" = "--version" ]; then v="$2"; fi
shift
done
if [ "$v" = "hangs" ]; then
sleep 10
exit 0
fi
echo "stub-ran:$v"
exit 0
`
dir, summaryFile := writeHarness(t, stub)
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "bash", "--noprofile", "--norc", "-eo", "pipefail", "-c", fastStep)
cmd.Dir = dir
cmd.Env = baseEnv(`["hangs","after"]`, summaryFile)
start := time.Now()
out, err := cmd.CombinedOutput()
elapsed := time.Since(start)
if ctx.Err() == context.DeadlineExceeded {
t.Fatalf("loop did not return within 20s; the timeout wrapper did not bound the hang. output:\n%s", out)
}
if elapsed > 8*time.Second {
t.Errorf("loop took %s to process a version meant to be killed after ~3s; output:\n%s", elapsed, out)
}
if _, ok := err.(*exec.ExitError); !ok && err != nil {
t.Fatalf("run loop: %v\noutput:\n%s", err, out)
}
if !strings.Contains(string(out), "stub-ran:after") {
t.Errorf("the version after the hang never ran; output:\n%s", out)
}
if !strings.Contains(string(out), "::error::historical upgrade failed for hangs") {
t.Errorf("the killed version was not reported as failed; output:\n%s", out)
}
})
}
// TestMigrationHarnessDoltRuntimeStepGatedByShard pins that the "Install Dolt
// test runtime" step's gate is exactly `matrix.shard == 'dolt-runtime'`, not
// some other formulation that might accidentally also match (or exclude) a
// different shard (F7c review fix S1, closes mutation M3).
func TestMigrationHarnessDoltRuntimeStepGatedByShard(t *testing.T) {
job := readCIWorkflow(t, "migration-test.yml").job(t, "historical-upgrades")
step := job.step(t, "Install Dolt test runtime")
const want = "matrix.shard == 'dolt-runtime'"
if step.If != want {
t.Errorf("migration-test.yml's Install Dolt test runtime if = %q, want %q", step.If, want)
}
}
// TestMigrationHarnessCacheHashMatchesDerivation pins that each shard's
// cache-hash is still sha256(join(",", <shard's per-version cache-identity
// strings>))[:16], computed from the pre-fold 14-entry identity table that
// existed before the 14 -> 3 shard fold (git show
// 362eecc6bf:.github/workflows/migration-test.yml), so a future shard
// rebalance or version-list edit can't silently leave a stale cache key
// behind (F7c review fix S1, closes mutation M5).
func TestMigrationHarnessCacheHashMatchesDerivation(t *testing.T) {
identities := map[string]string{
"v0.9.1": "v0.9.1-source-e3c8554fa2c3e4b9caf7e296e9c8abbe24211a72-go1.25.0-cgo1",
"v0.17.0": "v0.17.0-d4d08617a324c85b45c9628bc519d659a9ff9c7c37da67aa48727e0af7f19a75",
"v0.49.6": "v0.49.6-8546dc9a47e11dc31ac2bc9a0224a9c690975e91850932cbb62623053fbb7db8",
"v0.50.3": "v0.50.3-e94b09e0b6a9324bbc0e81ea36bccaaa42172a926bfedfb389e9a26dedb63184",
"v0.55.4": "v0.55.4-e0fa25456dd82890230eef17653448a0bf995104c78864be91c5ed84426a5f49",
"v0.56.1": "v0.56.1-4f9f6cc44465a11613ff529009901eaaf841c6b1f91c15e002b0ecda2015a15c",
"v0.57.0": "v0.57.0-f8629d5627bed7d25f06f92334addc171d679f9aed9d08c5d42a9684205dc04b",
"v0.62.0": "v0.62.0-4cca7265b22e5c3ca8d62ab0b9752bec31f68b7f5fa636282a4c7e5454c35535",
"v0.63.3": "v0.63.3-5f4efd2e010209b3f381dbcd783b2a3a652f50ea72f40ef04c8ba434d408bf9e",
"v1.0.0": "v1.0.0-7057db1e92428fcf5c08d5dc6b07ead57e588b262cba78b9a26893d55bd29fdb",
"v1.0.1": "v1.0.1-1d2364d5d7083a4634a9e734ca87822fb79c2b6625988f9f791e3376313b1b77",
"v1.1.0": "v1.1.0-b0f3dd607c3fb989ee08d0a6854fba80d0402971eb108f9af6170bc14d491a34",
"v1.1.2": "v1.1.2-a72d71ed374955dc9f83a0f90b54bd7b6a0016709dd1676ae2e368651ed401c2",
"v1.2.2": "v1.2.2-8140098a51d3b81d5548d1c5e6db1a2d9930e5d141efe2a4bff7d079c4d321e8",
}
job := readCIWorkflow(t, "migration-test.yml").job(t, "historical-upgrades")
for _, leg := range job.Strategy.Matrix.Include {
shard, _ := leg.Extra["shard"].(string)
versionsJSON, _ := leg.Extra["versions"].(string)
var versions []string
if err := json.Unmarshal([]byte(versionsJSON), &versions); err != nil {
t.Fatalf("shard %q versions %q: %v", shard, versionsJSON, err)
}
var parts []string
for _, v := range versions {
id, ok := identities[v]
if !ok {
t.Fatalf("shard %q version %q has no known pre-fold cache-identity; update the table in this test", shard, v)
}
parts = append(parts, id)
}
sum := sha256.Sum256([]byte(strings.Join(parts, ",")))
want := hex.EncodeToString(sum[:])[:16]
gotAny, ok := leg.Extra["cache-hash"]
if !ok {
t.Fatalf("shard %q has no cache-hash field", shard)
}
if got := fmt.Sprint(gotAny); got != want {
t.Errorf("shard %q cache-hash = %q, want sha256(join(\",\", identities))[:16] = %q", shard, got, want)
}
}
}
func mapKeys[K comparable, V any](m map[K]V) []K {
keys := make([]K, 0, len(m))
for k := range m {
@ -694,7 +329,6 @@ func TestCrossVersionSmokeChunksEveryResolvedVersion(t *testing.T) {
// blacksmithAdvisoryWorkflows are every workflow file F7c moved a job onto a
// same-repo-PR (or push-only, for main.yml's seed) Blacksmith runner.
var blacksmithAdvisoryWorkflows = []string{
"migration-test.yml",
"cross-version-smoke.yml",
"docs-mintlify.yml",
"main.yml",
@ -738,18 +372,13 @@ func containsSecretRef(s string) bool {
// advisoryBlacksmithRunnerJobs pins which F7c advisory job runs on which
// same-repo Blacksmith expression, and what label each falls back to when
// the same-repo/merge_group condition is false. migration-test.yml's
// historical-upgrades job is the one case with a non-ubuntu-latest fallback
// (sameRepoBlacksmith4vcpuNoble, F7c review fix B1): it must still resolve to
// the literal ubuntu-24.04 label, not the usual ubuntu-latest, for forks/
// Dependabot/push.
// the same-repo/merge_group condition is false.
var advisoryBlacksmithRunnerJobs = []struct {
file string
job string
blacksmithLabel string
fallback string
}{
{"migration-test.yml", "historical-upgrades", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-24.04"},
{"cross-version-smoke.yml", "smoke", "blacksmith-4vcpu-ubuntu-2404", "ubuntu-latest"},
{"cross-version-smoke.yml", "versions", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"},
{"docs-mintlify.yml", "broken-links", "blacksmith-2vcpu-ubuntu-2404", "ubuntu-latest"},
@ -818,16 +447,15 @@ func TestF7cAdvisorySameRepoBlacksmithExpressionSemantics(t *testing.T) {
// self-defined `blacksmith-sg-v1-` setup-go cache main.yml's
// blacksmith-setup-go-cache job seeds (B2, F7c implementation report).
var blacksmithSetupGoCacheConsumers = map[string][]string{
"migration-test.yml": {"historical-upgrades"},
"cross-version-smoke.yml": {"smoke"},
}
// blacksmithSetupGoCacheKeyNamespace is the self-defined cache key prefix
// (not setup-go's own implicit key) that the seeder and every consumer share,
// so the "no save in a consumer job" checks below can scope to exactly this
// cache without also flagging an unrelated, legitimately-caching step (e.g.
// migration-test.yml's historical-dolt-* cache, which is a distinct
// restore/save pair of its own - see advisoryBinaryCaches below).
// cache without also flagging an unrelated, legitimately-caching step (a
// binary cache with a distinct restore/save pair of its own - see
// advisoryBinaryCaches below).
const blacksmithSetupGoCacheKeyNamespace = "blacksmith-sg-v1-"
// TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers pins that main.yml's
@ -1008,11 +636,10 @@ func TestMainWorkflowHasNoSameRepoPRReachableTrigger(t *testing.T) {
// main.yml's seeder may ever write it, so a same-repo PR run can read the
// cache but never poison what another PR or main's seeder reads back. The
// "no save" check is scoped to the blacksmith-sg-v1- key namespace
// specifically (not "no actions/cache/save in this job at all"), since
// migration-test.yml's historical-upgrades job keeps its own historical-dolt-*
// cache in a different namespace - governed by its own restore/save-gated
// pair below (advisoryBinaryCaches / F7c review fix X1), not an exemption
// from this one.
// specifically (not "no actions/cache/save in this job at all"), since a
// binary cache in a different namespace is governed by its own
// restore/save-gated pair below (advisoryBinaryCaches / F7c review fix X1),
// not an exemption from this one.
func TestAdvisoryBlacksmithConsumersAreCacheRestoreOnly(t *testing.T) {
for file, jobNames := range blacksmithSetupGoCacheConsumers {
workflow := readCIWorkflow(t, file)
@ -1157,7 +784,6 @@ var advisoryBinaryCaches = []struct {
wantSaveIf string
}{
{"cross-version-smoke.yml", "smoke", "Restore previous release binaries cache", "Save previous release binaries cache", "smoke-binaries-", "github.event_name == 'push' || github.event_name == 'workflow_dispatch'"},
{"migration-test.yml", "historical-upgrades", "Restore pinned historical release cache", "Save pinned historical release cache", "historical-dolt-", "github.event_name == 'push' || github.event_name == 'workflow_dispatch'"},
}
// TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated pins the restore/save

View file

@ -48,17 +48,6 @@ func (b aptStepBudget) loopHeader() string {
func aptStepBudgets() []aptStepBudget {
return []aptStepBudget{
{
workflow: "migration-test.yml",
job: "historical-upgrades",
step: "Install test dependencies",
updateAttempts: 3,
updateTimeout: 120,
backoff: 10,
installTimeout: 180,
installPkgs: "jq libicu74",
capMinutes: 11,
},
{
workflow: "release.yml",
job: "goreleaser",

View file

@ -2286,10 +2286,10 @@ type ciWorkflowMatrixInclude struct {
// F7b review fix (S1): no typed `shard` field here. pr-preflight-
// platforms' Windows shard split (the one thing in this file that ever
// used it) was reverted as not measurably helpful, and a typed field
// here is shared by every workflow's `strategy.matrix.include` tuples —
// migration-test.yml's shard: src/pre-dolt/dolt-runtime entries use a
// STRING, not an int, so a mismatched type here would break parsing of
// an unrelated workflow. Untyped matrix keys fall through to Extra.
// here is shared by every workflow's `strategy.matrix.include` tuples,
// whose keys may be any type, so a mismatched type here would break
// parsing of an unrelated workflow. Untyped matrix keys fall through to
// Extra.
Extra map[string]any `yaml:",inline"`
}
@ -2709,22 +2709,9 @@ const mainWindowsTestBinariesCacheRunsOn = "${{ matrix.runner == 'blacksmith' &&
// F7c: used to define its own sameRepoBlacksmith4vcpu here (same expression
// as sameRepoBlacksmith2vcpu with the 4 vCPU label, for advisory jobs that
// compile Go: migration-test.yml,
// cross-version-smoke.yml). F7a independently
// defined the same const; both are now served by the single shared
// sameRepoBlacksmith4vcpu in ci_blacksmith_runner_test.go. migration-test.yml's
// historical-upgrades job uses sameRepoBlacksmith4vcpuNoble instead (below).
// F7c review fix (B1): migration-test.yml's historical-upgrades job cannot
// use the plain ubuntu-latest fallback that sameRepoBlacksmith4vcpu uses,
// because the v0.55.4 fixture is dynamically linked against the ICU ABI
// shipped by Ubuntu 24.04's libicu74 package, and GitHub's ubuntu-latest
// label is scheduled to move from 24.04 to 26.04 starting 2026-10-19 (see
// actions/runner-images#14748); 26.04 drops libicu74 entirely. The fallback
// here is pinned to the literal `ubuntu-24.04` label instead so the
// non-Blacksmith path keeps working after that migration regardless of when
// it lands. Do not fold this back into sameRepoBlacksmith4vcpu.
const sameRepoBlacksmith4vcpuNoble = "${{ (github.event_name == 'merge_group' || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')) && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}"
// compile Go: cross-version-smoke.yml). F7a independently defined the same
// const; both are now served by the single shared sameRepoBlacksmith4vcpu in
// ci_blacksmith_runner_test.go.
// F7b: venue-matrix runs-on for main.yml jobs that seed a Blacksmith GOCACHE
// alongside their GitHub-hosted seed ("venue: [blacksmith, github]"):
@ -3051,7 +3038,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) {
// is cheap enough to size at 2 vCPU per spec-f7.md §2.2) and moves
// to Blacksmith for same-repo PRs/merge_group only; forks,
// Dependabot and push stay on ubuntu-latest.
"migration-test.yml": {"historical-upgrades": sameRepoBlacksmith4vcpuNoble},
"cross-version-smoke.yml": {"smoke": sameRepoBlacksmith4vcpu, "versions": sameRepoBlacksmith2vcpu},
"docs-mintlify.yml": {"broken-links": sameRepoBlacksmith2vcpu},
// main.yml's seeder job (B2, F7c implementation report) is the one
@ -3117,7 +3103,6 @@ func TestSameRepoBlacksmithRunners(t *testing.T) {
// Blacksmith label beyond the jobs listed in `want` above.
for _, file := range []string{
"pr.yml", "pr-risk.yml", "main.yml",
"migration-test.yml",
"cross-version-smoke.yml", "docs-mintlify.yml",
} {
workflow := readCIWorkflow(t, file)

View file

@ -1,7 +1,8 @@
# Historical release catalog
`../release-catalog.json` pins the exact historical release universe used by the
v1.2 upgrade census: 122 stable module versions authenticated through the Go
`../release-catalog.json` pins the exact historical release universe (v0.9.1
through v1.3.1; the v1.2 upgrade census plus the releases since): 127 stable
module versions authenticated through the Go
proxy and SumDB, each with module / `go.mod` / source-zip / proxy-origin
provenance, explicit proxy prerelease and repository-only tag exclusions, and
the current repository tag drift.
@ -18,13 +19,22 @@ artifact (`usage: catalog generate|validate <manifest.json>`):
release APIs (network plus `go`/`gh` required). This is a manual maintenance
path, not a CI step.
## Adding a release
Raise `maximumVersion` in `main.go`, run `generate`, and update the count
constants and `expectedCatalogSHA256` the validator then reports. Every
release with a linux/amd64 asset digest automatically becomes a pinned Bazel
repository (`@bd_releases//:<version>`, `tools/bazel/bd_releases.bzl`).
## Status: self-validated reference anchor
Today the catalog is a reviewed-identity anchor. Its only automated enforcement
is `TestCheckedCatalogIsCanonicalAndComplete` in `main_test.go`, which runs
under `go test ./...` and holds the checked JSON to the pinned digest and its
invariants. Nothing in the migration-test harness (`run.sh`,
`historical-dolt-upgrade-test.sh`, `legacy-bridge-test.sh`) consumes it yet.
invariants. Bazel consumes it: `tools/bazel/bd_releases.bzl` declares one
sha256-verified repository per release asset it records, which the
regression, migration (`tests/migration`) and cross-version targets run
instead of downloading release binaries.
The intended consumer is the upgrade-coverage work stacked on top of this
change: runtime schema observations and family/path qualification will drive

View file

@ -23,21 +23,22 @@ import (
const (
modulePath = "github.com/steveyegge/beads"
minimumVersion = "v0.9.1"
maximumVersion = "v1.1.2"
maximumVersion = "v1.3.1"
proxyURL = "https://proxy.golang.org"
githubRepository = "gastownhall/beads"
expectedVersions = 122
expectedVersions = 127
expectedRepoOnly = 49
expectedRepoPre = 3
expectedReleases = 92
expectedLinuxAssets = 89
expectedProxyPre = 7
expectedReleases = 95
expectedLinuxAssets = 92
expectedTagDrift = 13
expectedDriftAssets = 11
// This pins the complete reviewed catalog identity. A generator run may
// expose upstream drift, but offline validation rejects even well-formed
// substitutions until this digest is deliberately updated.
expectedCatalogSHA256 = "298dd489a6274d80ac42e1fb14c993444159f3193a290f8da37afcb3e2eaf10d"
expectedCatalogSHA256 = "edd01ba3352ec38260a87637d8ec4abb6689df7b2b57f100b18b06842024f858"
)
var (
@ -511,7 +512,7 @@ func validateSourceRelations(versions []Entry, driftSet map[string]bool) error {
// validateExclusions checks the proxy-prerelease and repository-only exclusion
// lists against the included version set.
func validateExclusions(ex Exclusions, seen map[string]bool) error {
if err := validateExcluded("proxy prereleases", ex.ProxyPrereleases, 2, isPrerelease, seen); err != nil {
if err := validateExcluded("proxy prereleases", ex.ProxyPrereleases, expectedProxyPre, isPrerelease, seen); err != nil {
return err
}
if err := validateExcluded("repository-only stable tags", ex.RepositoryOnlyStable, expectedRepoOnly, isStable, seen); err != nil {

View file

@ -13,9 +13,9 @@ import (
func TestCheckedCatalogIsCanonicalAndComplete(t *testing.T) {
catalog := readCheckedCatalog(t)
if len(catalog.Versions) != 122 || len(catalog.Exclusions.RepositoryOnlyStable) != 49 ||
if len(catalog.Versions) != 127 || len(catalog.Exclusions.RepositoryOnlyStable) != 49 ||
len(catalog.Exclusions.RepositoryOnlyPrereleases) != 3 {
t.Fatalf("catalog counts = %d/%d/%d, want 122/49/3",
t.Fatalf("catalog counts = %d/%d/%d, want 127/49/3",
len(catalog.Versions), len(catalog.Exclusions.RepositoryOnlyStable),
len(catalog.Exclusions.RepositoryOnlyPrereleases))
}
@ -64,7 +64,7 @@ func TestCheckedCatalogRejectsWellFormedIdentitySubstitution(t *testing.T) {
func TestClassifyVersionsUsesProxyAsStableUniverse(t *testing.T) {
stable, excluded, err := classifyVersions(
[]string{"v1.1.2", "v1.1.0-rc.2", "v0.56.0", "v1.1.0-rc.1", "v0.9.1", "v1.2.0"},
[]string{"v1.1.2", "v1.1.0-rc.2", "v0.56.0", "v1.1.0-rc.1", "v0.9.1", "v1.4.0"},
[]string{"v0.9.1", "v0.57.12", "v0.58.8-nosqlite", "v1.1.0-rc.1", "v1.1.2", "2026.218.0"},
)
if err != nil {

View file

@ -126,9 +126,28 @@ verify_release_binary_version() (
fi
)
# HISTORICAL_RELEASE_DIR (set by the Bazel targets in tests/migration) holds
# the pinned release binaries as <dir>/<version>, each already verified
# against the release catalog's sha256 when Bazel fetched it, so nothing is
# downloaded. A version the directory lacks is an error, never a download.
prepinned_release_binary() {
local version="$1" binary
binary="$HISTORICAL_RELEASE_DIR/$version"
if [ ! -f "$binary" ]; then
echo "ERROR: HISTORICAL_RELEASE_DIR has no pinned $version binary (declare it in the target's releases)" >&2
return 1
fi
verify_release_binary_version "$version" "$binary" || return 1
resolve_existing_path "$binary"
}
download_verified_release_binary() {
local version="$1"
local asset expected release_dir archive binary
if [ -n "${HISTORICAL_RELEASE_DIR:-}" ]; then
prepinned_release_binary "$version"
return
fi
asset=$(strict_release_asset "$version" "$OS" "$ARCH") || {
echo "ERROR: no strict release manifest for $version ($OS/$ARCH)" >&2
return 1
@ -180,6 +199,13 @@ download_verified_release_binary() {
# v0.9.1 has no release asset. Its historical CLI reports 0.9.0 (dev), so
# qualification is the module checksum and VCS origin, never self-reporting.
build_verified_v091_source_binary() (
# SOURCE_TAG_SQLITE_BIN (set by //tests/migration's v0.9.1 target) is
# this same build, done offline by Bazel from the sha256-pinned module
# files with the same toolchain (//tests/migration:bd_source_tag).
if [ -n "${SOURCE_TAG_SQLITE_BIN:-}" ]; then
resolve_existing_path "$SOURCE_TAG_SQLITE_BIN"
return
fi
local scratch module_json source_dir build_dir binary temporary toolchain_root toolchain_go toolchain
scratch=$(mktemp -d) || return 1
# Canonicalize immediately: macOS's default TMPDIR (/var/folders/...) is

View file

@ -49,8 +49,8 @@ declare -ar EMBEDDED_DOLT_VERSIONS=(
# ignored-0011. Both regimes are replayed over populated rows by the candidate's
# main 0054-0066 + ignored 0012-0025 chain. v1.2.2 is the v1.1.2 tree re-tagged
# by the recovery release, so it adds no new regime — it is here because it is
# the tag people in the field actually upgrade from. Also gates the external
# Dolt oracle download in migration-test.yml.
# the tag people in the field actually upgrade from. Also selects the
# external Dolt oracle (DOLT_BIN) in tests/migration's targets.
declare -ar WISP_PLANE_VERSIONS=(
"v1.0.1"
"v1.1.0"

View file

@ -2400,6 +2400,98 @@
"digest": "sha256:a72d71ed374955dc9f83a0f90b54bd7b6a0016709dd1676ae2e368651ed401c2"
}
}
},
{
"version": "v1.2.0",
"sum": "h1:ybRxPVim24oaMdC19xobwBaoFMkgt7HrYZVOGR0Sts4=",
"go_mod_sum": "h1:iMHTYp47/cVnXX+SXweYlG9M9dZhOopTFCBLpyOT69E=",
"origin": {
"hash": "fa3d8cc5ceda129b76f8284231f34a3b6fe30b86",
"ref": "refs/tags/v1.2.0"
},
"source_zip": {
"sha256": "ad670f9c652f7d54b381fcc79e0e76a9506750c44465fb019260a2f8de259d17",
"size": 11232684
}
},
{
"version": "v1.2.1",
"sum": "h1:23vau/bJYyjRUo3CzvZyfm63yxeV1SHalaPRpVFNIpY=",
"go_mod_sum": "h1:iMHTYp47/cVnXX+SXweYlG9M9dZhOopTFCBLpyOT69E=",
"origin": {
"hash": "634cbbc4bc580fa5124f63fdf65d137a46d5b4ff",
"ref": "refs/tags/v1.2.1"
},
"source_zip": {
"sha256": "c10110f0e61087f8643d78a8ef6644375524a2ef3cead384809b394fad08d2ba",
"size": 11235296
}
},
{
"version": "v1.2.2",
"sum": "h1:2K5ki3k1GGGKuYdFIuJtqHHziGQcepN+H2f/EYfE0X4=",
"go_mod_sum": "h1:oavjy+j75Szj3AIHLe0rv6fMyMZ8O+vGpPFh0nq+T08=",
"origin": {
"hash": "6c124203e771433a3550c348771a5b5e27fd3c21",
"ref": "refs/tags/v1.2.2"
},
"source_zip": {
"sha256": "4e26133ada66d4df2b5a225eed07e2a67d39f95d1e0f7af24a55a921552d2856",
"size": 7610482
},
"github_release": {
"html_url": "https://github.com/gastownhall/beads/releases/tag/v1.2.2",
"source_relation": "matches_proxy_origin",
"linux_amd64_asset": {
"size": 49107375,
"name": "beads_1.2.2_linux_amd64.tar.gz",
"digest": "sha256:8140098a51d3b81d5548d1c5e6db1a2d9930e5d141efe2a4bff7d079c4d321e8"
}
}
},
{
"version": "v1.3.0",
"sum": "h1:QPvdw+hRA6EDx31eTzBe2pl/B5Z7yXczq7bC8KHPY0Q=",
"go_mod_sum": "h1:bw++9IlggPhjAeyNOv6cmfOlfyV+ERA5BiRgsqDvyTc=",
"origin": {
"hash": "f45b249ce6b40ba62aecc03949e6371e8f7c79d8",
"ref": "refs/tags/v1.3.0"
},
"source_zip": {
"sha256": "09dc04bcca035e194d9c0bce3ff4e2caffbed30795d12bc757bc06cdb66757d6",
"size": 12150899
},
"github_release": {
"html_url": "https://github.com/gastownhall/beads/releases/tag/v1.3.0",
"source_relation": "matches_proxy_origin",
"linux_amd64_asset": {
"size": 53179375,
"name": "beads_1.3.0_linux_amd64.tar.gz",
"digest": "sha256:2f92b904ecf35b607e44dc5c39229173af69c54f1183e8d709f1773540cdcf3b"
}
}
},
{
"version": "v1.3.1",
"sum": "h1:Ec9DhMDHo/prSd8PDYts+4VF5TY9hI0PrZ46bYnxj1Y=",
"go_mod_sum": "h1:z7FHqG/z403/Sjr6j36QaeWdFWFpEDwQVhWZt9C3Mug=",
"origin": {
"hash": "c1c4b642ac1c08d8c828007a1c2f96e47e43ef7c",
"ref": "refs/tags/v1.3.1"
},
"source_zip": {
"sha256": "018ce9faca318c0d8c54bde035ddf62bdfc7f45980ef0c92359dc1369bea9c24",
"size": 12443826
},
"github_release": {
"html_url": "https://github.com/gastownhall/beads/releases/tag/v1.3.1",
"source_relation": "matches_proxy_origin",
"linux_amd64_asset": {
"size": 53304363,
"name": "beads_1.3.1_linux_amd64.tar.gz",
"digest": "sha256:3219443a9734b89b93fb16ee8d65844759fa1b3cd3cf139c606b7353cfb0715c"
}
}
}
],
"repository_tag_drift": [
@ -2459,7 +2551,12 @@
"exclusions": {
"proxy_prereleases": [
"v1.1.0-rc.1",
"v1.1.0-rc.2"
"v1.1.0-rc.2",
"v1.2.2-rc.1",
"v1.3.0-rc.1",
"v1.3.0-rc.2",
"v1.3.1-rc.1",
"v1.3.1-rc.2"
],
"repository_only_stable_tags": [
"v0.57.12",

View file

@ -0,0 +1,187 @@
package scripts_test
import (
"encoding/json"
"regexp"
"slices"
"strings"
"testing"
)
// The historical-upgrade corpus runs as //tests/migration's per-release
// targets. Their release list and their pinned inputs live in Bazel files;
// scripts/migration-test/lib/versions.sh stays the reviewed definition of the
// corpus. These checks keep the two equal, so a release added to the corpus
// gets a target and a pinned binary, and a target never runs a binary whose
// digest differs from the one the harness would have verified.
// bashArray returns the elements of `declare -ar NAME=( ... )` in versions.sh.
func bashArray(t *testing.T, src, name string) []string {
t.Helper()
m := regexp.MustCompile(`(?s)declare -ar ` + name + `=\((.*?)\)`).FindStringSubmatch(src)
if m == nil {
t.Fatalf("versions.sh has no array %s", name)
}
return regexp.MustCompile(`"([^"]+)"`).FindAllString(m[1], -1)
}
// bashScalar returns the value of `readonly NAME="..."` in versions.sh.
func bashScalar(t *testing.T, src, name string) string {
t.Helper()
m := regexp.MustCompile(`(?m)^readonly ` + name + `="([^"]*)"$`).FindStringSubmatch(src)
if m == nil {
t.Fatalf("versions.sh has no scalar %s", name)
}
return m[1]
}
// starlarkList returns the string elements of `NAME = [ ... ]` in a BUILD file.
func starlarkList(t *testing.T, src, name string) []string {
t.Helper()
m := regexp.MustCompile(`(?s)\n` + name + ` = \[(.*?)\]`).FindStringSubmatch(src)
if m == nil {
t.Fatalf("tests/migration/BUILD.bazel has no list %s", name)
}
var out []string
for _, q := range regexp.MustCompile(`"([^"]+)"`).FindAllStringSubmatch(m[1], -1) {
out = append(out, q[1])
}
return out
}
func unquote(items []string) []string {
out := make([]string, len(items))
for i, s := range items {
out[i] = strings.Trim(s, `"`)
}
return out
}
func sortedSet(items ...[]string) []string {
var out []string
for _, list := range items {
out = append(out, list...)
}
slices.Sort(out)
return slices.Compact(out)
}
func TestMigrationCorpusBazelTargetsMatchVersionsSh(t *testing.T) {
root := sourceRepoRoot(t)
versions := readPolicyFile(t, root, "scripts/migration-test/lib/versions.sh")
build := readPolicyFile(t, root, "tests/migration/BUILD.bazel")
sqlite := []string{
bashScalar(t, versions, "SOURCE_TAG_SQLITE_VERSION"),
bashScalar(t, versions, "PRE_CANONICAL_SQLITE_VERSION"),
bashScalar(t, versions, "CLASSIC_SQLITE_VERSION"),
bashScalar(t, versions, "CONFIGURED_SQLITE_VERSION"),
}
server := unquote(bashArray(t, versions, "HISTORICAL_DOLT_VERSIONS"))
embedded := unquote(bashArray(t, versions, "EMBEDDED_DOLT_VERSIONS"))
wisp := unquote(bashArray(t, versions, "WISP_PLANE_VERSIONS"))
if got, want := sortedSet(starlarkList(t, build, "_CORPUS")), sortedSet(sqlite, server, embedded); !slices.Equal(got, want) {
t.Errorf("tests/migration _CORPUS = %v, want versions.sh's corpus %v", got, want)
}
if got, want := sortedSet(starlarkList(t, build, "_DOLT_RUNTIME")), sortedSet(server, wisp); !slices.Equal(got, want) {
t.Errorf("tests/migration _DOLT_RUNTIME = %v, want HISTORICAL_DOLT_VERSIONS + WISP_PLANE_VERSIONS %v", got, want)
}
if m := regexp.MustCompile(`(?m)^_SOURCE_TAG = "([^"]+)"$`).FindStringSubmatch(build); m == nil || m[1] != sqlite[0] {
t.Errorf("tests/migration _SOURCE_TAG = %v, want SOURCE_TAG_SQLITE_VERSION %s", m, sqlite[0])
}
if !strings.Contains(build, `"github.com/steveyegge/beads `+sqlite[0]+` `+bashScalar(t, versions, "SOURCE_TAG_SQLITE_GO_TOOLCHAIN")+`"`) {
t.Errorf("tests/migration:bd_source_tag does not build %s@%s with %s",
bashScalar(t, versions, "SOURCE_TAG_SQLITE_MODULE"), sqlite[0], bashScalar(t, versions, "SOURCE_TAG_SQLITE_GO_TOOLCHAIN"))
}
}
// Every pinned release a target runs is the archive lib/binary.sh would have
// downloaded and verified: the catalog's linux/amd64 digest (what
// @bd_releases checks) equals STRICT_RELEASE_SHA256.
func TestMigrationCorpusCatalogDigestsMatchStrictPins(t *testing.T) {
root := sourceRepoRoot(t)
versions := readPolicyFile(t, root, "scripts/migration-test/lib/versions.sh")
var catalog struct {
Versions []struct {
Version string `json:"version"`
GithubRelease *struct {
Asset *struct {
Name string `json:"name"`
Digest string `json:"digest"`
} `json:"linux_amd64_asset"`
} `json:"github_release"`
SourceZip struct {
SHA256 string `json:"sha256"`
} `json:"source_zip"`
} `json:"versions"`
}
if err := json.Unmarshal([]byte(readPolicyFile(t, root, "scripts/migration-test/release-catalog.json")), &catalog); err != nil {
t.Fatal(err)
}
digests, sourceZips := map[string]string{}, map[string]string{}
for _, v := range catalog.Versions {
sourceZips[v.Version] = v.SourceZip.SHA256
if v.GithubRelease != nil && v.GithubRelease.Asset != nil {
digests[v.Version] = strings.TrimPrefix(v.GithubRelease.Asset.Digest, "sha256:")
}
}
pins := regexp.MustCompile(`\["(v[^|"]+)\|linux\|amd64"\]="([0-9a-f]{64})"`).FindAllStringSubmatch(versions, -1)
if len(pins) == 0 {
t.Fatal("versions.sh has no linux/amd64 STRICT_RELEASE_SHA256 pins")
}
for _, p := range pins {
if got := digests[p[1]]; got != p[2] {
t.Errorf("%s: catalog linux_amd64 digest %q, versions.sh STRICT_RELEASE_SHA256 %q", p[1], got, p[2])
}
}
// The source-tag build's own module zip is the catalog's source_zip.
var lock struct {
Module string `json:"module"`
Version string `json:"version"`
Modules []struct {
Path string `json:"path"`
Version string `json:"version"`
ZipSHA256 string `json:"zip_sha256"`
} `json:"modules"`
}
if err := json.Unmarshal([]byte(readPolicyFile(t, root, "tests/migration/source_tag_modules.json")), &lock); err != nil {
t.Fatal(err)
}
if want := bashScalar(t, versions, "SOURCE_TAG_SQLITE_VERSION"); lock.Version != want || lock.Module != bashScalar(t, versions, "SOURCE_TAG_SQLITE_MODULE") {
t.Errorf("source_tag_modules.json pins %s@%s, want %s@%s", lock.Module, lock.Version, bashScalar(t, versions, "SOURCE_TAG_SQLITE_MODULE"), want)
}
found := false
for _, m := range lock.Modules {
if m.Path == lock.Module && m.Version == lock.Version {
found = true
if m.ZipSHA256 != sourceZips[lock.Version] {
t.Errorf("source_tag_modules.json %s zip %q, catalog source_zip %q", lock.Version, m.ZipSHA256, sourceZips[lock.Version])
}
}
}
if !found {
t.Errorf("source_tag_modules.json does not pin %s@%s itself", lock.Module, lock.Version)
}
}
// The external Dolt runtime the server-Dolt and wisp-plane lanes run is the
// one versions.sh pins: same release, same archive digest.
func TestMigrationDoltRuntimeMatchesVersionsSh(t *testing.T) {
root := sourceRepoRoot(t)
versions := readPolicyFile(t, root, "scripts/migration-test/lib/versions.sh")
version := strings.TrimPrefix(bashScalar(t, versions, "DOLT_TEST_RUNTIME_VERSION"), "v")
digest := bashScalar(t, versions, "DOLT_TEST_RUNTIME_SHA256")
module := readPolicyFile(t, root, "MODULE.bazel")
m := regexp.MustCompile(`(?s)name = "dolt_test_runtime_linux_amd64",\s*platform = "linux-amd64",\s*version = "([^"]+)",`).FindStringSubmatch(module)
if m == nil || m[1] != version {
t.Errorf("MODULE.bazel's dolt_test_runtime_linux_amd64 = %v, want version %q", m, version)
}
bzl := readPolicyFile(t, root, "tools/bazel/dolt.bzl")
d := regexp.MustCompile(`(?s)"` + regexp.QuoteMeta(version) + `": \{\s*"linux-amd64": "([0-9a-f]{64})",`).FindStringSubmatch(bzl)
if d == nil || d[1] != digest {
t.Errorf("tools/bazel/dolt.bzl DOLT_SHA256[%q][linux-amd64] = %v, want %s", version, d, digest)
}
}

131
tests/migration/BUILD.bazel Normal file
View file

@ -0,0 +1,131 @@
load("@rules_shell//shell:sh_test.bzl", "sh_test")
load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64")
load(":defs.bzl", "historical_upgrade_tests")
# The historical-upgrade corpus (formerly migration-test.yml), one target per
# reviewed release: historical_upgrade_test.sh runs
# scripts/migration-test/historical-dolt-upgrade-test.sh --version <release>
# against the non-race bd_for_tests. Every historical bd a release's lanes run
# is pinned: the published releases from the release catalog (@bd_releases,
# sha256-verified when Bazel fetches them), the source-tag v0.9.1 built
# offline below (:bd_source_tag). Where the lanes host a server-Dolt fixture
# or read the wisp plane, the pinned external Dolt runtime
# (@dolt_test_runtime_linux_amd64) is DOLT_BIN. Nothing is downloaded at test
# time, so the targets run remotely like any other, in the PR-core lane
# (bazel-test), and a target re-runs only when the candidate, the harness or
# a pinned binary changes.
#
# _CORPUS must equal scripts/migration-test/lib/versions.sh's corpus (the
# SQLite bridges, HISTORICAL_DOLT_VERSIONS and EMBEDDED_DOLT_VERSIONS), and
# _DOLT_RUNTIME its HISTORICAL_DOLT_VERSIONS plus WISP_PLANE_VERSIONS
# (scripts/migration_corpus_bazel_test.go). _HELPERS are the releases a lane
# runs besides its own: the classic SQLite bridge (v0.49.6) and the
# server-Dolt canonicalizer (v0.62.0).
_SOURCE_TAG = "v0.9.1"
_CORPUS = [
"v0.9.1",
"v0.17.0",
"v0.49.6",
"v0.50.3",
"v0.55.4",
"v0.56.1",
"v0.57.0",
"v0.62.0",
"v0.63.3",
"v1.0.0",
"v1.0.1",
"v1.1.0",
"v1.1.2",
"v1.2.2",
]
_DOLT_RUNTIME = [
"v0.55.4",
"v0.56.1",
"v0.57.0",
"v0.62.0",
"v1.0.1",
"v1.1.0",
"v1.1.2",
"v1.2.2",
]
_HELPERS = [
"v0.49.6",
"v0.62.0",
]
# Tags:
# pr-core-only: shell tests against the release-shaped bd_for_tests (no
# integration tag); the integration lane, which requires Go testcases,
# leaves them out.
historical_upgrade_tests(
corpus = _CORPUS,
dolt_runtime = _DOLT_RUNTIME,
helpers = _HELPERS,
source_tag = _SOURCE_TAG,
source_tag_bin = ":bd_source_tag",
tags = ["pr-core-only"],
)
exports_files(
["source_tag_modules.json"],
visibility = ["//visibility:public"],
)
# The source-tag SQLite release (v0.9.1: no release asset; its CLI reports
# 0.9.0 (dev)), built from its module source with Go 1.25.0 and CGO like
# lib/binary.sh's build_verified_v091_source_binary, but offline: the module
# zips and go.mod files come from source_tag_modules.json (the release's own
# zip is the catalog's source_zip), and go checks them against the release's
# go.sum. Regenerate the lock with gen_source_tag_lock.py.
genrule(
name = "bd_source_tag",
srcs = [
"@bd_source_tag_goproxy//:proxy",
"@go_1_25_0_linux_amd64//:go/bin/go",
"@go_1_25_0_linux_amd64//:sdk",
],
outs = ["bd_source_tag/bd"],
cmd = "$(execpath build_source_tag.sh) $@ $(execpath @go_1_25_0_linux_amd64//:go/bin/go) " +
"$$(echo $(execpaths @bd_source_tag_goproxy//:proxy) | cut -d' ' -f1) " +
"github.com/steveyegge/beads v0.9.1 go1.25.0",
target_compatible_with = LINUX_AMD64,
tools = ["build_source_tag.sh"],
)
# The public sealed-copy SQLite bridge's fast safety checks (fake binaries
# only), which run.sh ran before every release.
#
# Tags:
# pr-core-only: a shell test; the integration lane, which requires Go
# testcases, leaves it out.
sh_test(
name = "legacy_bridge_test",
srcs = ["legacy_bridge_test.sh"],
data = ["//scripts:migration_harness"],
tags = ["pr-core-only"],
)
# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) ---
filegroup(
name = "repo_files",
srcs = glob(
["**"],
exclude = [
"**/*.db",
"**/*.exe",
"**/*.out",
"**/*.prof",
"**/*.pyc",
"**/*.test",
"**/__pycache__/**",
"**/node_modules/**",
],
),
visibility = ["//:__pkg__"],
)
# --- end repo_files ---

View file

@ -0,0 +1,38 @@
#!/usr/bin/env bash
# build-tags: allow-bare (builds a historical release from its own module
# source with its own go.mod, exactly as lib/binary.sh does; that release
# predates gms_pure_go and must not be built with today's tags)
# Builds a historical bd from its module source, offline, the way
# scripts/migration-test/lib/binary.sh's build_verified_v091_source_binary
# does with network access: `go mod download` the module, then `go build
# -trimpath ./cmd/bd` in its directory with CGO and a writable copy of its
# go.mod, using the Go release it was qualified on.
#
# Usage: build_source_tag.sh <out> <go binary> <a file under the proxy tree> <module> <version> <go version>
# Paths are execroot-relative ($(execpath ...)). GOPROXY is the pinned
# file:// tree (tools/bazel/go_module_proxy.bzl), so every module is a
# sha256-verified input, and go checks each dependency against the module's
# go.sum; GOSUMDB is off because there is no network to reach it.
set -euo pipefail
out="$1" go_bin="$2" proxy_file="$3" module="$4" version="$5" go_version="$6"
root="$PWD"
tmp="$(mktemp -d)"
trap 'chmod -R u+w "$tmp" 2>/dev/null || true; rm -rf "$tmp"' EXIT
export GOROOT="$root/${go_bin%/bin/go}"
export PATH="$GOROOT/bin:$PATH"
export HOME="$tmp/home" GOPATH="$tmp/gopath" GOMODCACHE="$tmp/mod" GOCACHE="$tmp/cache"
export GOPROXY="file://$root/${proxy_file%%/proxy/*}/proxy" GOSUMDB=off GONOSUMDB= GOPRIVATE= GONOPROXY=
export GOFLAGS=-modcacherw GOWORK=off GOTOOLCHAIN=local GO111MODULE=on
export CGO_ENABLED=1 GOOS=linux GOARCH=amd64
mkdir -p "$HOME" "$tmp/scratch"
got="$(cd "$tmp/scratch" && go env GOVERSION)"
if [[ "$got" != "$go_version" ]]; then
echo "build_source_tag: $module@$version needs $go_version, the pinned SDK is $got" >&2
exit 1
fi
(cd "$tmp/scratch" && go mod download "$module@$version")
src="$GOMODCACHE/$module@$version"
cp -f "$src/go.mod" "$tmp/source.mod"
cp -f "$src/go.sum" "$tmp/source.sum"
chmod u+w "$tmp/source.mod" "$tmp/source.sum"
(cd "$src" && go build -trimpath -modfile="$tmp/source.mod" -o "$root/$out" ./cmd/bd)

52
tests/migration/defs.bzl Normal file
View file

@ -0,0 +1,52 @@
"""One Bazel test per historical-upgrade corpus release (see BUILD.bazel)."""
load("@rules_shell//shell:sh_test.bzl", "sh_test")
load("//tools/bazel:bd_releases.bzl", "LINUX_AMD64")
load("//tools/bazel:dolt.bzl", "hermetic_bin")
_DOLT_RUNTIME_LABEL = "@dolt_test_runtime_linux_amd64//:dolt"
def historical_upgrade_tests(corpus, dolt_runtime, helpers, source_tag, source_tag_bin, tags):
"""Declares historical_upgrade_<release>_test and its release directory per corpus release.
Args:
corpus: every release the corpus upgrades from.
dolt_runtime: the releases whose lanes need the external Dolt runtime.
helpers: releases every lane may run besides its own (bridges).
source_tag: the release built from source (no @bd_releases entry).
source_tag_bin: the label of that source build.
tags: the tests' tags.
"""
for v in corpus:
suffix = v.replace(".", "_")
releases = {r: None for r in [v] + helpers if r != source_tag}
hermetic_bin(
name = "releases_" + suffix,
target_compatible_with = LINUX_AMD64,
tools = {"@bd_releases//:" + r: r for r in sorted(releases.keys())},
)
data = [
":releases_" + suffix,
"//cmd/bd:bd_for_tests",
"//scripts:migration_harness",
]
env = {
"CANDIDATE_BIN": "$(rootpath //cmd/bd:bd_for_tests)",
"HISTORICAL_RELEASE_DIR": "{}/releases_{}".format(native.package_name(), suffix),
}
if v in dolt_runtime:
data.append(_DOLT_RUNTIME_LABEL)
env["DOLT_BIN"] = "$(rootpath {})".format(_DOLT_RUNTIME_LABEL)
if v == source_tag:
data.append(source_tag_bin)
env["SOURCE_TAG_SQLITE_BIN"] = "$(rootpath {})".format(source_tag_bin)
sh_test(
name = "historical_upgrade_{}_test".format(suffix),
size = "large",
srcs = ["historical_upgrade_test.sh"],
args = [v],
data = data,
env = env,
tags = tags,
target_compatible_with = LINUX_AMD64,
)

View file

@ -0,0 +1,87 @@
#!/usr/bin/env python3
"""Regenerate source_tag_modules.json: the Go module proxy files that build the
source-tag SQLite release (v0.9.1, which has no release asset) offline.
Usage: python3 tests/migration/gen_source_tag_lock.py
It fetches the release's module zip from proxy.golang.org, checks it against
the release catalog's source_zip sha256, reads the go.sum inside it, and
records the sha256 of every .mod and .zip that go.sum names (plus the
release's own .mod and .zip). tools/bazel/go_module_proxy.bzl downloads
exactly these files, verified, into a file:// GOPROXY tree; go then checks
every dependency against that go.sum as well. Network required; run it by
hand when the source-tag release changes, never in CI.
"""
import hashlib
import io
import json
import os
import sys
import urllib.request
import zipfile
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
CATALOG = os.path.join(ROOT, "scripts", "migration-test", "release-catalog.json")
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "source_tag_modules.json")
PROXY = "https://proxy.golang.org"
MODULE = "github.com/steveyegge/beads"
VERSION = "v0.9.1"
def escape(path):
"""Module path case-encoding of the proxy protocol (go.dev/ref/mod#goproxy-protocol)."""
return "".join("!" + c.lower() if c.isupper() else c for c in path)
def fetch(url):
with urllib.request.urlopen(url, timeout=60) as resp:
return resp.read()
def sha256(data):
return hashlib.sha256(data).hexdigest()
def main():
with open(CATALOG) as f:
catalog = json.load(f)
entry = next(e for e in catalog["versions"] if e["version"] == VERSION)
base = f"{PROXY}/{escape(MODULE)}/@v/{VERSION}"
zip_data = fetch(base + ".zip")
if sha256(zip_data) != entry["source_zip"]["sha256"]:
sys.exit(f"{MODULE}@{VERSION}.zip does not match the catalog's source_zip sha256")
gosum = zipfile.ZipFile(io.BytesIO(zip_data)).read(f"{MODULE}@{VERSION}/go.sum").decode()
wants = {} # (path, version) -> needs zip
for line in gosum.splitlines():
fields = line.split()
if len(fields) != 3:
continue
path, version = fields[0], fields[1]
if version.endswith("/go.mod"):
wants.setdefault((path, version[: -len("/go.mod")]), False)
else:
wants[(path, version)] = True
modules = [{
"path": MODULE,
"version": VERSION,
"mod_sha256": sha256(fetch(base + ".mod")),
"zip_sha256": entry["source_zip"]["sha256"],
}]
for (path, version), needs_zip in sorted(wants.items()):
url = f"{PROXY}/{escape(path)}/@v/{version}"
m = {"path": path, "version": version, "mod_sha256": sha256(fetch(url + ".mod"))}
if needs_zip:
m["zip_sha256"] = sha256(fetch(url + ".zip"))
modules.append(m)
print(f"{path}@{version}", file=sys.stderr)
with open(OUT, "w") as f:
json.dump({"module": MODULE, "version": VERSION, "modules": modules}, f, indent=2, sort_keys=True)
f.write("\n")
if __name__ == "__main__":
main()

View file

@ -0,0 +1,29 @@
#!/usr/bin/env bash
# One release of the historical-upgrade corpus (formerly migration-test.yml's
# `scripts/migration-test/run.sh --version <release>` loop), as a Bazel test:
# scripts/migration-test/historical-dolt-upgrade-test.sh against the
# candidate, with every historical binary it runs pre-pinned in
# HISTORICAL_RELEASE_DIR (lib/binary.sh downloads nothing then), the
# source-built v0.9.1 as SOURCE_TAG_SQLITE_BIN for that release, and, for the
# releases that need it, the pinned external Dolt runtime as DOLT_BIN.
#
# The first argument is the release. The BUILD file passes the paths as
# runfiles-relative paths ($(rootpath ...), relative to the working
# directory Bazel starts the test in); they are made absolute here, before
# the harness changes directory. Any further arguments (a lane's
# --test_arg flags, meant for Go test binaries) are ignored.
set -euo pipefail
release="$1"
abs() { printf '%s/%s\n' "$PWD" "$1"; }
CANDIDATE_BIN="$(abs "${CANDIDATE_BIN:?}")"
HISTORICAL_RELEASE_DIR="$(abs "${HISTORICAL_RELEASE_DIR:?}")"
export CANDIDATE_BIN HISTORICAL_RELEASE_DIR
if [[ -n "${DOLT_BIN:-}" ]]; then
DOLT_BIN="$(abs "$DOLT_BIN")"
export DOLT_BIN
fi
if [[ -n "${SOURCE_TAG_SQLITE_BIN:-}" ]]; then
SOURCE_TAG_SQLITE_BIN="$(abs "$SOURCE_TAG_SQLITE_BIN")"
export SOURCE_TAG_SQLITE_BIN
fi
exec scripts/migration-test/historical-dolt-upgrade-test.sh --version "$release"

View file

@ -0,0 +1,7 @@
#!/usr/bin/env bash
# The public SQLite bridge's fast safety checks (fake binaries only), which
# scripts/migration-test/run.sh runs before the corpus. Runs the harness in
# place, so it finds its siblings (lib/, ../migrate-legacy-to-current.sh);
# arguments are ignored.
set -euo pipefail
exec scripts/migration-test/legacy-bridge-test.sh

View file

@ -0,0 +1,166 @@
{
"module": "github.com/steveyegge/beads",
"modules": [
{
"mod_sha256": "7951983a1500203e1281e3e60e113105ef53a8f69e02d2aca3c2c190f77abada",
"path": "github.com/steveyegge/beads",
"version": "v0.9.1",
"zip_sha256": "a45a701fd720b89c08ce0993cb6771c5d3300dc8f96f5669e8a85aa30bc80de1"
},
{
"mod_sha256": "bb81a9e2d14d9a586f0591853bdaad230d8cdf0d8d6ffa71dddae249ad573cc9",
"path": "github.com/cpuguy83/go-md2man/v2",
"version": "v2.0.6"
},
{
"mod_sha256": "8732da74f45ec4c0ea5a7a3072c4d458aafb6102af75155d459dcb70afce6ae6",
"path": "github.com/fatih/color",
"version": "v1.18.0",
"zip_sha256": "4dc6907b7f6807b37c21ad527d966437d398f03472115f471eaa8fe5875363d3"
},
{
"mod_sha256": "da3713c5c57d252d25520de0e6b397b033858b3aff5dddf21164abe3297c9ff9",
"path": "github.com/fsnotify/fsnotify",
"version": "v1.9.0",
"zip_sha256": "94e64824b23591b7aa504e3321d5ec3f57de88f4d49882eaefbbf6820dab746e"
},
{
"mod_sha256": "5393fe364f1a200c6a84ee2c7bf727f02c9e0b1f844ca13b188ca925f4718b08",
"path": "github.com/go-viper/mapstructure/v2",
"version": "v2.4.0",
"zip_sha256": "33e0398e34eca4b1b8c9d91318dd3be871811aee6e2ababc82c02280dcca296a"
},
{
"mod_sha256": "1c1b15911aa8a0bd5cd0f6670ec23d1086496f3615fdf53306aff2b476ce9be1",
"path": "github.com/inconshreveable/mousetrap",
"version": "v1.1.0",
"zip_sha256": "526674de624d7db108cfe7653ef110ccdfd97bc85026254224815567928ed243"
},
{
"mod_sha256": "d6743ae2eae65716122f12b42c9f3f83a805c0ae32bd3c553d4a6f1bcb9243a5",
"path": "github.com/mattn/go-colorable",
"version": "v0.1.13",
"zip_sha256": "08be322dcc584a9fcfde5caf0cf878b4e11cd98f252e32bc704e92c5a4ba9d15"
},
{
"mod_sha256": "714766ec82d67db895ade8af6b8054ad430c3cea5e15b3a888f23ed2f3e1b863",
"path": "github.com/mattn/go-isatty",
"version": "v0.0.16"
},
{
"mod_sha256": "1eeabef1b4d98db73c6c2073289163649bc9a044552aca86b9d592fe84eaa1c8",
"path": "github.com/mattn/go-isatty",
"version": "v0.0.20",
"zip_sha256": "f2d5f89ca451577e17464b9bb596dc0d0ecececb5eaa63622c41b57cd0b7b8cc"
},
{
"mod_sha256": "bc9235ee6e4e60243f1e7dcacf0caa4d480afcdf06ed78e481fd127e9e3119df",
"path": "github.com/mattn/go-sqlite3",
"version": "v1.14.32",
"zip_sha256": "c70101000256656bc0b7286807cf177852c753a001a3aa3c55d5373575fb81aa"
},
{
"mod_sha256": "72b2ddd20583039d110a50147c63fd1e419a5c4acc1ba8f28881f1df4486abae",
"path": "github.com/pelletier/go-toml/v2",
"version": "v2.2.4",
"zip_sha256": "82162be7b85077f3be49bfe0f1e01f308b9c9554c72a980a22a34432f94885f0"
},
{
"mod_sha256": "c39547cce6e866e8cfa395d2937bce30801cd07f0b901c2133248f1fba12f58d",
"path": "github.com/russross/blackfriday/v2",
"version": "v2.1.0"
},
{
"mod_sha256": "61f9956a081c0abc80468039819c9bc5c1d51236b48f864bce6540cf0e5cbfa4",
"path": "github.com/sagikazarmark/locafero",
"version": "v0.11.0",
"zip_sha256": "6c2a9bdf9567f63caebf47694d94acd4350b1d4d69f13c3188f4e7e82939f6a5"
},
{
"mod_sha256": "1a03e9553a071b4371a6e72483b440fda7b591693403531d090286da73c7d1fc",
"path": "github.com/sourcegraph/conc",
"version": "v0.3.1-0.20240121214520-5f936abd7ae8",
"zip_sha256": "6db56fe285f78ba994a77d676d790b1db85a47cc8948a42d4c2f6ca6200b9413"
},
{
"mod_sha256": "b9ecbcbeeccb9da252f2b75a7f05a10c21f874a448a5e1294f115e47ba893e22",
"path": "github.com/spf13/afero",
"version": "v1.15.0",
"zip_sha256": "d03a0bbce87eeaa1c23f0c0276707bb2d59f46efee875d7d7f1d570201be6e1f"
},
{
"mod_sha256": "fef5e4baa2ea6235860469e13b663f5fb26a7047983e05fdb3bf60ea843530f1",
"path": "github.com/spf13/cast",
"version": "v1.10.0",
"zip_sha256": "7c7ba27f9979ada980c2a69f6b2fdf7b4ce1f2b766e245c0a4716f546ad7ddce"
},
{
"mod_sha256": "a501ef186772adafdaabb7aa78623b691890e87accd0715674e69949da6b586e",
"path": "github.com/spf13/cobra",
"version": "v1.10.1",
"zip_sha256": "00955783267c9ced54274df456377a15bc1b658362f93f85dfda0708b54f9a28"
},
{
"mod_sha256": "16c6dfc0eb58eb3c9113ce197d460c6725c647e4688ac26d31c8ed69f1ebbc68",
"path": "github.com/spf13/pflag",
"version": "v1.0.10",
"zip_sha256": "a246b8c9d3daa87d6c634cdfee1bdf7fa53d5817eeef4fa0c6da43edd12de4ca"
},
{
"mod_sha256": "16c6dfc0eb58eb3c9113ce197d460c6725c647e4688ac26d31c8ed69f1ebbc68",
"path": "github.com/spf13/pflag",
"version": "v1.0.9"
},
{
"mod_sha256": "671ff6030ff3b0300240c833789512a5b8819cee89a819db7bb20a6a44906b29",
"path": "github.com/spf13/viper",
"version": "v1.21.0",
"zip_sha256": "4096e4ccdbb74027c49f0dfede6d67193053b0d1c1ecce57543cf3dacc7b249e"
},
{
"mod_sha256": "8b2d8bc78b5def106360867a82ad82c5a4e9de93542f0a207087bb38384f55bf",
"path": "github.com/subosito/gotenv",
"version": "v1.6.0",
"zip_sha256": "142db3dd2328e744c157e85cf3291d027013b79f92a45984f860fe38bc0f1f8d"
},
{
"mod_sha256": "2989f8b800605890ad40cf49109b0b6a972bda79d92df236d40ddb798093127e",
"path": "go.yaml.in/yaml/v3",
"version": "v3.0.4",
"zip_sha256": "0317dae56eaa9e9954467c8768fd6a89ba2005f1534e2c26fce5163ad8d2ef31"
},
{
"mod_sha256": "f033333096fe198f3151deed93f2deba74e50bbfe7739134045bc3b7ce4a5024",
"path": "golang.org/x/sys",
"version": "v0.0.0-20220811171246-fbc7d0a398ab"
},
{
"mod_sha256": "d227b325f621f4ebe28d39ba773ea99b870f393b7c09c34592c365b16dd560de",
"path": "golang.org/x/sys",
"version": "v0.29.0",
"zip_sha256": "1d759c017d09328a4dacdd4a1a0170cf21641995c6e3ca1f96213369d73c6846"
},
{
"mod_sha256": "f033333096fe198f3151deed93f2deba74e50bbfe7739134045bc3b7ce4a5024",
"path": "golang.org/x/sys",
"version": "v0.6.0"
},
{
"mod_sha256": "5114acdbfab7e7a097808d0bd4cdf06e3a3043e4bc5e4e4974715d664502698a",
"path": "golang.org/x/text",
"version": "v0.28.0",
"zip_sha256": "46259e1416ae7ec6adf1867c5f9fab32af0476a148e3c95f1dfbb134f4acf48d"
},
{
"mod_sha256": "5c306e0d633cd66a11b40e2e5fbbc6da42110db7d72ea3c1524ceb45ee40c33f",
"path": "gopkg.in/check.v1",
"version": "v0.0.0-20161208181325-20d25e280405"
},
{
"mod_sha256": "21579860a20306fcf43b1bd234d1fba319499c77611b71c05f9bf3ba90dab939",
"path": "gopkg.in/yaml.v3",
"version": "v3.0.1"
}
],
"version": "v0.9.1"
}

View file

@ -18,6 +18,12 @@ DOLT_VERSION = "2.2.0"
# sha256 alone). They match the asset digests GitHub reports for the release
# (`gh api repos/dolthub/dolt/releases/tags/v<version>`).
DOLT_SHA256 = {
# The migration corpus's external Dolt runtime (DOLT_TEST_RUNTIME_VERSION
# and _SHA256 in scripts/migration-test/lib/versions.sh), linux/amd64
# only: @dolt_test_runtime_linux_amd64 in MODULE.bazel.
"2.1.8": {
"linux-amd64": "f66318f08ed66e409fc39363ae0fff8ce6fbf6dba9f5bac632b91527b9632a74",
},
"2.2.0": {
"darwin-amd64": "d40b57933e2a2c025a5a3c269eb87594b1aa71ead16c9902213d521529a19b02",
"darwin-arm64": "c6737dc2c5806e2eeef4839ad76c28167c861f878af3071df1242a6589d81267",
@ -34,13 +40,14 @@ exports_files(["dolt"])
def _dolt_release_impl(rctx):
platform = rctx.attr.platform
digests = DOLT_SHA256.get(DOLT_VERSION)
version = rctx.attr.version or DOLT_VERSION
digests = DOLT_SHA256.get(version)
if digests == None:
fail("DOLT_SHA256 has no digests for DOLT_VERSION {}".format(DOLT_VERSION))
fail("DOLT_SHA256 has no digests for dolt {}".format(version))
if platform not in digests:
fail("no pinned dolt {} release for platform {}".format(DOLT_VERSION, platform))
fail("no pinned dolt {} release for platform {}".format(version, platform))
url = "https://github.com/dolthub/dolt/releases/download/v{v}/dolt-{p}.tar.gz".format(
v = DOLT_VERSION,
v = version,
p = platform,
)
rctx.download_and_extract(
@ -59,8 +66,11 @@ dolt_release = repository_rule(
mandatory = True,
doc = "Release platform suffix, for example linux-amd64.",
),
"version": attr.string(
doc = "Dolt release to fetch (a DOLT_SHA256 key); empty means DOLT_VERSION.",
),
},
doc = "Downloads the pinned Dolt CLI release for one platform and exports its `dolt` binary.",
doc = "Downloads a pinned Dolt CLI release for one platform and exports its `dolt` binary.",
)
def _hermetic_bin_impl(ctx):

View file

@ -0,0 +1,55 @@
"""An offline Go module proxy tree, for building a pinned historical module.
go_module_proxy reads a lock file (tests/migration/source_tag_modules.json,
written by tests/migration/gen_source_tag_lock.py) and downloads every .mod
and .zip it lists from proxy.golang.org, each verified against its recorded
sha256, into proxy/<escaped path>/@v/<version>.{mod,zip} (plus a .info naming the
version): the layout a
`GOPROXY=file://<dir>/proxy` reads. A build action then needs no network,
and go additionally checks every dependency against the module's own go.sum.
The repository exports the tree as :proxy.
"""
_PROXY = "https://proxy.golang.org"
_BUILD = """\
filegroup(
name = "proxy",
srcs = glob(["proxy/**"]),
visibility = ["//visibility:public"],
)
"""
def _escape(path):
# Module path case-encoding (go.dev/ref/mod#goproxy-protocol).
out = ""
for c in path.elems():
out += "!" + c.lower() if c.isupper() else c
return out
def _go_module_proxy_impl(rctx):
lock = json.decode(rctx.read(rctx.attr.lock))
for m in lock["modules"]:
rel = "{}/@v/{}".format(_escape(m["path"]), m["version"])
for ext, key in ((".mod", "mod_sha256"), (".zip", "zip_sha256")):
if key not in m:
continue
url = "{}/{}{}".format(_PROXY, rel, ext)
rctx.download(
url = url,
output = "proxy/{}{}".format(rel, ext),
sha256 = m[key],
canonical_id = url,
)
# .info only names the version (the proxy's also carries a commit
# time); `go mod download module@version` reads it, the build does not.
rctx.file("proxy/{}.info".format(rel), json.encode({"Version": m["version"]}) + "\n")
rctx.file("BUILD.bazel", _BUILD)
go_module_proxy = repository_rule(
implementation = _go_module_proxy_impl,
attrs = {
"lock": attr.label(mandatory = True, allow_single_file = True, doc = "The module lock JSON."),
},
doc = "Downloads the sha256-pinned module proxy files a lock file lists into a file:// GOPROXY tree.",
)