fix(ci): pin the Dolt CLI — 2.3.0 broke DOLT_RESET on fresh databases (ga-m0dlk) (#5894)

* fix(dolt): git-remote tests need a sql-server that shares their filesystem

The four git-remote tests that push through the store API fail on every run
of the new test-server-storage-full lane (#5836), which is what has the
merge pipeline stalled: shards 1, 9, 10 and 15 all die with

  the remote: origin 'git+file:///tmp/embedded-git-remote-test-*/remote.git'
  could not be accessed; fatal: ... does not appear to be a git repository

setupEmbeddedGitRemote dates from when this package had an in-process
embedded engine. store.go now hardcodes serverMode:true, so New() connects
to the suite's shared Dolt server — which TestMain runs in a Docker
container whose only mount is the image's own /var/lib/dolt. CALL DOLT_PUSH
executes inside that container, so the host bare repo the test just created
does not exist for it, and neither does the store's own Path (which is why
TestGitRemotePushSkipsUserPrePushHook cannot find its cache mirror and
TestGitRemoteSyncRoundTrip's clone comes up empty). The dolt credential
hints in the log are generic advice, not a credential problem — nothing
here needs a credential.

Give these tests their own local dolt sql-server rooted at the store's data
directory, the arrangement TestGitRemoteExternalServerRouting,
TestSQLRemotePersistsAcrossExternalServerRestart and
TestCredentialCLIRoutingE2E already use. The server is spawned with
doltserver.ServerSpawnEnv(), the same environment bd gives the server it
starts: a `dolt` CLI command run against a directory a sql-server already
serves is proxied to that server, so the git subprocess is spawned by the
server and the CLI-side guards (git tracing, core.hooksPath) only take
effect if the server carries them too.

Two consequences fall out and are fixed here:

  - The setup now takes a genesis CommitAll. Commit() deliberately skips
    config (GH#2455), so issue_prefix stayed dirty forever: Pull() refuses
    to auto-commit a dirty internal config key, and a peer cloning the
    database got no prefix at all. The CLI sibling setupGitRemote has
    always done this with DOLT_COMMIT('-Am', 'Genesis: schema and config').

  - The gate is testutil.RequireDoltBinary rather than skipIfNoDolt: these
    tests no longer need the shared container, only dolt and git, so
    requiring the container would be a skip for no reason.

Setup asserts <Path>/<db>/.dolt exists on this filesystem, so if the store
ever drifts back onto a containerized server these tests fail loudly
instead of quietly testing nothing.

Verified against the real shard script on a live Dolt server: shard 1 goes
from 1 failure to 0, shard 9 from 1 to 0.

* test(dolt): pin what TestCreateIssueAfterPull's Pull actually delivered

Shard 1 still fails in CI after the topology fix, now at the last assertion:
'GetIssue(ai-clone-001) failed: not found'. Deterministic across two CI runs,
not reproducible locally on dolt 2.1.8 or 2.3.1 (the whole shard is green
here). The end-of-test assertion cannot distinguish 'Pull() reported success
but merged nothing' from 'Pull() delivered the row and the post-pull
CreateIssue dropped it', which are different bugs in different places.

Assert immediately after Pull, before the post-pull write, so the CI failure
names which one it is.

* fix(ci): pin the Dolt CLI — 2.3.0 broke DOLT_RESET on fresh databases

Test (Server Dolt Full Suite 12/16) has been failing on a coin flip with

  TestFreshBootstrapHealIncarnation/exact_creator_heals_its_interrupted_bootstrap
  creator-authorized migration: schema migration: pending schema migrations
  alter pre-existing dirty tables: issues; run 'bd dolt commit' ... (#4566)
      schema: fresh-bootstrap reset: Error 1105 (HY000): context canceled

That second line is the whole story, and it is not a beads bug. The heal was
authorized and its capability consumed; what failed is the
CALL DOLT_RESET('--hard') in schema.MigrateUpWithLock. Dolt 2.3.0 regressed
that procedure: roughly one freshly created database in twenty comes up with
it permanently unusable, answering "Error 1105 (HY000): context canceled" to
every session, from any connection, for the life of the server process. It is
not transient and not load-related — a beads-free probe that only does
CREATE DATABASE followed immediately by the procedure reproduces it:

  dolt 2.1.8   0/40 databases broken
  dolt 2.2.0   0/60
  dolt 2.3.0   3/60
  dolt 2.3.1   3/100

Every workflow installed the CLI by piping dolthub/dolt's releases/latest
install.sh, so the binary under test changed the moment 2.3.x shipped —
while internal/testutil/testdoltcommon.go had the sql-server container pinned
at 2.2.0 the whole time. Server-mode tests run both halves at once against the
same databases, so the suite was exercising a Dolt pair no release ever
shipped. Pin the CLI to the container's release and keep the two locked
together:

- scripts/ci/install-dolt.sh installs a pinned Dolt, verifies `dolt version`
  on PATH actually resolves to it, and records the per-version measurements
  and the criterion for raising the pin.
- All 25 install sites across six workflows route through it.
- TestWorkflowsInstallPinnedDolt rejects any reintroduction of
  releases/latest; TestPinnedDoltCLIMatchesContainerImage fails when the CLI
  pin, DoltDockerImage and pull-dolt-image.sh disagree.

The test itself is unchanged in what it asserts. It gains one preflight probe:
before exercising the heal it checks that this server can hard-reset the
database it just created, so a broken Dolt is named as such instead of
surfacing as the generic #4566 refusal — which reads like a beads defect and
sends the reader to "run 'bd dolt commit'", advice that cannot work when the
reset the heal needs is the thing that is broken. On a pristine database the
probe is a no-op, and over 60 fresh databases a successful first reset never
broke a later one.

Verification: 48 runs of the test (6 workers x 8) and two full shard-12 runs
on the pinned 2.2.0, zero failures. Same binary on 2.3.1: 3 of 24 runs still
fail, now in 0.04s with the explicit diagnosis instead of 1s in with the
misleading one.

Beyond this test: bd dolt compact, bd dolt flatten and mergesettle's rollback
all issue DOLT_RESET('--hard'), so they are unreliable for real users on any
Dolt 2.3.x server. That needs an upstream report; pinning CI does not fix it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-08-20 20:59:27 -07:00 • committed by GitHub
commit 825cd4c937
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 207 additions and 25 deletions

View file

@ -66,7 +66,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -113,7 +113,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -144,7 +144,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -177,7 +177,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -222,7 +222,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -287,7 +287,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -355,7 +355,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -423,7 +423,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -528,7 +528,7 @@ jobs:
path: prebuilt-bd
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -594,7 +594,7 @@ jobs:
go-version-file: 'go.mod'
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |

View file

@ -401,7 +401,7 @@ jobs:
restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -505,7 +505,7 @@ jobs:
restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-
- name: Install Dolt (Linux/macOS)
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Verify dolt on PATH
run: dolt version
@ -625,7 +625,7 @@ jobs:
cache: false
- name: Install Dolt CLI
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Verify dolt on PATH
run: dolt version
@ -703,7 +703,7 @@ jobs:
chmod +x bd-linux-gms-pure
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -787,7 +787,7 @@ jobs:
chmod +x bd-linux-gms-pure
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -878,7 +878,7 @@ jobs:
- name: Install Dolt CLI
# TestProxiedServerExternalCreate needs the dolt binary on PATH; the
# shared harness itself uses the dolt sql-server container image.
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |

View file

@ -59,7 +59,7 @@ jobs:
- name: Install Dolt
run: |
curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |

View file

@ -90,7 +90,7 @@ jobs:
- name: Install Dolt CLI
# TestProxiedServerExternalCreate needs the dolt binary on PATH; the
# shared harness itself uses the dolt sql-server container image.
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -303,7 +303,7 @@ jobs:
git config --global user.email "ci@beads.test"
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Dolt
run: |
@ -352,7 +352,7 @@ jobs:
git config --global user.email "ci@beads.test"
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Dolt
run: |

View file

@ -663,7 +663,7 @@ jobs:
restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |
@ -741,7 +741,7 @@ jobs:
cache: false
- name: Install Dolt CLI
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Verify dolt on PATH
run: dolt version
@ -842,7 +842,7 @@ jobs:
restore-keys: beads-go-build-v2-${{ runner.os }}-${{ runner.arch }}-go-${{ steps.setup-go.outputs.go-version }}-base-gms_pure_go-race-
- name: Install Dolt (Linux/macOS)
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Verify dolt on PATH
run: dolt version
@ -895,7 +895,7 @@ jobs:
cache: false
- name: Install Dolt CLI
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Git and Dolt identity
run: |

View file

@ -95,7 +95,7 @@ jobs:
git config --global user.email "ci@beads.test"
- name: Install Dolt
run: curl -fsSL https://github.com/dolthub/dolt/releases/latest/download/install.sh | sudo bash
run: ./scripts/ci/install-dolt.sh
- name: Configure Dolt
run: |

View file

@ -120,6 +120,7 @@ func TestFreshBootstrapHealIncarnation(t *testing.T) {
if facts.bootstrapHeal == nil {
t.Fatal("exact bare CREATE did not return a capability")
}
requireWorkingHardReset(t, ctx, db)
prepareFreshBootstrapV51Dirty(t, ctx, db, "creator_debris")
if _, err := initSchemaOnDBWithBootstrapHeal(
@ -229,6 +230,32 @@ func freshBootstrapIntegrationConfig(beadsDir string, port int, database, pathSu
}
}
// requireWorkingHardReset fails before the heal under test runs when this
// database's CALL DOLT_RESET('--hard') is already broken, so a broken Dolt is
// named as such instead of masquerading as a beads defect.
//
// Dolt 2.3.0 regressed the procedure: roughly one freshly created database in
// twenty comes up with it permanently unusable, answering
// "Error 1105 (HY000): context canceled" to every session for the life of the
// server process (measured 2026-08-20: 2.1.8 0/40, 2.2.0 0/60, 2.3.0 3/60,
// 2.3.1 3/100 fresh databases broken). Without this probe the defect surfaces
// only as the heal's generic #4566 dirty-table refusal, which reads like a
// beads bug and points the reader at "run 'bd dolt commit'" — advice that
// cannot work, because the reset the heal needs is the thing that is broken.
//
// The probe is safe on the database under test: on a pristine, just-created
// database the hard reset is a no-op, and over 60 fresh databases a successful
// first reset never broke a later one.
func requireWorkingHardReset(t *testing.T, ctx context.Context, db *sql.DB) {
t.Helper()
if err := schema.DrainCall(ctx, db, "CALL DOLT_RESET('--hard')"); err != nil {
t.Fatalf("this Dolt server cannot CALL DOLT_RESET('--hard') on a database it just created: %v\n"+
"That is the Dolt 2.3.0 regression, not a beads failure — the bootstrap heal this test "+
"exercises is unimplementable on such a server. Check the pinned CLI version in "+
"scripts/ci/install-dolt.sh and `dolt version`.", err)
}
}
func prepareFreshBootstrapV51Dirty(t *testing.T, ctx context.Context, db *sql.DB, debrisColumn string) {
t.Helper()
conn, err := db.Conn(ctx)

73
scripts/ci/install-dolt.sh Executable file
View file

@ -0,0 +1,73 @@
#!/usr/bin/env bash
set -euo pipefail
# Install a pinned Dolt CLI. Every CI job that needs `dolt` on PATH must go
# through this script instead of piping the upstream install.sh from
# releases/latest, because "latest" silently changes the binary under test.
#
# Keep this version in sync with internal/testutil/testdoltcommon.go:
# DoltDockerImage and scripts/ci/pull-dolt-image.sh. Tests run the CLI (the
# per-test sql-server that doltserver.Start launches) and the container
# side by side against the same databases; letting the two drift means the
# suite is exercising a Dolt pair no release ever shipped.
#
# Why pinned rather than latest: Dolt 2.3.0 (released 2026-08-13) regressed
# CALL DOLT_RESET('--hard') so that roughly one freshly created database in
# twenty comes up with the procedure permanently broken --
# "Error 1105 (HY000): context canceled" on every connection, from any
# session, for the life of the server process. Measured 2026-08-20 by
# creating fresh databases and immediately calling the procedure: 2.1.8 0/40
# broken, 2.2.0 0/60, 2.3.0 3/60, 2.3.1 3/100. That is what made
# TestFreshBootstrapHealIncarnation fail on a coin flip the day
# releases/latest moved to 2.3.x, and it also puts bd dolt compact/flatten
# and the #4566 fresh-bootstrap heal at risk on 2.3.x. Raise this pin only
# once a Dolt release is confirmed clean by that same measurement.
readonly version="2.2.0"
readonly max_attempts=3
readonly retry_delay_seconds=5
arch="$(uname -m)"
case "$arch" in
x86_64 | amd64) arch="amd64" ;;
aarch64 | arm64) arch="arm64" ;;
*)
printf 'Unsupported architecture for pinned dolt install: %s\n' "$arch" >&2
exit 1
;;
esac
os="$(uname -s | tr '[:upper:]' '[:lower:]')"
readonly asset="dolt-${os}-${arch}.tar.gz"
readonly url="https://github.com/dolthub/dolt/releases/download/v${version}/${asset}"
workdir="$(mktemp -d)"
trap 'rm -rf "$workdir"' EXIT
for ((attempt = 1; attempt <= max_attempts; attempt++)); do
if curl -fsSL -o "$workdir/$asset" "$url"; then
break
fi
status=$?
if ((attempt == max_attempts)); then
printf 'Failed to download %s after %d attempts.\n' "$url" "$max_attempts" >&2
exit "$status"
fi
printf 'Failed to download %s (attempt %d/%d); retrying in %d seconds.\n' \
"$url" "$attempt" "$max_attempts" "$retry_delay_seconds" >&2
sleep "$retry_delay_seconds"
done
tar -xzf "$workdir/$asset" -C "$workdir"
sudo install -m 0755 "$workdir/dolt-${os}-${arch}/bin/dolt" /usr/local/bin/dolt
# Fail loudly if PATH resolves to some other dolt: a stale runner-image copy
# earlier on PATH would silently put the suite back on an unpinned binary.
# No `| head` here: pipefail turns dolt's SIGPIPE into a failed install.
installed="$(dolt version)"
installed="${installed%%$'\n'*}"
if [[ "$installed" != *"$version"* ]]; then
printf 'dolt on PATH reports "%s", want version %s (which dolt: %s)\n' \
"$installed" "$version" "$(command -v dolt)" >&2
exit 1
fi
printf 'Installed pinned %s\n' "$installed"

View file

@ -1217,3 +1217,85 @@ func contains(items []string, want string) bool {
}
return false
}
// TestWorkflowsInstallPinnedDolt keeps the Dolt CLI under test pinned. Every
// workflow used to install it by piping dolthub/dolt's releases/latest
// install.sh, so the binary under test changed whenever upstream published —
// including backports, which can move "latest" backwards. When Dolt 2.3.0
// landed it regressed CALL DOLT_RESET('--hard'): roughly one freshly created
// database in twenty comes up with the procedure permanently broken
// ("Error 1105 (HY000): context canceled"), which made
// TestFreshBootstrapHealIncarnation fail on a coin flip. See
// scripts/ci/install-dolt.sh for the per-version measurements. The CLI is now
// pinned to the same release as the container image, so the two halves of
// every server-mode test (the per-test sql-server doltserver.Start launches,
// and the shared container) can never drift apart.
func TestWorkflowsInstallPinnedDolt(t *testing.T) {
workflowDir := filepath.Join(sourceRepoRoot(t), ".github", "workflows")
entries, err := os.ReadDir(workflowDir)
if err != nil {
t.Fatal(err)
}
installers := 0
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".yml" {
continue
}
path := filepath.Join(workflowDir, entry.Name())
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
body := string(data)
if strings.Contains(body, "dolt/releases/latest") {
t.Errorf("%s installs dolt from releases/latest; use ./scripts/ci/install-dolt.sh so the "+
"binary under test is pinned", entry.Name())
}
installers += strings.Count(body, "scripts/ci/install-dolt.sh")
}
if installers == 0 {
t.Fatal("no workflow installs dolt via scripts/ci/install-dolt.sh — the pin is not wired up")
}
}
// TestPinnedDoltCLIMatchesContainerImage keeps the CLI pin and the sql-server
// container pin on the same Dolt release. Server-mode tests run both at once
// against the same databases; a drifting pair tests a combination no release
// ever shipped.
func TestPinnedDoltCLIMatchesContainerImage(t *testing.T) {
root := sourceRepoRoot(t)
installer, err := os.ReadFile(filepath.Join(root, "scripts", "ci", "install-dolt.sh"))
if err != nil {
t.Fatal(err)
}
cliVersion := captureOne(t, `(?m)^readonly version="([0-9]+\.[0-9]+\.[0-9]+)"$`, string(installer), "scripts/ci/install-dolt.sh")
common, err := os.ReadFile(filepath.Join(root, "internal", "testutil", "testdoltcommon.go"))
if err != nil {
t.Fatal(err)
}
imageVersion := captureOne(t, `dolthub/dolt-sql-server:([0-9]+\.[0-9]+\.[0-9]+)`, string(common), "testdoltcommon.go:DoltDockerImage")
pullScript, err := os.ReadFile(filepath.Join(root, "scripts", "ci", "pull-dolt-image.sh"))
if err != nil {
t.Fatal(err)
}
pullVersion := captureOne(t, `dolthub/dolt-sql-server:([0-9]+\.[0-9]+\.[0-9]+)`, string(pullScript), "scripts/ci/pull-dolt-image.sh")
if cliVersion != imageVersion || cliVersion != pullVersion {
t.Errorf("dolt pins disagree: CLI %s, DoltDockerImage %s, pull-dolt-image.sh %s",
cliVersion, imageVersion, pullVersion)
}
}
func captureOne(t *testing.T, pattern, body, source string) string {
t.Helper()
matches := regexp.MustCompile(pattern).FindStringSubmatch(body)
if matches == nil {
t.Fatalf("%s does not match %s", source, pattern)
}
return matches[1]
}