feat(schema): soft watchdog warning for long-running schema migrations (#5997)

* test(feat): red — Add soft watchdog warning for long-running schema migrations (refs be-yyzzs)

* feat: green — Add soft watchdog warning for long-running schema migrations (refs be-yyzzs)

* fix(schema): address review follow-ups on migration watchdog doc citation (refs be-yyzzs, review be-sogif)

Round 1 review (be-sogif) flagged criterion 7: the soft-watchdog comment
at schema.go cited engdocs/plans/be-m65rs-migration-watchdog-scoping.md,
which was never landed to any reachable ref (only exists on PM's own
unmerged worktree branch). Replace the dangling file citation with an
inline summary of the rationale plus reachable citations: the existing
initSchema comment in store.go (migration 0047 as a concrete long-runner
example) and `bd show be-m65rs` for the full scoping decision. Comment-only
change; no functional/behavioral diff. gofmt/build/vet clean; full package
suite reconfirmed 132 PASS, 0 FAIL, 0 SKIP.

* test(schema): red — watchdog is silent off a tty, and ignores bare seconds

Both from review #5997.

TestRunMigrationsWatchdogSurvivesNonTTY drives runMigrations end to end
with a deliberately slow first migration body and a 10ms interval, with
the progress writer put in the state a non-tty caller really sees. It
fails with an empty buffer: runMigrations hands the watchdog the package
stderr var, which is io.Discard whenever os.Stderr is not a terminal, so
the warning is discarded under bd serve, systemd, CI and any piped
invocation — the exact cases where a migration running for twenty minutes
needs to be visible. On a tty it works, which is why this went unnoticed.
Asserting on the wiring rather than on a package variable, so the fix has
to actually reach the call site.

The interval table gains bare-seconds cases. store.go's parseTimeout — the
convention migrationWatchdogIntervalDuration's own doc comment claims to
follow — documents "bare numbers treated as seconds (e.g. \"90\")", and
this one does not, so an operator setting the value to 90 silently gets
5m.

Note the review's own example value, 300, cannot serve as the test input:
300s IS the 5m default, so it passes whether the value was parsed or
discarded. Using 90 and 600 instead, which distinguish.

watchdogStderr exists here but is not yet wired, so the red is behavioural
rather than a compile error.

Refs be-lotba.

* fix(schema): point the watchdog's last dangling citation at bd show be-m65rs

Review #5997 point 1. Round 2 replaced the citation on the call-site
comment but missed the one in runMigrationWithWatchdog's own doc comment,
which still named engdocs/plans/be-m65rs-migration-watchdog-scoping.md.
That path does not exist, and neither does engdocs/plans/ — so the comment
sent a reader after a file that was never committed.

Points it at bd show be-m65rs, matching what the call site already says.
This was the last engdocs/plans reference in the repo; a repo-wide grep for
that prefix is now empty, so the two comments can no longer disagree.

Refs be-lotba.

* fix(schema): send the migration watchdog WARN somewhere a non-tty caller sees

Review #5997 point 2. The package's stderr var is io.Discard whenever
os.Stderr is not a terminal (schema.go:26-37) — it was introduced for
routine per-migration progress lines, where discarding off a tty is
correct. Wiring the watchdog to it inherited that gating, so the WARN was
emitted only on an interactive terminal and silently dropped under bd
serve, systemd, CI and any piped invocation: precisely the situations
where a migration still running after twenty minutes is worth knowing
about. On a tty it worked, which is why it was not caught earlier.

Routes the watchdog through its own writer instead. This is not a new
convention for the package — remote_migrate_gate.go, smart_remote_migrate
_gate.go and both schema-skew notices in this file already write operator
warnings straight to os.Stderr, ungated, for the same reason. stderr keeps
its terminal gating and its progress lines.

Chose the writer over the alternative the review offered (scoping the doc
comment to "interactive runs"), since the feature's stated purpose in
be-m65rs is telling an operator reading logs that a migration is still
working — and logs are the non-tty case.

Turns TestRunMigrationsWatchdogSurvivesNonTTY green; it asserts through
runMigrations rather than on the package var, so the wiring at the call
site is what is under test.

Refs be-lotba.

* fix(schema): accept a bare-seconds watchdog interval, matching parseTimeout

Review #5997 nit. BEADS_MIGRATION_WATCHDOG_INTERVAL=300 fell back to the
5m default instead of meaning 300s, so an operator changing the value got
the number they were trying to change and no diagnostic.

The doc comment claimed to follow internal/storage/dolt/store.go's
timeoutFromEnv, but only its unset/invalid-falls-back shape. timeoutFromEnv
is a one-line wrapper; the contract lives in parseTimeout below it, which
retries an unparsable value as raw+"s" so bare numbers mean seconds. This
function skipped that half, so the comment promised a convention the code
did not implement. Now mirrors parseTimeout's full shape — including its
non-positive handling on both branches — and cites parseTimeout by name,
since that is where the behavior being mirrored actually is.

On the review's related note that d > 0 is the only floor: that is
deliberate and is also parseTimeout's only floor. Diverging here would
make the two disagree again, which is what produced this nit. Happy to add
a minimum on both together if you want one.

Not reused directly because parseTimeout is unexported in another package.

The interval table gains bare-seconds cases; 90 and 600 rather than the
review's 300, since 300s is the 5m default and so cannot distinguish
"parsed as seconds" from "fell back".

Refs be-lotba.

* docs(changelog): add the migration watchdog to Unreleased

Review #5997 nit: a new user-facing env var and a new WARN line earn an

Deliberately CHANGELOG only, not docs/reference/configuration.md — even
though BEADS_FSCK_TIMEOUT, the sibling this env var's parsing mirrors, has
a row in that page's table. docs/ describes the pinned release
(docs/cli-docs.pin is v1.2.2, which does not contain this feature) rather
than main, per engdocs/decisions/2026-07-17-docs-release-pin.md. The table
row belongs with the pin bump at release time; adding it now would
document a flag no released bd accepts.

Refs be-lotba.

* fix(bazel): list migration_watchdog_test.go in the schema go_test srcs

Base added internal/storage/schema/BUILD.bazel after this PR's merge base,
enumerating go_test srcs file by file. The PR's new migration_watchdog_test.go
was therefore absent from it, so under Bazel the target reported
"Executed 1 out of 1 test: 1 test passes" while running NONE of the new tests
("testing: warning: no tests to run") -- a silent green.

Measured both ways on the adopted head: without this entry 0 watchdog tests
run; with it, all 7 tests and 8 subtests execute and pass
(bazel test //internal/storage/schema:schema_test). The enumeration is
maintained at 100% repo-wide -- all 56 unlisted test files are
//go:build-tagged -- so this file would have been the only untagged omission.

Maintainer fixup applied during PR adoption's rebase gate. No contributor
code changed; the contributor could not have seen this file, which did not
exist at the merge base.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(schema): scope the watchdog promise to the migration body, stop the warner on panic

Maintainer fixup applied during PR adoption's review loop (iteration 1),
addressing the one major plus the three non-gating findings in the review
synthesis. No contributor commit was rewritten.

Major -- Change Impact / Blast Radius (scope). The watchdog wraps only
execMigrationBody, while a migration step also runs the dirty-table snapshot,
preMigrationRepair, the cursor INSERT and commitMigrationStep -- and this file
already says the per-step DOLT_ADD/DOLT_COMMIT is "the expensive, fallible part
of this step on the production embedded path". The CHANGELOG therefore
over-promised: it told users a wedged migration no longer "looked identical from
outside", which is false for a wedge in that commit. The synthesis offered two
arms and said explicitly not to do both. Taking the minimal-sufficient arm:
CHANGELOG.md now scopes the guarantee to the migration body and states that a
stall in the per-step commit is still quiet, and a comment at the call site
records that the narrow boundary is deliberate, and why. The extension arm was
declined on cost: widening the wrap means moving the snapshot/commit ordering
that carries this loop's atomic-commit contract (#4566, #4690), under a
strict-ordered sqlmock expectation set and the cross-package
migrateStepFaultHook tests -- more risk than an observability-only change should
carry. Measured: this fix leaves runMigrations at gocyclo 16 / gocognit 29,
byte-identical to before, whereas nesting the whole step in a closure would have
driven the cognitive score well past its current band.

Minor -- Resource Lifecycle & Cleanup. close(done)/wg.Wait() moved into a defer.
A panic out of fn skipped the straight-line close and orphaned the ticker
goroutine, which then went on warning about a migration that was no longer
running, for the life of the process, on the deliberately un-gated
watchdogStderr. Reachability is open rather than proven
(internal/httpapi/server.go:1406 recovers everything but http.ErrAbortHandler
and keeps serving), which is the argument for taking the one-keyword fix. Pinned
by a new test: against the pre-fix schema.go it reds with "3 WARN lines when the
panic unwound, 8 after a further 50ms"; with the defer it passes under -race.

Minor -- Test Evidence Quality. TestMigrationWatchdogIntervalDuration's
"unset_defaults_to_5m" arm never unset anything, so the ambient environment flowed
straight into the call. Two-arm control: with BEADS_MIGRATION_WATCHDOG_INTERVAL=600
exported the old arm reds ("= 10m0s; want 5m0s") and the fixed arm passes, with the
subtest confirmed to have actually run rather than being filtered out.

Nit -- Contract & Interface Fidelity. BEADS_MIGRATION_WATCHDOG_INTERVAL added to
docs/reference/configuration.md's env table, beside the BEADS_FSCK_TIMEOUT row
whose duration contract it mirrors.

Validation: go test ./internal/storage/schema/ -count=1 green (41s); 8 watchdog
tests + 8 subtests PASS, 0 SKIP, 0 FAIL; gofmt clean; go vet clean; git diff
--check clean; ./scripts/check-doc-freshness.sh PASSED.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(bazel): list golang.org/x/term in the schema go_test deps

migration_watchdog_test.go imports golang.org/x/term directly. Under Bazel
it compiled only through embed = [":schema"], but gazelle lists a test
file's own imports in the go_test's deps, so `bazel run //:gazelle`
rewrites this BUILD file. Since the base now gates PRs on the Bazel lanes,
BAZEL_TEST's "BUILD files in sync (gazelle, go_srcs, MODULE.bazel)" step
would fail on exactly this one line. This is the line gazelle produces.

Measured on the adopted head refreshed onto current main: the whole CI sync
step run locally (go_srcs.py --check, bazel run //:gazelle, go_srcs.py,
bazel mod tidy) leaves this line as the only drift in the repository.
With it, bazel test //internal/storage/schema:schema_test --config=ci
passes: 433 test cases, 0 failures, all 16 watchdog cases run (0 skipped),
and tools/bazel/check_testcases.py plus equivalence.py report no schema
divergence.

Maintainer fixup applied during PR adoption's pre-approval CI repair. No
contributor code changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
This commit is contained in:
Jim Wordelman 2026-09-30 20:23:14 -07:00 • committed by GitHub
commit 9407cba7ee
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 449 additions and 1 deletions

View file

@ -447,6 +447,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **A long-running schema migration now says so instead of going quiet**
([#5997](https://github.com/gastownhall/beads/pull/5997)). Migrations are
allowed to take a long time by design — migration 0047's full-table
`is_blocked` recompute is a real example — but until now a slow one and a
wedged one looked identical from outside: `bd` simply stopped printing. A
watchdog now emits a WARN naming the migration's version, name, and elapsed
time once the migration's own SQL has been running past the interval, and
repeats every interval while it keeps running, so an operator reading logs
can tell "still working" from "stopped emitting anything". Coverage is the
migration body specifically — the per-step Dolt commit that follows it on
the production embedded path is outside the watchdog, so a stall there is
still quiet. Set `BEADS_MIGRATION_WATCHDOG_INTERVAL` to change the
5-minute default; it accepts durations like `10m` and bare seconds like `90`,
and falls back to the default when unset or unparsable. This is observability
only and never a circuit breaker: the migration receives the caller's exact
context, its error is passed through unchanged, and nothing is aborted or
rolled back. The warning is deliberately not terminal-gated, so it survives
`bd serve`, systemd, CI, and piped invocations.
- **Auto-backup runs on a managed-local proxied-server workspace.** The
proxied arm of the post-command hook now calls auto-backup, so an explicit
`backup.enabled=true` (or `BD_BACKUP_ENABLED=1`) takes the same throttled,

View file

@ -503,6 +503,7 @@ Selected commonly-used variables:
| `BEADS_ACTOR` | Actor identity (preferred over `BD_ACTOR`, which is a deprecated alias) |
| `BEADS_IDENTITY` | Sender identity for `bd mail` |
| `BEADS_FSCK_TIMEOUT` | Runtime-only timeout for the pre-push `dolt fsck --quiet` integrity check (default `30s`) |
| `BEADS_MIGRATION_WATCHDOG_INTERVAL` | Interval between soft WARN lines while a schema migration's SQL keeps running (default `5m`; accepts durations like `10m` or bare seconds like `90`). Observability only — it never aborts a migration |
| `BEADS_DOLT_SERVER_MODE`, `BEADS_DOLT_SHARED_SERVER`, `BEADS_DOLT_DATA_DIR`, `BEADS_DOLT_PORT`, ... | Embedded/server Dolt overrides |
| `BEADS_DOLT_BIN` | Pin the exact external `dolt` CLI binary managed proxied-server mode spawns, overriding PATH lookup (highest precedence; an explicit path that fails validation is an error, not a silent fallback to PATH). On Windows the executable extension may be omitted — `C:\tools\dolt` finds `C:\tools\dolt.exe` via PATHEXT, though a file at the exact spelled path wins if both exist |

View file

@ -150,6 +150,7 @@ go_test(
"migration_0068_add_attribution_status_test.go",
"migration_0069_datetime_precision_test.go",
"migration_body_drain_test.go",
"migration_watchdog_test.go",
"progress_test.go",
"remote_migrate_gate_test.go",
"schema_skew_test.go",
@ -166,6 +167,7 @@ go_test(
"//internal/testutil",
"@com_github_data_dog_go_sqlmock//:go-sqlmock",
"@com_github_go_sql_driver_mysql//:mysql",
"@org_golang_x_term//:term",
],
)

View file

@ -0,0 +1,308 @@
package schema
import (
"bytes"
"context"
"database/sql"
"errors"
"fmt"
"io"
"os"
"strings"
"sync"
"testing"
"time"
"golang.org/x/term"
)
// TestRunMigrationWithWatchdog_NoWarningUnderThreshold covers be-yyzzs: a
// migration that finishes well inside the threshold must produce zero
// watchdog output — the warning is a "still working" signal, not a
// per-migration progress line (that already exists via runMigrations'
// "Applying migration..."/"done" prints).
func TestRunMigrationWithWatchdog_NoWarningUnderThreshold(t *testing.T) {
var buf bytes.Buffer
err := runMigrationWithWatchdog(context.Background(), &buf, 42, "add_date_indexes", 100*time.Millisecond,
func(ctx context.Context) error { return nil })
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got := buf.String(); got != "" {
t.Errorf("want no watchdog output for a fast migration, got %q", got)
}
}
// TestRunMigrationWithWatchdog_WarnsOnThresholdExceeded covers be-yyzzs: once
// a migration runs past the threshold, a WARN line naming the version, the
// human migration name, and the elapsed time must appear on the writer that
// runMigrations already uses for progress output.
func TestRunMigrationWithWatchdog_WarnsOnThresholdExceeded(t *testing.T) {
var buf bytes.Buffer
err := runMigrationWithWatchdog(context.Background(), &buf, 42, "add_date_indexes", 15*time.Millisecond,
func(ctx context.Context) error {
time.Sleep(40 * time.Millisecond)
return nil
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
got := buf.String()
if !strings.Contains(got, "WARN") {
t.Errorf("want a WARN-level line, got %q", got)
}
if !strings.Contains(got, fmt.Sprintf("%04d", 42)) {
t.Errorf("want the migration version in the warning, got %q", got)
}
if !strings.Contains(got, "add_date_indexes") {
t.Errorf("want the migration name in the warning, got %q", got)
}
if !strings.Contains(got, "still running") {
t.Errorf("want an elapsed-time/still-running marker, got %q", got)
}
}
// TestRunMigrationWithWatchdog_RepeatsWarningEveryInterval covers be-yyzzs:
// a migration that keeps running past multiple threshold intervals must get
// a fresh warning each interval, not just once — so an operator watching
// logs can distinguish "still working" from "stopped emitting anything".
func TestRunMigrationWithWatchdog_RepeatsWarningEveryInterval(t *testing.T) {
var buf bytes.Buffer
err := runMigrationWithWatchdog(context.Background(), &buf, 7, "backfill", 10*time.Millisecond,
func(ctx context.Context) error {
time.Sleep(55 * time.Millisecond)
return nil
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
count := strings.Count(buf.String(), "WARN")
if count < 3 {
t.Errorf("want at least 3 repeated warnings over ~5.5 intervals, got %d in %q", count, buf.String())
}
}
// TestRunMigrationWithWatchdog_ContextUnaffected covers be-yyzzs: the
// watchdog is observability only. It must hand fn the exact ctx it was
// given — no wrapping with a new timeout/cancel — and that ctx must still be
// live (Err() == nil) even after the watchdog has fired multiple warnings.
func TestRunMigrationWithWatchdog_ContextUnaffected(t *testing.T) {
type sentinelKey struct{}
ctx := context.WithValue(context.Background(), sentinelKey{}, "sentinel-value")
var sawValue any
var errDuringRun error
var buf bytes.Buffer
err := runMigrationWithWatchdog(ctx, &buf, 1, "initial", 10*time.Millisecond,
func(fnCtx context.Context) error {
time.Sleep(35 * time.Millisecond)
sawValue = fnCtx.Value(sentinelKey{})
errDuringRun = fnCtx.Err()
return nil
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if sawValue != "sentinel-value" {
t.Errorf("fn did not receive the original ctx (missing sentinel value); got %v", sawValue)
}
if errDuringRun != nil {
t.Errorf("ctx passed to fn was canceled/expired by the watchdog: %v", errDuringRun)
}
if ctx.Err() != nil {
t.Errorf("caller's ctx was canceled/expired by the watchdog: %v", ctx.Err())
}
}
// TestRunMigrationWithWatchdog_ErrorPassthrough covers be-yyzzs: the
// watchdog must never alter the migration's outcome. A migration that runs
// long AND ultimately fails must still surface the exact original error —
// the warning is purely additive, not a substitute for or wrapper around it.
func TestRunMigrationWithWatchdog_ErrorPassthrough(t *testing.T) {
sentinel := errors.New("boom: migration 0007 failed")
var buf bytes.Buffer
err := runMigrationWithWatchdog(context.Background(), &buf, 7, "backfill", 10*time.Millisecond,
func(ctx context.Context) error {
time.Sleep(25 * time.Millisecond)
return sentinel
})
if !errors.Is(err, sentinel) {
t.Errorf("want sentinel error passed through unchanged, got %v", err)
}
if !strings.Contains(buf.String(), "WARN") {
t.Errorf("want the watchdog to still have warned before the failure, got %q", buf.String())
}
}
// syncWriter serializes writes so the panic test below can read what the
// watchdog has emitted so far without racing it. The other tests in this file
// use a bare bytes.Buffer because the watchdog goroutine is always joined
// before they read; the panic case reads at a point where a leaked goroutine
// would still be writing, which is the leak it exists to catch.
type syncWriter struct {
mu sync.Mutex
buf bytes.Buffer
}
func (w *syncWriter) Write(p []byte) (int, error) {
w.mu.Lock()
defer w.mu.Unlock()
return w.buf.Write(p)
}
func (w *syncWriter) String() string {
w.mu.Lock()
defer w.mu.Unlock()
return w.buf.String()
}
// TestRunMigrationWithWatchdog_StopsWarnerWhenFnPanics pins the deferred
// close(done)/wg.Wait(): Dolt's embedded engine can panic out of DDL, and a
// caller that recovers instead of dying (internal/httpapi/server.go's
// middleware recovers everything except http.ErrAbortHandler and keeps
// serving) leaves the process alive afterwards. Stopping the warner on the
// straight-line path after fn returns is skipped by a panic, which orphans the
// ticker goroutine into warning about a migration that is no longer running —
// on the deliberately un-gated watchdogStderr — for the rest of that process's
// lifetime.
func TestRunMigrationWithWatchdog_StopsWarnerWhenFnPanics(t *testing.T) {
const interval = 10 * time.Millisecond
out := &syncWriter{}
func() {
defer func() {
if r := recover(); r == nil {
t.Errorf("want fn's panic to propagate to the caller unchanged, got none")
}
}()
_ = runMigrationWithWatchdog(context.Background(), out, 47, "recompute_mixed_is_blocked", interval,
func(ctx context.Context) error {
time.Sleep(35 * time.Millisecond)
panic("dolt engine panicked out of DDL")
})
}()
// The warner must already be joined by the time the panic reaches here, so
// the line count cannot grow any further.
settled := strings.Count(out.String(), "WARN")
if settled == 0 {
t.Fatalf("want at least one WARN before the panic, otherwise the check below is vacuous; got %q", out.String())
}
time.Sleep(5 * interval)
if after := strings.Count(out.String(), "WARN"); after != settled {
t.Errorf("watchdog goroutine outlived the panicking migration: %d WARN lines when the panic unwound, %d after a further %s", settled, after, 5*interval)
}
}
// TestMigrationWatchdogIntervalDuration covers be-yyzzs: the threshold must
// default to 5 minutes and be overridable via BEADS_MIGRATION_WATCHDOG_INTERVAL,
// following this codebase's existing timeoutFromEnv/BEADS_*_TIMEOUT convention
// (internal/storage/dolt/store.go's cliExecTimeoutEnv/timeoutFromEnv): unset,
// empty, or unparsable values fall back to the default.
func TestMigrationWatchdogIntervalDuration(t *testing.T) {
cases := []struct {
name string
envVal string
setEnv bool
want time.Duration
}{
{name: "unset_defaults_to_5m", setEnv: false, want: 5 * time.Minute},
{name: "empty_defaults_to_5m", setEnv: true, envVal: "", want: 5 * time.Minute},
{name: "valid_duration_string_overrides", setEnv: true, envVal: "10s", want: 10 * time.Second},
{name: "invalid_value_falls_back_to_default", setEnv: true, envVal: "not-a-duration", want: 5 * time.Minute},
// internal/storage/dolt/store.go's parseTimeout — the convention this
// function's doc comment claims to follow — documents "bare numbers
// treated as seconds (e.g. \"90\")". Without this, a plausible
// BEADS_MIGRATION_WATCHDOG_INTERVAL=300 silently means 5m rather than
// the 5m the operator was trying to change.
// NB: not "300" — 300s IS the 5m default, so that value cannot tell
// "parsed as seconds" from "fell back".
{name: "bare_seconds_treated_as_seconds", setEnv: true, envVal: "90", want: 90 * time.Second},
{name: "bare_seconds_larger_than_default", setEnv: true, envVal: "600", want: 10 * time.Minute},
{name: "bare_zero_falls_back_to_default", setEnv: true, envVal: "0", want: 5 * time.Minute},
{name: "negative_falls_back_to_default", setEnv: true, envVal: "-5s", want: 5 * time.Minute},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if tc.setEnv {
t.Setenv(migrationWatchdogIntervalEnv, tc.envVal)
} else {
// The "unset" arm has to actually unset: t.Setenv registers the
// restore of whatever the ambient value was, then os.Unsetenv
// removes it for the duration of the subtest. Without this,
// a shell exporting BEADS_MIGRATION_WATCHDOG_INTERVAL (this
// repo has a history of ambient BEADS_* reddening tests) flows
// straight into the call and the compiled-in default this
// subtest exists to pin is never exercised.
t.Setenv(migrationWatchdogIntervalEnv, "")
os.Unsetenv(migrationWatchdogIntervalEnv)
}
if got := migrationWatchdogIntervalDuration(); got != tc.want {
t.Errorf("migrationWatchdogIntervalDuration() = %v; want %v", got, tc.want)
}
})
}
}
// slowMigrationDB makes the first migration body take long enough for a
// short-interval watchdog to fire, so the wiring between runMigrations and
// runMigrationWithWatchdog can be exercised end to end rather than asserted
// on a package variable.
type slowMigrationDB struct {
mockDB
once sync.Once
d time.Duration
}
func (s *slowMigrationDB) ExecContext(ctx context.Context, q string, a ...any) (sql.Result, error) {
s.once.Do(func() { time.Sleep(s.d) })
return s.mockDB.ExecContext(ctx, q, a...)
}
// TestRunMigrationsWatchdogSurvivesNonTTY covers the review point on #5997:
// the package's progress writer is io.Discard whenever os.Stderr is not a
// terminal, so wiring the watchdog to it makes the warning silent under
// bd serve, systemd, CI, and any piped invocation — precisely the situations
// where a migration that has been running for twenty minutes needs to be
// visible. On a tty it worked fine, which is why this went unnoticed.
func TestRunMigrationsWatchdogSurvivesNonTTY(t *testing.T) {
if term.IsTerminal(int(os.Stderr.Fd())) {
t.Skip("stderr is a terminal here; this test asserts the non-tty path")
}
// Put the progress writer in the state a non-tty caller really sees.
origStderr := stderr
stderr = defaultStderr()
defer func() { stderr = origStderr }()
if stderr != io.Discard {
t.Fatalf("precondition: progress writer off a tty = %T, want io.Discard", stderr)
}
var watchdogBuf bytes.Buffer
origWatchdog := watchdogStderr
watchdogStderr = &watchdogBuf
defer func() { watchdogStderr = origWatchdog }()
origCounter := issueRowCounter
issueRowCounter = func(context.Context, DBConn) (int64, error) { return 0, nil }
defer func() { issueRowCounter = origCounter }()
t.Setenv(migrationWatchdogIntervalEnv, "10ms")
db := &slowMigrationDB{d: 80 * time.Millisecond}
if _, err := runMigrations(context.Background(), db, mainSource, 0, 39, false); err != nil {
t.Fatalf("runMigrations: %v", err)
}
if got := watchdogBuf.String(); !strings.Contains(got, "WARN") {
t.Errorf("no watchdog WARN reached the non-tty writer: the warning is discarded exactly where operators need it. got %q", got)
}
}

View file

@ -36,6 +36,19 @@ func defaultStderr() io.Writer {
return io.Discard
}
// watchdogStderr is where the migration watchdog's WARN lines go. Unlike
// stderr it is deliberately NOT terminal-gated: bd serve, systemd, CI and any
// piped invocation are all non-tty, and they are exactly the cases where a
// migration that has been running for twenty minutes needs to be visible.
// Routing it through stderr would make the warning io.Discard precisely there.
//
// This matches how this package already emits operator warnings that must not
// be swallowed (remote_migrate_gate.go, smart_remote_migrate_gate.go, and the
// skew notices in this file all write to os.Stderr directly); stderr remains
// for the routine per-migration progress lines it was introduced for.
// Overridable in tests.
var watchdogStderr io.Writer = os.Stderr
const largeRigThreshold = 10000
// issueRowCounter returns the current issues-table row count, or an error if
@ -60,6 +73,90 @@ func emitLargeRigNotice(out io.Writer, count int64, err error) {
fmt.Fprintf(out, "Large rig detected (%d issues). This migration may take up to 90 seconds; do not interrupt.\n", count)
}
// migrationWatchdogInterval is the default interval between soft watchdog
// warnings for a single long-running migration.
const migrationWatchdogInterval = 5 * time.Minute
// migrationWatchdogIntervalEnv overrides migrationWatchdogInterval.
const migrationWatchdogIntervalEnv = "BEADS_MIGRATION_WATCHDOG_INTERVAL"
// migrationWatchdogIntervalDuration returns the configured watchdog interval.
// BEADS_MIGRATION_WATCHDOG_INTERVAL overrides the compiled-in default. Valid
// time.ParseDuration strings ("10m", "90s") and bare numbers treated as
// seconds ("90") are both accepted; unset, empty, unparsable, or non-positive
// values fall back to migrationWatchdogInterval. This is the full contract of
// internal/storage/dolt/store.go's parseTimeout, bare-seconds included —
// mirrored rather than reused because that function is unexported in another
// package.
func migrationWatchdogIntervalDuration() time.Duration {
raw := strings.TrimSpace(os.Getenv(migrationWatchdogIntervalEnv))
if raw == "" {
return migrationWatchdogInterval
}
if d, err := time.ParseDuration(raw); err == nil {
if d > 0 {
return d
}
return migrationWatchdogInterval
}
// Bare numbers mean seconds, matching parseTimeout. Without this an
// operator setting the value to 90 gets 5m and no diagnostic.
if d, err := time.ParseDuration(raw + "s"); err == nil && d > 0 {
return d
}
return migrationWatchdogInterval
}
// runMigrationWithWatchdog runs fn — the same call runMigrations already
// makes to execMigrationBody — while a background timer watches elapsed
// time. If fn is still running after interval, it logs a WARN line to out
// naming the migration's version, name, and elapsed time, and repeats every
// additional interval for as long as fn keeps running, so an operator
// watching logs can tell "still working" from "stopped emitting anything".
//
// This is observability only, never a circuit breaker: fn receives exactly
// the ctx this function received (no wrapping timeout/cancel), and fn's
// returned error is passed through unchanged. Migrations are allowed to run
// arbitrarily long by design — store.go's own initSchema comment cites
// migration 0047's full-table is_blocked recompute as a real example — and
// no confirmed production hang exists today. See bd show be-m65rs for the
// full rationale before turning this into a hard abort.
func runMigrationWithWatchdog(ctx context.Context, out io.Writer, version int, name string, interval time.Duration, fn func(ctx context.Context) error) error {
start := time.Now()
done := make(chan struct{})
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-done:
return
case <-ticker.C:
fmt.Fprintf(out, "WARN: migration %04d (%s) still running after %s — watchdog check every %s, this is not an error\n",
version, name, time.Since(start).Round(time.Millisecond), interval)
}
}
}()
// Deferred so the warner is stopped on every exit path, including a panic
// out of fn — Dolt's embedded engine can panic out of DDL, and a caller
// that recovers rather than dying (internal/httpapi/server.go's middleware
// recovers everything except http.ErrAbortHandler and keeps serving) would
// otherwise orphan the ticker goroutine, which goes on writing "still
// running" to the un-gated watchdogStderr for a migration that is no longer
// running. A straight-line close/wait after fn returns leaks in exactly
// that case.
defer func() {
close(done)
wg.Wait()
}()
return fn(ctx)
}
// humanMigrationName turns "0033_add_date_indexes.up.sql" into
// "add_date_indexes" for the progress line.
func humanMigrationName(filename string) string {
@ -1813,7 +1910,28 @@ func runMigrations(ctx context.Context, db DBConn, src migrationSource, minVersi
fmt.Fprintf(stderr, "Applying migration %04d: %s…\n", mf.version, humanMigrationName(mf.name))
start := time.Now()
if err := execMigrationBody(ctx, db, string(data)); err != nil {
// Soft warning only, by design — never turn this into a hard
// timeout/abort. Migrations can legitimately run arbitrarily long
// (see the initSchema comment in internal/storage/dolt/store.go,
// e.g. migration 0047's full-table recompute); aborting mid-DDL
// risks leaving Dolt in a partially-migrated state with no clean
// rollback. Full scoping rationale and rejected alternatives:
// bd show be-m65rs.
//
// The wrap deliberately covers the migration's own SQL body and
// nothing else: the dirty-table snapshot and preMigrationRepair above
// and the cursor INSERT and commitMigrationStep below all run outside
// it, including the per-step DOLT_ADD/DOLT_COMMIT that the comment
// further down calls the expensive, fallible part of this step on the
// production embedded path. Widening the boundary to the whole step
// means moving this loop's snapshot/commit ordering — the atomicity
// contract documented on runMigrations and in commitMigrationStep
// (#4566, #4690) — which is more risk than an observability change
// should carry. So a wedge inside the per-step commit is still silent;
// CHANGELOG.md scopes the user-facing promise to match.
if err := runMigrationWithWatchdog(ctx, watchdogStderr, mf.version, humanMigrationName(mf.name), migrationWatchdogIntervalDuration(),
func(ctx context.Context) error { return execMigrationBody(ctx, db, string(data)) },
); err != nil {
return count, fmt.Errorf("migration %s: %w", mf.name, err)
}
sum := sha256.Sum256(data)