diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index 9d033d94a..439186a24 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -529,6 +529,7 @@ 15 15 TestEmbeddedDependencyEditorRefusesSelfDependencyWithTheProbeSkipped 15 15 TestEmbeddedDependencyEditorRepeatsWithinOneRequestCollapse 15 15 TestEmbeddedIssueOperationsUpdateLabelPatchOrdering +15 15 TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers 15 15 TestEmbeddedMigrateSelfHealsAfterMidPassFault_4566 15 15 TestEmbeddedMoveIssuePersistenceEphemeralNoHistorySmoke 15 15 TestEmbeddedOpenForRemoteSync_DataBehind diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e6aa83e5..91fd0bdd7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -131,6 +131,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `scripts/check-versions.sh` and `bd preflight` list the three that work: with `cmd/bd/version.go` already at the release version, that re-run rewrites only the `.githooks` markers and `uv.lock` and leaves any other drifted file as it was. +- **BREAKING (fail-closed): bd no longer migrates an existing database without + explicit consent** (Beads 1.2 release remediation). The 1.2.0/1.2.1 releases + applied migrations 54–65 at store open on ANY command — including pure reads + like `bd list` — with no prompt, and older binaries then refuse the migrated + database (schema-skew guard). A command that would apply pending + main-sequence migrations to an existing database at store open (in embedded + mode that includes reads) now refuses with guidance instead, and the refused + open performs ZERO writes (dolt_ignore seeding and clone-local migrations + included). Server-mode read commands open without running migrations, so + they keep reading at the current schema. `bd dolt commit` and `bd vc commit` + warn and commit at the current schema without migrating, so the dirty-table + recovery (commit, then `bd migrate schema`) still works on a database that + is both behind and dirty. Consent is any of: `bd migrate schema`, the verb + that names the migration (a preview flag withholds it); `--force` on + `bd migrate` or `bd migrate schema`; or `BD_ALLOW_MIGRATE=1` for scripts/CI + (`BD_ALLOW_REMOTE_MIGRATE=1` is honored as an alias). Bare `bd migrate`, + including `--update-repo-id`, `--inspect` and `--dry-run`, does not consent. + Fresh databases (`bd init`) still migrate to latest — creating a database is + consent for its schema, kept across the retry of an interrupted server + bootstrap — and the same creation principle covers a workspace the + invocation just cloned into existence (`bd init --remote`, `bd bootstrap` + sync), so the #4259 behind-remote flows keep their exact behavior. A + database already at this binary's latest version never consults the gate; + one left at v65 by 1.2.x or at v66 by 1.3.x still has migrations pending + under this release, so it asks once. The remote-backed case keeps all of the + #4259 remote-migrate gate's additional protections (smart gate, adopt + fast-forward, fork-skew analysis) unchanged. JSON consumers: the refusal + renders as a structured error whose `error` states the refusal and whose + `hint` is a non-runnable operator directive (neither names a migration + command or a consent variable), plus a `migrate_consent` object + (`current_version`, `required_version`, `pending`, `severity: "blocking"`, + `human_decision_required: true`, and `options`: `migrate` runs + `bd migrate schema`, or `bd migrate schema --global` under `--global`; + `keep` has no command). +- **Shipped migrations are frozen by CI** (same remediation): the content hash + of every released up-migration (main 0001–0066, ignored 0001–0026, through + the 1.3.x releases) is pinned by `TestShippedMigrationsAreFrozen`. Any + amendment to a released migration must land as a new migration (0070+, or + ignored 0028+, as of this change) so installed schemas and freshly-migrated + ones cannot fork. ### Fixed - **PRs based on `hotfix/**` branches now run full CI, not just diff --git a/README.md b/README.md index 0984804ed..bfa15f1bf 100644 --- a/README.md +++ b/README.md @@ -109,8 +109,8 @@ version: sync remote-backed databases with your current `bd`, back up with `bd export --all`, upgrade the binary, then run `bd info --whats-new`, `bd hooks install`, and `bd version`. If the upgrade crosses a schema migration on a remote-backed database, exactly one designated clone runs -`bd migrate` and `bd dolt push`; other clones install the new binary -and run `bd bootstrap`. See the full +`bd migrate schema` and `bd dolt push`; other clones install the new +binary and run `bd bootstrap`. See the full [upgrade guide](https://beads.gascity.com/getting-started/upgrading) or [docs/getting-started/installation.md](docs/getting-started/installation.md#updating-bd). diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index cb48242fb..b3bfafcc1 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -213,6 +213,7 @@ go_library( "metadata_flag.go", "metrics.go", "migrate.go", + "migrate_consent.go", "migrate_dolt_mode.go", "migrate_hooks.go", "migrate_hooks_apply.go", @@ -935,6 +936,7 @@ go_test( "metadata_ready_test.go", "metadataless_server_workspace_test.go", "metrics_test.go", + "migrate_consent_test.go", "migrate_dolt_mode_frontdoor_integration_test.go", "migrate_dolt_mode_test.go", "migrate_embedded_test.go", diff --git a/cmd/bd/bootstrap.go b/cmd/bd/bootstrap.go index 305989c43..25af0a9d4 100644 --- a/cmd/bd/bootstrap.go +++ b/cmd/bd/bootstrap.go @@ -1246,6 +1246,15 @@ func executeSyncAction(ctx context.Context, plan BootstrapPlan, cfg *configfile. return err } + // Creation-consent for the migration-consent gate + // (schema/migrate_consent.go): this sync action just CLONED the database + // fresh, so the operator's explicit bootstrap is consent for its schema — + // the same principle that exempts a fresh `bd init`. The #4259 + // remote-migrate gate below is unaffected (consent is not its + // designated-migrator override), so the behind-remote refusal keeps its + // exact bd-6dnrw.31 behavior. + schema.SetLocalMigrateConsent(true) + // Open and close the store to ensure dolt_ignore'd wisp tables are // created in the working set. Clone does not include these tables // (they are never committed), so they must be recreated after clone. diff --git a/cmd/bd/init.go b/cmd/bd/init.go index 8e3cf7435..6377f3779 100644 --- a/cmd/bd/init.go +++ b/cmd/bd/init.go @@ -1350,6 +1350,15 @@ Non-interactive mode (--non-interactive or BD_NON_INTERACTIVE=1): syncFromRemote = false } else { bootstrappedFromRemote = true + // Creation-consent for the migration-consent gate + // (schema/migrate_consent.go): this invocation just CLONED the + // database into a workspace that did not exist before, so the + // operator's `bd init --remote` is consent for its schema — + // the same principle that exempts a fresh `bd init`. The + // #4259 remote-migrate gate is unaffected (consent is not its + // designated-migrator override), so the smart-gate / + // designated-migrator flows keep their exact behavior. + schema.SetLocalMigrateConsent(true) if !quiet { fmt.Printf(" %s Bootstrapped from remote: %s\n", ui.RenderPass("✓"), syncURL) } diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 032768c49..7743549a0 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -368,7 +368,8 @@ func isForcedMigrate(cmd *cobra.Command) bool { // schema.SharedConsentCommandForced), so retargeting the bare form would hand // the operator a global-scoped command that still cannot succeed. This mirrors // the retarget in handleRemoteMigrateGateJSON. A nil error keeps the -// pre-existing bare-verb wording. +// pre-existing bare-verb wording, which is also the verb the +// migration-consent refusal prescribes, so that refusal passes nil. func printGlobalDatabaseConsentHint(w io.Writer, e *schema.RemoteMigrateGateError) { if !globalFlag { return @@ -415,13 +416,28 @@ func renderTypedOpenError(err error) bool { } return true } + // The migration-consent gate blocks silent in-place migration of + // ANY existing database (1.2 release remediation) and tells the + // operator how to consent. + var consentErr *schema.MigrateConsentError + if errors.As(err, &consentErr) { + if jsonOutput { + handleMigrateConsentJSON(consentErr) + } else { + fmt.Fprint(os.Stderr, consentErr.UserMessage()) + printGlobalDatabaseConsentHint(os.Stderr, nil) + } + return true + } return false } // isSchemaMigrateVerb reports whether cmd is `bd migrate schema` — the one // invocation in which the operator asked for a schema migration by name. That // request is the consent the shared-store gate wants for a database with no -// remote (#5920); see schema.SetSharedMigrateConsent. +// remote (#5920), and the consent the migration-consent gate wants for any +// existing database; see schema.SetSharedMigrateConsent and +// schema.SetLocalMigrateConsent. // // Deliberately just this one command, not the `bd migrate` tree. Bare // `bd migrate` reconciles version/repo-id/clone-id metadata and never applies @@ -436,6 +452,17 @@ func isSchemaMigrateVerb(cmd *cobra.Command) bool { return cmd == migrateSchemaCmd } +// isMigrateConsentCommand reports whether invoking cmd is consent to apply +// pending schema migrations to an existing database (migrate_consent.go). +// That is `bd migrate schema` alone, the verb that names the migration: bare +// `bd migrate` does not consent, for the reasons isSchemaMigrateVerb gives, +// and --force on either migrate command consents separately, through the gate +// override. A preview withholds the consent, as it does the shared-store one: +// a --dry-run/--inspect must not consent to the work it only inspects. +func isMigrateConsentCommand(cmd *cobra.Command) bool { + return isSchemaMigrateVerb(cmd) && !isPreviewCommand(cmd) +} + // forcedMigratePreviewFlag returns the name of a preview flag (--dry-run, // --inspect) that conflicts with --force on a forced migrate invocation, or "" // when there is no conflict. The combination must be rejected BEFORE the store @@ -1748,6 +1775,12 @@ var rootCmd = &cobra.Command{ // discipline as the --force override above. schema.SetSharedMigrateConsent(isSchemaMigrateVerb(cmd) && !previewMode) + // The same verb carries migration consent on its own + // (migrate_consent.go): every other command refuses to apply pending + // schema migrations to an existing database. Same unconditional + // set-or-clear discipline as the gate override above. + schema.SetLocalMigrateConsent(isMigrateConsentCommand(cmd)) + // Auto-migrate database on version bump (bd-jgxi). // Runs for ALL non-preview commands (including read-only ones) because // the migration opens its own store connection, writes the version diff --git a/cmd/bd/migrate.go b/cmd/bd/migrate.go index db9596c7d..7ac75b2f1 100644 --- a/cmd/bd/migrate.go +++ b/cmd/bd/migrate.go @@ -880,9 +880,15 @@ var migrateSchemaCmd = &cobra.Command{ Short: "Apply pending schema migrations (idempotent)", Long: `Apply pending schema migrations idempotently. -Schema migrations also run automatically on store open, so this subcommand -is typically a no-op. It exists to make migration explicit and observable -in CI, release gates, and recovery scenarios. +bd does not apply pending schema migrations to an existing database without +the operator's consent, and running this command is that consent +(BD_ALLOW_MIGRATE=1 is the scripted form). On a remote-backed database the +remote-migrate gate still applies: --force confirms you are the single +designated migrator. Migrating is one-way: bd releases built for an older +schema refuse the migrated database. + +On an up-to-date database this is a no-op, so it is safe to run in CI, +release gates, and recovery scenarios. Example: bd migrate schema diff --git a/cmd/bd/migrate_consent.go b/cmd/bd/migrate_consent.go new file mode 100644 index 000000000..bd0255e6f --- /dev/null +++ b/cmd/bd/migrate_consent.go @@ -0,0 +1,56 @@ +package main + +import ( + "encoding/json" + "os" + + "github.com/steveyegge/beads/internal/storage/schema" +) + +// handleMigrateConsentJSON renders the migration-consent refusal as a +// structured JSON error block for agent consumption. +// +// It follows handleRemoteMigrateGateJSON's convention, for the same reason: +// migrating is one-way, so neither "error" nor the top-level "hint" carries a +// runnable remedy. "error" states the refusal, "hint" is the non-runnable +// operator-decision directive, and the migrate command lives only inside +// migrate_consent.options[migrate], gated on its precondition and annotated +// with its risk. Handing an agent `bd migrate schema` (or the consent +// environment variable) as "the fix" lets it migrate a database that an older +// bd still has to open; the env var stays out of this payload entirely. +func handleMigrateConsentJSON(e *schema.MigrateConsentError) { + outer := buildJSONError(e.Refusal(), e.AgentDirective()) + if m, ok := outer.(map[string]interface{}); ok { + opts := make([]map[string]interface{}, 0, len(e.Options())) + for _, o := range e.Options() { + commands := make([]string, len(o.Commands)) + for i, c := range o.Commands { + // Under --global the refused open aimed at beads_global, so the + // project-scoped verb would migrate the WRONG database and leave + // the refusal in place. Same retarget as the text path + // (printGlobalDatabaseConsentHint) and the shared-store gate. + if globalFlag && c == schema.SharedConsentCommand { + c = schema.SharedConsentCommandGlobal + } + commands[i] = c + } + opts = append(opts, map[string]interface{}{ + "id": o.ID, + "when": o.When, + "commands": commands, + "risk": o.Risk, + }) + } + m["migrate_consent"] = map[string]interface{}{ + "current_version": e.CurrentVersion, + "required_version": e.LatestVersion, + "pending": e.Pending, + "severity": "blocking", + "human_decision_required": true, + "options": opts, + } + } + encoder := json.NewEncoder(os.Stderr) + encoder.SetIndent("", " ") + _ = encoder.Encode(outer) +} diff --git a/cmd/bd/migrate_consent_test.go b/cmd/bd/migrate_consent_test.go new file mode 100644 index 000000000..02a21cf01 --- /dev/null +++ b/cmd/bd/migrate_consent_test.go @@ -0,0 +1,297 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "testing" + + sqlmock "github.com/DATA-DOG/go-sqlmock" + "github.com/spf13/pflag" + "github.com/steveyegge/beads/internal/storage/schema" +) + +// renderMigrateConsentJSON routes a wrapped consent refusal through +// renderTypedOpenError in --json mode, the way a refused store open reaches +// it, and returns the decoded payload and the raw text. +func renderMigrateConsentJSON(t *testing.T, refusal *schema.MigrateConsentError) (map[string]interface{}, string) { + t.Helper() + pinJSONOutput(t, true) + t.Setenv("BD_JSON_ENVELOPE", "") + + var handled bool + out := captureNoticeStderr(t, func() { + handled = renderTypedOpenError(fmt.Errorf("failed to open database: %w", refusal)) + }) + if !handled { + t.Fatal("renderTypedOpenError did not handle a wrapped MigrateConsentError") + } + var parsed map[string]interface{} + if err := json.Unmarshal([]byte(out), &parsed); err != nil { + t.Fatalf("json.Unmarshal stderr: %v\nstderr was: %s", err, out) + } + return parsed, out +} + +// migrateConsentOptions returns migrate_consent.options keyed by id. +func migrateConsentOptions(t *testing.T, parsed map[string]interface{}) map[string]map[string]interface{} { + t.Helper() + obj, ok := parsed["migrate_consent"].(map[string]interface{}) + if !ok { + t.Fatalf("migrate_consent key missing or wrong type: %T", parsed["migrate_consent"]) + } + rawOpts, ok := obj["options"].([]interface{}) + if !ok { + t.Fatalf("migrate_consent.options missing or wrong type: %T", obj["options"]) + } + byID := map[string]map[string]interface{}{} + for _, raw := range rawOpts { + o, ok := raw.(map[string]interface{}) + if !ok { + t.Fatalf("option has wrong type: %T", raw) + } + id, _ := o["id"].(string) + byID[id] = o + } + return byID +} + +func optionCommands(t *testing.T, o map[string]interface{}) []string { + t.Helper() + raw, ok := o["commands"].([]interface{}) + if !ok { + t.Fatalf("option %v: commands missing or not an array: %T", o["id"], o["commands"]) + } + commands := make([]string, 0, len(raw)) + for _, c := range raw { + s, _ := c.(string) + commands = append(commands, s) + } + return commands +} + +// TestHandleMigrateConsentJSON_Shape pins the agent-facing refusal to the +// remote-migrate gate's convention (#4259). Migrating is one-way, so the +// payload must not hand an agent a runnable migration as "the fix": the error +// names the refusal, the hint is the non-runnable directive, and the migrate +// command appears exactly once, inside the operator-gated migrate option. The +// consent environment variable must not appear at all. +func TestHandleMigrateConsentJSON_Shape(t *testing.T) { + orig := globalFlag + globalFlag = false + t.Cleanup(func() { globalFlag = orig }) + + refusal := &schema.MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + parsed, raw := renderMigrateConsentJSON(t, refusal) + + if got := parsed["error"]; got != refusal.Refusal() { + t.Errorf("error = %v, want the remedy-free refusal %q", got, refusal.Refusal()) + } + if got := parsed["hint"]; got != refusal.AgentDirective() { + t.Errorf("hint = %v, want the directive %q", got, refusal.AgentDirective()) + } + for _, field := range []string{"error", "hint"} { + s, _ := parsed[field].(string) + if strings.Contains(s, schema.SharedConsentCommand) { + t.Errorf("%s must not carry the runnable migration %q: %q", field, schema.SharedConsentCommand, s) + } + } + if strings.Contains(raw, "BD_ALLOW") { + t.Errorf("the payload must not name a consent environment variable:\n%s", raw) + } + if n := strings.Count(raw, schema.SharedConsentCommand); n != 1 { + t.Errorf("%q appears %d times, want exactly once (inside options[migrate]):\n%s", + schema.SharedConsentCommand, n, raw) + } + + obj := parsed["migrate_consent"].(map[string]interface{}) + for key, want := range map[string]float64{"current_version": 65, "required_version": 69, "pending": 4} { + if got, ok := obj[key].(float64); !ok || got != want { + t.Errorf("%s = %v, want %v", key, obj[key], want) + } + } + if got := obj["severity"]; got != "blocking" { + t.Errorf("severity = %v, want \"blocking\"", got) + } + if got := obj["human_decision_required"]; got != true { + t.Errorf("human_decision_required = %v, want true", got) + } + + opts := migrateConsentOptions(t, parsed) + if len(opts) != 2 || opts["migrate"] == nil || opts["keep"] == nil { + t.Fatalf("options = %v, want exactly migrate and keep", opts) + } + if got := optionCommands(t, opts["migrate"]); len(got) != 1 || got[0] != schema.SharedConsentCommand { + t.Errorf("options[migrate].commands = %v, want [%q]", got, schema.SharedConsentCommand) + } + if got := optionCommands(t, opts["keep"]); len(got) != 0 { + t.Errorf("options[keep].commands = %v, want none", got) + } + if risk, _ := opts["migrate"]["risk"].(string); !strings.Contains(risk, "one-way") { + t.Errorf("options[migrate].risk must name the one-way risk: %q", risk) + } + for id, o := range opts { + if when, _ := o["when"].(string); when == "" { + t.Errorf("options[%s].when is empty; every option is gated on its precondition", id) + } + } +} + +// TestHandleMigrateConsentJSON_GlobalRetarget pins the --global retarget: the +// refused open aimed at beads_global, so the project-scoped verb would migrate +// the wrong database and leave the refusal in place. The command lives only +// inside the option, so that is where the retarget has to land. +func TestHandleMigrateConsentJSON_GlobalRetarget(t *testing.T) { + orig := globalFlag + globalFlag = true + t.Cleanup(func() { globalFlag = orig }) + + parsed, raw := renderMigrateConsentJSON(t, &schema.MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4}) + + opts := migrateConsentOptions(t, parsed) + if got := optionCommands(t, opts["migrate"]); len(got) != 1 || got[0] != schema.SharedConsentCommandGlobal { + t.Errorf("options[migrate].commands = %v, want [%q]", got, schema.SharedConsentCommandGlobal) + } + if strings.Contains(raw, `"`+schema.SharedConsentCommand+`"`) { + t.Errorf("--global payload still names the project-scoped %q:\n%s", schema.SharedConsentCommand, raw) + } +} + +// TestRenderTypedOpenErrorMigrateConsentText covers the human path: the full +// UserMessage, plus the --global verb when the invocation targeted the global +// database, since the message itself cannot know which database was refused. +func TestRenderTypedOpenErrorMigrateConsentText(t *testing.T) { + pinJSONOutput(t, false) + orig := globalFlag + t.Cleanup(func() { globalFlag = orig }) + + refusal := &schema.MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + wrapped := fmt.Errorf("failed to open database: %w", refusal) + + render := func() string { + t.Helper() + var handled bool + out := captureNoticeStderr(t, func() { handled = renderTypedOpenError(wrapped) }) + if !handled { + t.Fatal("renderTypedOpenError did not handle a wrapped MigrateConsentError") + } + return out + } + + globalFlag = false + if out := render(); out != refusal.UserMessage() { + t.Errorf("without --global the refusal is the UserMessage alone, got:\n%s", out) + } + + globalFlag = true + out := render() + if !strings.HasPrefix(out, refusal.UserMessage()) { + t.Errorf("under --global the UserMessage must still lead, got:\n%s", out) + } + if !strings.Contains(out, schema.SharedConsentCommandGlobal) { + t.Errorf("under --global the refusal must name %q, got:\n%s", schema.SharedConsentCommandGlobal, out) + } +} + +// TestLocalMigrateConsentScope pins which command lines consent to migrating +// an existing database. Only the verb that names the migration does, plus +// --force on either migrate command. Bare `bd migrate` reconciles metadata and +// never needed the gate open, so letting it consent made +// `bd migrate --update-repo-id` (repo-fingerprint surgery) a one-way schema +// migration as a side effect. +// +// Each line is resolved through the same cobra lookup and fed to the same +// predicates as the root pre-run (isForcedMigrate, isMigrateConsentCommand), +// then the real gate runs against an existing database one migration behind. +func TestLocalMigrateConsentScope(t *testing.T) { + for _, tt := range []struct { + cmdline string + consents bool + }{ + {cmdline: "bd migrate schema", consents: true}, + {cmdline: "bd migrate schema --force", consents: true}, + {cmdline: "bd migrate --force", consents: true}, + {cmdline: "bd migrate", consents: false}, + {cmdline: "bd migrate --update-repo-id", consents: false}, + {cmdline: "bd migrate --inspect", consents: false}, + {cmdline: "bd migrate --dry-run", consents: false}, + {cmdline: "bd list", consents: false}, + } { + t.Run(tt.cmdline, func(t *testing.T) { + err := runLocalConsentGate(t, tt.cmdline, tt.consents) + if tt.consents { + // Programmatic consent short-circuits the gate before its version + // reads, and those reads would fail here with "migrate consent:" + // (the mock expects nothing). Whatever MigrateUp hit past the gate + // is not this test's concern. + if schema.IsMigrateConsentError(err) || (err != nil && strings.Contains(err.Error(), "migrate consent:")) { + t.Fatalf("%q must consent, but the gate ran: %v", tt.cmdline, err) + } + return + } + if !schema.IsMigrateConsentError(err) { + t.Fatalf("%q must not consent; want the consent refusal, got %T: %v", tt.cmdline, err, err) + } + }) + } +} + +// runLocalConsentGate replays the root pre-run's two consent lines for +// cmdline and runs MigrateUp. A consenting line must not reach the gate's +// reads, so its mock expects nothing; a refusing one gets the gate's reads +// for an existing database one migration behind. +func runLocalConsentGate(t *testing.T, cmdline string, consents bool) error { + t.Helper() + t.Setenv(schema.AllowMigrateEnv, "") + t.Setenv(schema.AllowRemoteMigrateEnv, "") + + args := strings.Fields(strings.TrimPrefix(cmdline, "bd ")) + target, rest, err := rootCmd.Find(args) + if err != nil { + t.Fatalf("%q does not resolve to a command: %v", cmdline, err) + } + if err := target.ParseFlags(rest); err != nil { + t.Fatalf("%q: flags do not parse: %v", cmdline, err) + } + t.Cleanup(func() { + // Package-level cobra singletons: a parsed flag would outlive this + // test and silently consent for the next one. + target.Flags().VisitAll(func(f *pflag.Flag) { + if !f.Changed { + return + } + _ = f.Value.Set(f.DefValue) + f.Changed = false + }) + schema.SetForceAllowRemoteMigrate(false) + schema.SetLocalMigrateConsent(false) + }) + + // The two lines the root pre-run runs, verbatim in effect. + schema.SetForceAllowRemoteMigrate(isForcedMigrate(target)) + schema.SetLocalMigrateConsent(isMigrateConsentCommand(target)) + + db, mock, err := sqlmock.New() + if err != nil { + t.Fatal(err) + } + defer db.Close() + if !consents { + for range 2 { + mock.ExpectQuery(`SELECT COUNT\(\*\) FROM information_schema\.tables`). + WithArgs("schema_migrations"). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) + mock.ExpectQuery(`SELECT COALESCE\(MAX\(version\), 0\) FROM schema_migrations`). + WillReturnRows(sqlmock.NewRows([]string{"version"}).AddRow(schema.LatestVersion() - 1)) + } + } + + _, gateErr := schema.MigrateUp(context.Background(), db) + if !consents { + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations for %q: %v", cmdline, err) + } + } + return gateErr +} diff --git a/cmd/bd/store_factory.go b/cmd/bd/store_factory.go index 4b7a0c588..1d7564a54 100644 --- a/cmd/bd/store_factory.go +++ b/cmd/bd/store_factory.go @@ -102,7 +102,8 @@ func newDoltStore(ctx context.Context, cfg *dolt.Config) (s storage.DoltStorage, // Working-set-reconcile commands (bd dolt commit, bd vc commit) must // not be bricked by a pending-migration dirty-table refusal: that // refusal's documented recovery is exactly the commit these commands - // run, so failing the open here would deadlock (#4566). The server + // run, so failing the open here would deadlock (#4566). The same goes + // for the migration-consent refusal that precedes it. The server // arm above honors the same cfg.LenientOpen inside dolt.New; this // branch is the embedded half of one policy, not the whole of it. return embeddeddolt.OpenForWorkingSetReconcile(ctx, cfg.BeadsDir, cfg.Database, "main") diff --git a/docs/docs.json b/docs/docs.json index fbd0bc97b..c0d78f016 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -276,6 +276,7 @@ "recovery/circular-dependencies", "recovery/sync-failures", "recovery/history-squash", + "recovery/unmigrate-schema-v65", "recovery/uninstalling" ] }, diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index a020167a1..89a25474b 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -448,7 +448,7 @@ Upgrade checklist: `bd version` 5. If crossing a schema migration on a remote-backed database, only the designated migrator runs: - `bd migrate` + `bd migrate schema` `bd dolt push` Other clones should install the new binary and run `bd bootstrap`, not diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 814e01902..2da0e3199 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -312,7 +312,7 @@ In server mode, data is managed by the external `dolt sql-server`. ## Migrating databases -After upgrading bd, use `bd migrate` to check for and migrate old database files: +After upgrading bd, check for pending database migrations and apply them: ```bash # Inspect migration plan (AI agents) @@ -324,8 +324,8 @@ bd info --schema --json # Preview migration changes bd migrate --dry-run -# Migrate old databases to beads.db -bd migrate +# Apply pending schema migrations +bd migrate schema # Migrate and clean up old files bd migrate --yes diff --git a/docs/getting-started/upgrading.md b/docs/getting-started/upgrading.md index da9c5330e..717dfbd91 100644 --- a/docs/getting-started/upgrading.md +++ b/docs/getting-started/upgrading.md @@ -34,7 +34,7 @@ bd info --whats-new `bd version` 5. If crossing a schema migration on a remote-backed database, only the designated migrator runs: - `bd migrate` + `bd migrate schema` `bd dolt push` Other clones must not migrate independently. After the migrator pushes, each @@ -139,7 +139,7 @@ bd migrate --inspect --json bd migrate --dry-run # Apply migrations -bd migrate +bd migrate schema # Migrate and clean up old files bd migrate --yes @@ -307,7 +307,7 @@ also copy the `.beads` directory (or `dolt backup` in server mode) while no bd dolt push # 1. CURRENT binary: publish all local work bd export --all -o .beads/backup/pre-migrate.jsonl # 2. backup (see above) # 3. install the new binary (see Upgrading above) -bd migrate # 4. migrate as the designated migrator +bd migrate schema # 4. migrate as the designated migrator bd dolt push # 5. publish the migrated schema bd version # 6. confirm the new version is active ``` @@ -317,10 +317,10 @@ migrating here as the designated migrator, adopting the remote's already-migrate database, or recovering a fork — and asks for an explicit operator decision. Follow the guidance it prints. -For scripted or CI upgrades where nobody reads the output, run `bd migrate` as -an explicit step in exactly one job, never in all of them. If the gate blocks a -run it prints both the available options and the scripted override that fits -the situation. +For scripted or CI upgrades where nobody reads the output, run +`bd migrate schema` as an explicit step in exactly one job, never in all of +them. If the gate blocks a run it prints both the available options and the +scripted override that fits the situation. **Multiple clones sharing one remote:** @@ -332,7 +332,7 @@ bd dolt pull bd export --all -o .beads/backup/pre-migrate.jsonl # 2. Designated migrator ONLY: install the new binary, then migrate and publish. -bd migrate +bd migrate schema bd dolt push # 3. Every OTHER clone, after the migrator has pushed: pull the migrated @@ -538,9 +538,11 @@ bridge below accepts SQLite sources only. ### `.beads/embeddeddolt/`: direct upgrade -Upgrade the binary and run: +Upgrade the binary, apply the pending schema migrations, and update the +database metadata: ```bash +bd migrate schema bd migrate ``` @@ -589,8 +591,8 @@ metadata is missing or leaves `dolt_mode` blank/`embedded`; it does not override an explicit server selection with a local root. Compatibility admission cannot prove that a workspace is modern. If you know it was created by v0.55.4 through v0.62.0, use this explicit bridge even when its witness was lost or damaged. -Otherwise, follow the normal `bd migrate --dry-run` and `bd migrate` flow for -an admitted server workspace. +Otherwise, follow the normal `bd migrate --dry-run` and `bd migrate schema` +flow for an admitted server workspace. ### One `.beads/*.db` file: sealed SQLite bridge @@ -653,7 +655,7 @@ bd hooks install ```bash bd migrate --dry-run -bd migrate +bd migrate schema ``` ### Recovery after upgrade diff --git a/docs/recovery/accidental-1-2-1-release.md b/docs/recovery/accidental-1-2-1-release.md index c5e0f8d47..2e9a7061b 100644 --- a/docs/recovery/accidental-1-2-1-release.md +++ b/docs/recovery/accidental-1-2-1-release.md @@ -118,3 +118,7 @@ database directory, install v1.2.2, then `bd import backup.jsonl`. Caveats: issues deleted after the upgrade come back (import cannot re-delete), and audit events recorded while on v1.2.1 are lost. Most users should prefer the cursor rollback above. + +To rebuild a 1.1.2-compatible workspace from a full export instead, keeping +the migrated original aside, see +[Un-migrate from Schema v65](/recovery/unmigrate-schema-v65). diff --git a/docs/recovery/index.md b/docs/recovery/index.md index 647d6ff0d..b23632602 100644 --- a/docs/recovery/index.md +++ b/docs/recovery/index.md @@ -10,6 +10,7 @@ This section provides step-by-step recovery procedures for common Beads issues. | Issue | Symptoms | Runbook | |-------|----------|---------| | Schema Version Mismatch | `bd` refuses with `schema version mismatch: database is at vNN, binary knows up to vNN` | [Accidental v1.2.1 Release](/recovery/accidental-1-2-1-release) | +| Going Back from v65 | A database bd 1.2.x migrated to v65 has to work with bd 1.1.2 again, rebuilt from a full export | [Un-migrate from Schema v65](/recovery/unmigrate-schema-v65) | | Init Safety Refusals | `bd init` or `bd dolt` refuses with a pattern code like `pk-fork-refused` | [Recovery Playbooks](/recovery/init-safety) | | Database Corruption | Database errors, missing data | [Database Corruption](/recovery/database-corruption) | | Merge Conflicts | Dolt conflicts during sync | [Merge Conflicts](/recovery/merge-conflicts) | diff --git a/docs/recovery/unmigrate-schema-v65.md b/docs/recovery/unmigrate-schema-v65.md new file mode 100644 index 000000000..14cb62d25 --- /dev/null +++ b/docs/recovery/unmigrate-schema-v65.md @@ -0,0 +1,91 @@ +--- +title: Un-migrate from Schema v65 +description: Move a database that bd 1.2.x auto-migrated back to a 1.1.2-compatible workspace, keeping all data +--- + +The 1.2.0/1.2.1 releases migrated existing databases to schema v65 +automatically, on any command — including reads. If your database was +migrated and you want to go back to bd 1.1.2 (whose binaries refuse a v65 +database with `schema version mismatch: database is at v65, binary knows up +to v53`), this runbook rebuilds a 1.1.2-compatible workspace from a full +export. It is written to be executable by an agent as-is. + +There is no in-place down-migration. The procedure is export → set aside → +fresh init → import. If you only need the 1.1 line to open the database +again, the cursor rollback in +[Accidental v1.2.1 Release](/recovery/accidental-1-2-1-release) is shorter +and keeps the Dolt history in place. + +**What survives** (verified): issues with statuses and close reasons, +dependencies, comments, notes, labels, memories, and wisps (with +`--all`). +**What does not**: Dolt commit history and the events/telemetry audit +tables. Both remain readable in the set-aside `.beads.v65.bak` directory — +nothing is deleted. + + +Run every step with the binaries it names: the EXPORT runs under the 1.2.x +binary that matches the database; the INIT and IMPORT run under the 1.1.2 +binary you are returning to. Mixing them up fails closed (schema-skew guard), +but re-running the whole sequence from the top is the only supported retry. + + +## Preconditions + +1. **Stop anything that writes to this workspace** — supervisors, watchers, + hooks on a timer. +2. **Server mode: stop the dolt sql-server** serving this database + (`bd dolt stop`, or stop the process that owns it). For a SHARED server + database, coordinate with the other clients first — after this procedure + they must re-clone or run it too; do not un-migrate a shared database + unilaterally. +3. Install (or locate) a **bd 1.1.2 binary** alongside the current one. GitHub + releases: tag `v1.1.2`. Keep both paths handy; the steps name which one to + use. +4. Run from the directory that CONTAINS `.beads/`. + +## Procedure + +```bash +# 1. Full export, with the 1.2.x binary the database currently matches. +bd export --all -o snapshot.jsonl +# Sanity: the file is non-empty and its line count is plausible for +# your workspace. +wc -l snapshot.jsonl + +# 2. Set the migrated workspace aside. Salvage-first: rename, never delete. +mv .beads .beads.v65.bak + +# 3. Fresh init under the 1.1.2 binary (creates a v53-schema workspace). +# Use the same issue prefix as before. +bd-1.1.2 init --prefix + +# 4. Import the snapshot under the 1.1.2 binary. +bd-1.1.2 import snapshot.jsonl +``` + +## Verify + +```bash +# Counts match your expectations (compare against the export): +bd-1.1.2 list -n 0 --status all | tail -3 +# A known issue round-trips with its comments/labels/deps: +bd-1.1.2 show +# Memories survived: +bd-1.1.2 memories | head +``` + +If anything is missing, the migrated original is intact in `.beads.v65.bak`: +restore it with `mv .beads .beads.failed && mv .beads.v65.bak .beads` and use +a 1.2.x binary with it while you retry. + +## Afterwards + +- Keep `.beads.v65.bak` until you are confident; it holds the Dolt history + and audit tables the import does not carry. +- Pin your install to 1.1.2 until a release whose default does not migrate + your database is available; see the release notes of the version that sent + you here. +- If this workspace had a Dolt remote, the remote still holds the MIGRATED + history. Do not push the rebuilt workspace over it without deciding which + side is canonical; a fresh remote is the simple safe choice. diff --git a/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go b/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go index e8eaa0727..d7beeec97 100644 --- a/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go +++ b/internal/storage/dolt/lenient_open_dirty_tables_integration_test.go @@ -122,6 +122,71 @@ func TestServerModeLenientOpenDirtyTableGate(t *testing.T) { } }) + // The migration-consent refusal fires before the dirty guard, so on a + // database that is both behind and dirty the unconsented commit the + // guard prescribes would rebuild this deadlock one check earlier. + t.Run("behind and dirty database without consent recovers through a lenient open", func(t *testing.T) { + const database = "lenient_consent" + admin := prepareLenientOpenDatabaseAtV51(t, ctx, state.Port, database) + defer admin.Close() + + if _, err := admin.ExecContext(ctx, + "INSERT INTO issues (id, title, description, design, acceptance_criteria, notes) VALUES (?, ?, '', '', '', '')", + database+"-1", "uncommitted issue", + ); err != nil { + t.Fatalf("dirty issues table: %v", err) + } + + schema.SetLocalMigrateConsent(false) + t.Cleanup(func() { schema.SetLocalMigrateConsent(true) }) + t.Setenv(schema.AllowMigrateEnv, "") + t.Setenv(schema.AllowRemoteMigrateEnv, "") + + refuses := func(name string, cfg *Config) { + t.Helper() + s, err := New(ctx, cfg) + if err == nil { + s.Close() + t.Fatalf("New (%s) = nil, want *schema.MigrateConsentError", name) + } + if !schema.IsMigrateConsentError(err) { + t.Fatalf("New (%s) error = %T (%v), want *schema.MigrateConsentError", name, err, err) + } + } + refuses("plain writable open", lenientOpenConfig(beadsDir, state.Port, database)) + syncCfg := lenientOpenConfig(beadsDir, state.Port, database) + syncCfg.RemoteSyncOpen = true + refuses("RemoteSyncOpen", syncCfg) + + lenientCfg := lenientOpenConfig(beadsDir, state.Port, database) + lenientCfg.LenientOpen = true + store, err := New(ctx, lenientCfg) + if err != nil { + t.Fatalf("New (LenientOpen) = %v, want it to continue past the consent refusal", err) + } + defer store.Close() + if got := lenientOpenSchemaVersion(t, ctx, store.db); got != 51 { + t.Fatalf("schema version after lenient open = %d, want 51 (no consent, so no migration)", got) + } + if err := store.Commit(ctx, "checkpoint before migration"); err != nil { + t.Fatalf("Commit: %v", err) + } + if got := lenientOpenDirtyCount(t, ctx, store.db); got != 0 { + t.Fatalf("dolt_status rows after commit = %d, want 0", got) + } + store.Close() + + schema.SetLocalMigrateConsent(true) + migrated, err := New(ctx, lenientOpenConfig(beadsDir, state.Port, database)) + if err != nil { + t.Fatalf("New (consented, post-commit): %v", err) + } + defer migrated.Close() + if got := lenientOpenSchemaVersion(t, ctx, migrated.db); got != schema.LatestVersion() { + t.Fatalf("schema version after consented open = %d, want latest %d", got, schema.LatestVersion()) + } + }) + t.Run("clean database still migrates through a lenient open", func(t *testing.T) { const database = "lenient_clean" admin := prepareLenientOpenDatabaseAtV51(t, ctx, state.Port, database) diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index 2dc8e0a13..e04352b54 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -392,14 +392,15 @@ type Config struct { // would make it eligible to sweep. ClassifiedRead bool - // LenientOpen opens the store leniently: a migration gate refusal (#4259) - // or a dirty-working-set refusal (#4566) skips the migration instead of - // failing the open. Set for working-set-reconcile commands (bd dolt - // commit, bd vc commit; #4566), whose entire purpose is to clear the - // working set that the migration would otherwise refuse to touch. - // Honored in embedded and server mode alike. Migrations still RUN on a - // lenient open — only those two refusals are tolerated — so a lenient - // open of a clean database converges normally. + // LenientOpen opens the store leniently: a migration gate refusal (#4259), + // a dirty-working-set refusal (#4566) or a migration-consent refusal + // skips the migration instead of failing the open. Set for + // working-set-reconcile commands (bd dolt commit, bd vc commit; #4566), + // whose entire purpose is to clear the working set that the migration + // would otherwise refuse to touch. Honored in embedded and server mode + // alike. Migrations still RUN on a lenient open — only those refusals are + // tolerated — so a lenient open of a clean database with consent to + // migrate converges normally. LenientOpen bool // RemoteSyncOpen is LenientOpen's narrow sibling for the #6575 @@ -2261,8 +2262,9 @@ func newServerMode(ctx context.Context, cfg *Config) (*DoltStore, error) { return nil, fmt.Errorf("failed to initialize schema: %w", err) } // A tolerated refusal still reports what the aborted pass - // applied (0 for both guards today, since each refuses before - // migrating), so the rebuild below stays correct either way. + // applied (0 for every tolerated refusal today, since each + // refuses before migrating), so the rebuild below stays correct + // either way. } // initSchema runs migrations over a separate pool (openMigrationDB). // The Ping above already pinned a connection in store.db to the @@ -2300,14 +2302,15 @@ func newServerMode(ctx context.Context, cfg *Config) (*DoltStore, error) { // warnLenientOpenRefusal reports whether a lenient open (Config.LenientOpen) // may continue past err instead of failing, warning on stderr when it may. // -// Server mode reaches the same two pending-migration refusals embedded mode +// Server mode reaches the same pending-migration refusals embedded mode // relaxes for this intent (embeddeddolt's openWorkingSetReconcile): the #4566 // dirty-table guard, whose documented recovery IS the commit these opens exist -// to run, and the #4259 remote-migrate gate, a coordination stop with no -// business blocking a commit of the local working set. Against an external -// server the operator cannot sidestep either by deleting a local database, so -// leaving them fatal here left the refusals with no in-band recovery at all -// (#5781). +// to run; the migration-consent refusal, which fires before that guard and so +// would rebuild the same deadlock on a database both behind and dirty; and +// the #4259 remote-migrate gate, a coordination stop with no business blocking +// a commit of the local working set. Against an external server the operator +// cannot sidestep any of them by deleting a local database, so leaving them +// fatal here left the refusals with no in-band recovery at all (#5781). // // Every other migration failure still fails the open, and the schema-skew and // identity guards run before this point either way: lenient relaxes migration, @@ -2318,10 +2321,15 @@ func warnLenientOpenRefusal(err error) bool { fmt.Fprintf(os.Stderr, "Warning: %v\n"+ " Committing the working set at the current schema; when it completes,\n"+ - " re-run 'bd migrate'.\n", + " re-run 'bd migrate schema'.\n", dirtyErr) return true } + var consentErr *schema.MigrateConsentError + if errors.As(err, &consentErr) { + fmt.Fprint(os.Stderr, consentErr.WorkingSetWarning()) + return true + } var gateErr *schema.RemoteMigrateGateError if errors.As(err, &gateErr) { fmt.Fprintf(os.Stderr, diff --git a/internal/storage/dolt/testmain_test.go b/internal/storage/dolt/testmain_test.go index 12ce7eb7b..2acb6e4af 100644 --- a/internal/storage/dolt/testmain_test.go +++ b/internal/storage/dolt/testmain_test.go @@ -9,6 +9,7 @@ import ( "testing" "github.com/steveyegge/beads/internal/doltserver" + "github.com/steveyegge/beads/internal/storage/schema" "github.com/steveyegge/beads/internal/testutil" "github.com/steveyegge/beads/internal/testutil/credentialcmd" ) @@ -57,6 +58,11 @@ func TestMain(m *testing.M) { } func testMainInner(m *testing.M) int { + // Package-wide migration consent: these tests exercise the machinery that + // runs BELOW the consent gate (migrate_consent.go), exactly as production + // does once an operator has consented. Gate-behavior tests manage the + // consent state themselves. + schema.SetLocalMigrateConsent(true) os.Setenv("BEADS_TEST_MODE", "1") // BEADS_TEST_PDEATHSIG=1 protects TestMultiProcessSchemaInit_DoltVerify // (initschema_multiprocess_test.go), which calls doltserver.Start diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index a2fb5266a..181e37e32 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -176,6 +176,7 @@ go_test( "merge_blockers_test.go", "merge_strategy_test.go", "metadata_cas_contract_test.go", + "migrate_consent_gate_test.go", "migrate_dep_rekey_dedup_test.go", "migrate_events_flip_test.go", "migrate_frozen_guard_convergence_test.go", diff --git a/internal/storage/embeddeddolt/cache.go b/internal/storage/embeddeddolt/cache.go index 72d9166e7..cf7aa5709 100644 --- a/internal/storage/embeddeddolt/cache.go +++ b/internal/storage/embeddeddolt/cache.go @@ -48,8 +48,10 @@ func OpenForReadOnlyCommand(ctx context.Context, beadsDir, database, branch stri // is to reconcile the Dolt working set (bd dolt commit, bd vc commit) must be // able to open the store even when pending migrations touch dirty tables - // otherwise the commit that would clear the dirty state and unblock the -// migration can never run. The returned store is otherwise a normal writable -// store. +// migration can never run. A schema.MigrateConsentError, which fires first on +// a database that is also behind, is skipped the same way +// (toleratesConsentRefusal). The returned store is otherwise a normal +// writable store. func OpenForWorkingSetReconcile(ctx context.Context, beadsDir, database, branch string) (*EmbeddedDoltStore, error) { return openCached(ctx, beadsDir, database, branch, openWorkingSetReconcile) } diff --git a/internal/storage/embeddeddolt/migrate_consent_gate_test.go b/internal/storage/embeddeddolt/migrate_consent_gate_test.go new file mode 100644 index 000000000..2d7860231 --- /dev/null +++ b/internal/storage/embeddeddolt/migrate_consent_gate_test.go @@ -0,0 +1,190 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/storage/schema" + "github.com/steveyegge/beads/internal/types" +) + +// TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers covers +// the #4566 deadlock the migration-consent gate would otherwise rebuild one +// check earlier. On a database that is both behind and dirty, `bd migrate +// schema` consents but is refused by the dirty-table guard, whose remedy is +// `bd dolt commit`; that commit opens the store without consent, and MigrateUp +// checks consent before the dirty guard. The working-set-reconcile open must +// warn and commit at the current schema, strict and read-only opens must keep +// refusing, and the consented migration must then converge. +func TestEmbeddedMigrateConsent_BehindAndDirtyWorkingSetReconcileRecovers(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests") + } + + ctx := t.Context() + beadsDir := filepath.Join(t.TempDir(), ".beads") + dataDir := filepath.Join(beadsDir, "embeddeddolt") + + // Build the fixture under the package-wide consent (TestMain): create + // and commit the database, leave `issues` dirty, and regress the cursor + // to v51 (absolute, for the reasons dirty_tables_gate_test.go gives). + store, err := embeddeddolt.Open(ctx, beadsDir, "testdb", "main") + if err != nil { + t.Fatalf("Open: %v", err) + } + if err := store.SetConfig(ctx, "issue_prefix", "testdb"); err != nil { + store.Close() + t.Fatalf("SetConfig(issue_prefix): %v", err) + } + if err := store.Commit(ctx, "bd init"); err != nil { + store.Close() + t.Fatalf("Commit (init): %v", err) + } + issue := &types.Issue{ + ID: "testdb-1", + Title: "dirty working set issue", + Status: types.StatusOpen, + Priority: 1, + IssueType: types.TypeTask, + } + if err := store.CreateIssue(ctx, issue, "tester"); err != nil { + store.Close() + t.Fatalf("CreateIssue: %v", err) + } + db, cleanup, err := embeddeddolt.OpenSQL(ctx, dataDir, "testdb", "main") + if err != nil { + store.Close() + t.Fatalf("OpenSQL: %v", err) + } + const regressedVersion = 51 + if _, err := db.ExecContext(ctx, + "DELETE FROM schema_migrations WHERE version > ?", regressedVersion); err != nil { + t.Fatalf("regress schema_migrations: %v", err) + } + _ = cleanup() + store.Close() + + // From here on no consent source is set: this is the `bd dolt commit` + // an operator runs on the dirty guard's advice. + schema.SetLocalMigrateConsent(false) + schema.SetForceAllowRemoteMigrate(false) + t.Cleanup(func() { schema.SetLocalMigrateConsent(true) }) + t.Setenv(schema.AllowMigrateEnv, "") + t.Setenv(schema.AllowRemoteMigrateEnv, "") + + type opener func(context.Context, string, string, string) (*embeddeddolt.EmbeddedDoltStore, error) + refuses := func(name string, open opener) { + t.Helper() + s, err := open(ctx, beadsDir, "testdb", "main") + if err == nil { + s.Close() + t.Fatalf("%s = nil, want *schema.MigrateConsentError on a behind database without consent", name) + } + if !schema.IsMigrateConsentError(err) { + t.Fatalf("%s error = %T (%v), want *schema.MigrateConsentError (consent is checked before the dirty guard)", name, err, err) + } + } + before := refusalSnapshot(t, dataDir, "testdb") + refuses("Open", embeddeddolt.Open) + refuses("OpenForReadOnlyCommand", embeddeddolt.OpenForReadOnlyCommand) + if after := refusalSnapshot(t, dataDir, "testdb"); after != before { + t.Fatalf("a refused open wrote to the database:\nbefore:\n%s\nafter:\n%s", before, after) + } + + var reconcileStore *embeddeddolt.EmbeddedDoltStore + var openErr error + warning := captureStderr(t, func() { + reconcileStore, openErr = embeddeddolt.OpenForWorkingSetReconcile(ctx, beadsDir, "testdb", "main") + }) + if openErr != nil { + t.Fatalf("OpenForWorkingSetReconcile = %v, want it to warn and continue past the consent refusal", openErr) + } + if want := "Working-set reconcile command: continuing on schema v51 without"; !strings.Contains(warning, want) { + reconcileStore.Close() + t.Fatalf("warning is missing %q; got:\n%s", want, warning) + } + if current, err := schemaVersion(ctx, dataDir, "testdb"); err != nil { + reconcileStore.Close() + t.Fatalf("read schema version: %v", err) + } else if current != regressedVersion { + reconcileStore.Close() + t.Fatalf("schema version = %d, want %d (no consent, so no migration)", current, regressedVersion) + } + if err := reconcileStore.Commit(ctx, "checkpoint"); err != nil { + reconcileStore.Close() + t.Fatalf("Commit: %v", err) + } + reconcileStore.Close() + + // The commit consented to nothing. + refuses("Open (after commit)", embeddeddolt.Open) + + // The consent `bd migrate schema` records now converges: the dirty guard + // that refused it has nothing left to refuse. + schema.SetLocalMigrateConsent(true) + migrated, err := embeddeddolt.Open(ctx, beadsDir, "testdb", "main") + if err != nil { + t.Fatalf("Open (consented, post-commit): %v", err) + } + defer migrated.Close() + if current, err := schemaVersion(ctx, dataDir, "testdb"); err != nil { + t.Fatalf("read schema version: %v", err) + } else if current != schema.LatestVersion() { + t.Fatalf("schema version after consented reopen = %d, want latest %d", current, schema.LatestVersion()) + } +} + +// refusalSnapshot renders what a refused open must leave untouched: the +// working-set status, the dolt_ignore patterns MigrateUp seeds, and the +// schema cursor. +func refusalSnapshot(t *testing.T, dataDir, database string) string { + t.Helper() + ctx := t.Context() + db, cleanup, err := embeddeddolt.OpenSQL(ctx, dataDir, database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer func() { _ = cleanup() }() + var b strings.Builder + for _, q := range []string{ + "SELECT table_name, staged, status FROM dolt_status ORDER BY table_name, staged", + "SELECT pattern, ignored FROM dolt_ignore ORDER BY pattern", + "SELECT version FROM schema_migrations ORDER BY version", + } { + rows, err := db.QueryContext(ctx, q) + if err != nil { + t.Fatalf("%s: %v", q, err) + } + cols, err := rows.Columns() + if err != nil { + rows.Close() + t.Fatalf("%s: columns: %v", q, err) + } + fmt.Fprintf(&b, "%s\n", q) + for rows.Next() { + vals := make([]any, len(cols)) + for i := range vals { + vals[i] = new(any) + } + if err := rows.Scan(vals...); err != nil { + rows.Close() + t.Fatalf("%s: scan: %v", q, err) + } + for _, v := range vals { + fmt.Fprintf(&b, " %v", *(v.(*any))) + } + b.WriteString("\n") + } + if err := rows.Close(); err != nil { + t.Fatalf("%s: close: %v", q, err) + } + } + return b.String() +} diff --git a/internal/storage/embeddeddolt/store.go b/internal/storage/embeddeddolt/store.go index 4e8a121a9..842717273 100644 --- a/internal/storage/embeddeddolt/store.go +++ b/internal/storage/embeddeddolt/store.go @@ -98,8 +98,9 @@ func lenientGateWarningBody(gateErr *schema.RemoteMigrateGateError) string { // open on any pending-migration refusal; openReadOnlyCommand and // openWorkingSetReconcile relax both the #4259 remote-migrate gate refusal and // the #4566 dirty-table refusal, each with its own warning text (see -// initSchema); openRemoteSync relaxes exactly one gate refusal and nothing -// else. +// initSchema), and openWorkingSetReconcile alone also relaxes the +// migration-consent refusal (toleratesConsentRefusal); openRemoteSync relaxes +// exactly one gate refusal and nothing else. type openIntent int const ( @@ -115,7 +116,9 @@ const ( // commands (bd dolt commit, bd vc commit): their entire purpose is to // clear the dirty working set that a migration would otherwise refuse to // touch, so failing the open here would deadlock the documented recovery - // (#4566). Used by OpenForWorkingSetReconcile. + // (#4566). For the same reason it relaxes the migration-consent refusal, + // which fires before the dirty-table guard. Used by + // OpenForWorkingSetReconcile. openWorkingSetReconcile // openRemoteSync is the same shape of deadlock break as // openWorkingSetReconcile, for the #6575 data-behind gate refusal: that @@ -240,6 +243,27 @@ func (s *EmbeddedDoltStore) toleratesMigrationRefusal() bool { return s.intent == openReadOnlyCommand || s.intent == openWorkingSetReconcile } +// toleratesConsentRefusal reports whether this open's intent may warn and +// continue on the current schema past the migration-consent refusal +// (schema.MigrateConsentError) instead of failing the open. +// +// Only openWorkingSetReconcile may. MigrateUp checks consent before the #4566 +// dirty-table guard, so on a database that is both behind and dirty a strict +// commit open rebuilds the #4566 deadlock one check earlier: `bd migrate +// schema` is refused as dirty and names `bd dolt commit`, which is refused +// for consent. The commit needs no migration, and MigrateUp refuses before +// any write, so it runs on the current schema. +// +// openReadOnlyCommand stays strict on purpose. The remote-migrate gate lets +// reads ride through because its migrate-or-adopt decision is cross-clone +// coordination that takes a while to settle; this decision is one local +// command either way (`bd migrate schema`, or keep the matching release), and +// the refusal is where the operator learns it is needed. openRemoteSync stays +// strict for the reason toleratesMigrationRefusal gives. +func (s *EmbeddedDoltStore) toleratesConsentRefusal() bool { + return s.intent == openWorkingSetReconcile +} + // errClosed is returned when a method is called after Close. var errClosed = errors.New("embeddeddolt: store is closed") @@ -637,6 +661,11 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { // Embedded mode relies on the dolthub/driver/v2's local file/concurrency // controls; schema.MigrateUpWithLock requires a sql-server session lock. if _, err := schema.MigrateUp(ctx, conn); err != nil { + var consentErr *schema.MigrateConsentError + if s.toleratesConsentRefusal() && errors.As(err, &consentErr) { + fmt.Fprint(os.Stderr, consentErr.WorkingSetWarning()) + return nil + } var dirtyErr *schema.DirtyTablesError if s.toleratesMigrationRefusal() && errors.As(err, &dirtyErr) { // The guard exists to keep dirty user data from being entangled @@ -651,13 +680,13 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { fmt.Fprintf(os.Stderr, "Warning: %v\n"+ " Committing the working set at the current schema; when it completes,\n"+ - " re-run 'bd migrate'.\n", + " re-run 'bd migrate schema'.\n", dirtyErr) default: // openReadOnlyCommand fmt.Fprintf(os.Stderr, "Warning: %v\n"+ " Continuing without migrating. Run 'bd dolt commit' to commit the\n"+ - " working set at the current schema, then re-run 'bd migrate'.\n", + " working set at the current schema, then re-run 'bd migrate schema'.\n", dirtyErr) } return nil diff --git a/internal/storage/embeddeddolt/test_fixture_test.go b/internal/storage/embeddeddolt/test_fixture_test.go index 78b71e1a5..8f6820546 100644 --- a/internal/storage/embeddeddolt/test_fixture_test.go +++ b/internal/storage/embeddeddolt/test_fixture_test.go @@ -16,6 +16,7 @@ import ( "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/storage/schema" "github.com/steveyegge/beads/internal/types" ) @@ -49,6 +50,12 @@ func TestMain(m *testing.M) { } embeddedDoltFixtureRoot = root + // Package-wide migration consent: these tests exercise the machinery that + // runs BELOW the consent gate (migrate_consent.go), exactly as production + // does once an operator has consented. Gate-behavior tests manage the + // consent state themselves. + schema.SetLocalMigrateConsent(true) + code := m.Run() if err := os.RemoveAll(root); err != nil && code == 0 { fmt.Fprintln(os.Stderr, "remove embedded Dolt test fixture root:", err) diff --git a/internal/storage/schema/BUILD.bazel b/internal/storage/schema/BUILD.bazel index c3deff533..7d663e2e7 100644 --- a/internal/storage/schema/BUILD.bazel +++ b/internal/storage/schema/BUILD.bazel @@ -13,6 +13,7 @@ go_library( "dirty_tables_error.go", "ignored_cursor_untrack.go", "lock.go", + "migrate_consent.go", "migration_content_hashes.go", "migration_repairs.go", "remote_migrate_gate.go", @@ -146,10 +147,12 @@ go_test( "dolt_ignore_seed_test.go", "ignored_cursor_untrack_test.go", "lock_test.go", + "migrate_consent_test.go", "migration_0067_versioned_beads_test.go", "migration_0068_add_attribution_status_test.go", "migration_0069_datetime_precision_test.go", "migration_body_drain_test.go", + "migration_freeze_test.go", "migration_watchdog_test.go", "progress_test.go", "remote_migrate_gate_test.go", diff --git a/internal/storage/schema/lock.go b/internal/storage/schema/lock.go index e6c104b00..5f6602229 100644 --- a/internal/storage/schema/lock.go +++ b/internal/storage/schema/lock.go @@ -326,7 +326,14 @@ func MigrateUpWithLock(ctx context.Context, conn *sql.Conn, databaseName string, } } - applied, err = MigrateUp(ctx, conn) + // The same authority is consent for MigrateUp's own gate, which is that + // version-only test: an init that never sets local consent would otherwise + // be refused the retry of its own interrupted bootstrap. + migrate := MigrateUp + if o.freshBootstrapHeal != nil { + migrate = migrateUpConsented + } + applied, err = migrate(ctx, conn) var dirtyErr *DirtyTablesError if err != nil && o.freshBootstrapHeal != nil && errors.As(err, &dirtyErr) { // Authorization is checked after the dirty guard fires and while the @@ -347,7 +354,7 @@ func MigrateUpWithLock(ctx context.Context, conn *sql.Conn, databaseName string, if resetErr := drainFreshBootstrapReset(ctx, conn); resetErr != nil { return applied, errors.Join(err, fmt.Errorf("schema: fresh-bootstrap reset: %w", resetErr)) } - applied, err = MigrateUp(ctx, conn) + applied, err = migrate(ctx, conn) } return applied, err } diff --git a/internal/storage/schema/lock_test.go b/internal/storage/schema/lock_test.go index 2b0c1841a..0e926dcb2 100644 --- a/internal/storage/schema/lock_test.go +++ b/internal/storage/schema/lock_test.go @@ -1013,6 +1013,124 @@ func TestMigrateUpWithLockFreshBootstrapHealCapabilityIsOneShot(t *testing.T) { } } +// expectMigrateConsentReads mocks checkMigrateConsent's reads when no consent +// is held: the main cursor once for the fresh-database exemption, and again +// for the pending count. +func expectMigrateConsentReads(mock sqlmock.Sqlmock, cursor int) { + for range 2 { + expectCursorProbe(mock, "schema_migrations", true) + expectScalar(mock, "SELECT COALESCE(MAX(version), 0) FROM schema_migrations", "version", cursor) + } +} + +// TestMigrateUpWithLockFreshBootstrapHealIsMigrateConsent pins that heal +// authority is consent for MigrateUp's own gate, not just for the caller's +// migration gate. A retried server bootstrap finds the non-zero cursor its +// first pass left behind, and server-mode `bd init` records no local consent, +// so a version-only consent check would refuse the init the database it just +// created. Both migration passes are covered; sqlmock's ordered expectations +// fail either case on a consent read. +func TestMigrateUpWithLockFreshBootstrapHealIsMigrateConsent(t *testing.T) { + cases := map[string]func(t *testing.T, mock sqlmock.Sqlmock){ + "partially migrated": func(t *testing.T, mock sqlmock.Sqlmock) { + expectOnePendingMigration(t, mock) + }, + "partially migrated with dirty debris": func(t *testing.T, mock sqlmock.Sqlmock) { + expectDirtyGuardRefusal(t, mock) + expectFreshBootstrapIdentityMatch(mock) + mock.ExpectQuery(regexp.QuoteMeta("CALL DOLT_RESET('--hard')")). + WillReturnRows(sqlmock.NewRows([]string{"status"})) + expectOnePendingMigration(t, mock) + }, + } + for name, expectPasses := range cases { + t.Run(name, func(t *testing.T) { + failOnSwallowedAdvisory(t) + resetConsentState(t) + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("create sql mock: %v", err) + } + defer db.Close() + + ctx := context.Background() + conn, err := db.Conn(ctx) + if err != nil { + t.Fatalf("pin mock connection: %v", err) + } + defer conn.Close() + + lockName := MigrationLockName("testdb") + mock.ExpectQuery(regexp.QuoteMeta("SELECT GET_LOCK(?, ?)")). + WithArgs(lockName, migrationLockAcquireTimeoutSeconds). + WillReturnRows(sqlmock.NewRows([]string{"locked"}).AddRow(1)) + expectPasses(t, mock) + mock.ExpectQuery(regexp.QuoteMeta("SELECT RELEASE_LOCK(?)")). + WithArgs(lockName). + WillReturnRows(sqlmock.NewRows([]string{"released"}).AddRow(1)) + + applied, err := MigrateUpWithLock(ctx, conn, "testdb", + WithFreshBootstrapHeal(testFreshBootstrapHealCapability(), testBootstrapEndpoint)) + if err != nil { + t.Fatalf("MigrateUpWithLock() error = %v", err) + } + if applied != 1 { + t.Fatalf("MigrateUpWithLock() applied = %d, want 1", applied) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + }) + } +} + +// TestMigrateUpWithLockRefusesUnconsentedMigration is the control for the +// test above: the same behind database, opened without heal authority or +// consent, is refused under the lock before any write, and the lock is still +// released. +func TestMigrateUpWithLockRefusesUnconsentedMigration(t *testing.T) { + failOnSwallowedAdvisory(t) + resetConsentState(t) + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("create sql mock: %v", err) + } + defer db.Close() + + ctx := context.Background() + conn, err := db.Conn(ctx) + if err != nil { + t.Fatalf("pin mock connection: %v", err) + } + defer conn.Close() + + lockName := MigrationLockName("testdb") + expectConvergedFastPathMiss(mock, "testdb") + mock.ExpectQuery(regexp.QuoteMeta("SELECT GET_LOCK(?, ?)")). + WithArgs(lockName, migrationLockAcquireTimeoutSeconds). + WillReturnRows(sqlmock.NewRows([]string{"locked"}).AddRow(1)) + expectMigrateConsentReads(mock, LatestVersion()-1) + mock.ExpectQuery(regexp.QuoteMeta("SELECT RELEASE_LOCK(?)")). + WithArgs(lockName). + WillReturnRows(sqlmock.NewRows([]string{"released"}).AddRow(1)) + + applied, err := MigrateUpWithLock(ctx, conn, "testdb", WithDatabaseSelector(testDatabaseSelector)) + if applied != 0 { + t.Fatalf("MigrateUpWithLock() applied = %d, want 0", applied) + } + var consentErr *MigrateConsentError + if !errors.As(err, &consentErr) { + t.Fatalf("MigrateUpWithLock() error = %v, want *MigrateConsentError", err) + } + if consentErr.CurrentVersion != LatestVersion()-1 || consentErr.Pending != 1 { + t.Fatalf("refusal = v%d with %d pending, want v%d with 1 pending", + consentErr.CurrentVersion, consentErr.Pending, LatestVersion()-1) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } +} + // TestMigrateUpWithLockMigrationGate pins where WithMigrationGate's callback // sits: after the lock and after locked preparation (so the gate reads a // prepared, selected database), before MigrateUp (so a refusal applies no diff --git a/internal/storage/schema/migrate_consent.go b/internal/storage/schema/migrate_consent.go new file mode 100644 index 000000000..a8fcc9eba --- /dev/null +++ b/internal/storage/schema/migrate_consent.go @@ -0,0 +1,235 @@ +package schema + +import ( + "context" + "errors" + "fmt" + "os" + "strconv" +) + +// Migration consent (gastownhall Beads 1.2 release remediation). +// +// bd used to auto-apply pending schema migrations the first time a new binary +// opened an existing database — on ANY command, including reads. The 1.2.x +// releases turned that into a trap: users upgraded by a package manager had +// their databases migrated in place with no prompt, and older binaries then +// refuse the migrated database (schema-skew guard). The remote-migrate gate +// (#4259) already refuses the silent migration for remote-backed databases; +// this gate extends the same contract to EVERY existing database: bd never +// initiates a schema migration without explicit operator consent. +// +// Consent is any of: +// - `bd migrate schema` (the verb that names the migration; the root +// command records the intent via SetLocalMigrateConsent before the store +// opens — bare `bd migrate` reconciles metadata and does not consent), +// - --force on `bd migrate` or `bd migrate schema` (the remote-gate +// override, a strict superset), or +// - BD_ALLOW_MIGRATE=1 (scripted/CI form; BD_ALLOW_REMOTE_MIGRATE=1 is +// honored as an alias for continuity with the remote gate's hatch). +// +// Fresh databases (no schema_migrations table, version 0) are exempt: creating +// a database is consent for its schema. A server bootstrap holding +// fresh-bootstrap heal authority keeps that consent across a retry whose +// interrupted first pass left a non-zero cursor (MigrateUpWithLock). The same +// creation principle covers a workspace this invocation just CLONED into +// existence (`bd init --remote`, `bd bootstrap` sync) — those paths record the +// consent explicitly, and the #4259 remote-migrate gate still applies to them +// unchanged. Databases already at this binary's latest version never consult +// the gate. The ignored (clone-local, dolt_ignored) migration sequence is +// deliberately NOT gated on its own: it only materializes per-clone +// bookkeeping tables and runs on every fresh clone — but when the MAIN +// sequence is pending and unconsented, MigrateUp refuses before ANY write, +// ignored sequence and dolt_ignore seeding included. +// +// A refusal fails the open, except for the working-set-reconcile commands +// (bd dolt commit, bd vc commit), which commit at the current schema without +// migrating: the refusal comes before the #4566 dirty-table guard, whose +// documented recovery is that commit (WorkingSetWarning). + +// AllowMigrateEnv, when set to a boolean true ("1", "true", ...), consents to +// applying pending schema migrations to an existing database. It is consulted +// only when migrations are actually pending, so exporting it permanently does +// not warn on every store open. +const AllowMigrateEnv = "BD_ALLOW_MIGRATE" + +// localMigrateConsent is the programmatic consent recorded by +// `bd migrate schema` and by the paths that just cloned the database into +// existence. Process-local by design, like forceAllowRemoteMigrate: it cannot +// leak into child processes. +var localMigrateConsent bool + +// SetLocalMigrateConsent records (or clears) the in-process consent to apply +// pending schema migrations to an existing database. Set by the root command +// when the invoked command is `bd migrate schema`, before both +// autoMigrateOnVersionBump and the main store open. External test packages may +// reset it to false after each test case. +func SetLocalMigrateConsent(v bool) { localMigrateConsent = v } + +// MigrateConsentError is returned when bd would apply pending schema +// migrations to an existing database without operator consent. +type MigrateConsentError struct { + CurrentVersion int + LatestVersion int + Pending int + // UnrecognizedEnv carries a BD_ALLOW_MIGRATE value that was set but not + // understood (only boolean values consent), so a typo'd hatch fails with + // a hint instead of silently staying locked (0054-gate precedent, + // bd-6dnrw.34). + UnrecognizedEnv string +} + +func (e *MigrateConsentError) Error() string { + return fmt.Sprintf("%s — run `bd migrate schema`, or keep using a bd release that matches schema v%d", + e.Refusal(), e.CurrentVersion) +} + +// Refusal states what was refused without prescribing a remedy, for surfaces +// that carry the remedy separately and must not hand an agent a runnable +// migration (cmd/bd's JSON refusal, whose remedies live in Options). +func (e *MigrateConsentError) Refusal() string { + return fmt.Sprintf( + "database schema is at v%d; this bd requires v%d (%d pending migration(s)), and bd no longer migrates a database without explicit consent", + e.CurrentVersion, e.LatestVersion, e.Pending) +} + +// AgentDirective is the non-runnable instruction surfaced to agents in place +// of a ready-to-run migrate command, following the remote-migrate gate's +// convention (RemoteMigrateGateError.AgentDirective). Migrating is one-way, +// and whether anything still has to open this database with an older bd is +// not observable from this process, so the decision goes to the operator. +func (e *MigrateConsentError) AgentDirective() string { + return fmt.Sprintf( + "Operator decision required: this database is at schema v%d and this bd requires v%d. Migrating is ONE-WAY — "+ + "bd releases built for an older schema refuse the migrated database, and whether one still has to open it "+ + "is not observable from this process. Do NOT auto-run the migration or set a consent environment variable "+ + "to get past this refusal: surface migrate_consent.options to the operator and let them choose.", + e.CurrentVersion, e.LatestVersion) +} + +// Options returns the two mutually exclusive paths, migrate or keep the +// current schema, each gated on its precondition and annotated with its risk. +// The migrate command is reachable only through the operator's decision, +// never as a top-level hint. Its verb is the shared-store gate's consent verb +// too, so cmd/bd retargets both with the same --global rewrite. +func (e *MigrateConsentError) Options() []GateOption { + return []GateOption{ + { + ID: "migrate", + When: "the operator chooses to upgrade this database, and nothing that still opens it runs a bd release built for an older schema (confirmed with the operator)", + Commands: []string{SharedConsentCommand}, + Risk: fmt.Sprintf("one-way: v%d -> v%d; bd releases built for an older schema refuse the migrated database", e.CurrentVersion, e.LatestVersion), + }, + { + ID: "keep", + When: fmt.Sprintf("something still has to open this database with a bd release built for schema v%d", e.CurrentVersion), + Commands: []string{}, + Risk: "none to the database; this bd release refuses it until it is migrated, so keep using the release that matches it", + }, + } +} + +// UserMessage renders the full operator-facing refusal. +func (e *MigrateConsentError) UserMessage() string { + msg := fmt.Sprintf( + "This database is at schema v%d; this bd release uses schema v%d.\n"+ + "bd no longer upgrades a database automatically (see the release notes).\n"+ + "\n"+ + " To upgrade this database (one-way; older bd releases cannot read it):\n"+ + " bd migrate schema\n"+ + "\n"+ + " To keep the current schema, keep using the bd release that created it.\n"+ + "\n"+ + " Scripted/CI consent: BD_ALLOW_MIGRATE=1 bd \n", + e.CurrentVersion, e.LatestVersion) + if e.UnrecognizedEnv != "" { + msg += fmt.Sprintf( + "\nNote: %s=%q was set but not understood (use a boolean value, e.g. \"1\" or \"true\"), so it did not consent.\n", + AllowMigrateEnv, e.UnrecognizedEnv) + } + return msg +} + +// WorkingSetWarning renders the stderr warning for a working-set-reconcile +// open (bd dolt commit, bd vc commit) that continues past this refusal. The +// commit needs no migration, and MigrateUp refused before any write, so it +// runs on the current schema; the migration still waits for consent. Both +// storage modes print it (embeddeddolt's openWorkingSetReconcile, server +// mode's LenientOpen). +func (e *MigrateConsentError) WorkingSetWarning() string { + return fmt.Sprintf( + "Warning: %s.\n"+ + " Working-set reconcile command: continuing on schema v%d without\n"+ + " migrating; the commit applies to the working set at the current schema.\n"+ + " To upgrade this database afterwards (one-way): bd migrate schema\n", + e.Refusal(), e.CurrentVersion) +} + +// IsMigrateConsentError reports whether err (or any error it wraps) is a +// *MigrateConsentError. +func IsMigrateConsentError(err error) bool { + var e *MigrateConsentError + return errors.As(err, &e) +} + +// checkMigrateConsent refuses an unconsented migration of an existing +// database. Called at the top of MigrateUp, before any write (dolt_ignore +// seeding included). Returns nil for a fresh database (version 0), a database +// with no pending main-sequence migrations, or when consent was given. +func checkMigrateConsent(ctx context.Context, db DBConn) error { + if localMigrateConsent || forceAllowRemoteMigrate { + return nil // programmatic consent — skip the version reads entirely + } + current, err := mainSource.currentVersion(ctx, db) + if err != nil { + return fmt.Errorf("migrate consent: read current version: %w", err) + } + if current == 0 { + return nil // fresh database — creating it is consent for its schema + } + pending, err := mainSource.pendingVersions(ctx, db) + if err != nil { + return fmt.Errorf("migrate consent: read pending versions: %w", err) + } + return migrateConsentDecision(current, len(pending)) +} + +// migrateConsentDecision is checkMigrateConsent's pure decision core: nil when +// nothing is pending or consent was given (programmatic or env), a typed +// *MigrateConsentError otherwise. current is known to be > 0 here except when +// callers short-circuit earlier; a zero current is still treated as consent +// (fresh database). +func migrateConsentDecision(current, pending int) error { + if current == 0 || pending == 0 { + return nil + } + if localMigrateConsent || forceAllowRemoteMigrate { + return nil + } + unrecognized := "" + for _, env := range []string{AllowMigrateEnv, AllowRemoteMigrateEnv} { + v := os.Getenv(env) + if v == "" { + continue + } + allowed, perr := strconv.ParseBool(v) + if perr != nil { + if env == AllowMigrateEnv { + unrecognized = v + } + continue + } + if allowed { + fmt.Fprintf(defaultStderr(), + "Applying %d pending schema migration(s) (%s=%s): v%d -> v%d\n", + pending, env, v, current, LatestVersion()) + return nil + } + } + return &MigrateConsentError{ + CurrentVersion: current, + LatestVersion: LatestVersion(), + Pending: pending, + UnrecognizedEnv: unrecognized, + } +} diff --git a/internal/storage/schema/migrate_consent_test.go b/internal/storage/schema/migrate_consent_test.go new file mode 100644 index 000000000..40e75b08d --- /dev/null +++ b/internal/storage/schema/migrate_consent_test.go @@ -0,0 +1,164 @@ +package schema + +import ( + "errors" + "os" + "strings" + "testing" +) + +// TestMain grants migration consent for the whole package: the MigrateUp +// machinery tests exercise what runs BELOW the consent gate, exactly as +// production does once an operator has consented. The consent tests below +// clear the grant themselves via resetConsentState. +func TestMain(m *testing.M) { + SetLocalMigrateConsent(true) + os.Exit(m.Run()) +} + +// resetConsentState clears every consent source for one test case (undoing +// TestMain's package-wide grant) and restores it afterwards, so consent tests +// see a clean slate and machinery tests keep their grant. +func resetConsentState(t *testing.T) { + t.Helper() + SetLocalMigrateConsent(false) + SetForceAllowRemoteMigrate(false) + t.Cleanup(func() { + SetLocalMigrateConsent(true) + SetForceAllowRemoteMigrate(false) + }) + t.Setenv(AllowMigrateEnv, "") + t.Setenv(AllowRemoteMigrateEnv, "") +} + +func TestMigrateConsentDecision_FreshDB_Allows(t *testing.T) { + resetConsentState(t) + if err := migrateConsentDecision(0, 12); err != nil { + t.Fatalf("decision = %v, want nil for fresh database (current=0)", err) + } +} + +func TestMigrateConsentDecision_NothingPending_Allows(t *testing.T) { + resetConsentState(t) + if err := migrateConsentDecision(LatestVersion(), 0); err != nil { + t.Fatalf("decision = %v, want nil when nothing is pending", err) + } +} + +func TestMigrateConsentDecision_PendingNoConsent_Refuses(t *testing.T) { + resetConsentState(t) + err := migrateConsentDecision(53, 12) + var consentErr *MigrateConsentError + if !errors.As(err, &consentErr) { + t.Fatalf("decision = %v, want *MigrateConsentError", err) + } + if consentErr.CurrentVersion != 53 || consentErr.Pending != 12 { + t.Fatalf("error fields = v%d/%d pending, want v53/12", consentErr.CurrentVersion, consentErr.Pending) + } + if !IsMigrateConsentError(err) { + t.Fatalf("IsMigrateConsentError(err) = false, want true") + } + msg := consentErr.UserMessage() + for _, want := range []string{"bd migrate schema", AllowMigrateEnv} { + if !strings.Contains(msg, want) { + t.Fatalf("UserMessage missing %q:\n%s", want, msg) + } + } +} + +func TestMigrateConsentDecision_LocalConsent_Allows(t *testing.T) { + resetConsentState(t) + SetLocalMigrateConsent(true) + if err := migrateConsentDecision(53, 12); err != nil { + t.Fatalf("decision = %v, want nil with local consent set", err) + } +} + +func TestMigrateConsentDecision_ForceOverride_Allows(t *testing.T) { + resetConsentState(t) + SetForceAllowRemoteMigrate(true) + if err := migrateConsentDecision(53, 12); err != nil { + t.Fatalf("decision = %v, want nil with the migrate --force override set", err) + } +} + +func TestMigrateConsentDecision_EnvConsent_Allows(t *testing.T) { + for _, env := range []string{AllowMigrateEnv, AllowRemoteMigrateEnv} { + for _, v := range []string{"1", "true", "TRUE"} { + t.Run(env+"="+v, func(t *testing.T) { + resetConsentState(t) + t.Setenv(env, v) + if err := migrateConsentDecision(53, 12); err != nil { + t.Fatalf("decision = %v, want nil with %s=%s", err, env, v) + } + }) + } + } +} + +func TestMigrateConsentDecision_EnvFalse_Refuses(t *testing.T) { + resetConsentState(t) + t.Setenv(AllowMigrateEnv, "0") + if !IsMigrateConsentError(migrateConsentDecision(53, 12)) { + t.Fatalf("want refusal with %s=0", AllowMigrateEnv) + } +} + +func TestMigrateConsentDecision_UnparseableEnv_RefusesWithHint(t *testing.T) { + resetConsentState(t) + t.Setenv(AllowMigrateEnv, "yes-please") + err := migrateConsentDecision(53, 12) + var consentErr *MigrateConsentError + if !errors.As(err, &consentErr) { + t.Fatalf("decision = %v, want *MigrateConsentError", err) + } + if consentErr.UnrecognizedEnv != "yes-please" { + t.Fatalf("UnrecognizedEnv = %q, want the unparseable value surfaced", consentErr.UnrecognizedEnv) + } + if !strings.Contains(consentErr.UserMessage(), "yes-please") { + t.Fatalf("UserMessage does not surface the unparseable env value:\n%s", consentErr.UserMessage()) + } +} + +// TestMigrateConsentError_AgentSurfacesNameNoRemedy pins the #4259 directive +// convention cmd/bd's JSON refusal is built on: Refusal and AgentDirective are +// the top-level fields an agent reads, so neither may carry a runnable +// migration or a consent env var; the migrate command lives only in the +// operator-gated "migrate" option. +func TestMigrateConsentError_AgentSurfacesNameNoRemedy(t *testing.T) { + e := &MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + for name, s := range map[string]string{"Refusal": e.Refusal(), "AgentDirective": e.AgentDirective()} { + for _, unwanted := range []string{SharedConsentCommand, "BD_ALLOW"} { + if strings.Contains(s, unwanted) { + t.Errorf("%s carries %q:\n%s", name, unwanted, s) + } + } + } + opts := e.Options() + if len(opts) != 2 || opts[0].ID != "migrate" || opts[1].ID != "keep" { + t.Fatalf("Options = %+v, want exactly migrate then keep", opts) + } + if len(opts[0].Commands) != 1 || opts[0].Commands[0] != SharedConsentCommand { + t.Errorf("migrate option commands = %q, want [%q]", opts[0].Commands, SharedConsentCommand) + } + if len(opts[1].Commands) != 0 { + t.Errorf("keep option commands = %q, want none", opts[1].Commands) + } +} + +// TestMigrateConsentError_WorkingSetWarning pins the warning a +// working-set-reconcile open prints when it continues past the refusal: it +// states the refusal and that the commit runs on the current schema, and it +// never offers the consent env var as the way forward. +func TestMigrateConsentError_WorkingSetWarning(t *testing.T) { + e := &MigrateConsentError{CurrentVersion: 65, LatestVersion: 69, Pending: 4} + w := e.WorkingSetWarning() + for _, want := range []string{e.Refusal(), "Working-set reconcile command: continuing on schema v65 without"} { + if !strings.Contains(w, want) { + t.Errorf("WorkingSetWarning missing %q:\n%s", want, w) + } + } + if strings.Contains(w, "BD_ALLOW") { + t.Errorf("WorkingSetWarning offers a consent env var:\n%s", w) + } +} diff --git a/internal/storage/schema/migration_freeze_test.go b/internal/storage/schema/migration_freeze_test.go new file mode 100644 index 000000000..678762e11 --- /dev/null +++ b/internal/storage/schema/migration_freeze_test.go @@ -0,0 +1,183 @@ +package schema + +import ( + "crypto/sha256" + "encoding/hex" + "io/fs" + "testing" +) + +// Migration freeze (Beads 1.2 release remediation). +// +// Migrations 0001–0066 (and ignored/ 0001–0026) are SHIPPED: the 1.2.x +// releases published schema v65 to thousands of installs and the 1.3.x +// releases added 0066 and ignored/ 0025–0026, so these files are now the +// de-facto on-disk contract of every migrated database. Amending a shipped +// migration forks installed schemas from freshly-migrated ones — the exact +// skew the smart gate's fork-skew detection exists to catch at runtime. This +// test catches it in CI instead: the content hash of every shipped +// up-migration is pinned; any change must land as a NEW migration, numbered +// after the newest file in its directory. When a release ships new +// migrations, raise the floors below and pin their hashes. +// +// If this test fails because you edited a shipped migration: revert the edit +// and write a new migration. There is no legitimate reason to update a pinned +// hash for an already-released version. + +const ( + frozenMainThrough = 66 + frozenIgnoredThrough = 26 +) + +var frozenMigrationHashes = map[string]string{ + "0001_create_issues.up.sql": "0c9bc4d97e2c5ee5d276c952333b934cd7d7d18210997d6bfd72d17f0fdb14f0", + "0002_create_dependencies.up.sql": "9ca011fa0b24ddf26f689eebf347cc2f80dc90256b099d91b993ec82c4d74da5", + "0003_create_labels.up.sql": "1780f0dc941d97b74af53d5603646d05e83d94eee5a02ac68ccf929200deaec0", + "0004_create_comments.up.sql": "35ac76b3d0b2d6287cad1329c77c8a0562bb6310ba0ac34ae273d72d4bed0e2a", + "0005_create_events.up.sql": "c8cbe4bebbe0f92c8a0ccc2a1a0d11ad442635811c2dab093c34c9341a8499b2", + "0006_create_config.up.sql": "435f5ef129dd4999a9b393c98d633038f0e443280a26e63f7d4315bf025dab1f", + "0007_create_metadata.up.sql": "f92c4d3dd32cae489142de34c4706900736c91cdc53b0d54e87e71656b828dc9", + "0008_create_child_counters.up.sql": "99db9b4b5a785ddd209dff628364c4c1eff549c3ca5ec48a64c604b15ef95bfb", + "0009_create_issue_snapshots.up.sql": "269ef83f083766b7f2fd2dfd2814ef36a585ae4aa18bcab0468a713d48595e48", + "0010_create_compaction_snapshots.up.sql": "cb34cbe311fd89a17d72067f449a65089f2fc9a7c4b3774d54448db5ebc3d659", + "0011_create_repo_mtimes.up.sql": "bde5204466541dddb53a652fe58561c08b6b3c1fa7feb38a0a582d432d76bda6", + "0012_create_routes.up.sql": "68e8ef1fc336e8ca6973f6b927b09d3206a7ebd96db9c4cdceab52fb8498c431", + "0013_create_issue_counter.up.sql": "7ddb30dabe32a66197de78f62400f41d000cf43d0ba8ba2a3943536504dddcdb", + "0014_create_interactions.up.sql": "0e17f1cb9fc5ee5a7c43ca8e02b9fd59814f8fd4372f9ffb17b50c34388614ce", + "0015_create_federation_peers.up.sql": "6aa04b3f66566b243c9a4fc09136af5c17f0ffe947a489245e3c451b995a59b9", + "0016_default_config.up.sql": "5d556c01404eb06acfb9df2b20fde36a2e91bd8a7d14462acbd3c0ae928ea286", + "0017_create_ready_issues_view.up.sql": "4bab7acbe2d8e441f156cc5912f421dcb214b9b08d7958eacc8e3a7318ab5f57", + "0018_create_blocked_issues_view.up.sql": "84dc6a3950a5447e6665e3d98b43f11a7631dc4e53bab1be332d135e51c9511f", + "0019_wisps_dolt_ignore.up.sql": "3df660feb6b64c84ef74c408fe8f5840a2815936534362ed123122f795a21c26", + "0020_create_wisps.up.sql": "bca95662441635454b879bb44ca1e607df9c5efd8fd802be3d53a7371a5ced86", + "0021_create_wisp_auxiliary.up.sql": "d19284a2be70a448db7746ae009cb264dc86dac0d9094aae9fe80a713a08ee70", + "0022_wisp_dep_type_index.up.sql": "fc1375e64620cd4264516718d8f924d88131881f4c00c51471b7630c11b74a45", + "0023_add_no_history_column.up.sql": "d11cd7cb462239e49c0b2f9b93ac73771106d5450ebc3422bb7737748c6f967b", + "0024_create_custom_status_type_tables.up.sql": "816da3feeed531ca9787dca8d4b8072dd94b64d573cf4b6751c61f2390c88a68", + "0025_update_ready_issues_view.up.sql": "09782d195f22d620ba41802856f9eb6e1a10f68bc5bcc8f3616624f8c28b576a", + "0026_update_blocked_issues_view.up.sql": "33daef6b5a07364fc5f59ab59b0f5c677dcf98d9bd0866c7c3bea874f1f01348", + "0027_add_started_at.up.sql": "dd3b58a8bbed4c49e855a6002981a285f10b6094938534dd3da9b121155f24d6", + "0028_local_state_dolt_ignore.up.sql": "24a816406a6f40c66db98697869d172b2b4366b107788fdd00f44b99abee45c3", + "0029_create_local_metadata.up.sql": "b989fc49ab03e06ff6520f150d6958efdbd3f444b69b824f7b4bb283ca6f2b65", + "0030_migrate_local_metadata_keys.up.sql": "8d55135274c1c75e722e802b250911cc574d52a4ba74fbb13edde84f30f9b3a4", + "0031_wisp_events_created_at_index.up.sql": "2d5cf3a6d4245b48c1b893efcc17b9fa25f1ea807e61290cf54064451fc58d93", + "0032_drop_schema_migrations_applied_at.up.sql": "42b86d6b7352f7e158532cc67f4afa142dc2503ce9f780733c06fc9da3478751", + "0033_add_wisp_type_column.up.sql": "9256746bb0d6ad9ed9afba0f0501259f0a3599b7d7da507dd7d808a13c688c75", + "0034_add_spec_id_column.up.sql": "7add95ead971c57618cfbc52e5cb6771d872785bfc035bde12cd27a76358848e", + "0035_migrate_infra_to_wisps.up.sql": "f917fe319540155572f5c51618cbbe20ec8b9b669d5926d68e3a62629c317c5a", + "0036_cleanup_autopush_metadata.up.sql": "e55f1870c2bc031c574b495ebca650365a9c8088d7a8968fb86a19cab81267e8", + "0037_uuid_primary_keys.up.sql": "7526beadee13aa7d6ee2175e9a543210b07c1a4426c382f251ebd31eebe32745", + "0038_drop_hop_columns.up.sql": "002dc637d45840487c35a01ce15cee04872baa45c21f8e20170a4339135a38e7", + "0039_drop_child_counters_fk.up.sql": "db65f241d1e82da41f34b960abc912f78b72e7dc1cb6dd2298974308edf8b768", + "0040_ignored_tables_also_nonlocal_tables.up.sql": "c7068d67ad3e2b4852549e04d97f25814121de83db919e220bdcb9bdd9b61e5e", + "0041_split_dependencies_target.up.sql": "e84ef393576df31550570fd3217579b1880850c0645b7d2b6ad8cf55a18ac1c0", + "0042_add_on_update_cascade.up.sql": "9d0fbb3d02955d25442c4a628ac518236068a97ecf164edbc814216bd5a55adc", + "0043_drop_dependencies_generated_column.up.sql": "548b8a69ab82179b7ee7ab3cc87ef00b3ae28d5a3a4976f42364f6939e384668", + "0044_update_views_drop_depends_on_id.up.sql": "531524854cc0e2361817d1ad94ba9470b0e81e378a5422ffb8bce71f3500adf9", + "0045_update_blocked_view_drop_depends_on_id.up.sql": "395dcbf186deedb592d8ed144e0b7953455efa0ccf02076d3b63e5e5d05dc241", + "0046_add_is_blocked.up.sql": "4d6a5a5fc5e386f2c538b4f3574b82fb2ba025176997953a0a4fbd2a91cddf68", + "0047_recompute_mixed_is_blocked.up.sql": "4aff5a7cc2e1e0250ce4cdb911d010c936681871fb627bceee4b2dc4647dd0a1", + "0048_widen_event_value_columns.up.sql": "093c7a41145508879844d16cbe5acfa69baddbb19bf3e05f00bc08720adea657", + "0049_longtext_large_content_columns.up.sql": "3359343efb7c28e5803de07e9088670489615d151843d7f23705f3ca8dd6a1e8", + "0050_dependencies_deterministic_id.up.sql": "96265ff898753f3bb34549fafa18bbaeb67519705b8cf4e32e67f1058df6841a", + "0051_drop_aux_id_defaults.up.sql": "deb741d5ac5fc1e0cd2e60eb47e84f384ed3455b18d3e2f12cb4e935fe461059", + "0052_add_date_indexes.up.sql": "60ae32bc26c41da2e743c1172f73930d18482b8a98aea08b9de85f6009f9c01d", + "0053_repair_rig_wisps.up.sql": "f13909f0db89f48db694a782d1805eee7ee1b2b2e4dcb3d4b12352333cbcb330", + "0054_add_lease_columns.up.sql": "2e51058ba15be9d87415949bb03c2c7c2f0e0743c5ae0d3e819b617d95f61680", + "0055_move_leases_to_table.up.sql": "2237ddbe13f5a783c65da3bb1292d9a5d89a5f22fddb0c33d2db57f1492df22c", + "0056_add_comments_keyset_index.up.sql": "acc28e9b1aa7272c16c86fcb9fd4a2038ae47729a60b53bb75f3a9df05cd3a82", + "0057_events_value_columns_idempotent_longtext.up.sql": "f86e0f77be2e58affe1c485767b8dd450408d782c0413a41948ca0c0b6d72d06", + "0058_heal_wisp_dependencies_split_constraints.up.sql": "9c1c6d557a03e71cf6b49123831c9f5b5f910fcea3ca51e982ec4acce04ea6ec", + "0059_recompute_null_gate_is_blocked.up.sql": "f0186029a1c87d579b67e0540ec22ad0bd0fc59793b07f7d24a24e122b77c394", + "0060_add_storage_class.up.sql": "40419632ba75d92dfcf08e16e77a780f014172a17068f6021937df9086a24588", + "0061_content_derived_aux_ids.up.sql": "d2d65b92789f137da8ee586067c862a9c7744921ccac421e1b793f42050b62a2", + "0062_events_dolt_ignore.up.sql": "c7de6089997344784a71e7d82ea5e417baaf66320b4533d225a4a6bc3c74222a", + "0063_create_provenance_events.up.sql": "1ef173be07ca58d17cb09ba3b3f5ca0c96edfcace8d52ae3adde2fd27e8248bd", + "0064_create_events_journal.up.sql": "25ea9ad8e849dc7e32e7c5850631bedc63a5c5da3bdc58ccb2762fd7a6c758fc", + "0065_widen_wisp_comments_text.up.sql": "c7341bb977daff92e073db2579e63058cef88d43f1e09b13fa153a772caac2a6", + "0066_add_events_journal_actor.up.sql": "a524d42ba0eced9350edbf6402dc2952957ff8f75a896bc8339c4cbc8596c793", +} + +var frozenIgnoredMigrationHashes = map[string]string{ + "0001_create_local_state_tables.up.sql": "a73e9a486124f89e90d50f4597fd1d7ad08e5d0d654459ec87b7520b47f1bb5f", + "0002_create_wisp_child_counters.up.sql": "c575c6710169f2e697f3aca60f5f698acc8a64af0edb1e2305b9d377fd845f45", + "0003_split_wisp_dependencies_target.up.sql": "f82947deeafe589598e2a1802eddf17712bd322692f924bb5653b0fbd5c46a0d", + "0004_add_wisp_aux_fks.up.sql": "12e18a8bdd214d6f96625e78998433ccc9c2d0ef4d1a35307f2653e725c800b9", + "0005_drop_wisp_dependencies_generated_column.up.sql": "08780b28ddef6c15f5de558ea08d821b52610f810555739ee1d737779986c607", + "0006_add_wisp_is_blocked.up.sql": "4746f10d7040d0b7a9e127437356de87765d12d2d27ee37c9800191e6b619d14", + "0007_recompute_wisp_is_blocked.up.sql": "c3c8d92846a54183cc99afc09f779f6f5ba8030c16ce46b3ee750314bf134627", + "0008_widen_wisp_event_value_columns.up.sql": "51375854ec081d2e0b1434b5ff8b6d87092b891636214ebb9f1dbef22438cc1a", + "0009_aux_row_id_rekey_marker.up.sql": "3d1ee4f9dde3342d110474c8090a30c474edafe2290e4372a3fc4607c1fdc030", + "0010_drop_wisp_id_defaults.up.sql": "59d6644667b7e34f9d23261cb694d461ddd4953969e75519d370b0cce8afd0e8", + "0011_cleanup_orphaned_child_counters.up.sql": "0efade0ecb620420ee42127d3f67230da108e31f2ff3c4281855a8dc9e187ecd", + "0012_create_leases.up.sql": "e128a1301a2bcb967f684b09318aa5cf7ed9adb5f6dd7d412de3d67d1de2b28d", + "0013_add_wisp_row_lock.up.sql": "127a4af7012769e5f016b13b198b7a57ff8386a10f8a68ffb9ff204989564d1b", + "0014_add_wisp_comments_keyset_index.up.sql": "29ad3654218700481b9541331c714ecbbe96dc81c04dc83fba6bf8d35bed7fa4", + "0015_recompute_null_gate_wisp_is_blocked.up.sql": "e761d5c3ceb9610edbb2e1d153e82d235cc7d7f0b87f28a45bec43835e6a4e19", + "0016_add_lease_granted_node.up.sql": "31a55421b78d8c90db6cfba5bf8e99e87d6123271575282f761aa077e5d36386", + "0017_add_wisps_defer_until_index.up.sql": "1f031aaf2037280e62cbf604748a4157bf25e3a60b3392af7d74835c5059bdd8", + "0018_aux_row_id_rekey2_marker.up.sql": "3c2120bf55d15e71a9f96972007c3d914154d17ea37a2b51b44412e3e3df6ef1", + "0019_create_events.up.sql": "db072d67a403e60d5bcbe924b84e473adb09f2326d36ede82fcdab17bb1e2a87", + "0020_add_wisp_storage_class.up.sql": "9ff476e6c73a3a477aec95143d5cbf180517a9484f647447392576f13181a015", + "0021_widen_wisp_large_content_columns.up.sql": "3cfa1e07a3d02eb7f791814d9642a93b9af3bf63728943afa05dcb0e489919c8", + "0022_create_events_journal.up.sql": "3d0adcb9f5833b4ba70fb48f88e1043250796feb7173abe000ea4de85eb8b3a5", + "0023_repair_events_journal_shape.up.sql": "d0edd83ca4cfbc46bd17799c6497a70c8c011d130d08a00c71daf792ee1dcd51", + "0024_widen_wisp_comments_text.up.sql": "69a96330e00643dfeddc4bf6ae9bf3b743a3cfddd86f16dc7b8912758b1b570d", + "0025_add_events_journal_actor.up.sql": "c81ca9edc80c25f6ab4f92d05d6ea412a780b108d1ed83745027440e9723dceb", + "0026_dep_rekey_dedup_marker.up.sql": "dc5c1b9a74e42184451796068ecba1ca11d8ebd0fe7e8c4bd4900d0cfee30446", +} + +func verifyFrozenMigrations(t *testing.T, fsys fs.FS, dir string, frozen map[string]string, frozenThrough int) { + t.Helper() + entries, err := fs.ReadDir(fsys, dir) + if err != nil { + t.Fatalf("reading embedded %s: %v", dir, err) + } + seen := map[string]bool{} + for _, entry := range entries { + if entry.IsDir() { + continue + } + name := entry.Name() + version, err := parseVersion(name) + if err != nil { + t.Fatalf("unparseable migration filename %q: %v", name, err) + } + if version > frozenThrough { + continue // not yet released — frozen when a release ships it + } + want, ok := frozen[name] + if !ok { + t.Errorf("shipped migration %s (v%d <= frozen floor %d) has no pinned hash — a released version was renamed or replaced; restore the original file", name, version, frozenThrough) + continue + } + seen[name] = true + content, err := fs.ReadFile(fsys, dir+"/"+name) + if err != nil { + t.Fatalf("reading embedded %s/%s: %v", dir, name, err) + } + sum := sha256.Sum256(content) + if got := hex.EncodeToString(sum[:]); got != want { + t.Errorf("shipped migration %s was AMENDED (hash %s, pinned %s) — shipped migrations are frozen; land the change as a new migration instead", name, got, want) + } + } + for name := range frozen { + if !seen[name] { + t.Errorf("pinned migration %s is missing from the embedded FS — a released version was deleted or renamed", name) + } + } +} + +// TestShippedMigrationsAreFrozen pins the content of every up-migration that a +// release has shipped. See the file comment for the policy. +func TestShippedMigrationsAreFrozen(t *testing.T) { + verifyFrozenMigrations(t, upMigrations, "migrations", frozenMigrationHashes, frozenMainThrough) +} + +// TestShippedIgnoredMigrationsAreFrozen is the ignored-sequence twin. The +// ignored tables are clone-local, so an amendment does not fork synced +// history — but it DOES make freshly-materialized clones diverge from +// long-lived ones, the same class of bug one plane over. +func TestShippedIgnoredMigrationsAreFrozen(t *testing.T) { + verifyFrozenMigrations(t, upIgnoredMigrations, "migrations/ignored", frozenIgnoredMigrationHashes, frozenIgnoredThrough) +} diff --git a/internal/storage/schema/remote_migrate_gate.go b/internal/storage/schema/remote_migrate_gate.go index 5ca3d2a15..13630f4e9 100644 --- a/internal/storage/schema/remote_migrate_gate.go +++ b/internal/storage/schema/remote_migrate_gate.go @@ -523,7 +523,8 @@ func (e *RemoteMigrateGateError) AgentDirective() string { } } -// GateOption is one conditional remediation path for the remote-migrate gate. +// GateOption is one conditional remediation path for the remote-migrate gate +// (and the migration-consent gate, MigrateConsentError.Options). // It is intentionally conditional (When) and carries its Risk, so an agent // cannot treat any single command as the unconditional fix. type GateOption struct { diff --git a/internal/storage/schema/schema.go b/internal/storage/schema/schema.go index b9dae90b6..29187c54b 100644 --- a/internal/storage/schema/schema.go +++ b/internal/storage/schema/schema.go @@ -269,8 +269,8 @@ type SchemaBehindError struct { } func (e *SchemaBehindError) Error() string { - return fmt.Sprintf("schema version mismatch: database is at v%d, binary expects v%d, and the read-only open cannot migrate it; run any bd write command in that workspace to migrate, or set BD_IGNORE_SCHEMA_SKEW=1 to read anyway (queries touching newer schema may fail)", - e.DBVersion, e.BinaryVersion) + return fmt.Sprintf("schema version mismatch: database is at v%d, binary expects v%d; bd does not migrate a database without explicit consent — run `bd migrate schema` in that workspace to migrate, keep using a bd release that matches schema v%d, or set BD_IGNORE_SCHEMA_SKEW=1 to read anyway (queries touching newer schema may fail)", + e.DBVersion, e.BinaryVersion, e.DBVersion) } // IsSchemaBehindError reports whether err (or any error it wraps) is a @@ -734,6 +734,21 @@ func MigrateUpTo(ctx context.Context, db DBConn, maxVersion int) (int, error) { } func MigrateUp(ctx context.Context, db DBConn) (int, error) { + // Consent gate first, before ANY write — dolt_ignore seeding included: an + // existing database with pending main-sequence migrations must stay + // byte-identical when the operator has not consented (see + // migrate_consent.go). Fresh and already-current databases pass through. + if err := checkMigrateConsent(ctx, db); err != nil { + return 0, err + } + return migrateUpConsented(ctx, db) +} + +// migrateUpConsented is MigrateUp past its consent gate. It exists for the one +// caller whose consent the gate's version test cannot see: an open holding +// fresh-bootstrap heal authority, which proves it created the database it is +// migrating (see MigrateUpWithLock). Every other caller goes through MigrateUp. +func migrateUpConsented(ctx context.Context, db DBConn) (int, error) { // Re-assert the canonical dolt_ignore patterns before anything else, and // in particular before the migrationWorkNeeded short-circuit: a database // whose migration cursors arrived at-latest without executing the seeding diff --git a/internal/storage/uow/team_server_integration_test.go b/internal/storage/uow/team_server_integration_test.go index 2f1df42fc..20615be64 100644 --- a/internal/storage/uow/team_server_integration_test.go +++ b/internal/storage/uow/team_server_integration_test.go @@ -241,7 +241,7 @@ func TestTeamServerMode_Integration(t *testing.T) { require.Error(t, err) assert.Nil(t, p) assert.Contains(t, err.Error(), "bts migrate") - assert.NotContains(t, err.Error(), "run any bd write command", + assert.NotContains(t, err.Error(), "bd migrate schema", "must not suggest bd-driven migration on a bts-owned schema") assert.Equal(t, rolledBack, snapshotMigrations(t, ctx, direct), diff --git a/internal/storage/uow/team_server_schema.go b/internal/storage/uow/team_server_schema.go index d83aca3e1..f91ca3b0d 100644 --- a/internal/storage/uow/team_server_schema.go +++ b/internal/storage/uow/team_server_schema.go @@ -26,7 +26,7 @@ func checkTeamServerSchema(ctx context.Context, conn schema.DBConn, database str case current < latest: // No BD_IGNORE_SCHEMA_SKEW hatch here: it would let a newer bd write // against an older bts schema. Not SchemaBehindError either: its "run - // any bd write command to migrate" advice is wrong for a bts-owned schema. + // `bd migrate schema`" advice is wrong for a bts-owned schema. return fmt.Errorf( "uow: database %q is at schema v%d, this bd expects v%d; the schema is managed by beads-team-server — ask your operator to run 'bts migrate', or use a bd built against schema v%d", database, current, latest, current) diff --git a/internal/storage/uow/team_server_schema_test.go b/internal/storage/uow/team_server_schema_test.go index fb35bcb56..2b866eb23 100644 --- a/internal/storage/uow/team_server_schema_test.go +++ b/internal/storage/uow/team_server_schema_test.go @@ -108,7 +108,7 @@ func TestCheckTeamServerSchema_Behind_RefusesWithBtsMigrate(t *testing.T) { if !strings.Contains(err.Error(), "bts migrate") { t.Errorf("error %q should point at 'bts migrate'", err) } - if strings.Contains(err.Error(), "run any bd write command") { + if strings.Contains(err.Error(), "bd migrate schema") { t.Errorf("error %q must not suggest bd-driven migration on a bts-owned schema", err) } } diff --git a/internal/storage/uow/testmain_test.go b/internal/storage/uow/testmain_test.go index 165b8b644..880451b46 100644 --- a/internal/storage/uow/testmain_test.go +++ b/internal/storage/uow/testmain_test.go @@ -6,6 +6,7 @@ import ( "testing" "github.com/steveyegge/beads/internal/doltserver" + "github.com/steveyegge/beads/internal/storage/schema" "github.com/steveyegge/beads/internal/testutil" ) @@ -33,6 +34,12 @@ func TestMain(m *testing.M) { } func testMainInner(m *testing.M) int { + // Package-wide migration consent: these tests exercise the + // provider/bootstrap machinery that runs BELOW the consent gate + // (schema/migrate_consent.go), exactly as production does once an operator + // has consented. + schema.SetLocalMigrateConsent(true) + // Clear out the roots of earlier runs of this suite whose process is // gone, before claiming one of our own. Roots with no owner marker, and // roots whose owner is still running (a parallel package under diff --git a/scripts/migration-test/historical-dolt-upgrade-test.sh b/scripts/migration-test/historical-dolt-upgrade-test.sh index 927b1388e..9dfb45223 100755 --- a/scripts/migration-test/historical-dolt-upgrade-test.sh +++ b/scripts/migration-test/historical-dolt-upgrade-test.sh @@ -1038,6 +1038,18 @@ run_embedded_dolt_upgrade() { jq -e '.backend == "dolt" and .dolt_mode == "embedded"' "$metadata" >/dev/null || die "$version: source metadata does not select embedded Dolt" metadata_sha=$(sha256_file "$metadata") || die "$version: could not fingerprint source metadata" + # Migration-consent gate (Beads 1.2 remediation): the candidate no longer + # migrates an existing database from ordinary commands, so the explicit + # schema migration is now the FIRST candidate step — previously it ran + # later and was already a no-op because the first candidate open had + # silently migrated. Same effective sequence, now stated. Before + # consenting, assert the gate actually refuses an ordinary command. + if run_in_workspace "$candidate" list >/dev/null 2>"$workspace/consent-refusal.err"; then + die "$version: candidate operated an un-migrated database without consent" + fi + grep -q 'bd migrate schema' "$workspace/consent-refusal.err" || + die "$version: consent refusal did not carry migration guidance" + migrate_schema_current "$version" first verify_surviving_fixture "$version" direct if [ "$version" = v1.0.0 ]; then [ "$(run_in_workspace "$candidate" config get status.custom)" = review:active ] || die "$version: candidate did not preserve custom status config" @@ -1056,7 +1068,6 @@ run_embedded_dolt_upgrade() { blocker=$(sed -n '2p' "$workspace/fixture-ids") run_in_workspace "$candidate" show "$blocker" --json > "$workspace/after-first.json" jq -S . "$workspace/after-first.json" > "$workspace/after-first-canonical.json" - migrate_schema_current "$version" first verify_idempotent_migration "$version" direct run_in_workspace "$candidate" show "$task" --json --include-comments | jq -S . > "$workspace/direct-after-noops.json" @@ -1692,10 +1703,12 @@ run_wisp_plane_upgrade() { # First candidate touch runs the full MigrateUp under the standard per-op # timeout. A dirty-table refusal or a migration error here is the headline - # failure class for this lane. + # failure class for this lane. The migration-consent gate makes an ordinary + # command refuse an un-migrated database, so the explicit schema migration + # is that first touch (as in run_embedded_dolt_upgrade); the open follows. + migrate_schema_current "$version" wisp-first run_in_workspace "$candidate" list --json -n 0 --all > "$workspace/wisp-first-open.json" || die "$version: candidate could not open the seeded wisp workspace" - migrate_schema_current "$version" wisp-first snapshot_wisp_plane "$version" after # Read-only assertions first: the events count must be observed before any @@ -1767,8 +1780,10 @@ run_dirty_tracked_wisp_scenario() { before=$(oracle_count "$version" 'SELECT COUNT(*) FROM wisps') output="$workspace/dirty-tracked-open.out" # run_in_workspace already wraps every call in `timeout`, so a hard-lock - # surfaces as a timeout kill rather than hanging the job. - if run_in_workspace "$candidate" list --json -n 0 --all > "$output" 2>&1; then + # surfaces as a timeout kill rather than hanging the job. The open is the + # consented `migrate schema`: an ordinary command now stops at the + # migration-consent gate before MigrateUp ever reaches the dirty tables. + if run_in_workspace "$candidate" migrate schema > "$output" 2>&1; then printf ' · candidate accepted the dirty tracked wisp tables\n' else grep -Fq 'dirty tables' "$output" || diff --git a/scripts/upgrade-smoke-test.sh b/scripts/upgrade-smoke-test.sh index 52b05fbf2..256284339 100755 --- a/scripts/upgrade-smoke-test.sh +++ b/scripts/upgrade-smoke-test.sh @@ -223,6 +223,17 @@ cand() { (cd "$WS" && "$CAND_BIN" "$@"); } prev_init() { prev init --quiet --non-interactive --skip-hooks --skip-agents; } cand_init() { cand init --quiet --non-interactive --skip-hooks --skip-agents; } +# Upgrade an existing workspace to the candidate. Since the migration-consent +# gate (Beads 1.2 release remediation) the candidate never migrates an +# existing database from ordinary commands — `bd migrate schema` is the +# consent step every real upgrade now includes, so the smoke upgrade includes +# it too. Tolerates a workspace the old binary could not initialize (the +# migrate is a no-op failure there and cand_init creates a fresh database). +cand_upgrade() { + cand migrate schema >/dev/null 2>&1 || true + cand_init 2>/dev/null || true +} + # Create an issue with the previous binary, tolerating missing --silent flag. # Older binaries don't have --silent; we just need to know creation succeeded. # Sets _CREATED_ID to a non-empty value on success. @@ -260,8 +271,27 @@ prev_create --title "Pre-upgrade issue" --type task --priority 1 || true ID1="${_CREATED_ID}" prev_create --title "Another issue" --type bug || true -# Upgrade: run candidate init (simulates upgrade) -cand_init 2>/dev/null || true +# Migration-consent gate: before consenting, an ordinary candidate command on +# the old-schema database must either work as-is (previous release already +# ships the candidate's schema — nothing pending) or refuse WITH the consent +# guidance. A failure without the guidance is a regression. Graded only when +# the old binary left a database to gate: releases before v1.0 have no +# `init --non-interactive`, so prev_init creates nothing there. +if ! embedded_db_exists; then + pass "Consent gate check skipped (old binary could not initialize a database)" +else + GATE_OUT=$(cand list 2>&1) && GATE_RC=0 || GATE_RC=$? + if [ "$GATE_RC" -eq 0 ]; then + pass "Candidate operates database directly (schema already current)" + elif echo "$GATE_OUT" | grep -q "bd migrate schema"; then + pass "Candidate refuses un-migrated database with consent guidance" + else + fail "Candidate failed on old database without consent guidance" + fi +fi + +# Upgrade: consent + candidate init (simulates upgrade) +cand_upgrade # Verify ROLE=$(git -C "$WS" config --get beads.role 2>/dev/null || echo "MISSING") @@ -307,7 +337,7 @@ prev_init 2>/dev/null || true git -C "$WS" config beads.role contributor # Upgrade -cand_init 2>/dev/null || true +cand_upgrade ROLE=$(git -C "$WS" config --get beads.role 2>/dev/null || echo "MISSING") if [ "$ROLE" = "contributor" ]; then @@ -341,7 +371,7 @@ else fi # Upgrade with candidate (always runs — verifies candidate defaults to embedded) -cand_init 2>/dev/null || true +cand_upgrade # Verify candidate created an embedded DB if embedded_db_exists; then @@ -411,8 +441,8 @@ if [ -z "${MUT_ID:-}" ]; then else pass "Issue created with previous binary (id: $MUT_ID)" - # Upgrade: run candidate init - cand_init 2>/dev/null || true + # Upgrade: consent + candidate init + cand_upgrade # Mutate using the candidate binary cand update "$MUT_ID" --notes "smoke test mutation" 2>/dev/null || true @@ -479,8 +509,8 @@ if ! $DEP_CREATED; then else pass "Dependency created with previous binary ($BLOCKED_ID depends on $BLOCKER_ID)" - # Upgrade (runs migrations 0041–0047, including the data-copy). - cand_init 2>/dev/null || true + # Upgrade (consents to and runs migrations 0041+, including the data-copy). + cand_upgrade # 1. Blocker queries must not error on the migrated dependency rows. if cand ready >/dev/null 2>&1; then @@ -573,8 +603,8 @@ else pass "Wisp dependency skipped (old binary could not block a wisp on a task)" fi - # Upgrade. - cand_init 2>/dev/null || true + # Upgrade: consent + candidate init + cand_upgrade # 1. The wisp survived the 0054-0066 + ignored 0012-0025 replay. --all # because the default listing hides closed wisps.