perf(schema): D4v2 composite (status, updated_at) + defer_until indexes (#3662)

* perf(schema): D4v2 composite (status,updated_at) + defer_until indexes (be-s54)

Migration 0043 replaces the legacy single-column idx_issues_status with
a composite idx_issues_status_updated_at and adds standalone
idx_issues_defer_until on the issues table. Targets bd stale's
status+updated_at predicate and bd ready's deferred-parents
defer_until lookup.

Rebased from be-nx7 (originally migration 0033; renumbered to 0043 to
avoid conflict with main's existing 0033_add_wisp_type_column migration).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(schema): renumber date-indexes migration 0043 -> 0052 (version collision)

This branch was cut when main was at 0042 and claimed slot 0043. main has
since taken 0043 (0043_drop_dependencies_generated_column) and advanced to
0051. Two version-0043 migrations merge textually clean, but the loader keys
on the integer version (internal/storage/schema/schema.go): on existing DBs
at version >= 43, MigrateUp early-returns and 0043_add_date_indexes never
runs -- the composite/defer_until indexes are silently never created on
upgrade. Fresh DBs happened to run both 0043 files (no object overlap),
masking it, and the only schema guard checks MAX(version)==LatestVersion(),
which stays true with a duplicate.

Renumber to 0052 (next free slot after main's max 0051); SQL unchanged.
Update CHANGELOG prose 0043 -> 0052. No application-code changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(changelog): remove false claim about stderr migration progress

schema.migrate() produces no stderr output during migration application;
the "with progress shown on stderr" phrase was inaccurate and would
mislead users into waiting for output that never appears.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(changelog): move migration 0052 note to Unreleased

The D4v2 index changelog entry (#3662) was placed under the dated
[1.0.5] section; move it to [Unreleased] -> Added since 0052 is an
unreleased migration. No code change.

Agent-Signature: claude-opus-4-8-high on behalf of maphew

---------

Co-authored-by: Jim Wordelman <jim@wordelman.name>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
This commit is contained in:
Jim Wordelman 2026-06-20 20:42:11 -07:00 • committed by GitHub
commit 0268ba894c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 84 additions and 0 deletions

View file

@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **perf(schema):** composite `(status, updated_at)` and standalone `defer_until` indexes on `issues` (migration 0052). Speeds up `bd stale` and `bd ready` on large rigs. First run after upgrade applies the migration — expect a one-time 10–25 s pause on rigs with >10K issues. Do not interrupt.
- **Per-migration content hash.** `schema_migrations` now records the SHA-256 of each migration's file content alongside its version (`content_hash`), so two clones at the same `MAX(version)` but with divergent migration content become detectable (reporter fix No.2 for [#4259](https://github.com/gastownhall/beads/issues/4259)). The column is added to fresh databases via the bootstrap schema and idempotently to existing databases at migrate time; already-applied rows keep a NULL hash. The column definition and the hashes are deterministic, so `schema_migrations` still merges cleanly across clones.
- **`bd doctor` migration-content-skew check.** Using the recorded hashes, `bd doctor` now compares the local `schema_migrations` against the cached remote-tracking ref (no network fetch) and warns when this database and its remote applied different content for the same migration version — the silent schema fork from [#4259](https://github.com/gastownhall/beads/issues/4259), surfaced as a clear advisory instead of a cryptic merge failure. Read-only diagnostic (it does not gate push/pull); the comparison primitive (`schema.ContentHashSkew`) is reusable.
- **Remote-migrate prevention gate.** `bd` now refuses to silently auto-apply pending schema migrations to an existing database that has a remote configured (in both server and embedded mode), and tells the operator to choose: migrate (as the single designated migrator, then `bd dolt push`) or adopt the already-migrated database from the remote (`bd bootstrap`). Migrating each clone independently forks the schema and breaks `bd dolt pull` ([#4259](https://github.com/gastownhall/beads/issues/4259)). The gate is a no-op for fresh databases, databases already at the binary's version, databases with no remote, and read-only opens. The designated migrator proceeds with `BD_ALLOW_REMOTE_MIGRATE=1`. In server mode the gate also detects remotes persisted on disk in `.dolt/config`, so a freshly (auto-)started server — whose in-memory `dolt_remotes` table is not yet populated — cannot slip a remote-backed database past the gate.

View file

@ -6,6 +6,7 @@ import (
"database/sql"
"os"
"path/filepath"
"strings"
"testing"
"github.com/steveyegge/beads/internal/storage/embeddeddolt"
@ -32,6 +33,17 @@ func TestSchemaAfterInit(t *testing.T) {
}
t.Cleanup(func() { _ = cleanup() })
// Verify D4v2 indexes exist on the issues table.
var ignoredName, createStmt string
if err := db.QueryRowContext(ctx, "SHOW CREATE TABLE `issues`").Scan(&ignoredName, &createStmt); err != nil {
t.Fatalf("SHOW CREATE TABLE issues: %v", err)
}
for _, idx := range []string{"idx_issues_status_updated_at", "idx_issues_defer_until"} {
if !strings.Contains(createStmt, idx) {
t.Errorf("issues table missing index %q", idx)
}
}
var maxVersion int
if err := db.QueryRowContext(ctx, "SELECT MAX(version) FROM schema_migrations").Scan(&maxVersion); err != nil {
t.Fatalf("reading max schema_migrations version: %v", err)

View file

@ -0,0 +1,32 @@
-- Reverse 0052: drop the composite + defer_until indexes and restore the
-- single-column status index. Guarded for the same idempotency reason as up.
SET @has_du = (
SELECT IF(COUNT(*) > 0, 1, 0)
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'issues'
AND INDEX_NAME = 'idx_issues_defer_until'
);
SET @sql = IF(@has_du = 1, 'DROP INDEX idx_issues_defer_until ON issues', 'SELECT 1');
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
SET @has_su = (
SELECT IF(COUNT(*) > 0, 1, 0)
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'issues'
AND INDEX_NAME = 'idx_issues_status_updated_at'
);
SET @sql = IF(@has_su = 1, 'DROP INDEX idx_issues_status_updated_at ON issues', 'SELECT 1');
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
SET @needs_old = (
SELECT IF(COUNT(*) = 0, 1, 0)
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'issues'
AND INDEX_NAME = 'idx_issues_status'
);
SET @sql = IF(@needs_old = 1, 'CREATE INDEX idx_issues_status ON issues (status)', 'SELECT 1');
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;

View file

@ -0,0 +1,39 @@
-- Replace the single-column status index with a (status, updated_at) composite
-- and add a defer_until index (D4v2 query plan). Guarded against the current
-- schema so the migration is idempotent: a clone may have its schema_migrations
-- row for this version regressed without its DDL rolled back (see bd-4mpy7
-- remote_behind_schema_gates), in which case the designated migrator re-applies
-- this file and it must not error on already-present/already-absent indexes.
-- Drop the old single-column status index only if it is still present.
SET @has_old = (
SELECT IF(COUNT(*) > 0, 1, 0)
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'issues'
AND INDEX_NAME = 'idx_issues_status'
);
SET @sql = IF(@has_old = 1, 'DROP INDEX idx_issues_status ON issues', 'SELECT 1');
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
-- Create the (status, updated_at) composite index only if missing.
SET @needs_su = (
SELECT IF(COUNT(*) = 0, 1, 0)
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'issues'
AND INDEX_NAME = 'idx_issues_status_updated_at'
);
SET @sql = IF(@needs_su = 1, 'CREATE INDEX idx_issues_status_updated_at ON issues (status, updated_at)', 'SELECT 1');
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
-- Create the defer_until index only if missing.
SET @needs_du = (
SELECT IF(COUNT(*) = 0, 1, 0)
FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'issues'
AND INDEX_NAME = 'idx_issues_defer_until'
);
SET @sql = IF(@needs_du = 1, 'CREATE INDEX idx_issues_defer_until ON issues (defer_until)', 'SELECT 1');
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;