feat(storage): keyset reads — comments page, issue-list cursor, IsBlocked filter (#4922)
* feat(comments): keyset-paged comments read (GetIssueCommentsPage) The comments read surface was whole-thread only: GetIssueComments materializes every comment, so a consumer that wants a page of a long thread must load it all and page in memory (and cap it, serving long threads incomplete). Add a keyset-paged read with a stable order and an index-backed resume predicate. - Storage.GetIssueCommentsPage(ctx, issueID, after CommentPageCursor, limit): (created_at ASC, id ASC) order; resume via the sargable predicate created_at >= ? AND ((created_at > ?) OR (id > ?)); limit defaults to 100 and caps at 500 (documented, with the len(page)<limit idiom hazard called out — terminate on an empty page). Cursor comes from a previously read row; the forward-only-walk semantics for backdated inserts are documented. Wisp-routed like the whole-thread read. Page-walk equals the full read exactly, including same-second id tie-breaks (comment ids are lowercase UUIDv7, so the CHAR(36) collation cannot reorder them). - Migration 0056 adds idx_comments_issue_created_id (issue_id, created_at, id) on comments and — table-guarded, since wisp tables are clone-local and main migrations run before the ignored chain materializes them (the 0035/0037/0053 precedent, with a matching no-op pin test) — on wisp_comments. Fresh clones get the wisp index from the canonical clone-local schema (ignored/0001), the same two-path design earlier wisp indexes use. An EXPLAIN guard pins the composite index by its column-list signature, single-sourced from the exported production query. - AddIssueCommentInTx now truncates the returned CreatedAt to seconds so the returned comment matches what the DATETIME(0) column stored and is usable as a page cursor (the stored value is unchanged). - Implemented on DoltStore and EmbeddedDoltStore; InstrumentedStorage wraps it; beads re-exports the cursor type. Tests: page-walk==full-read (same-second groups), tie-break, first/past-end/empty/default-limit, wisp routing, the migration no-op pin, and the EXPLAIN guard — on both stores. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(storage): (created_at, id) keyset issue-list predicate on IssueFilter `bd list` and the library's own callers over the public Storage surface need true keyset paging over issues with second-granularity timestamps, which the surface lacked: IssueFilter had CreatedBefore + Limit but no (created_at, id) tuple cursor, so a same-second group larger than one page could not page without dropping or duplicating its overflow. Add an additive keyset to types.IssueFilter — AfterCreatedAt *time.Time + AfterID string — meaning "rows strictly after the (created_at DESC, id ASC) position (AfterCreatedAt, AfterID)". id is the primary key, so the tie-break is total. It composes with every existing filter (CreatedBefore keeps working) and is honored by BOTH SearchIssues and SearchIssuesWithCounts on every backend, because it is emitted through the shared sqlbuild.BuildIssueFilterClauses path that every storage backend goes through. Pair it with SortBy="created" for the matching ORDER BY created_at DESC, id ASC. The predicate is single-sourced as sqlbuild.KeysetCreatedAtIDPredicate: (created_at <= ? AND ((created_at < ?) OR (id > ?))). This is the keyset form the planner can drive through an index seek rather than a scan; a dolt EXPLAIN guard pins the plan against the exact production constant, a sqlbuild unit test pins clause/args emission, and same-second-overflow paging regressions run on both the server and embedded Dolt backends through the public SearchIssues. (cherry picked from commit 167e25b3af3002939fe27d4bbfc1598fff21c586) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(storage): IssueFilter.IsBlocked filter — index-backed on every backend `bd list`/`bd ready` and the library's own callers could not filter on the denormalized is_blocked column server-side: the column and the idx_issues_is_blocked(is_blocked, status) index already exist, but the projection column was not a filter — callers had to over-fetch and post-filter in Go. Make it one. Purely additive: an optional IssueFilter.IsBlocked *bool (nil = any; no signature change, no new interface method). Wired into the shared sqlbuild filter clause as `is_blocked = ?` (bound int 0/1; the column is NOT NULL DEFAULT 0 so no IS NULL arm), so BOTH SearchIssues(WithCounts) AND CountIssues/CountIssuesByGroup honor it on every backend (dolt, embeddeddolt) through the one BuildIssueFilterClauses path. Tests: clause/arg unit test (true→1, false→0, nil→no clause); multi-backend conformance — CountIssuesByGroup(IsBlocked=true) returns only blocked beads per status, false the complement, nil the whole population; and a dolt EXPLAIN guard that the filter drives the is_blocked index. (cherry picked from commit 96c0b03024f0df08fcada9f312669f970500eea8) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(schema): move the wisp comments keyset index to a new ignored-chain migration CI's migration-hygiene gate froze ignored/0001 (it exists on main; clones have recorded its content hash), so the wisp twin of the 0056 comments keyset index cannot be baked into the canonical wisp schema there. Add it as ignored/0014 instead — guarded exactly like ignored/0006 (table exists AND index absent) — which is strictly better: the ignored chain runs per clone with its own cursor, so fresh clones AND already-materialized workspaces both pick the index up; 0056's guarded half remains for workspaces whose wisp tables existed when the main chain ran. ignored/0001 reverts to its main content. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: CI Bot <ci@beads.test> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Test User <test@test.com>
This commit is contained in:
parent
607df586d1
commit
873e3040bf
21 changed files with 1428 additions and 1 deletions
6
beads.go
6
beads.go
|
|
@ -71,6 +71,12 @@ type CloseIssueResult = storage.CloseIssueResult
|
|||
// name it without importing internal/storage.
|
||||
type UpdateIssueOptions = storage.UpdateIssueOptions
|
||||
|
||||
// CommentPageCursor is the resume position for Storage.GetIssueCommentsPage — the
|
||||
// (created_at, id) of the last comment already returned, with the zero value
|
||||
// starting a walk from the beginning of the thread. Exported so consumers can
|
||||
// name it without importing internal/storage.
|
||||
type CommentPageCursor = storage.CommentPageCursor
|
||||
|
||||
// RemoteStore provides dolt remote management and replication operations.
|
||||
// Use type assertion on a Storage value to access these methods:
|
||||
//
|
||||
|
|
|
|||
|
|
@ -720,6 +720,10 @@ func (s *configStore) AddIssueComment(_ context.Context, _, _, _ string) (*types
|
|||
func (s *configStore) GetIssueComments(_ context.Context, _ string) ([]*types.Comment, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (s *configStore) GetIssueCommentsPage(_ context.Context, _ string, _ storage.CommentPageCursor, _ int) ([]*types.Comment, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (s *configStore) GetEvents(_ context.Context, _ string, _ int) ([]*types.Event, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -126,6 +126,7 @@ func RunAll(t *testing.T, factory Factory) {
|
|||
t.Run("SearchLimit", func(t *testing.T) { testSearchLimit(t, factory) })
|
||||
t.Run("CountIssues", func(t *testing.T) { testCountIssues(t, factory) })
|
||||
t.Run("CountByGroup", func(t *testing.T) { testCountByGroup(t, factory) })
|
||||
t.Run("CountByGroupIsBlockedFilter", func(t *testing.T) { testCountByGroupIsBlockedFilter(t, factory) })
|
||||
|
||||
// Dependencies
|
||||
t.Run("AddAndGetDeps", func(t *testing.T) { testAddAndGetDeps(t, factory) })
|
||||
|
|
@ -495,6 +496,66 @@ func testCountByGroup(t *testing.T, f Factory) {
|
|||
}
|
||||
}
|
||||
|
||||
// testCountByGroupIsBlockedFilter proves the additive IssueFilter.IsBlocked predicate honors the
|
||||
// denormalized is_blocked column in the grouped-count path on every backend: IsBlocked=&true returns
|
||||
// only is_blocked=true beads per status, &false the complement, and nil the whole population
|
||||
// (unchanged from testCountByGroup). Blockedness is produced the same way a real writer does — an open
|
||||
// blocker on a blocks-type dependency, which the write path denormalizes onto is_blocked.
|
||||
func testCountByGroupIsBlockedFilter(t *testing.T, f Factory) {
|
||||
s := f(t)
|
||||
// Two open beads blocked by an open blocker (is_blocked=true after AddDependency), one open
|
||||
// bead left unblocked, and one closed bead (is_blocked=false) — a (status × is_blocked) mix.
|
||||
must(t, s.CreateIssue(ctx(), withDefaults(&types.Issue{ID: "ibf-blk", Title: "Blocker", Status: types.StatusOpen}), "a"))
|
||||
must(t, s.CreateIssue(ctx(), withDefaults(&types.Issue{ID: "ibf-w1", Title: "Waiter1", Status: types.StatusOpen}), "a"))
|
||||
must(t, s.CreateIssue(ctx(), withDefaults(&types.Issue{ID: "ibf-w2", Title: "Waiter2", Status: types.StatusOpen}), "a"))
|
||||
must(t, s.CreateIssue(ctx(), withDefaults(&types.Issue{ID: "ibf-free", Title: "Free", Status: types.StatusOpen}), "a"))
|
||||
must(t, s.CreateIssue(ctx(), withDefaults(&types.Issue{ID: "ibf-done", Title: "Done", Status: types.StatusClosed}), "a"))
|
||||
must(t, s.AddDependency(ctx(), &types.Dependency{IssueID: "ibf-w1", DependsOnID: "ibf-blk", Type: types.DepBlocks}, "a"))
|
||||
must(t, s.AddDependency(ctx(), &types.Dependency{IssueID: "ibf-w2", DependsOnID: "ibf-blk", Type: types.DepBlocks}, "a"))
|
||||
|
||||
blockedTrue := true
|
||||
blockedFalse := false
|
||||
|
||||
// IsBlocked=&true: only the two blocked waiters, both status=open.
|
||||
onlyBlocked, err := s.CountIssuesByGroup(ctx(), types.IssueFilter{IsBlocked: &blockedTrue}, "status")
|
||||
if err != nil {
|
||||
t.Fatalf("CountIssuesByGroup(IsBlocked=true): %v", err)
|
||||
}
|
||||
if onlyBlocked["open"] != 2 {
|
||||
t.Errorf("IsBlocked=true open = %d, want 2 (the blocked waiters only)", onlyBlocked["open"])
|
||||
}
|
||||
if onlyBlocked["closed"] != 0 {
|
||||
t.Errorf("IsBlocked=true closed = %d, want 0", onlyBlocked["closed"])
|
||||
}
|
||||
|
||||
// IsBlocked=&false: the complement — the blocker + the free bead (open) and the done bead (closed).
|
||||
onlyUnblocked, err := s.CountIssuesByGroup(ctx(), types.IssueFilter{IsBlocked: &blockedFalse}, "status")
|
||||
if err != nil {
|
||||
t.Fatalf("CountIssuesByGroup(IsBlocked=false): %v", err)
|
||||
}
|
||||
if onlyUnblocked["open"] != 2 {
|
||||
t.Errorf("IsBlocked=false open = %d, want 2 (blocker + free)", onlyUnblocked["open"])
|
||||
}
|
||||
if onlyUnblocked["closed"] != 1 {
|
||||
t.Errorf("IsBlocked=false closed = %d, want 1 (done)", onlyUnblocked["closed"])
|
||||
}
|
||||
|
||||
// nil: the whole population, and the true/false split partitions it exactly (no double-count).
|
||||
all, err := s.CountIssuesByGroup(ctx(), types.IssueFilter{}, "status")
|
||||
if err != nil {
|
||||
t.Fatalf("CountIssuesByGroup(nil): %v", err)
|
||||
}
|
||||
if all["open"] != 4 || all["closed"] != 1 {
|
||||
t.Errorf("nil filter open/closed = %d/%d, want 4/1", all["open"], all["closed"])
|
||||
}
|
||||
for _, status := range []string{"open", "closed"} {
|
||||
if onlyBlocked[status]+onlyUnblocked[status] != all[status] {
|
||||
t.Errorf("status %q: blocked(%d) + unblocked(%d) != all(%d) — the is_blocked split must partition",
|
||||
status, onlyBlocked[status], onlyUnblocked[status], all[status])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- Dependencies ---
|
||||
|
||||
func testAddAndGetDeps(t *testing.T, f Factory) {
|
||||
|
|
|
|||
346
internal/storage/dolt/comments_page_test.go
Normal file
346
internal/storage/dolt/comments_page_test.go
Normal file
|
|
@ -0,0 +1,346 @@
|
|||
package dolt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// cmtIDs projects comment IDs in result order.
|
||||
func cmtIDs(cs []*types.Comment) []string {
|
||||
out := make([]string, len(cs))
|
||||
for i, c := range cs {
|
||||
out[i] = c.ID
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cmtEqual(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if a[i] != b[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// seedGroupedComments inserts n comments on issueID with ids that sort lexically
|
||||
// by index and created_at bucketed into same-second groups of groupSize (forcing
|
||||
// the (created_at, id) tie-break). It returns the whole-second base time. Raw
|
||||
// INSERTs give the test full control of both id and created_at.
|
||||
func seedGroupedComments(t *testing.T, ctx context.Context, db *sql.DB, issueID string, n, groupSize int) time.Time {
|
||||
t.Helper()
|
||||
base := time.Date(2024, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
for i := 0; i < n; i++ {
|
||||
id := fmt.Sprintf("00000000-0000-0000-0000-%012d", i)
|
||||
at := base.Add(time.Duration(i/groupSize) * time.Second)
|
||||
if _, err := db.ExecContext(ctx,
|
||||
"INSERT INTO comments (id, issue_id, author, text, created_at) VALUES (?, ?, ?, ?, ?)",
|
||||
id, issueID, "tester", fmt.Sprintf("c%d", i), at); err != nil {
|
||||
t.Fatalf("insert comment %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
// walkCommentsPage pages the whole thread with the given limit, feeding each
|
||||
// page's last (created_at, id) back in as the cursor, and returns the collected
|
||||
// ids. It fails on any duplicate id or an over-long page.
|
||||
func walkCommentsPage(t *testing.T, ctx context.Context, s *DoltStore, issueID string, limit int) []string {
|
||||
t.Helper()
|
||||
var collected []string
|
||||
seen := map[string]bool{}
|
||||
var after storage.CommentPageCursor
|
||||
for i := 0; i < 1000; i++ {
|
||||
page, err := s.GetIssueCommentsPage(ctx, issueID, after, limit)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(page %d): %v", i, err)
|
||||
}
|
||||
if len(page) == 0 {
|
||||
break
|
||||
}
|
||||
if len(page) > limit {
|
||||
t.Fatalf("page %d size = %d, want <= %d", i, len(page), limit)
|
||||
}
|
||||
for _, c := range page {
|
||||
if seen[c.ID] {
|
||||
t.Fatalf("duplicate id %q across pages — same-second overflow was lost", c.ID)
|
||||
}
|
||||
seen[c.ID] = true
|
||||
collected = append(collected, c.ID)
|
||||
}
|
||||
last := page[len(page)-1]
|
||||
after = storage.CommentPageCursor{CreatedAt: last.CreatedAt, ID: last.ID}
|
||||
}
|
||||
return collected
|
||||
}
|
||||
|
||||
// TestGetIssueCommentsPageWalkEqualsFullRead is the money property: a keyset walk
|
||||
// with a page size smaller than a same-second group reproduces GetIssueComments
|
||||
// exactly — same order, no dropped or duplicated comment, no gap — even across
|
||||
// same-second collisions where a created_at-only cursor would lose the overflow.
|
||||
func TestGetIssueCommentsPageWalkEqualsFullRead(t *testing.T) {
|
||||
store, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
iss := &types.Issue{ID: "cm-page", Title: "cm", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}
|
||||
if err := store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("create issue: %v", err)
|
||||
}
|
||||
// 25 comments in 5 same-second groups of 5 (groupSize 5 > page size 4).
|
||||
const n, groupSize, pageSize = 25, 5, 4
|
||||
seedGroupedComments(t, ctx, store.db, iss.ID, n, groupSize)
|
||||
|
||||
full, err := store.GetIssueComments(ctx, iss.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueComments: %v", err)
|
||||
}
|
||||
if len(full) != n {
|
||||
t.Fatalf("full read = %d comments, want %d", len(full), n)
|
||||
}
|
||||
|
||||
// Same-second tie-break, explicit: the first group shares one created_at and
|
||||
// is returned in strictly ascending id order (the secondary sort key).
|
||||
for i := 1; i < groupSize; i++ {
|
||||
if !full[i].CreatedAt.Equal(full[0].CreatedAt) {
|
||||
t.Fatalf("group 0 not same-second: full[%d].CreatedAt=%v != full[0]=%v", i, full[i].CreatedAt, full[0].CreatedAt)
|
||||
}
|
||||
if full[i].ID <= full[i-1].ID {
|
||||
t.Fatalf("same-second group not id-ascending at %d: %q <= %q", i, full[i].ID, full[i-1].ID)
|
||||
}
|
||||
}
|
||||
|
||||
// The walk (page size < group size) equals the full read exactly.
|
||||
walked := walkCommentsPage(t, ctx, store, iss.ID, pageSize)
|
||||
if want := cmtIDs(full); !cmtEqual(walked, want) {
|
||||
t.Fatalf("keyset walk = %v,\nwant full read %v", walked, want)
|
||||
}
|
||||
|
||||
// First page (zero cursor) is exactly the first pageSize in order.
|
||||
first, err := store.GetIssueCommentsPage(ctx, iss.ID, storage.CommentPageCursor{}, pageSize)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(first): %v", err)
|
||||
}
|
||||
if got, want := cmtIDs(first), cmtIDs(full)[:pageSize]; !cmtEqual(got, want) {
|
||||
t.Fatalf("first page = %v, want %v", got, want)
|
||||
}
|
||||
|
||||
// A cursor planted on the last comment (past-the-end) yields an empty page.
|
||||
lastC := full[len(full)-1]
|
||||
end, err := store.GetIssueCommentsPage(ctx, iss.ID, storage.CommentPageCursor{CreatedAt: lastC.CreatedAt, ID: lastC.ID}, pageSize)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(past-the-end): %v", err)
|
||||
}
|
||||
if len(end) != 0 {
|
||||
t.Fatalf("past-the-end page = %v, want empty", cmtIDs(end))
|
||||
}
|
||||
|
||||
// limit <= 0 falls back to the store default (>= n here), returning the whole
|
||||
// thread in one page identical to the full read.
|
||||
def, err := store.GetIssueCommentsPage(ctx, iss.ID, storage.CommentPageCursor{}, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(limit=0): %v", err)
|
||||
}
|
||||
if got, want := cmtIDs(def), cmtIDs(full); !cmtEqual(got, want) {
|
||||
t.Fatalf("default-limit page = %v, want full read %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetIssueCommentsPageEmptyThread pins the empty-thread and missing-issue
|
||||
// semantics: both return an empty page with no error, matching GetIssueComments.
|
||||
func TestGetIssueCommentsPageEmptyThread(t *testing.T) {
|
||||
store, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
iss := &types.Issue{ID: "cm-empty", Title: "cm", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}
|
||||
if err := store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("create issue: %v", err)
|
||||
}
|
||||
|
||||
empty, err := store.GetIssueCommentsPage(ctx, iss.ID, storage.CommentPageCursor{}, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(empty thread): %v", err)
|
||||
}
|
||||
if len(empty) != 0 {
|
||||
t.Fatalf("empty thread page = %v, want empty", cmtIDs(empty))
|
||||
}
|
||||
|
||||
missing, err := store.GetIssueCommentsPage(ctx, "does-not-exist", storage.CommentPageCursor{}, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(missing issue): %v", err)
|
||||
}
|
||||
if len(missing) != 0 {
|
||||
t.Fatalf("missing-issue page = %v, want empty", cmtIDs(missing))
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetIssueCommentsPageWisp verifies the wisp route: a page on an active wisp
|
||||
// reads wisp_comments (not comments) and matches GetIssueComments for that wisp.
|
||||
func TestGetIssueCommentsPageWisp(t *testing.T) {
|
||||
store, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
wisp := &types.Issue{ID: "cm-wisp", Title: "wisp", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, Ephemeral: true}
|
||||
if err := store.CreateIssue(ctx, wisp, "tester"); err != nil {
|
||||
t.Fatalf("create wisp: %v", err)
|
||||
}
|
||||
base := time.Date(2024, 5, 6, 7, 8, 9, 0, time.UTC)
|
||||
for i := 0; i < 6; i++ {
|
||||
if _, err := store.ImportIssueComment(ctx, wisp.ID, "tester", fmt.Sprintf("w%d", i), base.Add(time.Duration(i)*time.Second)); err != nil {
|
||||
t.Fatalf("import wisp comment %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Nothing landed in the durable comments table — routing hit wisp_comments.
|
||||
var durable int
|
||||
if err := store.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM comments WHERE issue_id = ?", wisp.ID).Scan(&durable); err != nil {
|
||||
t.Fatalf("count durable comments: %v", err)
|
||||
}
|
||||
if durable != 0 {
|
||||
t.Fatalf("wisp comments leaked into durable table: %d rows", durable)
|
||||
}
|
||||
|
||||
full, err := store.GetIssueComments(ctx, wisp.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueComments(wisp): %v", err)
|
||||
}
|
||||
if len(full) != 6 {
|
||||
t.Fatalf("wisp full read = %d, want 6", len(full))
|
||||
}
|
||||
walked := walkCommentsPage(t, ctx, store, wisp.ID, 2)
|
||||
if want := cmtIDs(full); !cmtEqual(walked, want) {
|
||||
t.Fatalf("wisp keyset walk = %v, want %v", walked, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetIssueCommentsPagePlanIsIndexed is the sargability regression guard: the
|
||||
// keyset page predicate must seek the (issue_id, created_at, id) index
|
||||
// (IndexedTableAccess), not full-scan-and-filter the issue's comments. The
|
||||
// redundant `created_at >= ?` lower bound in CommentsKeysetPredicate is what keeps
|
||||
// the Dolt planner on the index. It single-sources the guarded SQL from
|
||||
// production (issueops.CommentsPageQuery, which embeds CommentsKeysetPredicate)
|
||||
// and skips rather than fails if the EXPLAIN format is unrecognizable.
|
||||
func TestGetIssueCommentsPagePlanIsIndexed(t *testing.T) {
|
||||
store, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
iss := &types.Issue{ID: "cm-plan", Title: "cm", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}
|
||||
if err := store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("create issue: %v", err)
|
||||
}
|
||||
seedGroupedComments(t, ctx, store.db, iss.ID, 10, 2)
|
||||
|
||||
// Single-source: the production query string must embed the exported
|
||||
// predicate, so a drift in CommentsKeysetPredicate breaks both.
|
||||
prod := issueops.CommentsPageQuery("comments", true, 100)
|
||||
if !strings.Contains(prod, issueops.CommentsKeysetPredicate) {
|
||||
t.Fatalf("production page query does not embed CommentsKeysetPredicate:\n%s", prod)
|
||||
}
|
||||
|
||||
const cur = "2024-01-02 03:04:05"
|
||||
// Four ? placeholders: issue_id, created_at (lower bound), created_at
|
||||
// (strict), id (tie-break).
|
||||
plan := explainPlan(t, ctx, store.db, literalizeParams(prod, "'"+iss.ID+"'", "'"+cur+"'", "'"+cur+"'", "''"))
|
||||
|
||||
if !looksLikeDoltPlan(plan) {
|
||||
t.Skipf("EXPLAIN output not in a recognized Dolt plan format, skipping sargability assertion; plan=\n%s", plan)
|
||||
}
|
||||
// Pin the COMPOSITE index specifically, not merely "some index". Dolt's
|
||||
// EXPLAIN identifies an index by its column list rather than its name, so the
|
||||
// composite idx_comments_issue_created_id shows as
|
||||
// [comments.issue_id,comments.created_at,comments.id]. The single-column
|
||||
// idx_comments_issue shows only [comments.issue_id] (verified by a
|
||||
// drop-index probe during authoring) — so this exact signature fails if the
|
||||
// planner regresses to the issue-only index (leaving the created_at range +
|
||||
// id tie-break to a filter+sort) or to a full scan.
|
||||
const compositeIndexCols = "[comments.issue_id,comments.created_at,comments.id]"
|
||||
if !strings.Contains(plan, "IndexedTableAccess") || !strings.Contains(plan, compositeIndexCols) {
|
||||
t.Fatalf("comment page predicate does not seek the composite index (want IndexedTableAccess on %s) — regressed to the issue-only index or a full Table scan.\nplan:\n%s", compositeIndexCols, plan)
|
||||
}
|
||||
}
|
||||
|
||||
// literalizeParams replaces each ? placeholder in query, in order, with the
|
||||
// corresponding literal — for EXPLAINing a production ?-bound SQL string whose
|
||||
// planner shape is under test. It panics on an arity mismatch so a drifted
|
||||
// placeholder count fails loudly rather than EXPLAINing malformed SQL.
|
||||
func literalizeParams(query string, literals ...string) string {
|
||||
for _, lit := range literals {
|
||||
if !strings.Contains(query, "?") {
|
||||
panic("literalizeParams: more literals than ? placeholders in query")
|
||||
}
|
||||
query = strings.Replace(query, "?", lit, 1)
|
||||
}
|
||||
if strings.Contains(query, "?") {
|
||||
panic("literalizeParams: unbound ? placeholder(s) remain in query")
|
||||
}
|
||||
return query
|
||||
}
|
||||
|
||||
// explainPlan runs EXPLAIN FORMAT=TREE <query> and joins the plan tree into one
|
||||
// string. FORMAT=TREE yields the go-mysql-server node tree in a single "plan"
|
||||
// column (IndexedTableAccess / Filter / TopN); the default tabular EXPLAIN is
|
||||
// avoided because its numeric columns break the MySQL driver's row decode.
|
||||
func explainPlan(t *testing.T, ctx context.Context, db *sql.DB, query string) string {
|
||||
t.Helper()
|
||||
rows, err := db.QueryContext(ctx, "EXPLAIN FORMAT=TREE "+query)
|
||||
if err != nil {
|
||||
t.Fatalf("EXPLAIN: %v", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
cols, err := rows.Columns()
|
||||
if err != nil {
|
||||
t.Fatalf("EXPLAIN columns: %v", err)
|
||||
}
|
||||
var out strings.Builder
|
||||
for rows.Next() {
|
||||
// RawBytes bypasses the driver's per-column type coercion (Dolt's EXPLAIN
|
||||
// can carry numeric columns that arrive as the literal text "NULL").
|
||||
cells := make([]any, len(cols))
|
||||
holders := make([]sql.RawBytes, len(cols))
|
||||
for i := range cells {
|
||||
cells[i] = &holders[i]
|
||||
}
|
||||
if err := rows.Scan(cells...); err != nil {
|
||||
t.Fatalf("EXPLAIN scan: %v", err)
|
||||
}
|
||||
for _, h := range holders {
|
||||
if len(h) > 0 {
|
||||
out.Write(h)
|
||||
out.WriteString("\n")
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
t.Fatalf("EXPLAIN rows: %v", err)
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
|
||||
// looksLikeDoltPlan reports whether s resembles a go-mysql-server / Dolt EXPLAIN
|
||||
// plan tree, so the sargability assertion can skip cleanly if the format shifts.
|
||||
func looksLikeDoltPlan(s string) bool {
|
||||
for _, tok := range []string{"TableAccess", "Table", "Project", "Filter", "Sort", "Limit", "TopN"} {
|
||||
if strings.Contains(s, tok) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
|
@ -6,6 +6,7 @@ import (
|
|||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
|
@ -103,6 +104,20 @@ func (s *DoltStore) GetIssueComments(ctx context.Context, issueID string) ([]*ty
|
|||
return scanComments(rows)
|
||||
}
|
||||
|
||||
// GetIssueCommentsPage returns one keyset page of an issue's comments in
|
||||
// (created_at ASC, id ASC) order, resuming strictly after the cursor. See the
|
||||
// storage.Storage doc for the ordering, sargability, and page-walk-equals-full-
|
||||
// read contract.
|
||||
func (s *DoltStore) GetIssueCommentsPage(ctx context.Context, issueID string, after storage.CommentPageCursor, limit int) ([]*types.Comment, error) {
|
||||
var result []*types.Comment
|
||||
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
|
||||
var err error
|
||||
result, err = issueops.GetIssueCommentsPageInTx(ctx, tx, issueID, after, limit)
|
||||
return err
|
||||
})
|
||||
return result, err
|
||||
}
|
||||
|
||||
// GetCommentsForIssues retrieves comments for multiple issues
|
||||
func (s *DoltStore) GetCommentsForIssues(ctx context.Context, issueIDs []string) (map[string][]*types.Comment, error) {
|
||||
var result map[string][]*types.Comment
|
||||
|
|
|
|||
38
internal/storage/dolt/is_blocked_filter_test.go
Normal file
38
internal/storage/dolt/is_blocked_filter_test.go
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
package dolt
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// TestIsBlockedFilterPlanIsIndexed is the sargability guard for the
|
||||
// IssueFilter.IsBlocked predicate: `is_blocked = ?` must seek
|
||||
// idx_issues_is_blocked(is_blocked, status) (IndexedTableAccess), not
|
||||
// full-scan-and-filter, so a (status × is_blocked) count matrix stays cheap
|
||||
// on the reference Dolt backend. It EXPLAINs the exact production predicate shape
|
||||
// and skips rather than fails if the EXPLAIN format is unrecognizable.
|
||||
func TestIsBlockedFilterPlanIsIndexed(t *testing.T) {
|
||||
store, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
iss := &types.Issue{ID: "ibp-" + string(rune('a'+i)), Title: "ibp", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}
|
||||
if err := store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("create seed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The production filter emits `is_blocked = ?`; EXPLAIN the same predicate with
|
||||
// a literal so the planner shape is under test (callers bind 1 and 0).
|
||||
plan := explainPlan(t, ctx, store.db, "SELECT id FROM issues WHERE is_blocked = 1")
|
||||
if !looksLikeDoltPlan(plan) {
|
||||
t.Skipf("EXPLAIN output not in a recognized Dolt plan format, skipping sargability assertion; plan=\n%s", plan)
|
||||
}
|
||||
if !strings.Contains(plan, "IndexedTableAccess") || !strings.Contains(plan, "issues.is_blocked") {
|
||||
t.Fatalf("is_blocked predicate does not seek idx_issues_is_blocked (want IndexedTableAccess on [issues.is_blocked]) — the filter regressed to a full Table scan.\nplan:\n%s", plan)
|
||||
}
|
||||
}
|
||||
163
internal/storage/dolt/keyset_search_test.go
Normal file
163
internal/storage/dolt/keyset_search_test.go
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
package dolt
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/sqlbuild"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// ksIDs projects issue IDs in result order.
|
||||
func ksIDs(issues []*types.Issue) []string {
|
||||
out := make([]string, len(issues))
|
||||
for i, iss := range issues {
|
||||
out[i] = iss.ID
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func ksEqual(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if a[i] != b[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// TestSearchIssuesKeysetSameSecondOverflow is the same-second-overflow keyset regression: a same-second
|
||||
// group LARGER than one page must page completely under the (created_at DESC,
|
||||
// id ASC) keyset with no dropped or duplicated row — the exact case a
|
||||
// created_at-only cursor loses. It also pins the cross-second DESC + id-ASC order
|
||||
// and that the keyset resumes exactly at the boundary. Exercised through the
|
||||
// public SearchIssues (SLICE stack), which routes the new IssueFilter keyset
|
||||
// fields into the shared sqlbuild predicate.
|
||||
func TestSearchIssuesKeysetSameSecondOverflow(t *testing.T) {
|
||||
store, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
// A 5-row same-second group at T, one row a second newer, one a second older.
|
||||
// Under created_at DESC, id ASC the total order is: newer, a1..a5, older.
|
||||
base := time.Date(2024, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
seeds := []struct {
|
||||
id string
|
||||
at time.Time
|
||||
}{
|
||||
{"k-newer", base.Add(time.Second)},
|
||||
{"k-a1", base}, {"k-a2", base}, {"k-a3", base}, {"k-a4", base}, {"k-a5", base},
|
||||
{"k-older", base.Add(-time.Second)},
|
||||
}
|
||||
for _, s := range seeds {
|
||||
iss := &types.Issue{ID: s.id, Title: s.id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, CreatedAt: s.at}
|
||||
if err := store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("create %s: %v", s.id, err)
|
||||
}
|
||||
}
|
||||
|
||||
wantOrder := []string{"k-newer", "k-a1", "k-a2", "k-a3", "k-a4", "k-a5", "k-older"}
|
||||
|
||||
// One-shot ordered read (no keyset): pins created_at DESC, id ASC.
|
||||
full, err := store.SearchIssues(ctx, "", types.IssueFilter{
|
||||
IDPrefix: "k-", SkipWisps: true, SortBy: "created", Limit: 100,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("SearchIssues(full): %v", err)
|
||||
}
|
||||
if got := ksIDs(full); !ksEqual(got, wantOrder) {
|
||||
t.Fatalf("full order = %v, want %v", got, wantOrder)
|
||||
}
|
||||
|
||||
// Keyset walk, page size 2 (< the 5-row same-second group). The union across
|
||||
// pages must be exactly wantOrder in order, with no id seen twice.
|
||||
const pageSize = 2
|
||||
var collected []string
|
||||
seen := map[string]bool{}
|
||||
var afterCreatedAt *time.Time
|
||||
afterID := ""
|
||||
for i := 0; i < 100; i++ {
|
||||
f := types.IssueFilter{
|
||||
IDPrefix: "k-", SkipWisps: true, SortBy: "created", Limit: pageSize,
|
||||
AfterCreatedAt: afterCreatedAt, AfterID: afterID,
|
||||
}
|
||||
page, err := store.SearchIssues(ctx, "", f)
|
||||
if err != nil {
|
||||
t.Fatalf("SearchIssues(page %d): %v", i, err)
|
||||
}
|
||||
if len(page) == 0 {
|
||||
break
|
||||
}
|
||||
if len(page) > pageSize {
|
||||
t.Fatalf("page %d size = %d, want <= %d", i, len(page), pageSize)
|
||||
}
|
||||
for _, iss := range page {
|
||||
if seen[iss.ID] {
|
||||
t.Fatalf("duplicate id %q across pages — same-second overflow was lost", iss.ID)
|
||||
}
|
||||
seen[iss.ID] = true
|
||||
collected = append(collected, iss.ID)
|
||||
}
|
||||
last := page[len(page)-1]
|
||||
at := last.CreatedAt.UTC()
|
||||
afterCreatedAt = &at
|
||||
afterID = last.ID
|
||||
}
|
||||
if !ksEqual(collected, wantOrder) {
|
||||
t.Fatalf("keyset paged order = %v, want %v (no drop/dup)", collected, wantOrder)
|
||||
}
|
||||
|
||||
// Resume-at-boundary: a cursor planted in the middle of the same-second group
|
||||
// (after k-a3) yields exactly the strictly-later tail (a4, a5, older).
|
||||
afterA3 := base.UTC()
|
||||
tail, err := store.SearchIssues(ctx, "", types.IssueFilter{
|
||||
IDPrefix: "k-", SkipWisps: true, SortBy: "created", Limit: 100,
|
||||
AfterCreatedAt: &afterA3, AfterID: "k-a3",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("SearchIssues(after k-a3): %v", err)
|
||||
}
|
||||
if got := ksIDs(tail); !ksEqual(got, []string{"k-a4", "k-a5", "k-older"}) {
|
||||
t.Fatalf("resume after k-a3 = %v, want [k-a4 k-a5 k-older]", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSearchIssuesKeysetPlanIsIndexed is the sargability regression guard: the
|
||||
// keyset predicate must seek idx_issues_created_at (IndexedTableAccess), not
|
||||
// full-scan-and-filter. The redundant `created_at <= ?` leading bound is what
|
||||
// keeps the Dolt planner on the index. It EXPLAINs the exact production predicate
|
||||
// (single-sourced from sqlbuild.KeysetCreatedAtIDPredicate) with literals, and
|
||||
// skips rather than fails if the EXPLAIN format is unrecognizable.
|
||||
func TestSearchIssuesKeysetPlanIsIndexed(t *testing.T) {
|
||||
store, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
base := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
for i := 0; i < 5; i++ {
|
||||
iss := &types.Issue{ID: "kp-" + string(rune('a'+i)), Title: "kp", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, CreatedAt: base.Add(time.Duration(i) * time.Second)}
|
||||
if err := store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("create seed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
const cur = "2023-01-01 00:00:00"
|
||||
// Single-source the guarded SQL from production: the three placeholders bind
|
||||
// created_at (sargable upper bound), created_at (strict), id (tie-break).
|
||||
pred := literalizeParams(sqlbuild.KeysetCreatedAtIDPredicate, "'"+cur+"'", "'"+cur+"'", "''")
|
||||
//nolint:gosec // G202: pred is a literalized production constant, no user input.
|
||||
plan := explainPlan(t, ctx, store.db, "SELECT id FROM issues WHERE "+pred+" ORDER BY created_at DESC, id ASC LIMIT 100")
|
||||
|
||||
if !looksLikeDoltPlan(plan) {
|
||||
t.Skipf("EXPLAIN output not in a recognized Dolt plan format, skipping sargability assertion; plan=\n%s", plan)
|
||||
}
|
||||
if !strings.Contains(plan, "IndexedTableAccess") || !strings.Contains(plan, "issues.created_at") {
|
||||
t.Fatalf("keyset predicate does not seek idx_issues_created_at (want IndexedTableAccess on [issues.created_at]) — the sargable upper bound regressed to a full Table scan.\nplan:\n%s", plan)
|
||||
}
|
||||
}
|
||||
181
internal/storage/embeddeddolt/comments_page_test.go
Normal file
181
internal/storage/embeddeddolt/comments_page_test.go
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
//go:build cgo
|
||||
|
||||
package embeddeddolt_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// cpIDs projects comment IDs in result order.
|
||||
func cpIDs(cs []*types.Comment) []string {
|
||||
out := make([]string, len(cs))
|
||||
for i, c := range cs {
|
||||
out[i] = c.ID
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cpEqual(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if a[i] != b[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// TestGetIssueCommentsPageEmbedded is the embedded-backend twin of the dolt
|
||||
// suite's page-walk property: a keyset walk with a page size smaller than a
|
||||
// same-second group reproduces GetIssueComments exactly (order + content, no
|
||||
// drop/dup), and the wisp route pages wisp_comments. Comments are seeded through
|
||||
// the public ImportIssueComment with a controlled created_at so several share a
|
||||
// second, exercising the (created_at, id) tie-break; the external test package
|
||||
// shares no connection with the store, so writes go through store ops.
|
||||
func TestGetIssueCommentsPageEmbedded(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
|
||||
te := newTestEnv(t, "cp")
|
||||
ctx := t.Context()
|
||||
|
||||
iss := &types.Issue{ID: "cp-1", Title: "cp", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}
|
||||
if err := te.store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("CreateIssue: %v", err)
|
||||
}
|
||||
|
||||
// 25 comments in 5 same-second groups of 5 (group size > page size 4).
|
||||
const n, groupSize, pageSize = 25, 5, 4
|
||||
base := time.Date(2024, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
for i := 0; i < n; i++ {
|
||||
at := base.Add(time.Duration(i/groupSize) * time.Second)
|
||||
if _, err := te.store.ImportIssueComment(ctx, iss.ID, "tester", fmt.Sprintf("c%d", i), at); err != nil {
|
||||
t.Fatalf("ImportIssueComment %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
full, err := te.store.GetIssueComments(ctx, iss.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueComments: %v", err)
|
||||
}
|
||||
if len(full) != n {
|
||||
t.Fatalf("full read = %d comments, want %d", len(full), n)
|
||||
}
|
||||
|
||||
// A same-second collision was actually exercised (adjacent equal created_at).
|
||||
sawCollision := false
|
||||
for i := 1; i < len(full); i++ {
|
||||
if full[i].CreatedAt.Equal(full[i-1].CreatedAt) {
|
||||
sawCollision = true
|
||||
if full[i].ID <= full[i-1].ID {
|
||||
t.Fatalf("same-second group not id-ascending at %d: %q <= %q", i, full[i].ID, full[i-1].ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !sawCollision {
|
||||
t.Fatalf("no same-second created_at collision in the seeded thread — tie-break not exercised")
|
||||
}
|
||||
|
||||
// Keyset walk (page size < group size) equals the full read exactly.
|
||||
var collected []string
|
||||
seen := map[string]bool{}
|
||||
var after storage.CommentPageCursor
|
||||
for i := 0; i < 1000; i++ {
|
||||
page, err := te.store.GetIssueCommentsPage(ctx, iss.ID, after, pageSize)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(page %d): %v", i, err)
|
||||
}
|
||||
if len(page) == 0 {
|
||||
break
|
||||
}
|
||||
if len(page) > pageSize {
|
||||
t.Fatalf("page %d size = %d, want <= %d", i, len(page), pageSize)
|
||||
}
|
||||
for _, c := range page {
|
||||
if seen[c.ID] {
|
||||
t.Fatalf("duplicate id %q across pages", c.ID)
|
||||
}
|
||||
seen[c.ID] = true
|
||||
collected = append(collected, c.ID)
|
||||
}
|
||||
last := page[len(page)-1]
|
||||
after = storage.CommentPageCursor{CreatedAt: last.CreatedAt, ID: last.ID}
|
||||
}
|
||||
if want := cpIDs(full); !cpEqual(collected, want) {
|
||||
t.Fatalf("keyset walk = %v,\nwant full read %v", collected, want)
|
||||
}
|
||||
|
||||
// First page (zero cursor) is the first pageSize in order; a cursor on the
|
||||
// last comment yields an empty page.
|
||||
first, err := te.store.GetIssueCommentsPage(ctx, iss.ID, storage.CommentPageCursor{}, pageSize)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(first): %v", err)
|
||||
}
|
||||
if got, want := cpIDs(first), cpIDs(full)[:pageSize]; !cpEqual(got, want) {
|
||||
t.Fatalf("first page = %v, want %v", got, want)
|
||||
}
|
||||
lastC := full[len(full)-1]
|
||||
end, err := te.store.GetIssueCommentsPage(ctx, iss.ID, storage.CommentPageCursor{CreatedAt: lastC.CreatedAt, ID: lastC.ID}, pageSize)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(past-the-end): %v", err)
|
||||
}
|
||||
if len(end) != 0 {
|
||||
t.Fatalf("past-the-end page = %v, want empty", cpIDs(end))
|
||||
}
|
||||
|
||||
// Empty thread -> empty page.
|
||||
empty := &types.Issue{ID: "cp-empty", Title: "cp", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}
|
||||
if err := te.store.CreateIssue(ctx, empty, "tester"); err != nil {
|
||||
t.Fatalf("CreateIssue(empty): %v", err)
|
||||
}
|
||||
emptyPage, err := te.store.GetIssueCommentsPage(ctx, empty.ID, storage.CommentPageCursor{}, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(empty): %v", err)
|
||||
}
|
||||
if len(emptyPage) != 0 {
|
||||
t.Fatalf("empty thread page = %v, want empty", cpIDs(emptyPage))
|
||||
}
|
||||
|
||||
// Wisp route: an active wisp pages wisp_comments and matches its full read.
|
||||
wisp := &types.Issue{ID: "cp-wisp", Title: "wisp", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, Ephemeral: true}
|
||||
if err := te.store.CreateIssue(ctx, wisp, "tester"); err != nil {
|
||||
t.Fatalf("CreateIssue(wisp): %v", err)
|
||||
}
|
||||
for i := 0; i < 6; i++ {
|
||||
if _, err := te.store.ImportIssueComment(ctx, wisp.ID, "tester", fmt.Sprintf("w%d", i), base.Add(time.Duration(i)*time.Second)); err != nil {
|
||||
t.Fatalf("ImportIssueComment(wisp) %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
wispFull, err := te.store.GetIssueComments(ctx, wisp.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueComments(wisp): %v", err)
|
||||
}
|
||||
if len(wispFull) != 6 {
|
||||
t.Fatalf("wisp full read = %d, want 6", len(wispFull))
|
||||
}
|
||||
var wispWalk []string
|
||||
after = storage.CommentPageCursor{}
|
||||
for i := 0; i < 1000; i++ {
|
||||
page, err := te.store.GetIssueCommentsPage(ctx, wisp.ID, after, 2)
|
||||
if err != nil {
|
||||
t.Fatalf("GetIssueCommentsPage(wisp page %d): %v", i, err)
|
||||
}
|
||||
if len(page) == 0 {
|
||||
break
|
||||
}
|
||||
for _, c := range page {
|
||||
wispWalk = append(wispWalk, c.ID)
|
||||
}
|
||||
last := page[len(page)-1]
|
||||
after = storage.CommentPageCursor{CreatedAt: last.CreatedAt, ID: last.ID}
|
||||
}
|
||||
if want := cpIDs(wispFull); !cpEqual(wispWalk, want) {
|
||||
t.Fatalf("wisp keyset walk = %v, want %v", wispWalk, want)
|
||||
}
|
||||
}
|
||||
95
internal/storage/embeddeddolt/keyset_search_test.go
Normal file
95
internal/storage/embeddeddolt/keyset_search_test.go
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
//go:build cgo
|
||||
|
||||
package embeddeddolt_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// TestSearchIssuesKeysetEmbedded is the same-second-overflow keyset regression
|
||||
// on the embedded Dolt backend: a same-second group larger than a page pages completely under the
|
||||
// (created_at DESC, id ASC) keyset with no drop/dup, routed through the public
|
||||
// SearchIssues.
|
||||
func TestSearchIssuesKeysetEmbedded(t *testing.T) {
|
||||
te := newTestEnv(t, "ks")
|
||||
ctx := t.Context()
|
||||
|
||||
base := time.Date(2024, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
seeds := []struct {
|
||||
id string
|
||||
at time.Time
|
||||
}{
|
||||
{"k-newer", base.Add(time.Second)},
|
||||
{"k-a1", base}, {"k-a2", base}, {"k-a3", base}, {"k-a4", base}, {"k-a5", base},
|
||||
{"k-older", base.Add(-time.Second)},
|
||||
}
|
||||
for _, s := range seeds {
|
||||
iss := &types.Issue{ID: s.id, Title: s.id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, CreatedAt: s.at}
|
||||
if err := te.store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("create %s: %v", s.id, err)
|
||||
}
|
||||
}
|
||||
|
||||
want := []string{"k-newer", "k-a1", "k-a2", "k-a3", "k-a4", "k-a5", "k-older"}
|
||||
ids := func(issues []*types.Issue) []string {
|
||||
out := make([]string, len(issues))
|
||||
for i, iss := range issues {
|
||||
out[i] = iss.ID
|
||||
}
|
||||
return out
|
||||
}
|
||||
eq := func(got, exp []string) bool {
|
||||
if len(got) != len(exp) {
|
||||
return false
|
||||
}
|
||||
for i := range exp {
|
||||
if got[i] != exp[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
full, err := te.store.SearchIssues(ctx, "", types.IssueFilter{IDPrefix: "k-", SkipWisps: true, SortBy: "created", Limit: 100})
|
||||
if err != nil {
|
||||
t.Fatalf("SearchIssues(full): %v", err)
|
||||
}
|
||||
if got := ids(full); !eq(got, want) {
|
||||
t.Fatalf("full order = %v, want %v", got, want)
|
||||
}
|
||||
|
||||
const pageSize = 2
|
||||
var collected []string
|
||||
seen := map[string]bool{}
|
||||
var afterCreatedAt *time.Time
|
||||
afterID := ""
|
||||
for i := 0; i < 100; i++ {
|
||||
page, err := te.store.SearchIssues(ctx, "", types.IssueFilter{
|
||||
IDPrefix: "k-", SkipWisps: true, SortBy: "created", Limit: pageSize,
|
||||
AfterCreatedAt: afterCreatedAt, AfterID: afterID,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("SearchIssues(page %d): %v", i, err)
|
||||
}
|
||||
if len(page) == 0 {
|
||||
break
|
||||
}
|
||||
for _, iss := range page {
|
||||
if seen[iss.ID] {
|
||||
t.Fatalf("duplicate id %q across pages — same-second overflow lost", iss.ID)
|
||||
}
|
||||
seen[iss.ID] = true
|
||||
collected = append(collected, iss.ID)
|
||||
}
|
||||
last := page[len(page)-1]
|
||||
at := last.CreatedAt.UTC()
|
||||
afterCreatedAt = &at
|
||||
afterID = last.ID
|
||||
}
|
||||
if !eq(collected, want) {
|
||||
t.Fatalf("keyset paged order = %v, want %v (no drop/dup)", collected, want)
|
||||
}
|
||||
}
|
||||
|
|
@ -537,6 +537,20 @@ func (s *EmbeddedDoltStore) GetIssueComments(ctx context.Context, issueID string
|
|||
return result, err
|
||||
}
|
||||
|
||||
// GetIssueCommentsPage returns one keyset page of an issue's comments in
|
||||
// (created_at ASC, id ASC) order, resuming strictly after the cursor. See the
|
||||
// storage.Storage doc for the ordering, sargability, and page-walk-equals-full-
|
||||
// read contract.
|
||||
func (s *EmbeddedDoltStore) GetIssueCommentsPage(ctx context.Context, issueID string, after storage.CommentPageCursor, limit int) ([]*types.Comment, error) {
|
||||
var result []*types.Comment
|
||||
err := s.withConn(ctx, false, func(tx *sql.Tx) error {
|
||||
var err error
|
||||
result, err = issueops.GetIssueCommentsPageInTx(ctx, tx, issueID, after, limit)
|
||||
return err
|
||||
})
|
||||
return result, err
|
||||
}
|
||||
|
||||
func (s *EmbeddedDoltStore) GetEvents(ctx context.Context, issueID string, limit int) ([]*types.Event, error) {
|
||||
var result []*types.Event
|
||||
err := s.withConn(ctx, false, func(tx *sql.Tx) error {
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import (
|
|||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
|
|
@ -43,6 +44,114 @@ func GetIssueCommentsInTx(ctx context.Context, tx *sql.Tx, issueID string) ([]*t
|
|||
return comments, rows.Err()
|
||||
}
|
||||
|
||||
// Comment page-read tuning. Mirrors the EventsSince keyset clamp: an unbounded
|
||||
// page defeats the purpose of paging a long thread, so a non-positive limit
|
||||
// falls back to the default and any larger request is capped.
|
||||
const (
|
||||
defaultCommentsPageLimit = 100
|
||||
maxCommentsPageLimit = 500
|
||||
)
|
||||
|
||||
// CommentsKeysetPredicate is the SARGABLE (created_at ASC, id ASC) keyset resume
|
||||
// predicate GetIssueCommentsPageInTx ANDs in once a page cursor is set. Its three
|
||||
// ? placeholders bind, in order: created_at (the sargable lower bound),
|
||||
// created_at (strict), and id (the same-second tie-break).
|
||||
//
|
||||
// It is logically "(created_at, id) is strictly after the cursor" under
|
||||
// created_at ASC, id ASC — i.e. (created_at > ?) OR (created_at = ? AND id > ?)
|
||||
// — but rewritten with a redundant `created_at >= ?` leading bound so the
|
||||
// planner seeks the (issue_id, created_at, id) index (an IndexedTableAccess
|
||||
// range on Dolt) instead of scanning the issue's comments and filtering. The two
|
||||
// forms select the same rows: created_at >= C is true whenever the OR is, and
|
||||
// prunes only created_at < C, which the OR already excludes. It is exported so
|
||||
// the backend sargability guard EXPLAINs this exact string rather than a
|
||||
// hand-copied literal — a change here then breaks the guard.
|
||||
const CommentsKeysetPredicate = `created_at >= ? AND ((created_at > ?) OR (id > ?))`
|
||||
|
||||
// CommentsPageQuery returns the exact SQL GetIssueCommentsPageInTx executes for
|
||||
// the given comment table (comments or wisp_comments), cursor presence, and
|
||||
// already-clamped limit. The ? placeholders bind in order: issue_id, then — when
|
||||
// hasCursor — the three CommentsKeysetPredicate binds (created_at, created_at,
|
||||
// id). It is exported so the sargability guard EXPLAINs this production string
|
||||
// (with CommentsKeysetPredicate embedded) rather than a copy — a drift in either
|
||||
// the surrounding query or the predicate then breaks the guard.
|
||||
//
|
||||
//nolint:gosec // G201: table is a hardcoded routing constant and limit is an int the caller clamps; every runtime value is a bound parameter.
|
||||
func CommentsPageQuery(table string, hasCursor bool, limit int) string {
|
||||
query := fmt.Sprintf(`
|
||||
SELECT id, issue_id, author, text, created_at
|
||||
FROM %s
|
||||
WHERE issue_id = ?`, table)
|
||||
if hasCursor {
|
||||
query += " AND " + CommentsKeysetPredicate
|
||||
}
|
||||
query += fmt.Sprintf(" ORDER BY created_at ASC, id ASC LIMIT %d", limit)
|
||||
return query
|
||||
}
|
||||
|
||||
// GetIssueCommentsPageInTx returns one keyset page of an issue's comments within
|
||||
// an existing transaction, ordered by (created_at ASC, id ASC). It routes to
|
||||
// wisp_comments when issueID is an active wisp, exactly like GetIssueCommentsInTx.
|
||||
//
|
||||
// after is the resume position — the (created_at, id) of the last comment already
|
||||
// seen. The zero cursor starts from the beginning of the thread. Walking the
|
||||
// pages, feeding each page's last (created_at, id) back in as after, yields
|
||||
// exactly the same comments in the same order as GetIssueCommentsInTx, with no
|
||||
// dropped or duplicated comment even when several comments share a created_at
|
||||
// second, because id (the primary key) is a total tie-break.
|
||||
//
|
||||
// The cursor must come from a previously returned comment: created_at is a
|
||||
// DATETIME(0) column, so a sub-second cursor CreatedAt sorts after same-second
|
||||
// rows stored at the truncated second and skips them on resume (AddIssueCommentInTx
|
||||
// truncates its returned CreatedAt to avoid exactly this). And like an audit
|
||||
// feed, a comment inserted with a backdated created_at behind an in-progress
|
||||
// cursor is not seen by that forward-only walk.
|
||||
//
|
||||
// limit <= 0 falls back to defaultCommentsPageLimit (100); a larger limit is
|
||||
// capped at maxCommentsPageLimit (500), so a caller that pages until
|
||||
// len(page) < limit must keep limit <= 500 or terminate on an empty page
|
||||
// instead. A missing issue routes to comments and returns an empty page with no
|
||||
// error, matching GetIssueCommentsInTx.
|
||||
func GetIssueCommentsPageInTx(ctx context.Context, tx *sql.Tx, issueID string, after storage.CommentPageCursor, limit int) ([]*types.Comment, error) {
|
||||
if limit <= 0 {
|
||||
limit = defaultCommentsPageLimit
|
||||
}
|
||||
if limit > maxCommentsPageLimit {
|
||||
limit = maxCommentsPageLimit
|
||||
}
|
||||
|
||||
table := "comments"
|
||||
if IsActiveWispInTx(ctx, tx, issueID) {
|
||||
table = "wisp_comments"
|
||||
}
|
||||
|
||||
hasCursor := !after.CreatedAt.IsZero() || after.ID != ""
|
||||
args := []any{issueID}
|
||||
if hasCursor {
|
||||
// Bind the cursor time as time.Time, not a formatted string: created_at
|
||||
// is a DATETIME column, so a time.Time value compares correctly on every
|
||||
// backend while an RFC3339 string can mis-compare. Bound twice (the
|
||||
// sargable lower bound and the strict bound), then the id tie-break.
|
||||
args = append(args, after.CreatedAt, after.CreatedAt, after.ID)
|
||||
}
|
||||
|
||||
rows, err := tx.QueryContext(ctx, CommentsPageQuery(table, hasCursor, limit), args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get issue comments page from %s (after %v/%q): %w", table, after.CreatedAt, after.ID, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var comments []*types.Comment
|
||||
for rows.Next() {
|
||||
var c types.Comment
|
||||
if err := rows.Scan(&c.ID, &c.IssueID, &c.Author, &c.Text, &c.CreatedAt); err != nil {
|
||||
return nil, fmt.Errorf("get issue comments page: scan: %w", err)
|
||||
}
|
||||
comments = append(comments, &c)
|
||||
}
|
||||
return comments, rows.Err()
|
||||
}
|
||||
|
||||
// GetCommentCountsInTx returns comment counts per issue ID within a transaction.
|
||||
// Routes each ID to comments or wisp_comments based on wisp status.
|
||||
// Uses batched IN clauses (queryBatchSize) to avoid query-planner spikes.
|
||||
|
|
@ -109,9 +218,15 @@ func GetCommentCountsInTx(ctx context.Context, tx *sql.Tx, issueIDs []string) (m
|
|||
// AddIssueCommentInTx adds a structured comment to an issue within a transaction.
|
||||
// Routes to comments or wisp_comments based on wisp status.
|
||||
//
|
||||
// The insert timestamp is truncated to whole seconds to match the created_at
|
||||
// column's DATETIME(0) precision, so the returned Comment.CreatedAt equals the
|
||||
// stored value. That makes the returned comment safe to use directly as a
|
||||
// GetIssueCommentsPage cursor: an un-truncated sub-second CreatedAt would sort
|
||||
// after same-second rows stored at the truncated second and skip them on resume.
|
||||
//
|
||||
//nolint:gosec // G201: table names come from hardcoded constants
|
||||
func AddIssueCommentInTx(ctx context.Context, tx *sql.Tx, issueID, author, text string) (*types.Comment, error) {
|
||||
return ImportIssueCommentInTx(ctx, tx, issueID, author, text, time.Now().UTC())
|
||||
return ImportIssueCommentInTx(ctx, tx, issueID, author, text, time.Now().UTC().Truncate(time.Second))
|
||||
}
|
||||
|
||||
// ImportIssueCommentInTx adds a comment preserving the original timestamp.
|
||||
|
|
|
|||
|
|
@ -389,6 +389,50 @@ func TestBuildIssueFilterClauses_PinnedFilter(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestBuildIssueFilterClauses_IsBlockedFilter(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
blockedTrue := true
|
||||
blockedFalse := false
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
isBlocked *bool
|
||||
wantSQL string
|
||||
wantArg int
|
||||
}{
|
||||
{name: "is_blocked=true", isBlocked: &blockedTrue, wantSQL: "is_blocked = ?", wantArg: 1},
|
||||
{name: "is_blocked=false", isBlocked: &blockedFalse, wantSQL: "is_blocked = ?", wantArg: 0},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
clauses, args, err := BuildIssueFilterClauses("", types.IssueFilter{IsBlocked: tt.isBlocked}, IssuesFilterTables)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if len(clauses) != 1 || clauses[0] != tt.wantSQL {
|
||||
t.Fatalf("got clauses %v, want [%q]", clauses, tt.wantSQL)
|
||||
}
|
||||
if len(args) != 1 || args[0] != tt.wantArg {
|
||||
t.Errorf("got args %v, want [%d] (index-backed integer bind)", args, tt.wantArg)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// nil is the unset case: no is_blocked clause is emitted (filter is inert).
|
||||
clauses, _, err := BuildIssueFilterClauses("", types.IssueFilter{IsBlocked: nil}, IssuesFilterTables)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
for _, c := range clauses {
|
||||
if strings.Contains(c, "is_blocked") {
|
||||
t.Errorf("nil IsBlocked emitted an is_blocked clause %q, want none", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildIssueFilterClauses_IDFilters(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,25 @@
|
|||
-- Reverse 0056: drop the (issue_id, created_at, id) composite indexes on
|
||||
-- comments and wisp_comments. Guarded for the same idempotency reason as up so
|
||||
-- it must not error when the index is already absent. The "index present" guard
|
||||
-- also covers a missing wisp_comments table (an absent table has no index), so
|
||||
-- the wisp half is a no-op on a fresh clone too.
|
||||
|
||||
SET @has_c = (
|
||||
SELECT IF(COUNT(*) > 0, 1, 0)
|
||||
FROM INFORMATION_SCHEMA.STATISTICS
|
||||
WHERE TABLE_SCHEMA = DATABASE()
|
||||
AND TABLE_NAME = 'comments'
|
||||
AND INDEX_NAME = 'idx_comments_issue_created_id'
|
||||
);
|
||||
SET @sql = IF(@has_c = 1, 'DROP INDEX idx_comments_issue_created_id ON comments', 'SELECT 1');
|
||||
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
||||
|
||||
SET @has_w = (
|
||||
SELECT IF(COUNT(*) > 0, 1, 0)
|
||||
FROM INFORMATION_SCHEMA.STATISTICS
|
||||
WHERE TABLE_SCHEMA = DATABASE()
|
||||
AND TABLE_NAME = 'wisp_comments'
|
||||
AND INDEX_NAME = 'idx_wisp_comments_issue_created_id'
|
||||
);
|
||||
SET @sql = IF(@has_w = 1, 'DROP INDEX idx_wisp_comments_issue_created_id ON wisp_comments', 'SELECT 1');
|
||||
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
-- Add a (issue_id, created_at, id) composite index to comments and wisp_comments
|
||||
-- so the keyset comment-page read (GetIssueCommentsPage) seeks the index for a
|
||||
-- single issue's thread in (created_at, id) order instead of scanning the issue's
|
||||
-- comments and sorting. The pre-existing single-column idx_comments_issue /
|
||||
-- idx_wisp_comments_issue indexes can seek issue_id but leave the created_at
|
||||
-- range and the id tie-break to a filter+sort.
|
||||
--
|
||||
-- Numbered 0056: the next contiguous version after main's 0055 (the migration
|
||||
-- loader requires gap-free versions — COUNT == MAX, enforced by
|
||||
-- TestAllMigrationsSQLAppliesThroughDoltCLIAndRecordsLatestVersion — so we
|
||||
-- cannot pre-emptively skip to 0057). Several in-flight branches (claim-fence,
|
||||
-- holder-token, revision) also add an 0056 on their own 0055 base; the loader
|
||||
-- panics on a duplicate version (schema.go checkNoDuplicateVersions), so
|
||||
-- whichever lands second rebases onto the new main and bumps to the next free
|
||||
-- number. That collision is resolved loudly at merge, never silently.
|
||||
--
|
||||
-- Guarded against the current schema so the migration is idempotent (a clone may
|
||||
-- re-apply it; see 0052) — it must not error when the index is already present.
|
||||
--
|
||||
-- The wisp half is additionally guarded on the wisp_comments table EXISTING:
|
||||
-- wisp_% tables are dolt-ignored clone-local state that the ignored-migration
|
||||
-- chain materializes AFTER the main migrations run, so on a fresh clone's first
|
||||
-- writable open wisp_comments does not yet exist when this migration runs.
|
||||
-- Treat that as a no-op (exactly like 0035/0037/0053) — the fresh wisp_comments
|
||||
-- is created with this composite index directly by the canonical wisp schema in
|
||||
-- ignored/0001_create_local_state_tables.up.sql, while this migration upgrades
|
||||
-- the wisp_comments of already-materialized (existing) workspaces.
|
||||
|
||||
-- comments is a durable table present in every workspace once 0004 has run, so
|
||||
-- it needs only the index-missing guard.
|
||||
SET @needs_c = (
|
||||
SELECT IF(COUNT(*) = 0, 1, 0)
|
||||
FROM INFORMATION_SCHEMA.STATISTICS
|
||||
WHERE TABLE_SCHEMA = DATABASE()
|
||||
AND TABLE_NAME = 'comments'
|
||||
AND INDEX_NAME = 'idx_comments_issue_created_id'
|
||||
);
|
||||
SET @sql = IF(@needs_c = 1, 'CREATE INDEX idx_comments_issue_created_id ON comments (issue_id, created_at, id)', 'SELECT 1');
|
||||
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
||||
|
||||
-- wisp_comments may not exist yet on a fresh clone: guard on table existence AND
|
||||
-- index-missing so the CREATE INDEX (and its PREPARE) is skipped entirely when
|
||||
-- the table is absent.
|
||||
SET @has_wisp_comments = (
|
||||
SELECT COUNT(*)
|
||||
FROM INFORMATION_SCHEMA.TABLES
|
||||
WHERE TABLE_SCHEMA = DATABASE()
|
||||
AND TABLE_NAME = 'wisp_comments'
|
||||
);
|
||||
SET @has_wisp_index = (
|
||||
SELECT COUNT(*)
|
||||
FROM INFORMATION_SCHEMA.STATISTICS
|
||||
WHERE TABLE_SCHEMA = DATABASE()
|
||||
AND TABLE_NAME = 'wisp_comments'
|
||||
AND INDEX_NAME = 'idx_wisp_comments_issue_created_id'
|
||||
);
|
||||
SET @sql = IF(@has_wisp_comments > 0 AND @has_wisp_index = 0, 'CREATE INDEX idx_wisp_comments_issue_created_id ON wisp_comments (issue_id, created_at, id)', 'SELECT 1');
|
||||
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
-- Keyset comment-page index on wisp_comments (issue_id, created_at, id) — the
|
||||
-- wisp twin of main migration 0056's index on durable comments. The ignored
|
||||
-- chain runs per clone with its own cursor, so BOTH fresh clones (which
|
||||
-- materialize wisp_comments via ignored/0001) and already-upgraded workspaces
|
||||
-- pick this up; main 0056 covers only workspaces whose wisp tables existed
|
||||
-- when it ran. Guarded like ignored/0006: create only when the table exists
|
||||
-- and the index is absent.
|
||||
SET @needs_index = IF(
|
||||
(SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLES
|
||||
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'wisp_comments') > 0
|
||||
AND
|
||||
(SELECT COUNT(*) FROM INFORMATION_SCHEMA.STATISTICS
|
||||
WHERE TABLE_SCHEMA = DATABASE()
|
||||
AND TABLE_NAME = 'wisp_comments'
|
||||
AND INDEX_NAME = 'idx_wisp_comments_issue_created_id') = 0,
|
||||
1, 0
|
||||
);
|
||||
SET @sql = IF(@needs_index = 1,
|
||||
'CREATE INDEX idx_wisp_comments_issue_created_id ON wisp_comments(issue_id, created_at, id)',
|
||||
'SELECT 1');
|
||||
PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
||||
|
|
@ -425,6 +425,44 @@ DELETE FROM schema_migrations WHERE version = %d;
|
|||
`SELECT COUNT(*) AS c FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'wisps'`, "0")
|
||||
}
|
||||
|
||||
// TestMigration0056NoopsWithoutWispCommentsThroughDoltCLI pins the fresh-clone
|
||||
// guard for the comments-keyset index: main migrations run before the ignored
|
||||
// chain materializes the clone-local wisp_% tables, so migration 0056 must
|
||||
// no-op its wisp half when wisp_comments is absent. A bare
|
||||
// CREATE INDEX ... ON wisp_comments would fail at PREPARE ("table not found")
|
||||
// and brick the first writable open. The durable comments half still runs.
|
||||
// AllMigrationsSQL is main-source only (it never creates wisp_comments), so
|
||||
// applying it already runs 0056 against an absent wisp_comments; the isolated
|
||||
// re-apply then asserts the no-op explicitly.
|
||||
func TestMigration0056NoopsWithoutWispCommentsThroughDoltCLI(t *testing.T) {
|
||||
testutil.RequireDoltBinary(t)
|
||||
|
||||
dir := filepath.Join(t.TempDir(), "comments-keyset-no-wisps")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatalf("create no-wisps dir: %v", err)
|
||||
}
|
||||
runDoltCommand(t, dir, "init", "--name", "test", "--email", "test@example.com")
|
||||
runDoltSQL(t, dir, AllMigrationsSQL())
|
||||
|
||||
seedSQL := fmt.Sprintf(`
|
||||
DROP TABLE IF EXISTS wisp_comments;
|
||||
DELETE FROM schema_migrations WHERE version = %d;
|
||||
`, LatestVersion())
|
||||
migrationSQL, err := mainSource.files.ReadFile("migrations/0056_add_comments_keyset_index.up.sql")
|
||||
if err != nil {
|
||||
t.Fatalf("read 0056 migration: %v", err)
|
||||
}
|
||||
runDoltSQL(t, dir, seedSQL+"\n"+string(migrationSQL))
|
||||
|
||||
// Wisp half no-oped: no wisp_comments table was created, no error raised.
|
||||
requireDoltCount(t, dir,
|
||||
`SELECT COUNT(*) AS c FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'wisp_comments'`, "0")
|
||||
// Comments half ran: the durable composite index is present (one distinct
|
||||
// index name; a composite spans three STATISTICS rows).
|
||||
requireDoltCount(t, dir,
|
||||
`SELECT COUNT(DISTINCT INDEX_NAME) AS c FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'comments' AND INDEX_NAME = 'idx_comments_issue_created_id'`, "1")
|
||||
}
|
||||
|
||||
func TestMigration0053RepairsRigWispsThroughDoltCLI(t *testing.T) {
|
||||
testutil.RequireDoltBinary(t)
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,23 @@ import (
|
|||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// KeysetCreatedAtIDPredicate is the SARGABLE (created_at DESC, id ASC) keyset
|
||||
// predicate emitted for IssueFilter.AfterCreatedAt/AfterID. Its three ?
|
||||
// placeholders bind, in order: created_at (the sargable upper bound), created_at
|
||||
// (strict, drops the same-second rows already returned), and id (the same-second
|
||||
// tie-break).
|
||||
//
|
||||
// It is logically "(created_at, id) is strictly after (cursor)" under
|
||||
// created_at DESC, id ASC — i.e. (created_at < ?) OR (created_at = ? AND id > ?)
|
||||
// — but rewritten with a redundant `created_at <= ?` leading bound so the planner
|
||||
// seeks idx_issues_created_at (an IndexedTableAccess range on Dolt, an index/
|
||||
// BitmapOr scan on Postgres) instead of full-scanning and filtering. The two
|
||||
// forms select the same rows: created_at <= C is true whenever the OR is, and
|
||||
// prunes only created_at > C, which the OR already excludes. It is exported so
|
||||
// the backend sargability guards EXPLAIN this exact string rather than a copy —
|
||||
// a change here then breaks the guard.
|
||||
const KeysetCreatedAtIDPredicate = "(created_at <= ? AND ((created_at < ?) OR (id > ?)))"
|
||||
|
||||
// BuildIssueFilterClauses builds WHERE clause fragments and args from a query
|
||||
// string and IssueFilter. The tables parameter controls which table names are
|
||||
// referenced in subqueries (issues vs wisps).
|
||||
|
|
@ -192,6 +209,19 @@ func BuildIssueFilterClauses(query string, filter types.IssueFilter, tables Filt
|
|||
whereClauses = append(whereClauses, "(is_template = 0 OR is_template IS NULL)")
|
||||
}
|
||||
}
|
||||
if filter.IsBlocked != nil {
|
||||
// is_blocked is NOT NULL DEFAULT 0 on both issues and wisps, so a plain
|
||||
// equality is exact (no IS NULL arm needed) and index-backed by
|
||||
// idx_issues_is_blocked(is_blocked, status). Bound as an int so the same
|
||||
// clause is portable across every backend (Dolt/MySQL/SQLite native, and
|
||||
// pgdialect rewrites ? → $n).
|
||||
blocked := 0
|
||||
if *filter.IsBlocked {
|
||||
blocked = 1
|
||||
}
|
||||
whereClauses = append(whereClauses, "is_blocked = ?")
|
||||
args = append(args, blocked)
|
||||
}
|
||||
|
||||
if filter.EmptyDescription {
|
||||
whereClauses = append(whereClauses, "(description IS NULL OR description = '')")
|
||||
|
|
@ -223,6 +253,18 @@ func BuildIssueFilterClauses(query string, filter types.IssueFilter, tables Filt
|
|||
}
|
||||
}
|
||||
|
||||
if filter.AfterCreatedAt != nil {
|
||||
// Bind the cursor time as time.Time, not a formatted string: the issues/
|
||||
// wisps created_at columns are DATETIME (NUMERIC affinity), so an RFC3339
|
||||
// string parameter mis-compares on the SQLite backend, while a time.Time
|
||||
// value compares correctly on every backend — the same binding EventsSince
|
||||
// uses. Bound twice (the sargable upper bound and the strict bound), then
|
||||
// the id tie-break.
|
||||
ac := *filter.AfterCreatedAt
|
||||
whereClauses = append(whereClauses, KeysetCreatedAtIDPredicate)
|
||||
args = append(args, ac, ac, filter.AfterID)
|
||||
}
|
||||
|
||||
if filter.Deferred {
|
||||
whereClauses = append(whereClauses, "(defer_until IS NOT NULL OR status = ?)")
|
||||
args = append(args, types.StatusDeferred)
|
||||
|
|
|
|||
92
internal/storage/sqlbuild/keyset_test.go
Normal file
92
internal/storage/sqlbuild/keyset_test.go
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
package sqlbuild
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// TestKeysetPredicateEmission pins the (created_at DESC, id ASC) keyset predicate
|
||||
// BuildIssueFilterClauses emits for IssueFilter.AfterCreatedAt/AfterID: the exact
|
||||
// sargable SQL fragment (single-sourced from KeysetCreatedAtIDPredicate) and its
|
||||
// three bound args in order (created_at, created_at, id).
|
||||
func TestKeysetPredicateEmission(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cur := time.Date(2024, 3, 2, 1, 0, 0, 0, time.UTC)
|
||||
|
||||
// No keyset set: predicate absent.
|
||||
clauses, args, err := BuildIssueFilterClauses("", types.IssueFilter{}, IssuesFilterTables)
|
||||
if err != nil {
|
||||
t.Fatalf("BuildIssueFilterClauses (no keyset): %v", err)
|
||||
}
|
||||
for _, c := range clauses {
|
||||
if strings.Contains(c, KeysetCreatedAtIDPredicate) {
|
||||
t.Fatalf("keyset predicate emitted with no AfterCreatedAt set: %v", clauses)
|
||||
}
|
||||
}
|
||||
_ = args
|
||||
|
||||
// Keyset set: exactly one predicate clause equal to the single-sourced
|
||||
// constant, with three args in bind order.
|
||||
clauses, args, err = BuildIssueFilterClauses("", types.IssueFilter{
|
||||
AfterCreatedAt: &cur,
|
||||
AfterID: "bd-42",
|
||||
}, IssuesFilterTables)
|
||||
if err != nil {
|
||||
t.Fatalf("BuildIssueFilterClauses (keyset): %v", err)
|
||||
}
|
||||
found := 0
|
||||
for _, c := range clauses {
|
||||
if c == KeysetCreatedAtIDPredicate {
|
||||
found++
|
||||
}
|
||||
}
|
||||
if found != 1 {
|
||||
t.Fatalf("keyset predicate clause count = %d, want 1; clauses=%v", found, clauses)
|
||||
}
|
||||
// The cursor time binds as time.Time (twice: sargable + strict bound), then
|
||||
// the id — bound as a value, not a formatted string, so the DATETIME columns
|
||||
// compare correctly on every backend.
|
||||
want := []any{cur, cur, "bd-42"}
|
||||
if len(args) != len(want) {
|
||||
t.Fatalf("args = %v, want %v", args, want)
|
||||
}
|
||||
for i := range want {
|
||||
if args[i] != want[i] {
|
||||
t.Fatalf("arg[%d] = %v (%T), want %v (%T)", i, args[i], args[i], want[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestKeysetComposesWithCreatedBefore proves the new keyset field does not
|
||||
// displace CreatedBefore: both predicates are emitted, and the keyset upper
|
||||
// bound (created_at <=) is distinct from CreatedBefore's (created_at <).
|
||||
func TestKeysetComposesWithCreatedBefore(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cur := time.Date(2024, 3, 2, 1, 0, 0, 0, time.UTC)
|
||||
before := time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
clauses, args, err := BuildIssueFilterClauses("", types.IssueFilter{
|
||||
CreatedBefore: &before,
|
||||
AfterCreatedAt: &cur,
|
||||
AfterID: "bd-7",
|
||||
}, IssuesFilterTables)
|
||||
if err != nil {
|
||||
t.Fatalf("BuildIssueFilterClauses: %v", err)
|
||||
}
|
||||
joined := strings.Join(clauses, " AND ")
|
||||
if !strings.Contains(joined, KeysetCreatedAtIDPredicate) {
|
||||
t.Fatalf("keyset predicate missing when composed with CreatedBefore: %v", clauses)
|
||||
}
|
||||
if !strings.Contains(joined, "created_at < ?") {
|
||||
t.Fatalf("CreatedBefore predicate (created_at < ?) missing: %v", clauses)
|
||||
}
|
||||
// CreatedBefore contributes one arg, keyset contributes three.
|
||||
if len(args) != 4 {
|
||||
t.Fatalf("arg count = %d, want 4 (1 CreatedBefore + 3 keyset)", len(args))
|
||||
}
|
||||
}
|
||||
|
|
@ -56,6 +56,18 @@ var ErrCloseBlocked = errors.New("cannot close blocked issue")
|
|||
// precondition from other errors.
|
||||
var ErrVersionMismatch = errors.New("version mismatch")
|
||||
|
||||
// CommentPageCursor is the resume position for a keyset page of an issue's
|
||||
// comments: the (created_at, id) of the last comment already returned. The zero
|
||||
// value starts a walk from the beginning of the thread.
|
||||
//
|
||||
// It lives in the storage package (rather than issueops) because issueops
|
||||
// imports storage — the reverse would be an import cycle — so the shared cursor
|
||||
// type is defined here and referenced from the issueops query layer.
|
||||
type CommentPageCursor struct {
|
||||
CreatedAt time.Time
|
||||
ID string
|
||||
}
|
||||
|
||||
// Storage is the interface satisfied by *dolt.DoltStore.
|
||||
// Consumers depend on this interface rather than on the concrete type so that
|
||||
// alternative implementations (mocks, proxies, etc.) can be substituted.
|
||||
|
|
@ -146,6 +158,32 @@ type Storage interface {
|
|||
// Comments and events
|
||||
AddIssueComment(ctx context.Context, issueID, author, text string) (*types.Comment, error)
|
||||
GetIssueComments(ctx context.Context, issueID string) ([]*types.Comment, error)
|
||||
// GetIssueCommentsPage returns one keyset page of an issue's comments in the
|
||||
// stable (created_at ASC, id ASC) total order, resuming strictly after the
|
||||
// after cursor (the zero cursor starts from the beginning of the thread).
|
||||
// id is the primary key, so the same-second tie-break is total: a thread
|
||||
// with several comments in the same created_at second still pages
|
||||
// completely, and concatenating every page of a full walk yields exactly the
|
||||
// same comments in the same order as GetIssueComments — no dropped or
|
||||
// duplicated comment. The resume predicate is sargable: it seeks the
|
||||
// (issue_id, created_at, id) index rather than scanning the whole thread.
|
||||
//
|
||||
// The after cursor MUST come from a comment previously returned by a read
|
||||
// (this method or GetIssueComments), whose CreatedAt matches the stored
|
||||
// DATETIME second. Feeding a cursor with a sub-second CreatedAt can skip
|
||||
// same-second rows (AddIssueComment already truncates its returned CreatedAt
|
||||
// for this reason).
|
||||
//
|
||||
// Keyset semantics, like an audit feed: a comment inserted with a backdated
|
||||
// created_at that lands behind an in-progress cursor is not seen by that
|
||||
// walk — the walk only moves forward. A whole-thread read or a fresh walk
|
||||
// still returns it.
|
||||
//
|
||||
// limit <= 0 uses a store default (100); a larger limit is capped at 500. A
|
||||
// caller that pages until len(page) < limit must therefore keep limit <= 500
|
||||
// or use empty-page termination instead: a request for limit > 500 always
|
||||
// returns at most 500 rows and would stop a len-based loop one page early.
|
||||
GetIssueCommentsPage(ctx context.Context, issueID string, after CommentPageCursor, limit int) ([]*types.Comment, error)
|
||||
GetEvents(ctx context.Context, issueID string, limit int) ([]*types.Event, error)
|
||||
GetAllEventsSince(ctx context.Context, since time.Time) ([]*types.Event, error)
|
||||
|
||||
|
|
|
|||
|
|
@ -466,6 +466,14 @@ func (s *InstrumentedStorage) GetIssueComments(ctx context.Context, issueID stri
|
|||
return v, err
|
||||
}
|
||||
|
||||
func (s *InstrumentedStorage) GetIssueCommentsPage(ctx context.Context, issueID string, after storage.CommentPageCursor, limit int) ([]*types.Comment, error) {
|
||||
attrs := []attribute.KeyValue{attribute.String("bd.issue.id", issueID)}
|
||||
ctx, span, t := s.op(ctx, "GetIssueCommentsPage", attrs...)
|
||||
v, err := s.inner.GetIssueCommentsPage(ctx, issueID, after, limit)
|
||||
s.done(ctx, span, t, err, attrs...)
|
||||
return v, err
|
||||
}
|
||||
|
||||
func (s *InstrumentedStorage) GetEvents(ctx context.Context, issueID string, limit int) ([]*types.Event, error) {
|
||||
attrs := []attribute.KeyValue{attribute.String("bd.issue.id", issueID)}
|
||||
ctx, span, t := s.op(ctx, "GetEvents", attrs...)
|
||||
|
|
|
|||
|
|
@ -1298,6 +1298,23 @@ type IssueFilter struct {
|
|||
StartedAfter *time.Time
|
||||
StartedBefore *time.Time
|
||||
|
||||
// Keyset pagination over the (created_at DESC, id ASC) total order.
|
||||
//
|
||||
// When AfterCreatedAt != nil the query is restricted to rows strictly after
|
||||
// the keyset position (AfterCreatedAt, AfterID) under that order — i.e.
|
||||
// (created_at < AfterCreatedAt) OR (created_at = AfterCreatedAt AND id > AfterID).
|
||||
// id is the primary key, so the tie-break is total: a same-second group
|
||||
// larger than one page still pages completely with no dropped or duplicated
|
||||
// row (unlike a created_at-only cursor, which loses same-second overflow).
|
||||
// AfterID is meaningful only when AfterCreatedAt is set; "" starts the
|
||||
// same-second group from its first id.
|
||||
//
|
||||
// This composes with every other filter (including CreatedBefore, which it
|
||||
// does not replace). Pair it with SortBy="created", SortDesc=false so the
|
||||
// ORDER BY is created_at DESC, id ASC — the order the predicate assumes.
|
||||
AfterCreatedAt *time.Time
|
||||
AfterID string
|
||||
|
||||
// Empty/null checks
|
||||
EmptyDescription bool
|
||||
NoAssignee bool
|
||||
|
|
@ -1316,6 +1333,12 @@ type IssueFilter struct {
|
|||
// Pinned filtering
|
||||
Pinned *bool // Filter by pinned flag (nil = any, true = only pinned, false = only non-pinned)
|
||||
|
||||
// Blocked filtering: the denormalized, transitive is_blocked column (direct ∨
|
||||
// inherited parent-child ∨ waits-for gate), maintained by the write paths and
|
||||
// index-backed by idx_issues_is_blocked(is_blocked, status). The projection
|
||||
// column alone is not a filter; this optional predicate makes it one.
|
||||
IsBlocked *bool // nil = any, true = only is_blocked, false = only unblocked
|
||||
|
||||
// Template filtering
|
||||
IsTemplate *bool // Filter by template flag (nil = any, true = only templates, false = exclude templates)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue