From 8aaefe2d4faae7b95391dd67086ee17c0dd5726a Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 27 Aug 2026 09:21:12 -0700 Subject: [PATCH] feat(httpapi): serve `bd list`'s order with `sort` on listIssues (#5666) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(httpapi): serve `bd list`'s order with `sort` on listIssues The issue listing served one order, `(created_at DESC, id ASC)`, because the cursor is a keyset position in it. The spec stated the constraint in its own words: "the sort order is welded to the cursor contract, and a new order needs new surface." That welding put the cost on every client wanting `bd list`'s ordering — the only way to get it was to page the whole result set and sort locally, which is the one cost on this surface that grows with the project. Measured against a real `bd serve` over 1400 rows at the 200-row page an HTTP client actually uses: walk-and-sort is 7 requests plus a client-side comparator, `?sort=priority` is 1. Both are checked against `bd list --json` row for row, because the claim that matters is not "fewer requests" but "the same order". `sort` takes `created` and `priority` and nothing else. Each value is a keyset contract — a position shape, a strictly-after predicate, an index — not a display preference, and both keys are provably total (priority and created_at are non-null, id is unique) so a page boundary inside a run of equal keys resolves on id. The seven other orders `bd list --sort` takes have no such key: `id` is natural-numeric and no database expresses it, `updated` moves on every write, `closed` is nullable, and status/title/type/assignee are mutable and unindexed. `priority` here is (priority ASC, created_at DESC, id ASC), which is both `bd list`'s flagless order and what `--sort priority` produces — a stable priority sort over the created order is the same sequence — so one served order retires the walk for both. Absent `sort` still means `created`, permanently: it is the compatibility contract for every client written before the parameter existed. THE CURSOR NOW CARRIES ITS ORDER. This is the half that makes the parameter safe rather than merely useful, and the obvious implementation gets it wrong. The token was base64 of `{t,i}` with no binding, so a created-order cursor replayed under `sort=priority` decoded perfectly and was read as a position in a different total order — a page that skips and duplicates rows, served with a 200, undetectable by the client. Tokens are now `v2` with an explicit order member, and decode takes the wanted order as a parameter so a mismatch is unrepresentable at the seam rather than merely detectable past it. Outstanding `v1` tokens stay readable as the created-order positions they are, so no traversal in flight has to restart. This does not contradict "the token carries no filters": filters select the set, the order decides what the position means. Storage grows one field. IssueFilter.AfterPriority extends the existing position; the predicate nests the created one rather than restating it, because the created/id comparison is only reachable at the cursor's own priority. Flattening it the way the created predicate's leading-bound trick invites looks equivalent and is not — it re-delivers rows at the cursor's priority created after it, which the new conformance walk catches against real Dolt. NO INDEX MIGRATION, by measurement rather than by omission. A composite (priority, created_at, id) index was tried on Dolt 2.2.0 against the production predicate, in both the plain ascending and the mixed-direction spelling Dolt's DDL accepts: the plan is byte-identical in all three cases, keeping the single-column priority seek and the TopN. It would have bought write amplification on every insert and every priority update for no measured change. The sargability guard pins what is achieved — an IndexedTableAccess range on idx_issues_priority — and records the measurement to redo if Dolt learns to stream the composite. Agent-Signature: claude-opus-5 Co-authored-by: CI Bot * fix(changelog): drop leftover diff3 conflict marker in sort entry The rebased adopt-PR head left a stray '|||||||' merge-base marker on the first line of the new `sort` CHANGELOG entry (CHANGELOG.md:381), which `git diff --check` flagged as a leftover conflict marker. Remove only the marker line; the resolved entry content is unchanged. Maintainer fixup for PR #5666 adoption review. Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: CI Bot Co-authored-by: Claude Opus 4.8 --- .github/scripts/proxied-cmd-test-shards.txt | 7 + CHANGELOG.md | 47 +++ backend/conformance/reader_contract.go | 135 ++++++++ backend/conformance/role_bundle_cases.go | 1 + ...erve_list_sort_proxied_integration_test.go | 317 ++++++++++++++++++ cmd/bd/serve_reads_parity_integration_test.go | 51 ++- engdocs/design/bd-serve-v0.md | 71 +++- internal/httpapi/apigen/types.gen.go | 38 ++- internal/httpapi/cursor.go | 139 +++++++- internal/httpapi/list_sort_test.go | 283 ++++++++++++++++ internal/httpapi/reads.go | 34 +- internal/httpapi/reads_test.go | 45 ++- internal/httpapi/spec/openapi.v0.yaml | 149 ++++++-- internal/storage/dolt/keyset_search_test.go | 54 +++ internal/storage/dolt/reader_contract_test.go | 9 + .../embeddeddolt/reader_contract_test.go | 6 + internal/storage/sqlbuild/filter.go | 43 ++- internal/storage/sqlbuild/keyset_test.go | 97 ++++++ internal/storage/uow/reader_contract_test.go | 8 + internal/types/types.go | 27 ++ internal/workapi/list.go | 1 + internal/workapi/list_keyset_test.go | 84 +++++ .../workapi/testdata/list_filter_golden.json | 39 +++ issueops/reader.go | 16 + issueops/reader_ready_scope.go | 8 +- 25 files changed, 1600 insertions(+), 109 deletions(-) create mode 100644 cmd/bd/serve_list_sort_proxied_integration_test.go create mode 100644 internal/httpapi/list_sort_test.go create mode 100644 internal/workapi/list_keyset_test.go diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index e725e37de..1c5e9dbed 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -14,12 +14,19 @@ # oracle — the behavioural half of the anti-drift claim — so it being IN this # lane, and this lane being required by PR Risk's CI gate, is the thing that # makes that claim enforced on PRs rather than asserted in a doc comment. +# +# TestProxiedServerListSortRetiresTheWalk is listed the same way, at the shard +# `cksum` was already giving it. It is the most expensive entry in the file — +# it seeds 1400 rows and walks the listing three ways — and it is here for the +# same reason as the parity oracle: it is the measurement behind the claim that +# `sort=priority` retires the client-side walk, so it has to run on PRs. 15 1 TestProxiedServerCount 15 1 TestProxiedServerDelete 15 1 TestProxiedServerDoltRemoteRemove 15 1 TestProxiedServerExport 15 1 TestProxiedServerGateLifecycle +15 1 TestProxiedServerListSortRetiresTheWalk 15 1 TestProxiedServerMolCurrent 15 1 TestProxiedServerPathHelpers 15 1 TestProxiedServerTodo diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f6c18930..5748fe559 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -432,6 +432,53 @@ stopgap (`BD_IGNORE_SCHEMA_SKEW=1`). `go install github.com/steveyegge/beads/cmd/bd@latest` resolves to this release instead of the accidental one. (The retract block is also carried on main so future tags keep the retractions.) +- **`sort` on `GET /v0/beads/issues`.** The listing served one order — + `(created_at DESC, id ASC)` — because the cursor is a keyset position in it, + and the spec said so in its own words: "the sort order is welded to the cursor + contract, and a new order needs new surface." The cost of that welding fell on + every client that wanted `bd list`'s ordering, because the only way to get it + was to page the whole result set and re-sort locally. Measured over a + 1400-row store at the 200-row page an HTTP client actually uses, that is + **7 requests plus a client-side comparator; `?sort=priority` is 1** + (`TestProxiedServerListSortRetiresTheWalk`, which runs both strategies against + a real `bd serve` and checks both against `bd list --json` row for row). It is + the one cost on this surface that got worse as a project grew. + + **Two values, and the set is closed.** `created` is the existing order, now + spellable. `priority` is `(priority ASC, created_at DESC, id ASC)` — `bd + list`'s flagless ordering, which is also what `bd list --sort priority` + produces, so one served order retires the walk for both. The other seven + orders `bd list --sort` takes are not offered: each value here is a cursor + contract needing a key proven total, and `id` is a natural-numeric order no + database expresses, `updated` moves on every write, `closed` is nullable, and + `status`/`title`/`type`/`assignee` are mutable and unindexed. + + **Absent `sort` still means `created`, permanently.** It is the compatibility + contract for every client written before the parameter existed; changing it + would alter which rows a truncated page contains, with no error to notice it + by. A server that predates the parameter answers `param: "sort"` with + `reason: "unknown_parameter"`, which is the per-parameter capability probe a + client dispatches on to fall back. + + **The cursor now carries its order, and a mismatch is refused.** This is the + half that makes the parameter safe rather than merely useful. The token was + base64 of `{t,i}` with no binding, so a `created`-order cursor replayed under + `sort=priority` would have decoded perfectly and been read as a position in a + different total order — a page that both skips and duplicates rows, served + with a 200, undetectable by the client. Tokens are now `v2` with an explicit + order member and decode refuses any token whose order differs from the + request's, as `invalid_cursor` (documented recovery: restart paging). + Outstanding `v1` tokens stay readable as the `created`-order positions they + are, so **no traversal in flight has to restart**. This does not contradict + "the token carries no filters": filters select the set, the order decides what + the position means. + + **`priority` is a mutable key**, which `created_at` is not, and the spec says + so: under `created` only new rows move relative to a walk, while a priority + update moves an existing row too, so it can be seen twice or missed. That is + the already-documented "a cursor pins a position, not a snapshot" caveat + reached by a second route, not a new class of error — unchanged data never + skips or repeats under either order. ## [1.2.1] - 2026-08-11 diff --git a/backend/conformance/reader_contract.go b/backend/conformance/reader_contract.go index 0bb2f55c1..6910a5bbb 100644 --- a/backend/conformance/reader_contract.go +++ b/backend/conformance/reader_contract.go @@ -2397,6 +2397,141 @@ func RunReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing(t assertReaderPageIDs(t, `List from (the group's second, "")`, fromGroupStart, append(slices.Clone(group), older)) } +// RunReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing +// is the walk over the SECOND served order, (priority ASC, created_at DESC, +// id ASC), and the proof that its three-part key is total. +// +// WHY A SECOND WALK CASE AND NOT A PARAMETER ON THE FIRST: the created order's +// key is (created_at, id) and this one's is (priority, created_at, id), so the +// property at risk is different. There the tie-break is reached whenever two +// rows share a second; here it is reached only when two rows share BOTH a +// priority and a second, and the leg above it — created_at DESC WITHIN one +// priority — does not exist there at all. A body can pass the created walk and +// still get this one wrong in two ways that both look like a correct page: +// carrying only the (created_at, id) half of the position, which drops every +// row of a higher-numbered priority created after the cursor; or flattening the +// predicate's priority-equal arm, which re-delivers rows at the cursor's own +// priority created after it. +// +// THE FIXTURE IS BUILT SO THE PAGE BREAK LANDS INSIDE THE EQUAL-KEY RUN, which +// is the input that makes totality observable rather than merely asserted. +// Five rows share one (priority, created_at) and the page is two, so two of the +// three boundaries fall mid-run and the walk can only continue on `id`. A +// fixture whose runs happened to align with its pages would pass against a +// two-part key and prove nothing. +// +// THE NEIGHBORS ARE THE OTHER HALF OF THE FIXTURE. `sooner` shares the run's +// priority and is NEWER, so it must be delivered before the run and never +// again after it — it is the row a flattened predicate re-delivers. `worse` is +// a higher-numbered priority and is OLDER than nothing in particular; it must +// come last however its timestamp compares, which is what fails when only the +// created half of the position survives. +// +// WHAT IT DEPENDS ON FROM OUTSIDE ITSELF: the id set it scopes on, and nothing +// about the workspace. Every timestamp is a whole second, deliberately: the +// column has no fractional part, so a fixture written in milliseconds would +// have the engine, not the case, decide which rows tie. +func RunReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t *testing.T, ctx context.Context, fixture ReaderFixture) { + t.Helper() + id := func(name string) string { return readerID(fixture, "pkswalk", name) } + better, sooner, worse := id("better"), id("sooner"), id("worse") + run := []string{id("a1"), id("a2"), id("a3"), id("a4"), id("a5")} + + runSecond := time.Now().UTC().Truncate(time.Second).Add(-1 * time.Hour) + seed := func(memberID string, priority int, at time.Time) { + issue := readerIssue(memberID, types.TypeTask, "") + issue.Priority = priority + issue.CreatedAt = at + issue.UpdatedAt = at + seedReaderIssue(t, ctx, fixture, issue) + } + // A lower-numbered priority sorts FIRST, whatever its instant: this row is + // the OLDEST in the fixture and must still lead the answer. + seed(better, 1, runSecond.Add(-time.Hour)) + // Same priority as the run, one second newer — the row a priority-equal arm + // that forgot its created_at bound would hand out twice. + seed(sooner, 2, runSecond.Add(time.Second)) + for _, member := range run { + seed(member, 2, runSecond) + } + // A higher-numbered priority sorts LAST, and it is the NEWEST row in the + // fixture so that a body carrying only the created half of the position + // cannot reach it at all. + seed(worse, 3, runSecond.Add(time.Hour)) + + want := append([]string{better, sooner}, run...) + want = append(want, worse) + idScope := readerIDFilter(want...) + + // THE ONE-SHOT ORDER IS READ FIRST, so a backend that agrees with itself + // but orders differently from the reference fails on the sequence rather + // than on the walk — two different defects that would otherwise report the + // same way. + oneShot, err := fixture.Reader.List(ctx, publicops.ListRequest{IDFilter: idScope, SortBy: "priority"}) + if err != nil { + t.Fatalf("List unpaged: %v", err) + } + assertReaderPageIDs(t, "List unpaged", oneShot, want) + + const pageSize = 2 + var walked []string + seen := make(map[string]bool, len(want)) + var afterPriority *int + var afterCreatedAt *time.Time + afterID := "" + for page := 0; page <= len(want); page++ { + got, pageErr := fixture.Reader.List(ctx, publicops.ListRequest{ + IDFilter: idScope, SortBy: "priority", Limit: readerLimit(pageSize), + AfterPriority: afterPriority, AfterCreatedAt: afterCreatedAt, AfterID: afterID, + }) + if pageErr != nil { + t.Fatalf("List page %d: %v", page, pageErr) + } + if len(got.Items) == 0 { + if got.HasMore { + t.Errorf("List page %d came back empty with HasMore set", page) + } + break + } + if len(got.Items) > pageSize { + t.Fatalf("List page %d answered %d rows over a Limit of %d", page, len(got.Items), pageSize) + } + for _, item := range got.Items { + if item == nil || item.Issue == nil { + t.Fatalf("List page %d returned a nil row", page) + } + if seen[item.ID] { + t.Fatalf("List page %d repeated %s: the equal-key run is larger than the page, and the position re-delivered a row it had already handed out", + page, item.ID) + } + seen[item.ID] = true + walked = append(walked, item.ID) + } + last := got.Items[len(got.Items)-1] + at := last.CreatedAt.UTC() + priority := last.Priority + afterPriority, afterCreatedAt, afterID = &priority, &at, last.ID + } + if !slices.Equal(walked, want) { + t.Errorf("the priority keyset walk delivered %v, want the one-shot sequence %v with nothing dropped and nothing repeated", walked, want) + } + + // The position is a CONJUNCT here too: resuming from the run's first member + // keeps the rest of the run and the higher-numbered priority behind it, and + // drops the two rows that sort ahead of it — including `sooner`, which + // shares the cursor's priority and is only excluded by the created_at leg. + cursorAt := runSecond + cursorPriority := 2 + resumed, err := fixture.Reader.List(ctx, publicops.ListRequest{ + IDFilter: idScope, SortBy: "priority", + AfterPriority: &cursorPriority, AfterCreatedAt: &cursorAt, AfterID: run[0], + }) + if err != nil { + t.Fatalf("List resumed from inside the run: %v", err) + } + assertReaderPageIDs(t, "List resumed from inside the run", resumed, append(slices.Clone(run[1:]), worse)) +} + // RunReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates pins the // keyset position as a CONJUNCT. It narrows what the rest of the request // matched; it does not become the request. diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index dac3febbd..fa457a011 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -533,6 +533,7 @@ var roleContractCases = []roleContract{ RunReaderReadyParentScopesToItsTransitiveDescendants, RunReaderListParentReachesEveryDescendantAndOnlyItsOwn, RunReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing, + RunReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing, RunReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates, RunReaderListIncludeEphemeralMergesThePlanesIntoOneOrder, RunReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt, diff --git a/cmd/bd/serve_list_sort_proxied_integration_test.go b/cmd/bd/serve_list_sort_proxied_integration_test.go new file mode 100644 index 000000000..fa29ca03c --- /dev/null +++ b/cmd/bd/serve_list_sort_proxied_integration_test.go @@ -0,0 +1,317 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "reflect" + "sort" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/types" +) + +// walkPageSize is the page an HTTP client walking this operation actually uses. +// It is the value the enterprise httpstore client pages with, and it is the +// denominator in the cost this test exists to measure: a listing of N rows in +// `bd list`'s default order used to cost ceil(N/walkPageSize) requests because +// the order could not be asked for. +const walkPageSize = 200 + +// listSortFixtureRows is large enough that the old cost is unambiguously +// multi-page — seven walk requests at walkPageSize, measured — and small enough +// that a single `bd import` seeds it in one subprocess call. +const listSortFixtureRows = 1400 + +// listPage is one decoded page of GET /v0/beads/issues. +type listPage struct { + Items []map[string]any `json:"items"` + HasMore bool `json:"has_more"` + NextCursor *string `json:"next_cursor"` +} + +// countingPage fetches one page and reports it, so a caller can count the +// requests a strategy costs rather than assert on a number it computed itself. +func (sp *serveProcess) countingPage(t *testing.T, path string, requests *int) listPage { + t.Helper() + *requests++ + resp, err := sp.client.Get(sp.url(path)) + if err != nil { + t.Fatalf("GET %s: %v\nstderr:\n%s", path, err, sp.stderr.String()) + } + defer func() { _ = resp.Body.Close() }() + raw, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + if resp.StatusCode != http.StatusOK { + t.Fatalf("GET %s: status = %d, body = %s\nstderr:\n%s", path, resp.StatusCode, raw, sp.stderr.String()) + } + var page listPage + if err := json.Unmarshal(raw, &page); err != nil { + t.Fatalf("decode %s: %v\nraw: %s", path, err, raw) + } + return page +} + +// itemPriorities projects the priority column, for asserting an order rather +// than a membership. +func itemPriorities(items []map[string]any) []int { + out := make([]int, 0, len(items)) + for _, item := range items { + p, _ := item["priority"].(float64) + out = append(out, int(p)) + } + return out +} + +// TestProxiedServerListSortRetiresTheWalk is the acceptance measurement for +// serving `bd list`'s default order, taken against real Dolt through a real +// `bd serve` subprocess and a real `bd list`. +// +// THE COST IT MEASURES. Before `sort`, an HTTP client that wanted `bd list`'s +// ordering had exactly one option: page the whole result set in the served +// created order and re-sort it locally, because the cursor was a keyset +// position in that order and nothing else could be asked for. That is +// ceil(N/walkPageSize) requests plus a client-side comparator, and it is the +// only cost on this surface that gets WORSE as a project grows. This test runs +// both strategies over the same 1400-row fixture and compares the request +// counts and the delivered sequences: seven requests against one. +// +// WHY THE TWO SEQUENCES MUST BE IDENTICAL, and why that is the load-bearing +// claim rather than the request count: `bd list --sort priority` is a pure +// priority comparator applied as a STABLE sort over the walked created order, +// which yields priority ASC, then created DESC, then id ASC within ties — the +// same sequence as `bd list` with no flags, and the same sequence the served +// `sort=priority` renders in SQL. If those three ever diverge, the parameter is +// not retiring the walk, it is answering a different question faster. So the +// oracle here is `bd list --json` itself: the CLI is the reference, and both +// HTTP strategies are checked against it. +// +// THE FIXTURE IS BUILT TO BREAK A NAIVE IMPLEMENTATION. Priorities repeat +// across the whole range, and timestamps repeat WITHIN a priority in runs of +// four, so equal (priority, created_at) keys straddle every page boundary a +// walk can land on. A two-part position, or a predicate whose priority-equal +// arm lost its created_at bound, drops or repeats rows here; both look like a +// successful 200 and neither changes the row COUNT enough to notice by eye, +// which is why the assertion is the full sequence. +func TestProxiedServerListSortRetiresTheWalk(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "lstsort") + sp := startServe(t, bd, p.dir, bdProxiedEnv(p.dir)) + defer sp.shutdown(t) + + base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + issues := make([]*types.Issue, 0, listSortFixtureRows) + for i := range listSortFixtureRows { + // Priority cycles over the whole 0..4 range and the instant advances + // only every fourth row, so each (priority, created_at) key names a run + // of rows that a 200-row page cannot be assumed to contain whole. + at := base.Add(time.Duration(i/4) * time.Second) + issues = append(issues, &types.Issue{ + ID: fmt.Sprintf("%s-%04d", p.prefix, i), + Title: fmt.Sprintf("row %04d", i), + Status: types.StatusOpen, + Priority: i % 5, + IssueType: types.TypeTask, + CreatedAt: at, + UpdatedAt: at, + }) + } + if _, stderr, err := bdProxiedImportWithInput(t, bd, p.dir, importFixtureJSONL(t, issues), "-"); err != nil { + t.Fatalf("seed %d issues: %v\n%s", listSortFixtureRows, err, stderr) + } + + // THE REFERENCE. `bd list` with no sort flag, unlimited: the order this + // whole exercise exists to reproduce over HTTP. + reference := cliItems(t, bd, p.dir, "list", "--json", "--limit", "0") + if len(reference) != listSortFixtureRows { + t.Fatalf("`bd list --limit 0` returned %d rows, want the %d seeded — the fixture did not land, so every comparison below would be vacuous", + len(reference), listSortFixtureRows) + } + // The reference has to actually be priority-first, or it is not the order + // under test. A fixture whose priorities happened to be uniform would make + // every sequence comparison below pass against a created-order server. + if priorities := itemPriorities(reference); !sort.IntsAreSorted(priorities) { + t.Fatalf("`bd list`'s default order is not priority-first in this fixture; the oracle is wrong, not the server") + } else if priorities[0] == priorities[len(priorities)-1] { + t.Fatal("every seeded row has the same priority, so a created-order answer would be indistinguishable from a priority-order one") + } + + // THE OLD STRATEGY: page the created order to exhaustion, then apply the + // client-side comparator. Spelled out here in full rather than described, + // because what is being measured is its cost against the alternative's. + oldRequests := 0 + var walked []map[string]any + path := fmt.Sprintf("/v0/beads/issues?limit=%d", walkPageSize) + for { + page := sp.countingPage(t, path, &oldRequests) + walked = append(walked, page.Items...) + if !page.HasMore { + break + } + if page.NextCursor == nil { + t.Fatal("has_more is true but next_cursor is absent; the walk cannot continue") + } + path = fmt.Sprintf("/v0/beads/issues?limit=%d&cursor=%s", walkPageSize, *page.NextCursor) + if oldRequests > listSortFixtureRows { + t.Fatal("the created-order walk did not terminate") + } + } + if len(walked) != listSortFixtureRows { + t.Fatalf("the created-order walk delivered %d rows, want %d", len(walked), listSortFixtureRows) + } + // STABLE, and by priority ALONE. That is the whole client-side comparator: + // stability is what preserves the walked created order inside a priority + // tie, and it is why one served order answers both `bd list` and + // `bd list --sort priority`. + sort.SliceStable(walked, func(i, j int) bool { + a, _ := walked[i]["priority"].(float64) + b, _ := walked[j]["priority"].(float64) + return a < b + }) + + // THE NEW STRATEGY: ask for the order. + newRequests := 0 + served := sp.countingPage(t, "/v0/beads/issues?sort=priority&limit=0", &newRequests) + if served.HasMore { + t.Error("an unlimited page reported has_more; the walk is not actually retired") + } + + // ceil(N/pageSize), NOT 1+ceil(N/pageSize). The extra trailing request the + // arithmetic invites is not spent: this server over-fetches one probe row + // per page to decide `has_more`, so the seventh full page already reports + // has_more:false and the walk stops there instead of asking for an eighth, + // empty one. The number here is the MEASURED cost, and it was wrong in this + // test before the measurement corrected it — a walk cost asserted from a + // formula rather than from a run is exactly the kind of number that gets + // quoted for years. + wantRequests := (listSortFixtureRows + walkPageSize - 1) / walkPageSize + if oldRequests != wantRequests { + t.Errorf("the created-order walk cost %d requests, want the %d this measurement is calibrated against (ceil(%d/%d))", + oldRequests, wantRequests, listSortFixtureRows, walkPageSize) + } + if newRequests != 1 { + t.Errorf("`sort=priority` cost %d requests, want 1", newRequests) + } + t.Logf("%d rows in `bd list`'s default order: walk-and-sort = %d requests, sort=priority = %d request", + listSortFixtureRows, oldRequests, newRequests) + + // The three sequences, row for row. + refIDs := itemIDs(reference) + if got := itemIDs(served.Items); !reflect.DeepEqual(got, refIDs) { + t.Errorf("`sort=priority` does not reproduce `bd list`'s default order\nfirst divergence at %s", firstDivergence(got, refIDs)) + } + if got := itemIDs(walked); !reflect.DeepEqual(got, refIDs) { + t.Errorf("the walk-and-sort strategy does not reproduce `bd list`'s default order either, so the oracle is not what this test assumes\nfirst divergence at %s", + firstDivergence(got, refIDs)) + } + + // AND THE SERVED ORDER PAGES. One request is the point, but a client with a + // limit still has to be able to walk this order, and a walk is where a + // position that lost its priority half — or a predicate whose + // priority-equal arm lost its created_at bound — stops agreeing with the + // one-shot answer. The page is deliberately not a divisor of the equal-key + // run length, so boundaries land inside runs. + pagedRequests := 0 + var paged []map[string]any + seen := map[string]bool{} + path = "/v0/beads/issues?sort=priority&limit=97" + for { + page := sp.countingPage(t, path, &pagedRequests) + for _, item := range page.Items { + id, _ := item["id"].(string) + if seen[id] { + t.Fatalf("the priority walk repeated %s at request %d; the equal-key run straddles the page boundary and the position re-delivered a row", + id, pagedRequests) + } + seen[id] = true + } + paged = append(paged, page.Items...) + if !page.HasMore { + if page.NextCursor != nil { + t.Error("has_more is false but next_cursor is present; the document makes those a biconditional") + } + break + } + if page.NextCursor == nil { + t.Fatal("has_more is true but next_cursor is absent; the priority walk cannot continue") + } + path = fmt.Sprintf("/v0/beads/issues?sort=priority&limit=97&cursor=%s", *page.NextCursor) + if pagedRequests > listSortFixtureRows { + t.Fatal("the priority walk did not terminate") + } + } + if got := itemIDs(paged); !reflect.DeepEqual(got, refIDs) { + t.Errorf("a cursored walk in priority order did not reproduce the one-shot answer\nfirst divergence at %s", firstDivergence(got, refIDs)) + } + + t.Run("a cursor minted in one order is refused in the other", func(t *testing.T) { + // Against the real server, over the real wire: the token is legible + // base64 and the two orders' positions are the same shape, so this is + // the refusal that stands between a client and a silently + // skipped-and-duplicated page. + first := sp.countingPage(t, "/v0/beads/issues?sort=priority&limit=10", new(int)) + if first.NextCursor == nil { + t.Fatal("no cursor to replay") + } + status, body := sp.object(t, "/v0/beads/issues?limit=10&cursor="+*first.NextCursor) + if status != http.StatusBadRequest { + t.Fatalf("a priority cursor under the default order: status = %d, want 400: %v", status, body) + } + if body["code"] != "invalid_cursor" { + t.Errorf("code = %v, want invalid_cursor", body["code"]) + } + }) +} + +// firstDivergence names where two id sequences part company, because a diff of +// two 1400-element slices is unreadable and the position of the first +// disagreement is what identifies the defect. +func firstDivergence(got, want []string) string { + for i := range min(len(got), len(want)) { + if got[i] != want[i] { + return fmt.Sprintf("index %d: got %s, want %s (got %d rows, want %d)", i, got[i], want[i], len(got), len(want)) + } + } + if len(got) == len(want) { + return "nowhere: the sequences are equal" + } + return fmt.Sprintf("index %d: one sequence ended (got %d rows, want %d); missing/extra: %s", + min(len(got), len(want)), len(got), len(want), strings.Join(symmetricDifference(got, want), ",")) +} + +// symmetricDifference names the ids one sequence has and the other does not, +// bounded so a wholesale mismatch does not print 1400 ids. +func symmetricDifference(a, b []string) []string { + inB := make(map[string]bool, len(b)) + for _, id := range b { + inB[id] = true + } + inA := make(map[string]bool, len(a)) + for _, id := range a { + inA[id] = true + } + var out []string + for _, id := range a { + if !inB[id] { + out = append(out, "+"+id) + } + } + for _, id := range b { + if !inA[id] { + out = append(out, "-"+id) + } + } + if len(out) > 10 { + out = append(out[:10], "…") + } + return out +} diff --git a/cmd/bd/serve_reads_parity_integration_test.go b/cmd/bd/serve_reads_parity_integration_test.go index 980161d57..a45495665 100644 --- a/cmd/bd/serve_reads_parity_integration_test.go +++ b/cmd/bd/serve_reads_parity_integration_test.go @@ -49,9 +49,14 @@ import ( // // - `bd show --json` emits an array of one; GET /v0/beads/issues/{id} emits // the object. The CLI envelope is byte-pinned by the protocol corpus. -// - `bd list`'s default order is priority-first; the list endpoint's is -// (created_at DESC, id ASC), because the cursor is a keyset position in -// the created order. The ITEM SET and every item's JSON still match. +// - `bd list`'s default order is priority-first; the list endpoint's DEFAULT +// is (created_at DESC, id ASC), because that is what the operation served +// before it had a `sort` parameter and the default is its compatibility +// contract. It is no longer a divergence a client has to live with: +// `sort=priority` serves `bd list`'s own order, row for row, and +// TestProxiedServerListSortRetiresTheWalk measures that against this same +// CLI over 1400 rows. The ITEM SET and every item's JSON match under +// either order. // - THE DEFAULT LIMIT. The endpoint always defaults to // workapi.DefaultListLimit; `bd list` resolves a five-way CLI policy // first (an explicit --limit, --all, a configured list.limit, piped @@ -295,13 +300,13 @@ func TestProxiedServerServeReadParity(t *testing.T) { // read epilogue (fetch, sort, trim, has_more) can differ without // changing the unlimited answer. // - // `--sort created` is what makes the two surfaces comparable under a - // limit: the endpoint's order is welded to the cursor contract and is - // always (created_at DESC, id ASC), so asking the CLI for the same - // order is the only way a truncated page can be compared row for row - // rather than by count. + // Naming the order on BOTH sides is what makes a truncated page + // comparable row for row rather than by count: the two surfaces have + // different defaults (the endpoint keeps `created` for compatibility, + // the CLI is priority-first), so a truncation comparison that named + // neither would be comparing two different top-2s. cli := cliItems(t, bd, p.dir, "list", "--json", "--limit", "2", "--sort", "created") - got := sp.items(t, "/v0/beads/issues?limit=2") + got := sp.items(t, "/v0/beads/issues?limit=2&sort=created") if len(cli) != 2 { t.Fatalf("`bd list --limit 2` returned %d items, want 2", len(cli)) } @@ -309,6 +314,16 @@ func TestProxiedServerServeReadParity(t *testing.T) { t.Fatalf("GET issues?limit=2 returned %d items, want 2", len(got)) } assertItemsMatch(t, "list --limit 2", cli, got) + + // The other order, under the same truncation. This is where the two + // defaults used to force a client to choose between the CLI's ordering + // and a bounded page; now the same two rows come back on both sides. + cliByPriority := cliItems(t, bd, p.dir, "list", "--json", "--limit", "2", "--sort", "priority") + gotByPriority := sp.items(t, "/v0/beads/issues?limit=2&sort=priority") + assertItemsMatch(t, "list --sort priority --limit 2", cliByPriority, gotByPriority) + if a, b := itemIDs(cliByPriority), itemIDs(gotByPriority); !reflect.DeepEqual(a, b) { + t.Errorf("a truncated priority page differs between the surfaces\n cli: %v\nhttp: %v", a, b) + } }) t.Run("a limit the database cannot push down still truncates", func(t *testing.T) { @@ -319,14 +334,26 @@ func TestProxiedServerServeReadParity(t *testing.T) { // this command answered with every row while the direct route answered // with two. // - // There is no HTTP half to compare against: the list endpoint takes no - // `sort` parameter. What is being pinned is that the CLI's two routes - // agree, which is the same property by a different pair. + // There is no HTTP half to compare against, and that is deliberate + // rather than an oversight: `id` is precisely the order the list + // endpoint's `sort` refuses, because a sort SQL cannot express has no + // keyset predicate and so cannot be paged with a cursor. What is being + // pinned is that the CLI's two routes agree, which is the same property + // by a different pair. cli := cliItems(t, bd, p.dir, "list", "--json", "--limit", "2", "--sort", "id") if len(cli) != 2 { t.Errorf("`bd list --sort id --limit 2` returned %d items (%v), want 2 — a sort SQL cannot express still has to respect the limit", len(cli), itemIDs(cli)) } + // The endpoint turns it down by name, so a client that tried to move + // this command onto HTTP learns why rather than receiving a page in an + // order it did not ask for. + status, body := sp.object(t, "/v0/beads/issues?limit=2&sort=id") + if status != http.StatusBadRequest { + t.Errorf("GET issues?sort=id: status = %d, want 400: %v", status, body) + } else if body["param"] != "sort" || body["reason"] != "invalid_value" { + t.Errorf("GET issues?sort=id: body = %v, want param=sort reason=invalid_value", body) + } }) t.Run("show", func(t *testing.T) { diff --git a/engdocs/design/bd-serve-v0.md b/engdocs/design/bd-serve-v0.md index 2543ca3f4..7e5d050da 100644 --- a/engdocs/design/bd-serve-v0.md +++ b/engdocs/design/bd-serve-v0.md @@ -160,27 +160,64 @@ which is legible enough that someone will read it — so the contract is enforce by the version prefix rather than by obscurity. A client that mints its own token gets `invalid_cursor` the moment the encoding moves. -**No lifetime.** The token carries a position and a private encoding version, -and nothing else. The server keeps no state for it, so it does not expire, does -not become invalid across a restart, and is not tied to the connection that -issued it. The only thing that invalidates one is an encoding change. +**No lifetime.** The token carries a position, the ORDER that position is in, +and a private encoding version — and nothing else. The server keeps no state +for it, so it does not expire, does not become invalid across a restart, and is +not tied to the connection that issued it. The only thing that invalidates one +is an encoding change, and the one encoding change made so far (`v1` → `v2`, +adding the order tag) kept every outstanding `v1` token readable as the +`created`-order position it is, so no traversal in flight had to restart. **One recovery.** Every failure mode — wrong version, undecodable base64, -malformed JSON, an empty position — is the same answer, because it is the same -client situation: restart paging with no `cursor`. Re-sending the value cannot -succeed. +malformed JSON, an empty position, a position in another order — is the same +answer, because it is the same client situation: restart paging with no +`cursor`. Re-sending the value cannot succeed. -**Misuse is not detectable.** Because the token carries no filters, a page -fetched with a cursor minted under different filters is *not* refused. The -server applies the current request's filters from the old request's position, -silently skipping every row the new filter set would have placed before it. -Repeat every filter verbatim for the whole traversal, and start a new traversal -when they change. +**Filter misuse is not detectable; ORDER misuse is.** Those are two different +properties and the difference is the whole of the design here. -That last property is a deliberate trade. Embedding the filters would make the -token a second, opaque copy of the request that can disagree with the request -itself; keeping it a bare position makes the failure mode a documented client -obligation instead of a hidden server-side reconciliation. +Because the token carries no filters, a page fetched with a cursor minted under +different filters is *not* refused. The server applies the current request's +filters from the old request's position, silently skipping every row the new +filter set would have placed before it. Repeat every filter verbatim for the +whole traversal, and start a new traversal when they change. That is a +deliberate trade: embedding the filters would make the token a second, opaque +copy of the request that can disagree with the request itself, and keeping it a +bare position makes the failure mode a documented client obligation instead of +a hidden server-side reconciliation. + +The ORDER is not a filter, and it does not get the same treatment. A filter +selects the SET; the order decides what the position MEANS. The same instant +and id name one row's place under `sort=created` and a different row's place +under `sort=priority`, and there is nothing in the bytes to say which was +intended — so a token replayed under a different `sort` is refused with +`invalid_cursor` rather than reinterpreted. This is not a second copy of the +request; it is the position's own type tag, and without it adding `sort` at all +would mean serving skipped-and-duplicated pages with a 200 and no way for a +client to notice. + +**Only orders with a proven total key are served.** `sort` takes `created` and +`priority` and nothing else, because each value is a keyset contract — a +position shape, a strictly-after predicate, an index — rather than a display +preference. Both of these keys are total (`priority` and `created_at` are +non-null, `id` is unique), so a page boundary inside a run of equal keys +resolves on `id` with no dropped and no repeated row. The seven other orders +`bd list --sort` accepts have no such key: `id` is a natural-numeric order no +database expresses, `updated` moves on every write, `closed` is nullable, and +`status`/`title`/`type`/`assignee` are mutable and unindexed. A client that +wants one of those still pages in `created` order and sorts what it received. + +`priority` is itself mutable, which `created_at` is not, and the consequence is +stated rather than hidden: under `created` only new rows move relative to a +walk, while under `priority` a priority update moves an existing row too, so it +can be seen twice or missed. That is the same "a cursor pins a position, not a +snapshot" caveat reached by a second route, not a new class of error — +unchanged data never skips or repeats under either order. + +`GET /v0/beads/ready` still publishes a `sort` with a wider vocabulary and no +cursor at all, which is consistent rather than contradictory: without a cursor +an order is only a display decision, and nothing has to be able to resume into +the middle of it. ## The loopback posture diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index 171607a4d..75b256751 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -136,6 +136,24 @@ func (e CountDependencyEdgesParamsDirection) Valid() bool { } } +// Defines values for ListIssuesParamsSort. +const ( + ListIssuesParamsSortCreated ListIssuesParamsSort = "created" + ListIssuesParamsSortPriority ListIssuesParamsSort = "priority" +) + +// Valid indicates whether the value is a known member of the ListIssuesParamsSort enum. +func (e ListIssuesParamsSort) Valid() bool { + switch e { + case ListIssuesParamsSortCreated: + return true + case ListIssuesParamsSortPriority: + return true + default: + return false + } +} + // Defines values for ListRelatedIssuesParamsDirection. const ( ListRelatedIssuesParamsDirectionIn ListRelatedIssuesParamsDirection = "in" @@ -2112,7 +2130,7 @@ type ListIssuesParams struct { // IncludeInfra Include the workspace's configured infrastructure issue types. This also admits the ephemeral plane those types live in, so it is strictly wider than `include_ephemeral`. IncludeInfra *bool `form:"include_infra,omitempty" json:"include_infra,omitempty"` - // IncludeEphemeral Include the ephemeral tier — ephemeral rows and the non-synced rows stored beside them — merged into the same `(created_at DESC, id ASC)` order as the durable ones. + // IncludeEphemeral Include the ephemeral tier — ephemeral rows and the non-synced rows stored beside them — merged into the same order as the durable ones, whichever order `sort` named. // // It admits a TIER and takes no TYPE exclusion off, so a row whose type this operation already hides stays hidden. That includes the configured infrastructure types: ephemeral `agent`, `role` and `message` rows need `include_infra` as well as, or instead of, this one. What `include_ephemeral` alone reaches is the ephemeral rows of the types a listing already shows. IncludeEphemeral *bool `form:"include_ephemeral,omitempty" json:"include_ephemeral,omitempty"` @@ -2129,11 +2147,22 @@ type ListIssuesParams struct { // HasMetadataKey Only issues carrying this top-level metadata key. HasMetadataKey *string `form:"has_metadata_key,omitempty" json:"has_metadata_key,omitempty"` + // Sort Which of the two served total orders this page is in. `created` is `(created_at DESC, id ASC)`; `priority` is `(priority ASC, created_at DESC, id ASC)` — `bd list`'s flagless ordering, and the order `bd list --sort priority` produces. + // + // THE DEFAULT IS `created` AND WILL NOT CHANGE. It is what this operation served before this parameter existed, so moving it would alter which rows a truncated page contains for every client written against that behavior, with no error to notice it by. + // + // THE VOCABULARY IS CLOSED, and deliberately smaller than the nine values `bd list --sort` and `GET /v0/beads/issues:query` take. Each value here is a cursor contract, not a display preference: it needs a keyset predicate and a key proven total. See the operation description for why the other seven have neither. + // + // A value outside the enum is a 400 `invalid_argument` with `param: "sort"` and `reason: "invalid_value"`. A server that predates this parameter answers `param: "sort"` with `reason: "unknown_parameter"` instead, which is the per-parameter capability probe a client can dispatch on to fall back to paging in `created` order and sorting client-side. + Sort *ListIssuesParamsSort `form:"sort,omitempty" json:"sort,omitempty"` + // Cursor Opaque keyset position, taken verbatim from a previous response's `next_cursor`. Clients MUST NOT construct, parse or mutate it: its encoding is server-private and versioned, and an undecodable or unknown-version value is refused with 400 `invalid_cursor`. The recovery for that refusal is normative: restart paging with no `cursor` at all — the position cannot be salvaged, and re-sending the same value cannot succeed. // - // LIFETIME: a cursor holds a position and a private encoding version, and nothing else. The server keeps no state for it, so it does not expire, does not become invalid when the server restarts, and is not tied to the connection that issued it; the only thing that invalidates one is a change to the encoding, which surfaces as `invalid_cursor`. + // LIFETIME: a cursor holds a position, the ORDER that position is in, and a private encoding version — and nothing else. The server keeps no state for it, so it does not expire, does not become invalid when the server restarts, and is not tied to the connection that issued it; the only thing that invalidates one is a change to the encoding, which surfaces as `invalid_cursor`. A token minted before `sort` existed is still accepted, as the `created`-order position it is, so no traversal in flight across a server upgrade has to be restarted. // - // MISUSE IS NOT DETECTABLE, which is why repeating the filters matters. Because the token carries no filters, a page fetched with a cursor minted under DIFFERENT filters is not refused: the server applies the filters of the current request from the position of the old one, silently skipping every row the new filter set would have placed before that position. Repeat every filter verbatim for the whole traversal, and start a new traversal when they change. + // THE ORDER MUST MATCH, AND THAT MISMATCH IS DETECTED. A cursor minted under one `sort` and re-sent under another is refused with 400 `invalid_cursor`; the same instant and id name a different row in a different order, so resuming from it would skip and duplicate rows silently. Send the same `sort` for the whole traversal. + // + // FILTER MISUSE IS *NOT* DETECTABLE, which is the difference, and why repeating the filters matters. Because the token carries no filters, a page fetched with a cursor minted under DIFFERENT filters is not refused: the server applies the filters of the current request from the position of the old one, silently skipping every row the new filter set would have placed before that position. Repeat every filter verbatim for the whole traversal, and start a new traversal when they change. Cursor *string `form:"cursor,omitempty" json:"cursor,omitempty"` // Limit Maximum number of items to return. `0` means unlimited, exactly as `bd list --limit 0` does — the two surfaces read the same shared default and the same zero semantics, so they cannot diverge. An unlimited page reports `has_more: false` and carries no `next_cursor`. A negative value is a 400. @@ -2149,6 +2178,9 @@ type ListIssuesParams struct { Brief *bool `form:"brief,omitempty" json:"brief,omitempty"` } +// ListIssuesParamsSort defines parameters for ListIssues. +type ListIssuesParamsSort string + // GetIssueParams defines parameters for GetIssue. type GetIssueParams struct { // IncludeComments Populate `comments` with the issue's full comment bodies. When it is honored `comments_omitted` is false, so a client is never left to guess whether an absent list means "no comments" or "not asked for". diff --git a/internal/httpapi/cursor.go b/internal/httpapi/cursor.go index 5e83aa891..bc64c969b 100644 --- a/internal/httpapi/cursor.go +++ b/internal/httpapi/cursor.go @@ -9,25 +9,86 @@ import ( "github.com/steveyegge/beads/internal/types" ) +// listOrder names one of the total orders the issue listing serves. It is the +// `sort` vocabulary and the cursor's order tag in one type, so the value a +// request asks for and the value a position was minted in are the same kind of +// thing and can be compared directly. +// +// THE SET IS CLOSED, and every member of it costs a position shape, a +// strictly-after predicate, an index and a conformance case. An order with no +// proven total key cannot be served with a cursor at all — it can only be +// walked to exhaustion and sorted client-side, which is what this type exists +// to stop doing for the two orders below. +type listOrder string + +const ( + // orderCreated is (created_at DESC, id ASC): the order this operation has + // always served, and what an absent `sort` means. Permanently. + orderCreated listOrder = "created" + // orderPriority is (priority ASC, created_at DESC, id ASC): the order + // `bd list` renders with no flags, which is also what `bd list --sort + // priority` produces — a pure priority comparator applied STABLY over the + // created order is the same sequence. One served order therefore answers + // both, in one request instead of a walk. + orderPriority listOrder = "priority" +) + +// listOrders is the documented vocabulary, in the document's order. +var listOrders = []string{string(orderCreated), string(orderPriority)} + +// listOrderDefault is the order an absent `sort` selects. +// +// IT CANNOT CHANGE. Every v0 client — including the walking clients this +// parameter exists to retire, which resume with their own created-ordered +// bookkeeping — was written when this was the only order there was. Moving it +// would alter which rows a truncated page contains for all of them, with no +// error and no version to notice it by. +const listOrderDefault = orderCreated + // cursorVersion prefixes every token this server mints. It is the ONLY thing // that invalidates a cursor: the token carries a position and nothing else, so // it does not expire, does not die with a restart, and is not tied to the // connection that issued it. Bumping this is how an encoding change is made // safely — every older token then decodes to invalid_cursor, whose documented // recovery is to restart paging. -const cursorVersion = "v1" +const cursorVersion = "v2" -// cursorPosition is the keyset position a token carries: one row's place in -// the (created_at DESC, id ASC) total order. +// cursorVersionV1 is the retired encoding: base64 of `{t,i}`, minted when +// (created_at DESC, id ASC) was the only order this operation served. +// +// It stays DECODABLE, as exactly that order, so bumping to v2 costs no client a +// restarted traversal — a client mid-walk when a server upgrades is holding one +// of these. It is never minted again, and it is refused under `sort=priority` +// for the same reason a v2 created-order token is: it is a position in the +// created order and means nothing else. +const cursorVersionV1 = "v1" + +// cursorPosition is the keyset position a token carries: one row's place in a +// named total order. // // The token deliberately does NOT carry the filters it was minted under. That // is a documented property with a documented consequence — a cursor reused // under different filters is not refused, it just resumes from the old // position — and it is what keeps the token from becoming a second, opaque // copy of the request that can disagree with the request itself. +// +// THE ORDER IS NOT A FILTER, which is why it is here and they are not. A +// filter selects the SET; the order decides what the position MEANS. The bytes +// of `{"t":…,"i":…}` are a position in the created order, a position in the +// priority order, or nothing at all, and there is no way to tell which from the +// bytes. Carrying the order is therefore not a copy of the request — it is the +// position's own type tag, and without it a created-order token replayed under +// `sort=priority` decodes cleanly and is silently misread as a priority +// position, answering 200 with a page that both skips and duplicates rows. type cursorPosition struct { + Order listOrder `json:"o"` CreatedAt time.Time `json:"t"` ID string `json:"i"` + // Priority is present if and only if Order is orderPriority, because that + // order's position is the triple (priority, created_at, id). A pointer so + // "the token carried none" is distinguishable from priority 0, which is a + // real and reachable value. + Priority *int `json:"p,omitempty"` } // encodeCursor mints the opaque token for a page's last row. @@ -40,27 +101,70 @@ type cursorPosition struct { func encodeCursor(pos cursorPosition) string { blob, err := json.Marshal(pos) if err != nil { - // Unreachable: the struct is a time and a string. + // Unreachable: the struct is a string, a time, a string and an int. return "" } return cursorVersion + "." + base64.RawURLEncoding.EncodeToString(blob) } -// decodeCursor reads a token this server minted. Every failure mode — -// wrong version, undecodable base64, malformed JSON, an empty position — is -// the same answer, because they are the same client situation: the position -// cannot be salvaged and re-sending the value cannot succeed. -func decodeCursor(token string) (cursorPosition, bool) { +// decodeCursor reads a token this server minted, IN THE ORDER THE REQUEST IS +// ASKING FOR. Every failure mode — wrong version, undecodable base64, malformed +// JSON, an empty position, a position in another order, a priority order +// missing its priority — is the same answer, because they are the same client +// situation: the position cannot be salvaged and re-sending the value cannot +// succeed. +// +// THE ORDER IS A PARAMETER RATHER THAN A FIELD THE CALLER CHECKS AFTERWARDS. +// A decoder that returned the position and left the comparison to its caller +// would be correct at every call site that remembered, and the one that forgot +// would not fail loudly — it would serve a wrong page with a 200. Passing +// `want` in makes the mismatch unrepresentable at the seam instead of merely +// detectable somewhere past it. +func decodeCursor(token string, want listOrder) (cursorPosition, bool) { + if rest, ok := strings.CutPrefix(token, cursorVersionV1+"."); ok { + // The retired encoding named no order because it had none to name. + // Every v1 token in existence is a created-order position. + if want != orderCreated { + return cursorPosition{}, false + } + pos, ok := decodeCursorBlob(rest) + if !ok { + return cursorPosition{}, false + } + pos.Order = orderCreated + pos.Priority = nil + return pos, true + } rest, ok := strings.CutPrefix(token, cursorVersion+".") if !ok { return cursorPosition{}, false } - blob, err := base64.RawURLEncoding.DecodeString(rest) + pos, ok := decodeCursorBlob(rest) + if !ok { + return cursorPosition{}, false + } + if pos.Order != want { + return cursorPosition{}, false + } + // A priority-order token with no priority is two thirds of a position, and + // a created-order one with a priority is a position in an order it does not + // name. Neither is mintable here, which is precisely why both are refused: + // the token is base64 of legible JSON, so the shapes a client hand-rolls + // are the shapes that have to be turned down. + if (pos.Order == orderPriority) != (pos.Priority != nil) { + return cursorPosition{}, false + } + return pos, true +} + +// decodeCursorBlob reads the base64 JSON body shared by both encodings. +func decodeCursorBlob(blob string) (cursorPosition, bool) { + raw, err := base64.RawURLEncoding.DecodeString(blob) if err != nil { return cursorPosition{}, false } var pos cursorPosition - if err := json.Unmarshal(blob, &pos); err != nil { + if err := json.Unmarshal(raw, &pos); err != nil { return cursorPosition{}, false } if pos.CreatedAt.IsZero() { @@ -69,9 +173,9 @@ func decodeCursor(token string) (cursorPosition, bool) { return pos, true } -// cursorFor returns the token that resumes after the last row of a page, or "" -// when there is nothing to resume from. -func cursorFor(items []*types.IssueWithCounts) string { +// cursorFor returns the token that resumes after the last row of a page in the +// order that page was served in, or "" when there is nothing to resume from. +func cursorFor(items []*types.IssueWithCounts, order listOrder) string { if len(items) == 0 { return "" } @@ -79,5 +183,10 @@ func cursorFor(items []*types.IssueWithCounts) string { if last == nil || last.Issue == nil { return "" } - return encodeCursor(cursorPosition{CreatedAt: last.CreatedAt, ID: last.ID}) + pos := cursorPosition{Order: order, CreatedAt: last.CreatedAt, ID: last.ID} + if order == orderPriority { + priority := last.Priority + pos.Priority = &priority + } + return encodeCursor(pos) } diff --git a/internal/httpapi/list_sort_test.go b/internal/httpapi/list_sort_test.go new file mode 100644 index 000000000..7facb0a0e --- /dev/null +++ b/internal/httpapi/list_sort_test.go @@ -0,0 +1,283 @@ +package httpapi + +import ( + "encoding/base64" + "encoding/json" + "net/http" + "testing" + "time" + + "github.com/steveyegge/beads/internal/types" +) + +// listSortOf drives GET /v0/beads/issues once and returns the filter the +// handler built, which is the only place the order and the position are +// observable — the response body shows rows, not what was asked for. +func listSortOf(t *testing.T, query string) types.IssueFilter { + t.Helper() + ts, rec := newReadServer(t, Config{}) + resp := ts.get(t, "/v0/beads/issues"+query) + if resp.StatusCode != http.StatusOK { + t.Fatalf("GET /v0/beads/issues%s: status = %d, want 200; body=%v", query, resp.StatusCode, decodeBody(t, resp)) + } + filters := rec.searchFilters() + if len(filters) != 1 { + t.Fatalf("GET /v0/beads/issues%s built %d filters, want 1", query, len(filters)) + } + return filters[0] +} + +// TestListSortDefaultsToCreatedOrderForever is the compatibility guard on the +// one decision this parameter must NOT quietly make. +// +// Absent `sort` has always meant `(created_at DESC, id ASC)`, and every client +// written against v0 — including the walk-and-sort clients this parameter +// exists to retire — assumes it. Changing the default would be a silent +// wire-behaviour change for all of them: no error, no version bump, just +// different rows once `limit` truncates. The literal is asserted rather than +// the constant the handler reads, because comparing the handler against its own +// default passes for every value that default could take. +func TestListSortDefaultsToCreatedOrderForever(t *testing.T) { + if got := listSortOf(t, "").SortBy; got != "created" { + t.Errorf("an absent `sort` reached storage as %q, want %q — the default is the compatibility contract for every v0 client", got, "created") + } + if got := listSortOf(t, "?sort=created").SortBy; got != "created" { + t.Errorf("`sort=created` reached storage as %q, want %q", got, "created") + } + // An EMPTY value is the absent spelling, which is this surface's existing + // reading of one — `ready` resolves `?sort=` to its own default the same + // way, through the same decoder. It is pinned here rather than left + // implicit because the alternative readings are both bad: refusing it would + // make `?sort=` a 400 on one operation and a default on its sibling, and + // passing "" through to storage would silently adopt whatever the empty + // sort key maps to there. + if got := listSortOf(t, "?sort=").SortBy; got != "created" { + t.Errorf("`sort=` (empty) reached storage as %q, want %q — an empty value is the absent spelling here, as it is on `ready`", got, "created") + } +} + +// TestListSortPriorityReachesStorageAsThePriorityOrder pins the whole point of +// the parameter: `sort=priority` must reach storage as the order `bd list` +// renders flaglessly (priority ASC, created_at DESC, id ASC), so one request +// answers what a walk-to-exhaustion plus a client-side comparator answered +// before. +// +// The assertion is on the filter and not on the row order because the fixture's +// rows come back in whatever order the fake hands them over. What is at risk is +// the ORDER THE QUERY ASKED FOR; a handler that decoded the parameter and then +// forwarded "created" anyway would serve a correct-looking page in the wrong +// order and no response-shaped assertion would see it. +func TestListSortPriorityReachesStorageAsThePriorityOrder(t *testing.T) { + filter := listSortOf(t, "?sort=priority") + if filter.SortBy != "priority" { + t.Errorf("`sort=priority` reached storage as SortBy=%q, want %q", filter.SortBy, "priority") + } + if filter.SortDesc { + t.Error("`sort=priority` reached storage reversed; this operation publishes no direction knob and each direction is a different keyset") + } +} + +// TestListSortRefusesAnOrderItCannotPage pins the CLOSED enum. Every served +// order is a keyset contract — a position shape, a strictly-after predicate and +// an index — so an order with no proven total key cannot be accepted and then +// paged. The refusal is by name, and it is the same `unknown_parameter`-adjacent +// machinery a client dispatches on. +// +// The values chosen are the ones a caller is most likely to try, because +// `bd list --sort` and the query operation on this same surface both take them: +// `updated` (maximally mutable key), `id` (an order the database cannot +// express) and `status` (unindexed). Accepting any of them would mean either a +// silently unordered page or a server-side walk — the very thing this parameter +// removes. +func TestListSortRefusesAnOrderItCannotPage(t *testing.T) { + for _, bad := range []string{"updated", "id", "status", "title", "closed", "assignee", "type", "PRIORITY", "created,priority", "ready", "hybrid"} { + ts, rec := newReadServer(t, Config{}) + resp := ts.get(t, "/v0/beads/issues?sort="+bad) + if resp.StatusCode != http.StatusBadRequest { + t.Errorf("GET ?sort=%q: status = %d, want 400", bad, resp.StatusCode) + continue + } + body := decodeBody(t, resp) + if body["code"] != string(CodeInvalidArgument) { + t.Errorf("GET ?sort=%q: code = %v, want invalid_argument", bad, body["code"]) + } + if body["param"] != "sort" { + t.Errorf("GET ?sort=%q: param = %v, want \"sort\" — param is what a client dispatches on", bad, body["param"]) + } + if body["reason"] != string(ReasonInvalidValue) { + t.Errorf("GET ?sort=%q: reason = %v, want invalid_value", bad, body["reason"]) + } + if n := len(rec.searchFilters()); n != 0 { + t.Errorf("GET ?sort=%q reached storage", bad) + } + } +} + +// TestListCursorCarriesItsOwnOrder is the test that decides whether this +// parameter is safe to ship, and it is the one the design's own audit says the +// obvious implementation fails. +// +// bd serve's cursor is base64 of `{t,i}` with no seal and no binding. Add +// `sort` without touching it and a token minted under the created order decodes +// perfectly under `sort=priority`: the server reads its instant and its id, +// applies them as a position in a DIFFERENT total order, and answers 200 with a +// page that skips and duplicates rows. Nothing refuses, nothing logs, and the +// client cannot tell. +// +// A position is only interpretable together with the order it was minted in, so +// the token carries that order and decode refuses a mismatch. This does not +// contradict "the token carries no filters": filters select the SET, the order +// defines what the position MEANS. +func TestListCursorCarriesItsOwnOrder(t *testing.T) { + created := time.Now().UTC().Truncate(time.Second) + items := []*types.IssueWithCounts{{Issue: &types.Issue{ID: "bd-7", CreatedAt: created, Priority: 2}}} + + createdToken := cursorFor(items, orderCreated) + priorityToken := cursorFor(items, orderPriority) + if createdToken == "" || priorityToken == "" { + t.Fatal("cursorFor returned no token for a nonempty page") + } + if createdToken == priorityToken { + t.Fatal("the two orders mint the SAME token, so a position cannot say which order it is a position in — this is the skip-and-duplicate hole") + } + + for _, tc := range []struct { + name string + query string + token string + }{ + {"a created-order token replayed under sort=priority", "?sort=priority&cursor=", createdToken}, + {"a priority-order token replayed under the default order", "?cursor=", priorityToken}, + {"a priority-order token replayed under sort=created", "?sort=created&cursor=", priorityToken}, + } { + ts, rec := newReadServer(t, Config{}) + resp := ts.get(t, "/v0/beads/issues"+tc.query+tc.token) + if resp.StatusCode != http.StatusBadRequest { + t.Errorf("%s: status = %d, want 400 — it decoded and was silently reinterpreted as a position in the wrong order", tc.name, resp.StatusCode) + continue + } + if body := decodeBody(t, resp); body["code"] != string(CodeInvalidCursor) { + t.Errorf("%s: code = %v, want invalid_cursor (documented recovery: restart paging)", tc.name, body["code"]) + } + if n := len(rec.searchFilters()); n != 0 { + t.Errorf("%s reached storage", tc.name) + } + } + + // The matching legs, so the refusal above is shown to be about the ORDER + // and not about cursors in general. + for _, tc := range []struct { + name string + query string + }{ + {"a created-order token under the default order", "?cursor=" + createdToken}, + {"a created-order token under sort=created", "?sort=created&cursor=" + createdToken}, + {"a priority-order token under sort=priority", "?sort=priority&cursor=" + priorityToken}, + } { + if filter := listSortOf(t, tc.query); filter.AfterCreatedAt == nil { + t.Errorf("%s: the position did not reach storage", tc.name) + } + } +} + +// TestListPriorityCursorReachesStorageAsAWholePosition pins the plumbing the +// order tag exists to protect. Under `sort=priority` the position is +// (priority, created_at, id); a handler that decoded the priority and forwarded +// only the instant and the id would page a priority-ordered ORDER BY with a +// created-order predicate, dropping every row of a higher-numbered priority +// created after the cursor — silently, with a 200. +func TestListPriorityCursorReachesStorageAsAWholePosition(t *testing.T) { + created := time.Now().UTC().Truncate(time.Second) + items := []*types.IssueWithCounts{{Issue: &types.Issue{ID: "bd-7", CreatedAt: created, Priority: 3}}} + + filter := listSortOf(t, "?sort=priority&cursor="+cursorFor(items, orderPriority)) + if filter.AfterCreatedAt == nil || !filter.AfterCreatedAt.Equal(created) { + t.Fatalf("filter.AfterCreatedAt = %v, want %v", filter.AfterCreatedAt, created) + } + if filter.AfterID != "bd-7" { + t.Errorf("filter.AfterID = %q, want %q", filter.AfterID, "bd-7") + } + if filter.AfterPriority == nil { + t.Fatal("filter.AfterPriority is nil: the priority half of the position was dropped between the cursor and storage") + } + if *filter.AfterPriority != 3 { + t.Errorf("filter.AfterPriority = %d, want 3", *filter.AfterPriority) + } + + // The created order does NOT position by priority: sending one there would + // AND a priority bound onto a created-order walk and drop rows the caller + // asked for. + if p := listSortOf(t, "?cursor="+cursorFor(items, orderCreated)).AfterPriority; p != nil { + t.Errorf("the created order forwarded AfterPriority=%d; that order's position is (created_at, id) and nothing else", *p) + } +} + +// TestOutstandingV1CursorsStillPage is the no-flag-day guard. Bumping the +// encoding is the ONE thing the document says invalidates a cursor, and a +// client mid-traversal when this server upgrades holds a `v1.` token. Those +// tokens were all minted in the created order — it was the only order there +// was — so they stay readable as exactly that, and the upgrade costs no +// restarted traversal. +// +// The same rule closes the door behind them: a `v1.` token is a created-order +// position, so replaying one under `sort=priority` is the same mismatch a `v2.` +// created token is, and is refused the same way. +func TestOutstandingV1CursorsStillPage(t *testing.T) { + created := time.Now().UTC().Truncate(time.Second) + legacy := legacyV1Cursor(created, "bd-7") + + filter := listSortOf(t, "?cursor="+legacy) + if filter.AfterCreatedAt == nil || !filter.AfterCreatedAt.Equal(created) { + t.Fatalf("a v1 token did not resume: AfterCreatedAt = %v, want %v", filter.AfterCreatedAt, created) + } + if filter.AfterID != "bd-7" { + t.Errorf("a v1 token resumed at id %q, want %q", filter.AfterID, "bd-7") + } + if filter.AfterPriority != nil { + t.Errorf("a v1 token positioned by priority (%d); it was minted in the created order and means nothing else", *filter.AfterPriority) + } + + ts, rec := newReadServer(t, Config{}) + resp := ts.get(t, "/v0/beads/issues?sort=priority&cursor="+legacy) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("a v1 token under sort=priority: status = %d, want 400", resp.StatusCode) + } + if body := decodeBody(t, resp); body["code"] != string(CodeInvalidCursor) { + t.Errorf("a v1 token under sort=priority: code = %v, want invalid_cursor", body["code"]) + } + if n := len(rec.searchFilters()); n != 0 { + t.Error("a v1 token under sort=priority reached storage") + } +} + +// TestPriorityCursorNeedsItsPriority pins the decode of a token that names the +// priority order and carries no priority. It is not reachable from this +// server's own minting, which is exactly why it is asserted: the token is +// base64 of legible JSON, the document tells clients never to construct one, +// and a hand-rolled `{"o":"priority","t":…,"i":…}` would otherwise decode into +// a half position and page a priority order with a created-order predicate. +func TestPriorityCursorNeedsItsPriority(t *testing.T) { + created := time.Now().UTC().Truncate(time.Second) + handRolled := encodeCursor(cursorPosition{Order: orderPriority, CreatedAt: created, ID: "bd-7"}) + + if _, ok := decodeCursor(handRolled, orderPriority); ok { + t.Error("a priority-order token with no priority decoded; the position is (priority, created_at, id) and two thirds of one is not a position") + } +} + +// legacyV1Cursor mints a token in the retired v1 encoding — base64 of `{t,i}` +// under a `v1.` prefix — INDEPENDENTLY of production code. Minting it by +// calling the production encoder would make the compatibility assertion +// circular: the encoder no longer produces v1, and a helper that reached for +// whatever it does produce would assert only that a token round-trips through +// itself. This is the literal bytes a client is holding across the upgrade. +func legacyV1Cursor(createdAt time.Time, id string) string { + blob, err := json.Marshal(struct { + CreatedAt time.Time `json:"t"` + ID string `json:"i"` + }{createdAt, id}) + if err != nil { + panic(err) + } + return "v1." + base64.RawURLEncoding.EncodeToString(blob) +} diff --git a/internal/httpapi/reads.go b/internal/httpapi/reads.go index 0f7c617fc..1b48532b9 100644 --- a/internal/httpapi/reads.go +++ b/internal/httpapi/reads.go @@ -356,23 +356,35 @@ func (s *Server) handleListIssues(w http.ResponseWriter, r *http.Request) { MetadataFields: q.metadataFields("metadata_field"), HasMetadataKey: q.str("has_metadata_key"), - // ORDERING IS FIXED AND DIVERGES FROM `bd list` DELIBERATELY, which is - // why there is no `sort` parameter to decode. The cursor is a keyset - // position in the created order, so a first page under `bd list`'s - // priority-first default would make the second page skip and duplicate - // rows. The order is welded to the cursor contract. - SortBy: "created", - Limit: q.limit(), } + // THE ORDER AND THE CURSOR ARE ONE DECISION. Each served order is a keyset + // contract — its own position shape and its own strictly-after predicate — + // so `sort` selects the ORDER BY and, with it, what a position means. The + // vocabulary is closed for that reason and not for tidiness: the seven + // other orders `bd list --sort` takes have no proven total key (mutable, + // nullable, or not expressible in SQL at all), and serving one behind a + // cursor would page a walk that skips and repeats rows. + // + // SortBy takes the wire value verbatim, which is safe only because the two + // vocabularies coincide by construction: sqlbuild.SortDefs already spells + // these orders `created` and `priority`, and `priority` there is exactly + // (priority ASC, created_at DESC, id ASC) — `bd list`'s flagless order. + order := listOrder(q.oneOf("sort", string(listOrderDefault), listOrders...)) + req.SortBy = string(order) + token := q.str("cursor") if !s.acceptQuery(w, r, q) { return } if token != "" { - pos, ok := decodeCursor(token) + // Decoded AGAINST the order this request asked for. A token minted in + // the other order is refused rather than reinterpreted: its instant and + // its id would decode perfectly and mean something else, which is a + // skipped-and-duplicated page served with a 200. + pos, ok := decodeCursor(token, order) if !ok { requestInfo(r.Context()).refuse(token) s.fail(w, r, InvalidCursor()) @@ -380,6 +392,7 @@ func (s *Server) handleListIssues(w http.ResponseWriter, r *http.Request) { } req.AfterCreatedAt = &pos.CreatedAt req.AfterID = pos.ID + req.AfterPriority = pos.Priority } if !s.allowUnlimited(w, r, req.Limit) { return @@ -404,7 +417,10 @@ func (s *Server) handleListIssues(w http.ResponseWriter, r *http.Request) { // Present if and only if has_more, which the document states as a // biconditional: a client that sees one and not the other has no way // to know whether paging is finished. - if next := cursorFor(page.Items); next != "" { + // Minted in the order this page was SERVED in, so the token a client + // hands back is a position the next request can only be read against + // the same way. + if next := cursorFor(page.Items, order); next != "" { body.NextCursor = &next } } diff --git a/internal/httpapi/reads_test.go b/internal/httpapi/reads_test.go index ee09a3c0a..f4d5a6313 100644 --- a/internal/httpapi/reads_test.go +++ b/internal/httpapi/reads_test.go @@ -302,25 +302,38 @@ func TestGetIssueRefusesAnImpossibleIDFromTheEdge(t *testing.T) { // nothing, encodeCursor broken outright included. func TestCursorRoundTrips(t *testing.T) { created := time.Now().UTC().Truncate(time.Second) - items := []*types.IssueWithCounts{{Issue: &types.Issue{ID: "bd-7", CreatedAt: created}}} + items := []*types.IssueWithCounts{{Issue: &types.Issue{ID: "bd-7", CreatedAt: created, Priority: 2}}} - token := cursorFor(items) - if token == "" { - t.Fatal("cursorFor returned no token for a nonempty page") - } - pos, ok := decodeCursor(token) - if !ok { - t.Fatalf("a token this server minted did not decode: %q", token) - } - if pos.ID != "bd-7" { - t.Errorf("decoded id = %q, want bd-7", pos.ID) - } - if !pos.CreatedAt.Equal(created) { - t.Errorf("decoded created_at = %s, want %s — the position is a keyset predicate, so a lossy instant skips or repeats rows", pos.CreatedAt, created) + // Both served orders, because a round trip that held for one and lost a + // member of the other's position would still page — into the wrong rows. + for _, order := range []listOrder{orderCreated, orderPriority} { + token := cursorFor(items, order) + if token == "" { + t.Fatalf("%s: cursorFor returned no token for a nonempty page", order) + } + pos, ok := decodeCursor(token, order) + if !ok { + t.Fatalf("%s: a token this server minted did not decode: %q", order, token) + } + if pos.ID != "bd-7" { + t.Errorf("%s: decoded id = %q, want bd-7", order, pos.ID) + } + if !pos.CreatedAt.Equal(created) { + t.Errorf("%s: decoded created_at = %s, want %s — the position is a keyset predicate, so a lossy instant skips or repeats rows", order, pos.CreatedAt, created) + } + if order == orderPriority { + if pos.Priority == nil { + t.Errorf("%s: decoded no priority; this order's position is (priority, created_at, id)", order) + } else if *pos.Priority != 2 { + t.Errorf("%s: decoded priority = %d, want 2", order, *pos.Priority) + } + } else if pos.Priority != nil { + t.Errorf("%s: decoded a priority (%d) it has no place for", order, *pos.Priority) + } } - for _, bad := range []string{"", "v0.abc", "v1.!!!", "v1.", "not-a-cursor"} { - if _, ok := decodeCursor(bad); ok { + for _, bad := range []string{"", "v0.abc", "v1.!!!", "v1.", "v2.!!!", "v2.", "v3.abc", "not-a-cursor"} { + if _, ok := decodeCursor(bad, orderCreated); ok { t.Errorf("decodeCursor(%q) succeeded; every unreadable token is the same client situation", bad) } } diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index c990419e6..ed508c621 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -1002,25 +1002,59 @@ paths: types), each carrying dependency, dependent and comment counts. - ORDERING IS FIXED AND DIVERGES FROM `bd list` DELIBERATELY. Every page - of this operation — cursored or not — is ordered by - `(created_at DESC, id ASC)`, while `bd list`'s default ordering is - priority-first. The divergence is the price of a correct cursor: the - cursor is a keyset position in the created order, so if the first page - fell back to priority ordering the second page would skip and duplicate - rows. That is also why v0 exposes no `sort` parameter — the sort order - is welded to the cursor contract, and a new order needs new surface. + ORDERING IS NAMED BY `sort`, AND THE VOCABULARY IS CLOSED. Two orders + are served: `created` — `(created_at DESC, id ASC)`, the default and + the only order earlier revisions had — and `priority` — + `(priority ASC, created_at DESC, id ASC)`, which is `bd list`'s + flagless ordering and also what `bd list --sort priority` produces. + Absent `sort` means `created`, permanently: it is the compatibility + contract for every client written before this parameter existed. - The item set, and each item's JSON, are identical to `bd list --json`; - only the order differs. + THE ORDER AND THE CURSOR ARE ONE DECISION, which is why the vocabulary + is closed rather than the nine-value display vocabulary `bd list --sort` + and `GET /v0/beads/issues:query` take. A `cursor` is a keyset POSITION + in a total order, so each served order needs its own strictly-after + predicate and its own proof that its key is total. Both of these have + one: `priority` and `created_at` are non-null and `id` is unique, so a + page boundary landing inside a run of equal keys resolves on `id` with + no dropped and no repeated row. The other seven orders do not — `id` is + a natural-numeric order no database expresses, `updated` moves on every + write, `closed` is nullable, and `status`/`title`/`type`/`assignee` are + mutable and unindexed — so they are not offered here at all rather than + offered behind a cursor that would silently skip and duplicate. + + + A CURSOR IS A POSITION IN THE ORDER IT WAS MINTED IN, and re-sending + one under a different `sort` is refused with 400 `invalid_cursor` + (recovery: restart paging). This is the one thing about a cursor that + is NOT "just repeat your request": mismatched FILTERS are undetectable + and silently resume from the old position, while a mismatched ORDER is + detected and refused, because a position whose order is unknown is not + interpretable at all. + + + `priority` IS A MUTABLE KEY, which `created_at` is not, and a paging + client should know what that costs. Under `created` only new rows move + relative to a walk; under `priority` an `bd update --priority` moves an + existing row too, so it can be seen twice or missed. That is the + already-documented consequence of a cursor pinning a position rather + than a snapshot, reached by updates as well as by creations — not a new + class of error. Unchanged data never skips or repeats under either + order. + + + The item set, and each item's JSON, are identical to `bd list --json`. + Under `sort=priority` so is the order, so a client that used to page + this operation to exhaustion and re-sort client-side to reproduce + `bd list` can now ask for one page. EPHEMERAL ROWS ARE OUT OF SCOPE BY DEFAULT and are admitted by `include_ephemeral`, which merges that tier IN ADDITION to the durable - one. The merged page is ordered by the same `(created_at DESC, id ASC)` - key across both tiers as if they were one table, and `cursor` pages - across the merge without skipping or repeating a row. + one. The merged page is ordered by the requested `sort` across both + tiers as if they were one table, and `cursor` pages across the merge + without skipping or repeating a row. `include_ephemeral` and `include_infra` are INDEPENDENT and compose: @@ -1165,8 +1199,8 @@ paths: in: query description: >- Include the ephemeral tier — ephemeral rows and the non-synced - rows stored beside them — merged into the same - `(created_at DESC, id ASC)` order as the durable ones. + rows stored beside them — merged into the same order as the durable + ones, whichever order `sort` named. It admits a TIER and takes no TYPE exclusion off, so a row whose @@ -1206,6 +1240,50 @@ paths: description: Only issues carrying this top-level metadata key. schema: type: string + - name: sort + in: query + description: >- + Which of the two served total orders this page is in. `created` is + `(created_at DESC, id ASC)`; `priority` is + `(priority ASC, created_at DESC, id ASC)` — `bd list`'s flagless + ordering, and the order `bd list --sort priority` produces. + + + THE DEFAULT IS `created` AND WILL NOT CHANGE. It is what this + operation served before this parameter existed, so moving it would + alter which rows a truncated page contains for every client written + against that behavior, with no error to notice it by. + + + THE VOCABULARY IS CLOSED, and deliberately smaller than the nine + values `bd list --sort` and `GET /v0/beads/issues:query` take. Each + value here is a cursor contract, not a display preference: it needs + a keyset predicate and a key proven total. See the operation + description for why the other seven have neither. + + + A value outside the enum is a 400 `invalid_argument` with + `param: "sort"` and `reason: "invalid_value"`. A server that + predates this parameter answers `param: "sort"` with + `reason: "unknown_parameter"` instead, which is the per-parameter + capability probe a client can dispatch on to fall back to paging in + `created` order and sorting client-side. + schema: + type: string + enum: [created, priority] + default: created + # CODEGEN TRAP, verified against oapi-codegen v2.6.0: these are the + # names the generator ALREADY produces for this enum, spelled out so + # it stops renaming a DIFFERENT one. It resolves colliding constant + # names by prefixing the type name, and adding a third `priority` + # value to the package flipped that resolution for + # `ListReadyWorkParamsSort`, whose constants regenerated as bare + # `Hybrid`/`Oldest`/`Priority` — a breaking rename of exported + # identifiers, in a package whose whole job is to be a stable + # generated client, caused by an unrelated parameter. Pinning the + # names here makes the regeneration purely additive. Removing this + # line reproduces the rename; that is the check, not a guess. + x-enum-varnames: [ListIssuesParamsSortCreated, ListIssuesParamsSortPriority] - name: cursor in: query description: >- @@ -1218,21 +1296,32 @@ paths: the same value cannot succeed. - LIFETIME: a cursor holds a position and a private encoding version, - and nothing else. The server keeps no state for it, so it does not - expire, does not become invalid when the server restarts, and is - not tied to the connection that issued it; the only thing that - invalidates one is a change to the encoding, which surfaces as - `invalid_cursor`. + LIFETIME: a cursor holds a position, the ORDER that position is in, + and a private encoding version — and nothing else. The server keeps + no state for it, so it does not expire, does not become invalid + when the server restarts, and is not tied to the connection that + issued it; the only thing that invalidates one is a change to the + encoding, which surfaces as `invalid_cursor`. A token minted before + `sort` existed is still accepted, as the `created`-order position + it is, so no traversal in flight across a server upgrade has to be + restarted. - MISUSE IS NOT DETECTABLE, which is why repeating the filters - matters. Because the token carries no filters, a page fetched with - a cursor minted under DIFFERENT filters is not refused: the server - applies the filters of the current request from the position of the - old one, silently skipping every row the new filter set would have - placed before that position. Repeat every filter verbatim for the - whole traversal, and start a new traversal when they change. + THE ORDER MUST MATCH, AND THAT MISMATCH IS DETECTED. A cursor + minted under one `sort` and re-sent under another is refused with + 400 `invalid_cursor`; the same instant and id name a different row + in a different order, so resuming from it would skip and duplicate + rows silently. Send the same `sort` for the whole traversal. + + + FILTER MISUSE IS *NOT* DETECTABLE, which is the difference, and why + repeating the filters matters. Because the token carries no + filters, a page fetched with a cursor minted under DIFFERENT + filters is not refused: the server applies the filters of the + current request from the position of the old one, silently skipping + every row the new filter set would have placed before that + position. Repeat every filter verbatim for the whole traversal, and + start a new traversal when they change. schema: type: string - name: limit @@ -3332,7 +3421,7 @@ paths: A collection-level custom method rather than `DELETE - /v0/beads/issues/{id}`: this acts on a SET, its behaviour turns on + /v0/beads/issues/{id}`: this acts on a SET, its behavior turns on three flags, and a `DELETE` carrying those in a query string is the shape where a dropped parameter changes what is erased. Sending a body makes every one of them a member the server refuses by name if it does @@ -4578,7 +4667,7 @@ paths: command's `--json` emits. - THESE ARE ROWS, NOT NEIGHBOURS, and that is the difference from the + THESE ARE ROWS, NOT NEIGHBORS, and that is the difference from the `dependencies` member of `GET /v0/beads/issues/{id}`'s issue body. A dependency target may be an `external:` reference or an id belonging to another repository, and this database holds no issue for either; such diff --git a/internal/storage/dolt/keyset_search_test.go b/internal/storage/dolt/keyset_search_test.go index 60871b3c5..ea756bd4a 100644 --- a/internal/storage/dolt/keyset_search_test.go +++ b/internal/storage/dolt/keyset_search_test.go @@ -49,3 +49,57 @@ func TestSearchIssuesKeysetPlanIsIndexed(t *testing.T) { t.Fatalf("keyset predicate does not seek idx_issues_created_at (want IndexedTableAccess on [issues.created_at]) — the sargable upper bound regressed to a full Table scan.\nplan:\n%s", plan) } } + +// TestSearchIssuesPriorityKeysetPlanIsIndexed is the sargability guard for the +// second served order, and the record of a MEASUREMENT that decided against a +// migration. +// +// What it pins: the redundant `priority >= ?` leading bound keeps the priority +// keyset on an IndexedTableAccess range over idx_issues_priority. Drop that +// bound — leaving the bare OR — and Dolt has nothing to seek and full-scans. +// +// WHAT IT DOES NOT PIN, DELIBERATELY: an index-STREAMED page. The plan under +// this predicate is an index range plus a Filter and a TopN, i.e. a bounded +// per-page sort rather than a walk down a btree in the ORDER BY's own order. +// The obvious fix is a composite (priority, created_at, id) index, and it was +// measured on Dolt 2.2.0 against this exact query: BOTH the plain ascending +// composite and the mixed-direction (priority ASC, created_at DESC, id ASC) +// spelling — which Dolt's DDL accepts — leave the plan byte-identical. The +// planner keeps the single-column priority seek and the TopN in all three +// cases. A migration would therefore have bought one more btree's worth of +// write amplification, on every insert and every `bd update --priority`, for no +// measured change to any plan. It is not shipped, and this comment is why; if a +// later Dolt learns to stream the composite, the measurement to redo is this +// one. +func TestSearchIssuesPriorityKeysetPlanIsIndexed(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + base := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC) + for i := 0; i < 8; i++ { + iss := &types.Issue{ + ID: "pkp-" + string(rune('a'+i)), Title: "pkp", Status: types.StatusOpen, + Priority: i % 4, IssueType: types.TypeTask, CreatedAt: base.Add(time.Duration(i) * time.Second), + } + if err := store.CreateIssue(ctx, iss, "tester"); err != nil { + t.Fatalf("create seed: %v", err) + } + } + + const cur = "2023-01-01 00:00:04" + // Single-source the guarded SQL from production: the five placeholders bind + // priority (sargable lower bound), priority (strict), created_at (sargable + // upper bound), created_at (strict), id (tie-break). + pred := literalizeParams(sqlbuild.KeysetPriorityCreatedAtIDPredicate, "2", "2", "'"+cur+"'", "'"+cur+"'", "''") + //nolint:gosec // G202: pred is a literalized production constant, no user input. + plan := explainPlan(t, ctx, store.db, "SELECT id FROM issues WHERE "+pred+" ORDER BY priority ASC, created_at DESC, id ASC LIMIT 100") + + if !looksLikeDoltPlan(plan) { + t.Skipf("EXPLAIN output not in a recognized Dolt plan format, skipping sargability assertion; plan=\n%s", plan) + } + if !strings.Contains(plan, "IndexedTableAccess") || !strings.Contains(plan, "issues.priority") { + t.Fatalf("the priority keyset predicate does not seek idx_issues_priority (want IndexedTableAccess on [issues.priority]) — the sargable leading bound regressed to a full Table scan.\nplan:\n%s", plan) + } +} diff --git a/internal/storage/dolt/reader_contract_test.go b/internal/storage/dolt/reader_contract_test.go index 100693fac..2bc7a4124 100644 --- a/internal/storage/dolt/reader_contract_test.go +++ b/internal/storage/dolt/reader_contract_test.go @@ -224,6 +224,15 @@ func TestReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing(t conformance.RunReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing(t, ctx, fixture) } +// The same probe-row rule under the second served order, where the next +// position is a triple and the row it is read from is still the last DELIVERED +// one. +func TestReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t *testing.T) { + fixture, ctx, cleanup := newDoltReaderFixture(t, "rdr") + defer cleanup() + conformance.RunReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t, ctx, fixture) +} + func TestReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates(t *testing.T) { fixture, ctx, cleanup := newDoltReaderFixture(t, "rdr") defer cleanup() diff --git a/internal/storage/embeddeddolt/reader_contract_test.go b/internal/storage/embeddeddolt/reader_contract_test.go index 4ee7f3bb7..7d792ad2c 100644 --- a/internal/storage/embeddeddolt/reader_contract_test.go +++ b/internal/storage/embeddeddolt/reader_contract_test.go @@ -217,6 +217,12 @@ func TestEmbeddedReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsN conformance.RunReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing(t, ctx, newEmbeddedReaderFixture(t, "rdr")) } +func TestEmbeddedReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t *testing.T) { + skipUnlessEmbeddedDolt(t) + ctx := t.Context() + conformance.RunReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t, ctx, newEmbeddedReaderFixture(t, "rdr")) +} + func TestEmbeddedReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates(t *testing.T) { skipUnlessEmbeddedDolt(t) ctx := t.Context() diff --git a/internal/storage/sqlbuild/filter.go b/internal/storage/sqlbuild/filter.go index b1375429e..777f1fa9f 100644 --- a/internal/storage/sqlbuild/filter.go +++ b/internal/storage/sqlbuild/filter.go @@ -27,6 +27,29 @@ import ( // a change here then breaks the guard. const KeysetCreatedAtIDPredicate = "(created_at <= ? AND ((created_at < ?) OR (id > ?)))" +// KeysetPriorityCreatedAtIDPredicate is the SARGABLE (priority ASC, +// created_at DESC, id ASC) keyset predicate emitted when the position carries a +// priority (IssueFilter.AfterPriority set alongside AfterCreatedAt/AfterID). +// Its five ? placeholders bind, in order: priority (the sargable lower bound), +// priority (strict), created_at (the sargable upper bound), created_at +// (strict), id (the tie-break). +// +// IT NESTS THE CREATED PREDICATE RATHER THAN RESTATING IT, which is both a +// single-sourcing choice and the correctness argument. The created/id +// comparison is only reachable for rows at the cursor's OWN priority, so it has +// to sit inside the priority-equal arm. Flattening it the way the created +// predicate's own leading-bound trick invites — +// `(priority >= ? AND ((priority > ?) OR (created_at < ?) OR (id > ?)))` — +// looks equivalent and is not: at the cursor's priority it admits a row created +// AFTER the cursor whenever that row's id sorts later, and that row was already +// delivered on an earlier page. +// +// The leading `priority >= ?` bound is redundant in the same way and for the +// same reason as the created predicate's: it prunes only priority < P, which +// the OR already excludes, and it gives the planner an index range to seek +// instead of a full scan. +const KeysetPriorityCreatedAtIDPredicate = "(priority >= ? AND ((priority > ?) OR " + KeysetCreatedAtIDPredicate + "))" + // BuildIssueFilterClauses builds WHERE clause fragments and args from a query // string and IssueFilter. The tables parameter controls which table names are // referenced in subqueries (issues vs wisps). @@ -283,11 +306,23 @@ func BuildIssueFilterClauses(query string, filter types.IssueFilter, tables Filt // wisps created_at columns are DATETIME (NUMERIC affinity), so an RFC3339 // string parameter mis-compares on the SQLite backend, while a time.Time // value compares correctly on every backend — the same binding EventsSince - // uses. Bound twice (the sargable upper bound and the strict bound), then - // the id tie-break. + // uses. It is bound twice under either order (the sargable upper bound + // and the strict bound), followed by the id tie-break. ac := *filter.AfterCreatedAt - whereClauses = append(whereClauses, KeysetCreatedAtIDPredicate) - args = append(args, ac, ac, filter.AfterID) + // AfterCreatedAt alone decides that a position was supplied; AfterPriority + // decides which ORDER it is a position in. The two predicates are + // ALTERNATIVES, never conjuncts: ANDing them would keep the created + // predicate's `created_at <= ?` bound in force across every priority, + // which excludes exactly the rows a priority walk exists to reach — + // everything of a higher-numbered priority created after the cursor. + if filter.AfterPriority != nil { + ap := *filter.AfterPriority + whereClauses = append(whereClauses, KeysetPriorityCreatedAtIDPredicate) + args = append(args, ap, ap, ac, ac, filter.AfterID) + } else { + whereClauses = append(whereClauses, KeysetCreatedAtIDPredicate) + args = append(args, ac, ac, filter.AfterID) + } } if filter.Deferred { diff --git a/internal/storage/sqlbuild/keyset_test.go b/internal/storage/sqlbuild/keyset_test.go index a6486fd64..962e841d7 100644 --- a/internal/storage/sqlbuild/keyset_test.go +++ b/internal/storage/sqlbuild/keyset_test.go @@ -90,3 +90,100 @@ func TestKeysetComposesWithCreatedBefore(t *testing.T) { t.Fatalf("arg count = %d, want 4 (1 CreatedBefore + 3 keyset)", len(args)) } } + +// TestPriorityKeysetPredicateEmission pins the (priority ASC, created_at DESC, +// id ASC) keyset predicate BuildIssueFilterClauses emits when the position +// carries a priority: the exact sargable fragment (single-sourced from +// KeysetPriorityCreatedAtIDPredicate) and its five bound args in order +// (priority, priority, created_at, created_at, id). +// +// The created-order predicate must NOT also be emitted as a standalone clause: +// the two are alternatives, and ANDing them would exclude every row of a +// higher-numbered priority whose created_at is newer than the cursor's — the +// exact rows a priority walk exists to reach. +func TestPriorityKeysetPredicateEmission(t *testing.T) { + t.Parallel() + + cur := time.Date(2024, 3, 2, 1, 0, 0, 0, time.UTC) + p := 2 + + clauses, args, err := BuildIssueFilterClauses("", types.IssueFilter{ + AfterPriority: &p, + AfterCreatedAt: &cur, + AfterID: "bd-42", + }, IssuesFilterTables) + if err != nil { + t.Fatalf("BuildIssueFilterClauses (priority keyset): %v", err) + } + found, plain := 0, 0 + for _, c := range clauses { + switch c { + case KeysetPriorityCreatedAtIDPredicate: + found++ + case KeysetCreatedAtIDPredicate: + plain++ + } + } + if found != 1 { + t.Fatalf("priority keyset predicate clause count = %d, want 1; clauses=%v", found, clauses) + } + if plain != 0 { + t.Fatalf("the created-order predicate was ALSO emitted (%d times); the two orders are alternatives, not conjuncts: %v", plain, clauses) + } + want := []any{p, p, cur, cur, "bd-42"} + if len(args) != len(want) { + t.Fatalf("args = %v, want %v", args, want) + } + for i := range want { + if args[i] != want[i] { + t.Fatalf("arg[%d] = %v (%T), want %v (%T)", i, args[i], args[i], want[i], want[i]) + } + } +} + +// TestPriorityKeysetNeedsBothHalvesOfThePosition pins that AfterPriority alone +// is not a position. The timestamp half is what decides whether a position was +// supplied — the same rule AfterID already follows — so a request carrying a +// priority and no instant emits no keyset predicate at all rather than a +// half-built one that would silently drop every row of the cursor's own +// priority. +func TestPriorityKeysetNeedsBothHalvesOfThePosition(t *testing.T) { + t.Parallel() + + p := 2 + clauses, args, err := BuildIssueFilterClauses("", types.IssueFilter{AfterPriority: &p}, IssuesFilterTables) + if err != nil { + t.Fatalf("BuildIssueFilterClauses (priority only): %v", err) + } + joined := strings.Join(clauses, " AND ") + if strings.Contains(joined, "priority >= ?") { + t.Fatalf("a priority with no instant emitted a keyset predicate: %v", clauses) + } + if len(args) != 0 { + t.Fatalf("args = %v, want none", args) + } +} + +// TestPriorityKeysetPredicateNestsTheCreatedOne pins the SHAPE that makes the +// predicate correct, not merely its text. Under (priority ASC, created_at DESC, +// id ASC) the created/id comparison is only reachable for rows of the cursor's +// OWN priority, so it must sit inside the priority-equal arm. A flattened +// `(priority > ?) OR (created_at < ?) OR (id > ?)` — the shape the created-only +// predicate's own leading-bound trick invites — admits a row at the cursor's +// priority created AFTER the cursor whenever its id sorts later, which is a +// duplicate on the next page. +func TestPriorityKeysetPredicateNestsTheCreatedOne(t *testing.T) { + t.Parallel() + + if !strings.Contains(KeysetPriorityCreatedAtIDPredicate, KeysetCreatedAtIDPredicate) { + t.Fatalf("the priority predicate does not contain the created one verbatim, so the two can drift:\n priority: %s\n created: %s", + KeysetPriorityCreatedAtIDPredicate, KeysetCreatedAtIDPredicate) + } + if !strings.HasPrefix(KeysetPriorityCreatedAtIDPredicate, "(priority >= ?") { + t.Fatalf("the priority predicate lost its sargable leading bound: %s", KeysetPriorityCreatedAtIDPredicate) + } + if strings.Count(KeysetPriorityCreatedAtIDPredicate, "?") != 5 { + t.Fatalf("the priority predicate binds %d placeholders, want 5 (priority, priority, created_at, created_at, id): %s", + strings.Count(KeysetPriorityCreatedAtIDPredicate, "?"), KeysetPriorityCreatedAtIDPredicate) + } +} diff --git a/internal/storage/uow/reader_contract_test.go b/internal/storage/uow/reader_contract_test.go index 121d4b2ac..427219c4e 100644 --- a/internal/storage/uow/reader_contract_test.go +++ b/internal/storage/uow/reader_contract_test.go @@ -144,6 +144,14 @@ func TestReaderContract(t *testing.T) { t.Run("ListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing", func(t *testing.T) { conformance.RunReaderListKeysetWalkOverAnOversizedGroupLosesNothingAndRepeatsNothing(t, ctx, fixture) }) + // The second served order's walk. This seam ORDERs a SQL union, so the + // priority-equal arm of the predicate and the union's ORDER BY have to agree + // about what "after" means across both legs — a place the created-order walk + // never exercises, because there the tie-break is reached by a shared second + // alone. + t.Run("ListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing", func(t *testing.T) { + conformance.RunReaderListPriorityKeysetWalkOverAnOversizedEqualKeyRunLosesNothingAndRepeatsNothing(t, ctx, fixture) + }) t.Run("ListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates", func(t *testing.T) { conformance.RunReaderListKeysetPositionNarrowsWithoutReplacingTheOtherPredicates(t, ctx, fixture) }) diff --git a/internal/types/types.go b/internal/types/types.go index 731a238d9..82bc0b243 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -1901,6 +1901,33 @@ type IssueFilter struct { AfterCreatedAt *time.Time AfterID string + // AfterPriority EXTENDS the position above to the (priority ASC, + // created_at DESC, id ASC) order — the order SortBy="priority" (and the + // empty default) renders. When it is set the restriction becomes + // (priority > AfterPriority) + // OR (priority = AfterPriority AND created_at < AfterCreatedAt) + // OR (priority = AfterPriority AND created_at = AfterCreatedAt AND id > AfterID), + // which is total for the same reason the pair above is: priority and + // created_at are NOT NULL and id is the primary key, so a page boundary + // inside a run of equal (priority, created_at) resolves on id with no + // dropped and no duplicated row. + // + // IT IS THE SAME POSITION, NOT A SECOND ONE. AfterCreatedAt still decides + // whether a position was supplied at all; a priority with no instant is + // half a position and is ignored, exactly as AfterID alone is. Set it only + // under the priority order — pairing it with SortBy="created" positions in + // an order the ORDER BY does not render, which pages a walk through rows + // in an order neither side agrees on. + // + // THE KEY IS MUTABLE, which created_at is not, and that changes what a + // walk can promise. `bd update --priority` moves a row between pages + // mid-walk, so a row can be seen twice or missed — the already-documented + // consequence of pinning a position rather than a snapshot, reached here + // by updates as well as by creations. What totality buys is that + // UNCHANGED data never skips or duplicates, which is what welding the + // listing to the created order originally bought. + AfterPriority *int + // Empty/null checks EmptyDescription bool NoAssignee bool diff --git a/internal/workapi/list.go b/internal/workapi/list.go index 49cf30dab..761d822d2 100644 --- a/internal/workapi/list.go +++ b/internal/workapi/list.go @@ -208,6 +208,7 @@ func BuildListFilter(in issueops.ListRequest, cfg ListConfig) (types.IssueFilter SortDesc: in.Reverse, AfterCreatedAt: in.AfterCreatedAt, AfterID: in.AfterID, + AfterPriority: in.AfterPriority, // The defensive cap travels ON the request, so this builder is the // only writer of the filter's two cap fields. `bd list` used to stamp // them onto the filter after the builder returned, which is the diff --git a/internal/workapi/list_keyset_test.go b/internal/workapi/list_keyset_test.go new file mode 100644 index 000000000..e38e84254 --- /dev/null +++ b/internal/workapi/list_keyset_test.go @@ -0,0 +1,84 @@ +package workapi + +import ( + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/issueops" +) + +// TestBuildListFilterCarriesTheWholeKeysetPosition pins that the position +// reaches the filter INTACT. The builder is the single seam every +// implementation of issueops.Reader passes through, so a half-carried position +// — the instant and the id but not the priority — is not a partial feature: it +// is a priority-ordered ORDER BY walked with a created-order predicate, which +// silently drops every row of a higher-numbered priority created after the +// cursor. +func TestBuildListFilterCarriesTheWholeKeysetPosition(t *testing.T) { + var cfg ListConfig + + at := time.Date(2024, 5, 6, 7, 8, 9, 0, time.UTC) + priority := 2 + + filter, err := BuildListFilter(issueops.ListRequest{ + SortBy: "priority", + AfterCreatedAt: &at, + AfterID: "bd-9", + AfterPriority: &priority, + }, cfg) + if err != nil { + t.Fatalf("BuildListFilter: %v", err) + } + if filter.AfterCreatedAt == nil || !filter.AfterCreatedAt.Equal(at) { + t.Errorf("filter.AfterCreatedAt = %v, want %v", filter.AfterCreatedAt, at) + } + if filter.AfterID != "bd-9" { + t.Errorf("filter.AfterID = %q, want %q", filter.AfterID, "bd-9") + } + if filter.AfterPriority == nil { + t.Fatal("filter.AfterPriority is nil: the priority half of the position was dropped, so a priority-ordered page would be walked with a created-order predicate") + } + if *filter.AfterPriority != priority { + t.Errorf("filter.AfterPriority = %d, want %d", *filter.AfterPriority, priority) + } + if filter.SortBy != "priority" { + t.Errorf("filter.SortBy = %q, want %q — the predicate and the ORDER BY must name the same order", filter.SortBy, "priority") + } + + // A request with no position carries none, so the absent case cannot be + // satisfied by a builder that stamps a zero priority onto every filter. + unpositioned, err := BuildListFilter(issueops.ListRequest{SortBy: "priority"}, cfg) + if err != nil { + t.Fatalf("BuildListFilter (unpositioned): %v", err) + } + if unpositioned.AfterPriority != nil { + t.Errorf("filter.AfterPriority = %d on a request that sent no cursor, want nil", *unpositioned.AfterPriority) + } +} + +// TestReadyFlagRefusesThePriorityHalfOfThePosition pins that the priority half +// is refused by the --ready arm for the same reason the instant half is: the +// blocker-aware query cannot carry a keyset position at all, and a refusal that +// named only two thirds of one would let a caller "fix" their request by +// dropping the fields it named and get the wider set anyway. +func TestReadyFlagRefusesThePriorityHalfOfThePosition(t *testing.T) { + var cfg ListConfig + + at := time.Now().UTC().Add(-time.Hour) + priority := 1 + _, err := BuildListFilter(issueops.ListRequest{ + ReadyFlag: true, + AfterCreatedAt: &at, + AfterID: "bd-9", + AfterPriority: &priority, + }, cfg) + if err == nil { + t.Fatal("BuildListFilter accepted a keyset position under --ready; want ErrValidation") + } + for _, want := range []string{"AfterCreatedAt", "AfterPriority"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("the refusal does not name %s, so a caller cannot tell which part of their position was the problem: %v", want, err) + } + } +} diff --git a/internal/workapi/testdata/list_filter_golden.json b/internal/workapi/testdata/list_filter_golden.json index dcec6ef34..d7fa4f5f2 100644 --- a/internal/workapi/testdata/list_filter_golden.json +++ b/internal/workapi/testdata/list_filter_golden.json @@ -36,6 +36,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -119,6 +120,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -199,6 +201,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -279,6 +282,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -362,6 +366,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -442,6 +447,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -522,6 +528,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -602,6 +609,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -682,6 +690,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -765,6 +774,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -859,6 +869,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -944,6 +955,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1025,6 +1037,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1105,6 +1118,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1189,6 +1203,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1270,6 +1285,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1351,6 +1367,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1433,6 +1450,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1520,6 +1538,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1604,6 +1623,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1687,6 +1707,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1768,6 +1789,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1848,6 +1870,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -1928,6 +1951,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2011,6 +2035,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2098,6 +2123,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2182,6 +2208,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2267,6 +2294,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2383,6 +2411,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2471,6 +2500,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": true, "NoAssignee": true, "NoLabels": true, @@ -2556,6 +2586,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2650,6 +2681,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2734,6 +2766,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2822,6 +2855,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2909,6 +2943,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -2993,6 +3028,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -3076,6 +3112,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -3159,6 +3196,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, @@ -3243,6 +3281,7 @@ "StartedBefore": null, "AfterCreatedAt": null, "AfterID": "", + "AfterPriority": null, "EmptyDescription": false, "NoAssignee": false, "NoLabels": false, diff --git a/issueops/reader.go b/issueops/reader.go index 0f6f44d02..80db1527d 100644 --- a/issueops/reader.go +++ b/issueops/reader.go @@ -398,8 +398,24 @@ type ListRequest struct { // AfterCreatedAt and AfterID carry a decoded keyset position in the // (created_at DESC, id ASC) order. The opaque token that encodes them is a // transport concern and never reaches this contract. + // + // AfterPriority EXTENDS that position to the (priority ASC, created_at + // DESC, id ASC) order — the order SortBy="priority" and the empty default + // render — and is honored by every implementation, on BOTH tier legs, the + // same way the pair above is. It is the same position and not a second + // one: AfterCreatedAt alone decides whether one was supplied, so a + // priority with no instant is ignored exactly as an AfterID with no instant + // is. + // + // SET IT ONLY UNDER THE ORDER IT NAMES. A priority position under + // SortBy="created" positions in an order the ORDER BY does not render, and + // the page that comes back is a walk through rows neither side agrees on. + // The order the request asks for and the order the position was minted in + // are one decision; the transport above this contract is what keeps them + // from drifting. AfterCreatedAt *time.Time AfterID string + AfterPriority *int // MaxRows is a DEFENSIVE CAP rather than a page. It bounds how many rows // the query may match before the whole answer is refused; 0 disables it. diff --git a/issueops/reader_ready_scope.go b/issueops/reader_ready_scope.go index 011aa70ff..bfcda1b3d 100644 --- a/issueops/reader_ready_scope.go +++ b/issueops/reader_ready_scope.go @@ -91,9 +91,11 @@ var readyScopeFields = []readyScopeField{ {"PriorityMin", "--priority-min", func(r ListRequest) bool { return r.PriorityMin != nil }}, {"PriorityMax", "--priority-max", func(r ListRequest) bool { return r.PriorityMax != nil }}, - // The keyset position is a pair and only the timestamp half decides - // whether one was supplied; AfterID alone is not a position. - {"AfterCreatedAt/AfterID", "cursor", func(r ListRequest) bool { return r.AfterCreatedAt != nil }}, + // The keyset position is a tuple and only the timestamp half decides + // whether one was supplied; neither AfterID nor AfterPriority alone is a + // position. All three are named in the refusal because a caller who dropped + // only the ones named would still be sending part of a position. + {"AfterCreatedAt/AfterID/AfterPriority", "cursor", func(r ListRequest) bool { return r.AfterCreatedAt != nil }}, } // ValidateReadyFlagScope refuses a list request that combines ReadyFlag with a