perf(ready): answer the page and its total in one pass (#6731)
* perf(issueops): answer a ready page and its total in one pass `bd ready --json --limit 1` against a remote SQL store (~55 ms per round trip) took 5.4 s and ~100 sequential round trips; `--limit 0` (557 rows) took 3.8 s. Asking for one row was slower than asking for all of them. The data is small, so the cost was purely statement count: - a capped page re-ran the whole ready predicate in a SECOND transaction (CountReadyWorkInTx, behind a second defer-wake sweep) just to print "Showing 1 of 556"; - every leg re-probed the same facts: wisp_dependencies existence, wisps emptiness, and the future-deferred-parent scan (2 probes + 4 child queries), once per table family and again in the count pass. This change: - Adds a per-transaction probe (probeReadyWorkInTx): one statement of EXISTS subqueries answers wisps-present, wisp_dependencies-exists, future-deferred-parent-exists and (when sizing) issues/wisps ID collision; the deferred-parent children are then read in one UNION ALL instead of four queries. The facts and the rendered ID sets are shared by both families. A missing-table error (pre-wisp database) falls back to the old per-table probes, so their tolerance and error classification are unchanged. - Adds GetReadyWorkWithCountsAndTotalInTx and DoltStorage.GetReadyWorkWithCountsAndTotal: the capped page's ID query carries COUNT(*) OVER (), evaluated over the full predicate before LIMIT, so the total costs no extra statement. The overlap count only runs when the probe saw an ID in both tables. Window functions are supported by every engine this code runs on (Dolt/go-mysql-server server and embedded, MySQL 8, Postgres, SQLite >= 3.25). The externaldeps decorator overrides the new method so its exclusions apply to page and total alike, and now also overrides ReadyCounter() so the text-mode total honors the same exclusions (it used to pass through to the inner store and count externally blocked issues); telemetry instruments the new method. - CountReadyWorkInTx (ReadyCounter role, text `bd ready`) uses the probe and counts both families in one statement. Whole-command statements for `bd ready --json`, counted from a local dolt sql-server query log (13 of each are startup/config reads this change does not touch): store --limit 1 --limit 0 issues only 28 -> 18 20 -> 17 issues + wisps 34 -> 19 23 -> 18 issues + wisps + deferred epic 46 -> 20 29 -> 19 TestReadyWorkStatementBudget holds the per-transaction statement count at or under today's budget on embedded Dolt. TestReadyWorkPageTotal and TestReadyWorkPageTotalFilterMatrix check the page and total against hand-written expected ready sets (wisps, deferred parent, blocked, ID collision; sorts, label/assignee/priority/type/parent/deferred/exclude filters) and every capped page against the unbounded listing's prefix. stdout (plain and BD_JSON_ENVELOPE=1) and the stderr hint were byte-identical before/after for every case above. The one intended output change: with an external dependency present, the --json total (and the text "of N") now excludes externally blocked issues, as the page already did; before, the total counted them. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf(ready): take the --json total in-band instead of a second count pass `bd ready --json --limit N` listed a page, then - when the page came back full - opened a second store call (ReadyCounter.CountReady: a new read transaction plus its own defer-wake sweep) only to print "Showing N of M" on stderr and the pagination total in the JSON envelope. On a remote store (~55 ms per round trip) that pass is why `--limit 1` (5.4 s) was slower than `--limit 0` (3.8 s). The --json branch now calls GetReadyWorkWithCountsAndTotal, which returns the page and the size of the whole ready set from the same transaction. The total is the same identity the ReadyCounter role promises, taken over the listing's own filter, so the hint text, the envelope's pagination and the stdout JSON are identical except that the total now honors external-dependency exclusions (the externaldeps decorator applies them to page and total alike; the old second pass counted externally blocked issues). The text-mode branch still asks the ReadyCounter role, which the previous commit made cheaper and made apply the same exclusions, so text and --json report one number. The embedded CLI test now pins the exact "Showing 2 of N" hint and the envelope's pagination total against an unbounded listing. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
c507f3b4c9
commit
3c30a28d07
16 changed files with 1089 additions and 110 deletions
|
|
@ -174,7 +174,15 @@ This is useful for agents executing molecules to see which steps can run next.`,
|
|||
}
|
||||
|
||||
if jsonOutput {
|
||||
results, err := activeStore.GetReadyWorkWithCounts(ctx, filter)
|
||||
// The page and the size of the whole ready set come back from ONE
|
||||
// read transaction: the total rides the page's own ID query, so a
|
||||
// capped listing no longer pays for a second counting pass (and a
|
||||
// second defer-wake sweep) just to print "Showing N of M". Against
|
||||
// a remote SQL server each of that pass's statements was a
|
||||
// sequential round trip. The total is the same number the
|
||||
// ReadyCounter role answers (storage.DoltStorage documents the
|
||||
// identity), taken over the listing's own filter.
|
||||
results, total, err := activeStore.GetReadyWorkWithCountsAndTotal(ctx, filter)
|
||||
if err != nil {
|
||||
if capErr := handleMaxRowsError(err); capErr != nil {
|
||||
return capErr
|
||||
|
|
@ -183,15 +191,9 @@ This is useful for agents executing molecules to see which steps can run next.`,
|
|||
}
|
||||
totalReady := len(results)
|
||||
truncated := false
|
||||
if filter.Limit > 0 && len(results) == filter.Limit {
|
||||
// The page is full, so there may be more ready work. The
|
||||
// ReadyCounter role promises its answer equals
|
||||
// len(Reader.Ready(Limit=0).Items), which is what makes this
|
||||
// total describe the page above it.
|
||||
if n, countErr := readyTotal(ctx, activeStore, in); countErr == nil && n > len(results) {
|
||||
totalReady = n
|
||||
truncated = true
|
||||
}
|
||||
if filter.Limit > 0 && len(results) == filter.Limit && total > len(results) {
|
||||
totalReady = total
|
||||
truncated = true
|
||||
}
|
||||
if results == nil {
|
||||
results = []*types.IssueWithCounts{}
|
||||
|
|
@ -224,9 +226,10 @@ This is useful for agents executing molecules to see which steps can run next.`,
|
|||
totalReady := len(issues)
|
||||
truncated := false
|
||||
if filter.Limit > 0 && len(issues) == filter.Limit {
|
||||
// The same question the --json branch asks, through the same role,
|
||||
// so the "Showing X of N" a human reads and the total a script
|
||||
// parses are one number.
|
||||
// The same question the --json branch answers in-band, asked here
|
||||
// of the ReadyCounter role, whose answer is the same identity, so
|
||||
// the "Showing X of N" a human reads and the total a script parses
|
||||
// are one number.
|
||||
if n, countErr := readyTotal(ctx, activeStore, in); countErr == nil && n > len(issues) {
|
||||
totalReady = n
|
||||
truncated = true
|
||||
|
|
@ -402,10 +405,13 @@ var blockedCmd = &cobra.Command{
|
|||
// readyTotal sizes the whole ready set for the request `bd ready` just listed
|
||||
// a page of, through the store's own ReadyCounter accessor.
|
||||
//
|
||||
// BOTH OUTPUT MODES CALL IT and only when the page came back full, which is
|
||||
// THE TEXT OUTPUT CALLS IT, and only when the page came back full, which is
|
||||
// the one situation where the answer can differ from what is already on
|
||||
// screen. The role has no --max-rows field to honor and needs none: the cap
|
||||
// bounds a page this machine materializes, and a count materializes no rows.
|
||||
// screen. The --json listing does not: it takes its total in-band from
|
||||
// GetReadyWorkWithCountsAndTotal, in the page's own transaction.
|
||||
//
|
||||
// The role has no --max-rows field to honor and needs none: the cap bounds a
|
||||
// page this machine materializes, and a count materializes no rows.
|
||||
//
|
||||
// A failed count is not a failed command — the page is already correct; all
|
||||
// that is lost is the "of N" beside it.
|
||||
|
|
|
|||
|
|
@ -107,6 +107,44 @@ func TestEmbeddedReady(t *testing.T) {
|
|||
if !strings.Contains(stderr.String(), "Use --limit 0 for all") {
|
||||
t.Fatalf("expected truncation hint on stderr, got: %q", stderr.String())
|
||||
}
|
||||
|
||||
// The "of N" is the whole ready set, now taken in the page's own
|
||||
// transaction: it must be exactly what an unbounded listing returns,
|
||||
// in the hint and in the envelope's pagination alike.
|
||||
all := exec.Command(bd, "ready", "--json", "--limit", "0")
|
||||
all.Dir = dir
|
||||
all.Env = bdEnv(dir)
|
||||
allOut, err := all.Output()
|
||||
if err != nil {
|
||||
t.Fatalf("bd ready --json --limit 0 failed: %v", err)
|
||||
}
|
||||
var unbounded []types.IssueWithCounts
|
||||
if err := json.Unmarshal(bytes.TrimSpace(allOut), &unbounded); err != nil {
|
||||
t.Fatalf("parse unbounded ready JSON: %v\n%s", err, allOut)
|
||||
}
|
||||
wantHint := fmt.Sprintf("Showing 2 of %d ready issues.", len(unbounded))
|
||||
if !strings.Contains(stderr.String(), wantHint) {
|
||||
t.Fatalf("hint = %q, want it to contain %q", stderr.String(), wantHint)
|
||||
}
|
||||
|
||||
env := exec.Command(bd, "ready", "--json", "--limit", "2")
|
||||
env.Dir = dir
|
||||
env.Env = append(bdEnv(dir), "BD_JSON_ENVELOPE=1")
|
||||
envOut, err := env.Output()
|
||||
if err != nil {
|
||||
t.Fatalf("bd ready --json --limit 2 (envelope) failed: %v", err)
|
||||
}
|
||||
var envelope struct {
|
||||
Data []types.IssueWithCounts `json:"data"`
|
||||
Pagination *PaginationMeta `json:"pagination"`
|
||||
}
|
||||
if err := json.Unmarshal(bytes.TrimSpace(envOut), &envelope); err != nil {
|
||||
t.Fatalf("parse envelope: %v\n%s", err, envOut)
|
||||
}
|
||||
if envelope.Pagination == nil || !envelope.Pagination.Truncated ||
|
||||
envelope.Pagination.Returned != 2 || envelope.Pagination.Total != len(unbounded) {
|
||||
t.Fatalf("pagination = %+v, want returned=2 total=%d truncated", envelope.Pagination, len(unbounded))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("ready_claim_json", func(t *testing.T) {
|
||||
|
|
|
|||
|
|
@ -172,7 +172,7 @@ are inventoried here rather than repeated.
|
|||
There are two families.
|
||||
|
||||
**Direct storage operations** — `storage.<Method>`, one per storage method
|
||||
(72 spans):
|
||||
(73 spans):
|
||||
|
||||
```
|
||||
AddDependency AddIssueComment AddLabel CloseIssue CloseIssueChecked
|
||||
|
|
@ -183,15 +183,16 @@ GetDependenciesWithMetadata GetDependencyTree GetDependents
|
|||
GetDependentsWithMetadata GetEpicsEligibleForClosure GetEvents GetIssue
|
||||
GetIssueByExternalRef GetIssueComments GetIssueCommentsPage GetIssuesByIDs
|
||||
GetIssuesByLabel GetLabels GetLocalMetadata GetProvenanceByRef
|
||||
GetProvenanceEvents GetReadyWork GetReadyWorkWithCounts GetStatistics
|
||||
IterAllEventsSince IterBlockedIssues IterDependenciesWithMetadata
|
||||
IterDependentsWithMetadata IterEvents IterIssueComments IterIssues
|
||||
IterReadyWork IterWisps ListWisps MergeMetadata MergeSlotAcquire
|
||||
MergeSlotCheck MergeSlotCreate MergeSlotRelease RecordProvenanceEvent
|
||||
RemoveDependency RemoveLabel ReopenIssue RunInIssueLifecycleTransaction
|
||||
RunInTransaction SearchIssueIDs SearchIssues SearchIssuesWithCounts SetConfig
|
||||
SetLocalMetadata SlotClear SlotGet SlotSet UnclaimIssue UnclaimIssueIfAssignee
|
||||
UpdateIssue UpdateIssueChecked UpdateIssueType
|
||||
GetProvenanceEvents GetReadyWork GetReadyWorkWithCounts
|
||||
GetReadyWorkWithCountsAndTotal GetStatistics IterAllEventsSince
|
||||
IterBlockedIssues IterDependenciesWithMetadata IterDependentsWithMetadata
|
||||
IterEvents IterIssueComments IterIssues IterReadyWork IterWisps ListWisps
|
||||
MergeMetadata MergeSlotAcquire MergeSlotCheck MergeSlotCreate MergeSlotRelease
|
||||
RecordProvenanceEvent RemoveDependency RemoveLabel ReopenIssue
|
||||
RunInIssueLifecycleTransaction RunInTransaction SearchIssueIDs SearchIssues
|
||||
SearchIssuesWithCounts SetConfig SetLocalMetadata SlotClear SlotGet SlotSet
|
||||
UnclaimIssue UnclaimIssueIfAssignee UpdateIssue UpdateIssueChecked
|
||||
UpdateIssueType
|
||||
```
|
||||
|
||||
**Issueops role operations** — `storage.<Role>.<Method>`, emitted when a caller
|
||||
|
|
|
|||
|
|
@ -117,6 +117,21 @@ func (s *DoltStore) GetReadyWorkWithCounts(ctx context.Context, filter types.Wor
|
|||
return result, err
|
||||
}
|
||||
|
||||
// GetReadyWorkWithCountsAndTotal returns the ready page and the size of the
|
||||
// whole ready set in one read transaction (and one defer-wake sweep), so a
|
||||
// capped `bd ready` needs no separate count pass.
|
||||
func (s *DoltStore) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) {
|
||||
s.wakeExpiredDefers(ctx)
|
||||
var result []*types.IssueWithCounts
|
||||
var total int
|
||||
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
|
||||
var err error
|
||||
result, total, err = issueops.GetReadyWorkWithCountsAndTotalInTx(ctx, tx, filter)
|
||||
return err
|
||||
})
|
||||
return result, total, err
|
||||
}
|
||||
|
||||
// CountReadyWork returns the total ready-work count for filter. It is identical
|
||||
// to len(GetReadyWorkWithCounts(filter with Limit=0)) but sizes the total with
|
||||
// cheap indexed COUNT(*)s instead of re-running the counts mega-query. Backs the
|
||||
|
|
|
|||
|
|
@ -70,6 +70,21 @@ func (s *EmbeddedDoltStore) GetReadyWorkWithCounts(ctx context.Context, filter t
|
|||
return result, err
|
||||
}
|
||||
|
||||
// GetReadyWorkWithCountsAndTotal returns the ready page and the size of the
|
||||
// whole ready set in one read transaction (and one defer-wake sweep), so a
|
||||
// capped `bd ready` needs no separate count pass.
|
||||
func (s *EmbeddedDoltStore) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) {
|
||||
s.wakeExpiredDefers(ctx)
|
||||
var result []*types.IssueWithCounts
|
||||
var total int
|
||||
err := s.withConn(ctx, false, func(tx *sql.Tx) error {
|
||||
var err error
|
||||
result, total, err = issueops.GetReadyWorkWithCountsAndTotalInTx(ctx, tx, filter)
|
||||
return err
|
||||
})
|
||||
return result, total, err
|
||||
}
|
||||
|
||||
// CountReadyWork returns the total ready-work count for filter. It is identical
|
||||
// to len(GetReadyWorkWithCounts(filter with Limit=0)) but sizes the total with
|
||||
// cheap indexed COUNT(*)s instead of re-running the counts mega-query. Backs the
|
||||
|
|
|
|||
396
internal/storage/embeddeddolt/ready_total_test.go
Normal file
396
internal/storage/embeddeddolt/ready_total_test.go
Normal file
|
|
@ -0,0 +1,396 @@
|
|||
//go:build cgo
|
||||
|
||||
package embeddeddolt_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/embeddeddolt"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// countingTx counts the statements a read issues. Against a remote SQL server
|
||||
// every statement is one sequential round trip, so this number — not rows
|
||||
// scanned — is what `bd ready` latency is made of.
|
||||
type countingTx struct {
|
||||
tx *sql.Tx
|
||||
stmts []string
|
||||
}
|
||||
|
||||
func (c *countingTx) record(q string) {
|
||||
c.stmts = append(c.stmts, strings.Join(strings.Fields(q), " "))
|
||||
}
|
||||
|
||||
func (c *countingTx) ExecContext(ctx context.Context, q string, args ...any) (sql.Result, error) {
|
||||
c.record(q)
|
||||
return c.tx.ExecContext(ctx, q, args...)
|
||||
}
|
||||
|
||||
func (c *countingTx) QueryContext(ctx context.Context, q string, args ...any) (*sql.Rows, error) {
|
||||
c.record(q)
|
||||
return c.tx.QueryContext(ctx, q, args...)
|
||||
}
|
||||
|
||||
func (c *countingTx) QueryRowContext(ctx context.Context, q string, args ...any) *sql.Row {
|
||||
c.record(q)
|
||||
return c.tx.QueryRowContext(ctx, q, args...)
|
||||
}
|
||||
|
||||
// withCountingReadTx runs fn inside one read transaction on a raw connection
|
||||
// to te's database and returns the statements fn issued.
|
||||
func withCountingReadTx(t *testing.T, te *testEnv, fn func(tx issueops.DBTX) error) []string {
|
||||
t.Helper()
|
||||
ctx := t.Context()
|
||||
db, cleanup, err := embeddeddolt.OpenSQL(ctx, te.dataDir, te.database, "main")
|
||||
if err != nil {
|
||||
t.Fatalf("OpenSQL: %v", err)
|
||||
}
|
||||
defer func() { _ = cleanup() }()
|
||||
tx, err := db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("begin: %v", err)
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
c := &countingTx{tx: tx}
|
||||
if err := fn(c); err != nil {
|
||||
t.Fatalf("read: %v", err)
|
||||
}
|
||||
return c.stmts
|
||||
}
|
||||
|
||||
func readyIDs(items []*types.IssueWithCounts) []string {
|
||||
ids := make([]string, 0, len(items))
|
||||
for _, it := range items {
|
||||
ids = append(ids, it.Issue.ID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
type readyTotalWorld struct {
|
||||
wisps bool
|
||||
deferredParent bool
|
||||
collision bool
|
||||
}
|
||||
|
||||
// seedReadyTotalWorld builds a ready front with every shape the total has to
|
||||
// get right: plain ready issues, a blocked issue (its blocker stays ready), a
|
||||
// future-deferred parent whose children are hidden, ready wisps, and an ID
|
||||
// present in both tables (the overlap the merge dedupes wisp-wins).
|
||||
func seedReadyTotalWorld(t *testing.T, te *testEnv, prefix string, w readyTotalWorld) {
|
||||
t.Helper()
|
||||
ctx := t.Context()
|
||||
create := func(iss *types.Issue) {
|
||||
t.Helper()
|
||||
if iss.Status == "" {
|
||||
iss.Status = types.StatusOpen
|
||||
}
|
||||
if iss.IssueType == "" {
|
||||
iss.IssueType = types.TypeTask
|
||||
}
|
||||
if err := te.store.CreateIssue(ctx, iss, "tester"); err != nil {
|
||||
t.Fatalf("CreateIssue %s: %v", iss.ID, err)
|
||||
}
|
||||
}
|
||||
dep := func(from, to string, typ types.DependencyType) {
|
||||
t.Helper()
|
||||
if err := te.store.AddDependency(ctx, &types.Dependency{IssueID: from, DependsOnID: to, Type: typ}, "tester"); err != nil {
|
||||
t.Fatalf("AddDependency %s -> %s: %v", from, to, err)
|
||||
}
|
||||
}
|
||||
for i := 0; i < 5; i++ {
|
||||
create(&types.Issue{ID: fmt.Sprintf("%s-r%d", prefix, i), Title: fmt.Sprintf("ready %d", i), Priority: i % 3})
|
||||
}
|
||||
create(&types.Issue{ID: prefix + "-blocker", Title: "blocker", Priority: 1})
|
||||
create(&types.Issue{ID: prefix + "-blocked", Title: "blocked", Priority: 1})
|
||||
dep(prefix+"-blocked", prefix+"-blocker", types.DepBlocks)
|
||||
|
||||
if w.deferredParent {
|
||||
future := time.Now().UTC().Add(72 * time.Hour)
|
||||
create(&types.Issue{ID: prefix + "-dparent", Title: "deferred parent", IssueType: types.TypeEpic, DeferUntil: &future})
|
||||
create(&types.Issue{ID: prefix + "-dchild", Title: "child of deferred", Priority: 0})
|
||||
dep(prefix+"-dchild", prefix+"-dparent", types.DepParentChild)
|
||||
}
|
||||
if w.wisps {
|
||||
for i := 0; i < 3; i++ {
|
||||
create(&types.Issue{ID: fmt.Sprintf("%s-w%d", prefix, i), Title: fmt.Sprintf("wisp %d", i), Priority: 1, Ephemeral: true})
|
||||
}
|
||||
if w.deferredParent {
|
||||
create(&types.Issue{ID: prefix + "-wdchild", Title: "wisp child of deferred", Priority: 0, Ephemeral: true})
|
||||
dep(prefix+"-wdchild", prefix+"-dparent", types.DepParentChild)
|
||||
}
|
||||
}
|
||||
if w.collision {
|
||||
// GH#4455 made creates refuse this, so it is written the way a store
|
||||
// corrupted before that fix holds it: the same row in both tables.
|
||||
te.exec(t, ctx, "INSERT INTO wisps (id, title, status, priority, issue_type, ephemeral, created_at, updated_at) "+
|
||||
"SELECT id, title, status, priority, issue_type, 1, created_at, updated_at FROM issues WHERE id = ?", prefix+"-r0")
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadyWorkPageTotal pins the in-band total GetReadyWorkWithCountsAndTotal
|
||||
// hands `bd ready`: for every page size it must equal both the ReadyCounter
|
||||
// identity (CountReadyWorkInTx) and the length of the unbounded page, and the
|
||||
// page itself must be the unbounded page's prefix — the counting pass it
|
||||
// replaced proved nothing more.
|
||||
func TestReadyWorkPageTotal(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
|
||||
// The expected ready sets are written out by hand rather than derived
|
||||
// from another reader in this package: GetReadyWorkWithCountsInTx,
|
||||
// CountReadyWorkInTx and the method under test all share
|
||||
// probeReadyWorkInTx, so agreement among them alone could not catch a
|
||||
// probe that hid (or leaked) a row in all three at once. Keyed by
|
||||
// IncludeEphemeral; suffixes are appended to the world's prefix.
|
||||
plain := []string{"blocker", "r0", "r1", "r2", "r3", "r4"}
|
||||
withWisps := []string{"blocker", "r0", "r1", "r2", "r3", "r4", "w0", "w1", "w2"}
|
||||
worlds := []struct {
|
||||
name string
|
||||
world readyTotalWorld
|
||||
want map[bool][]string
|
||||
}{
|
||||
{"issues_only", readyTotalWorld{}, map[bool][]string{false: plain, true: plain}},
|
||||
// dparent is future-deferred, so it and its child are hidden.
|
||||
{"issues_deferred_parent", readyTotalWorld{deferredParent: true}, map[bool][]string{false: plain, true: plain}},
|
||||
{"wisps", readyTotalWorld{wisps: true}, map[bool][]string{false: plain, true: withWisps}},
|
||||
// The wisp child of the deferred parent is hidden too.
|
||||
{"wisps_deferred_parent", readyTotalWorld{wisps: true, deferredParent: true}, map[bool][]string{false: plain, true: withWisps}},
|
||||
// r0 sits in both tables; it is listed (and counted) exactly once.
|
||||
{"wisps_collision", readyTotalWorld{wisps: true, collision: true}, map[bool][]string{false: plain, true: withWisps}},
|
||||
}
|
||||
for i, wc := range worlds {
|
||||
t.Run(wc.name, func(t *testing.T) {
|
||||
prefix := fmt.Sprintf("rt%d", i)
|
||||
te := newTestEnv(t, prefix)
|
||||
seedReadyTotalWorld(t, te, prefix, wc.world)
|
||||
|
||||
for _, includeEphemeral := range []bool{false, true} {
|
||||
base := types.WorkFilter{IncludeEphemeral: includeEphemeral}
|
||||
var unbounded []*types.IssueWithCounts
|
||||
var counted int
|
||||
withCountingReadTx(t, te, func(tx issueops.DBTX) error {
|
||||
var err error
|
||||
if unbounded, err = issueops.GetReadyWorkWithCountsInTx(t.Context(), tx, base); err != nil {
|
||||
return err
|
||||
}
|
||||
counted, err = issueops.CountReadyWorkInTx(t.Context(), tx, base)
|
||||
return err
|
||||
})
|
||||
if counted != len(unbounded) {
|
||||
t.Fatalf("ephemeral=%v: CountReadyWorkInTx = %d, unbounded page has %d rows %v",
|
||||
includeEphemeral, counted, len(unbounded), readyIDs(unbounded))
|
||||
}
|
||||
all := readyIDs(unbounded)
|
||||
if got, want := sortedIDs(all), prefixed(prefix, wc.want[includeEphemeral]); strings.Join(got, ",") != strings.Join(want, ",") {
|
||||
t.Fatalf("ephemeral=%v: ready set = %v, want %v", includeEphemeral, got, want)
|
||||
}
|
||||
if len(all) < 3 {
|
||||
t.Fatalf("ephemeral=%v: seeded world too small to page: %v", includeEphemeral, all)
|
||||
}
|
||||
sawWisp := false
|
||||
for _, id := range all {
|
||||
if strings.Contains(id, "-w") {
|
||||
sawWisp = true
|
||||
}
|
||||
if strings.HasSuffix(id, "-blocked") || strings.Contains(id, "dchild") {
|
||||
t.Fatalf("ephemeral=%v: %s is not ready but was listed: %v", includeEphemeral, id, all)
|
||||
}
|
||||
}
|
||||
|
||||
// Identity alone cannot see a wisp family that was never read:
|
||||
// the page and the count would agree on leaving it out.
|
||||
if wantWisp := wc.world.wisps && includeEphemeral; sawWisp != wantWisp {
|
||||
t.Fatalf("ephemeral=%v: wisps listed = %v, want %v: %v", includeEphemeral, sawWisp, wantWisp, all)
|
||||
}
|
||||
|
||||
for _, limit := range []int{0, 1, 2, len(all) - 1, len(all), len(all) + 3, len(all) + 5} {
|
||||
filter := base
|
||||
filter.Limit = limit
|
||||
var page []*types.IssueWithCounts
|
||||
var total int
|
||||
withCountingReadTx(t, te, func(tx issueops.DBTX) error {
|
||||
var err error
|
||||
page, total, err = issueops.GetReadyWorkWithCountsAndTotalInTx(t.Context(), tx, filter)
|
||||
return err
|
||||
})
|
||||
if total != len(all) {
|
||||
t.Errorf("ephemeral=%v limit=%d: total = %d, want %d", includeEphemeral, limit, total, len(all))
|
||||
}
|
||||
want := all
|
||||
if limit > 0 && limit < len(all) {
|
||||
want = all[:limit]
|
||||
}
|
||||
if got := readyIDs(page); strings.Join(got, ",") != strings.Join(want, ",") {
|
||||
t.Errorf("ephemeral=%v limit=%d: page = %v, want %v", includeEphemeral, limit, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadyWorkStatementBudget is the round-trip regression guard. `bd ready
|
||||
// --json --limit 1` against a remote store with ~55 ms per round trip took
|
||||
// 5.4 s — slower than --limit 0 — because a capped page re-ran every probe per
|
||||
// table family and then sized the set in a second pass. These budgets are the
|
||||
// statements ONE read transaction issues for the page AND its total; the
|
||||
// separate count transaction is gone entirely.
|
||||
func TestReadyWorkStatementBudget(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
world readyTotalWorld
|
||||
limit int
|
||||
want int
|
||||
}{
|
||||
// probe + mega-query
|
||||
{"issues_only/limit0", readyTotalWorld{}, 0, 2},
|
||||
// probe + (ID page with window total + counts hydration)
|
||||
{"issues_only/limit1", readyTotalWorld{}, 1, 3},
|
||||
// probe + mega-query per family
|
||||
{"wisps/limit0", readyTotalWorld{wisps: true}, 0, 3},
|
||||
// probe + (ID page + hydration) per family
|
||||
{"wisps/limit1", readyTotalWorld{wisps: true}, 1, 5},
|
||||
// + one statement for every deferred-parent child leg
|
||||
{"wisps_deferred_parent/limit0", readyTotalWorld{wisps: true, deferredParent: true}, 0, 4},
|
||||
{"wisps_deferred_parent/limit1", readyTotalWorld{wisps: true, deferredParent: true}, 1, 6},
|
||||
// + the overlap count, only because an ID sits in both tables
|
||||
{"wisps_collision/limit1", readyTotalWorld{wisps: true, collision: true}, 1, 6},
|
||||
}
|
||||
for i, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
prefix := fmt.Sprintf("rb%d", i)
|
||||
te := newTestEnv(t, prefix)
|
||||
seedReadyTotalWorld(t, te, prefix, tc.world)
|
||||
|
||||
filter := types.WorkFilter{IncludeEphemeral: true, Limit: tc.limit}
|
||||
var total int
|
||||
stmts := withCountingReadTx(t, te, func(tx issueops.DBTX) error {
|
||||
var err error
|
||||
_, total, err = issueops.GetReadyWorkWithCountsAndTotalInTx(t.Context(), tx, filter)
|
||||
return err
|
||||
})
|
||||
if total == 0 {
|
||||
t.Fatalf("total = 0; the seeded world has ready work")
|
||||
}
|
||||
// A ceiling, not an exact pin: a change that saves a statement
|
||||
// should not have to edit this table, but one that adds a round
|
||||
// trip fails. The budgets are today's counts, so there is no slack.
|
||||
if len(stmts) > tc.want {
|
||||
t.Errorf("page+total issued %d statements, budget %d (+%d):\n %s",
|
||||
len(stmts), tc.want, len(stmts)-tc.want, strings.Join(stmts, "\n "))
|
||||
} else if len(stmts) < tc.want {
|
||||
t.Logf("page+total issued %d statements, under budget %d by %d; consider tightening:\n %s",
|
||||
len(stmts), tc.want, tc.want-len(stmts), strings.Join(stmts, "\n "))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func sortedIDs(ids []string) []string {
|
||||
out := slices.Clone(ids)
|
||||
slices.Sort(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func prefixed(prefix string, suffixes []string) []string {
|
||||
out := make([]string, 0, len(suffixes))
|
||||
for _, s := range suffixes {
|
||||
out = append(out, prefix+"-"+s)
|
||||
}
|
||||
slices.Sort(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// TestReadyWorkPageTotalFilterMatrix runs the in-band total across the
|
||||
// WorkFilter fields `bd ready` forwards. Each filter's ready set is written
|
||||
// out by hand (the oracle); the total must equal the unbounded listing's
|
||||
// length, and every capped page must be that listing's prefix, so a filter
|
||||
// the window total ignored — or applied to one family only — fails here.
|
||||
func TestReadyWorkPageTotalFilterMatrix(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
|
||||
const prefix = "rm"
|
||||
te := newTestEnv(t, prefix)
|
||||
seedReadyTotalWorld(t, te, prefix, readyTotalWorld{wisps: true, deferredParent: true})
|
||||
ctx := t.Context()
|
||||
id := func(s string) string { return prefix + "-" + s }
|
||||
for issue, labels := range map[string][]string{"r0": {"a", "b"}, "r1": {"a"}, "blocker": {"a"}, "w0": {"b"}} {
|
||||
for _, l := range labels {
|
||||
if err := te.store.AddLabel(ctx, id(issue), l, "tester"); err != nil {
|
||||
t.Fatalf("AddLabel %s %s: %v", issue, l, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, issue := range []string{"r2", "w1"} {
|
||||
if err := te.store.UpdateIssue(ctx, id(issue), map[string]interface{}{"assignee": "alice"}, "tester"); err != nil {
|
||||
t.Fatalf("assign %s: %v", issue, err)
|
||||
}
|
||||
}
|
||||
|
||||
all := []string{"blocker", "r0", "r1", "r2", "r3", "r4", "w0", "w1", "w2"}
|
||||
alice, zero := "alice", 0
|
||||
parent := id("dparent")
|
||||
cases := []struct {
|
||||
name string
|
||||
filter types.WorkFilter
|
||||
want []string
|
||||
}{
|
||||
{"hybrid", types.WorkFilter{}, all},
|
||||
{"oldest", types.WorkFilter{SortPolicy: types.SortPolicyOldest}, all},
|
||||
{"priority", types.WorkFilter{SortPolicy: types.SortPolicyPriority}, all},
|
||||
{"labels_all", types.WorkFilter{Labels: []string{"a", "b"}}, []string{"r0"}},
|
||||
{"labels_any", types.WorkFilter{LabelsAny: []string{"a", "b"}}, []string{"blocker", "r0", "r1", "w0"}},
|
||||
{"assignee", types.WorkFilter{Assignee: &alice}, []string{"r2", "w1"}},
|
||||
{"unassigned", types.WorkFilter{Unassigned: true}, []string{"blocker", "r0", "r1", "r3", "r4", "w0", "w2"}},
|
||||
{"priority_0", types.WorkFilter{Priority: &zero}, []string{"r0", "r3"}},
|
||||
{"exclude_ids", types.WorkFilter{ExcludeIDs: []string{id("r1"), id("w2")}}, []string{"blocker", "r0", "r2", "r3", "r4", "w0", "w1"}},
|
||||
{"include_deferred", types.WorkFilter{IncludeDeferred: true}, append(slices.Clone(all), "dchild", "dparent", "wdchild")},
|
||||
{"parent_deferred", types.WorkFilter{ParentID: &parent, IncludeDeferred: true}, []string{"dchild", "wdchild"}},
|
||||
{"max_rows", types.WorkFilter{MaxRows: 50}, all},
|
||||
{"type_epic", types.WorkFilter{Type: string(types.TypeEpic), IncludeDeferred: true}, []string{"dparent"}},
|
||||
{"type_task_oldest", types.WorkFilter{Type: string(types.TypeTask), SortPolicy: types.SortPolicyOldest}, all},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
base := tc.filter
|
||||
base.IncludeEphemeral = true
|
||||
var unbounded []*types.IssueWithCounts
|
||||
withCountingReadTx(t, te, func(tx issueops.DBTX) error {
|
||||
var err error
|
||||
unbounded, err = issueops.GetReadyWorkWithCountsInTx(t.Context(), tx, base)
|
||||
return err
|
||||
})
|
||||
listed := readyIDs(unbounded)
|
||||
if got, want := sortedIDs(listed), prefixed(prefix, tc.want); strings.Join(got, ",") != strings.Join(want, ",") {
|
||||
t.Fatalf("ready set = %v, want %v", got, want)
|
||||
}
|
||||
n := len(listed)
|
||||
for _, limit := range []int{1, 2, n, n + 3} {
|
||||
filter := base
|
||||
filter.Limit = limit
|
||||
var page []*types.IssueWithCounts
|
||||
var total int
|
||||
withCountingReadTx(t, te, func(tx issueops.DBTX) error {
|
||||
var err error
|
||||
page, total, err = issueops.GetReadyWorkWithCountsAndTotalInTx(t.Context(), tx, filter)
|
||||
return err
|
||||
})
|
||||
if total != n {
|
||||
t.Errorf("limit=%d: total = %d, want %d", limit, total, n)
|
||||
}
|
||||
want := listed[:min(limit, n)]
|
||||
if got := readyIDs(page); strings.Join(got, ",") != strings.Join(want, ",") {
|
||||
t.Errorf("limit=%d: page = %v, want prefix %v", limit, got, want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -11,6 +11,7 @@ import (
|
|||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
"github.com/steveyegge/beads/internal/workapi/storereadycounter"
|
||||
publicops "github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
|
|
@ -184,6 +185,19 @@ func (s *Store) GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFil
|
|||
return s.inner.GetReadyWorkWithCounts(ctx, filter)
|
||||
}
|
||||
|
||||
// GetReadyWorkWithCountsAndTotal applies the same external exclusions as
|
||||
// GetReadyWorkWithCounts, so the page and its total describe one ready set.
|
||||
// It must be overridden here: the embedded passthrough would reach the inner
|
||||
// store without the exclusions.
|
||||
func (s *Store) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) {
|
||||
state, err := s.loadBlockingState(ctx)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
filter = withExternalExclusions(filter, state.refsByIssue)
|
||||
return s.inner.GetReadyWorkWithCountsAndTotal(ctx, filter)
|
||||
}
|
||||
|
||||
func withExternalExclusions(filter types.WorkFilter, refsByIssue map[string][]string) types.WorkFilter {
|
||||
filter.ExcludeIDs = slices.Clone(filter.ExcludeIDs)
|
||||
newIDs := make([]string, 0, len(refsByIssue))
|
||||
|
|
@ -216,6 +230,16 @@ func (s *Store) CountReadyWork(ctx context.Context, filter types.WorkFilter) (in
|
|||
return len(issues), nil
|
||||
}
|
||||
|
||||
// ReadyCounter sizes the ready set through CountReadyWork above, so the total
|
||||
// text-mode `bd ready` prints honors the same external exclusions as the page
|
||||
// and as the in-band total `bd ready --json` takes from
|
||||
// GetReadyWorkWithCountsAndTotal. It must be overridden here: the embedded
|
||||
// passthrough would hand back the inner store's counter, which counts
|
||||
// externally blocked issues as ready.
|
||||
func (s *Store) ReadyCounter() (publicops.ReadyCounter, error) {
|
||||
return storereadycounter.New(s)
|
||||
}
|
||||
|
||||
// ClaimReadyIssue resolves external blockers before using the existing local
|
||||
// compare-and-swap claim operation. Cross-project state cannot be atomic with
|
||||
// the local claim, but local claim ownership remains race-safe.
|
||||
|
|
|
|||
|
|
@ -68,6 +68,20 @@ func (f *fakeStore) GetReadyWorkWithCounts(ctx context.Context, filter types.Wor
|
|||
return result, nil
|
||||
}
|
||||
|
||||
func (f *fakeStore) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) {
|
||||
items, err := f.GetReadyWorkWithCounts(ctx, filter)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
unbounded := filter
|
||||
unbounded.Limit, unbounded.Offset = 0, 0
|
||||
all, err := f.GetReadyWork(ctx, unbounded)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
return items, len(all), nil
|
||||
}
|
||||
|
||||
func (f *fakeStore) GetBlockedIssues(_ context.Context, _ types.WorkFilter) ([]*types.BlockedIssue, error) {
|
||||
return slices.Clone(f.blocked), nil
|
||||
}
|
||||
|
|
@ -380,6 +394,64 @@ func TestCountReadyWorkUsesExternalPolicy(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestReadyPageTotalUsesExternalPolicy: the page `bd ready --json` lists and
|
||||
// the total it prints beside it must both exclude externally blocked work. The
|
||||
// decorator embeds the inner store, so without its own override this method
|
||||
// would pass straight through and count the excluded rows.
|
||||
func TestReadyPageTotalUsesExternalPolicy(t *testing.T) {
|
||||
a, b, c := issue("be-a"), issue("be-b"), issue("be-c")
|
||||
raw := &fakeStore{
|
||||
ready: []*types.Issue{a, b, c},
|
||||
deps: map[string][]*types.Dependency{
|
||||
a.ID: {externalDep(a.ID, "external:remote:payments", types.DepBlocks)},
|
||||
},
|
||||
}
|
||||
store := testStore(raw, &fakeStore{}, true)
|
||||
|
||||
items, total, err := store.GetReadyWorkWithCountsAndTotal(t.Context(), types.WorkFilter{Limit: 1})
|
||||
if err != nil {
|
||||
t.Fatalf("GetReadyWorkWithCountsAndTotal: %v", err)
|
||||
}
|
||||
if len(items) != 1 || items[0].ID != b.ID {
|
||||
t.Fatalf("page = %v, want [%s]", items, b.ID)
|
||||
}
|
||||
if total != 2 {
|
||||
t.Fatalf("total = %d, want 2 (be-a is externally blocked)", total)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadyTotalsAgreeAcrossOutputModes: text `bd ready` sizes a full page
|
||||
// through the ReadyCounter role, `bd ready --json` through the in-band total.
|
||||
// With an externally blocked issue present both must exclude it; before the
|
||||
// decorator overrode ReadyCounter, the text total passed through to the inner
|
||||
// store and counted it.
|
||||
func TestReadyTotalsAgreeAcrossOutputModes(t *testing.T) {
|
||||
a, b, c := issue("be-a"), issue("be-b"), issue("be-c")
|
||||
raw := &fakeStore{
|
||||
ready: []*types.Issue{a, b, c},
|
||||
deps: map[string][]*types.Dependency{
|
||||
a.ID: {externalDep(a.ID, "external:remote:payments", types.DepBlocks)},
|
||||
},
|
||||
}
|
||||
store := testStore(raw, &fakeStore{}, true)
|
||||
|
||||
counter, err := store.ReadyCounter()
|
||||
if err != nil {
|
||||
t.Fatalf("ReadyCounter: %v", err)
|
||||
}
|
||||
text, err := counter.CountReady(t.Context(), publicops.ReadyRequest{})
|
||||
if err != nil {
|
||||
t.Fatalf("CountReady: %v", err)
|
||||
}
|
||||
_, jsonTotal, err := store.GetReadyWorkWithCountsAndTotal(t.Context(), types.WorkFilter{Limit: 1})
|
||||
if err != nil {
|
||||
t.Fatalf("GetReadyWorkWithCountsAndTotal: %v", err)
|
||||
}
|
||||
if text.Total != 2 || int(text.Total) != jsonTotal {
|
||||
t.Fatalf("text total = %d, json total = %d; want both 2 (be-a is externally blocked)", text.Total, jsonTotal)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetBlockedIssuesAddsUnsatisfiedExternalRefs(t *testing.T) {
|
||||
a, c := issue("be-a"), issue("be-c")
|
||||
raw := &fakeStore{
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ package issueops
|
|||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
|
@ -40,6 +41,21 @@ type countsEntryPoint struct {
|
|||
wantTolerated string
|
||||
}
|
||||
|
||||
// readyProbeSQL matches the one-statement ready probe (probeReadyWorkInTx).
|
||||
const readyProbeSQL = `^SELECT EXISTS \(SELECT 1 FROM wisps\), EXISTS \(SELECT 1 FROM wisp_dependencies\)`
|
||||
|
||||
// readyProbeRows answers the ready probe: wisps present, wisp_dependencies
|
||||
// present, then any further flags the caller's probe asks for.
|
||||
func readyProbeRows(flags ...bool) *sqlmock.Rows {
|
||||
cols := make([]string, len(flags))
|
||||
vals := make([]driver.Value, len(flags))
|
||||
for i, f := range flags {
|
||||
cols[i] = fmt.Sprintf("f%d", i)
|
||||
vals[i] = f
|
||||
}
|
||||
return sqlmock.NewRows(cols).AddRow(vals...)
|
||||
}
|
||||
|
||||
func scalarCountRows(n int) *sqlmock.Rows {
|
||||
return sqlmock.NewRows([]string{"count"}).AddRow(n)
|
||||
}
|
||||
|
|
@ -160,11 +176,8 @@ var countsEntryPoints = []countsEntryPoint{
|
|||
name: "GetReadyWorkWithCountsInTx",
|
||||
missing: countsMegaQueryTables,
|
||||
prime: func(mock sqlmock.Sqlmock, wispErr error) {
|
||||
mock.ExpectQuery(`SELECT 1 FROM wisp_dependencies LIMIT 1`).
|
||||
WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1))
|
||||
mock.ExpectQuery(readyProbeSQL).WillReturnRows(readyProbeRows(true, true))
|
||||
mock.ExpectQuery(`(?s)FROM issues i`).WillReturnRows(emptyCountsRows())
|
||||
mock.ExpectQuery(`SELECT 1 FROM wisps LIMIT 1`).
|
||||
WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1))
|
||||
mock.ExpectQuery(`(?s)FROM wisps i`).WillReturnError(wispErr)
|
||||
},
|
||||
run: func(ctx context.Context, tx *sql.Tx) (string, error) {
|
||||
|
|
@ -177,12 +190,12 @@ var countsEntryPoints = []countsEntryPoint{
|
|||
name: "CountReadyWorkInTx",
|
||||
missing: []string{"wisp_labels"},
|
||||
prime: func(mock sqlmock.Sqlmock, wispErr error) {
|
||||
mock.ExpectQuery(`SELECT 1 FROM wisp_dependencies LIMIT 1`).
|
||||
WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1))
|
||||
// Both families are counted in one statement; when its wisp half
|
||||
// names a table the database may lack, the issues count is taken
|
||||
// on its own.
|
||||
mock.ExpectQuery(readyProbeSQL).WillReturnRows(readyProbeRows(true, true, false))
|
||||
mock.ExpectQuery(`SELECT \(SELECT COUNT\(\*\) FROM issues .*\), \(SELECT COUNT\(\*\) FROM wisps`).WillReturnError(wispErr)
|
||||
mock.ExpectQuery(`SELECT COUNT\(\*\) FROM issues`).WillReturnRows(scalarCountRows(1))
|
||||
mock.ExpectQuery(`SELECT 1 FROM wisps LIMIT 1`).
|
||||
WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1))
|
||||
mock.ExpectQuery(`SELECT COUNT\(\*\) FROM wisps`).WillReturnError(wispErr)
|
||||
},
|
||||
run: func(ctx context.Context, tx *sql.Tx) (string, error) {
|
||||
n, err := CountReadyWorkInTx(ctx, tx, readyFilter())
|
||||
|
|
@ -323,3 +336,63 @@ func TestCountsHealthyPlaneStillMerges(t *testing.T) {
|
|||
t.Fatalf("unmet sql expectations: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadyProbeFallsBackOnPreWispDatabase pins the one-statement ready probe's
|
||||
// escape hatch. A pre-migration database has no wisps table, so the combined
|
||||
// probe (which names it) fails as a whole; the ready read must then fall back
|
||||
// to the per-table probes — whose missing-table tolerance is the documented
|
||||
// contract — and answer issues-only, for the page, its in-band total and the
|
||||
// separate count alike.
|
||||
func TestReadyProbeFallsBackOnPreWispDatabase(t *testing.T) {
|
||||
primeFallback := func(mock sqlmock.Sqlmock) {
|
||||
mock.ExpectQuery(readyProbeSQL).WillReturnError(tableNotFound("wisps"))
|
||||
mock.ExpectQuery(`SELECT 1 FROM wisp_dependencies LIMIT 1`).WillReturnError(tableNotFound("wisp_dependencies"))
|
||||
mock.ExpectQuery(`SELECT 1 FROM wisps LIMIT 1`).WillReturnError(tableNotFound("wisps"))
|
||||
}
|
||||
|
||||
t.Run("page_and_total", func(t *testing.T) {
|
||||
_, mock, tx := beginMockTx(t)
|
||||
primeFallback(mock)
|
||||
mock.ExpectQuery(`^SELECT id, COUNT\(\*\) OVER \(\) FROM issues`).
|
||||
WillReturnRows(sqlmock.NewRows([]string{"id", "total"}))
|
||||
filter := readyFilter()
|
||||
filter.Limit = 1
|
||||
out, total, err := GetReadyWorkWithCountsAndTotalInTx(context.Background(), tx, filter)
|
||||
if err != nil {
|
||||
t.Fatalf("GetReadyWorkWithCountsAndTotalInTx on a pre-wisp database: %v", err)
|
||||
}
|
||||
if len(out) != 0 || total != 0 {
|
||||
t.Fatalf("got %d rows, total %d; want an empty issues-only answer", len(out), total)
|
||||
}
|
||||
if err := mock.ExpectationsWereMet(); err != nil {
|
||||
t.Fatalf("unmet sql expectations: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("count", func(t *testing.T) {
|
||||
_, mock, tx := beginMockTx(t)
|
||||
primeFallback(mock)
|
||||
mock.ExpectQuery(`^SELECT COUNT\(\*\) FROM issues WHERE`).WillReturnRows(scalarCountRows(4))
|
||||
n, err := CountReadyWorkInTx(context.Background(), tx, readyFilter())
|
||||
if err != nil {
|
||||
t.Fatalf("CountReadyWorkInTx on a pre-wisp database: %v", err)
|
||||
}
|
||||
if n != 4 {
|
||||
t.Fatalf("CountReadyWorkInTx = %d, want the issues-only 4", n)
|
||||
}
|
||||
if err := mock.ExpectationsWereMet(); err != nil {
|
||||
t.Fatalf("unmet sql expectations: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
// The fallback is for a MISSING table only; any other probe failure is
|
||||
// the caller's error, never a quietly issues-only answer.
|
||||
t.Run("unrelated_error_propagates", func(t *testing.T) {
|
||||
_, mock, tx := beginMockTx(t)
|
||||
boom := errors.New("connection refused")
|
||||
mock.ExpectQuery(readyProbeSQL).WillReturnError(boom)
|
||||
if _, _, err := GetReadyWorkWithCountsAndTotalInTx(context.Background(), tx, readyFilter()); !errors.Is(err, boom) {
|
||||
t.Fatalf("probe failure not propagated: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -64,7 +64,15 @@ func buildReadyWorkPredicates(ctx context.Context, tx DBTX, filter types.WorkFil
|
|||
}
|
||||
inputs.ParentDescendantIDs = descendantIDs
|
||||
}
|
||||
return buildReadyWorkPredicatesFrom(filter, tables, inputs)
|
||||
}
|
||||
|
||||
// buildReadyWorkPredicatesFrom renders the ready-work predicates for one table
|
||||
// family from ID sets the caller already resolved. It issues no SQL, which is
|
||||
// what lets a caller that renders BOTH families (and the total) resolve the
|
||||
// deferred-parent children and parent descendants once per transaction
|
||||
// instead of once per family.
|
||||
func buildReadyWorkPredicatesFrom(filter types.WorkFilter, tables FilterTables, inputs sqlbuild.ReadyWorkWhereInputs) (*readyWorkPredicates, error) {
|
||||
whereSQL, whereArgs, err := sqlbuild.BuildReadyWorkWhere(filter, tables, inputs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -506,6 +514,60 @@ func queryReadyIssueIDPage(ctx context.Context, tx DBTX, query string, args []in
|
|||
return issueIDs, nil
|
||||
}
|
||||
|
||||
var (
|
||||
deferredChildDepTables = []string{"dependencies", "wisp_dependencies"}
|
||||
deferredChildIssueTables = []string{"issues", "wisps"}
|
||||
)
|
||||
|
||||
// deferredChildrenQuery selects the children, recorded in depTable, of parents
|
||||
// in issueTable whose defer_until is still in the future.
|
||||
func deferredChildrenQuery(depTable, issueTable string) string {
|
||||
targetCol := "depends_on_issue_id"
|
||||
if issueTable == "wisps" {
|
||||
targetCol = "depends_on_wisp_id"
|
||||
}
|
||||
return fmt.Sprintf(`
|
||||
SELECT dep.issue_id
|
||||
FROM %s dep
|
||||
JOIN %s parent ON parent.id = dep.%s
|
||||
WHERE dep.type = 'parent-child'
|
||||
AND parent.defer_until IS NOT NULL
|
||||
AND parent.defer_until > UTC_TIMESTAMP()
|
||||
`, depTable, issueTable, targetCol)
|
||||
}
|
||||
|
||||
// getDeferredChildrenAllTablesInTx is getChildrenOfDeferredParentsInTx's
|
||||
// child scan for a database known to have the whole wisp plane: the four
|
||||
// dependency-table x parent-table legs go out as ONE statement (UNION ALL)
|
||||
// instead of four. Callers must already know every table exists — the
|
||||
// per-leg missing-table tolerance of the loop form cannot apply to a single
|
||||
// statement.
|
||||
func getDeferredChildrenAllTablesInTx(ctx context.Context, tx DBTX) ([]string, error) {
|
||||
legs := make([]string, 0, len(deferredChildDepTables)*len(deferredChildIssueTables))
|
||||
for _, depTable := range deferredChildDepTables {
|
||||
for _, issueTable := range deferredChildIssueTables {
|
||||
legs = append(legs, deferredChildrenQuery(depTable, issueTable))
|
||||
}
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx, strings.Join(legs, " UNION ALL "))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("deferred parents: get deferred children: %w", err)
|
||||
}
|
||||
defer func() { _ = rows.Close() }()
|
||||
var childIDs []string
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
return nil, fmt.Errorf("deferred parents: scan deferred child: %w", err)
|
||||
}
|
||||
childIDs = append(childIDs, id)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("deferred parents: child rows: %w", err)
|
||||
}
|
||||
return childIDs, nil
|
||||
}
|
||||
|
||||
// getChildrenOfDeferredParentsInTx returns IDs of issues whose parent has a
|
||||
// future defer_until. Works within an existing transaction.
|
||||
//
|
||||
|
|
@ -538,20 +600,9 @@ func getChildrenOfDeferredParentsInTx(ctx context.Context, tx DBTX) ([]string, e
|
|||
}
|
||||
|
||||
var childIDs []string
|
||||
for _, depTable := range []string{"dependencies", "wisp_dependencies"} {
|
||||
for _, issueTable := range []string{"issues", "wisps"} {
|
||||
targetCol := "depends_on_issue_id"
|
||||
if issueTable == "wisps" {
|
||||
targetCol = "depends_on_wisp_id"
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx, fmt.Sprintf(`
|
||||
SELECT dep.issue_id
|
||||
FROM %s dep
|
||||
JOIN %s parent ON parent.id = dep.%s
|
||||
WHERE dep.type = 'parent-child'
|
||||
AND parent.defer_until IS NOT NULL
|
||||
AND parent.defer_until > UTC_TIMESTAMP()
|
||||
`, depTable, issueTable, targetCol))
|
||||
for _, depTable := range deferredChildDepTables {
|
||||
for _, issueTable := range deferredChildIssueTables {
|
||||
rows, err := tx.QueryContext(ctx, deferredChildrenQuery(depTable, issueTable))
|
||||
if err != nil {
|
||||
// This FROM names two tables at once, so the gate has to key on
|
||||
// which one the driver reports missing. The wisp plane is the
|
||||
|
|
|
|||
|
|
@ -27,45 +27,94 @@ func readyHydrationFor(filter types.WorkFilter) sqlbuild.CountsHydration {
|
|||
return sqlbuild.CountsHydration{Lite: filter.Lite}
|
||||
}
|
||||
|
||||
func GetReadyWorkWithCountsInTx(ctx context.Context, tx *sql.Tx, filter types.WorkFilter) ([]*types.IssueWithCounts, error) {
|
||||
wispDepsExist, err := optionalTableExistsInTx(ctx, tx, "wisp_dependencies")
|
||||
// GetReadyWorkWithCountsInTx returns the ready-work page for filter, hydrated
|
||||
// with counts. It is GetReadyWorkWithCountsAndTotalInTx without the total.
|
||||
func GetReadyWorkWithCountsInTx(ctx context.Context, tx DBTX, filter types.WorkFilter) ([]*types.IssueWithCounts, error) {
|
||||
items, _, err := getReadyWorkWithCountsInTx(ctx, tx, filter, false)
|
||||
return items, err
|
||||
}
|
||||
|
||||
// GetReadyWorkWithCountsAndTotalInTx returns the ready-work page for filter
|
||||
// together with the size of the whole ready set the page was cut from —
|
||||
// identical to CountReadyWorkInTx(filter), and so to
|
||||
// len(GetReadyWorkWithCountsInTx(filter with Limit=0)) — resolved inside the
|
||||
// SAME statements that select the page.
|
||||
//
|
||||
// It exists because `bd ready --limit N` used to answer its "Showing N of M"
|
||||
// with a second, separate pass (CountReadyWorkInTx in a new transaction,
|
||||
// behind a second defer-wake sweep) that re-ran every probe and predicate the
|
||||
// page had just run. Against a remote SQL server each statement is a
|
||||
// sequential round trip, so asking for ONE row took longer than asking for
|
||||
// all of them. Here the total rides the page's ID query as a window count
|
||||
// (COUNT(*) OVER ()), evaluated over the full predicate before LIMIT, so a
|
||||
// capped page costs no extra statement for its total. For an unbounded page
|
||||
// (Limit <= 0) the total is simply the page's length.
|
||||
//
|
||||
// The two families are summed and their overlap (an ID ready as both an issue
|
||||
// and a wisp, which the merge dedupes wisp-wins) subtracted, exactly as
|
||||
// CountReadyWorkInTx does; the overlap statement runs only when the probe saw
|
||||
// an ID present in both tables.
|
||||
func GetReadyWorkWithCountsAndTotalInTx(ctx context.Context, tx DBTX, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) {
|
||||
return getReadyWorkWithCountsInTx(ctx, tx, filter, true)
|
||||
}
|
||||
|
||||
func getReadyWorkWithCountsInTx(ctx context.Context, tx DBTX, filter types.WorkFilter, wantTotal bool) ([]*types.IssueWithCounts, int, error) {
|
||||
// A capped page needs the collision fact to size the merged set; an
|
||||
// unbounded one is its own total.
|
||||
sized := wantTotal && filter.Limit > 0
|
||||
probe, err := probeReadyWorkInTx(ctx, tx, filter, sized)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("get ready work with counts: wisp dependency probe: %w", err)
|
||||
return nil, 0, fmt.Errorf("get ready work with counts: %w", err)
|
||||
}
|
||||
|
||||
issuePreds, err := buildReadyWorkPredicates(ctx, tx, filter, IssuesFilterTables)
|
||||
issuePreds, err := buildReadyWorkPredicatesFrom(filter, IssuesFilterTables, probe.inputs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return nil, 0, err
|
||||
}
|
||||
out, err := runReadyCountsInTx(ctx, tx, IssuesFilterTables, filter.Limit, issuePreds, wispDepsExist, readyHydrationFor(filter))
|
||||
out, issueTotal, err := runReadyCountsInTx(ctx, tx, IssuesFilterTables, filter.Limit, issuePreds, probe.wispDepsExist, readyHydrationFor(filter), sized)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return nil, 0, err
|
||||
}
|
||||
|
||||
empty, probeErr := wispsTableEmptyOrMissingInTx(ctx, tx)
|
||||
if probeErr != nil {
|
||||
return nil, fmt.Errorf("get ready work with counts: wisp probe: %w", probeErr)
|
||||
}
|
||||
if empty {
|
||||
return finishReadyWorkWithCounts(out, filter)
|
||||
}
|
||||
if !wispDepsExist {
|
||||
return finishReadyWorkWithCounts(out, filter)
|
||||
finish := func(items []*types.IssueWithCounts, total int) ([]*types.IssueWithCounts, int, error) {
|
||||
if !sized {
|
||||
// Unbounded: every ready row is on the page (the MaxRows cap
|
||||
// refuses rather than truncates), so the page IS the set.
|
||||
total = len(items)
|
||||
}
|
||||
items, err := finishReadyWorkWithCounts(items, filter)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
return items, total, nil
|
||||
}
|
||||
|
||||
wispPreds, err := buildReadyWorkPredicates(ctx, tx, filter, WispsFilterTables)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
if !probe.readsWisps() {
|
||||
return finish(out, issueTotal)
|
||||
}
|
||||
wisps, err := runReadyCountsInTx(ctx, tx, WispsFilterTables, filter.Limit, wispPreds, true, readyHydrationFor(filter))
|
||||
|
||||
wispPreds, err := buildReadyWorkPredicatesFrom(filter, WispsFilterTables, probe.inputs)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
wisps, wispTotal, err := runReadyCountsInTx(ctx, tx, WispsFilterTables, filter.Limit, wispPreds, true, readyHydrationFor(filter), sized)
|
||||
if err != nil {
|
||||
if missingOptionalWispTable(err) {
|
||||
return finishReadyWorkWithCounts(out, filter)
|
||||
return finish(out, issueTotal)
|
||||
}
|
||||
return nil, err
|
||||
return nil, 0, err
|
||||
}
|
||||
if len(wisps) == 0 {
|
||||
return finishReadyWorkWithCounts(out, filter)
|
||||
return finish(out, issueTotal)
|
||||
}
|
||||
|
||||
total := issueTotal + wispTotal
|
||||
if sized && probe.idCollision && issueTotal > 0 {
|
||||
overlap, err := countReadyOverlapInTx(ctx, tx, issuePreds, wispPreds)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("get ready work with counts: overlap: %w", err)
|
||||
}
|
||||
total -= overlap
|
||||
}
|
||||
|
||||
// Prefer the canonical wisp record when an ID exists in both tables (be-iabdi).
|
||||
|
|
@ -87,7 +136,7 @@ func GetReadyWorkWithCountsInTx(ctx context.Context, tx *sql.Tx, filter types.Wo
|
|||
}
|
||||
kept = append(kept, wisps...)
|
||||
sortIssuesWithCountsByPolicy(kept, filter.SortPolicy)
|
||||
return finishReadyWorkWithCounts(kept, filter)
|
||||
return finish(kept, total)
|
||||
}
|
||||
|
||||
// finishReadyWorkWithCounts is the terminal hook every
|
||||
|
|
@ -144,6 +193,14 @@ func finishReadyWorkWithCounts(items []*types.IssueWithCounts, filter types.Work
|
|||
// For limit <= 0 (unbounded) there is no page to push down, so it runs the
|
||||
// predicate-form mega-query unchanged.
|
||||
//
|
||||
// withTotal (bounded pages only) also returns the number of rows the family's
|
||||
// ready predicate admits, as a window count over the same ID query:
|
||||
// COUNT(*) OVER () is evaluated over every row the WHERE admits, before ORDER
|
||||
// BY … LIMIT cut the page, so it is exactly `SELECT COUNT(*) … WHERE` — the
|
||||
// count CountReadyWorkInTx takes — at no extra statement. The ID query has no
|
||||
// DISTINCT, so the window counts rows and IDs alike. Otherwise the returned
|
||||
// total is len(result).
|
||||
//
|
||||
// Both callers pass readyHydrationFor(filter), which carries Lite and nothing
|
||||
// else: ready work always hydrates labels and cardinalities, because
|
||||
// types.WorkFilter carries neither opt-out — the projection that builds it
|
||||
|
|
@ -151,18 +208,28 @@ func finishReadyWorkWithCounts(items []*types.IssueWithCounts, filter types.Work
|
|||
// SkipLabels and SkipCounts are not carried onto the ReadyFlag arm.
|
||||
//
|
||||
//nolint:gosec // G201: whereSQL/orderBySQL/limitSQL are hardcoded fragments; user input rides ? placeholders.
|
||||
func runReadyCountsInTx(ctx context.Context, tx *sql.Tx, tables FilterTables, limit int, preds *readyWorkPredicates, includeWispReverseDeps bool, hyd sqlbuild.CountsHydration) ([]*types.IssueWithCounts, error) {
|
||||
func runReadyCountsInTx(ctx context.Context, tx DBTX, tables FilterTables, limit int, preds *readyWorkPredicates, includeWispReverseDeps bool, hyd sqlbuild.CountsHydration, withTotal bool) ([]*types.IssueWithCounts, int, error) {
|
||||
if limit <= 0 {
|
||||
return runSearchQueryInTx(ctx, tx, tables, preds.whereSQL, preds.orderBySQL, preds.limitSQL, preds.args, includeWispReverseDeps, hyd)
|
||||
out, err := runSearchQueryInTx(ctx, tx, tables, preds.whereSQL, preds.orderBySQL, preds.limitSQL, preds.args, includeWispReverseDeps, hyd)
|
||||
return out, len(out), err
|
||||
}
|
||||
|
||||
idQuery := fmt.Sprintf("SELECT id FROM %s %s %s %s", tables.Main, preds.whereSQL, preds.orderBySQL, preds.limitSQL)
|
||||
pageIDs, err := queryReadyIssueIDPage(ctx, tx, idQuery, preds.args)
|
||||
var pageIDs []string
|
||||
var total int
|
||||
var err error
|
||||
if withTotal {
|
||||
idQuery := fmt.Sprintf("SELECT id, COUNT(*) OVER () FROM %s %s %s %s", tables.Main, preds.whereSQL, preds.orderBySQL, preds.limitSQL)
|
||||
pageIDs, total, err = queryReadyIDPageWithTotal(ctx, tx, idQuery, preds.args)
|
||||
} else {
|
||||
idQuery := fmt.Sprintf("SELECT id FROM %s %s %s %s", tables.Main, preds.whereSQL, preds.orderBySQL, preds.limitSQL)
|
||||
pageIDs, err = queryReadyIssueIDPage(ctx, tx, idQuery, preds.args)
|
||||
total = len(pageIDs)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return nil, 0, err
|
||||
}
|
||||
if len(pageIDs) == 0 {
|
||||
return nil, nil
|
||||
return nil, 0, nil
|
||||
}
|
||||
|
||||
// Hydrate the counts for the resolved page, chunking the IN-list. The page
|
||||
|
|
@ -176,7 +243,7 @@ func runReadyCountsInTx(ctx context.Context, tx *sql.Tx, tables FilterTables, li
|
|||
countsSQL, idArgs := sqlbuild.SearchCountsSQL(tables, pageIDs[start:end], "", "", "", includeWispReverseDeps, hyd)
|
||||
rows, scanErr := scanCountsRowsInTx(ctx, tx, tables.Main, countsSQL, idArgs, hyd)
|
||||
if scanErr != nil {
|
||||
return nil, scanErr
|
||||
return nil, 0, scanErr
|
||||
}
|
||||
for _, r := range rows {
|
||||
if r != nil && r.Issue != nil {
|
||||
|
|
@ -193,60 +260,92 @@ func runReadyCountsInTx(ctx context.Context, tx *sql.Tx, tables FilterTables, li
|
|||
ordered = append(ordered, r)
|
||||
}
|
||||
}
|
||||
return ordered, nil
|
||||
return ordered, total, nil
|
||||
}
|
||||
|
||||
// queryReadyIDPageWithTotal runs a `SELECT id, COUNT(*) OVER () …` page query
|
||||
// and returns the page IDs with the window total. An empty page carries no
|
||||
// row to read the total from, and means the predicate admitted nothing.
|
||||
func queryReadyIDPageWithTotal(ctx context.Context, tx DBTX, query string, args []interface{}) ([]string, int, error) {
|
||||
rows, err := tx.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, 0, fmt.Errorf("failed to get ready work: %w", err)
|
||||
}
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var ids []string
|
||||
var total int64
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id, &total); err != nil {
|
||||
return nil, 0, fmt.Errorf("get ready work: scan id: %w", err)
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, 0, fmt.Errorf("get ready work: rows: %w", err)
|
||||
}
|
||||
return ids, int(total), nil
|
||||
}
|
||||
|
||||
// CountReadyWorkInTx returns the number of ready-work items — identical to
|
||||
// len(GetReadyWorkWithCountsInTx(filter with Limit=0)) — without materializing
|
||||
// the counts mega-query. The ready set is a union of the issues and wisps that
|
||||
// match the ready predicate, so it sizes each family with a single indexed
|
||||
// COUNT(*) over that predicate and subtracts the overlap (IDs present in both
|
||||
// ready sets, which GetReadyWorkWithCountsInTx dedupes wisp-wins). It never
|
||||
// re-runs the mega-query, so a single wisp no longer disables the fast path.
|
||||
// This backs the "Showing X of N" total `bd ready` prints when the page is
|
||||
// capped.
|
||||
func CountReadyWorkInTx(ctx context.Context, tx *sql.Tx, filter types.WorkFilter) (int, error) {
|
||||
// match the ready predicate, so it sizes each family with an indexed COUNT(*)
|
||||
// over that predicate and subtracts the overlap (IDs present in both ready
|
||||
// sets, which GetReadyWorkWithCountsInTx dedupes wisp-wins). It never re-runs
|
||||
// the mega-query, so a single wisp no longer disables the fast path.
|
||||
//
|
||||
// Statement budget: the shared probe (one statement; plus one for the
|
||||
// deferred-parent children when a future-deferred parent exists), then both
|
||||
// family counts in ONE statement of scalar subqueries, then the overlap only
|
||||
// when the probe saw an ID present in both tables. `bd ready` itself sizes a
|
||||
// capped --json page in-band (GetReadyWorkWithCountsAndTotalInTx); this backs
|
||||
// the ReadyCounter role and the human-readable "Showing X of N".
|
||||
func CountReadyWorkInTx(ctx context.Context, tx DBTX, filter types.WorkFilter) (int, error) {
|
||||
countFilter := filter
|
||||
countFilter.Limit = 0
|
||||
|
||||
wispDepsExist, err := optionalTableExistsInTx(ctx, tx, "wisp_dependencies")
|
||||
probe, err := probeReadyWorkInTx(ctx, tx, countFilter, true)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("count ready work: wisp dependency probe: %w", err)
|
||||
return 0, fmt.Errorf("count ready work: %w", err)
|
||||
}
|
||||
|
||||
issuePreds, err := buildReadyWorkPredicates(ctx, tx, countFilter, IssuesFilterTables)
|
||||
issuePreds, err := buildReadyWorkPredicatesFrom(countFilter, IssuesFilterTables, probe.inputs)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
issueCount, err := countReadyPredicateInTx(ctx, tx, "issues", issuePreds.whereSQL, issuePreds.whereArgs)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("count ready work: issues: %w", err)
|
||||
}
|
||||
|
||||
// Mirror GetReadyWorkWithCountsInTx's wisp gating: an empty/missing wisps
|
||||
// table or absent wisp_dependencies means the ready set is issues-only.
|
||||
empty, err := wispsTableEmptyOrMissingInTx(ctx, tx)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("count ready work: wisp probe: %w", err)
|
||||
}
|
||||
if empty || !wispDepsExist {
|
||||
if !probe.readsWisps() {
|
||||
issueCount, err := countReadyPredicateInTx(ctx, tx, "issues", issuePreds.whereSQL, issuePreds.whereArgs)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("count ready work: issues: %w", err)
|
||||
}
|
||||
return issueCount, nil
|
||||
}
|
||||
|
||||
wispPreds, err := buildReadyWorkPredicates(ctx, tx, countFilter, WispsFilterTables)
|
||||
wispPreds, err := buildReadyWorkPredicatesFrom(countFilter, WispsFilterTables, probe.inputs)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
wispCount, err := countReadyPredicateInTx(ctx, tx, "wisps", wispPreds.whereSQL, wispPreds.whereArgs)
|
||||
issueCount, wispCount, err := countReadyFamiliesInTx(ctx, tx, issuePreds, wispPreds)
|
||||
if err != nil {
|
||||
if missingOptionalWispTable(err) {
|
||||
return issueCount, nil
|
||||
if !missingOptionalWispTable(err) {
|
||||
return 0, fmt.Errorf("count ready work: %w", err)
|
||||
}
|
||||
// A wisp plane the database may legitimately lack: issues-only, and
|
||||
// the issues count has to be taken on its own.
|
||||
issueCount, err = countReadyPredicateInTx(ctx, tx, "issues", issuePreds.whereSQL, issuePreds.whereArgs)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("count ready work: issues: %w", err)
|
||||
}
|
||||
return 0, fmt.Errorf("count ready work: wisps: %w", err)
|
||||
}
|
||||
if wispCount == 0 {
|
||||
return issueCount, nil
|
||||
}
|
||||
if wispCount == 0 || issueCount == 0 || !probe.idCollision {
|
||||
return issueCount + wispCount, nil
|
||||
}
|
||||
|
||||
overlap, err := countReadyOverlapInTx(ctx, tx, issuePreds, wispPreds)
|
||||
if err != nil {
|
||||
|
|
@ -255,12 +354,29 @@ func CountReadyWorkInTx(ctx context.Context, tx *sql.Tx, filter types.WorkFilter
|
|||
return issueCount + wispCount - overlap, nil
|
||||
}
|
||||
|
||||
// countReadyFamiliesInTx counts both families' ready rows in one statement of
|
||||
// two scalar subqueries (portable: no FROM-less dialect extension beyond
|
||||
// SELECT of subqueries, which every supported backend accepts).
|
||||
//
|
||||
//nolint:gosec // G201: whereSQL fragments are hardcoded; user input rides ? placeholders.
|
||||
func countReadyFamiliesInTx(ctx context.Context, tx DBTX, issuePreds, wispPreds *readyWorkPredicates) (int, int, error) {
|
||||
q := fmt.Sprintf("SELECT (SELECT COUNT(*) FROM issues %s), (SELECT COUNT(*) FROM wisps %s)", issuePreds.whereSQL, wispPreds.whereSQL)
|
||||
args := make([]interface{}, 0, len(issuePreds.whereArgs)+len(wispPreds.whereArgs))
|
||||
args = append(args, issuePreds.whereArgs...)
|
||||
args = append(args, wispPreds.whereArgs...)
|
||||
var issues, wisps int
|
||||
if err := tx.QueryRowContext(ctx, q, args...).Scan(&issues, &wisps); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
return issues, wisps, nil
|
||||
}
|
||||
|
||||
// countReadyPredicateInTx counts the rows in one table family that match the
|
||||
// ready predicate. whereSQL already begins with "WHERE " and whereArgs binds
|
||||
// only its placeholders (no ORDER BY params).
|
||||
//
|
||||
//nolint:gosec // G201: whereSQL is hardcoded fragments; user input rides ? placeholders.
|
||||
func countReadyPredicateInTx(ctx context.Context, tx *sql.Tx, table, whereSQL string, whereArgs []interface{}) (int, error) {
|
||||
func countReadyPredicateInTx(ctx context.Context, tx DBTX, table, whereSQL string, whereArgs []interface{}) (int, error) {
|
||||
var n int
|
||||
if err := tx.QueryRowContext(ctx, fmt.Sprintf("SELECT COUNT(*) FROM %s %s", table, whereSQL), whereArgs...).Scan(&n); err != nil {
|
||||
return 0, err
|
||||
|
|
@ -273,7 +389,7 @@ func countReadyPredicateInTx(ctx context.Context, tx *sql.Tx, table, whereSQL st
|
|||
// the issue row for such an ID, so |ready| = issueCount + wispCount - overlap.
|
||||
//
|
||||
//nolint:gosec // G201: whereSQL fragments are hardcoded; user input rides ? placeholders.
|
||||
func countReadyOverlapInTx(ctx context.Context, tx *sql.Tx, issuePreds, wispPreds *readyWorkPredicates) (int, error) {
|
||||
func countReadyOverlapInTx(ctx context.Context, tx DBTX, issuePreds, wispPreds *readyWorkPredicates) (int, error) {
|
||||
q := fmt.Sprintf("SELECT COUNT(*) FROM issues %s AND id IN (SELECT id FROM wisps %s)", issuePreds.whereSQL, wispPreds.whereSQL)
|
||||
args := make([]interface{}, 0, len(issuePreds.whereArgs)+len(wispPreds.whereArgs))
|
||||
args = append(args, issuePreds.whereArgs...)
|
||||
|
|
|
|||
156
internal/storage/issueops/ready_work_probe.go
Normal file
156
internal/storage/issueops/ready_work_probe.go
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
package issueops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/sqlbuild"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// readyWorkProbe holds the per-transaction facts every leg of a ready read
|
||||
// consults: which optional wisp-plane tables exist and hold rows, and the ID
|
||||
// sets the ready predicate is rendered from. A ready read renders the
|
||||
// predicate for two table families (issues, wisps) and — for a capped page —
|
||||
// sizes the whole set as well; each of those used to re-derive these facts
|
||||
// with its own statements. Against a remote SQL server every statement is a
|
||||
// sequential round trip, so the facts are resolved ONCE per transaction here
|
||||
// and shared by every leg.
|
||||
type readyWorkProbe struct {
|
||||
// wispsPresent: the wisps table exists and holds at least one row. False
|
||||
// means the ready set is issues-only (wispsTableEmptyOrMissingInTx).
|
||||
wispsPresent bool
|
||||
// wispDepsExist: the wisp_dependencies table exists
|
||||
// (optionalTableExistsInTx). The counts mega-query joins it, so the wisp
|
||||
// family is only read when it does.
|
||||
wispDepsExist bool
|
||||
// idCollision: some ID exists in both issues and wisps. Only then can the
|
||||
// two ready sets overlap, so only then does sizing the merged set need the
|
||||
// overlap count. Probed only when the caller sizes the set; conservatively
|
||||
// true whenever it was not probed exactly.
|
||||
idCollision bool
|
||||
// inputs are the ID sets the ready WHERE clause is rendered from
|
||||
// (children of future-deferred parents, descendants of filter.ParentID),
|
||||
// shared by both families.
|
||||
inputs sqlbuild.ReadyWorkWhereInputs
|
||||
}
|
||||
|
||||
const futureDeferredPredicate = "defer_until IS NOT NULL AND defer_until > UTC_TIMESTAMP()"
|
||||
|
||||
// probeReadyWorkInTx resolves a readyWorkProbe for filter.
|
||||
//
|
||||
// The existence facts are answered by ONE statement of EXISTS subqueries,
|
||||
// which is portable across every SQL dialect the shared issueops code runs on
|
||||
// (no window functions, no FROM-less dialect quirks beyond a SELECT of
|
||||
// scalar subqueries). EXISTS rather than COUNT so each leg stops at its first
|
||||
// row.
|
||||
//
|
||||
// A pre-migration database may legitimately lack the wisp plane, and a single
|
||||
// statement cannot tolerate one missing table while reading the rest, so a
|
||||
// missing-table failure falls back to the per-table probes — the exact
|
||||
// statements, tolerances and error classification these facts were always
|
||||
// resolved with.
|
||||
//
|
||||
// wantCollision asks for the issues/wisps ID-collision fact, which only a
|
||||
// caller sizing the merged ready set needs.
|
||||
func probeReadyWorkInTx(ctx context.Context, tx DBTX, filter types.WorkFilter, wantCollision bool) (*readyWorkProbe, error) {
|
||||
p := &readyWorkProbe{}
|
||||
excludeDeferred := !filter.IncludeDeferred
|
||||
|
||||
cols := []string{
|
||||
"EXISTS (SELECT 1 FROM wisps)",
|
||||
// Only the table REFERENCE matters here: the statement fails with a
|
||||
// missing-table error when wisp_dependencies is absent, and an empty
|
||||
// table still exists (optionalTableExistsInTx's contract). The value
|
||||
// is not read.
|
||||
"EXISTS (SELECT 1 FROM wisp_dependencies)",
|
||||
}
|
||||
if excludeDeferred {
|
||||
cols = append(cols,
|
||||
"EXISTS (SELECT 1 FROM issues WHERE "+futureDeferredPredicate+")",
|
||||
"EXISTS (SELECT 1 FROM wisps WHERE "+futureDeferredPredicate+")")
|
||||
}
|
||||
if wantCollision {
|
||||
cols = append(cols, "EXISTS (SELECT 1 FROM issues JOIN wisps ON wisps.id = issues.id)")
|
||||
}
|
||||
flags := make([]bool, len(cols))
|
||||
dest := make([]any, len(cols))
|
||||
for i := range flags {
|
||||
dest[i] = &flags[i]
|
||||
}
|
||||
|
||||
hasDeferredParent := false
|
||||
err := tx.QueryRowContext(ctx, "SELECT "+strings.Join(cols, ", ")).Scan(dest...)
|
||||
switch {
|
||||
case err == nil:
|
||||
p.wispsPresent = flags[0]
|
||||
// The statement ran, so every table it names exists.
|
||||
p.wispDepsExist = true
|
||||
next := 2
|
||||
if excludeDeferred {
|
||||
hasDeferredParent = flags[2] || flags[3]
|
||||
next = 4
|
||||
}
|
||||
// Without a wisp row there is nothing to collide with.
|
||||
p.idCollision = wantCollision && p.wispsPresent && flags[next]
|
||||
if hasDeferredParent {
|
||||
// Every table the child scan reads is known to exist, so its
|
||||
// four legs can go out as one statement.
|
||||
children, dcErr := getDeferredChildrenAllTablesInTx(ctx, tx)
|
||||
if dcErr != nil {
|
||||
return nil, fmt.Errorf("compute deferred parent children: %w", dcErr)
|
||||
}
|
||||
p.inputs.DeferredChildIDs = children
|
||||
}
|
||||
case isTableNotExistError(err):
|
||||
if err := p.probeTableByTableInTx(ctx, tx, excludeDeferred); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Not probed exactly; assume the worst so a total is never
|
||||
// under-corrected.
|
||||
p.idCollision = wantCollision && p.wispsPresent
|
||||
default:
|
||||
return nil, fmt.Errorf("ready probe: %w", err)
|
||||
}
|
||||
|
||||
// Parent filtering: all transitive descendants of parentID (GH#3396).
|
||||
if filter.ParentID != nil {
|
||||
descendantIDs, descErr := GetDescendantIDsInTx(ctx, tx, *filter.ParentID, 0)
|
||||
if descErr != nil {
|
||||
return nil, fmt.Errorf("get parent descendants: %w", descErr)
|
||||
}
|
||||
p.inputs.ParentDescendantIDs = descendantIDs
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// probeTableByTableInTx is the per-table fallback for a database missing part
|
||||
// of the wisp plane.
|
||||
func (p *readyWorkProbe) probeTableByTableInTx(ctx context.Context, tx DBTX, excludeDeferred bool) error {
|
||||
wispDepsExist, err := optionalTableExistsInTx(ctx, tx, "wisp_dependencies")
|
||||
if err != nil {
|
||||
return fmt.Errorf("wisp dependency probe: %w", err)
|
||||
}
|
||||
p.wispDepsExist = wispDepsExist
|
||||
empty, err := wispsTableEmptyOrMissingInTx(ctx, tx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("wisp probe: %w", err)
|
||||
}
|
||||
p.wispsPresent = !empty
|
||||
if excludeDeferred {
|
||||
children, err := getChildrenOfDeferredParentsInTx(ctx, tx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("compute deferred parent children: %w", err)
|
||||
}
|
||||
p.inputs.DeferredChildIDs = children
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// readsWisps reports whether the wisp family takes part in the ready set:
|
||||
// GetReadyWorkWithCountsInTx has always read it only when the wisps table has
|
||||
// rows and wisp_dependencies exists.
|
||||
func (p *readyWorkProbe) readsWisps() bool {
|
||||
return p.wispsPresent && p.wispDepsExist
|
||||
}
|
||||
|
|
@ -142,7 +142,7 @@ func runFilterSearchQueryInTx(ctx context.Context, tx *sql.Tx, query string, fil
|
|||
}
|
||||
|
||||
//nolint:gosec // G201: SQL fragments are caller-built from hardcoded shapes
|
||||
func runSearchQueryInTx(ctx context.Context, tx *sql.Tx, tables FilterTables, whereSQL, orderBySQL, limitSQL string, args []interface{}, includeWispReverseDeps bool, hyd sqlbuild.CountsHydration) ([]*types.IssueWithCounts, error) {
|
||||
func runSearchQueryInTx(ctx context.Context, tx DBTX, tables FilterTables, whereSQL, orderBySQL, limitSQL string, args []interface{}, includeWispReverseDeps bool, hyd sqlbuild.CountsHydration) ([]*types.IssueWithCounts, error) {
|
||||
searchSQL, _ := sqlbuild.SearchCountsSQL(tables, nil, whereSQL, orderBySQL, limitSQL, includeWispReverseDeps, hyd)
|
||||
return scanCountsRowsInTx(ctx, tx, tables.Main, searchSQL, args, hyd)
|
||||
}
|
||||
|
|
@ -153,7 +153,7 @@ func runSearchQueryInTx(ctx context.Context, tx *sql.Tx, tables FilterTables, wh
|
|||
// ready-counts path.
|
||||
//
|
||||
//nolint:gosec // G201: query is builder-produced; user input rides ? placeholders.
|
||||
func scanCountsRowsInTx(ctx context.Context, tx *sql.Tx, mainTable, query string, args []interface{}, hyd sqlbuild.CountsHydration) ([]*types.IssueWithCounts, error) {
|
||||
func scanCountsRowsInTx(ctx context.Context, tx DBTX, mainTable, query string, args []interface{}, hyd sqlbuild.CountsHydration) ([]*types.IssueWithCounts, error) {
|
||||
rows, err := tx.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("search count %s: %w", mainTable, err)
|
||||
|
|
|
|||
|
|
@ -38,7 +38,7 @@ func wispsTableEmptyOrMissingInTx(ctx context.Context, tx DBTX) (bool, error) {
|
|||
}
|
||||
|
||||
//nolint:gosec // table is selected by callers from fixed optional wisp tables.
|
||||
func optionalTableExistsInTx(ctx context.Context, tx *sql.Tx, table string) (bool, error) {
|
||||
func optionalTableExistsInTx(ctx context.Context, tx DBTX, table string) (bool, error) {
|
||||
var probe int
|
||||
err := tx.QueryRowContext(ctx, fmt.Sprintf("SELECT 1 FROM %s LIMIT 1", table)).Scan(&probe)
|
||||
switch {
|
||||
|
|
|
|||
|
|
@ -411,6 +411,12 @@ type Storage interface {
|
|||
// Work queries
|
||||
GetReadyWork(ctx context.Context, filter types.WorkFilter) ([]*types.Issue, error)
|
||||
GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, error)
|
||||
// GetReadyWorkWithCountsAndTotal is GetReadyWorkWithCounts plus the size
|
||||
// of the whole ready set the page was cut from — the same number
|
||||
// ReadyWorkCounter.CountReadyWork(filter) answers — resolved in the same
|
||||
// read transaction and statements as the page. It is what lets
|
||||
// `bd ready --limit N` print "Showing N of M" without a second pass.
|
||||
GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error)
|
||||
GetBlockedIssues(ctx context.Context, filter types.WorkFilter) ([]*types.BlockedIssue, error)
|
||||
GetEpicsEligibleForClosure(ctx context.Context) ([]*types.EpicStatus, error)
|
||||
|
||||
|
|
|
|||
|
|
@ -428,6 +428,16 @@ func (s *InstrumentedStorage) GetReadyWorkWithCounts(ctx context.Context, filter
|
|||
return v, err
|
||||
}
|
||||
|
||||
func (s *InstrumentedStorage) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) {
|
||||
ctx, span, t := s.op(ctx, "GetReadyWorkWithCountsAndTotal")
|
||||
v, total, err := s.inner.GetReadyWorkWithCountsAndTotal(ctx, filter)
|
||||
if err == nil {
|
||||
span.SetAttributes(attribute.Int("bd.result.count", len(v)))
|
||||
}
|
||||
s.done(ctx, span, t, err)
|
||||
return v, total, err
|
||||
}
|
||||
|
||||
func (s *InstrumentedStorage) GetBlockedIssues(ctx context.Context, filter types.WorkFilter) ([]*types.BlockedIssue, error) {
|
||||
ctx, span, t := s.op(ctx, "GetBlockedIssues")
|
||||
v, err := s.inner.GetBlockedIssues(ctx, filter)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue