beads/internal/workapi/sort.go
Julian Knutsen 9a2c918b0e
Admit the ephemeral plane from a list request, on the facade and on the wire (#5476)
* feat(reader): admit the ephemeral plane from a list request

`issueops.Reader.Ready` has had an ephemeral gate since it was written;
`List` has not. The wisp plane was reachable from a list request only as
a SIDE EFFECT of `IncludeInfra`, which also drops the infra-type
exclusions, or by naming an infra type, which routes to that plane
ALONE. A caller that wants the ephemeral rows of ordinary types — a
message bead, say — could ask for them on the ready query and not on the
listing, and the wire said so in prose: listIssues published no
`include_ephemeral`, and the document called the asymmetry out.

ListRequest.IncludeEphemeral closes it, with the same semantics
ReadyRequest.IncludeEphemeral already has.

  false (zero value)  durable rows only — today's listing, unchanged
  true                the wisp plane merged IN ADDITION; the answer is a
                      superset of the false answer under the same
                      filters, and never ephemeral-only
  WispType            a predicate on the wisp_type COLUMN, which BOTH
                      tables carry, so it selects no plane: it narrows
                      whatever was admitted. WispType alone answers with
                      nothing, since no durable row carries a
                      classification; WispType + IncludeEphemeral is
                      lawful and composes as an ordinary AND
  order               ONE order across both planes, by SortBy, as if
                      they were a single table — not durable rows
                      followed by ephemeral ones. A Limit truncates the
                      merged order, and the keyset position is applied
                      to BOTH legs, so a cursored walk over the merge
                      drops no row and repeats none. The non-snapshot
                      caveat is the one every multi-page walk already
                      has; the ephemeral plane only turns over faster
  --ready             CARRIED, not dropped. The ready query has an
                      ephemeral gate of its own, so
                      ReadyFilterFromIssueFilter now reads the plane bit
                      off the list filter. That also carries
                      IncludeInfra's plane half, which the projection
                      used to drop silently — `bd list --ready
                      --include-infra` now answers with the wisps `bd
                      count --include-infra` already counts

It is a PLANE knob and not a TYPE knob, which is the whole difference
from the three fields beside it: templates, gates and infra types stay
hidden by their own predicates. IncludeInfra remains strictly wider.

The wire gets `include_ephemeral` on listIssues, spec-first. There is no
capability-token change: it is an ordinary optional query parameter, so
an older server answers it with 400 `unknown_parameter` — the designed
skew signal, whose recovery is to degrade rather than retry.

Conformance runs on all three legs (server-backed store, embedded store,
unit of work) and covers admission, the merged order, every page bound
across the interleaving, a cursored walk across the plane boundary, and
the WispType interaction. The two seams arrange the merge differently —
one merge-sorts two independently ordered legs in Go and trims, the
other ORDERs a SQL UNION ALL — so the bounded and cursored arms are a
genuine two-body case rather than one body twice.

* fix(reader): correct the flag's reach, and settle the merge seam it lands on

Review follow-up. Three of these are documentation being wrong about the
code; one is the code itself.

THE MOTIVATING EXAMPLE WAS FALSE. `message` is one of the DEFAULT INFRA
TYPES (agent, role, message), so "the ephemeral rows of ordinary types —
a message bead, say" named the one row this flag cannot reach: the
infra-type exclusion is a TYPE predicate and survives the plane knob, as
the golden already recorded and nothing asserted. `bd ready` cannot
deliver it either — ReadyWorkExcludeTypes appends the infra set
unconditionally. The field doc, the operation description and the
parameter text now say that ephemeral agent/role/message rows need
IncludeInfra, and what the flag alone reaches is the ephemeral rows of
the types a listing already shows.

The contract case is what makes it more than a corrected sentence. The
fixture's only wisp was a task, which comes back under EITHER knob, so a
body that wired the plane flag to IncludeInfra's branch answered every
row correctly. An infra-typed wisp is hidden twice over — once by the
plane, once by the type — and is the only row that can tell the two
apart. Three rows now pin it: absent under IncludeEphemeral, present
under IncludeInfra, present under both. Wiring the flag into the
type branch fails it.

TWO MORE DOC CORRECTIONS on the same field. It admits the wisps TABLE,
which holds no-history rows (ephemeral = 0) alongside true ephemerals —
stated, rather than left as "the ephemeral plane". And it is NOT
identical to ReadyRequest.IncludeEphemeral: ready reads both planes
either way and adds a per-ROW ephemeral predicate when unset, where this
selects which TABLES are read. The difference is observable on exactly
those no-history rows, which a DEFAULT ready answer already contains and
a default listing does not.

The ready projection's predicate is negative, on a field whose zero
value means admit, so it fires for a filter that merely never set
SkipWisps rather than for one that asked. Every caller is handed a
BuildListFilter result, which decides that field for every request, so
"unset" is unreachable today — named at the site so the next hand-built
filter does not walk into it.

THE SEAM DISAGREEMENT, which is the code change. A cross-plane duplicate
id was tolerated by the per-table seam (collapse to the canonical wisp,
answer) and HARD-ERRORED by the union seam, and this flag routes every
wisp-inclusive read onto the union. Two implementations of one role
answering a page and an error is a divergence, not a design, and leaving
it silent while making it reachable is worse than either behavior.

The union now resolves it the way its sibling does, at all four sites,
and the strict flag is gone rather than left as a footgun. Three things
say that is the right verdict and not merely the convenient one: the
per-table seam has said so since be-iabdi ("hard-erroring breaks every
lookup city-wide"); `bd doctor` detects the state with a query of its
own, so answering blinds no detector; and its fix deletes the stale
ISSUES copy, which is exactly the row the union now drops. Its own check
text names the hard failure as the damage — "stale issues-table copies
break every lookup for the affected IDs". The four tests that pinned the
error now pin the wisp-canonical answer; inverting the preference fails
all four.

Preferring the wisp needs a second pass: the outer ORDER BY can place
either copy first, so which leg produced a row is known in full only
after the scan.

Last, a stale comment: issueops/count.go claimed SearchIssuesInTx was
the corruption detector. It has not been one since be-iabdi. The count
trusts the disjoint-table invariant and no read enforces it, so a
dual-resident id is counted twice and listed once; doctor is the
detector and the repair.

---------

Co-authored-by: Review Sim <ci@beads.test>
2026-08-09 05:06:36 -07:00

174 lines
7.2 KiB
Go

package workapi
import (
"cmp"
"strings"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/internal/utils"
)
// CompareIssuesBy orders two issues by one of `bd list --sort`'s fields.
// An unknown field compares equal, which leaves the underlying order intact.
func CompareIssuesBy(a, b *types.Issue, sortBy string) int {
switch sortBy {
case "priority":
return cmp.Compare(a.Priority, b.Priority)
case "created":
return b.CreatedAt.Compare(a.CreatedAt)
case "updated":
return b.UpdatedAt.Compare(a.UpdatedAt)
case "closed":
if a.ClosedAt == nil && b.ClosedAt == nil {
return 0
} else if a.ClosedAt == nil {
return 1
} else if b.ClosedAt == nil {
return -1
}
return b.ClosedAt.Compare(*a.ClosedAt)
case "status":
return cmp.Compare(a.Status, b.Status)
case "id":
return utils.NaturalCompareIDs(a.ID, b.ID)
case "title":
return cmp.Compare(strings.ToLower(a.Title), strings.ToLower(b.Title))
case "type":
return cmp.Compare(a.IssueType, b.IssueType)
case "assignee":
return cmp.Compare(a.Assignee, b.Assignee)
}
return 0
}
// ReadyFilterFromIssueFilter projects a list filter onto the ready-work
// filter, which is how `--ready` reuses the whole list vocabulary against the
// blocker-aware query. WorkFilter is the narrower of the two, so this is a
// deliberate lossy projection: what it does not carry, ready work does not
// filter on.
func ReadyFilterFromIssueFilter(filter types.IssueFilter) types.WorkFilter {
wf := types.WorkFilter{
Status: types.StatusOpen,
Limit: filter.Limit,
Offset: filter.Offset,
Labels: filter.Labels,
LabelsAny: filter.LabelsAny,
ExcludeLabels: filter.ExcludeLabels,
LabelPattern: filter.LabelPattern,
LabelRegex: filter.LabelRegex,
ParentID: filter.ParentID,
MolType: filter.MolType,
WispType: filter.WispType,
ExcludeTypes: filter.ExcludeTypes,
MetadataFields: filter.MetadataFields,
HasMetadataKey: filter.HasMetadataKey,
MaxRows: filter.MaxRows,
MaxRowsSource: filter.MaxRowsSource,
}
if filter.IssueType != nil {
wf.Type = string(*filter.IssueType)
}
if filter.Priority != nil {
wf.Priority = filter.Priority
}
if filter.Assignee != nil {
wf.Assignee = filter.Assignee
}
if filter.NoAssignee {
wf.Unassigned = true
}
// The ephemeral PLANE, which the two filters spell differently: a list
// filter admits it by leaving SkipWisps off (or, for an infra type, by
// routing to it outright), and a work filter by setting IncludeEphemeral.
// Reading both spellings is what carries ListRequest.IncludeEphemeral —
// and IncludeInfra's plane half with it — onto the --ready arm instead of
// dropping it there, which would answer a request that named the plane
// with the durable set and no error.
//
// THE PREDICATE IS NEGATIVE, on a field whose ZERO VALUE means admit. That
// is deliberate but it is a footgun worth naming: this fires for any filter
// that merely never set SkipWisps, not only for one that asked. Every
// caller today is handed a filter built by BuildListFilter, which decides
// that field for every request, so "unset" is not reachable here — a
// hand-built types.IssueFilter{} passed in from somewhere new would get the
// wisp plane it never asked for. Keep the construction on the builder.
if (filter.Ephemeral != nil && *filter.Ephemeral) || !filter.SkipWisps {
wf.IncludeEphemeral = true
}
return wf
}
// WithFetchOneExtra bumps Limit by one so the caller can detect "more results
// than the limit" (GH#3212) by comparing the row count against the original
// limit.
//
// `--limit N --max-rows N` (equal) needs special handling so this bump doesn't
// collide with the MaxRows cap check: EffectiveSearchLimit treats
// Limit==MaxRows as "no overage sniff" (returns Limit verbatim, no +1 — see
// EffectiveSearchLimit's doc and the WithLimit_LimitAtCap_NoError storage
// test), but bumping Limit to N+1 alone flips that to Limit>MaxRows, which
// *does* sniff for overage — so the probe row itself would trip
// EnforceMaxRowsCap(N+1, N, ...) even though the delivered set is always
// trimmed back to N. Bumping MaxRows by one in lockstep, only in this exact
// N==M case, keeps the probe row inside the (temporarily N+1) cap so
// EnforceMaxRowsCap never fires on it while still fetching N+1 rows for the
// truncation check. This can't false-negative a real violation: the query's
// own LIMIT is capped at N+1 either way, so "more than N+1 matches exist" was
// already undetectable at Limit==MaxRows before this bump ever existed (the
// same "no overage detection above the equal cap" contract EffectiveSearchLimit
// documents for the unbumped case).
//
// This does not change the Limit>MaxRows (tighter-cap) or Limit<MaxRows cases:
// EffectiveSearchLimit's branch selection there is unaffected by a one-row bump
// to Limit, and MaxRows is left untouched, so a genuine tighter-cap violation
// still fires with the correct (unbumped) Cap value in the error message.
// The pair of steps this performs — bump the cap in lockstep at Limit==MaxRows,
// then bump Limit — is also written as one function for the seam that renders
// its probe row inside the query rather than onto a filter; see
// internal/storage/issueops.SearchProbeLimit. Change one and change the other,
// or the two implementations of issueops.Reader.List stop agreeing about when a
// cap fires.
func WithFetchOneExtra(filter types.IssueFilter) types.IssueFilter {
if filter.Limit > 0 {
if filter.MaxRows > 0 && filter.Limit == filter.MaxRows {
filter.MaxRows++
}
filter.Limit++
}
return filter
}
// WithRowsBeforeThePage moves an OFFSET off the query and onto the epilogue: it
// widens the filter's row bound to cover the rows the epilogue will skip, and
// clears the Offset the builder carried so the query does not skip them too.
// An unbounded filter needs no widening — it already carries every matching row.
//
// It is the filter half of FinishPageAt, and the reason both halves exist is
// there: the skip belongs after the display order, and the rows it drops are
// rows the MaxRows cap has to have counted. Apply it BEFORE WithFetchOneExtra,
// which sizes the probe row against the bound this leaves.
//
// The builders still write Offset onto the filter, for the callers that consume
// it as a VALUE and run their own query — `bd list --watch`, the proxied
// hierarchical --parent walk, proxied `bd ready` — where there is no shared
// epilogue to move it to. Those callers get the same cap boundary anyway: the
// seam beneath them widens its own bound by the offset and keeps the skip
// whenever a cap is set, for the reason stated above (internal/storage/domain/db,
// searchWindow).
func WithRowsBeforeThePage(filter types.IssueFilter, offset int) types.IssueFilter {
filter.Offset = 0
if offset > 0 && filter.Limit > 0 {
filter.Limit += offset
}
return filter
}
// WithReadyRowsBeforeThePage is WithRowsBeforeThePage for the ready-work filter,
// which is a different type carrying the same two fields.
func WithReadyRowsBeforeThePage(filter types.WorkFilter, offset int) types.WorkFilter {
filter.Offset = 0
if offset > 0 && filter.Limit > 0 {
filter.Limit += offset
}
return filter
}