* perf(ready): recurse the descendant walk off indexed base tables (be-qfm)
`bd ready --parent` and `bd blocked --parent` compute the parent's
transitive descendants with a recursive CTE whose edge relation was a
non-recursive `parent_edges` CTE over `dependencies UNION ALL
wisp_dependencies` with the typed target columns COALESCE'd into one.
Dolt cannot index through that wrapper: it materializes the relation and
re-scans it once per recursion row, so the walk costs (parent-child rows)
x (descendants) — 7.46 s for 483 descendants on a 4.6k-edge database,
about 14 ms per descendant, and past the shared pool's read deadline on
a busy server.
Build the walk in sqlbuild.DescendantWalkQuery instead: one anchor and
one recursive member per (dependency table x typed target column), each
joining the base table on a bare indexed column, with per-column IS NULL
guards reproducing the COALESCE precedence and the LOCATE cycle guard,
depth guard and parent-child filter carried onto every member. The
recursive members carry JOIN_ORDER(d,e) LOOKUP_JOIN(d,e): the cycle guard
references both join sides, and without the hints Dolt's planner keeps
the recursive working table on the inner side and scans the edge table
per iteration, which measured no faster than the wrapper shape and
flipped plans between runs. With them the frontier is the outer side and
every member is an index probe.
issueops.GetDescendantIDsInTx and domain/db.getDescendantIDs call the
builder; they had carried the same query text separately.
Measured on a local Dolt 2.2.4 sql-server with a generated forest of
4,541 issues, 4,500 parent-child rows and 500 descendants under the
probed parent (BenchmarkDescendantWalk, new): 1.26–2.28 s/op on main,
32–36 ms/op with this change. Same rows on every target kind, cycle
and depth limit (TestDescendantWalkReachesEveryTargetKindAndStopsOnCycles,
new; the conformance ReadyParentScopesToItsTransitiveDescendants cases on
all three backends).
Refs #6128
Co-authored-by: qcore/syl <q@withq.ai>
Agent-Signature: claude-code-claude-fable-5-max on behalf of a3ackerman
* test(embeddeddolt): gate the descendant-walk reference test on cgo (be-qfm)
The embedded-Dolt test tier is cgo-only and every sibling file carries
the build tag; this one did not, so the CGO_ENABLED=0 boundary job
compiled it against a package whose helpers were excluded and failed on
an undefined newTestEnv.
Refs #6128
Agent-Signature: claude-code-claude-fable-5-max on behalf of a3ackerman
* fix(review): sync BUILD.bazel srcs and resolve every dep target column in the descendants CTE
Maintainer fixup from the PR #6130 adoption review, iterations 1 and 2.
Bazel srcs gap (major, claude+codex). The four new .go files were absent from
the gazelle-managed go_library/go_test srcs lists, so
//internal/storage/sqlbuild did not contain DescendantWalkQuery and both
consumers failed to compile under Bazel:
internal/storage/issueops/blocked.go:170:13: undefined: sqlbuild.DescendantWalkQuery
internal/storage/issueops/blocked.go:171:13: undefined: sqlbuild.DescendantWalkArgs
`make bazel-sync` adds exactly the four srcs rows the review measured; the
fifth BUILD.bazel row is a deps entry this commit's own new test requires.
A second sync leaves the tree clean. Note that `make bazel-sync-check` and
tools/bazel/go_srcs.py --check pass falsely on a srcs gap -- they manage only
the go_srcs glob filegroups -- and bazel-autofix cannot absorb this one
because bazel.yml:313 guards the sync step on steps.bazel.outcome ==
'success', which a compile break never reaches.
buildDescendantsCTE target columns (major, codex). GetDescendants' walk
resolved a row's parent through COALESCE(depends_on_issue_id,
depends_on_wisp_id), omitting depends_on_external, while the classic ParentID
filter it mirrors resolves all three and the reference test this PR adds makes
a cross-prefix parent a pinned contract for the sibling walk. ck_dep_one_target
(migration 0041) leaves exactly one target column set, so an external-target
row coalesced to NULL and could never match an edge: adding the column is
strictly additive, and it closes a gap where the proxied-server
bd list --parent tree (and --watch) could not see a cross-prefix child or
its subtree, while the direct route's per-level ParentID filter listed them.
TestBuildDescendantsCTEResolvesEveryDepTargetColumn derives the expected
column set from sqlbuild.DepTargetExpr, so the two cannot drift again.
That finding's other half -- migrating these members to the per-(table,
column) shape -- is deliberately deferred to bd-tztam and recorded at the
site. buildDescendantsCTE is a different query ((id, src, via) projection, a
dotted-ID fallback recursion, hoisted per-level filter predicates, no depth
cap, no path cycle guard), sqlbuild exports no member-fragment API, and
per-column members would triple its branch count, which the predBundle
comment above it records as tripping the dolt 2.1.6 analyzer.
nullGuards precedence and column binding (minor, claude+codex; minor, codex).
TestDescendantWalkQueryReproducesDepTargetPrecedence pins the per-column
IS NULL guards, which no existing test could see: the shape test never asserts
on a guard and the reference test seeds exactly one target column per edge, so
emptying nullGuards left both new test files green. The expected counts are
derived from DepTargetExpr's column list rather than restated, so the same
test binds descendantWalkTargetCols to it. The review's prescribed
`DepTargetExpr = "COALESCE(" + strings.Join(descendantWalkTargetCols, ", ")
+ ")"` does not compile -- issueops/dependencies.go:58 and
domain/db/dependency.go:35,:37 consume it in const declarations -- so this
takes the unit-assertion arm the same finding offers.
LOOKUP_JOIN warning multiplicity (nit, codex). The doc comment now records
that the ignored-hint warning is raised once per hint comment, six per
includeWisps=true query.
Iteration 2 (two minors, gate impact none, on a 981/850 approve).
TestGetDescendantsCrossPrefixParent pins the widened COALESCE against real
rows, where the test above pins only query text. Its edges go through the
repository's own classifier, which routes any cross-prefix target to
depends_on_external even when both ends are local, and each of its four
edges is matched by a different edge member (issues and wisps, anchor and
recursive): reverting the column at any one member reds the case with a
different set of children missing. CHANGELOG.md gains a Fixed bullet for
the user-visible half of this change, the proxied-server bd list --parent
tree.
Validation: sqlbuild 31 PASS / 0 FAIL / 0 SKIP; full domain/db package green
against the real Dolt container (915 PASS / 0 FAIL / 0 SKIP, 49s),
including TestGetDescendantsCrossPrefixParent,
TestGetDescendantsDottedOrphans, TestGetDescendantsFilteredByStatus and
TestParityReadyBlockedDescendants; golangci-lint 2.10.1 reports 0 issues on
the three changed packages; go test -run Bazel ./scripts/ is 13 PASS / 0 FAIL,
including TestBazelTestTagsFollowTaxonomy; bazel test
//internal/storage/sqlbuild:sqlbuild_test PASSED and bazel build
//internal/storage/domain/db:db_test succeeds; a final make bazel-sync
leaves the tree unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: qcore/syl <q@withq.ai>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(bazel): add internal/testutil/bazeltest runfiles helpers
Repository-path resolution for tests that works identically under plain
go test (package cwd, full checkout), bazel test locally, and remote
execution (declared runfiles only):
- IsBazel, RepoRoot (BEADS_TEST_REPO_ROOT, then the runfiles workspace
root, then a go.mod walk), OverrideRoot (drop-in condition for existing
helpers; empty under plain go test), CallerDir
- Runfile for $(rlocationpath) values, GOROOT for type-checking tests
- ShardFreeEnv for helpers that re-exec a test binary
Every helper reduces to the pre-Bazel behavior under go test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(bazel): declare source/data reads for 14 scan-guard targets
Source-scanning and repo-file tests read files their Bazel targets never
declared, so they failed under `bazel test` (and could not run on remote
workers at all). Declare narrow inputs as `data` and resolve paths through
bazeltest where the test located the tree via runtime.Caller:
- tools/bazel/go_srcs.py (run by `make bazel-sync` after gazelle) keeps a
managed `go_srcs` filegroup in the packages whose Go source tests scan,
plus `//internal/storage:tree_go_srcs` for the storage-tree walk. Bazel
globs stop at package boundaries, so the aggregate needs one filegroup
per package; the script keeps it in sync and gazelle-stable.
- Hand-written narrow inputs: //internal/storage/schema:migrations,
//:docsync_files, //plugins/beads:plugin_files, and exports of
.gitattributes, .claude-plugin/marketplace.json, cmd/bd list.go and
ready.go, testutil/testdoltcommon.go, scripts README.md and the release
catalog, each visible only to its consumer.
- runtime.Caller-based roots in backend, backend/conformance,
internal/storage, internal/formula and test/docsync now go through
bazeltest.CallerDir / OverrideRoot; relative-path tests need data only.
- integrations/ leaves .bazelignore (it is still gazelle-excluded) so the
docsync link check can declare integrations/beads-mcp/PYPI.md.
Under --config=prcore every target reports the same PASS/SKIP/FAIL test
set as `go test -race -short` for its package, locally and remotely.
Plain `go test` behavior is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(bazel): guard generated go_srcs and harden bazeltest root override
- tools/bazel/go_srcs.py: add --check (print diff, exit 1 on stale blocks);
new `make bazel-sync-check` target runs it.
- scripts/bazel_policy_test.go: under go test, assert every tree_go_srcs
(//internal/storage) lists exactly the BUILD.bazel packages below its
root, run go_srcs.py --check, and forbid BUILD files under docs/ and
engdocs/ (they would cut subtrees out of //:docsync_files).
- bazeltest: honor BEADS_TEST_REPO_ROOT only when it holds go.mod
(RepoRoot fails loudly otherwise); drop the unused GOROOT() helper,
whose compiled-in branch could never fire under rules_go.
- scripts/ci/lib/test-env.sh: unset BEADS_TEST_REPO_ROOT.
- Document the hand-listed out-of-tree link targets in //:docsync_files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* build(bazel): re-sync BUILD files with main
Regenerated with make bazel-sync after commits that landed while the
scaffold PR was in CI (new internal/execenv, gitenv, gitignore packages
and new sources/deps).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(bazel): declare init_safety_help.go for docsync
TestInitSafetyCLIHelpCoversReCloneGotchas (#6431) reads
cmd/bd/init_safety_help.go; export it to //test/docsync only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* build(bazel): add side-by-side Bazel scaffold (module + gazelle)
Add a Bzlmod Bazel build alongside the Go toolchain. `go build` and
`go test` are unchanged; nothing in CI invokes Bazel yet.
- MODULE.bazel: rules_go 0.63.0, gazelle 0.53.0, Go SDK 1.26.7 (matches
the go.mod toolchain directive), go_deps from go.mod.
- ICU policy (engdocs/ICU-POLICY.md): a go-mysql-server gazelle_override
selects the pure-Go regex backend and excludes regex_cgo.go, and
.bazelrc builds with --@rules_go//go/config:tags=gms_pure_go, so no
target depends on go-icu-regex or links libicu.
- third_party/patches: three module_override patches for dolt deps that
ship WORKSPACE-era BUILD files, adapted from the gascity project.
- .bazelrc: prcore/ci test configs (inert for now; they match pr-core's
test selection, not yet its environment isolation) and a generic
remote-exec config with a bounded remote timeout. Executor endpoints and
TLS credentials come only from a gitignored .bazelrc.local/user.bazelrc
or a CI rc outside the workspace.
- .bazelversion pins Bazel 9.2.0; .bazelignore skips separate modules,
non-Go trees, nested agent worktrees (.claude, .worktrees) and local
data/output dirs (.beads, .dolt, dist, .runtime, tmp), so a primary
checkout neither builds nor gazelle-rewrites nested worktree copies.
- Gazelle-generated BUILD.bazel files for every Go package.
- Makefile: `make bazel-sync` (gazelle + bazel mod tidy).
- .gitignore: bazel-* output symlinks and machine-local rc files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(bazel): policy tests for the Bazel configuration
Add scripts/bazel_policy_test.go, plain Go tests that run under both
`go test ./scripts` and `bazel test //scripts:scripts_test`:
- MODULE.bazel go_sdk.download version equals the go.mod toolchain.
- ICU policy: .bazelrc sets gms_pure_go for build, the go-mysql-server
gazelle_override excludes regex_cgo.go, and the root BUILD.bazel
carries `# gazelle:build_tags gms_pure_go`.
- .bazelversion pins an exact Bazel release.
- .bazelrc.local and user.bazelrc are gitignored (parsed from .gitignore
and cross-checked with git check-ignore when in a checkout).
- No remote-execution endpoints or credentials in committed bazelrc-like
files, Markdown docs or .github/**: remote/BES/TLS flags
(--remote_executor, --remote_cache, --bes_backend, --remote_header,
--tls_client_certificate, ...) with a literal value, with or without a
URL scheme, and non-loopback gRPC URLs in rc/.github files. Placeholders,
$VARIABLES, and loopback are allowed; Markdown gRPC URLs (OTel exporter
docs) are not flagged. Skipped outside a git checkout, e.g. in the Bazel
sandbox.
Each check is a pure function also exercised against fixtures that break
the invariant. The files the tests read are declared as data via
//:bazel_policy_files so they resolve from runfiles under Bazel.
scripts/check-build-tags.sh now treats Bazel invocations as governed by
.bazelrc: the shell command segment that invokes Bazel (split on &&, ||,
;, |, $( and backticks) is exempt from the per-line `go` tag check, while
a bare `go` in another segment of the same line is still flagged. When
any scanned file invokes Bazel (or .bazelrc exists) .bazelrc must set
--@rules_go//go/config:tags=gms_pure_go for build.
scripts/check_build_tags_test.go covers both in a temp repository.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(scripts): split bazel lines in pure bash so check-build-tags works with BSD sed
BSD sed (macOS) does not expand \n in s/// replacements, so the segment
split never happened and the Bazel-exemption tests failed on macos-latest.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(scripts): skip check-build-tags tests when bash lacks mapfile
macOS runners use /bin/bash 3.2, which has no mapfile; check-build-tags.sh
has always required bash >= 4, so the new tests are skipped there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bd ready pins WorkFilter.Status to StatusOpen. BuildReadyWorkWhere treated
that pin as an exact match, so issues in a custom status with category
active never appeared — even though bd statuses --help, the ready_issues
view (migration 0025), and ClaimableSourceStatusesInTx all treat that
category as ready-eligible.
Widen the singular StatusOpen clause to the same predicate the view uses:
status = ? OR status IN (SELECT name FROM custom_statuses WHERE category = 'active')
Non-open singular statuses stay exact. Built-in open still matches. Custom
wip/done/frozen/unspecified statuses stay out, matching claim's GH-3570
anti-steal set.
Reported by Justin Clemens (@justin-candor).
Addresses #5831
External (cross-project) dependency blockers are now enforced everywhere a bead's readiness or closability is decided: `bd ready` / `--claim`, `bd blocked`, `bd close` / done-status `bd update` (direct and proxied), `bd dep tree`, and served HTTP ready/claim. The policy is a storage-boundary decorator (`internal/storage/externaldeps`) applied on both the DoltStorage chain and the UOW provider, with full capability-forwarding parity, `Unwrap` at every level, and the backend's advisory defer-wake preserved on the claim path.
Review trail: maphew rounds 1–4 (Jul–Aug), bee rounds 1–4 (Sep 2): D1 recording-parity exemption, D2 ready-filter goldens, D3 UOW-level Unwrap for the proxied importer, D4 claim-path defer wake. Verified at head 3468d62f on a clean merge with current main; CI fully green at that head.
Fixes#4769.
* feat(httpapi): serve `bd list`'s order with `sort` on listIssues
The issue listing served one order, `(created_at DESC, id ASC)`, because the
cursor is a keyset position in it. The spec stated the constraint in its own
words: "the sort order is welded to the cursor contract, and a new order needs
new surface." That welding put the cost on every client wanting `bd list`'s
ordering — the only way to get it was to page the whole result set and sort
locally, which is the one cost on this surface that grows with the project.
Measured against a real `bd serve` over 1400 rows at the 200-row page an HTTP
client actually uses: walk-and-sort is 7 requests plus a client-side
comparator, `?sort=priority` is 1. Both are checked against `bd list --json`
row for row, because the claim that matters is not "fewer requests" but "the
same order".
`sort` takes `created` and `priority` and nothing else. Each value is a keyset
contract — a position shape, a strictly-after predicate, an index — not a
display preference, and both keys are provably total (priority and created_at
are non-null, id is unique) so a page boundary inside a run of equal keys
resolves on id. The seven other orders `bd list --sort` takes have no such key:
`id` is natural-numeric and no database expresses it, `updated` moves on every
write, `closed` is nullable, and status/title/type/assignee are mutable and
unindexed. `priority` here is (priority ASC, created_at DESC, id ASC), which is
both `bd list`'s flagless order and what `--sort priority` produces — a stable
priority sort over the created order is the same sequence — so one served order
retires the walk for both.
Absent `sort` still means `created`, permanently: it is the compatibility
contract for every client written before the parameter existed.
THE CURSOR NOW CARRIES ITS ORDER. This is the half that makes the parameter
safe rather than merely useful, and the obvious implementation gets it wrong.
The token was base64 of `{t,i}` with no binding, so a created-order cursor
replayed under `sort=priority` decoded perfectly and was read as a position in
a different total order — a page that skips and duplicates rows, served with a
200, undetectable by the client. Tokens are now `v2` with an explicit order
member, and decode takes the wanted order as a parameter so a mismatch is
unrepresentable at the seam rather than merely detectable past it. Outstanding
`v1` tokens stay readable as the created-order positions they are, so no
traversal in flight has to restart. This does not contradict "the token carries
no filters": filters select the set, the order decides what the position means.
Storage grows one field. IssueFilter.AfterPriority extends the existing
position; the predicate nests the created one rather than restating it, because
the created/id comparison is only reachable at the cursor's own priority.
Flattening it the way the created predicate's leading-bound trick invites looks
equivalent and is not — it re-delivers rows at the cursor's priority created
after it, which the new conformance walk catches against real Dolt.
NO INDEX MIGRATION, by measurement rather than by omission. A composite
(priority, created_at, id) index was tried on Dolt 2.2.0 against the production
predicate, in both the plain ascending and the mixed-direction spelling Dolt's
DDL accepts: the plan is byte-identical in all three cases, keeping the
single-column priority seek and the TopN. It would have bought write
amplification on every insert and every priority update for no measured change.
The sargability guard pins what is achieved — an IndexedTableAccess range on
idx_issues_priority — and records the measurement to redo if Dolt learns to
stream the composite.
Agent-Signature: claude-opus-5
Co-authored-by: CI Bot <ci@beads.test>
* fix(changelog): drop leftover diff3 conflict marker in sort entry
The rebased adopt-PR head left a stray '|||||||' merge-base marker on the first line of the new `sort` CHANGELOG entry (CHANGELOG.md:381), which `git diff --check` flagged as a leftover conflict marker. Remove only the marker line; the resolved entry content is unchanged. Maintainer fixup for PR #5666 adoption review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: CI Bot <ci@beads.test>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sweep): select purge/prune candidates by tier, not the raw ephemeral flag
A typed wisp minted without the ephemeral flag was invisible to bd purge:
the candidate filter matched ephemeral=1 alone, and the Ephemeral=true
search route only looks at the wisps plane — so legacy typed wisps in the
issues table (858 rows / 7.3 MB in one production DB) survived every sweep
that existed to delete them, while still reporting 'No closed ephemeral
beads to purge'.
IssueFilter gains EphemeralTier: ephemeral-tier is ephemeral=1 OR a
non-empty wisp_type; the durable tier is the complement, so bd prune stops
silently owning rows purge should. The field deliberately does not take
the wisps-plane-only fast path — a tier query merges both planes, because
the rows that motivated it live in the issues table. NoHistory beads
(wisps plane, ephemeral=0, no wisp_type) stay durable-tier (GH#3649).
The list/ready golden filters gain the new field as null.
* fix(create): a wisp_type implies the ephemeral tier at mint
All three create paths (ExecuteCreate, the batch body, and the dolt
store's own createIssue routing) now infer ephemeral from a non-empty
wisp_type, the same way they already infer it from a configured infra
type — so no new mints produce the flag-less typed-wisp shape the sweep
fix has to catch. NoHistory keeps its own retention mode, as before.
* test(dolt): pin the tier boundary on the legacy typed-wisp shape
Three cases: a manufactured issues-plane row with wisp_type but no
ephemeral flag is taken by the ephemeral sweep and left by the durable
one; a NoHistory bead stays durable-tier; and a create carrying only a
wisp_type lands in the wisps plane marked ephemeral. The first and third
were verified red against their halves of the fix reverted.
* test(sweep): conformance tier-boundary cases + the remaining mint sites
Adds two cases to the sweeper conformance kit, wired in all three backends:
- RunSweeperTreatsALegacyTypedWispAsEphemeralTier — the row shape that
motivated the fix: a closed issues-plane row carrying wisp_type but not
the ephemeral flag (858 such rows / 7.3 MB survived every purge in one
production DB). Manufactured with a raw UPDATE through a new optional
Exec fixture hook, because every current create path now infers the flag.
- RunSweeperLeavesNoHistoryBeadsToTheDurableTier — the other edge: a
NoHistory bead (wisps plane, ephemeral=0, no wisp_type) is durable-tier
(GH#3649) and must survive the widened ephemeral sweep. Seeded through
CreateWisp because plane routing of NoHistory is the create verb's
concern and the uow kit's CreateIssue calls the domain-level create.
Also carries the wisp_type→ephemeral mint inference to the five remaining
create sites (uow issue_operations/batch_creator/batch_applier,
embeddeddolt create_issue) and slims the dolt-local typed-wisp test to the
mint case now that the sweep half lives in the conformance kit.
Verified: TestSweeperContract green on dolt (91s), uow (87s), and
embeddeddolt with BEADS_TEST_EMBEDDED_DOLT=1 (76s, 13/13 subtests, no
skips).
* test(sweep): dispatch the two new sweeper cases in roleContractCases
The conformance package's own coverage tripwire
(TestRoleContractCasesCoverEveryContractCase) failed in CI: a role contract
case that no roleContractCases row dispatches is invisible to an external
backend running RunRoleContracts. The legacy typed-wisp case still skips
cleanly on a bundle whose SweeperFixture carries no Exec hook.
* fix(storage): stop reporting a broken wisp plane as an empty one in search
A wisp search folded any table-not-exist error into "there are no wisps". The
tolerance is there for pre-migration databases, which have no wisp plane at
all, but the wisp query reads more tables than the wisp plane owns: its FROM
clause carries sqlbuild.LeaseJoin and its hydration reads wisp_labels. So a
database that has wisps but has lost wisp_labels is reported as having no wisps
-- with a nil error, from an ordinary list.
Reproduced end-to-end against embedded Dolt, with the row present in the same
transaction that is told it is not:
RENAME TABLE wisp_labels TO wisp_labels_renamed_away
SearchIssues(ephemeral) before: 1 row after: 0 rows, nil error
SearchIssues(no filter) before: 2 rows after: 1 row, nil error
GetIssueInTx already surfaces a missing wisp_labels as an error (pinned by the
existing embedded-Dolt wisp_label_table_error_propagates case), so the two read
paths disagreed today about whether that database is broken or empty.
Narrow the tolerance to the tables the wisp plane may legitimately not have.
sqlbuild.OptionalWispTable becomes the one place that set is written down --
issueops.optionalBlockedTable now reads it, so the classic and domain/db stacks
cannot drift on what a missing table means, which is the same reason the query
text itself lives in sqlbuild. The check is gated on the existing
IsTableNotExist classification via dberrors.MissingTableName, so nothing that
was previously an error becomes tolerated.
The leases half of the same over-broad check is included for the same reason
but is not, on its own, a reachable wrong answer today: the ephemeral branch
falls through to an issues-plane query over the same join, which re-raises the
missing table, so the caller does get an error. That was verified rather than
assumed, and the tests pin it at the guard rather than claiming a user-visible
repro.
Tests are red-first: sqlmock cases on both stacks and an end-to-end
RENAME TABLE against real embedded Dolt. Both directions have controls -- a
pre-migration database missing wisps must still search to an empty result, and
a missing wisp_dependencies must stay invisible -- because over-tightening to
"never tolerate" passes every must-error assertion and only those controls
catch it.
* test(storage): cover the union hydrate-wisps guard, which nothing reached
The review council mutated all four guarded call sites and found one that
no test in the repository reaches: the domain/db searchUnion hydrate-wisps
check. Reverting it to the old blanket table-not-exist test survived the
full domain tree and the nine-minute uow contract suite with embedded Dolt
enabled.
It is the site that matters most. The ephemeral branch needs an explicit
filter; the union is what a plain `bd list` runs. And the union is exactly
where the two stacks diverge: the leg's FROM clause is plan.FromSQL, with
no lease join and no label read, so a database missing leases or
wisp_labels answers the union query successfully and returns wisp ids --
the breakage surfaces only in fetchIssuesByIDs, which adds
sqlbuild.LeaseJoin and hydrates labels. Tolerating it there sends every
wisp through reassembleBySrc as a miss and drops it behind a nil error.
Also adds the fail-open control the domain/db file was missing. The
council's other surviving mutant made the guard return true for any error
it could not parse a table name out of; issueops caught it, domain/db was
green across the whole package.
Red before green (real edits, restored by trap):
union guard -> blanket IsTableNotExist TestSearchUnionBrokenWispPlane
IsAnError, both subtests; every
other test stays green
fail open on unparseable errors both UnrelatedErrorPropagates
tests, ephemeral and union
The e2e wisp_dependencies control keeps its name but loses its overclaim:
a search that never asks for dependencies cannot see that table missing,
so it stays green whatever the tolerance set is. The sqlmock control is
what pins that entry; this one proves the query still answers end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(stale,blocked): add label filtering to bd stale and bd blocked
bd stale and bd blocked were the last two issue-listing commands with no way
to scope output by label; bd list, bd ready, bd orphans, bd count and bd
search all have it. Add -l/--label (AND), --label-any (OR) and
--exclude-label (NOT) to both, reusing the existing help text so the flags
mean the same thing wherever a user meets them.
bd blocked was also silently discarding label filters it already accepted:
types.WorkFilter has carried Labels/LabelsAny/ExcludeLabels all along and
blockedCmd builds a WorkFilter, but GetBlockedIssuesInTx only ever read
ParentID, so a caller setting them got unfiltered results back and no error.
New sqlbuild.LabelSetClauses is the IN-subquery form of the three predicates,
for callers whose FROM clause is fixed and cannot take the JOIN that
BuildLabelDrivenSearch builds. A test pins it against the equivalent blocks in
BuildIssueFilterClauses so the two cannot drift: --label should mean the same
thing whichever command it is passed to.
On stale the predicates go in the WHERE clause rather than onto the hydrated
rows, because filter.Limit is this query's LIMIT and post-filtering would make
--label x --limit 10 return fewer than ten matching issues. On blocked they go
in the per-table scan, so issues filter against labels and wisps against
wisp_labels, which also spares the blocker-dependency and hydration passes any
work on rows that cannot survive.
Both the direct and the proxied-server path now build the blocked filter
through one blockedFilterFromFlags helper rather than assembling it twice.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(stale,blocked): normalize the new label filters, as every sibling does
Review feedback on #5822. The new flags built exact `label = ?` clauses from
raw pflag values, while every other label filter in bd normalizes first --
list_input.go:293-295, search.go:106, orphans.go:56, workapi/ready.go:56-58.
That gap is the read-side twin of #5812, and #5813 does not cure it:
normalize-on-write only protects labels a fixed bd wrote, whereas a filter
input still needs trimming wherever it comes from.
Two shapes, both verified on the embedded engine:
- `--label ' theme:a'` returned 0 rows while `bd list --label ' theme:a'`
returned the issue. This is the everyday case, not an exotic one: pflag's CSV
split leaves the space on the second value of `--label 'a, b'`.
- `--label 'a,,b'` ANDed in a `label = ''` clause, which nothing can satisfy,
turning a valid filter into an empty result.
Both fail silently -- a filtered list short of rows reads exactly like a
complete one -- and both cut against the promise LabelSetClauses is documented
to keep, that --label means the same thing whichever command you reach for.
Normalization sits at the two shared choke points, so no route is missed:
blockedFilterFromFlags (direct + proxied blocked) and stale's RunE, which
builds the StaleFilter both its routes use.
Six new embedded subtests across bd stale and bd blocked pin trimming, the
empty-element case, and that both apply to --label-any and --exclude-label.
Verified failing with this commit's normalization reverted (6 fail, the nine
pre-existing label subtests still pass) and passing with it.
CHANGELOG: the flag addition, plus the WorkFilter silent-drop fix found en
route -- bd blocked accepted Labels/LabelsAny/ExcludeLabels and read only
ParentID, so programmatic callers got unfiltered results and no error.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(search): order Go-side-sort pages before every trim; honor sortDesc for id
Two live defects, both named-but-deferred in #5587's commit message and
both reachable from bd query '<expr>' --sort id (DefaultQueryLimit means a
bound is always set):
1. sqlbuild.Less ignored sortDesc for "id" — the one Go-side sort key —
so every merge sort feeding a reversed id page kept the byte-FIRST
rows. idSrcPage.sortGoSide's doc named this sibling bug explicitly.
2. #5587's fix (a page bound is only ever pushed under an order the query
can express) covered the union seam only. The per-table seams
(searchTable/scanFilterIDs, and the counts twin's
finishSearchCountsPage) received the full matching set — searchWindowFor
already withholds the page bound — but trimmed it with no sort anywhere:
measured, 10 rows sort id desc limit 3 answered [000 001 002] where
[009 008 007] was owed. The store-backed seam (issueops.searchTableInTxT)
was worse off: it still pushed LIMIT into the leg SQL with no ORDER BY,
so each leg contributed an arbitrary engine-ordered subset.
Fixes, one per seam, all converging on the same rule:
- sqlbuild.Less: sortDesc now flips the id comparison (byte order, matching
the SQL the key would render if it could).
- domain/db: new sortRowsGoSide helper (the per-table copy of
idSrcPage.sortGoSide) runs before finishWindow at searchTable's wide
path, scanFilterIDs, and finishSearchCountsPage.
- issueops.searchTableInTxT: a Go-side sort leg no longer carries a SQL
LIMIT; the leg scans the complete matching set, orders it in Go, and
applies the same EffectiveSearchLimit bound after the order exists
(goSideSortAndTrim), so cap accounting is unchanged and Pattern B
hydrates only surviving rows.
- idProjection gains a comparator for the one key its payload can order
("id"), removing the merged-id-search landmine the old nil documented.
Tests: red-before-green verified on all three seams. sqlbuild unit test
pins Less; sqlmock tests pin the store-backed leg shape (no LIMIT under a
Go-side sort, bound still pushed under a SQL sort) and the ordered+trimmed
page; the domain/db suite adds per-table membership cases (asc, the
measured desc repro, and a four-route parity check: wisps-empty probe,
SkipWisps, wide path, counts twin must answer one page) plus a reversed
union page fence.
(bd-jao3t, bd-69c1a)
* fix(search): extend the Go-side-sort bound rule to the store-backed counts seam; one LessID comparator
Review round (adversarial pass on PR #5639):
F1 — runFilterSearchQueryInTx (issueops/search_counts.go) had the identical
defect the first commit fixed in searchTableInTxT: LIMIT eff pushed
unconditionally while sqlbuild.OrderBy renders no ORDER BY for "id". This
is the seam bd query '<expr>' --sort id actually reaches on the
store-shaped backends (storequerier → SearchIssuesWithCounts →
SearchIssuesWithCountsInTx), where BuildQueryPlan always pushes a bound —
so the first commit's repro claim held only for the uow/domain-db bodies.
Now the counts leg scans the complete matching set under a Go-side sort
and applies the same eff bound after ordering (goSideSortAndTrim),
leaving merge/trim/cap accounting exactly what the SQL LIMIT handed them.
bd list was already safe here (workapi.SQLLimit zeroes the pushed limit
for --sort id); only the querier path pushed a bound.
F2 — the byte-order id comparison existed in five hand-rolled copies
(Less, idProjection.less, goSideSortAndTrim, sortRowsGoSide,
idSrcPage.sortGoSide) — the exact duplication that let Less drift from
sortGoSide in the first place, and that hid F1's sixth site. All five now
call the one exported comparator, sqlbuild.LessID.
F3 — the ascending per-table suite case is annotated as the weak witness
it is (engine order for a PK-clustered prefix scan tends to be
byte-ascending already); the DESC cases are the red-before-green fences.
New shape test pins the counts legs: under --sort id neither the issues
nor the wisps counts query may carry a LIMIT (red on the previous commit:
the leg query carried LIMIT 3).
(bd-jao3t, bd-69c1a)
STORAGE LAYER ONLY, per CONTRIBUTING_PR_GUIDELINES.md. issueops.ListRequest
and issueops.ReadyRequest gain Brief; no CLI flag and no HTTP parameter sets
it yet, so no front door's payload changes. The wiring stacks on top.
WHAT IT DROPS is the free-form text: description, design, acceptance_criteria,
notes, payload and waiters are not selected, and the row comes back carrying
types.Issue.IsLitePartial. #5546 measured a 16,051-issue store where
`bd ready --json` returned 454,021 bytes and notes plus description were 76% of
it, on the call an agent makes to decide what to work on. Measured through
issueops.Reader on 100 rows sized from those per-row ratios: 362,301 bytes
full, 24,001 brief, a 93.4% reduction.
THE PRIMITIVE WAS HALF-BUILT AND UNREACHABLE. #3906 landed
types.IssueFilter.Lite, IssueSelectColumnsLite, HeavyDropList,
ScanIssueLiteFrom and Issue.IsLitePartial, then wired filter.Lite into exactly
one place: SearchIssuesInTx, the UNCOUNTED store-backed search. Every read that
answers with IssueWithCounts, which is every --json route of both commands,
both implementations of issueops.Reader, and both HTTP reads, runs the counts
mega-query instead and ignored the field. A request that set it would have come
back correct, fully hydrated, and with nothing on it to say the projection had
not happened. That silent no-op is what this closes.
THE APPLICATION POINT IS THE COUNTS MEGA-QUERY, and it is one point rather than
four because sqlbuild.SearchCountsSQL and issueops.ScanReadyWorkRowWithCounts
are already the pair every counted read passes through: the store seam's list
(search_counts.go) and ready (ready_work_counts.go), and the unit-of-work
seam's list (issue_search_counts.go) and ready (ready_work_union.go, via
fetchCountsByIDs). sqlbuild.CountsHydration is already the value that carries
hydration opt-outs across that pair on both seams, so Lite is a third field on
it rather than a new channel.
IT CHANGES THE PROJECTION'S SHAPE, which SkipLabels and SkipCounts do not, and
that is why ScanReadyWorkRowWithCounts now takes the whole hydration instead of
a bool: the value that chose the SELECT list is the value that drives the scan,
so the two cannot be given different answers. Substituting empty strings in the
heavy positions would have kept one scan path and was rejected, because the row
would carry blank text with IsLitePartial=false, which is indistinguishable
from a genuinely textless row, the ambiguity ga-clgh and CommentsOmitted
already record.
WHERE IS INDEPENDENT OF SELECT, so this bounds the size of a row and never
which rows match: DescContains, NotesContains and EmptyDesc keep selecting
exactly what they select today, and no sort key, keyset cursor or wisp-merge
dedup reads a dropped column (CompareIssuesBy orders on nine fields, none of
them heavy).
THE CLAIM REFUSES IT, which is not the same call the page makes and is the
sharper case of the rule Limit and Offset already state. A claim does not read
its row through the page's query at all: ExecuteClaimNext refetches the winning
row whole with GetIssueInTx and hydrates its counts itself, so the projection
has nothing to apply to. Accepting the field would answer a MUTATING request
with a fully hydrated row carrying IsLitePartial=false, the caller's one signal
that it did not get what it asked for, reporting that it did. The check lives
in issueops.ValidateClaimNextRequest, which is the shared rule both ready
claimers and BatchCloser's claim already run, so all four backends refuse it on
the same line.
BuildReadyCountFilter CARRIES it, which is the opposite treatment and the
opposite reason. A count reads no field of any row, so the projection cannot
make the number wrong, and the count is not always cheap enough for that to be
the end of it: the unit-of-work seam has no COUNT(*) over the ready predicate
and sizes the set by running the unbounded page and taking its length
(uow/ready_counter.go), so dropping the field there would hydrate every heavy
column of the whole ready set to answer `bd ready --brief`, a larger read than
the page it was asked for. Carrying it also keeps that builder what its doc
says it is: the listing's filter with the PAGE removed and nothing else.
TWO ASYMMETRIES on the page path, both deliberate and both stated in code:
- Brief IS carried onto the --ready arm (ReadyFilterFromIssueFilter) where
SkipLabels and SkipCounts are dropped. Those two hide a number the ready
renderings print; this drops a body no listing prints, so carrying it is
what makes `bd list --ready` and `bd ready` answer one request one way.
- types.WorkFilter carries Lite and neither of the other two, so the ready
paths get readyHydrationFor rather than hydrationFor. Before this, both
uow ready call sites passed a hardcoded CountsHydration{}, a literal that
could not carry a filter's answer at all, which is how that path would have
been missed.
CONFORMANCE at all four registered sites, one Test per case, for the two page
cases and for the claim refusal. Every field the cases assert stripped is
seeded heavy first and the premise is checked, so a backend that never
round-trips one of the six fails loudly instead of turning that clause into one
that cannot fail. Identity is asserted too (ID, Title, Status, IssueType,
Priority), or a body that stripped the row to its id would satisfy every
emptiness clause. IsLitePartial is asserted in both directions.
THE LITE COLUMN LIST IS PINNED BY ORDER, not only by membership. Both lists are
scanned positionally, so two columns of the same SQL type can be transposed
with no membership change at all: the pre-existing set-based parity test stays
green, the scan succeeds, and every row silently carries one column's value in
another's field. TestIssueSelectColumnsLite_IsFullMinusHeavyInOrder derives the
expected list by deleting HeavyDropList from IssueSelectColumns in place, which
makes the lite list a derivation rather than a second hand-maintained copy.
RED BEFORE GREEN, five sabotages, each verified and reverted:
- deleting the lite branch from CountsHydration.IssueColumns fails both page
cases with "expected 58 destination arguments in Scan, not 52";
- accepting Brief without plumbing it to either filter, the silent no-op this
change exists to close, fails both page cases on all six text fields and on
IsLitePartial, which is the run that proves the assertions are
load-bearing;
- transposing created_by and owner in IssueSelectColumnsLite fails only the
new order test, on IssueSelectColumnsLite[9] = owner, want created_by,
while the set-based test stays green, which is the gap it was added for;
- dropping the claim's refusal fails the claimer contract with "ClaimNext
with brief: error = <nil>, want ErrValidation"; and
- clearing Brief in BuildReadyCountFilter fails
TestBuildReadyCountFilterIsTheUnboundedReadyFilter, which already existed to
catch a ReadyRequest field the count builder forgets to carry and gains
Brief in its fixture here.
The recorded ready-filter golden gains "Lite": false in 62 cases because
types.WorkFilter gained the field and the test compares marshaled JSON
byte-for-byte. False is what a request that did not ask for the projection
resolves that field to, so the replay stays exact; the test comment says so.
Gates: go build ./..., go vet ./..., gofmt, internal/storage/... (dolt,
embeddeddolt with BEADS_TEST_EMBEDDED_DOLT=1, uow, issueops, sqlbuild,
domain/db), internal/workapi/..., backend/conformance/..., issueops/...,
make api-check.
Refs: #5546, #3906
Co-authored-by: sjarmak <t@t.co>
Ten agents over 216 files, one directory-balanced group each, working to a
rubric that named what to cut (duplication narration, restatement of the next
line, the governing rule echoed in every command file, throat-clearing) and
what to keep (traps, invariants, security, tracker ids, behaviour changes, and
the "why a naive simplification destroys this assertion" rationale).
Net 1,523 lines. Density against this repo's own per-directory norm on main:
cmd/bd 33.6% -> 27.3% (main 10.7%)
internal/storage 26.1% -> 23.1% (main 14.1%)
backend/conformance 34.0% -> 29.6% (main 22.8%)
internal/httpapi 30.9% -> 27.8% (main 30.6%, now below)
overall 34.7% -> 31.6%
Every agent landed at 18-25% rather than the 30% asked for, and every one gave
the same reason independently: what remained was on the keep list. Ten separate
judgements agreeing is the most useful result here — the prose is dense, but it
is mostly load-bearing, and the shortfall is the honest answer rather than a
failure to try.
TWO REVERTS, both to protect something worth more than the lines.
issueops/ is back to its pre-trim state, giving up 610 lines. The conformance
contracts cite the leaf docs BY LINE — "pins treewalker.go:52-66" — 155 times.
Trimming the cited files silently repointed those at whatever text moved into
those numbers, across agent boundaries so no agent could see it. A semantic
remapper recovered 141 of them, but 23 needed judgement and several landed on
struct fields. The leaf docs are the spec the contracts are written against;
they are the one place in this change where dense prose IS the deliverable, and
keeping them stable is worth more than the 610 lines. tree_walker_contract.go
and version_reconciler_contract.go went back for the same reason: an agent had
deleted a paragraph containing a citation, so positional restoration could not
align them.
After the reverts, exactly 5 citations point into a still-trimmed file and 2 of
those are weak — both in cmd/bd/config_guard_test.go, which this series never
touched and whose citations were already wrong on main (init.go:361 pointed at
a closing brace; the guard it names is at init.go:1499).
VERIFICATION. A guard strips every comment and all whitespace from each of the
193 modified files at HEAD and in the worktree and requires the two to be
byte-identical, so a code change cannot hide in a comment diff. It handles `//`
inside string and raw-string literals, and was tested against a deliberate code
change before being trusted. gofmt clean, go build, go vet, golangci-lint 0
issues, and the two AST-based tests that PARSE trimmed files — the timed
provider binding pin over claim.go and the serve database-source gate over
serve.go — both pass.
One real defect found while trimming: checkDatabaseSource's doc block ran into
`// sourceRoles is ...` with no blank line, so godoc attached thirty-one lines
of the wrong prose to func sourceRoles and left checkDatabaseSource
undocumented. Fixed by separating them.
STAGE ONE OF TWO. Fields, not methods, and no route flips — those are stage 2
(bd-5k21g). `issueops.ListRequest` gains MaxRows, MaxRowsSource and SkipCounts,
each with a conformance contract running at all three backends and zero parks.
MAXROWS IS THE EXACT INVERSE OF OFFSET, which is why it needed no new shape.
Offset is honored by the unit-of-work body — it renders OFFSET — and refused by
the store-backed one with a typed *ErrUnsupported. The cap points the other way:
the store search path enforces it after the scan (EnforceMaxRowsCap), and the
unit-of-work query path (internal/storage/domain/db) reads no cap at all, so a
request carrying one would come back as the FULL uncapped answer with nothing on
it to say so — a caller that set a circuit breaker handed exactly the runaway
query it was guarding against. It refuses instead, naming the operation and the
backend. `bd list --max-rows --proxied-server` has always refused one layer up
in the CLI for that reason; this is the same refusal for the callers that are
not the CLI. RunReaderListMaxRowsIsHonoredOrRefused mirrors
RunReaderOffsetIsHonoredOrRefused, including the never-silently-ignored clause,
and asserts the complement too: a cap the result set FITS inside must still come
back as an ordinary page, or a body that refused every cap out of hand would
pass.
MAXROWSSOURCE is the third field and the bead named two. It is attribution
only — it decides no answer — but `handleMaxRowsError` falls back to
"BEADS_MAX_ROWS" when it is empty, so a request that carried the number without
the source would make stage 2's route flip start telling operators to raise the
wrong knob. The contract asserts it comes back on the cap error.
SKIPCOUNTS is the SkipLabels precedent one column over: the three cardinality
joins are dropped and DependencyCount/DependentCount/CommentCount come back
zero, to be read as unknown rather than as none. It has to land WITH MaxRows —
stage 2 moves `bd list`'s TEXT path onto the role, and that path calls
SearchIssues today and projects no counts at all, so arriving on a counted page
without a way to opt out is a straight perf regression. The expensive member is
the reverse-blocker aggregate, which joins on a COALESCE key the pure-Go
analyzer cannot auto-index — the known embedded sore point, and the reason the
knob is worth having at all.
The contract's second half is the load-bearing one: NOTHING ELSE MOVES. Same
rows, same order, same has-more, same Parent — asserted by running the identical
request both ways rather than against a written-out expectation. Parent is the
tripwire: it rides the same mega-query and is not a cardinality, so an
implementation that made an aggregate join inner, or suppressed the parent join
too, would answer a strict subset and still look like it had "skipped the
counts".
WHAT THE SQL DOES. sqlbuild.SearchCountsSQL's two trailing hydration bools
become one CountsHydration value, so a path cannot pick up one opt-out and drop
the other, and both stores plus the unit-of-work repository read the pair
through one helper. A suppressed aggregate is projected as a constant IN PLACE
rather than removed: the scan reads the six extra columns positionally, so the
shape is invariant. Ready work passes the zero value at every call site —
types.WorkFilter carries neither knob, which is what "SkipLabels and SkipCounts
are not carried onto the ReadyFlag arm" means, now stated where a caller reads
it. MaxRows IS carried there; reader_ready_scope.go already claimed so, and this
is the commit that makes the claim true, since until now the field existed only
on the filter.
ONE HAND-ROLL DELETED. `bd list` built the filter through the shared builder and
then reached in and stamped filter.MaxRows/MaxRowsSource onto it — the
"build it, then change it" half-step this programme exists to make unreachable,
and the reason the cap was invisible to every implementation of Reader.List. The
cap is resolved in gatherListInput onto the request, BuildListFilter is now the
only writer of those two filter fields, and the proxied refusal reads the
already-resolved value (rejectResolvedMaxRowsUnderProxiedServer) so a malformed
BEADS_MAX_ROWS still warns exactly once.
PROVABLY INERT AT THE CLI, measured rather than argued: 22 `bd list` / `bd
children` invocations — every output mode, both max-rows knobs, the malformed
and negative cases — run against a binary built from integration/facade-wave-2
and one built from this commit, over ONE shared workspace so both read identical
data, differ on exactly one line of 617. `bd list --max-rows -1 --status bogus`
now reports the malformed flag instead of the invalid status; both exit 1, it
needs two bad arguments to reach, and it is recorded as A-lst1-3.
NO OPENAPI OPERATION AND NO NEW QUERY PARAMETER, deliberately, recorded as
A-lst1-1. `bd list`'s operation (GET /v0/beads/issues) already exists and is
already on the role, and neither field can be published on it yet: `bd serve`
always builds a provider-backed server, which REFUSES max_rows, so the parameter
would be one the reference server can never honor; and skip_counts would set
three unconditional integers on a PINNED response component to zero, leaving a
client unable to tell "no dependencies" from "not computed" — the failure
comments_omitted exists to prevent on the detail view. `make api-check`
regenerates nothing.
Verified: build, vet, gofmt clean; golangci-lint 0 issues; the two new cases at
dolt, embeddeddolt (BEADS_TEST_EMBEDDED_DOLT=1) and uow, plus the whole Reader
contract at all three; the builder golden (35 recorded cases plus two new ones
for the new fields) and the counts-SQL shape test; internal/httpapi and
make api-check; internal/storage/{issueops,domain/...}, internal/workapi,
internal/types, internal/telemetry, the root package; the embedded
`bd list`/`bd children`/max-rows CLI suites, TestProxiedServerList under
BEADS_TEST_PROXIED_SERVER=1 (including reject_max_rows_flag,
reject_max_rows_env and allow_max_rows_zero), TestProxiedLookup*, and the
protocol corpus package.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds issueops.Lifecycle — Create, Update, Close, Reopen — reached from the storage handle via
store.IssueLifecycle(), implemented once and shared by the CLI and linked library consumers. Three backends
implement it and a shared conformance suite asserts them against one spec. The bd CLI's create, update,
close and reopen now route through it.
Fixes several defects that predate this change and became visible once the write paths were forced to
agree: swallowed metadata-edit failures, a claim applied before field validation, missing value validation
on title/priority/estimate, silently dropped create edges to missing targets, infra-typed creates landing in
the wrong table, a dropped SourceRepo, closed_by_session written but never read, and a compound update
writing twice.
Behavior changes needing release notes: create with an occupied explicit id now fails instead of silently
overwriting; a compound update is one atomic operation with one hook and one version commit; reparenting
replaces all parent edges; reopening something not closed reports nothing to do.
* feat(storage): storage_class marker column + types plumbing (bd-8rifr)
Protocol v0.1 §C (bd-7vb13): types.StorageClass (versioned|unversioned|
ephemeral, empty = unset), Issue.StorageClass with C1.2 defaulting via
EffectiveStorageClass, Validate coherence with the wisp-plane flags,
migration 0060 (nullable marker on issues+wisps, no row rewrite), and the
column threaded through IssueBaseColumns/Lite, both scans, and both
insert stacks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxHUGXk944MDbtQ9xCNygJ
* feat(create,config): --storage-class flag + storage-class.<type> defaults (bd-8rifr)
Create-time resolution per C1.3: explicit flag > per-type config default >
unset; versioned normalizes to the omitted/NULL marker form (C2.4);
--storage-class ephemeral routes to the wisp plane as the spelled-out
--ephemeral (C1.4). storage-class.<type> is a yaml-plane key (dolt.auto-commit
precedent) validated at set time in both config set and set-many. Unit tests
plus an embedded end-to-end contract test (flag, config default, override,
at-rest NULL rule, wisp routing, set-time validation).
Docs: deferred to the cli-docs.pin bump per docs policy (docs describe the
pinned release; #5052/#5089 precedent) — flag help text regenerates into the
CLI reference then.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxHUGXk944MDbtQ9xCNygJ
* fix(graph): per-node storage_class in graph plans (bd-8rifr)
TestGraphApplyNodeCoversCreateIssueParams caught the gap on CI:
createIssueParams grew StorageClass but graph plans could not address it.
Wire a per-node storage_class with single-issue create's exact C1.3
precedence (explicit node value > storage-class.<type> config > unset),
including the ephemeral spelling routing to the wisp plane with the same
mutual-exclusion errors, and extend the graph_full_fields round-trip to
pin the at-rest value.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(config): validate the storage-class.<type> key suffix at set time (bd-8rifr)
Lion's #5149 should-fix: the value was validated but the key suffix was
not, so an alias (storage-class.feat) or typo (storage-class.taks) passed
set-time validation and silently never matched — create-time lookup keys
on the Normalize()d type. Reject non-canonical suffixes with the canonical
key as the hint, and unknown types with a types.custom pointer. Also the
ephemeral-constant doc nit: class-ephemeral is typically TTL-bounded, not
definitionally (no-history rows are TTL-exempt).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perf(search): filter the main table before the counts aggregate joins (#5)
* perf(search): filter the main table before the counts aggregate joins
sqlbuild.SearchCountsSQL (the counts mega-query behind bd query/search --counts,
bd list --counts, and ready-work counts) joined six aggregate derived tables
(labels, dep/rdep/comment counts, parent, deps JSON) onto every main-table row
and only then applied the WHERE filter. For the narrow ephemeral-work searches
that dominate the hot path, Dolt materialized those aggregates across the full
tables before pruning -- the dominant cost for that access pattern.
Wrap the main table in an inner subquery that applies the WHERE filter first, so
the aggregate LEFT JOINs drive off the already-narrowed set:
FROM ( SELECT i.* FROM <main> i <whereSQL> ) i
LEFT JOIN ( ... ) ...
The WHERE clause references only issue columns, never the aggregates, and the
joins are all LEFT JOINs that preserve every main row, so filtering before vs.
after the joins yields the identical row set. ORDER BY and LIMIT deliberately
stay in their original position after the joins: the ready-work counts path
passes a parameterized ORDER BY, so it must render exactly once to keep the
placeholder/arg counts aligned, and callers that do not re-sort in Go (the
domain/db stack) depend on a post-join ORDER BY for the final row order.
Tests:
- TestSearchCountsFilterBeforeJoinParity (real Dolt): the new shape returns
byte-identical rows (ids, order, dep/rdep/comment counts, parent, labels, deps
JSON) to the prior filter-after-join shape across 11 filter/order/limit/skip
scenarios. The old shape is kept in test code as the reference.
- TestSearchCountsSQLShape: asserts the WHERE sits inside the derived subquery
(before the first LEFT JOIN) and that ORDER BY / LIMIT stay after the joins and
appear exactly once.
- Full sqlbuild, issueops, dolt counts/search/ready, and domain/db suites green.
* test(search): enforce WHERE-never-references-aggregates invariant
Add TestWhereClausesNeverReferenceAggregates plus contract comments on
both WHERE producers (BuildIssueFilterClauses, BuildReadyWorkWhere) and
SearchCountsSQL, pinning the load-bearing invariant behind the
filter-before-join shape: WHERE may reference only issue columns, never
the six counts aggregate aliases (out of scope in the pre-join subquery).
Drop the stale ORDER BY/LIMIT doc paragraph (described unchanged
behavior; the arg-count invariant is already pinned by
TestSearchCountsSQLShape).
* test(search): drop the maximally-populated WHERE-aggregate test
The enumeration test only checked fields it happened to populate, so a
new filter field referencing an aggregate would pass unnoticed — false
confidence plus a standing sync cost. The invariant is already guarded
structurally: TestSearchCountsSQLShape pins the WHERE inside the pre-join
subquery, which is exactly what makes a violation an out-of-scope column
reference that Dolt errors on (fail-loud). Reword the builder/contract
comments to lean on that rather than the deleted test.
* fix(search): keep plain driver on empty WHERE; by-IDs hydration in domain/db
Review follow-ups for the filter-before-joins reshape:
- SearchCountsSQL: an empty whereSQL keeps the plain "FROM <main> i"
driver instead of wrapping it in a no-op derived table, so the
unfiltered list/counts path keeps its pre-existing shape and plans.
Pinned by a new TestSearchCountsSQLShape assertion.
- domain/db fetchCountsByIDs: hydrate via the by-IDs form instead of
hand-building "WHERE i.id IN (...)" for the predicate form - that
clause now renders inside the derived subquery, silently coupling the
caller to the subquery's internal alias. The by-IDs form also
constrains every aggregate subquery to the page; IDs are chunked
(queryBatchSize) to stay within placeholder limits, mirroring
issueops.runReadyCountsInTx.
---------
Co-authored-by: Zook Bot <275398848+zook-bot@users.noreply.github.com>
Callers that need ready work across several statuses (e.g. Gas City's
native store Ready(), which spans open/blocked/deferred/pinned/hooked/
review/testing) currently issue one GetReadyWork per status. Each call
re-pays the deferred-parents pre-query, the wisps arm, and the
per-transaction round trips — measured at 30-70ms per call on a live
server-mode store, ~7x per Ready().
WorkFilter.Statuses (OR semantics) renders a single status IN (...)
clause through BuildReadyWorkWhere and propagates to the wisp-side
IssueFilter.Statuses, so multi-status callers pay the fixed per-call
work once. Singular Status keeps precedence; both empty keeps the
legacy ('open', 'in_progress') default.
Review round 2: a conformance audit case (ReadyMultiStatusFilter) pins
the contract at the result level against both backends — real issues
and wisps across several statuses, OR semantics in one call, singular-
Status precedence, the empty default, a custom status through the same
IN clause, and the ephemeral exclusion of wisps without
IncludeEphemeral. The library guide documents Statuses, and
BuildReadyWorkWhere computes the status clause and args in one pass.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(storage): wire --label-pattern/--label-regex into the WHERE builder (ga-hqchm)
IssueFilter.LabelPattern/LabelRegex and WorkFilter.LabelPattern/LabelRegex
were parsed from `bd list --label-pattern`/`--label-regex` (and threaded
through to `bd list --ready`) but never read by BuildIssueFilterClauses or
BuildReadyWorkWhere — the SQL query builder silently ignored both fields, so
a non-matching pattern returned every issue instead of an empty/filtered set.
Wire both fields into a label-subquery clause, mirroring the existing
Labels/LabelsAny/ExcludeLabels idiom: LabelPattern converts glob syntax (*, ?)
to a SQL LIKE pattern (escaping literal %, _, and \ in the input first);
LabelRegex passes through to SQL REGEXP directly.
Confirmed via direct binary execution, not just unit tests: built the fixed
bd, ran `bd list --label-pattern 'warrant*'` against a scratch workspace with
no matching label — now returns [] instead of 5 arbitrary unrelated issues,
matching the exact repro reported on ga-hqchm.
Also strengthens the existing `label_pattern` CLI test
(cmd/bd/list_embedded_test.go), which only asserted the matching issue was
present, never that a non-matching issue was absent — which is why it kept
passing despite the dead filter — and adds a sibling `label_regex` case.
Note: the standalone `bd ready` subcommand doesn't expose --label-pattern/
--label-regex as flags at all (separate gap from this bug; `bd list --ready`,
which shares WorkFilter/BuildReadyWorkWhere, works correctly with this fix).
Filed separately, not fixed here.
* fix(storage): drop LabelRegex/SQL REGEXP wiring, narrow to --label-pattern
engdocs/ICU-POLICY.md is explicit: "bd does not use SQL REGEXP functions,
so [excluding ICU] has zero practical impact" — that sentence is the stated
justification for shipping without libicu at all. The previous commit's
LabelRegex wiring used `WHERE label REGEXP ?`, which is real SQL REGEXP
usage and directly contradicts that invariant, even though it happens to
still work under gms_pure_go (RE2 via Go's regexp instead of ICU).
--label-regex needs a different implementation (Go-side regexp.MatchString
against fetched candidates, not a SQL REGEXP push-down) to stay consistent
with the policy, and ga-hqchm's reported bug only covers --label-pattern
(glob/LIKE) in the first place. Per CONTRIBUTING.md's "one issue per PR"
rule, splitting it out rather than doing the redesign inline here.
Rebuilt and re-verified end to end with the project's own canonical build
(`go build -tags gms_pure_go`, confirmed ICU-free via `otool -L`): both
`bd list --label-pattern` and `bd list --ready --label-pattern` now filter
correctly, matching-pattern issues included, non-matching issues excluded.
* fix(ready): wire --label-pattern and --label-regex into BuildReadyWorkWhere and bd ready
Upstream #3971 landed LabelPattern/LabelRegex for the list path; the
ready path still silently ignored both. Add the LIKE ESCAPE and REGEXP
subquery clauses to BuildReadyWorkWhere mirroring filter.go, and register
both flags on the standalone bd ready subcommand.
The bd ready flag registration absorbs the surviving scope of #4884 —
credit to athosmartins for both the pattern (this PR) and regex (#4884)
ready-path intent.
Addresses the 2026-07-23 review on gastownhall/beads#4882.
Agent-Signature: claude-sonnet-5-medium on behalf of maphew
* fix(ready): honor --label-pattern/--label-regex under the proxied path
Reviewer finding: flag reads happened after the proxied early-return and
gatherReadyInput never copied them into WorkFilter, so both filters were
silently ignored under --proxied-server. Hoist the reads, wire them
through readyInput, and pin with a gatherReadyInput unit test plus a
proxied integration subtest covering both filters.
Agent-Signature: codex-gpt-5.6-terra-medium on behalf of maphew
---------
Co-authored-by: matt wilkie <maphew@gmail.com>
A claim lease is only meaningful on the replica that granted it. Every other
replica's view of the holder's liveness is stale by up to one sync interval,
so a reaper over there can revert a unit that is very much alive here. Until
now the only defense was operator discipline: partition every supervisor's
reclaim by label and hope nobody runs an unscoped sweep.
Leases now record WHICH replica granted them (leases.granted_node, ignored
migration 0016, stamped from config.NodeID(): BEADS_NODE_ID / node_id config,
falling back to the hostname). bd reclaim skips a lease that positively names
another replica — on both the snapshot SELECT and the per-row DELETE
re-check — and names each skip on stderr, since silence there is
indistinguishable from "nothing to recover" while a genuinely dead remote
worker's unit sits in_progress forever. bd reclaim --any-replica is the
operator escape hatch for a replica that is gone for good.
The guard is deliberately fail-open on unknown provenance: granted_node ''
(a row predating the column, or a deployment whose NodeID is empty) is
treated as local, so the upgrade itself can never strand a stale lease the
reaper could previously recover. A heartbeat backfills an unknown row with
the local node and never overwrites a known one.
The one path that can materialize a FOREIGN lease row locally is
RestoreLeaseOnImportInTx — lease fields round-trip the JSONL interchange — so
the granting replica rides types.Issue.LeaseGrantedNode and a restored lease
keeps the replica the snapshot names, never this node. A snapshot from an
older binary carries none, restores as unknown, and behaves exactly as before.
What the guard cannot enforce is documented instead, in both bd reclaim --help
and docs/multi-agent/federation.md: grace window > sync interval, and lease
TTL > sync interval. A TTL shorter than the cadence at which replicas exchange
state is meaningless across the bridge — the remote view is a full interval
old by construction.
doc/beads-upstream-asks.md section 5.
bd reclaim was global-only: on a federated deployment one machine's reaper
can revert a stale-looking unit that is alive on the machine that granted its
lease. Add scope filters mirroring the claim-side label surface
(bd ready --claim) so a supervisor can point its reaper at exactly the
partition it claims from.
- types.ReclaimFilter (IDs/Assignees/Labels/LabelsAny/ExcludeLabels) + IsEmpty.
- sqlbuild.BuildReclaimScopeClauses/ReclaimScopeSQL: same SQL shapes the
claim/ready path emits, alias-qualified for the reclaim JOIN. Fails closed
(1=0) on a degenerate all-empty field so a scoped sweep can never silently
widen to global.
- issueops.ReclaimExpiredLeasesInTx threads the filter into the snapshot SELECT
only; the per-row revert re-checks by id from the already-scoped snapshot.
- Threaded ReclaimExpiredLeases through the Store/domain interfaces and every
backend (dolt, embedded dolt, sql repo, use case) and both CLI paths
(direct + proxied server). A zero ReclaimFilter reproduces the exact prior
global behavior.
- CLI hard-errors a scope flag supplied with no usable value (e.g. --label
"$LANE" with LANE unset) instead of degrading to a global sweep; --json now
reports a "scoped" boolean.
- Cross-backend conformance coverage (testReclaimScoped) + sqlbuild and
flag-parsing unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(list): suppress truncation hint in --json mode + lock-in label filter tests (be-acnquj)
Filter logic for --label/--label-any/--label-pattern/--title-contains was
already correct in tree (BuildIssueFilterClauses handles each case), so the
bug PM observed was a stale-binary artifact. To prevent regression, add a
unit test in TestListLabelFiltersAcnquj that exercises every acceptance
criterion:
AC#1 -l X returns ONLY beads with label X
AC#2 --label-any A,B (OR semantics)
AC#3 --label-pattern glob
AC#4 --title-contains case-insensitive
AC#5 -l NONEXISTENT returns empty
AC#6 AND/OR composition with multiple flags
For AC#7, gate printTruncationHint on jsonOutput so the trailing
"Showing N issues; more results matched..." line is suppressed in
--json mode. TestPrintTruncationHintJSONSuppression exercises both paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(filters): wire LabelPattern/LabelRegex through SQL builder (be-ucslk4)
Both fields had been on IssueFilter and set by the cobra layer for
`bd list`, but BuildIssueFilterClauses never consumed them — so
--label-pattern and --label-regex silently returned the full result
set instead of filtering. This was the live regression in be-ucslk4
(every agent's intake hook saw the entire open queue).
Fix:
- LabelPattern: convert glob (* / ?) to a SQL LIKE pattern, escape
literal %/_/| with ESCAPE '|', and AND-join an `id IN (SELECT
issue_id FROM <labels-table> WHERE label LIKE ? ESCAPE '|')`
subquery — same shape as the existing single-label clause.
- LabelRegex: pass through verbatim to a `label REGEXP ?` subquery.
- Helper globToLikePattern is unit-tested for stars, question marks,
literal wildcards, and the escape char itself.
Tests:
- filters_test.go: TestBuildIssueFilterClauses_LabelPattern (issues +
wisps tables), TestBuildIssueFilterClauses_LabelRegex,
TestGlobToLikePattern table test. These run without Dolt — they
pin the SQL emission and arg shape so the regression cannot recur.
- list_embedded_test.go: strengthened the existing label_pattern
subtest with negative-case assertions (feature/chore must NOT
match `back*`). The prior assertion only checked positive
inclusion, so it would pass even when the filter was a no-op.
Verified end-to-end on live city: --label-pattern, --label-regex,
--label, --label-any, --title-contains all return the correct counts
after the fix; trailing 'Showing N issues; more results matched...'
hint stays off stdout under --json.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: release gate PASS for be-ucslk4-dolt-ship (be-qd2vmt)
* docs(regression): mark BUG-28 label-pattern filter as fixed (PR #3971)
TestDiscovery_LabelPatternFilterDeadCode's assertions already described the
correct (filtered) behavior, so no assertion changes were needed — it now
passes against the candidate binary where it previously failed and
documented the dead-code bug. Update the test docstring and the DISCOVERY.md
BUG-28 entry/status table/index list to reflect the fix landing in this PR
rather than describing an open bug.
Agent-Signature: claude-sonnet-medium on behalf of matt wilkie
* fix(filters): dedupe globToLikePattern, pin test against live impl
TestGlobToLikePattern was pinning a byte-identical but unreachable copy of
globToLikePattern in internal/storage/issueops/filters.go — the production
caller is internal/storage/sqlbuild/filter.go:325 via BuildIssueFilterClauses.
An escaping regression in the sqlbuild copy (%, _, | handling) could have
shipped unseen since only the dead copy was tested.
Delete the issueops copy (and its now-unused "strings" import) and move
TestGlobToLikePattern into internal/storage/sqlbuild, in-package, so it
exercises the function BuildIssueFilterClauses actually calls.
Flagged by cross-vendor paired review (claude-opus + codex-gpt-5.6-sol) on
PR #3971.
Agent-Signature: claude-sonnet-medium on behalf of matt wilkie
* test(list): add store-level --label-regex coverage
The label REGEXP ? path was previously pinned only at the clause-string
level (TestBuildIssueFilterClauses_LabelRegex); no committed test exercised
it end-to-end against the embedded store, though a reviewer's ad-hoc probe
had confirmed it works. Add a label_regex subtest beside the existing
label_pattern case in TestEmbeddedList, following the same build-tag/env
gating (BEADS_TEST_EMBEDDED_DOLT=1), covering both a single-label regex and
an alternation across unrelated labels to confirm REGEXP semantics (not just
substring/glob matching).
Flagged by cross-vendor paired review (claude-opus + codex-gpt-5.6-sol) on
PR #3971.
Agent-Signature: claude-sonnet-medium on behalf of matt wilkie
* chore: drop contributor release-gate artifact from PR
release-gates/be-ucslk4-dolt-ship-gate.md is a contributor-process artifact
that doesn't belong in the upstream tree.
Flagged by cross-vendor paired review (claude-opus + codex-gpt-5.6-sol) on
PR #3971.
Agent-Signature: claude-sonnet-medium on behalf of matt wilkie
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
* feat(comments): keyset-paged comments read (GetIssueCommentsPage)
The comments read surface was whole-thread only: GetIssueComments materializes
every comment, so a consumer that wants a page of a long thread must load it
all and page in memory (and cap it, serving long threads incomplete). Add a
keyset-paged read with a stable order and an index-backed resume predicate.
- Storage.GetIssueCommentsPage(ctx, issueID, after CommentPageCursor, limit):
(created_at ASC, id ASC) order; resume via the sargable predicate
created_at >= ? AND ((created_at > ?) OR (id > ?)); limit defaults to 100 and
caps at 500 (documented, with the len(page)<limit idiom hazard called out —
terminate on an empty page). Cursor comes from a previously read row; the
forward-only-walk semantics for backdated inserts are documented. Wisp-routed
like the whole-thread read. Page-walk equals the full read exactly, including
same-second id tie-breaks (comment ids are lowercase UUIDv7, so the CHAR(36)
collation cannot reorder them).
- Migration 0056 adds idx_comments_issue_created_id (issue_id, created_at, id)
on comments and — table-guarded, since wisp tables are clone-local and main
migrations run before the ignored chain materializes them (the 0035/0037/0053
precedent, with a matching no-op pin test) — on wisp_comments. Fresh clones
get the wisp index from the canonical clone-local schema (ignored/0001), the
same two-path design earlier wisp indexes use. An EXPLAIN guard pins the
composite index by its column-list signature, single-sourced from the exported
production query.
- AddIssueCommentInTx now truncates the returned CreatedAt to seconds so the
returned comment matches what the DATETIME(0) column stored and is usable as
a page cursor (the stored value is unchanged).
- Implemented on DoltStore and EmbeddedDoltStore; InstrumentedStorage wraps it;
beads re-exports the cursor type. Tests: page-walk==full-read (same-second
groups), tie-break, first/past-end/empty/default-limit, wisp routing, the
migration no-op pin, and the EXPLAIN guard — on both stores.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(storage): (created_at, id) keyset issue-list predicate on IssueFilter
`bd list` and the library's own callers over the public Storage surface need
true keyset paging over issues with second-granularity timestamps, which the
surface lacked: IssueFilter had CreatedBefore + Limit but no (created_at, id)
tuple cursor, so a same-second group larger than one page could not page without
dropping or duplicating its overflow.
Add an additive keyset to types.IssueFilter — AfterCreatedAt *time.Time +
AfterID string — meaning "rows strictly after the (created_at DESC, id ASC)
position (AfterCreatedAt, AfterID)". id is the primary key, so the tie-break is
total. It composes with every existing filter (CreatedBefore keeps working) and
is honored by BOTH SearchIssues and SearchIssuesWithCounts on every backend,
because it is emitted through the shared sqlbuild.BuildIssueFilterClauses path
that every storage backend goes through. Pair it with SortBy="created" for the
matching ORDER BY created_at DESC, id ASC.
The predicate is single-sourced as sqlbuild.KeysetCreatedAtIDPredicate:
(created_at <= ? AND ((created_at < ?) OR (id > ?))). This is the keyset
form the planner can drive through an index seek rather than a scan; a dolt
EXPLAIN guard pins the plan against the exact production constant, a sqlbuild
unit test pins clause/args emission, and same-second-overflow paging
regressions run on both the server and embedded Dolt backends through the
public SearchIssues.
(cherry picked from commit 167e25b3af3002939fe27d4bbfc1598fff21c586)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(storage): IssueFilter.IsBlocked filter — index-backed on every backend
`bd list`/`bd ready` and the library's own callers could not filter on the
denormalized is_blocked column server-side: the column and the
idx_issues_is_blocked(is_blocked, status) index already exist, but the
projection column was not a filter — callers had to over-fetch and post-filter
in Go. Make it one.
Purely additive: an optional IssueFilter.IsBlocked *bool (nil = any; no
signature change, no new interface method). Wired into the shared sqlbuild
filter clause as `is_blocked = ?` (bound int 0/1; the column is NOT NULL
DEFAULT 0 so no IS NULL arm), so BOTH SearchIssues(WithCounts) AND
CountIssues/CountIssuesByGroup honor it on every backend (dolt, embeddeddolt)
through the one BuildIssueFilterClauses path.
Tests: clause/arg unit test (true→1, false→0, nil→no clause); multi-backend
conformance — CountIssuesByGroup(IsBlocked=true) returns only blocked beads per
status, false the complement, nil the whole population; and a dolt EXPLAIN
guard that the filter drives the is_blocked index.
(cherry picked from commit 96c0b03024f0df08fcada9f312669f970500eea8)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(schema): move the wisp comments keyset index to a new ignored-chain migration
CI's migration-hygiene gate froze ignored/0001 (it exists on main; clones have
recorded its content hash), so the wisp twin of the 0056 comments keyset index
cannot be baked into the canonical wisp schema there. Add it as ignored/0014
instead — guarded exactly like ignored/0006 (table exists AND index absent) —
which is strictly better: the ignored chain runs per clone with its own
cursor, so fresh clones AND already-materialized workspaces both pick the
index up; 0056's guarded half remains for workspaces whose wisp tables
existed when the main chain ran. ignored/0001 reverts to its main content.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: CI Bot <ci@beads.test>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Test User <test@test.com>
* feat(metadata): MergeMetadata — atomic single-key metadata merge
SlotSet did a read-modify-write across TWO transactions: GetIssue (tx 1),
metadata[key]=value in memory, then UpdateIssue (tx 2). Two concurrent SlotSet
calls — even on DIFFERENT keys — both read the same base metadata and each
wrote back its whole blob, so the second clobbered the first's key. SlotClear
had the same bug. Add MergeMetadata, which does the read-modify-write inside
ONE transaction (so a concurrent conflict is retried and re-reads), stores a
JSON value (nested objects/arrays, not just strings), and reimplement SlotSet
and SlotClear on top of it so they inherit the atomicity.
- issueops.MergeMetadataInTx / DeleteMetadataInTx: read the metadata (routed
issues/wisps; missing issue -> ErrNotFound), merge/delete the one key, and
write the whole object back THROUGH UpdateIssueInTx — so the operation keeps
everything the old SlotSet got from UpdateIssue (the EventUpdated history
event with actor attribution, the configured metadata-schema validation, and
updated_at), now atomic. The read and write share the caller's transaction,
so a concurrent merge of a different key is retried and re-read, never
clobbered.
- Storage.MergeMetadata implemented on DoltStore (perm withRetryTx +
DOLT_ADD(issues,events) + DOLT_COMMIT; wisp path no commit) and
EmbeddedDoltStore (withConn); forwarded on InstrumentedStorage; auto-promoted
on HookFiringStore; no-op on the configStore mock. SlotSet/SlotClear rebuilt
on it (a string value marshals to a JSON string, byte-compatible with the
historical rewrite).
- Cross-backend conformance covers MergeMetadata + atomic SlotClear; the
8-goroutine concurrent-no-clobber test proves the B1 fix on real Dolt, and
a test asserts the EventUpdated event and schema validation are preserved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(concurrency): expose a per-row version on Issue reads (RowVersion)
The engine's internal row_lock column — a random non-zero int64 the engine
rewrites on every status/ownership-mutating write and on the generic update
path — was not surfaced on the public types.Issue. Consumers that wanted a
fine-grained optimistic-concurrency token had only updated_at, which is stored
at second granularity, so two same-second writes were indistinguishable. Expose
row_lock read-only as types.Issue.RowVersion so a caller can tell same-second
writes apart.
- types.Issue gains RowVersion int64 (json:"-": a Go-only concurrency token,
never on the CLI/export JSON wire — it is opaque and equality-only). Added
row_lock to the canonical sqlbuild.IssueBaseColumns and a matching scan target
in issueops.ScanIssueFrom; both engine write stacks, wisps, and the counts /
dependents parallel scans share those, so the field hydrates everywhere from a
single-point change. Read-only — no CAS/write behavior (that is a follow-up).
- Doc is precise about coverage: RowVersion changes on claim/close/unclaim and
the generic update path; it does NOT change on some direct-UPDATE paths
(restore, compaction text), which bump updated_at — so a complete change key
combines RowVersion with updated_at, status, and labels. A created row is
already non-zero (create stamps row_lock); 0 only appears on legacy rows
backfilled by migration 0054's DEFAULT 0.
- Tests: RowVersion hydrates on read and on the list path, changes on a mutating
write, distinguishes two same-second writes (identical updated_at, distinct
RowVersion), and never appears on any JSON surface.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(deps): return a typed error on dependency type-conflict
Adding a dependency between a pair that already has an edge of a different type
is a deterministic rejection, but the two engine write paths disagreed on how
they surfaced it: one returned a typed *DependencyTypeConflictError, the other a
plain fmt.Errorf string. So a caller could errors.As the conflict on one path
but had to string-match the message on the other. Make both paths return the
same typed error, and re-export the two dependency-conflict types on the public
beads package so `bd` (and the library's own call sites) classify by errors.As
instead of parsing message text.
- issueops.AddDependencyInTx now returns *domain.DependencyTypeConflictError for
the type-conflict case (byte-identical message — the struct's Error() is the
exact former string, so no string-matcher changes). The hierarchy/cross-type
case already returned *domain.DependencyHierarchyConflictError on both paths;
self-dependency and cycle are already typed sentinels.
- beads.go re-exports DependencyTypeConflictError and
DependencyHierarchyConflictError as type aliases, so errors.As against the
public type matches the value the engine returns.
- Tests assert errors.As + the four fields + the byte-identical message on both
write paths (permanent issues->issues and the wisp-source BeginTx/Commit seam),
and lock the re-exports; the deterministic conflict is returned immediately
(never retried as a transient serialization error).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(deps): emit dependency_added / dependency_removed events
`bd dep add` and `bd dep remove` recorded no event, so the events feed never
surfaced dependency changes even though the event types (types.EventDependencyAdded
/ EventDependencyRemoved) have existed since the original SQLite backend, which
emitted them on every add/remove. The emission was dropped when that backend was
removed for the Dolt-only migration and never ported to the current write paths.
Restore it on both Dolt write plumbings, matching the original shape (an events
row on the source issue: "Added dependency: <src> <type> <target>" /
"Removed dependency on <target>").
- issueops.AddDependencyInTx / RemoveDependencyInTx record the event via
RecordEventInTable on the source's event table (wisp-routed), on the genuine
add/remove only — the idempotent same-type re-add and the no-op remove of a
missing edge record nothing.
- The event is committed with the edge: DoltStore.AddDependency/RemoveDependency
DOLT_ADD `events`, and the transaction-path (RunInTransaction / batch /
graph-apply) methods mark the source's event table dirty (events or wisp_events)
so StageAndCommit commits it — without this the event row would dangle in the
working set.
- Emission is gated to the explicit dep-add/remove verb on both plumbings: the
proxied-server repo only records when the use-case sets DepInsertOpts.EmitEvent,
which create-with-deps (implicit parent-child / --deps / waits-for, via a direct
repo Insert) does not — so `bd create --parent` produces the same history on
either backend (the embedded create path via PersistDependencies likewise emits
nothing).
- Tests on both plumbings: add/remove emit exactly once and commit (verified via
`events AS OF 'HEAD'`), idempotent re-add and no-op remove emit nothing, wisp
sources route to wisp_events, and create-with-deps emits no dependency_added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(close): optional ExpectedVersion CAS on CloseIssueChecked
CloseIssueChecked guards a close on is_blocked in-transaction, but a caller that
read an issue and then closes it has no way to say "only if it hasn't changed
since I read it" — a lost-update window. Add an optional compare-and-swap on the
row's version (the RowVersion / row_lock token): when the caller supplies an
ExpectedVersion, the close proceeds only if the row still has that version, else
it refuses with a typed ErrVersionMismatch. The version read and the close share
one transaction, so it is a true CAS with no read-then-write gap.
- storage.ErrVersionMismatch (re-exported as beads.ErrVersionMismatch) and
CloseIssueOptions.ExpectedVersion *int64 (nil disables the check; a pointer so
nil "no check" is distinct from "require version 0").
- issueops.CheckVersionInTx reads row_lock (wisp-routed; ErrNotFound on a missing
row) and returns ErrVersionMismatch on divergence. CloseIssueCheckedInTx runs
it FIRST — before the is_blocked guard and before the Force short-circuit, so
Force bypasses only the guard, never the CAS (the version check is an
orthogonal precondition). A mismatch returns before any write, so the tx rolls
back leaving the issue open with no `closed` event (atomic refuse).
- The CAS has two limbs: the read-side check catches a writer that committed
before the close began, and on the retry-wrapped permanent path a commit-time
row_lock conflict is replayed by withRetryTx so the re-read refuses.
- Tests on both stores: match closes, stale refuses atomically (still open, zero
closed events), a committed concurrent write invalidates a captured version,
nil is unchanged behavior, Force does not bypass, missing id → ErrNotFound,
wisp sources route to the wisps table, and an already-closed re-close with the
post-close version stays idempotent (Unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(close): document RowVersion CAS coverage boundary on ExpectedVersion
Both adoption reviewers flagged that the public ExpectedVersion godoc and the
shared CloseIssueCheckedInTx doc could be misread as a full-row "unchanged"
check. RowVersion (row_lock) only tracks lifecycle/ownership writes
(status/assignee/started_at), so concurrent label, dependency, rename, or
is_blocked writes leave it untouched and are intentionally outside this CAS
boundary. Document that on both sites, grounded in the freshRowLock invariant.
Doc-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(cli): bd close delegates to the engine's guarded close
The embedded `bd close` path ran its own is_blocked pre-check and then closed in
a separate call — a read-then-write TOCTOU that duplicated a guard the engine
provides atomically. The verb now builds CloseIssueOptions and delegates to
CloseIssueChecked, which runs the guard and the close in ONE transaction; the
CLI's duplicated guard is deleted. --force maps to Options.Force; a blocked
close refuses with the blockers named and the --force hint.
Behavior-preserving on purpose (verified against the previous binary):
- The engine guard now uses the exact historical CLI predicate — refuse only on
a LIVE direct blocker (blocked && len(blockers) > 0), not on the bare
denormalized is_blocked column. A transitively-blocked child (is_blocked
inherited from a blocked parent, zero direct blockers) still closes without
--force, and a stale is_blocked with since-closed blockers self-heals, exactly
as before.
- An already-closed issue stays an idempotent success: it still appears in the
--json array and the text report (same output shape), still exits 0, but no
longer produces a spurious closed->closed audit entry, a no-op commit, or the
real-close side effects (molecule auto-close, newly-unblocked, claim-next).
Tests cover: direct blocker refuses atomically (still open) and closes with
--force; transitively-blocked and stale-is_blocked beads close without --force;
already-closed emits in --json alone and in a mixed batch; idempotent exit 0 —
on the engine (both stores) and through the bd binary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(update): UpdateIssueChecked — optional ExpectedVersion CAS on the update
UpdateIssue applies a field-map update with no optimistic-concurrency option, so
a caller that wants "update only if unchanged since I read it" had to
read-then-write — a lost-update window. Add UpdateIssueChecked: when the caller
supplies an ExpectedVersion (the RowVersion / row_lock token), the version read
and the update share ONE transaction, refusing with the typed ErrVersionMismatch
if the row has moved — a true compare-and-swap. Nil disables the check and is
byte-identical to UpdateIssue, which is left untouched on the hot path.
- storage.UpdateIssueOptions{ExpectedVersion *int64} (aliased as
beads.UpdateIssueOptions) + Storage.UpdateIssueChecked, mirroring the
CloseIssueChecked pattern: CheckVersionInTx runs first inside the same
transaction on every route — the permanent withRetryTx path, the wisp path,
AND the demote route (an update carrying no_history/wisp routes through the
demote flow; a pure demoteToWispInTx extraction lets the check compose
atomically with the row move — verified byte-identical to the old
DemoteToWisp).
- Implemented on DoltStore and EmbeddedDoltStore; InstrumentedStorage wraps it;
HookFiringStore fires on_update only on success (never on a refused update).
- Tests on both stores: match updates and bumps RowVersion; stale refuses
atomically (field unchanged, zero updated events); a committed concurrent
write invalidates a captured version; nil behaves exactly like UpdateIssue;
wisp match/stale; demote-route match (migrates + applies) and stale (still in
issues, unchanged); missing id → ErrNotFound; the hook decorator fires once on
success and never on refusal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(cli): proxied bd close delegates to a library checked close
The proxied-server `bd close` path duplicated the close guard in the CLI:
it queried blockers via the dependency use-case, refused with its own message,
and then closed in a separate use-case call. Move the guard into the library —
IssueUseCase gains CloseIssueChecked / CloseWispChecked, which run the same
live-direct-blocker predicate (refuse only on blocked && len(blockers) > 0)
and then the untouched unchecked close — and thin the CLI verb to a single
delegated call. Both `bd close` paths now enforce the guard in the library and
emit the byte-identical refusal: the same storage.ErrCloseBlocked sentinel
("cannot close blocked issue: <id> is blocked by [...]") plus the
"(use --force to override)" hint.
- Guard and close run on the one pinned unit-of-work connection (the whole
close batch is a single transaction), and the guard is read-only and refuses
before any write — a refused id leaves the shared batch transaction clean,
and an all-refused batch skips the commit entirely, exactly as before.
- The unchecked CloseIssue / CloseWisp are untouched: internal closes (molecule
auto-close, gate closes, bd todo done) intentionally stay unguarded.
- Behavior-parity preserved: guard ordering (validate → epic-child → gate →
blocked), exit codes, JSON/batch shape, already-closed handling, audit fields,
and commit-message contents are unchanged; a transitively-blocked bead
(is_blocked with no direct open blocker) still closes without --force, and
--force still bypasses. The guard-failure diagnostic now uses the embedded
path's generic wording (further convergence; nothing matched the old text).
- Tests: use-case level on real Dolt (direct blocker refuses with blockers
named + still open, transitive-only closes, force closes, already-closed
parity, wisp refuse/force) plus the proxied CLI integration suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(review): cross-backend dependency-event + RowVersion parity
Maintainer review fixups for PR #4911 (adopt-pr review loop):
- RowVersion: the proxied/domain create (insertIssueRow) and generic Update
paths never stamped or rewrote row_lock, so an issue written through the
proxied-server backend read back RowVersion 0 on create and unchanged on
update -- a stale CAS token that violated the types.Issue.RowVersion
contract on that backend. Stamp a fresh non-zero row_lock on insert (and the
duplicate-key rewrite) and append RowLockClause() to the generic update,
mirroring the classic issueops insert/update invariant. Adds domain/db
regression tests proving CreateIssue/CreateWisp are non-zero and
UpdateIssue/UpdateWisp change it.
- Dependency events: embedded issueops AddDependencyInTx/RemoveDependencyInTx
recorded history unconditionally, so structural create-with-deps
(bd create --parent/--deps/--waits-for) and reparent (bd update --parent)
emitted dependency_added/removed on embedded but were silent on the proxied
backend, diverging issue history for identical commands. Gate the embedded
emit on EmitEvent and thread it through AddDependencyWithOptions /
RemoveDependencyWithOptions on both the store and transaction interfaces:
the plain AddDependency/RemoveDependency are the no-event structural default
(create-with-deps, reparent), while the explicit dep verbs (bd dep add/remove,
bd link, bd relate/unrelate) pass EmitEvent -- matching the proxied
DepInsertOpts.EmitEvent gate. DoltStore stages the events table only when an
event was actually written (GH#2455). Corrects the false
"parity with PersistDependencies" comments and adds symmetric embedded/proxied
dep-event coverage for create-with-parent/deps and structural removal.
- Retain iteration-1 fixups: heal molecules on already-closed re-close,
idempotent proxied checked-close, and documented public Storage growth.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(storage): drop always-constant event param from fireDependencyHookByID (unparam)
golangci-lint (latest, v2.12.2 in CI) unparam flags fireDependencyHookByID:
its `event` parameter always receives hooks.EventUpdate. The parameter was
already effectively constant before this PR (both base call sites passed
EventUpdate); this PR adds two more call sites, all EventUpdate. Remove the
dead parameter and hardcode hooks.EventUpdate at the single runner.Run call.
All four call sites pass hooks.EventUpdate, so behavior is unchanged; go
build, go vet, and a scoped golangci-lint run pass clean.
Maintainer fixup during adoption finalize (PR #4911).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Test User <test@test.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <ci@beads.test>
Claims/heartbeats no longer mint dolt commits: lease state (holder,
granted_at, lease_expires_at, heartbeat_at) moves off the issues table
into a leases table registered in dolt_ignore. Migration 0055 +
ignored/0012 move live leases over; all full-issue query sites hydrate
via LEFT JOIN; import never clobbers a live local lease.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roll back the direct adapters, dialect layers, credential handling, dependencies, CI services, and conformance profiles for general-purpose server databases. That support adds operational and testing complexity at odds with our goal of keeping Beads as simple as possible and consuming as few resources as possible.
Keep the Storage and DoltStorage interfaces unchanged. Embedded Dolt, Dolt server, and SQLite remain supported; the MySQL wire driver remains because Dolt server uses it.
Retain legacy metadata identifiers and hidden init flags only as compatibility tombstones. Removed, unknown, or corrupt backend metadata now fails safely across CLI, SDK, bootstrap, doctor, factories, and Dolt administration before storage is opened or modified, with guidance to export using a supporting build and reinitialize via bd help init-safety.
Agent-Signature: codex-gpt-5-unknown-reasoning on behalf of Test User
BuildReadyWorkWhere emitted clauses for --label (AND-set) and
--exclude-label but none at all for --label-any, so the OR-set filter was
silently dropped by `bd ready` and `bd ready --claim` on every backend
(with or without --parent), while `bd list`/`bd search` honored it.
On an atomic claim the silent drop is dangerous rather than merely wrong:
a worker fencing itself to its own lane
(`bd ready --claim --label-any lane-a --parent epic-1`) would claim
another lane's issue and believe it was fenced. Reproduced on current
main: the new conformance case claims a lane-b issue when lane-a was
requested.
Emit an OR-set membership clause (same shape as the list path in
sqlbuild/filter.go) that AND-combines with Labels, ExcludeLabels and the
parent filter. All claim paths (Dolt, embedded-Dolt, sqlkit SQL family)
funnel through ClaimReadyIssueInTx -> GetReadyWorkInTx ->
BuildReadyWorkWhere, so the single seam fixes every backend.
Tests: a SQL-shape unit test in sqlbuild, plus a behavioral conformance
case (ClaimReadyIssueLabelFilters) asserting the claim is fenced by the
OR-set alone, AND-combined with --label and --parent, and — the safety
property — that an exhausted lane claims NOTHING rather than falling back
to unfenced ready work.
Rebase of PR #4601 (feat/choose-your-own-backend) onto current origin/main,
squashed to a single commit because the original 73-commit history built and then
removed the uowStore derisk spike incrementally, which does not replay cleanly
over the 125 commits main has since advanced (notably #4679, the ready-work
counts fix, which merged to main and is now inherited rather than re-applied).
The full pre-rebase history remains on the original branch tip for reference.
Contents (multi-backend feature + the six review follow-ups):
- bd init --backend=postgres|mysql|sqlite provisioning behind the Dolt-parity
seam; the three SQL backends share *sqlkit.Store over pluggable dialects and
carry a generated typed-unsupported shell for the non-core surface.
- Passwords in init DSNs are redacted before persisting to metadata.json (driver
parser + re-parse verification), with a negative-test matrix proving no secret
survives (pgdialect + mysql).
- The uowStore derisk spike and its report artifacts are removed; the shell
generator now lives in internal/storage/unsupportedgen (neutral name).
- Compile-time guard `var _ storage.DoltStorage = (*Store)(nil)` on each backend
plus a regen-idempotence test keep the generated skip lists honest.
- sqlkit exposes CountReadyWork so every SQL backend satisfies
storage.ReadyWorkCounter and inherits the shared page-pushed ready-counts path.
- Commercial/product names scrubbed from the feature's own code for vendor
neutrality.
Rebase reconciliations: adopted main's metadata-load hardening in the store
factories alongside the backend dispatch; combined the config interface
(GetMetadata + ReconcileVersion) and the conformance registrations (ReadyCounts +
Claim/lease); took main's versions of the shared-infra concurrency (uow
doltserver_tx) and delete-cascade (domain/issue_delete) evolutions; implemented
the interface's new UnclaimIssue on sqlkit (delegating to issueops) and added it
to the skip lists.
Verified: go build (cgo + CGO_ENABLED=0) and go vet clean; unit tests
(sqlbuild/issueops/pgdialect/mysql/unsupportedgen) and the SQLite conformance
suite green (RunAll, incl. ReadyCounts equivalence and Claim/lease on the real
sqlkit backend).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`bd ready --json` had two O(N) hotspots on main. The counts mega-query's
reverse-blocker `rc` subquery keys on COALESCE(depends_on_issue_id, …), an
expression the pure-Go GMS analyzer cannot auto-index, so it re-scanned its
whole materialization once per candidate — O(candidates × blockers). And the
"Showing X of N" total re-ran that entire mega-query UNBOUNDED just to size N.
Changes:
- Generalize one counts builder: sqlbuild.SearchCountsSQL now takes an optional
`ids []string`. Empty ids = the predicate form (unchanged callers); non-empty
ids = the page-pushed form that constrains the driver and every count
subquery to the page, bounding rc by the page instead of the full graph.
Removes the drifted SearchCountsSQL/SearchCountsByIDsSQL duplication; the
projection order is defined once and DepTargetExpr is used throughout.
- issueops.runReadyCountsInTx resolves the ready-ID page with the cheap indexed
ID query, then hydrates counts constrained to those IDs, CHUNKED by
sqlbuild.QueryBatchSize so large pages stay within backend placeholder limits
(no quadratic fallback, no wasted ID pass). The page order is restored so the
result is byte-identical to the unbounded ORDER BY … LIMIT.
- issueops.CountReadyWorkInTx sizes N with indexed COUNT(*)s over the ready
predicate for the issues AND wisps families, minus the wisp-wins overlap —
identical to len(GetReadyWorkWithCounts(Limit=0)) even when wisps exist, with
no mega-query fallback.
- Add ReadyWorkCounter to internal/storage beside the other optional
capabilities; DoltStore and EmbeddedDoltStore implement CountReadyWork.
cmd/bd/ready.go zeroes the limit at the call site and unwraps to the
capability via storage.UnwrapStore, falling back to the unbounded query only
when a store predates it.
- Extract the shared scan/dedupe loop (scanCountsRowsInTx).
- Conformance tests: page counts byte-identical to unbounded, and
CountReadyWork == len(unbounded), with fixtures for wisps and a page that
spans the chunk boundary.
Co-authored-by: Test User <test@test.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve ready-work ordering against current main after upstream moved the shared ordering into sqlbuild. Priority ready work now breaks same-priority ties with created_at ASC, and regression coverage checks both the shared helper and ClaimReadyIssue.
Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
Add --external-ref flag to `bd list` for exact-match lookup by external
reference value. Also adds --external-contains to list for parity with
bd search.
Use case: AI coding agent hooks that need to find the bead associated
with a specific GitHub branch or Jira ticket without fetching all beads
and filtering client-side.
Closes#4341
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(dolt): retry mutate+commit methods on serialization conflict (1213/1205)
ClaimIssue, ClaimReadyIssue, UpdateIssue and CloseIssue each hand-rolled
their own tx (BeginTx + DOLT_ADD/DOLT_COMMIT + Commit) outside
withRetryTx, so a concurrent writer that lost Dolt's optimistic
commit-time merge surfaced MySQL 1213/1205 as a hard failure instead of
retrying. Dolt has no real row locking — FOR UPDATE / SKIP LOCKED are
parse-only no-ops (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/)
— so retry is the only safety net under N concurrent workers. These four
are the work-queue hot paths: `bd ready --claim`, claim, update, close.
Move each body into a withRetryTx closure (which owns BeginTx and the
final Commit and retries serialization failures with backoff), preserving
the single-tx CAS/mutation + DOLT_COMMIT atomicity exactly. On retry,
ClaimReadyIssue re-scans the ready front from a fresh snapshot and claims
the next available issue.
Tested: TestEmbeddedReadyClaimConcurrent, TestEmbeddedReadyConcurrent,
TestEmbeddedCloseConcurrent, TestEmbeddedUpdateConcurrent and the
non-concurrent embedded Update/Close/Ready tests pass; build + vet clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(dolt): add concurrent work-queue drain test (Gas Station claim invariant)
N workers concurrently dequeue one shared ready-front via ClaimReadyIssue
until empty; asserts every issue claimed exactly once, none stranded, no
double-claims, and no serialization error ever surfaces (withRetryTx absorbs
the 1213/1205 from the claim-CAS cell collision). This is the regression guard
for the multi-agent claim queue that #4491 makes safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(lease): claim-TTL + heartbeat + reclaim for dead-worker recovery
Gas Station v1.1 (wy-5r9j). A claim was previously permanent: a worker that
died mid-task stranded its issue in_progress forever. This adds a lease so a
claim can expire and be recovered.
Schema (migration 0054): add lease_expires_at, heartbeat_at (DATETIME) and
row_lock (BIGINT) to issues (and wisps, for uniform issueops routing), plus
idx_issues_lease for the reaper scan. assignee doubles as the lease owner.
Lifecycle (shared issueops, so both server-mode DoltStore and EmbeddedDoltStore
get it):
- Claim stamps lease_expires_at = now + TTL (default 5m, WithLeaseTTL to
override), heartbeat_at = now, and a fresh row_lock.
- bd heartbeat <id> — owner-only; pushes the lease forward and rewrites
row_lock. Fails (ErrNotClaimable/ErrAlreadyClaimed) once the lease is gone.
- bd reclaim --older-than <dur> — reverts in_progress issues whose lease
expired more than <dur> ago back to ready (clears assignee/started_at,
records a lease_reclaimed event). Default grace 2×TTL.
The row_lock trick: Dolt has no row locking and merges concurrent commits
cell-by-cell, so a heartbeat (writes heartbeat_at) racing a reclaim (writes
status) would silently cell-merge into a zombie — an open/unassigned issue that
still carries the worker's fresh heartbeat. Every mutating path (claim, close,
update, heartbeat, reclaim) now rewrites the shared row_lock cell, forcing the
loser to a 1213/1205 serialization conflict that withRetryTx replays. All lease
paths are withRetryTx/withConn wrapped.
Tests (real Dolt): TestRowLockForcesConflictOnDisjointCellWrites proves the bug
without row_lock (silent merge to zombie) and correctness with it (conflict);
TestConcurrentHeartbeatReclaimClose is the integration race (live workers
heartbeat+close vs a continuous reaper vs dead workers) asserting exactly-once,
no lost close, no zombie; plus claim/heartbeat/reclaim unit coverage on both
backends.
Note: lease columns are second-granular DATETIME (Dolt rounds), so sub-second
TTLs are meaningless — production TTL is minutes. Heartbeat writes a Dolt
commit, so cadence must stay ≪ claim cadence to avoid history bloat.
Fixes the test coupling in TestMigration0053PromotesRigWisps, which assumed the
rig-wisp repair was always LatestVersion().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci+docs: allowlist lease stress tests; scope the row_lock invariant comment
- Add TestConcurrentWorkQueueDrain and TestConcurrentHeartbeatReclaimClose
to the check-testing-short.sh allowlist; both are real-Dolt runtime/stress
skips and were failing ci-pr-policy.
- freshRowLock doc said "EVERY mutating path" rewrites row_lock, which
overstates it. Scope it to status/ownership/lease-mutating paths (the set
the reclaim/heartbeat races care about) and document the invariant + why
orthogonal-cell writers (is_blocked, compaction, deps, rename, reopen) are
intentionally exempt, so a future writer does not silently reintroduce the
zombie-merge bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(show): surface lease_expires_at/heartbeat_at in bd show (--json + text) (wy-9cdw)
Migration 0054 added lease_expires_at/heartbeat_at (and internal row_lock) to
the issues/wisps tables for the claim/heartbeat/reclaim leasing feature, but
bd show could not observe lease state — you had to drop to raw SQL.
- types.Issue: add LeaseExpiresAt/HeartbeatAt (*time.Time, omitempty), matching
the StartedAt nullable-timestamp convention. row_lock stays unexported.
- sqlbuild.IssueSelectColumns: append lease_expires_at, heartbeat_at to the
canonical hydration list (flows to every stack via the shared constant).
- issueops.ScanIssueFrom: scan + map the two nullable columns (before the
trailing extra dests, keeping positional parity across all scan sites).
- bd show text view: emit a muted 'Lease: expires <rel> (heartbeat <rel> ago)'
line only for in_progress issues holding an active lease; add formatTimeUntil
as the forward-looking mirror of formatTimeAgo.
- types_test: assert lease keys appear when set, are omitted when nil, and that
row_lock is never surfaced.
* fix(lease): settle v54 review blockers
Address #4537 review blockers before landing v54: keep heartbeat off wisps in both backends, keep direct status/assignee updates from leaving stale leases, guard the down migration, and emit reclaim JSON as snake_case.
Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
* fix(lease): cover lease fields in tests and hooks
Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
* Fix rig bead durability (bds-vwz)
* fix: keep rig beads out of infra migration
* fix: repair durable rig wisps
Restore migration 0035 to its frozen upstream shape and add a forward repair migration for existing type=rig rows that were routed into wisps.
Keep type=rig durable while excluding it from ready work, and cover the repair path with CLI-backed schema tests plus Dolt-server tests where available.
Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of matt wilkie
* fix: make rig wisp repair migration upgrade-safe
Guard migration 0053 so older workspaces without local wisp tables no-op instead of failing, while keeping the fresh-Dolt CLI migration bundle able to exercise rig wisp repair fixtures.
Regenerate CLI reference docs for the updated infra type help text.
Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
---------
Co-authored-by: silvanshade <silvanshade@users.noreply.github.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
Move all pure SQL-text builders shared by issueops (classic) and
domain/db (proxied) into one package; both stacks now delegate, so
ready/filter/counts query semantics can no longer drift (Seam A).
Execution orchestration stays per-stack per the extract-don't-unify
decision. Canonical SQL text is classic's; the only proxied-side text
deltas are IN-list whitespace.
Adversarial re-review (bd-6dnrw.47) verified all 10 attack items with
no production divergence, and adds an import-boundary test pinning the
layering (domain must never import issueops/sqlbuild) plus a single-use
constraint note on LabelSearchPlan.MergeInto. Follow-ups filed:
bd-emrs2 (+1-overfetch convention), bd-kzeme (IN-helper consolidation).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>