beads/internal/storage/issueops/create.go
Jim Wordelman ce70b261c5
storage: add participation_generation (migration 0070) and the versioned-history write fence (#6943)
* schema+storage: restore 0068, move participation_generation to 0070 (be-blvmg)

MAYOR RULING 2026-09-29 on tracker be-waare: migration 0068 shipped to
origin/main as attribution_status-only (be-v33pa). be-h89oq's lineage had
appended participation_generation steps 4-5 to that same file while it was
still unmerged, which check-migration-hygiene.sh Check C correctly rejects
once 0068 is frozen on main. This restores 0068 to origin/main's exact
content and re-homes the participation_generation column (issues + wisps,
design section 16.3 steps 4-5, be-dt74u amendment) in a new migration 0070,
with its clone-local ignored/0028 twin for wisps. Ports back the
participation_generation write-fence consumer code (RecordVersionInTx in
internal/storage/issueops/version_history.go and its callers) and the
dualwrite contract tests that exercise it.

RED: the CLI-bundle direct-DDL override for 0070's guarded PREPARE/ALTER
pair is not yet registered in cli_migrations.go, so
TestBundleMigrationsWithPreparedALTERAreOverriddenOrJustified,
TestMigration0070AddsParticipationGeneration, and
TestAllMigrationsSQLUsesDirectDDLForKnownCLIIncompatibilities fail --
correctly: a fresh CLI-built bundle would silently omit the column
(dolthub/dolt#11345) while a runtime-migrated store has it.

git diff origin/main -- internal/storage/schema/migrations/0068_*.sql is
empty, confirming 0068 is untouched.

Agent-Signature: model=claude-sonnet-5; reasoning=high

* schema: register CLI-bundle direct-DDL override for migration 0070 (be-blvmg)

GREEN: adds cliMigration0070AddParticipationGeneration and its dispatch
entries (cliCompatibleMigrationSQL, cliSubstituteAssumesWispTables), the
same pre-2.3 Dolt CLI PREPARE/ALTER escape hatch 0067's wisps.current_revision
and 0068's attribution_status already use (dolthub/dolt#11345). Makes
TestBundleMigrationsWithPreparedALTERAreOverriddenOrJustified,
TestMigration0070AddsParticipationGeneration, and
TestAllMigrationsSQLUsesDirectDDLForKnownCLIIncompatibilities pass: a
fresh CLI-built bundle now carries participation_generation on both
issues and wisps, matching what a runtime-migrated store gets.

scripts/check-migration-hygiene.sh checks A-D pass; 0068 remains
byte-identical to origin/main.

Agent-Signature: model=claude-sonnet-5; reasoning=high

* build: bazel-sync BUILD.bazel for migration 0070 files (be-blvmg)

make bazel-sync (gazelle) picked up two files the RED commit added that
the hand-ported BUILD.bazel hadn't listed yet: 0070's embedded .up.sql in
the go_library embedsrcs and migration_0070_add_participation_generation_test.go
in the go_test srcs.

Agent-Signature: model=claude-sonnet-5; reasoning=high

* test(feat): red — Fix: enable history before fixtures in TestDualWriteDeleteNeighborRewriteRidesTheDeletesDoltCommit — PR #6943's §16.2b write fence leaves its flag-off neighbor legacy, so the delete mints 0 rows (stacked on 44418b3) (refs be-fyaja)

* feat: green — Fix: enable history before fixtures in TestDualWriteDeleteNeighborRewriteRidesTheDeletesDoltCommit — PR #6943's §16.2b write fence leaves its flag-off neighbor legacy, so the delete mints 0 rows (stacked on 44418b3)

(refs be-fyaja)

* test(conformance): pin that a wisp reaching the issues plane begins its history (be-31mnz)

Review feedback on #6943, blocker 1: with the flag on, a promoted wisp is
never versioned. None of the five RunParticipationGeneration* cases reaches
a promotion, so nothing caught it.

Two contract cases, wired on all three legs:

- RunParticipationGenerationWispPromotionBeginsHistory: create a wisp with
  the flag on, promote it, and require a stamped participation_generation,
  one version row, and +1 on the next update.
- RunParticipationGenerationWispPersistenceMoveBeginsHistory: the same
  check for a persistence-mode update (bd update --persistent). This mints
  at a different site than promotion on every leg: the guarded update's one
  end-of-update mint on dolt and embeddeddolt, and
  MoveIssuePersistenceInTx's own on uow.

"Wisp" is in the names because this contract already says "promoted" in
R2.3's sense (ProceedsForPromotedRecord), which is a different operation.
The fixture gains CreateWisp, PromoteWisp and MovePersistent closures, and
each leg's FixtureKitIsWired checks them.

Red at this commit on the embedded leg: after the arrival there are 0 rows
and participation_generation is NULL, and after the update there are still
0 rows. A flag-on durable create goes 1 -> 2 in the same store.

* fix(issueops): mint a wisp's first issues-plane version create-shaped (be-31mnz)

Review feedback on #6943, blocker 1. A promotion, or a persistence move out
of the wisps table, inserts a new issues row. A wisp never declares
participation (FR-8), so that row's participation_generation is NULL. Both
sites minted through the update-shaped RecordVersionInTx, and the section
16.2b fence read the NULL as legacy and skipped the mint. Nothing stamps the
column later, so every later update skipped too, and a record created with
versioned history on was never versioned.

These sites now use RecordVersionForCreateInTx, which stamps the column and
mints version 1, as a create on the issues plane does:

- PromoteFromEphemeralInTx (promote.go).
- MoveIssuePersistenceInTx's move out of the wisps table.
- ExecuteUpdate's one end-of-update mint (execution.go), when its
  persistence patch did that move. This is the site bd update --persistent
  reaches on the dolt and embeddeddolt legs, and the review did not list
  it. PersistenceMoveResult.EnteredIssuesPlane tells ExecuteUpdate which
  entry point to use.

One site the review named stays update-shaped on purpose:
MoveIssuePersistenceInTx's same-plane repair, which clears a stray wisp
flag on a row already in issues. That row keeps its own
participation_generation. A row created with the flag on still mints. A
legacy row stays legacy, because a repair is an ordinary write and R2.3
says an ordinary write does not promote it. A create-shaped mint there
would stamp the legacy row and start versioning it.

* docs(schema): 0070 must precede turning versioned history on (be-31mnz)

Review note on #6943 (schema skew). The write fence reads
participation_generation on every flag-on mint of an issues-plane row. On a
store that has not applied 0070 the read fails, and the mutation fails with
it instead of skipping. The migration header now says so, and says that the
flag-off path (the default) returns before the read, so a store below 0070
writes exactly as before.

Comment only. 0070 is not on main, so check-migration-hygiene's frozen-file
check does not apply to it, and no test pins its bytes.

* test(dolt): a persistence repair keeps the row's own participation (be-31mnz)

Pins why MoveIssuePersistenceInTx's same-plane repair stays update-shaped
while promotion and a move out of the wisps table became create-shaped (the
previous commit). The repair clears a stray wisp flag on a row that already
lives in issues, so the row keeps its own participation_generation.

TestPersistenceRepairKeepsTheRowsOwnParticipation runs the repair with
versioned history on for a legacy row (created with the flag off) and a
participating row (created with it on), through both repair entry points:
MoveIssuePersistenceInTx and a guarded update with a persistence patch. They
mint at different sites. The legacy row must stay NULL and mint nothing,
since R2.3 says an ordinary write does not promote it. The participating row
must mint one version. participation_generation must not change in either
case.

It fails if the repair mint is create-shaped, which would stamp and start
versioning the legacy row.

* test(dolt): a delete's rewrite of a legacy neighbor mints nothing (be-31mnz)

Folded in from bee's R1 on #6961, note 1. Once
TestDualWriteDeleteNeighborRewriteRidesTheDeletesDoltCommit started creating
its neighbor with history on, nothing covered a delete whose neighbor was
created with history off. Under the section 16.2b fence, the expected result
for that case is different: the rewrite still runs, but it mints nothing.

TestDualWriteDeleteSkipsALegacyNeighborsRewrite creates the fixtures with
history off, turns history on, and force-deletes the cited target. It then
checks four things: the rewrite ran (ReferencesUpdated=1), the neighbor has
no issue_versions rows, its participation_generation is still NULL, and
dolt_status is clean for issue_versions, store_epoch and issues.

* test(embeddeddolt): batch-created rows are stamped with the participation generation

A batch create writes its rows first and mints each created row's first
version last, after the creation-time edges land. The mint has to stay
create-shaped for rows the batch inserted (stamping participation_generation
from the store epoch) and update-shaped for rows it only upserted over
(skipped while the row is a legacy row), so the per-row isNew has to survive
the deferral to the batch's final mint pass.

Pin that on the embedded engine:

- brand-new explicit-id rows are stamped with the store epoch and get one
  version row each, and match the same input run one row at a time with the
  batch-create fast paths off (5 rows, and 250 rows across three INSERT
  statements);
- a legacy row the batch upserts over keeps a NULL participation_generation
  and gets no version row, while the new rows beside it are stamped;
- with versioned history off, no row is stamped and no version row is written.

* ci(bazel): enroll the participation-generation tests in the embedded shards

The embedded-Dolt shard manifest has to name every top-level Test* in
internal/storage/embeddeddolt, or the exact-once coverage check fails.
The two participation-generation fence tests were written before that
check landed on main, and the batch-create stamping test is new, so none
of the three is listed yet. Add them to the 15-shard block as the
generator's incremental output: three inserted lines, no existing
assignment moves.

Also list the new batch-create test file in the embeddeddolt_test srcs,
in sorted position, so the Bazel target compiles it alongside the Go
build. Its imports are already among that target's deps.

Rebase note (maintainer): on current main the embedded storage manifest
uses a 40-shard block; the generator's incremental output placed these
three tests in shards 18, 19 and 39 and re-sorted one existing shard-12
line.

* fix(issueops): keep a created row's updated_at when its first version is minted

Minting a version is bookkeeping. It must not change a user-visible column
of the row it versions.

The create-shaped mint stamps participation_generation with an UPDATE of the
issues row. issues.updated_at is ON UPDATE CURRENT_TIMESTAMP, so an UPDATE
that changes any other column and does not name updated_at rewrites it with
the clock. Before the stamp, a row's first mint left current_revision at its
column default of 1, so the UPDATE changed nothing and the clause never
fired. The stamp moves participation_generation from NULL to the store
epoch, so the clause fires on every row whose first version is minted and
replaces the updated_at an import supplies.

Set updated_at to itself in the stamp, as the blocked-state recompute in
issueops/blocked_state.go already does, so the timestamp the row was created
with survives. Add a regression that batch-creates rows with a supplied
updated_at and requires both the stamp and the unchanged timestamp.

* test(createbatchequiv): re-record the digests the participation-generation fence moves

The fence does not re-mint rows that predate the stamp: a row whose
participation_generation is still NULL gets no new version when a batch
touches it. Main mints one, so six recorded entries move.

The issue_versions entry of five scenarios loses the versions main mints
for rows that predate the stamp: three rows in small, all eleven in
depadd, all fifteen in waitsfor, four in import458 and two in
import458-reject-stale. In depadd, eq-db1 and eq-dl are the only rows
main mints more than once, and each of those mints moves their updated_at
off the fixture date. The fence skips them, so the two rows keep the
fixture date and join the issues_updated_at entry.

No other table, list or counter digest changes, and the fast and per-row
bodies already agree in every scenario. The digests were recorded with
the embedded backend. The doc comment on GoldenDirEnv now says which
entries were re-recorded.

* test(storage): waive the participation-generation contracts on the http leg

main's #7247 (6d21d337ad, "a served http conformance leg") registered a
fourth leg, http, after this PR's original base. The PR's seven
RunParticipationGeneration* entrypoints are neither wired nor waived there,
so TestEveryLegWiresEveryRoleContract/http fails on the rebased branch.

ParticipationGenerationFixture toggles the store's own versioned-history
write flag and reads the raw participation_generation column back: store
write-fence bookkeeping (design §16.2b), not a publicops role the HTTP
client serves. Waive the seven cases on http with a named reason, the same
waiver the donnabox fork carries (09e7ce05af), and keep the gap visible
until a supported HTTP surface is designed and tested.

go test ./internal/storage/ -run TestEveryLegWiresEveryRoleContract now
passes on dolt, embeddeddolt, http and uow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(issueops): decide the fence's plane by the table that holds the row

recordVersionInTx chose the plane from the row's flags (IsWisp: Ephemeral
|| NoHistory) and then read participation_generation from the issues table
only. A wisps-TABLE row can carry neither flag, so it passed the IsWisp
return and the issues read failed with sql.ErrNoRows, aborting the whole
mutation whenever versioned history is on. Two producers:

  - bd mol squash clears an ephemeral root's flag in place
    (UpdateIssue {"wisp": false}, after the root's close, in the squash's
    one transaction); update.go and the uow repository's Update both mint
    after it unconditionally.
  - import's "wisp_plane" marker (WispPlaneOverride = &true, flags false)
    routes the row to wisps, and the batch create mints it create-shaped.

On origin/main the same paths do not fail but mint a stray issue_versions
row for a wisps-table row, an FR-8 leak (wisps are never versioned).

Fix: no issues row means nothing to version. ErrNoRows on the
participation_generation read now returns nil (no version, no error) for
update- and create-shaped calls alike. That read is the right signal and
IsActiveWispInTx is not: GetIssueInTx has already resolved the row, so
ErrNoRows on the issues read can only mean "not in issues", which is
exactly the question the fence has to answer before it stamps or advances
an issues row. It costs no extra query. IsActiveWispInTx asks the inverse
question ("is it in wisps?") with one more query per mint, and it folds
every error, including a transient one, into "not a wisp".

Tests come first, as three new conformance contracts with three new
ParticipationGenerationFixture hooks (ClearWispFlag, SquashWispRoot,
CreateUnflaggedWisp). They are wired on the dolt, embeddeddolt and uow legs
as subtests of each leg's TestParticipationGenerationFence (no new
top-level test, so the shard manifests do not change) and waived on http
with participationGenerationRawStoreWaiverReason:

  RunParticipationGenerationWispFlagClearMintsNothing      (create wisp,
      UpdateIssue {"wisp": false}: no error, 0 versions)
  RunParticipationGenerationWispRootSquashMintsNothing     (close +
      UpdateIssue {"wisp": false} in one transaction: no error, 0 versions)
  RunParticipationGenerationUnflaggedWispCreateMintsNothing (CreateIssues
      with WispPlaneOverride=&true, flags false: no error, 0 versions)

Red before the fix, green after, run both ways:
  embeddeddolt (BEADS_TEST_EMBEDDED_DOLT=1): all three FAIL with
    "versioned history: read participation_generation for pg-...: sql: no
    rows in result set"; all three PASS after.
  uow: all three FAIL with the same error; all three PASS after.
  dolt (BEADS_TEST_DOLT_SERVER=local): WispRootSquash and UnflaggedWispCreate
    FAIL with the same error; both PASS after. WispFlagClear passes both
    ways on this leg. The store-level wisp update (updateWisp) runs in a
    transaction versioned history never scopes, so it never reaches the
    mint. For the same reason the leg's SquashWispRoot hook turns the events
    journal on for its one transaction: that pins the wisp writes to the
    scoped regular transaction, as in a journal-enabled workspace, so that
    case does reach the seam.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(issueops): a create stamps participation even when the row is flagged no-history

A create-shaped mint (RecordVersionForCreateInTx) of a brand-new
ISSUES-TABLE row that carries NoHistory or Ephemeral returned at the IsWisp
check before the isCreate stamp, so participation_generation stayed NULL
although the row was created with versioned history on.

Import produces exactly that row: bd-r9uce keeps a promoted no-history
wisp's durable row on the issues plane with its stray no_history=true
(import_shared.go applyImportWispPlane: WispPlaneOverride = &false). When
bd update --persistent later repairs the flag, it is a same-plane normalize
whose mint is update-shaped by design (persistence.go: "A row created with
the flag on proceeds; a legacy row stays legacy"). The NULL column made the
fence read the row as legacy, so the repair and every later update were
fenced and the row was never versioned. origin/main versions it after the
repair (0, then 1, then 2 rows).

Fix: on a create-shaped call, a flagged row that the issues table holds
gets participation_generation stamped from store_epoch.epoch, the same
source the minting stamp uses, read in the same transaction. No version is
minted and updated_at is kept (SET ..., updated_at = updated_at), then the
call returns. Update-shaped calls on flagged rows still return before any
read. The table check from the previous commit now runs ahead of the IsWisp
branch for create-shaped calls, so a flagged wisps-table row (the common
wisp create) still mints and stamps nothing. That costs a create-shaped
call on a flagged wisp one primary-key miss on issues, and only while
versioned history is on. The store_epoch read/seed moves into
readStoreEpochInTx so both stamps share it.

Tests first:
  - New conformance contract
    RunParticipationGenerationNoHistoryIssueCreateDeclaresParticipation and
    hook CreateNoHistoryIssue. Wired on dolt, embeddeddolt and uow as a
    TestParticipationGenerationFence subtest and waived on http. With
    history on: create {NoHistory: true, WispPlaneOverride: &false} gives
    participation_generation non-NULL and 0 versions. The persistent repair
    (MovePersistent) then gives 1 version, and an update-shaped edit
    (MutateExisting, a title edit) gives 2.
  - New subtest
    TestCreateBatchParticipationGeneration/NoHistoryRowIsStampedUnversionedAndKeepsTheCallersUpdatedAt
    (embeddeddolt): a batch of such rows with a supplied updated_at comes
    out stamped, unversioned, and with updated_at unchanged.

Red before the fix, green after, run both ways:
  dolt (BEADS_TEST_DOLT_SERVER=local), embeddeddolt
  (BEADS_TEST_EMBEDDED_DOLT=1), uow: the contract FAILS on all three legs
    ("participation_generation for pg-no-history-create is NULL ...", then
    "= 0 after the persistent repair, want 1", then "went from 0 to 0 ...
    want exactly +1"). It PASSES on all three after.
  embeddeddolt subtest: FAILS before the fix ("store_epoch has no row after
    the batch, so nothing stamped the 5 rows"). With the fix minus the
    updated_at = updated_at clause it FAILS on every row's updated_at. It
    PASSES with the full fix.
No new top-level Test*, so the shard manifests do not change. The
createbatchequiv goldens are unchanged: the dolt and embeddeddolt
equivalence tests pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: fix comments the rebase left stale

Comment-only.

- createbatchequiv.GoldenDirEnv said "One deliberate behavior change since
  ... nothing else moved" and then listed a second re-record, the
  participation-generation fence's. It now says there are two deliberate
  changes, puts each in its own paragraph, and scopes "nothing else moved"
  to apply.json, the only golden the first change touched.
- issueops CreateIssueResult.isNew and deferredVersion still described one
  write path. After main's #7205, a row written by deferredCreates'
  multi-row INSERT reaches finishCreateIssueInTx(..., true) directly
  (create_batch_cache.go flush) and never passes through InsertIssueIfNew
  or CreateIssueInTxWithResult. isNew is InsertIssueIfNew's result on the
  per-row path and always true for a multi-row-INSERT row. deferredVersion
  is filled from both paths.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(issueops): pin the create/update shape split in the version guard

TestEveryVersionedEntryPointMints counts RecordVersionInTx and
RecordVersionForCreateInTx as one seam. A create-shaped site that fell back
to the update-shaped entry therefore stayed green, though the fence then
reads the new row's NULL participation_generation as legacy and the record
is never versioned. The reverse case also stayed green: an update path that
reached the create-shaped entry would stamp an existing legacy row and
promote it, which R2.3 forbids.

Add the split to the structural guard. It is two small tests over the same
journalscan fixpoint and mintVersion edge discipline, with no AST changes:

- TestCreateShapedEntryPointsMintCreateShaped: the create-shaped mint sites
  (finishCreateIssueInTx, CreateIssuesInTxWithContext,
  PromoteFromEphemeralInTx, moveIssuePersistenceInTx, executeUpdate) must
  each call RecordVersionForCreateInTx themselves. A delegate is not enough
  here, because executeUpdate runs its move with mintVersion=false and a
  batch switches finishCreateIssueInTx's mint off at run time. Every
  create-shaped entry point, the delegates and #7205's
  deferredCreates.flush included, must reach it.
- TestUpdateShapedEntryPointsNeverMintCreateShaped: every other
  versionedEntryPoint must not reach RecordVersionForCreateInTx. The
  exception is ApplyBatchInTx, which runs create and update items together
  and is listed as mixed with a reason.

Mutation-checked, each probe reverted afterwards. executeUpdate's
EnteredIssuesPlane mint, PromoteFromEphemeralInTx's mint, or the batch's
deferred create-shaped mint switched to RecordVersionInTx: each fails the
create half. update.go's end-of-update mint switched to
RecordVersionForCreateInTx: fails the update half for UpdateIssueInTx and
every entry point that writes through it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(issueops): hold the dual-shape mint sites to both halves of the split

f006c871c5 listed executeUpdate, moveIssuePersistenceInTx,
finishCreateIssueInTx and CreateIssuesInTxWithContext (and the
ExecuteUpdate/MoveIssuePersistenceInTx/create wrappers that delegate to
them) as create-shaped, so TestUpdateShapedEntryPointsNeverMintCreateShaped
skipped them. Each one also mints update-shaped, on its existing-row or
same-plane branch, which is the common path: a guarded update of an
existing row, the bd update --persistent repair of an issues-plane row's
flags, a create or batch create that resolved to a row already there.
Switching that branch to RecordVersionForCreateInTx stamped a legacy row
out of legacy (R2.3), and every guard stayed green.

Split the lists by what each function actually mints:

- versionCreateShapedMintSites keeps only the pure create-shaped site,
  PromoteFromEphemeralInTx. It must call RecordVersionForCreateInTx and
  must not call RecordVersionInTx.
- versionDualShapeMintSites names the four, each with its two branches.
  TestDualShapeMintSitesCallBothEntries requires each to call BOTH entries
  directly. A delegate does not count, for the same mintVersion=false and
  DeferVersionMint reasons as before.
- versionCreateShapedEntryPoints keeps PromoteFromEphemeralInTx and
  deferredCreates.flush, which passes isNew=true for every row.
- versionDualShapeEntryPoints replaces versionMixedShapeEntryPoints. It
  holds ApplyBatchInTx and the create, persistence-move and guarded-update
  entries that delegate to a dual-shape site, each with a reason. Each must
  reach both entries transitively.

The update half skips the create-shaped and dual-shape entry points, as
before. The update-shaped branch inside a dual-shape site is now the new
test's to pin.

Mutation-checked in a throwaway worktree. Switching the update-shaped mint
to RecordVersionForCreateInTx at execution.go:301, persistence.go:97,
create.go:260 or create.go:499 now fails
TestDualShapeMintSitesCallBothEntries for that site. Before this commit,
the whole issueops package stayed green under each of these switches.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(httpclient): run the participation-generation fence over the served wire

edd3b72a1a waived all eleven ParticipationGeneration contracts on the http
leg, saying the client had "no write flag to toggle and no raw column to
read". That reason was wrong. served_dualwrite_history_test.go already
flips the reference store's VersionedHistoryConfigurer and reads its raw
columns, with the client driving the mutation. The waiver also hid the
one harness that runs the fence behind the update a connected workspace
actually sends. Every store leg's MutateExisting is store.UpdateIssue,
but the served Lifecycle.Update reaches the guarded update,
ExecuteUpdate.

Add served_participation_generation_test.go on the same pattern:

- Bound to the client: Mutate is Lifecycle.Create, MutateExisting is
  Lifecycle.Update, MovePersistent is the persistence-mode
  Lifecycle.Update that bd update --persistent sends, and PromoteWisp is
  issueops.Promote over the client's Reader/Lifecycle/Commenter, the
  entry bd promote dials over http.
- Bound to the reference store, as before: SetFlag, the CreateWisp seed,
  and the raw reads.

TestServedParticipationGenerationFence runs seven cases:
CreateWithFlagOnStampsAValue, CreateWithFlagOffLeavesLegacyNull,
SkipsUpdateOnLegacyRecord, ProceedsForPromotedRecord,
FenceIgnoresSchemaSkewHatch, WispPromotionBeginsHistory and
WispPersistenceMoveBeginsHistory. All seven pass, and no case turned up a
server-side fence bug.

Four cases stay waived, now with a reason that names the actual wire gap
(participationGenerationWireGapWaiverReason):

- WispFlagClearMintsNothing: the in-place {"wisp": false} is an untyped
  UpdateIssue, and IssuePatch.Persistence moves the row instead.
- WispRootSquashMintsNothing: it needs RunInTransaction. That and
  UpdateIssue are both on the client's unsupported allowlist (D8).
- UnflaggedWispCreateMintsNothing and
  NoHistoryIssueCreateDeclaresParticipation: both need WispPlaneOverride,
  which every create shape on the wire drops
  (roleIgnoredCreateIssueMembers).

The new file is listed in httpclient_test's srcs. The served lane deals
unpinned tests round-robin, so no shard manifest names it.

Mutation-checked in a throwaway worktree. With execution.go:301 (the
guarded update's update-shaped mint) switched to
RecordVersionForCreateInTx, SkipsUpdateOnLegacyRecord and
FenceIgnoresSchemaSkewHatch fail here ("version row count for
hpg-legacy-skip went from 0 to 1"). The embeddeddolt and uow
Participation|DualWrite|Version suites stay green under the same probe.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(conformance): an update never stamps a legacy no-history row

7c00d65fa6 made the seam write participation_generation on a flagged
(NoHistory or Ephemeral) issues-table row when the call is create-shaped.
Only the early `if IsWisp(issue) && !isCreate { return nil }` keeps an
update-shaped call out of that branch, and nothing pinned it. The new
contract and its subtest covered only the positive, create-shaped side.
With the early return dropped, an ordinary update stamped a legacy
no-history row out of legacy (R2.3), and every suite stayed green.

Add RunParticipationGenerationLegacyNoHistoryIssueStaysLegacy beside
NoHistoryIssueCreateDeclaresParticipation as its negative twin:

1. Create the no-history issues-table row with the flag OFF.
2. Turn the flag on and run MutateExisting. participation_generation must
   stay NULL, with 0 versions and current_revision unchanged.
3. Run MovePersistent, the bd update --persistent repair that clears the
   flag in place. The repair is update-shaped too, so the same three
   things must still hold.

Wired on all four legs: dolt, embeddeddolt, uow and http. On http the
case exercises the client's Lifecycle.Update and persistence update.
Their precondition, the flag-off no-history create, is seeded through
the reference store, because no client create can land that row in the
issues table. That gap is also why the case whose SUBJECT is that create
stays waived on http.

Mutation-checked in a throwaway worktree, with the early return made
`if false && IsWisp(issue) && !isCreate`. The new case fails on all four
legs and is the only case that fails:
"participation_generation for pg-no-history-legacy = 1 after an
update-shaped mutation with the flag on, want NULL" (dolt server leg,
embeddeddolt, uow and http). At tip it passes on all four.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(issueops): say which table and which mint the fence comments mean

Comment and doc corrections. No code changes.

- RecordVersionForCreateInTx's doc comment covered a flagged issues-table
  row but not a wisps-table row. A row the wisps table holds mints and
  stamps nothing, whatever its flags say: there is no issues row to stamp
  (FR-8). That holds for an ordinary wisp create and for import's
  unflagged "wisp_plane" row.
- The no-history stamp branch said the bd update --persistent repair mints
  update-shaped "(MoveIssuePersistenceInTx)". That is true only on the
  unit-of-work path. On dolt and embeddeddolt the repair goes through the
  guarded update: executeUpdate runs moveIssuePersistenceInTx with
  mintVersion=false and mints once at its own end. The fixture's
  MovePersistent doc already says this.
- VERSIONED-BEADS-WRITE-PATHS.md row 13 still credited the whole wisp
  exclusion to `RecordVersionInTx returns on IsWisp` and called every case
  a NO-OP. After ec52d29030, the table holding the row decides its plane:
  only an update-shaped call on a flagged wisp returns on IsWisp. Every
  other wisps-table row, including an unflagged one, is excluded by the
  issues-table participation_generation read, which finds no row. After
  7c00d65fa6, a create-shaped call on a flagged issues-table row is
  stamped but not versioned, so the row says that too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(conformance): the wisps-table cases check the row is still a wisp

runWispTableRowMintsNothing asserted 0 versions but never checked its own
precondition, that the wisps table still holds the row. Suppose a step
moved the row onto the issues plane through an update-shaped mint. That
mint would hit the new row's NULL participation_generation and be fenced,
so the count stayed 0 and the case passed without saying anything about
a wisps-table row.

- Add an InWispsTable hook to ParticipationGenerationFixture and bind it
  on dolt, embeddeddolt and uow as a raw wisps-table count. The
  wisps-table body now Fatals if the row is not in the wisps table after
  its steps. Each leg's KitIsWired test checks the hook. On http it
  stays nil with the other wisps-table hooks, whose cases are waived
  there.
- The dolt ClearWispFlag hook now bites. A bare store.UpdateIssue on a
  wisp goes through updateWisp, whose transaction versioned history never
  scopes. So WispFlagClearMintsNothing passed on dolt with or without the
  ec52d29030 fix, as that commit's message admits. The hook now runs the
  lone flag clear in a RunInTransaction with the events journal on: the
  journal-enabled shape the SquashWispRoot hook already uses.
- Both hooks go through a new withEventsJournal helper. It restores the
  store's previous journal setting rather than unconditionally setting it
  to false afterwards, which assumed the store started with the journal
  off.

Mutation-checked in throwaway worktrees:
- With version_history.go at ec52d29030^, dolt's WispFlagClearMintsNothing
  now fails ("read participation_generation for pg-flag-clear: sql: no
  rows in result set"). It used to pass there.
- Embedded ClearWispFlag rerouted as a flag-off persistence move followed
  by a flag-on title update: the case fails on the new precondition ("is
  not in the wisps table after flag-clear"). With the precondition
  disabled, the same probe passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(storage): review fixups for the participation_generation fence

The review of d17b716115 approved the PR with no required changes and
raised four minors and three nits. Each one below says what changed and
why, or why it did not.

Minor 1, the upsert create's new-or-existing choice was untested. The
unit-of-work create that is not CreateOnly (bd batch, bd todo, tracker
sync and mol port) inserts through an upsert, and db.Insert's existed
probe picks the mint's shape; forcing it either way shipped green. The
new uow TestParticipationGenerationUpsertCreate pins all three outcomes:
a new row is stamped and begins its history, a legacy row the upsert
resolves to stays legacy with no version, and a participating row mints
exactly one more. No served-route case is added because every served
create is CreateOnly (uow createParams), so the http wire cannot reach
this branch. The cmd/bd proxied-server batch and todo creates call this
same use-case method.

Minor 2, an update-shaped mint still lets issues.updated_at move
through ON UPDATE CURRENT_TIMESTAMP. Not changed here. The maintainer
listed it on the PR as follow-up (2), and the approving review of
d17b716115 accepted carrying it as a follow-up. It does not change
the fence's contract as pinned.

Minor 3, the depadd and waitsfor goldens had been re-recorded to zero
issue_versions rows. Their seeds run with history off, so every seeded
row was legacy and the harness stopped checking the dependency-add,
close and reopen mints. Both scenarios now stamp their seeded
issues-table rows after the seed (stampParticipation; wisps stay
unstamped). Their goldens are back to main's pre-fence files byte for
byte: depadd mints 11 version rows and waitsfor 15. The fenced legacy
outcome stays pinned by small, import458 and import458-reject-stale. A
run scenario that mints no version row now fails rather than recording
an empty digest.

Minor 4, PromoteFromEphemeralInTx always minted create-shaped. When the
issues plane already held the id (the dual presence GH#4455 records),
its upsert resolved to that row and the create-shaped mint stamped a
legacy record (R2.3). It now keeps InsertIssueIfNew's isNew and mints
update-shaped for an existing row, as finishCreateIssueInTx does. The
version guard lists it as a dual-shape mint site and entry point, and
PromoteOverDualPresenceKeepsLegacy pins the behavior at run time.

Nit 1: with versioned history off, db.Insert skips the existed probe,
since both mint entries return before reading anything. A create costs
the round trips it did before the fence.

Nit 2: the MINT COST comment now counts the fence read (about 6 to 7
round trips) and no longer says the snapshot reads current_revision;
the fence read can select it for #6379's ordinal seed. Moving that read
now would only add a column nothing consumes yet.

Nit 3: the http leg's participation-generation wire-gap waiver opens as
PENDING and names the wire mutation that would retire each case.

Validation, in a scrubbed env with -tags=gms_pure_go on embedded and
server Dolt (all pass, none skipped):
- uow: TestParticipationGenerationFence, its FixtureKitIsWired check
  and TestParticipationGenerationUpsertCreate with its 4 subtests.
- embeddeddolt: the create-batch equivalence parts (small/depadd,
  apply/waitsfor), the parts-coverage guard, the large scenarios, and
  the fence and kit.
- dolt: TestCreateBatchFastPathsMatchPerRow_Dolt, the fence,
  FenceIgnoresSchemaSkewHatch and the kit.
- httpclient: TestServedParticipationGenerationFence.
- TestEveryLegWiresEveryRoleContract, the issueops structural guards,
  and the issueops, storage and domain/db packages.

Eleven mutants, graded in 13 runs, each fail the test meant to catch
them:
- upsert always create-shaped or always update-shaped;
- the probe guard inverted;
- promote always create-shaped or always update-shaped;
- the seed left unstamped;
- the dependency-add, close or reopen mint dropped;
- two on-disk single-entry promotes.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(storage): a wisp upsert must not stamp a legacy issues twin

domain/db's Insert chose the mint shape of its upsert (the create that
is not CreateOnly) from an existence probe of the target table. For a
wisps-table insert that probe asks the wrong plane: the seam snapshots
the issues table first (GetIssueInTx), so when the issues plane already
held the id as a legacy row, the wisp create took the create-shaped
mint and stamped and versioned that untouched twin. An ordinary wisp
write promoted a legacy record, which R2.3 forbids. bd mol port and the
proxied mol and wisp writers call CreateWisp this way. It is latent
until versioned history is enabled (#6661).

A wisps-table insert never creates an issues row, so it now mints
update-shaped, without the probe. The fence leaves a legacy twin alone,
a wisp-only id still mints nothing through either entry (FR-8), and a
participating twin mints the one version main already minted for it.
Insert's comment now records that last case: the dual presence
predates the fence, and this upsert does not refuse it the way
issueops' create path does.

TestParticipationGenerationUpsertCreate gains
WispOverLegacyIssuesRowKeepsItLegacy, which fails against the previous
dispatch (twin stamped, one version row). Also from review: the version
shape guard names the dual-shape site its parse cannot see, and the
row-lock exemption comment counts all three of the seam's UPDATE issues
statements.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
2026-10-18 13:10:57 +00:00

1397 lines
52 KiB
Go

package issueops
import (
"context"
"database/sql"
"errors"
"fmt"
"strings"
"time"
gmssql "github.com/dolthub/go-mysql-server/sql"
"github.com/go-sql-driver/mysql"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/depid"
"github.com/steveyegge/beads/internal/storage/domain"
"github.com/steveyegge/beads/internal/types"
)
// BatchContext holds per-batch state read once and reused for every issue.
type BatchContext struct {
CustomStatuses []string
CustomTypes []string
ConfigPrefix string
AllowedPrefixes string
Opts storage.BatchCreateOptions
// SkipChildCounterReconcile tells CreateIssueInTxWithResult to skip its
// per-issue ReconcileChildCounters call. CreateIssuesInTxWithResult sets
// this because it already runs one slice-wide ReconcileChildCounters over
// the whole accepted batch after the per-issue loop, which covers every
// issue the per-issue call would have handled; running it again per issue
// during a batch import was 3-4 redundant round trips per hierarchical
// issue for a result the caller discards. Singular creates leave this
// false so they keep reconciling immediately, per-issue.
SkipChildCounterReconcile bool
// DeferVersionMint tells CreateIssueInTxWithResult NOT to mint the
// issue's version row itself but to report the deferral on its result.
// CreateIssuesInTxWithContext sets this because it persists the batch's
// creation-time dependency edges AFTER the per-issue loop, and the first
// version must carry that outgoing edge set: it mints once per accepted
// issue after the edges (and the blocked-state recompute) have landed.
// Singular creates leave this false and mint in place — they never run
// the dependency pass.
DeferVersionMint bool
// cache, when set, answers the per-issue presence/label lookups from one
// up-front batch read and buffers the batch's audit events for one
// bulk write (see createBatchCache). CreateIssuesInTxWithContext sets it
// on its private copy for multi-issue batches; it is never set on a
// caller's context.
cache *createBatchCache
}
// NewBatchContext reads config from the database and returns a BatchContext.
func NewBatchContext(ctx context.Context, tx DBTX, opts storage.BatchCreateOptions) (*BatchContext, error) {
customStatuses, err := GetCustomStatusesTx(ctx, tx)
if err != nil {
return nil, fmt.Errorf("failed to get custom statuses: %w", err)
}
customTypes, err := ResolveCustomTypesInTx(ctx, tx)
if err != nil {
return nil, fmt.Errorf("failed to get custom types: %w", err)
}
configPrefix, err := ReadConfigPrefix(ctx, tx)
if err != nil {
return nil, err
}
var allowedPrefixes string
_ = tx.QueryRowContext(ctx, "SELECT value FROM config WHERE `key` = ?", "allowed_prefixes").Scan(&allowedPrefixes)
return &BatchContext{
CustomStatuses: customStatuses,
CustomTypes: customTypes,
ConfigPrefix: configPrefix,
AllowedPrefixes: allowedPrefixes,
Opts: opts,
}, nil
}
func CreateIssueInTx(ctx context.Context, tx DBTX, bc *BatchContext, issue *types.Issue, actor string) error {
_, err := CreateIssueInTxWithResult(ctx, tx, bc, issue, actor)
return err
}
// CreateIssueResult reports the tables actually written by CreateIssueInTx.
type CreateIssueResult struct {
ChangedTables map[string]bool
// StaleRejected reports that the RejectStaleUpserts guard kept the stored
// row: nothing was written, and the issue's aux data must not be
// persisted by later batch stages either (bd-578h9.8).
StaleRejected bool
persistedDependencies []persistedDependency
// persistedComments are the comments this create actually inserted, carried
// up to the entry point so their journal rows land AFTER the create's. A
// consumer must never see a comment for a bead it has not been told about.
persistedComments []EventComment
// versionDeferred reports that this create reached the version seam but
// left the mint to the batch entry point (BatchContext.DeferVersionMint).
// Set only on the path that would otherwise have minted, so the batch
// mints for exactly the issues a singular create would have.
versionDeferred bool
// isNew is whether this create actually inserted a new row, as opposed
// to resolving to an already-existing one. On the per-row path
// (CreateIssueInTxWithResult) it is InsertIssueIfNew's result; a row
// written by deferredCreates' multi-row INSERT reaches
// finishCreateIssueInTx directly and is always new, since only an id
// absent from both planes is deferrable. The participation_generation
// write-fence (design §16.2b) needs this to choose
// RecordVersionForCreateInTx over RecordVersionInTx; a batch that defers
// its mint (BatchContext.DeferVersionMint) carries it here since the
// per-issue isNew local is out of scope by the time the batch mints.
isNew bool
}
type persistedDependency struct {
source string
target string
depType types.DependencyType
sourceWisp bool
}
func (r *CreateIssueResult) markChanged(table string) {
if table == "" {
return
}
if r.ChangedTables == nil {
r.ChangedTables = map[string]bool{}
}
r.ChangedTables[table] = true
}
func mergeChangedTables(dst map[string]bool, src map[string]bool) map[string]bool {
for table := range src {
if dst == nil {
dst = map[string]bool{}
}
dst[table] = true
}
return dst
}
func CreateIssueInTxWithResult(ctx context.Context, tx DBTX, bc *BatchContext, issue *types.Issue, actor string) (CreateIssueResult, error) {
var result CreateIssueResult
issueTable, eventTable, skip, err := prepareCreateIssueInTx(ctx, tx, bc, issue, actor)
if err != nil || skip {
return result, err
}
isNew, staleRejected, err := insertIssueIfNewCached(ctx, tx, issueTable, issue, bc.Opts, bc.cache)
if err != nil {
return result, err
}
if staleRejected {
// The stored row is strictly newer than this snapshot: nothing was
// written, and the snapshot's labels/comments belong to the older
// version, so they must not merge in either (bd-578h9.8).
result.StaleRejected = true
if bc.Opts.OnStaleRejected != nil {
bc.Opts.OnStaleRejected(issue.ID)
}
return result, nil
}
return finishCreateIssueInTx(ctx, tx, bc, issue, actor, issueTable, eventTable, isNew)
}
// prepareCreateIssueInTx is the part of a create before its row write:
// normalization and validation, table routing, id assignment, the create-only
// guard and the cross-plane collision check. skip reports a collision the
// options tolerate (ConflictSkip): nothing is written for the issue.
func prepareCreateIssueInTx(ctx context.Context, tx DBTX, bc *BatchContext, issue *types.Issue, actor string) (issueTable, eventTable string, skip bool, err error) {
if err := PrepareIssueForInsert(issue, bc.CustomStatuses, bc.CustomTypes); err != nil {
return "", "", false, err
}
issueTable, eventTable = TableRouting(issue)
if err := assignCreateIssueIDInTx(ctx, tx, bc, issue, actor); err != nil {
return "", "", false, err
}
if bc.Opts.CreateOnly {
if err := EnsureIssueIDAvailableInTx(ctx, tx, issue.ID); err != nil {
return "", "", false, err
}
}
skip, err = checkCrossTableIDCollisionCached(ctx, tx, issue.ID, issueTable, bc.Opts, bc.cache)
if err != nil {
return "", "", false, err
}
return issueTable, eventTable, skip, nil
}
// finishCreateIssueInTx is the part of a create after its row is written:
// lease reconciliation, the created event, labels, comments, the child
// counter, the journal entry and the version seam.
func finishCreateIssueInTx(ctx context.Context, tx DBTX, bc *BatchContext, issue *types.Issue, actor, issueTable, eventTable string, isNew bool) (CreateIssueResult, error) {
var result CreateIssueResult
result.markChanged(issueTable)
result.isNew = isNew
// Reconcile the ephemeral lease row with the accepted issue state
// (restore an imported lease / drop an orphaned one — see
// RestoreLeaseOnImportInTx). Wisps are never leased. The leases table is
// dolt_ignored, so this is deliberately not marked as a changed table.
if issueTable == "issues" {
if err := RestoreLeaseOnImportInTx(ctx, tx, issue, isNew); err != nil {
return result, err
}
}
if isNew {
if bc.cache != nil {
bc.cache.bufferEvent(eventTable, createdAuxEvent(issue.ID, actor))
} else if err := RecordEventInTable(ctx, tx, eventTable, issue.ID, types.EventCreated, actor, ""); err != nil {
return result, fmt.Errorf("failed to record event for %s: %w", issue.ID, err)
}
result.markChanged(eventTable)
}
labelResult, err := persistLabelsCached(ctx, tx, issue, actor, eventTable, bc.cache)
if err != nil {
return result, err
}
result.ChangedTables = mergeChangedTables(result.ChangedTables, labelResult.ChangedTables)
commentResult, err := PersistComments(ctx, tx, issue)
if err != nil {
return result, err
}
result.ChangedTables = mergeChangedTables(result.ChangedTables, commentResult.ChangedTables)
result.persistedComments = append(result.persistedComments, commentResult.persistedComments...)
// Advance child_counters when a singular create materializes a hierarchical
// ID (e.g. bd create --id P.8). The batch path already calls
// ReconcileChildCounters after CreateIssuesInTx; without this, explicit --id
// creates leave last_child behind the live suffix high-water mark and the
// next bd create --parent can recycle lower suffixes (GH#4750).
if isNew && !bc.SkipChildCounterReconcile {
if _, childNum, ok := ParseHierarchicalID(issue.ID); ok && childNum > 0 {
changedCounters, err := ReconcileChildCounters(ctx, tx, []*types.Issue{issue})
if err != nil {
return result, err
}
result.ChangedTables = mergeChangedTables(result.ChangedTables, changedCounters)
}
}
// Journal the create once, after labels and comments are in the row's
// transaction, so the snapshot is the complete bead. The early returns above
// (collision skip, stale reject) wrote nothing and journal nothing.
if err := RecordEventInTx(ctx, tx, EventCreate, issue.ID, actor); err != nil {
return result, err
}
// The version row is the create's LAST durable-state write. A batch
// create persists this issue's outgoing edges after this function returns,
// so it defers the mint to its own end (DeferVersionMint); a singular
// create has nothing after this point and mints here.
if bc.DeferVersionMint {
result.versionDeferred = true
} else if isNew {
if err := RecordVersionForCreateInTx(ctx, tx, issue.ID, actor); err != nil {
return result, err
}
} else if err := RecordVersionInTx(ctx, tx, issue.ID, actor); err != nil {
return result, err
}
// Creation-time comments (import/interchange carries them inline) are
// replayable content the create snapshot does NOT contain — issue hydration
// joins labels but not comments — so each inserted comment gets its own op,
// emitted after the create so a consumer is never told about a comment on a
// bead it has not seen created. Dedup hits above inserted nothing and emit
// nothing.
for i := range result.persistedComments {
if err := RecordCommentEventInTx(ctx, tx, issue.ID, &result.persistedComments[i]); err != nil {
return result, err
}
}
return result, nil
}
func assignCreateIssueIDInTx(ctx context.Context, tx DBTX, bc *BatchContext, issue *types.Issue, actor string) error {
if issue.ID == "" {
issueTable, _ := TableRouting(issue)
prefix := bc.ConfigPrefix
if issue.PrefixOverride != "" {
prefix = issue.PrefixOverride
} else if issue.IDPrefix != "" {
prefix = bc.ConfigPrefix + "-" + issue.IDPrefix
} else if IsWisp(issue) {
prefix = bc.ConfigPrefix + "-wisp"
}
var err error
issue.ID, err = GenerateIssueIDInTable(ctx, tx, issueTable, prefix, issue, actor)
if err != nil {
return fmt.Errorf("failed to generate issue ID: %w", err)
}
return nil
}
if !bc.Opts.SkipPrefixValidation {
if err := ValidateIssueIDPrefix(issue.ID, bc.ConfigPrefix, bc.AllowedPrefixes); err != nil {
return fmt.Errorf("prefix validation failed for %s: %w", issue.ID, err)
}
}
return nil
}
// CreateIssuesResult reports side effects that callers need for selective
// Dolt staging after CreateIssuesInTxWithResult returns.
type CreateIssuesResult struct {
ChangedTables map[string]bool
ChangedChildCounterTables map[string]bool
}
func (r *CreateIssuesResult) markChanged(table string) {
if table == "" {
return
}
if r.ChangedTables == nil {
r.ChangedTables = map[string]bool{}
}
r.ChangedTables[table] = true
}
func (r *CreateIssuesResult) merge(changed map[string]bool) {
r.ChangedTables = mergeChangedTables(r.ChangedTables, changed)
}
func CreateIssuesInTx(ctx context.Context, tx DBTX, issues []*types.Issue, actor string, opts storage.BatchCreateOptions) error {
_, err := CreateIssuesInTxWithResult(ctx, tx, issues, actor, opts)
return err
}
// CreateIssuesInTxWithResult creates issues and reports tables whose writes are
// only knowable after SQL reconciliation, such as child counter advances.
func CreateIssuesInTxWithResult(ctx context.Context, tx DBTX, issues []*types.Issue, actor string, opts storage.BatchCreateOptions) (CreateIssuesResult, error) {
bc, err := NewBatchContext(ctx, tx, opts)
if err != nil {
return CreateIssuesResult{}, err
}
return CreateIssuesInTxWithContext(ctx, tx, bc, issues, actor)
}
// deferredVersion carries a create's issue id and whether it was a genuinely
// new row (CreateIssueResult.isNew) from the per-issue loop in
// CreateIssuesInTxWithContext to that function's own final mint pass. The
// loop fills it from both of its write paths: a row CreateIssueInTxWithResult
// wrote on its own, and a row deferredCreates wrote in a multi-row INSERT and
// finished through finishCreateIssueInTx, which is always new.
type deferredVersion struct {
id string
isNew bool
}
// CreateIssuesInTxWithContext is CreateIssuesInTxWithResult with a
// caller-supplied BatchContext. Callers that split config reads from row
// writes across SQL sessions (doltTransaction's wisp tier) build the context
// on the session that sees in-transaction config writes and pass it here.
// The caller's bc is not modified, so one context can serve several calls.
func CreateIssuesInTxWithContext(ctx context.Context, tx DBTX, bc *BatchContext, issues []*types.Issue, actor string) (CreateIssuesResult, error) {
opts := bc.Opts
// The plane rule is a REFUSAL, never a filter. A strict batch (BatchCreator's
// contract) is refused whole before any row is written. A tolerant batch
// (SkipDependencyValidationErrors — the import mode) keeps every in-batch
// regular<->wisp edge: every row of BOTH planes is written on this one tx
// below before PersistDependenciesWithOptionsResult runs, and that pass
// resolves each target on its own plane, so the edge is writable — the
// old per-batch filter skip-reported it, and since a re-import upserts the
// rows unchanged the edge was never backfilled (wy-4276q8, wy-a648lq). A
// caller that writes the two planes on different SQL sessions
// (doltTransaction.CreateIssues) validates up front and passes one plane
// per call, so a mixed batch reaching here always shares one tx.
if !opts.SkipDependencyValidationErrors {
if err := validateCreateIssuesMixedBucketDependencies(issues); err != nil {
return CreateIssuesResult{}, err
}
}
// This function already runs a slice-wide ReconcileChildCounters below,
// covering every accepted issue; skip the redundant per-issue reconcile.
// Set the flag on a shallow copy so the caller's context keeps its own
// reconcile behavior.
batch := *bc
batch.SkipChildCounterReconcile = true
// The per-issue create leaves the version mint to this function: the
// creation-time edges below land after the per-issue loop, and the first
// version of each issue must carry them (one version per issue at
// creation, minted last).
batch.DeferVersionMint = true
if len(issues) >= createBatchCacheMinIssues && !createFastPathsDisabled.Load() {
cache, err := newCreateBatchCache(ctx, tx, issues)
if err != nil {
return CreateIssuesResult{}, err
}
batch.cache = cache
}
result := CreateIssuesResult{}
accepted := issues[:0:0]
var toVersion []deferredVersion
record := func(issue *types.Issue, issueResult CreateIssueResult) {
result.merge(issueResult.ChangedTables)
if issueResult.versionDeferred {
toVersion = append(toVersion, deferredVersion{id: issue.ID, isNew: issueResult.isNew})
}
if issueResult.StaleRejected {
return // stale snapshot: keep its deps out of the batch too
}
accepted = append(accepted, issue)
}
// Brand-new rows are written in multi-row INSERTs (see deferredCreates):
// a run of them is prepared in order, written together, then finished in
// order, before the next issue that is not one of them is touched.
var deferred deferredCreates
flush := func() error {
results, err := deferred.flush(ctx, tx, &batch, actor)
if err != nil {
return err
}
for i, issueResult := range results {
record(deferred.created[i].issue, issueResult)
}
deferred.created = deferred.created[:0]
return nil
}
for _, issue := range issues {
if batch.cache.deferrable(issue, opts) {
// deferrable requires the id absent from both planes, so the
// cross-plane collision check cannot ask for a skip here.
issueTable, eventTable, _, err := prepareCreateIssueInTx(ctx, tx, &batch, issue, actor)
if err != nil {
// The issues before this one are written (and can fail) first,
// as they would have been one at a time.
if flushErr := flush(); flushErr != nil {
return CreateIssuesResult{}, flushErr
}
return CreateIssuesResult{}, err
}
batch.cache.markInserted(issueTable, issue.ID)
deferred.created = append(deferred.created, deferredCreate{issue: issue, issueTable: issueTable, eventTable: eventTable})
continue
}
if err := flush(); err != nil {
return CreateIssuesResult{}, err
}
issueResult, err := CreateIssueInTxWithResult(ctx, tx, &batch, issue, actor)
if err != nil {
return CreateIssuesResult{}, err
}
record(issue, issueResult)
}
if err := flush(); err != nil {
return CreateIssuesResult{}, err
}
issues = accepted
// The buffered created/label_added events land before anything else in
// the batch could read the events tables (nothing in it does).
if err := batch.cache.flushEvents(ctx, tx); err != nil {
return CreateIssuesResult{}, err
}
depResult, err := PersistDependenciesWithOptionsResult(ctx, tx, issues, actor, opts)
if err != nil {
return CreateIssuesResult{}, err
}
result.merge(depResult.ChangedTables)
changedCounters, err := ReconcileChildCounters(ctx, tx, issues)
if err != nil {
return CreateIssuesResult{}, err
}
result.ChangedChildCounterTables = changedCounters
for table := range changedCounters {
result.markChanged(table)
}
issueIDs, wispIDs, err := createBlockedRecomputeIDs(ctx, tx, issues, depResult.persistedDependencies)
if err != nil {
return CreateIssuesResult{}, err
}
// The ids are this batch's rows, most of them fresh and edgeless: let the
// recompute skip the union statements for those (planRecomputeInTx).
recomputed, err := recomputeIsBlockedInTxWithResult(ctx, tx, issueIDs, wispIDs, true)
if err != nil {
return CreateIssuesResult{}, err
}
if recomputed.IssueRowsChanged {
result.markChanged("issues")
}
if recomputed.WispRowsChanged {
result.markChanged("wisps")
}
// Mint each created issue's first version LAST — after its creation-time
// edges and the blocked-state recompute — so durable_state carries the
// outgoing edge set. PersistDependenciesWithOptionsResult writes the edge
// rows directly and mints nothing itself, so this is the one version per
// issue at creation. Wisps are excluded by the seam.
for _, dv := range toVersion {
if dv.isNew {
if err := RecordVersionForCreateInTx(ctx, tx, dv.id, actor); err != nil {
return CreateIssuesResult{}, err
}
continue
}
if err := RecordVersionInTx(ctx, tx, dv.id, actor); err != nil {
return CreateIssuesResult{}, err
}
}
return result, nil
}
// CreateIssueDirtyTables returns the regular Dolt tables CreateIssueInTx may
// dirty for the given issue. Wisp tables are intentionally omitted because they
// are Dolt-ignored and cannot be staged.
func CreateIssueDirtyTables(ctx context.Context, issue *types.Issue, result CreateIssueResult) map[string]bool {
dirty := stageableChangedTables(result.ChangedTables)
if issue == nil {
return dirty
}
if parentID, childNum, ok := ParseHierarchicalID(issue.ID); ok &&
storage.HasReservedChildCounter(ctx, parentID, childNum) {
dirty["child_counters"] = true
}
return dirty
}
// CreateIssuesDirtyTables returns the regular Dolt tables CreateIssuesInTx may
// dirty, including child counters that reconciliation actually advanced.
func CreateIssuesDirtyTables(ctx context.Context, issues []*types.Issue, result CreateIssuesResult) map[string]bool {
dirty := stageableChangedTables(result.ChangedTables)
for _, issue := range issues {
if issue == nil {
continue
}
if parentID, childNum, ok := ParseHierarchicalID(issue.ID); ok &&
storage.HasReservedChildCounter(ctx, parentID, childNum) {
dirty["child_counters"] = true
}
}
return dirty
}
func stageableChangedTables(changed map[string]bool) map[string]bool {
dirty := map[string]bool{}
for table := range changed {
if table == "wisps" || strings.HasPrefix(table, "wisp_") {
continue
}
dirty[table] = true
}
return dirty
}
// ValidateCreateIssuesMixedBucketDependencies refuses same-batch dependency
// edges between regular issues and wisps. It is the plane rule of the STRICT
// batch (BatchCreator's contract, CrossPlaneBatchEdgeError): a caller whose
// regular and wisp rows are written on different SQL sessions
// (doltTransaction.CreateIssues) cannot create both ends of such an edge
// atomically, so the batch is refused whole before any row is written. It is
// deliberately NOT applied to a tolerant batch (SkipDependencyValidationErrors):
// on one tx the dependency pass runs after every row of both planes exists and
// writes the edge (see CreateIssuesInTxWithContext).
func ValidateCreateIssuesMixedBucketDependencies(issues []*types.Issue) error {
return validateCreateIssuesMixedBucketDependencies(issues)
}
func validateCreateIssuesMixedBucketDependencies(issues []*types.Issue) error {
batchWispByID := make(map[string]bool, len(issues))
hasRegular := false
hasWisp := false
for _, issue := range issues {
if issue == nil {
continue
}
isWisp := IsWisp(issue)
if isWisp {
hasWisp = true
} else {
hasRegular = true
}
if issue.ID != "" {
batchWispByID[issue.ID] = isWisp
}
}
if !hasRegular || !hasWisp {
return nil
}
for _, issue := range issues {
if issue == nil {
continue
}
for _, dep := range issue.Dependencies {
if dep == nil {
continue
}
sourceID := issue.ID
sourceIsWisp := IsWisp(issue)
if dep.IssueID != "" {
sourceID = dep.IssueID
if isWisp, ok := batchWispByID[sourceID]; ok {
sourceIsWisp = isWisp
}
}
targetIsWisp, targetInBatch := batchWispByID[dep.DependsOnID]
if targetInBatch && sourceIsWisp != targetIsWisp {
// Through the shared constructor, so the two bodies raise
// one message AND one sentinel: the role promises this
// refusal is the caller's fault, and an untyped error left
// callers classifying it by prose.
return CrossPlaneBatchEdgeError(sourceID, dep.DependsOnID)
}
}
}
return nil
}
func createBlockedRecomputeIDs(ctx context.Context, tx DBTX, issues []*types.Issue, dependencies []persistedDependency) ([]string, []string, error) {
issueSeen := make(map[string]bool, len(issues))
wispSeen := make(map[string]bool, len(issues))
issueIDs := make([]string, 0, len(issues))
wispIDs := make([]string, 0, len(issues))
add := func(id string, isWisp bool) {
if id == "" {
return
}
if isWisp {
if !wispSeen[id] {
wispSeen[id] = true
wispIDs = append(wispIDs, id)
}
return
}
if !issueSeen[id] {
issueSeen[id] = true
issueIDs = append(issueIDs, id)
}
}
for _, issue := range issues {
if issue == nil {
continue
}
isWisp := IsWisp(issue)
add(issue.ID, isWisp)
}
// The rows a created edge affects are AffectedByDepChange(ForWisp)InTx's:
// its source, plus the waiters on a parent-child edge's target, closed
// over parent-child descendants. That closure distributes over union, so
// the batch seeds every edge at once and expands once, rather than
// re-walking the descendants of each edge's source separately.
var depIssueSeed, depWispSeed, spawnerIDs []string
depIssueSeen, depWispSeen, spawnerSeen := map[string]bool{}, map[string]bool{}, map[string]bool{}
for _, dependency := range dependencies {
switch dependency.depType {
case types.DepBlocks, types.DepConditionalBlocks, types.DepWaitsFor, types.DepParentChild:
default:
continue
}
if dependency.sourceWisp {
if !depWispSeen[dependency.source] {
depWispSeen[dependency.source] = true
depWispSeed = append(depWispSeed, dependency.source)
}
} else if !depIssueSeen[dependency.source] {
depIssueSeen[dependency.source] = true
depIssueSeed = append(depIssueSeed, dependency.source)
}
if dependency.depType == types.DepParentChild && dependency.target != "" && !spawnerSeen[dependency.target] {
spawnerSeen[dependency.target] = true
spawnerIDs = append(spawnerIDs, dependency.target)
}
}
if len(depIssueSeed) > 0 || len(depWispSeed) > 0 {
if len(spawnerIDs) > 0 {
if err := loadWaitersOnSpawnerIDsInTx(ctx, tx, spawnerIDs, &depIssueSeed, depIssueSeen, &depWispSeed, depWispSeen); err != nil {
return nil, nil, fmt.Errorf("affected by created dependencies: %w", err)
}
}
affectedIssues, affectedWisps, err := expandByParentChildDescendantsInTx(ctx, tx, depIssueSeed, depWispSeed, depIssueSeen, depWispSeen)
if err != nil {
return nil, nil, fmt.Errorf("affected by created dependencies: %w", err)
}
for _, id := range affectedIssues {
add(id, false)
}
for _, id := range affectedWisps {
add(id, true)
}
}
return issueIDs, wispIDs, nil
}
// PrepareIssueForInsert normalizes timestamps, validates, and computes the content hash.
func PrepareIssueForInsert(issue *types.Issue, customStatuses, customTypes []string) error {
if err := ValidateMetadataIfConfigured(issue.Metadata); err != nil {
return fmt.Errorf("metadata validation failed for issue %s: %w", issue.ID, err)
}
// Normalize timestamps to UTC, defaulting to now.
now := time.Now().UTC()
if issue.CreatedAt.IsZero() {
issue.CreatedAt = now
} else {
issue.CreatedAt = issue.CreatedAt.UTC()
}
if issue.UpdatedAt.IsZero() {
issue.UpdatedAt = now
} else {
issue.UpdatedAt = issue.UpdatedAt.UTC()
}
// Optional timestamps (closed_at, started_at, due_at, …) may arrive from a
// JSONL import carrying a non-UTC offset; normalize them to UTC so the stored
// instant matches created_at/updated_at instead of keeping local wall-clock.
issue.NormalizeOptionalTimestampsToUTC()
// Ensure closed issues have a closed_at timestamp.
if issue.Status == types.StatusClosed && issue.ClosedAt == nil {
maxTime := issue.CreatedAt
if issue.UpdatedAt.After(maxTime) {
maxTime = issue.UpdatedAt
}
closedAt := maxTime.Add(time.Second)
issue.ClosedAt = &closedAt
}
if err := issue.ValidateWithCustom(customStatuses, customTypes); err != nil {
return fmt.Errorf("validation failed for issue %s: %w", issue.ID, err)
}
if issue.ContentHash == "" {
issue.ContentHash = issue.ComputeContentHash()
}
return nil
}
// ValidateIssueIDPrefix validates that the issue ID matches the configured prefix
// or any of the allowed_prefixes.
func ValidateIssueIDPrefix(id, prefix, allowedPrefixes string) error {
if strings.HasPrefix(id, prefix+"-") {
return nil
}
if allowedPrefixes != "" {
for _, allowed := range strings.Split(allowedPrefixes, ",") {
allowed = strings.TrimSpace(allowed)
if allowed != "" && strings.HasPrefix(id, allowed+"-") {
return nil
}
}
}
return fmt.Errorf("%w: issue ID %s does not match configured prefix %s", storage.ErrPrefixMismatch, id, prefix)
}
// ParseHierarchicalID checks if an ID is hierarchical (e.g., "bd-abc.1")
// and returns the parent ID and child number.
func ParseHierarchicalID(id string) (parentID string, childNum int, ok bool) {
lastDot := strings.LastIndex(id, ".")
if lastDot == -1 {
return "", 0, false
}
parentID = id[:lastDot]
var num int
if _, err := fmt.Sscanf(id[lastDot+1:], "%d", &num); err != nil {
return "", 0, false
}
return parentID, num, true
}
// AllWisps returns true if every issue in the slice should be routed to the
// wisps table (i.e., is ephemeral or no-history). Used to gate the fast path
// that skips Dolt versioning in batch creates.
func AllWisps(issues []*types.Issue) bool {
for _, issue := range issues {
if !issue.Ephemeral && !issue.NoHistory {
return false
}
}
return true
}
// checkCrossTableIDCollision rejects a create whose ID already lives in the
// sibling table (GH#4455). Issues and wisps share one ID space but live in
// separate tables; an ID present in both makes the merge-based lookups
// (bd ready/search) hard-error for the whole store. The target-table
// existence check in InsertIssueIfNew only sees one table, so nothing else in
// the create path closes this hole.
//
// Promotion (PromoteFromEphemeralInTx) deliberately inserts into issues while
// the wisp row still exists, then deletes the wisp — but it calls
// InsertIssueIfNew directly and never routes through here, so its transient
// dual-presence window is unaffected.
//
// ConflictSkip is the auto-import upgrade-recovery path (GH#3955), which must
// never hard-fail; there we skip the colliding row instead (lookups stay
// tolerant via GH#4163).
//
//nolint:gosec // G201: siblingTable is one of two hardcoded constants
func checkCrossTableIDCollision(ctx context.Context, tx DBTX, id, issueTable string, opts storage.BatchCreateOptions) (skip bool, err error) {
return checkCrossTableIDCollisionCached(ctx, tx, id, issueTable, opts, nil)
}
// checkCrossTableIDCollisionCached is checkCrossTableIDCollision reading the
// sibling plane's presence from cache when it covers id.
func checkCrossTableIDCollisionCached(ctx context.Context, tx DBTX, id, issueTable string, opts storage.BatchCreateOptions, cache *createBatchCache) (skip bool, err error) {
if id == "" {
return false, nil
}
siblingTable := "wisps"
if issueTable == "wisps" {
siblingTable = "issues"
}
siblingCount, err := cache.rowCount(ctx, tx, siblingTable, id)
if err != nil {
return false, fmt.Errorf("failed to check cross-table ID collision for %s: %w", id, err)
}
if siblingCount == 0 {
return false, nil
}
if opts.ConflictSkip {
return true, nil
}
return false, fmt.Errorf("cannot create %q: ID already exists in the %s table (issues and wisps share one ID space)", id, siblingTable)
}
// InsertIssueIfNew inserts the issue and returns whether it was genuinely new,
// and whether the RejectStaleUpserts guard rejected it.
//
// When opts.ConflictSkip is true and an issue with the same ID already exists,
// the row is left untouched (no UPSERT) and isNew is false. This is the
// auto-import upgrade-recovery guarantee (GH#3955): even if the emptiness
// guard in maybeAutoImportJSONL regresses, a stale issues.jsonl can never
// overwrite live rows — worst case is a no-op. Otherwise the INSERT … ON
// DUPLICATE KEY UPDATE runs, so explicit `bd import` keeps UPSERT semantics;
// with opts.RejectStaleUpserts the update half is conditional on the incoming
// row being strictly newer than the stored one (bd-pkim8, bd-hj85c).
// Staleness is decided by an explicit in-transaction read (stored updated_at
// strictly newer ⇒ rejected) so callers can skip aux persistence and count
// the row as skipped instead of created (bd-578h9.8). Equal-timestamp rows
// are deliberately NOT rejected here, even though the ODKU's
// VALUES(updated_at) > updated_at condition keeps every stored column for
// them: updated_at has second granularity, so a tie may be two distinct
// same-second updates — the local row must win the tie (an incoming row with
// an empty notes field must not wipe local notes), but its aux data
// (labels/comments/deps, which never bump updated_at) still merges
// additively (bd-hj85c).
//
//nolint:gosec // G201: table is a hardcoded constant
func InsertIssueIfNew(ctx context.Context, tx DBTX, issueTable string, issue *types.Issue, opts storage.BatchCreateOptions) (isNew bool, staleRejected bool, err error) {
return insertIssueIfNewCached(ctx, tx, issueTable, issue, opts, nil)
}
// insertIssueIfNewCached is InsertIssueIfNew reading the row's presence from
// cache when it covers the id, and recording the write in cache.
func insertIssueIfNewCached(ctx context.Context, tx DBTX, issueTable string, issue *types.Issue, opts storage.BatchCreateOptions, cache *createBatchCache) (isNew bool, staleRejected bool, err error) {
var existingCount int
if issue.ID != "" {
existingCount, err = cache.rowCount(ctx, tx, issueTable, issue.ID)
if err != nil {
return false, false, fmt.Errorf("failed to check issue existence for %s: %w", issue.ID, err)
}
}
if opts.ConflictSkip && existingCount > 0 {
return false, false, nil // issue already exists — skip, never overwrite
}
if opts.CreateOnly {
if err := insertIssueCreateOnly(ctx, tx, issueTable, issue); err != nil {
if isCreateOnlyDuplicateError(err) {
return false, false, fmt.Errorf("%w: %s", storage.ErrAlreadyExists, issue.ID)
}
return false, false, err
}
cache.markInserted(issueTable, issue.ID)
return true, false, nil
}
if opts.RejectStaleUpserts && existingCount > 0 {
var storedNewer int
if err := tx.QueryRowContext(ctx, fmt.Sprintf(`SELECT COUNT(*) FROM %s WHERE id = ? AND updated_at > ?`, issueTable), issue.ID, issue.UpdatedAt).Scan(&storedNewer); err != nil {
return false, false, fmt.Errorf("failed to check issue staleness for %s: %w", issue.ID, err)
}
if storedNewer > 0 {
// The conditional ODKU would keep every stored column anyway;
// skipping the no-op insert makes the rejection observable.
return false, true, nil
}
}
if err := insertIssueIntoTable(ctx, tx, issueTable, issue, opts.RejectStaleUpserts); err != nil {
return false, false, fmt.Errorf("failed to insert issue %s: %w", issue.ID, err)
}
cache.markInserted(issueTable, issue.ID)
return existingCount == 0, false, nil
}
func isCreateOnlyDuplicateError(err error) bool {
var mysqlError *mysql.MySQLError
if errors.As(err, &mysqlError) && mysqlError.Number == 1062 {
return true
}
return gmssql.ErrPrimaryKeyViolation.Is(err) || gmssql.ErrUniqueKeyViolation.Is(err)
}
// InsertIssueStrictInTx inserts one issue without probing either storage plane.
// Callers that move an aggregate use it while the source row necessarily still
// occupies the shared ID, so cross-plane create guards would reject a valid move.
func InsertIssueStrictInTx(ctx context.Context, tx DBTX, table string, issue *types.Issue) error {
if err := insertIssueCreateOnly(ctx, tx, table, issue); err != nil {
if isCreateOnlyDuplicateError(err) {
return fmt.Errorf("%w: %s", storage.ErrAlreadyExists, issue.ID)
}
return err
}
return nil
}
func PersistLabels(ctx context.Context, tx DBTX, issue *types.Issue, actor, eventTable string) (CreateIssueResult, error) {
return persistLabelsCached(ctx, tx, issue, actor, eventTable, nil)
}
// createdAuxEvent is the created event RecordEventInTable mints for a new
// issue, as an AuxEvent a batch can buffer.
func createdAuxEvent(issueID, actor string) AuxEvent {
return AuxEvent{
IssueID: issueID,
EventType: types.EventCreated,
Actor: actor,
OldValue: str(""),
NewValue: str(""),
}
}
// labelAddedAuxEvent is the label_added event PersistLabels records for a
// label its insert actually added.
func labelAddedAuxEvent(issueID, actor, label string) AuxEvent {
return AuxEvent{
IssueID: issueID,
EventType: types.EventLabelAdded,
Actor: actor,
Comment: str("Added label: " + label),
}
}
// persistLabelsCached is PersistLabels for a batch: with a cache it knows the
// issue's stored labels before inserting, so the labels that are new land in
// one multi-row INSERT and their label_added events are buffered, instead of
// one INSERT IGNORE (plus a RowsAffected probe) and one event write per label.
// The labels added, their order, and the events recorded are the same.
func persistLabelsCached(ctx context.Context, tx DBTX, issue *types.Issue, actor, eventTable string, cache *createBatchCache) (CreateIssueResult, error) {
if cache == nil {
return persistLabelsPerRow(ctx, tx, issue, actor, eventTable)
}
var result CreateIssueResult
if len(issue.Labels) == 0 {
return result, nil
}
labelTable := "labels"
if IsWisp(issue) {
labelTable = "wisp_labels"
}
stored, err := cache.storedLabels(ctx, tx, labelTable, issue.ID)
if err != nil {
return result, err
}
seen := make(map[string]struct{}, len(issue.Labels))
var added []string
for _, label := range issue.Labels {
if _, ok := seen[label]; ok {
continue
}
seen[label] = struct{}{}
// Same over-length refusal, in the same label order, as the per-row
// path; the enclosing transaction rolls everything back.
if err := types.CheckFieldLen("label", label); err != nil {
return result, err
}
if stored[label] {
continue
}
added = append(added, label)
}
if len(added) == 0 {
return result, nil
}
values := make([]string, len(added))
args := make([]any, 0, 2*len(added))
for i, label := range added {
values[i] = "(?, ?)"
args = append(args, issue.ID, label)
}
//nolint:gosec // G201: table is determined by ephemeral flag; only placeholders are formatted in.
if _, err := tx.ExecContext(ctx, fmt.Sprintf(`
INSERT IGNORE INTO %s (issue_id, label)
VALUES %s
`, labelTable, strings.Join(values, ", ")), args...); err != nil {
return result, fmt.Errorf("failed to insert labels for %s: %w", issue.ID, err)
}
result.markChanged(labelTable)
result.markChanged(eventTable)
for _, label := range added {
cache.addLabel(labelTable, issue.ID, label)
cache.bufferEvent(eventTable, labelAddedAuxEvent(issue.ID, actor, label))
}
return result, nil
}
func persistLabelsPerRow(ctx context.Context, tx DBTX, issue *types.Issue, actor, eventTable string) (CreateIssueResult, error) {
var result CreateIssueResult
if len(issue.Labels) == 0 {
return result, nil
}
labelTable := "labels"
if IsWisp(issue) {
labelTable = "wisp_labels"
}
seen := make(map[string]struct{}, len(issue.Labels))
for _, label := range issue.Labels {
if _, ok := seen[label]; ok {
continue
}
seen[label] = struct{}{}
// Reject an over-length label before the INSERT IGNORE, which would
// otherwise silently truncate it to VARCHAR(255). This is the create and
// import chokepoint (AddLabelInTx guards the bd label-add path). The whole
// create runs in one transaction, so returning here rolls it back — the
// issue and its labels are not persisted.
if err := types.CheckFieldLen("label", label); err != nil {
return result, err
}
//nolint:gosec // G201: table is determined by ephemeral flag
sqlResult, err := tx.ExecContext(ctx, fmt.Sprintf(`
INSERT IGNORE INTO %s (issue_id, label)
VALUES (?, ?)
`, labelTable), issue.ID, label)
if err != nil {
return result, fmt.Errorf("failed to insert label %q for %s: %w", label, issue.ID, err)
}
rowsAffected, err := sqlResult.RowsAffected()
if err != nil {
return result, fmt.Errorf("failed to check label insert result for %q on %s: %w", label, issue.ID, err)
}
if rowsAffected == 0 {
continue
}
result.markChanged(labelTable)
if err := InsertDerivedEvent(ctx, tx, eventTable, labelAddedAuxEvent(issue.ID, actor, label)); err != nil {
return result, fmt.Errorf("failed to record label event %q for %s: %w", label, issue.ID, err)
}
result.markChanged(eventTable)
}
return result, nil
}
func PersistComments(ctx context.Context, tx DBTX, issue *types.Issue) (CreateIssueResult, error) {
var result CreateIssueResult
if len(issue.Comments) == 0 {
return result, nil
}
commentTable := "comments"
if IsWisp(issue) {
commentTable = "wisp_comments"
}
for _, comment := range issue.Comments {
createdAt := comment.CreatedAt
if createdAt.IsZero() {
// No supplied timestamp: this is a live comment, so stamp it the
// same way AddIssueComment does — one second past the issue's
// newest comment when the clock second would collide. Otherwise
// several such comments in one create share a second and read back
// in content-digest order rather than the order they were listed.
stamped, err := NextLiveCommentTime(ctx, tx, commentTable, issue.ID, time.Now())
if err != nil {
return result, fmt.Errorf("failed to insert comment for %s: %w", issue.ID, err)
}
createdAt = stamped
}
createdAtText := FormatAuxTime(createdAt)
if comment.ID == "" {
// No incoming id (fresh comment): content-derived id, collapsing
// onto an identical existing row exactly like the import dedup.
id, existed, err := InsertDerivedComment(ctx, tx, commentTable, issue.ID, comment.Author, comment.Text, createdAtText)
if err != nil {
return result, fmt.Errorf("failed to insert comment for %s: %w", issue.ID, err)
}
comment.ID = id
if !existed {
result.markChanged(commentTable)
result.persistedComments = append(result.persistedComments, EventComment{
ID: id, Author: comment.Author, Text: comment.Text, CreatedAt: createdAt, Source: CommentSourceStructured,
})
}
continue
}
// Incoming id (import/interchange): preserve it, with the historical
// existence check preventing duplicates on re-import.
var exists int
//nolint:gosec // G201: table is determined by ephemeral flag
if err := tx.QueryRowContext(ctx, fmt.Sprintf(`
SELECT COUNT(*) FROM %s
WHERE issue_id = ? AND author = ? AND created_at = ? AND text = ?
`, commentTable), issue.ID, comment.Author, createdAtText, comment.Text).Scan(&exists); err != nil {
return result, fmt.Errorf("failed to check comment existence for %s: %w", issue.ID, err)
}
if exists > 0 {
continue
}
//nolint:gosec // G201: table is determined by ephemeral flag
_, err := tx.ExecContext(ctx, fmt.Sprintf(`
INSERT INTO %s (id, issue_id, author, text, created_at)
VALUES (?, ?, ?, ?, ?)
`, commentTable), comment.ID, issue.ID, comment.Author, comment.Text, createdAtText)
if err != nil {
return result, fmt.Errorf("failed to insert comment for %s: %w", issue.ID, err)
}
result.markChanged(commentTable)
result.persistedComments = append(result.persistedComments, EventComment{
ID: comment.ID, Author: comment.Author, Text: comment.Text, CreatedAt: createdAt, Source: CommentSourceStructured,
})
}
return result, nil
}
func PersistDependencies(ctx context.Context, tx DBTX, issues []*types.Issue, actor string) error {
_, err := PersistDependenciesWithResult(ctx, tx, issues, actor)
return err
}
func PersistDependenciesWithResult(ctx context.Context, tx DBTX, issues []*types.Issue, actor string) (CreateIssueResult, error) {
return PersistDependenciesWithOptionsResult(ctx, tx, issues, actor, storage.BatchCreateOptions{})
}
func PersistDependenciesWithOptionsResult(ctx context.Context, tx DBTX, issues []*types.Issue, actor string, opts storage.BatchCreateOptions) (CreateIssueResult, error) {
var result CreateIssueResult
type pendingDependency struct {
dep *types.Dependency
depTable string
}
var pending []pendingDependency
var deps []*types.Dependency
for _, issue := range issues {
for _, dep := range issue.Dependencies {
// Default IssueID to the owning issue when not pre-set (e.g.,
// markdown bulk create where the ID is auto-generated).
if dep.IssueID == "" {
dep.IssueID = issue.ID
}
deps = append(deps, dep)
}
}
// A multi-edge batch answers its per-edge routing, presence and graph
// reads from batch reads (depBatchLookups); nil keeps the per-edge reads.
var lookups *depBatchLookups
if len(deps) >= depBatchLookupsMinDeps && !createFastPathsDisabled.Load() {
var err error
if lookups, err = newDepBatchLookups(ctx, tx, deps); err != nil {
return result, err
}
}
for _, dep := range deps {
depTable := "dependencies"
if lookups.isWisp(ctx, tx, dep.IssueID) {
depTable = "wisp_dependencies"
}
pending = append(pending, pendingDependency{dep: dep, depTable: depTable})
}
// Persist hierarchy first so blocking edges in the same import see the full
// planned ancestry. The enclosing create transaction rolls this phase back
// if a later dependency is invalid.
for phase := 0; phase < 2; phase++ {
parentPhase := phase == 0
for _, item := range pending {
dep := item.dep
if (dep.Type == types.DepParentChild) != parentPhase {
continue
}
isCrossPrefix := types.ExtractPrefix(dep.IssueID) != types.ExtractPrefix(dep.DependsOnID)
kind := lookups.classify(ctx, tx, dep, isCrossPrefix)
if kind != DepTargetExternal {
exists, err := lookups.targetExists(ctx, tx, kind, dep.DependsOnID)
if err != nil {
return result, fmt.Errorf("failed to check dependency target %s for %s: %w", dep.DependsOnID, dep.IssueID, err)
}
if !exists {
recordSkippedDependency(opts, dep, "target not found")
continue
}
}
if kind != DepTargetExternal && types.ExtractPrefix(dep.IssueID) == types.ExtractPrefix(dep.DependsOnID) {
if err := lookups.checkHierarchy(ctx, tx, dep); err != nil {
if opts.SkipDependencyValidationErrors && isDependencyValidationRefusal(err) {
recordSkippedDependency(opts, dep, err.Error())
continue
}
return result, fmt.Errorf("invalid dependency %s -> %s: %w", dep.IssueID, dep.DependsOnID, err)
}
}
if err := lookups.checkCycle(ctx, tx, dep); err != nil {
if opts.SkipDependencyValidationErrors && isDependencyValidationRefusal(err) {
recordSkippedDependency(opts, dep, err.Error())
continue
}
return result, fmt.Errorf("invalid dependency %s -> %s: %w", dep.IssueID, dep.DependsOnID, err)
}
createdAt := dep.CreatedAt
if createdAt.IsZero() {
createdAt = time.Now().UTC()
}
// Deterministic id from (issue_id, target) keeps bulk-imported edges
// merge-safe across clones — two clones importing the same JSONL get the
// same primary key, not two random UUIDs that collide on uk_dep_* (#4259).
createdBy := dependencyCreatedBy(dep, actor)
metadata := dep.Metadata
if metadata == "" {
metadata = "{}"
}
//nolint:gosec // G201: item.depTable is one of two hardcoded constants; target column from DepTargetKind.Column()
sqlResult, err := tx.ExecContext(ctx, fmt.Sprintf(`
INSERT INTO %s (id, issue_id, %s, type, created_by, created_at, metadata, thread_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE type = type
`, item.depTable, kind.Column()), depid.New(dep.IssueID, dep.DependsOnID), dep.IssueID, dep.DependsOnID, dep.Type, createdBy, createdAt, metadata, dep.ThreadID)
if err != nil {
return result, fmt.Errorf("failed to insert dependency %s -> %s: %w", dep.IssueID, dep.DependsOnID, err)
}
rowsAffected, err := sqlResult.RowsAffected()
if err != nil {
return result, fmt.Errorf("failed to check dependency insert result for %s -> %s: %w", dep.IssueID, dep.DependsOnID, err)
}
if err := lookups.recordInsert(ctx, tx, item.depTable, dep, rowsAffected); err != nil {
return result, err
}
if rowsAffected > 0 {
result.markChanged(item.depTable)
result.persistedDependencies = append(result.persistedDependencies, persistedDependency{
source: dep.IssueID,
target: dep.DependsOnID,
depType: dep.Type,
sourceWisp: item.depTable == "wisp_dependencies",
})
if dep.Type == types.DepParentChild {
if err := TouchDependencyCoordinationTableInTx(ctx, tx, dep.DependsOnID, item.depTable); err != nil {
return result, err
}
}
// Creation-time edges are independently replayable operations; do
// not rely on the issue create payload's inline dependencies.
if err := RecordDepEventInTx(ctx, tx, EventDepAdd, dep.IssueID, string(dep.Type), dep.DependsOnID, metadata, actor); err != nil {
return result, err
}
}
}
}
return result, nil
}
// dependencyCreatedBy returns the author stamped on a dependency edge.
// Import/restore paths populate dep.CreatedBy from JSONL; interactive
// creation leaves it empty and falls back to the current actor.
func dependencyCreatedBy(dep *types.Dependency, actor string) string {
if dep != nil && dep.CreatedBy != "" {
return dep.CreatedBy
}
return actor
}
// isDependencyValidationRefusal reports whether err, from checkHierarchy or
// checkCycle, is a typed refusal of the edge itself (a self-dependency, a
// cycle, or a blocker that is the issue's own ancestor or descendant) — the
// only failures SkipDependencyValidationErrors tolerates. Every other error is
// a failed walk query (an untranslated statement, a timeout, a dropped
// connection): recording it as a skipped edge would commit the import without
// the edge and report the SQL error only as a skip reason. On a backend whose
// transaction aborts on a failed statement, the import would instead fail
// later at an unrelated statement, with the real cause hidden in the skip
// reason.
func isDependencyValidationRefusal(err error) bool {
if errors.Is(err, domain.ErrSelfDependency) || errors.Is(err, domain.ErrDependencyCycle) {
return true
}
var hierarchy *domain.DependencyHierarchyConflictError
return errors.As(err, &hierarchy)
}
func recordSkippedDependency(opts storage.BatchCreateOptions, dep *types.Dependency, reason string) {
if dep == nil {
return
}
recordSkippedDependencyEdge(opts, dep.IssueID, dep.DependsOnID, reason)
}
func recordSkippedDependencyEdge(opts storage.BatchCreateOptions, issueID, dependsOnID, reason string) {
if opts.OnSkippedDependency == nil {
return
}
opts.OnSkippedDependency(issueID, dependsOnID, reason)
}
func ReconcileChildCounters(ctx context.Context, tx DBTX, issues []*types.Issue) (map[string]bool, error) {
type bucket struct {
maxChild int
isWisp bool
known bool
}
parents := make(map[string]*bucket)
var changed map[string]bool
for _, issue := range issues {
if issue == nil {
continue
}
if IsWisp(issue) {
if b, ok := parents[issue.ID]; ok {
b.isWisp, b.known = true, true
} else {
parents[issue.ID] = &bucket{isWisp: true, known: true}
}
}
}
for _, issue := range issues {
if issue == nil {
continue
}
parentID, childNum, ok := ParseHierarchicalID(issue.ID)
if !ok {
continue
}
b, exists := parents[parentID]
if !exists {
b = &bucket{}
parents[parentID] = b
}
if childNum > b.maxChild {
b.maxChild = childNum
}
}
unknownParentIDs := make([]string, 0, len(parents))
for parentID, b := range parents {
if b.maxChild > 0 && !b.known {
unknownParentIDs = append(unknownParentIDs, parentID)
}
}
wispParents, err := WispIDSetInTx(ctx, tx, unknownParentIDs)
if err != nil {
return nil, fmt.Errorf("failed to route child counter parents: %w", err)
}
for _, parentID := range unknownParentIDs {
_, parents[parentID].isWisp = wispParents[parentID]
}
for parentID, b := range parents {
if b.maxChild == 0 {
continue
}
table := "child_counters"
parentTable := "issues"
if b.isWisp {
table = "wisp_child_counters"
parentTable = "wisps"
}
var parentExists int
// Orphaned hierarchical IDs are valid import input when the parent was
// deleted before export. Their auxiliary counter has no owner and must
// not be inserted: both counter tables enforce a parent foreign key.
//nolint:gosec // G201: parentTable is one of two hardcoded constants.
err := tx.QueryRowContext(ctx, fmt.Sprintf(`
SELECT 1 FROM %s WHERE id = ?
`, parentTable), parentID).Scan(&parentExists)
if err == sql.ErrNoRows {
continue
}
if err != nil {
return nil, fmt.Errorf("failed to check child counter parent %s: %w", parentID, err)
}
var current int
//nolint:gosec // G201: table is one of two hardcoded constants.
err = tx.QueryRowContext(ctx, fmt.Sprintf(`
SELECT last_child FROM %s WHERE parent_id = ?
`, table), parentID).Scan(&current)
if err != nil && err != sql.ErrNoRows {
return nil, fmt.Errorf("failed to read child counter for %s: %w", parentID, err)
}
if err == nil && current >= b.maxChild {
continue
}
// Qualify the existing-row column with the table name so the canonical
// MySQL form and SQLite's translated ON CONFLICT form both unambiguously
// refer to the target row rather than the incoming value.
//nolint:gosec // G201: table is one of two hardcoded constants.
if _, err := tx.ExecContext(ctx, fmt.Sprintf(`
INSERT INTO %[1]s (parent_id, last_child) VALUES (?, ?)
ON DUPLICATE KEY UPDATE last_child = GREATEST(%[1]s.last_child, ?)
`, table), parentID, b.maxChild, b.maxChild); err != nil {
return nil, fmt.Errorf("failed to reconcile child counter for %s: %w", parentID, err)
}
if changed == nil {
changed = map[string]bool{}
}
changed[table] = true
}
return changed, nil
}