feat(http): bd heartbeat over bd serve (issues.heartbeat), and close the re-arm orphan-lease race (#7600)
* feat(roles): issueops.Heartbeater role on the dolt, embedded and uow legs
Adds issueops.Heartbeater.Heartbeat(HeartbeatRequest{Actor, IssueID}) ->
HeartbeatResult{IssueID, Holder, HeartbeatAt, LeaseExpiresAt, Rearmed},
the lease refresh behind `bd heartbeat`, as its own role with its own
accessor.
- One body: internal/storage/issueops.ExecuteHeartbeatInTx validates
(*HeartbeatFieldError for a blank/over-long actor or empty id), refuses
a wisp, then runs RenewLeaseInTx, which is HeartbeatIssueInTx's body
reporting the renewal. HeartbeatIssueInTx is now a wrapper, so the legacy
HeartbeatIssue legs answer the same text and sentinels.
- The refresh is fenced on the holder AND on the live claim: after the
holder-fenced UPDATE, a read in the same transaction requires the issue
to be in_progress under the holder (the `update leases set` prefix and
the one IF() the Postgres dialect rewrites are unchanged; a Postgres port
should expect more serialization retries from that read). A refusal deletes the caller's orphaned lease
row, and the legs commit that refusal.
- A refresh writes only the lease row: no row_lock, version, updated_at,
event, journal entry, Dolt commit or hook. A re-arm (a live hand-doled
claim with no lease row the caller holds) is recorded as a mutation:
row_lock reminted, updated_at stamped, a lease_rearmed event, a journal
entry, a version, a Dolt commit (LeaseRearmVersionCommit) and on_update.
The remint makes a re-arm collide with a racing release or close instead
of leaving an orphaned lease on an open or closed issue (a pre-existing
race on every Dolt route). Rearmed is true only when a row was created:
a same-second refresh (0 changed rows on embedded Dolt) or a holder
spelled differently refreshes the existing row in place and keeps its
granted_at/granted_node. Reported times are rounded to whole seconds,
the stored values; the binds stay unrounded so a same-second refresh is
never a 0-row UPDATE on Dolt.
- Typed refusals: an absent id is ErrNotFound; a foreign holder is
*LeaseConflictError{Holder} wrapping ErrAlreadyClaimed; a wisp
(Ephemeral) or no live claim is *LeaseConflictError wrapping the new
ErrLeaseNotHeld, which also matches ErrNotClaimable.
- Legs: DoltStore and EmbeddedDoltStore through the issue-operation
funnel (DOLT_ADD of the named tables, then DOLT_COMMIT -m, after the SQL
commit); uow.NewHeartbeater through the new uow.RunTxScoped: a refresh or
a refusal commits with the plain SQL COMMIT (no working-set scan, no Dolt
commit, nothing another writer left uncommitted is swept), and a re-arm
commits through the new ScopedCommitter (DOLT_ADD of exactly its tables,
DOLT_COMMIT -m, never -Am). The legacy HeartbeatIssue doors commit a
re-arm the same way and keep a refusal's orphan delete, and the
proxied-server `bd heartbeat` route runs the role's body with the same
scoped commit.
- Census: storage.Storage gains Heartbeater(); HookFiringStore wraps it
(fires on_update for a re-arm only, RoleFiresHooks knows it), telemetry
wraps it, httpapi.timedProvider, uow.notifyingProvider (records a re-arm)
and externaldeps.uowProvider build it; the journal and version guards
list the heartbeat family as entry points; the httpclient unsupported
shell is regenerated and Heartbeater sits on its unsupported allowlist
until the wire lands.
Conformance: RunHeartbeater* (18 cases) on dolt, embedded dolt and uow:
refresh post-state with the stored times pinned to the result; no
row_lock/current_revision/updated_at/events/journal/issue_versions/
version-control/hook change; a guarded write with the pre-heartbeat
revision still lands; foreign holder refused against an aged lease;
refused after close/release/reclaim/status move; wisp refused; hand-doled
claim re-armed as a mutation; same-second heartbeat is not a re-arm;
orphaned leases on released, closed and reassigned issues refused and
deleted; a refresh and a refusal against a dirty working set make no
commit and leave the other writer's change uncommitted, and a re-arm
commits only its own tables; absent id; request rules; no request mutation. The http leg is
waived until the wire lands.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(httpapi): POST /v0/beads/issues/{id}:heartbeat behind issues.heartbeat
Publishes issueops.Heartbeater on bd serve as heartbeatIssue, the served
lease refresh behind `bd heartbeat`.
- Spec: the operation, HeartbeatIssueRequest {actor} and
HeartbeatIssueResponse {id, holder, heartbeat_at, lease_expires_at,
rearmed}; no `revision` (a refresh writes nothing to the row) and no
expected_version (the fence is the holder and the live claim).
issues.heartbeat joins the capability vocabulary and lease_not_held
(409) the Problem.code list; `reason: ephemeral` qualifies it for a
wisp. apigen and the wire-shape golden are regenerated; the golden diff
is additive only, so wire_revision stays 2.
- Server: route row on the custom-method dispatcher, handleHeartbeat in
heartbeat.go (claim's actor rules via bodyActor, unknown member and
query refusals), Config.Heartbeater as a required role (sourceRoles,
roleSourceNames, the per-request uow.HeartbeaterSource accessor).
failHeartbeat maps *LeaseConflictError typed fields: a foreign holder
is the claim's conflict shape (already_claimed + assignee +
issue_status); no live claim, a wisp included, is the new
lease_not_held (+ issue_status, + reason: ephemeral for a wisp), matched
before the generic arms so ErrLeaseNotHeld's ErrNotClaimable never
surfaces as not_claimable. Its detail names the one state the refusing
transaction found (closed, open and not claimed, an ephemeral wisp, ...).
- bd serve wires the role from its peeled store (serveRoleSource gains
Heartbeater); the serve stub stores, the corpus and e2e harnesses, the
served harness and the encode oracle bind it.
- Wire client vocabulary: OpHeartbeatIssue/issues.heartbeat and
MethodHeartbeat are CLASSIFIED (the capability set-equality gate),
ReasonEphemeral is pinned to the server's, and lease_not_held rebuilds
*LeaseConflictError{Status, Ephemeral} wrapping ErrLeaseNotHeld in the
native words (the code-table set-equality gate). No accessor yet.
Tests: heartbeat_test.go (request mapping and projection, edge refusals,
every role refusal with its code, members and state-specific detail, the
code row, the capability). The served-http Heartbeater leg
(served_heartbeater_test.go, 18 cases) runs the contract against a real
bd serve through a test-local dial over the store's own dispatch, and the
http-leg waivers are deleted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(http): bd heartbeat over http through httpclient.Heartbeater
The client leg of the served lease refresh, and `bd heartbeat` on the
role for every route.
- httpclient.Store.Heartbeater (heartbeater.go) dials POST
/v0/beads/issues/{id}:heartbeat through Store.dispatch, so
issues.heartbeat is preflighted: an older server is refused BEFORE the
dial with *wire.CapabilityError naming the token (unwraps to
*storage.ErrUnsupported{Capability: "issues.heartbeat"}); it never falls
back to a re-claim or a metadata stamp.
- The role's request rules are restated before the dial
(validateHeartbeatRequest, pinned to
storeops.ValidateHeartbeatRequest): blank/over-long actor, empty id, each
a *HeartbeatFieldError.
- Problem mapping: already_claimed (which the shared table rebuilds as the
claim's ClaimConflictError) becomes *issueops.LeaseConflictError naming
the holder and wrapping ErrAlreadyClaimed, in the native words ("issue
already claimed by <holder>"); lease_not_held already arrives as
LeaseConflictError (Ephemeral from `reason`); not_found is ErrNotFound;
invalid_argument on `actor` is the role's field error.
- Idempotent: true, the one write this client marks replayable (a repeat
only moves the expiry later; a replay after a reclaim fails the fence).
- Legacy Store.HeartbeatIssue rides the role. Both leave the generated
shell and completeness_test's unsupported allowlist.
- `bd heartbeat` takes Heartbeater() from the resolved store on the direct
routes (dolt, embedded, http) and from the provider's HeartbeaterSource
on the proxied route (exact id resolution in a read unit of work, then
the role); runHeartbeatProxiedServer and heartbeat_proxied_server.go are
deleted. Success output and exit codes are unchanged; refusal text
changes per route (see CHANGELOG).
Tests: heartbeater_test.go (stub server: preflight refusal naming the
token with no POST, request/answer projection, each refusal's typed
mapping including the wisp's reason, the 503 replay, pre-dial validation,
validator parity, the legacy door). served_heartbeater_test.go runs the
18-case contract through env.subject.Heartbeater(), with
HookedHeartbeater wired off a HookFiringStore. TestE2E_HeartbeatOverHTTP:
real bd against an in-process serve: holder ok with revision and
updated_at unchanged and a guarded write with the pre-heartbeat revision
landing; a non-holder refused against an aged lease that stays untouched;
closed, wisp and unclaimed refused, each naming its own state; an older
server's handshake refused naming issues.heartbeat with no write. Corpus
shapes gc-heartbeat, -non-holder, -closed, -wisp pending until
reference_server.json advertises the token. CHANGELOG entry for the whole
track, including the orphaned-lease race fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* build(bazel): auto-sync BUILD files
Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/38875641041
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.
* fix(heartbeat): roll back a no-issue refusal; restate refresh vs re-arm
Review fixes for #7600 (adopt-pr apply-fixes, iterations 1-2).
Legacy HeartbeatIssue doors commit a refusal only when it read the issue.
The dolt and embedded HeartbeatIssue callbacks committed every
LeaseRenewalRefusal, including Found=false (the issue read failed). That
refusal deletes nothing, but it can follow a fast-path UPDATE that
already extended an orphan lease row's expiry, so committing it kept the
extension. issueops.LeaseRenewalRefusalCommits now gates the commit on
Found, the rule the Heartbeater role's HeartbeatRefusalCommits applies.
New tests on both legs pin both arms: a no-issue orphan's expiry is
unchanged, and a released issue's orphan is deleted.
Refresh vs re-arm contract docs. Comments, Long help, the OpenAPI spec
and the generated apigen still said every heartbeat writes only the
lease row, makes no Dolt commit and fires no hook. A re-arm is recorded
as a mutation: it remints the revision, records an event and a version
entry, commits once (LeaseRearmVersionCommit) and fires on_update
through the wrapping hook decorator. Restated in domain, uow, issueops,
storage.go (why RoleFiresHooks answers true, so bd serve takes the
heartbeater off a peeled store), cmd/bd heartbeat and serve, httpapi,
httpclient, the spec (apigen regenerated), and the conformance and
contract-test docs.
Minor review findings:
- The 409 already_claimed and lease_not_held details now end "nothing
on the issue was written": the refusal may have deleted and committed
the caller's orphan lease row.
- docs/multi-agent/federation.md and the CHANGELOG entry describe the
spelling-mismatch heartbeat: the row is refreshed in place and keeps
granted_at and granted_node; only a re-arm stamps this node. The
embedded spelling test pins both columns, and the lease.go comments
say "refreshed in place" instead of "re-arms".
- The refusal's orphan delete judges the holder under actorMatches, as
the in-place refresh does (iteration 2). It matched holder = ? on the
actor as sent, so a dead claim's orphan held under another spelling
of the caller (gastown.mayor, heartbeat as gastown__mayor) survived
the refusal until TTL. The refused closure now reads the row's holder
and deletes that stored holder only when actorMatches(holder, actor).
Both TestHeartbeatIssueCommitsARefusalOnlyWhenItReadTheIssue twins
gain a cross-spelling arm (row deleted) and a control: another
identity's row (gastown.dog-3) survives.
Not changed: `bd heartbeat --json` still omits `rearmed`, since adding
it changes a CLI JSON contract (follow-up). The federation.md paragraph
post-dates the v1.2.2 docs pin, so documenting this main-only behaviour
there is a maintainer call under
engdocs/decisions/2026-07-17-docs-release-pin.md section 1.
Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(uow): run the scoped re-arm's Dolt commit after the SQL commit
The proxied-server heartbeat commits a re-arm with doltServerTx.CommitScoped,
which ran DOLT_ADD per table and DOLT_COMMIT('-m') INSIDE the open
transaction, before the SQL COMMIT. DOLT_ADD stages whole tables from the
session's transaction root, so that Dolt commit was built from the re-arm's
start snapshot: an issues row another session changed while the re-arm was
open (a write still only in the working set, as in batch mode or before that
writer's own Dolt commit) was staged at its old value, and HEAD disagreed
with the rows every reader sees. This is the LatentLabsSpace/NEXUS#92
ordering the DoltStore funnel (runIssueOperationTxWithMessage) already
avoids, and the review's major finding on this PR.
CommitScoped now commits the SQL transaction first, then stages exactly the
named tables (plus the versioned-history tables) and makes the Dolt commit on
the same pinned connection, the uow twin of doltAddAndCommitPostTx:
- a serialization failure of the SQL COMMIT is still returned, so
RunTxScoped replays the unit of work;
- the Dolt commit runs detached on its own budget and replays only conflicts
that leave nothing behind: serialization failures and Dolt's 1105
"@autocommit transaction rolled back" (new isDoltAutocommitRollbackError,
mirroring the DoltStore classifier of the same name);
- nothing staged, or "nothing to commit", is not an error: a concurrent
writer's Dolt commit already took the row. A "nothing to commit" past the
staged check is logged, as the funnel logs it, so the missing audit line
is explicable;
- any other failure is logged, never returned. The data is committed, and
returning it would make RunTxScoped replay a committed re-arm, which would
then report a refresh. The session is poisoned rather than pooled, since
its state is unknown. uow records no metrics, so that log line is the
drop's only signal; bd.db.post_tx_commit_dropped counts only the funnel's.
Statements go through schema.DrainCall, retiring the local drainCall fork
(review nit).
Tests: five sqlmock tests pin the order (SQL COMMIT before any DOLT_ADD),
the swallow-and-poison path, the 1105 retry, the returned serialization
failure and the empty-commit tolerance; each fails against the previous
CommitScoped. TestIsDoltAutocommitRollbackError carries the DoltStore
classifier's table over to the uow copy, plus the prefix boundary; dropping
the prefix branch, the prefix's period, the 1105 check or the type check
each fails a case. TestHeartbeaterRearmKeepsAConcurrentWrite is the
real-Dolt regression: a change to another issues row made while the
re-arm's transaction is open must be at HEAD after the re-arm's Dolt
commit. Its working-set-only writer fails on the old ordering (dolt
2.1.8); the writer that Dolt-commits its own change passes on both and is
kept as coverage.
Review follow-ups folded in (comments, a file move and a test; the
absorption log line above is the only behaviour change among them):
- the freshRowLock invariant and its row_lock_guard_test.go mirror name the
re-arm as the one deliberate remint without a lifecycle-field change,
load-bearing for the release/close collision;
- httpclient: "a write is never retried" (wire/client.go, stall_retry.go)
and "only a read's answer is yes" (dispatch.go) now allow for the
Idempotent heartbeat, and httpHeartbeater and issueops.HeartbeatResult
document that a replayed heartbeat can report Rearmed false after a
committed re-arm, with the guarded write's revision check as the backstop;
- the proxied route moves back to cmd/bd/heartbeat_proxied_server.go, the
file family cmd/bd/AGENTS.md prescribes;
- the ScopedCommitter and RunTxScoped docs no longer say every other
writer's uncommitted change stays out of the commit: a staged table is
committed whole, so such a change to a named table rides in the re-arm's
commit, as it does in the funnel, and only other tables keep theirs;
- commitStagedTables says its drop is log-only, the absorption is logged,
and the uow classifier gets its table test (see Tests).
Not folded in: the base-owned IssueUseCase.Heartbeat /
IssueSQLRepository.HeartbeatIssue door, dead in production since the
proxied route moved to the Heartbeater role. Its one caller is the
notifying hook-parity test (uow/notifying_test.go), and moving that case
onto the Heartbeater means re-deriving its RenewLease hook semantics, so it
is left for a follow-up.
Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
This commit is contained in:
parent
91a72ef7b5
commit
47d5db39ff
96 changed files with 5397 additions and 190 deletions
|
|
@ -271,6 +271,7 @@
|
|||
40 14 TestEmbeddedReaderReadyDefaultTypeExclusionsYieldToAnExplicitType
|
||||
40 14 TestEmbeddedReadyClaimerClaimsTheFrontRowAndReturnsThePostClaimState
|
||||
40 14 TestEmbeddedReadyClaimerDoesNotMutateTheCallerRequest
|
||||
40 14 TestHeartbeatIssueCommitsARefusalOnlyWhenItReadTheIssueEmbedded
|
||||
40 14 TestHistory_NullTextColumns
|
||||
|
||||
40 15 TestCreateIssuesRejectStaleUpserts
|
||||
|
|
@ -395,6 +396,7 @@
|
|||
40 25 TestEmbeddedReadyClaimerLeavesEphemeralRowsOutOfTheDefaultReadySet
|
||||
40 25 TestEmbeddedReadyClaimerRejectsLimitOffsetBriefAndEmptyActor
|
||||
40 25 TestGetDependentRecordsForIssuesEmbedded
|
||||
40 25 TestHeartbeaterContract
|
||||
40 25 TestIgnoredCursorHealIsIdempotent
|
||||
40 25 TestIsBlockedBatchEmbedded
|
||||
40 25 TestReadyWorkPageTotal
|
||||
|
|
|
|||
63
CHANGELOG.md
63
CHANGELOG.md
|
|
@ -32,6 +32,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
capability error naming the missing token (never sent without the
|
||||
member); a batchApply plan is checked whole, so no partial plan is sent.
|
||||
A request naming no gate member is unaffected by either token.
|
||||
- `bd heartbeat` works over a connected HTTP workspace.
|
||||
The lease refresh is a role, `issueops.Heartbeater`
|
||||
(`Heartbeat(HeartbeatRequest{Actor, IssueID}) -> HeartbeatResult{IssueID,
|
||||
Holder, HeartbeatAt, LeaseExpiresAt, Rearmed}`), on every backend leg, and
|
||||
`bd serve` publishes it as `POST /v0/beads/issues/{id}:heartbeat` behind the
|
||||
new `issues.heartbeat` capability token (additive; no wire_revision bump),
|
||||
with the new `lease_not_held` (409) problem code (`reason: ephemeral` for a
|
||||
wisp). A refresh writes ONLY the lease row: the issue's `revision` is not
|
||||
reminted (a guarded write against the pre-heartbeat token still lands), no
|
||||
version, event, journal entry or Dolt commit is recorded, and no hook
|
||||
fires. `Rearmed` is true only when the heartbeat created the caller's lease
|
||||
row (a hand-doled claim's first heartbeat). A re-arm is recorded as a
|
||||
mutation; a refresh is not: the re-arm remints `revision`, stamps
|
||||
`updated_at`, records a `lease_rearmed` event, a journal entry and a
|
||||
version, makes a Dolt commit and fires `on_update` (see Fixed); the
|
||||
reported times are whole seconds, exactly what the lease row stores. The
|
||||
HTTP backend's `Heartbeater()` and legacy `HeartbeatIssue` dial it;
|
||||
against a server that does not advertise the token they refuse before
|
||||
dialing with a typed capability error naming `issues.heartbeat`, and never
|
||||
fall back to a re-claim or a metadata stamp. A
|
||||
busy (503) answer is replayed, because a repeat only moves the expiry later.
|
||||
`bd heartbeat` now runs the role on every route (the proxied-server route
|
||||
through the provider's accessor), with the same
|
||||
`{"id","status":"heartbeat","owner"}` success output and the same exit
|
||||
codes. Refusal TEXT on stderr changes per route: the direct routes (Dolt
|
||||
server, embedded) gain a `lease not held: ` prefix
|
||||
(`heartbeat X: lease not held: issue not claimable: X status closed`); the
|
||||
proxied-server route loses its
|
||||
`Heartbeat: db: IssueSQLRepository.HeartbeatIssue: ` prefix and reads as the
|
||||
direct routes do; over HTTP the refusal is the problem's detail, which names
|
||||
the state found (`it is closed`, `an ephemeral wisp`, `open and not
|
||||
claimed`) or, for a foreign holder, the native `issue already claimed by
|
||||
<holder>`. An id that names no row answers not-found from the role rather
|
||||
than "not claimable" (the CLI resolves the id first, so its output is
|
||||
unchanged), and every refusal for a caller that holds no live claim
|
||||
(closed, released, reclaimed, moved off in_progress, or a wisp) is
|
||||
`issueops.ErrLeaseNotHeld`, which still matches `ErrNotClaimable`.
|
||||
|
||||
- `GET /v0/beads/dependencies/blocking` accepts `is_blocked`, behind the new
|
||||
`dependencies.blocking.isBlocked` capability token (additive; no
|
||||
wire_revision bump), and answers each item's stored, transitive
|
||||
|
|
@ -642,6 +680,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
1,000 listed rows: +0.07 s at 1,000 rows and +0.46 s at 4,000 on embedded
|
||||
Dolt. A store with many wisps, whose edges the old read also scanned, is
|
||||
faster at every page size.
|
||||
- A heartbeat can no longer resurrect a lease after a release or close. A
|
||||
heartbeat re-arming a hand-doled claim (in_progress with no lease row) could
|
||||
race a release or close on Dolt and leave an orphaned lease row on an open
|
||||
or closed issue; every later heartbeat by the former holder then refreshed
|
||||
it and answered success for a claim it no longer held. This affected every
|
||||
Dolt route, including releases before this one. The re-arm is now recorded
|
||||
as a mutation and remints the issue's row_lock, which every claim-ending
|
||||
path rewrites, so it collides with a racing release or close and one side
|
||||
is replayed against the other's result (the cost: a hand-doled claim's
|
||||
first heartbeat changes its `revision` once). The refresh also requires, in
|
||||
the same transaction, that the issue is still in_progress under the holder,
|
||||
and a refusal deletes the caller's orphaned lease row, so a lease row
|
||||
orphaned any other way never answers a heartbeat. A refresh and a refusal
|
||||
still make no Dolt commit on any route; a re-arm commits only its own
|
||||
tables (never `DOLT_COMMIT('-Am')`), so another writer's uncommitted
|
||||
batch-mode edits to other tables stay uncommitted (a re-arm stages its
|
||||
tables whole, as every other direct-route write does). On embedded Dolt, a heartbeat landing in
|
||||
the same second as the previous one no longer re-grants the lease
|
||||
(rewriting its `granted_at` / `granted_node`), and on every route a
|
||||
heartbeat whose actor is a different spelling of the lease's holder
|
||||
(`gastown.mayor` for `gastown__mayor`) refreshes the lease in place,
|
||||
keeping its `granted_at` and granting replica, instead of re-granting it
|
||||
under this node. For a Postgres port: the
|
||||
refresh now reads the issue row, so under SERIALIZABLE expect more retried
|
||||
serialization failures between a holder's own edits and its heartbeat.
|
||||
- **`bd gate list` works over a connected HTTP workspace** (S20). The
|
||||
workspace-wide listing built a raw `SearchIssues` filter, which the HTTP
|
||||
backend refuses, so gc's `renudge-stale-human-gates` order failed on every
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ go_library(
|
|||
"edge_reader_contract.go",
|
||||
"expected_revision_contract.go",
|
||||
"graph_counter_contract.go",
|
||||
"heartbeater_contract.go",
|
||||
"history_matching.go",
|
||||
"importer_contract.go",
|
||||
"issue_operations_contract.go",
|
||||
|
|
|
|||
837
backend/conformance/heartbeater_contract.go
Normal file
837
backend/conformance/heartbeater_contract.go
Normal file
|
|
@ -0,0 +1,837 @@
|
|||
package conformance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
storeops "github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
publicops "github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// This file holds the contract every implementation of publicops.Heartbeater
|
||||
// must satisfy. Each case asserts what issueops/heartbeater.go PROMISES, cited
|
||||
// by symbol.
|
||||
//
|
||||
// THREE LEGS, ONE BODY. dolt, embedded dolt and the unit-of-work provider all
|
||||
// reach internal/storage/issueops.ExecuteHeartbeatInTx, so a per-leg failure
|
||||
// here is a wrapper's failure: a commit the leg added, a hook it fired, a
|
||||
// refusal that stopped matching errors.Is.
|
||||
//
|
||||
// THE QUIET CASE IS THE POINT. A heartbeat is the hottest write in a fleet and
|
||||
// it is cheap because it writes the lease row and nothing else. Most of what
|
||||
// RunHeartbeaterChangesNothingOnTheRow asserts is an ABSENCE — no revision, no
|
||||
// version, no updated_at, no event, no journal entry, no history entry, no
|
||||
// hook — and each absence is something a plausible wrapper would add.
|
||||
//
|
||||
// THE LIVE-CLAIM FENCE IS AN INVARIANT AND A PREDICATE. The RefusesAfter*
|
||||
// cases hold the invariant: each ends a claim one way and asserts both that
|
||||
// the lease row is gone and that the holder's next heartbeat is refused. The
|
||||
// RefusesAnOrphanedLease* cases hold the predicate: a lease row that survives
|
||||
// on an issue that is no longer claimed by its holder (what a re-arm racing a
|
||||
// release or close can leave on Dolt) is refused AND deleted, never refreshed.
|
||||
type HeartbeaterFixture struct {
|
||||
// IssuePrefix namespaces the ids each assertion seeds, so several of them
|
||||
// can share one database.
|
||||
IssuePrefix string
|
||||
// Heartbeater is the surface under test.
|
||||
Heartbeater publicops.Heartbeater
|
||||
// CreateIssue seeds a durable issue in the issues plane.
|
||||
CreateIssue func(context.Context, *types.Issue, string) error
|
||||
// CreateWisp seeds an ephemeral issue in the wisps plane. A nil CreateWisp
|
||||
// SKIPS the wisp case loudly.
|
||||
CreateWisp func(context.Context, *types.Issue, string) error
|
||||
// Claimer takes the claim, and with it the lease row, a heartbeat
|
||||
// refreshes.
|
||||
Claimer publicops.Claimer
|
||||
// Lifecycle ends or moves a claim (close, a status move, a hand-doled
|
||||
// claim) and performs the guarded write the revision case lands.
|
||||
Lifecycle publicops.Lifecycle
|
||||
// Releaser gives a claim up for the release case. Nil SKIPS that case.
|
||||
Releaser publicops.Releaser
|
||||
// LeaseReclaimer reverts an aged claim for the reclaim case. Nil, or a nil
|
||||
// Exec, SKIPS that case.
|
||||
LeaseReclaimer publicops.LeaseReclaimer
|
||||
// Exec runs raw statements against the backing database: the cases use it
|
||||
// to age a lease.
|
||||
Exec func(ctx context.Context, statements []SQLStatement) error
|
||||
// ExecUncommitted runs raw statements as ANOTHER writer that leaves its
|
||||
// change uncommitted in the version-controlled working set (a batch-mode
|
||||
// edit): SQL-committed, no version-control commit. Nil means Exec already
|
||||
// behaves so.
|
||||
ExecUncommitted func(ctx context.Context, statements []SQLStatement) error
|
||||
// QueryScalar runs a single-row query and scans it, so the post-state is
|
||||
// read RAW rather than through the role being tested.
|
||||
QueryScalar func(context.Context, string, []any, ...any) error
|
||||
// CountHistory reports how many version-control entries the fixture's
|
||||
// branch has. Nil SKIPS that one assertion, loudly, in the quiet case.
|
||||
CountHistory func(context.Context) (int, error)
|
||||
// SetJournalEnabled turns on durable journaling, so the quiet case can
|
||||
// observe that a heartbeat records nothing in bd_events_journal. Nil SKIPS
|
||||
// that one assertion, loudly.
|
||||
SetJournalEnabled func(enabled bool)
|
||||
// SetVersionedHistoryEnabled turns on dual-write issue-version history, so
|
||||
// the quiet case can observe that a heartbeat records no issue_versions
|
||||
// row. Nil SKIPS that one assertion, loudly.
|
||||
SetVersionedHistoryEnabled func(enabled bool)
|
||||
// HookedHeartbeater is the same role taken off the leg's HOOK-FIRING chain
|
||||
// (storage.HookFiringStore on the store legs, uow.NewNotifyingProvider on
|
||||
// the unit-of-work leg), built over the runner FiredUpdates reads. Nil,
|
||||
// with FiredUpdates, SKIPS the no-hook assertion loudly.
|
||||
HookedHeartbeater publicops.Heartbeater
|
||||
// FiredUpdates reports, in firing order, the issue id of every on_update
|
||||
// the hooked chain ran so far.
|
||||
FiredUpdates func(context.Context) ([]string, error)
|
||||
}
|
||||
|
||||
const (
|
||||
heartbeaterHolder = "hb-holder"
|
||||
heartbeaterIntruder = "hb-intruder"
|
||||
heartbeaterReaper = "hb-reaper"
|
||||
)
|
||||
|
||||
// RunHeartbeaterRefreshesTheHoldersLease pins the refresh itself
|
||||
// (Heartbeater: "pushes the lease the caller holds forward by the store's
|
||||
// TTL") and every member of the HeartbeatResult, against the row READ RAW.
|
||||
func RunHeartbeaterRefreshesTheHoldersLease(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
id := fixture.IssuePrefix + "-refresh"
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
aged := time.Now().UTC().Add(-time.Minute).Truncate(time.Second)
|
||||
if err := fixture.Exec(ctx, []SQLStatement{{
|
||||
Query: "UPDATE leases SET lease_expires_at = ?, heartbeat_at = ? WHERE issue_id = ?",
|
||||
Args: []any{aged, aged, id},
|
||||
}}); err != nil {
|
||||
t.Fatalf("age the lease on %s: %v", id, err)
|
||||
}
|
||||
|
||||
before := time.Now().UTC().Add(-time.Second)
|
||||
result := heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
after := time.Now().UTC().Add(time.Second)
|
||||
|
||||
if result.IssueID != id || result.Holder != heartbeaterHolder || result.Rearmed {
|
||||
t.Fatalf("result = %+v, want {IssueID:%s Holder:%s Rearmed:false}", result, id, heartbeaterHolder)
|
||||
}
|
||||
if result.HeartbeatAt.Before(before) || result.HeartbeatAt.After(after) {
|
||||
t.Fatalf("HeartbeatAt = %v, want the server clock at the call (between %v and %v)", result.HeartbeatAt, before, after)
|
||||
}
|
||||
if ttl := result.LeaseExpiresAt.Sub(result.HeartbeatAt); heartbeaterAbs(ttl-storeops.DefaultLeaseTTL) > time.Second {
|
||||
t.Fatalf("LeaseExpiresAt - HeartbeatAt = %v, want the store's lease TTL %v", ttl, storeops.DefaultLeaseTTL)
|
||||
}
|
||||
if moved := heartbeaterScalar[int](t, ctx, fixture,
|
||||
"SELECT COUNT(*) FROM leases WHERE issue_id = ? AND holder = ? AND lease_expires_at > ? AND heartbeat_at > ?",
|
||||
id, heartbeaterHolder, aged, aged); moved != 1 {
|
||||
t.Fatal("the lease row's lease_expires_at and heartbeat_at did not move forward")
|
||||
}
|
||||
heartbeaterRequireStoredMatches(t, ctx, fixture, id, result, before, after)
|
||||
}
|
||||
|
||||
// RunHeartbeaterASameSecondHeartbeatIsNotARearm pins HeartbeatResult.Rearmed
|
||||
// against the refresh that changes no value: a heartbeat landing in the same
|
||||
// second as the claim or the previous heartbeat writes values identical to the
|
||||
// row's (the lease columns hold whole seconds), which some engines report as
|
||||
// zero rows affected. It is still a refresh of the caller's own row: Rearmed
|
||||
// is false, and the lease's granted_at and granted_node do not move.
|
||||
func RunHeartbeaterASameSecondHeartbeatIsNotARearm(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
for i := 0; i < 3; i++ {
|
||||
id := fixture.IssuePrefix + "-samesec-" + string(rune('a'+i))
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
granted := heartbeaterScalar[string](t, ctx, fixture,
|
||||
"SELECT CONCAT(CAST(granted_at AS CHAR), '|', COALESCE(granted_node, '')) FROM leases WHERE issue_id = ?", id)
|
||||
for beat := 0; beat < 2; beat++ {
|
||||
result := heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
if result.Rearmed {
|
||||
t.Fatalf("heartbeat %d on %s right after the claim = %+v, want Rearmed:false: the holder's lease row existed", beat+1, id, result)
|
||||
}
|
||||
}
|
||||
if got := heartbeaterScalar[string](t, ctx, fixture,
|
||||
"SELECT CONCAT(CAST(granted_at AS CHAR), '|', COALESCE(granted_node, '')) FROM leases WHERE issue_id = ?", id); got != granted {
|
||||
t.Fatalf("granted_at|granted_node on %s moved from %s to %s across refreshes; a refresh never re-grants", id, granted, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterChangesNothingOnTheRow pins the role's quiet promise
|
||||
// (Heartbeater: "A REFRESH WRITES ONLY THE LEASE ROW"): the issue row's revision
|
||||
// (row_lock), its current_revision, its updated_at, its events, the events
|
||||
// journal, issue_versions, the version-control history and the hooks are all
|
||||
// exactly where the claim left them.
|
||||
func RunHeartbeaterChangesNothingOnTheRow(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
if fixture.SetJournalEnabled != nil {
|
||||
fixture.SetJournalEnabled(true)
|
||||
defer fixture.SetJournalEnabled(false)
|
||||
} else {
|
||||
t.Log("HeartbeaterFixture.SetJournalEnabled is nil: the journal assertion is skipped on this backend")
|
||||
}
|
||||
if fixture.SetVersionedHistoryEnabled != nil {
|
||||
fixture.SetVersionedHistoryEnabled(true)
|
||||
defer fixture.SetVersionedHistoryEnabled(false)
|
||||
} else {
|
||||
t.Log("HeartbeaterFixture.SetVersionedHistoryEnabled is nil: the issue_versions assertion is skipped on this backend")
|
||||
}
|
||||
id := fixture.IssuePrefix + "-quiet"
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
// updated_at is stored at second precision on some backends; step past
|
||||
// the claim's second so a stamp the heartbeat added cannot hide in it.
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
|
||||
snapshot := func() heartbeaterRowSnapshot {
|
||||
return heartbeaterSnapshot(t, ctx, fixture, id)
|
||||
}
|
||||
before := snapshot()
|
||||
fired := heartbeaterFired(t, ctx, fixture, id)
|
||||
|
||||
surface := fixture.Heartbeater
|
||||
if fixture.HookedHeartbeater != nil && fixture.FiredUpdates != nil {
|
||||
surface = fixture.HookedHeartbeater
|
||||
} else {
|
||||
t.Log("HeartbeaterFixture.HookedHeartbeater/FiredUpdates is nil: the no-hook assertion is skipped on this backend")
|
||||
}
|
||||
heartbeat(t, ctx, surface, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
|
||||
after := snapshot()
|
||||
if before != after {
|
||||
t.Fatalf("a heartbeat changed the issue's durable state:\nbefore %+v\nafter %+v", before, after)
|
||||
}
|
||||
if got := heartbeaterFired(t, ctx, fixture, id); got != fired {
|
||||
t.Fatalf("on_update fired %d time(s) for %s across a refresh, want 0: a refresh fires no hook", got-fired, id)
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterAGuardedWriteWithThePreHeartbeatRevisionStillWins is the
|
||||
// reason the quiet case matters to a caller: the holder's revision token
|
||||
// survives its own heartbeats, so a guarded write it composed from the claim
|
||||
// still lands.
|
||||
func RunHeartbeaterAGuardedWriteWithThePreHeartbeatRevisionStillWins(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.Lifecycle == nil {
|
||||
t.Skip("HeartbeaterFixture.Lifecycle is nil: this backend cannot land a guarded write")
|
||||
}
|
||||
id := fixture.IssuePrefix + "-guarded"
|
||||
revision := heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
|
||||
heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
|
||||
if _, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{
|
||||
Actor: heartbeaterHolder, IssueID: id, ExpectedVersion: &revision,
|
||||
Patch: publicops.IssuePatch{Notes: publicops.Field[string]{Set: true, Value: "after two heartbeats"}},
|
||||
}); err != nil {
|
||||
t.Fatalf("guarded write with the claim's revision after heartbeats = %v, want it to land", err)
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterRefusesAForeignHolder pins the fence (Heartbeater REFUSALS:
|
||||
// "an issue another actor holds: a *LeaseConflictError naming the Holder,
|
||||
// wrapping ErrAlreadyClaimed"), and that the refusal writes nothing.
|
||||
func RunHeartbeaterRefusesAForeignHolder(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
id := fixture.IssuePrefix + "-foreign"
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
// Age the holder's lease first, so a heartbeat that ignored the holder
|
||||
// fence would really CHANGE the row (some engines count only changed rows,
|
||||
// and the columns hold whole seconds: an unaged row refreshed in the
|
||||
// claim's second changes nothing and would hide the missing fence).
|
||||
aged := time.Now().UTC().Add(-time.Minute).Truncate(time.Second)
|
||||
if err := fixture.Exec(ctx, []SQLStatement{{
|
||||
Query: "UPDATE leases SET lease_expires_at = ?, heartbeat_at = ? WHERE issue_id = ?",
|
||||
Args: []any{aged, aged, id},
|
||||
}}); err != nil {
|
||||
t.Fatalf("age the lease on %s: %v", id, err)
|
||||
}
|
||||
lease := heartbeaterScalar[string](t, ctx, fixture, "SELECT CONCAT(CAST(lease_expires_at AS CHAR), '|', CAST(heartbeat_at AS CHAR), '|', holder) FROM leases WHERE issue_id = ?", id)
|
||||
|
||||
_, err := fixture.Heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: heartbeaterIntruder, IssueID: id})
|
||||
if !errors.Is(err, publicops.ErrAlreadyClaimed) {
|
||||
t.Fatalf("heartbeat by a non-holder = %v, want ErrAlreadyClaimed", err)
|
||||
}
|
||||
if errors.Is(err, publicops.ErrLeaseNotHeld) {
|
||||
t.Fatalf("heartbeat by a non-holder = %v also matches ErrLeaseNotHeld; the two refusals must stay distinct", err)
|
||||
}
|
||||
conflict := heartbeaterConflict(t, err)
|
||||
if conflict.IssueID != id || conflict.Holder != heartbeaterHolder || conflict.Status != types.StatusInProgress {
|
||||
t.Fatalf("LeaseConflictError = %+v, want {IssueID:%s Holder:%s Status:%s}", conflict, id, heartbeaterHolder, types.StatusInProgress)
|
||||
}
|
||||
if got := heartbeaterScalar[string](t, ctx, fixture, "SELECT CONCAT(CAST(lease_expires_at AS CHAR), '|', CAST(heartbeat_at AS CHAR), '|', holder) FROM leases WHERE issue_id = ?", id); got != lease {
|
||||
t.Fatalf("a refused heartbeat moved the holder's lease from %s to %s", lease, got)
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterCommitsOnlyItsOwnTables pins the commit discipline against a
|
||||
// DIRTY working set (Heartbeater: "A REFRESH WRITES ONLY THE LEASE ROW"; "A
|
||||
// RE-ARM IS RECORDED AS A MUTATION"): another writer has left an uncommitted
|
||||
// versioned change (a batch-mode edit, here a config row). A refresh and a
|
||||
// refusal make no version-control commit and leave that change uncommitted;
|
||||
// a re-arm makes exactly one commit, of its own tables, and still leaves the
|
||||
// other writer's change uncommitted rather than sweeping it in under the
|
||||
// heartbeat's message.
|
||||
func RunHeartbeaterCommitsOnlyItsOwnTables(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
if fixture.CountHistory == nil || fixture.Lifecycle == nil {
|
||||
t.Skip("HeartbeaterFixture.CountHistory/Lifecycle is nil: this backend has no version-control history to observe")
|
||||
}
|
||||
refreshed := fixture.IssuePrefix + "-dirty-refresh"
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, refreshed)
|
||||
refused := fixture.IssuePrefix + "-dirty-refused"
|
||||
heartbeaterSeed(t, ctx, fixture, refused)
|
||||
rearmed := fixture.IssuePrefix + "-dirty-rearm"
|
||||
heartbeaterSeed(t, ctx, fixture, rearmed)
|
||||
if _, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{
|
||||
Actor: heartbeaterHolder, IssueID: rearmed,
|
||||
Patch: publicops.IssuePatch{
|
||||
Status: publicops.Field[publicops.Status]{Set: true, Value: types.StatusInProgress},
|
||||
Assignee: publicops.Field[string]{Set: true, Value: heartbeaterHolder},
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("hand-dole %s: %v", rearmed, err)
|
||||
}
|
||||
|
||||
probe := "heartbeat.dirty-probe." + fixture.IssuePrefix
|
||||
dirty := fixture.Exec
|
||||
if fixture.ExecUncommitted != nil {
|
||||
dirty = fixture.ExecUncommitted
|
||||
}
|
||||
if err := dirty(ctx, []SQLStatement{{
|
||||
Query: "REPLACE INTO config (`key`, value) VALUES (?, ?)", Args: []any{probe, "uncommitted"},
|
||||
}}); err != nil {
|
||||
t.Fatalf("dirty the working set: %v", err)
|
||||
}
|
||||
defer func() {
|
||||
_ = dirty(ctx, []SQLStatement{{Query: "DELETE FROM config WHERE `key` = ?", Args: []any{probe}}})
|
||||
}()
|
||||
foreignDirty := func(when string) {
|
||||
t.Helper()
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture,
|
||||
"SELECT COUNT(*) FROM dolt_status WHERE table_name = 'config'"); n == 0 {
|
||||
t.Fatalf("%s: the other writer's uncommitted config change was committed; a heartbeat commits only its own tables", when)
|
||||
}
|
||||
}
|
||||
history := func() int {
|
||||
t.Helper()
|
||||
n, err := fixture.CountHistory(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("CountHistory: %v", err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
foreignDirty("before any heartbeat")
|
||||
before := history()
|
||||
|
||||
heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: refreshed})
|
||||
if got := history(); got != before {
|
||||
t.Fatalf("a refresh moved the version-control history %d -> %d; a refresh makes no commit", before, got)
|
||||
}
|
||||
foreignDirty("after a refresh")
|
||||
|
||||
if _, err := fixture.Heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: refused}); err == nil {
|
||||
t.Fatalf("heartbeat on unclaimed %s succeeded, want a refusal", refused)
|
||||
}
|
||||
if got := history(); got != before {
|
||||
t.Fatalf("a refusal moved the version-control history %d -> %d; a refusal makes no commit", before, got)
|
||||
}
|
||||
foreignDirty("after a refusal")
|
||||
|
||||
if result := heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: rearmed}); !result.Rearmed {
|
||||
t.Fatalf("heartbeat on hand-doled %s = %+v, want Rearmed", rearmed, result)
|
||||
}
|
||||
if got := history(); got != before+1 {
|
||||
t.Fatalf("a re-arm moved the version-control history %d -> %d, want exactly one commit", before, got)
|
||||
}
|
||||
foreignDirty("after a re-arm")
|
||||
}
|
||||
|
||||
// RunHeartbeaterRefusesAnOrphanedLeaseOnAReleasedIssue and
|
||||
// ...OnAClosedIssue and ...OnAnIssueSomeoneElseHolds pin the live-claim
|
||||
// predicate (Heartbeater: "THE LIVE-CLAIM FENCE IS A PREDICATE"): a lease row
|
||||
// that names the caller but survives on an issue that is open, closed, or
|
||||
// in_progress under another assignee is refused, never refreshed, and the
|
||||
// refusal deletes it (Heartbeater: the one write a refusal makes). The row
|
||||
// is planted raw: it is the state a re-arm racing a release or close leaves.
|
||||
func RunHeartbeaterRefusesAnOrphanedLeaseOnAReleasedIssue(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
id := fixture.IssuePrefix + "-orphan-open"
|
||||
heartbeaterSeed(t, ctx, fixture, id)
|
||||
heartbeaterRefusesAnOrphan(t, ctx, fixture, id, types.StatusOpen, "")
|
||||
}
|
||||
|
||||
func RunHeartbeaterRefusesAnOrphanedLeaseOnAClosedIssue(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
if fixture.Lifecycle == nil {
|
||||
t.Skip("HeartbeaterFixture.Lifecycle is nil: this backend cannot close the claim")
|
||||
}
|
||||
id := fixture.IssuePrefix + "-orphan-closed"
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
if _, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{Actor: heartbeaterHolder, IssueID: id, Reason: "done"}); err != nil {
|
||||
t.Fatalf("close %s: %v", id, err)
|
||||
}
|
||||
heartbeaterRefusesAnOrphan(t, ctx, fixture, id, types.StatusClosed, "")
|
||||
}
|
||||
|
||||
func RunHeartbeaterRefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
requireHeartbeaterExec(t, fixture)
|
||||
if fixture.Lifecycle == nil {
|
||||
t.Skip("HeartbeaterFixture.Lifecycle is nil: this backend cannot hand-dole a claim")
|
||||
}
|
||||
id := fixture.IssuePrefix + "-orphan-moved"
|
||||
heartbeaterSeed(t, ctx, fixture, id)
|
||||
if _, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{
|
||||
Actor: heartbeaterIntruder, IssueID: id,
|
||||
Patch: publicops.IssuePatch{
|
||||
Status: publicops.Field[publicops.Status]{Set: true, Value: types.StatusInProgress},
|
||||
Assignee: publicops.Field[string]{Set: true, Value: heartbeaterIntruder},
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("hand-dole %s to %s: %v", id, heartbeaterIntruder, err)
|
||||
}
|
||||
heartbeaterRefusesAnOrphan(t, ctx, fixture, id, types.StatusInProgress, heartbeaterIntruder)
|
||||
}
|
||||
|
||||
// heartbeaterRefusesAnOrphan plants a lease row for heartbeaterHolder on id
|
||||
// (which the holder does not hold), heartbeats as the holder, and asserts the
|
||||
// refusal and that the row is gone.
|
||||
func heartbeaterRefusesAnOrphan(t *testing.T, ctx context.Context, fixture HeartbeaterFixture, id string, status types.Status, foreign string) {
|
||||
t.Helper()
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
if err := fixture.Exec(ctx, []SQLStatement{
|
||||
{Query: "DELETE FROM leases WHERE issue_id = ?", Args: []any{id}},
|
||||
{
|
||||
Query: "INSERT INTO leases (issue_id, holder, granted_at, lease_expires_at, heartbeat_at, granted_node) VALUES (?, ?, ?, ?, ?, '')",
|
||||
Args: []any{id, heartbeaterHolder, now.Add(-time.Minute), now.Add(time.Hour), now.Add(-time.Minute)},
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("plant an orphaned lease on %s: %v", id, err)
|
||||
}
|
||||
|
||||
_, err := fixture.Heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
conflict := heartbeaterConflict(t, err)
|
||||
if foreign != "" {
|
||||
if !errors.Is(err, publicops.ErrAlreadyClaimed) || conflict.Holder != foreign {
|
||||
t.Fatalf("heartbeat over an orphaned lease on an issue %s holds = %v (%+v), want ErrAlreadyClaimed naming %s", foreign, err, conflict, foreign)
|
||||
}
|
||||
} else if !errors.Is(err, publicops.ErrLeaseNotHeld) || conflict.Holder != "" {
|
||||
t.Fatalf("heartbeat over an orphaned lease on a %s issue = %v (%+v), want ErrLeaseNotHeld", status, err, conflict)
|
||||
}
|
||||
if conflict.IssueID != id || conflict.Status != status || conflict.Ephemeral {
|
||||
t.Fatalf("LeaseConflictError = %+v, want {IssueID:%s Status:%s Ephemeral:false}", conflict, id, status)
|
||||
}
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 0 {
|
||||
t.Fatalf("lease rows on %s after the refusal = %d, want 0: a refusal deletes the caller's orphaned lease", id, n)
|
||||
}
|
||||
// And it stays refused: the next heartbeat finds no row to refresh.
|
||||
if _, err := fixture.Heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id}); err == nil {
|
||||
t.Fatalf("a second heartbeat on %s succeeded after the orphan was refused", id)
|
||||
}
|
||||
}
|
||||
|
||||
// heartbeaterRequireStoredMatches pins the reported times to the stored row
|
||||
// (LeaseRenewal: "exactly what the lease row now holds"): lease_expires_at and
|
||||
// heartbeat_at as stored are the result's, within the columns' one-second
|
||||
// precision, and the stored expiry is the call's clock plus the store's TTL.
|
||||
func heartbeaterRequireStoredMatches(t *testing.T, ctx context.Context, fixture HeartbeaterFixture, id string, result publicops.HeartbeatResult, before, after time.Time) {
|
||||
t.Helper()
|
||||
window := func(at time.Time) (time.Time, time.Time) { return at.Add(-time.Second), at.Add(time.Second) }
|
||||
hbLo, hbHi := window(result.HeartbeatAt)
|
||||
exLo, exHi := window(result.LeaseExpiresAt)
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture,
|
||||
"SELECT COUNT(*) FROM leases WHERE issue_id = ? AND holder = ? AND heartbeat_at BETWEEN ? AND ? AND lease_expires_at BETWEEN ? AND ?",
|
||||
id, result.Holder, hbLo, hbHi, exLo, exHi); n != 1 {
|
||||
t.Fatalf("the stored lease row on %s does not hold the reported heartbeat_at %v and lease_expires_at %v (within 1s)",
|
||||
id, result.HeartbeatAt, result.LeaseExpiresAt)
|
||||
}
|
||||
ttlLo, ttlHi := before.Add(storeops.DefaultLeaseTTL), after.Add(storeops.DefaultLeaseTTL)
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture,
|
||||
"SELECT COUNT(*) FROM leases WHERE issue_id = ? AND lease_expires_at BETWEEN ? AND ?",
|
||||
id, ttlLo, ttlHi); n != 1 {
|
||||
t.Fatalf("the stored lease_expires_at on %s is not the call's clock plus the lease TTL %v", id, storeops.DefaultLeaseTTL)
|
||||
}
|
||||
if !result.HeartbeatAt.Equal(result.HeartbeatAt.Truncate(time.Second)) || !result.LeaseExpiresAt.Equal(result.LeaseExpiresAt.Truncate(time.Second)) {
|
||||
t.Fatalf("result times %v / %v carry sub-second digits the lease columns cannot hold", result.HeartbeatAt, result.LeaseExpiresAt)
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterRefusesAfterClose, ...AfterRelease, ...AfterReclaim and
|
||||
// ...AfterStatusMovedOffInProgress pin the in_progress fence through the
|
||||
// lease-row invariant (Heartbeater: "a lease row exists only while its issue
|
||||
// is in_progress with that holder as assignee"): each ends the claim one way,
|
||||
// asserts the lease row went with it, and that the former holder's heartbeat
|
||||
// is refused with ErrLeaseNotHeld, which also matches ErrNotClaimable.
|
||||
func RunHeartbeaterRefusesAfterClose(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.Lifecycle == nil {
|
||||
t.Skip("HeartbeaterFixture.Lifecycle is nil: this backend cannot close the claim")
|
||||
}
|
||||
heartbeaterRefusesAfter(t, ctx, fixture, "closed", types.StatusClosed, func(id string) error {
|
||||
_, err := fixture.Lifecycle.Close(ctx, publicops.CloseRequest{Actor: heartbeaterHolder, IssueID: id, Reason: "done"})
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
func RunHeartbeaterRefusesAfterRelease(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.Releaser == nil {
|
||||
t.Skip("HeartbeaterFixture.Releaser is nil: this backend cannot release the claim")
|
||||
}
|
||||
heartbeaterRefusesAfter(t, ctx, fixture, "released", types.StatusOpen, func(id string) error {
|
||||
_, err := fixture.Releaser.Release(ctx, publicops.ReleaseRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
func RunHeartbeaterRefusesAfterReclaim(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.LeaseReclaimer == nil || fixture.Exec == nil {
|
||||
t.Skip("HeartbeaterFixture.LeaseReclaimer/Exec is nil: this backend cannot reclaim the claim")
|
||||
}
|
||||
heartbeaterRefusesAfter(t, ctx, fixture, "reclaimed", types.StatusOpen, func(id string) error {
|
||||
if err := fixture.Exec(ctx, []SQLStatement{{
|
||||
Query: "UPDATE leases SET lease_expires_at = ? WHERE issue_id = ?",
|
||||
Args: []any{time.Now().UTC().Add(-time.Hour), id},
|
||||
}}); err != nil {
|
||||
return err
|
||||
}
|
||||
result, err := fixture.LeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{
|
||||
Actor: heartbeaterReaper, Filter: publicops.ReclaimFilter{IDs: []string{id}},
|
||||
})
|
||||
if err == nil && len(result.Reclaimed) != 1 {
|
||||
t.Fatalf("Reclaimed = %+v, want exactly %s", result.Reclaimed, id)
|
||||
}
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
func RunHeartbeaterRefusesAfterStatusMovedOffInProgress(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.Lifecycle == nil {
|
||||
t.Skip("HeartbeaterFixture.Lifecycle is nil: this backend cannot move the status")
|
||||
}
|
||||
heartbeaterRefusesAfter(t, ctx, fixture, "blocked", types.StatusBlocked, func(id string) error {
|
||||
_, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{
|
||||
Actor: heartbeaterHolder, IssueID: id,
|
||||
Patch: publicops.IssuePatch{Status: publicops.Field[publicops.Status]{Set: true, Value: types.StatusBlocked}},
|
||||
})
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
// RunHeartbeaterRefusesAWisp pins the plane rule (Heartbeater REFUSALS: "a
|
||||
// wisp ... ErrLeaseNotHeld"): wisps are never leased, so even a wisp the
|
||||
// caller holds in_progress is refused, and no lease row appears for it.
|
||||
func RunHeartbeaterRefusesAWisp(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.CreateWisp == nil || fixture.QueryScalar == nil {
|
||||
t.Skip("HeartbeaterFixture.CreateWisp/QueryScalar is nil: this backend cannot seed a wisp")
|
||||
}
|
||||
id := fixture.IssuePrefix + "-wisp"
|
||||
if err := fixture.CreateWisp(ctx, &types.Issue{
|
||||
ID: id, Title: id, Status: types.StatusInProgress, Assignee: heartbeaterHolder,
|
||||
Priority: 2, IssueType: types.TypeTask, Ephemeral: true,
|
||||
}, "seed"); err != nil {
|
||||
t.Fatalf("seed wisp %s: %v", id, err)
|
||||
}
|
||||
|
||||
_, err := fixture.Heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
if !errors.Is(err, publicops.ErrLeaseNotHeld) || !errors.Is(err, publicops.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat on a wisp = %v, want ErrLeaseNotHeld matching ErrNotClaimable", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "ephemeral") {
|
||||
t.Fatalf("heartbeat on a wisp = %q, want the refusal to say the issue is ephemeral", err)
|
||||
}
|
||||
if conflict := heartbeaterConflict(t, err); conflict.IssueID != id || conflict.Holder != "" {
|
||||
t.Fatalf("LeaseConflictError = %+v, want {IssueID:%s Holder:\"\"}", conflict, id)
|
||||
}
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 0 {
|
||||
t.Fatalf("lease rows for wisp %s = %d, want 0: wisps are never leased", id, n)
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterRearmsAHandDoledClaim pins the self-heal (HeartbeatResult.
|
||||
// Rearmed): an issue put in_progress with the caller as assignee by a generic
|
||||
// update holds no lease row, and the holder's heartbeat grants one rather than
|
||||
// refusing. The re-arm is recorded as a mutation (Heartbeater: "A RE-ARM IS
|
||||
// RECORDED AS A MUTATION"): it remints the revision, which is what makes it
|
||||
// collide with a racing release or close, and records a lease_rearmed event;
|
||||
// the refresh after it does neither.
|
||||
func RunHeartbeaterRearmsAHandDoledClaim(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.Lifecycle == nil || fixture.QueryScalar == nil {
|
||||
t.Skip("HeartbeaterFixture.Lifecycle/QueryScalar is nil: this backend cannot hand-dole a claim")
|
||||
}
|
||||
id := fixture.IssuePrefix + "-doled"
|
||||
heartbeaterSeed(t, ctx, fixture, id)
|
||||
if _, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{
|
||||
Actor: heartbeaterHolder, IssueID: id,
|
||||
Patch: publicops.IssuePatch{
|
||||
Status: publicops.Field[publicops.Status]{Set: true, Value: types.StatusInProgress},
|
||||
Assignee: publicops.Field[string]{Set: true, Value: heartbeaterHolder},
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("hand-dole %s: %v", id, err)
|
||||
}
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 0 {
|
||||
t.Fatalf("lease rows after a generic update = %d, want 0: the case needs a claim with no lease", n)
|
||||
}
|
||||
revision := heartbeaterScalar[int64](t, ctx, fixture, "SELECT row_lock FROM issues WHERE id = ?", id)
|
||||
|
||||
before := time.Now().UTC().Add(-time.Second)
|
||||
result := heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
after := time.Now().UTC().Add(time.Second)
|
||||
if !result.Rearmed || result.Holder != heartbeaterHolder {
|
||||
t.Fatalf("result = %+v, want Rearmed:true Holder:%s", result, heartbeaterHolder)
|
||||
}
|
||||
heartbeaterRequireStoredMatches(t, ctx, fixture, id, result, before, after)
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture,
|
||||
"SELECT COUNT(*) FROM leases WHERE issue_id = ? AND holder = ? AND lease_expires_at > ?",
|
||||
id, heartbeaterHolder, time.Now().UTC()); n != 1 {
|
||||
t.Fatal("the re-armed heartbeat left no live lease row for the holder")
|
||||
}
|
||||
rearmed := heartbeaterScalar[int64](t, ctx, fixture, "SELECT row_lock FROM issues WHERE id = ?", id)
|
||||
if rearmed == revision {
|
||||
t.Fatalf("row_lock stayed %d across a re-arming heartbeat; a re-arm remints it so a racing release or close collides with it", revision)
|
||||
}
|
||||
rearmEvents := func() int {
|
||||
return heartbeaterScalar[int](t, ctx, fixture,
|
||||
"SELECT COUNT(*) FROM events WHERE issue_id = ? AND event_type = ?", id, string(types.EventLeaseRearmed))
|
||||
}
|
||||
if n := rearmEvents(); n != 1 {
|
||||
t.Fatalf("lease_rearmed events on %s after a re-arm = %d, want 1: a re-arm is recorded as a mutation", id, n)
|
||||
}
|
||||
// The next heartbeat refreshes the lease it just granted: no re-arm, and
|
||||
// no revision change.
|
||||
if again := heartbeat(t, ctx, fixture.Heartbeater, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id}); again.Rearmed {
|
||||
t.Fatalf("second heartbeat after the re-arm = %+v, want Rearmed:false", again)
|
||||
}
|
||||
if got := heartbeaterScalar[int64](t, ctx, fixture, "SELECT row_lock FROM issues WHERE id = ?", id); got != rearmed {
|
||||
t.Fatalf("row_lock moved from %d to %d across a refresh after the re-arm", rearmed, got)
|
||||
}
|
||||
if n := rearmEvents(); n != 1 {
|
||||
t.Fatalf("lease_rearmed events on %s after a refresh = %d, want still 1: a refresh records nothing", id, n)
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterAnswersNotFoundForAnAbsentID pins Heartbeater REFUSALS: "an
|
||||
// IssueID that names no row on either plane: ErrNotFound".
|
||||
func RunHeartbeaterAnswersNotFoundForAnAbsentID(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
_, err := fixture.Heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{
|
||||
Actor: heartbeaterHolder, IssueID: fixture.IssuePrefix + "-absent",
|
||||
})
|
||||
if !errors.Is(err, publicops.ErrNotFound) {
|
||||
t.Fatalf("heartbeat on an absent id = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterRefusesABlankOrOverLongActor pins the request rules
|
||||
// (HeartbeatRequest.Actor, Heartbeater REFUSALS): each is a
|
||||
// *HeartbeatFieldError naming its field, matching ErrValidation.
|
||||
func RunHeartbeaterRefusesABlankOrOverLongActor(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
id := fixture.IssuePrefix + "-refusals"
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
request publicops.HeartbeatRequest
|
||||
field string
|
||||
}{
|
||||
{"an empty actor", publicops.HeartbeatRequest{IssueID: id}, publicops.HeartbeatFieldActor},
|
||||
{"a blank actor", publicops.HeartbeatRequest{Actor: " \t", IssueID: id}, publicops.HeartbeatFieldActor},
|
||||
{"an over-long actor", publicops.HeartbeatRequest{Actor: strings.Repeat("a", types.MaxFieldLen+1), IssueID: id}, publicops.HeartbeatFieldActor},
|
||||
{"an empty id", publicops.HeartbeatRequest{Actor: heartbeaterHolder}, publicops.HeartbeatFieldIssueID},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
_, err := fixture.Heartbeater.Heartbeat(ctx, test.request)
|
||||
if !errors.Is(err, publicops.ErrValidation) {
|
||||
t.Fatalf("Heartbeat(%s) = %v, want ErrValidation", test.name, err)
|
||||
}
|
||||
var field *publicops.HeartbeatFieldError
|
||||
if !errors.As(err, &field) || field.Field != test.field {
|
||||
t.Fatalf("Heartbeat(%s) = %v, want a *HeartbeatFieldError naming %q", test.name, err, test.field)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// RunHeartbeaterDoesNotMutateTheRequest pins the role's "never mutate
|
||||
// caller-owned request values".
|
||||
func RunHeartbeaterDoesNotMutateTheRequest(t *testing.T, ctx context.Context, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
id := fixture.IssuePrefix + "-nomutate"
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
request := publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id}
|
||||
want := request
|
||||
heartbeat(t, ctx, fixture.Heartbeater, request)
|
||||
if request != want {
|
||||
t.Fatalf("request after Heartbeat = %+v, want %+v", request, want)
|
||||
}
|
||||
}
|
||||
|
||||
// heartbeaterRefusesAfter seeds and claims one issue, ends the claim with end,
|
||||
// and asserts the invariant and the refusal.
|
||||
func heartbeaterRefusesAfter(t *testing.T, ctx context.Context, fixture HeartbeaterFixture, suffix string, status types.Status, end func(id string) error) {
|
||||
t.Helper()
|
||||
if fixture.QueryScalar == nil {
|
||||
t.Skip("HeartbeaterFixture.QueryScalar is nil: this backend cannot read the lease row")
|
||||
}
|
||||
id := fixture.IssuePrefix + "-after-" + suffix
|
||||
heartbeaterSeedClaimed(t, ctx, fixture, id)
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 1 {
|
||||
t.Fatalf("lease rows for claimed %s = %d, want 1", id, n)
|
||||
}
|
||||
if err := end(id); err != nil {
|
||||
t.Fatalf("end the claim on %s (%s): %v", id, suffix, err)
|
||||
}
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 0 {
|
||||
t.Fatalf("lease rows for %s %s = %d, want 0: the path that ended the claim left its lease row", suffix, id, n)
|
||||
}
|
||||
|
||||
_, err := fixture.Heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
if !errors.Is(err, publicops.ErrLeaseNotHeld) || !errors.Is(err, publicops.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat after the claim was %s = %v, want ErrLeaseNotHeld matching ErrNotClaimable", suffix, err)
|
||||
}
|
||||
if errors.Is(err, publicops.ErrAlreadyClaimed) {
|
||||
t.Fatalf("heartbeat after the claim was %s = %v also matches ErrAlreadyClaimed", suffix, err)
|
||||
}
|
||||
conflict := heartbeaterConflict(t, err)
|
||||
if conflict.IssueID != id || conflict.Holder != "" || conflict.Status != status {
|
||||
t.Fatalf("LeaseConflictError = %+v, want {IssueID:%s Holder:\"\" Status:%s}", conflict, id, status)
|
||||
}
|
||||
if n := heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 0 {
|
||||
t.Fatalf("a refused heartbeat left %d lease row(s) for %s", n, id)
|
||||
}
|
||||
}
|
||||
|
||||
// heartbeaterRowSnapshot is every durable trace a heartbeat must not leave.
|
||||
// A field the backend cannot observe holds -1.
|
||||
type heartbeaterRowSnapshot struct {
|
||||
RowLock int64
|
||||
CurrentRevision int64
|
||||
UpdatedAt string
|
||||
Events int
|
||||
JournalMaxSeq int64
|
||||
IssueVersions int
|
||||
History int
|
||||
}
|
||||
|
||||
func heartbeaterSnapshot(t *testing.T, ctx context.Context, fixture HeartbeaterFixture, id string) heartbeaterRowSnapshot {
|
||||
t.Helper()
|
||||
snap := heartbeaterRowSnapshot{
|
||||
RowLock: heartbeaterScalar[int64](t, ctx, fixture, "SELECT row_lock FROM issues WHERE id = ?", id),
|
||||
CurrentRevision: heartbeaterScalar[int64](t, ctx, fixture, "SELECT current_revision FROM issues WHERE id = ?", id),
|
||||
UpdatedAt: heartbeaterScalar[string](t, ctx, fixture, "SELECT CAST(updated_at AS CHAR) FROM issues WHERE id = ?", id),
|
||||
Events: heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM events WHERE issue_id = ?", id),
|
||||
JournalMaxSeq: -1,
|
||||
IssueVersions: -1,
|
||||
History: -1,
|
||||
}
|
||||
if fixture.SetJournalEnabled != nil {
|
||||
snap.JournalMaxSeq = heartbeaterScalar[int64](t, ctx, fixture, "SELECT COALESCE(MAX(seq), 0) FROM bd_events_journal")
|
||||
}
|
||||
if fixture.SetVersionedHistoryEnabled != nil {
|
||||
snap.IssueVersions = heartbeaterScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM issue_versions WHERE issue_id = ?", id)
|
||||
}
|
||||
if fixture.CountHistory != nil {
|
||||
n, err := fixture.CountHistory(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("CountHistory: %v", err)
|
||||
}
|
||||
snap.History = n
|
||||
}
|
||||
return snap
|
||||
}
|
||||
|
||||
func heartbeaterFired(t *testing.T, ctx context.Context, fixture HeartbeaterFixture, id string) int {
|
||||
t.Helper()
|
||||
if fixture.HookedHeartbeater == nil || fixture.FiredUpdates == nil {
|
||||
return 0
|
||||
}
|
||||
ids, err := fixture.FiredUpdates(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("FiredUpdates: %v", err)
|
||||
}
|
||||
n := 0
|
||||
for _, fired := range ids {
|
||||
if fired == id {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func requireHeartbeaterExec(t *testing.T, fixture HeartbeaterFixture) {
|
||||
t.Helper()
|
||||
if fixture.Exec == nil || fixture.QueryScalar == nil {
|
||||
t.Skip("HeartbeaterFixture.Exec/QueryScalar is nil: this backend cannot read or age a lease row")
|
||||
}
|
||||
}
|
||||
|
||||
func heartbeaterSeed(t *testing.T, ctx context.Context, fixture HeartbeaterFixture, id string) {
|
||||
t.Helper()
|
||||
if err := fixture.CreateIssue(ctx, &types.Issue{
|
||||
ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask,
|
||||
}, "seed"); err != nil {
|
||||
t.Fatalf("seed %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
|
||||
// heartbeaterSeedClaimed creates id and claims it as heartbeaterHolder through
|
||||
// the leg's Claimer, answering the claim's row version.
|
||||
func heartbeaterSeedClaimed(t *testing.T, ctx context.Context, fixture HeartbeaterFixture, id string) int64 {
|
||||
t.Helper()
|
||||
if fixture.Claimer == nil {
|
||||
t.Skip("HeartbeaterFixture.Claimer is nil: this backend cannot take the claim a heartbeat refreshes")
|
||||
}
|
||||
heartbeaterSeed(t, ctx, fixture, id)
|
||||
claimed, err := fixture.Claimer.Claim(ctx, publicops.ClaimRequest{Actor: heartbeaterHolder, IssueID: id})
|
||||
if err != nil {
|
||||
t.Fatalf("claim %s as %s: %v", id, heartbeaterHolder, err)
|
||||
}
|
||||
if claimed.Issue == nil {
|
||||
t.Fatalf("claim %s answered no issue", id)
|
||||
}
|
||||
return claimed.Issue.RowVersion
|
||||
}
|
||||
|
||||
func heartbeat(t *testing.T, ctx context.Context, surface publicops.Heartbeater, request publicops.HeartbeatRequest) publicops.HeartbeatResult {
|
||||
t.Helper()
|
||||
result, err := surface.Heartbeat(ctx, request)
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeat(%+v): %v", request, err)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func heartbeaterConflict(t *testing.T, err error) *publicops.LeaseConflictError {
|
||||
t.Helper()
|
||||
var conflict *publicops.LeaseConflictError
|
||||
if !errors.As(err, &conflict) {
|
||||
t.Fatalf("refusal %v (%T) is not a *LeaseConflictError", err, err)
|
||||
}
|
||||
return conflict
|
||||
}
|
||||
|
||||
func heartbeaterScalar[T any](t *testing.T, ctx context.Context, fixture HeartbeaterFixture, query string, args ...any) T {
|
||||
t.Helper()
|
||||
var value T
|
||||
if err := fixture.QueryScalar(ctx, query, args, &value); err != nil {
|
||||
t.Fatalf("%s %v: %v", query, args, err)
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func heartbeaterAbs(d time.Duration) time.Duration {
|
||||
if d < 0 {
|
||||
return -d
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
|
@ -33,6 +33,7 @@ go_test(
|
|||
"attach_test.go",
|
||||
"e2e_cli_test.go",
|
||||
"e2e_export_test.go",
|
||||
"e2e_heartbeat_test.go",
|
||||
"e2e_list_filters_test.go",
|
||||
"e2e_list_page_edges_test.go",
|
||||
"e2e_mol_test.go",
|
||||
|
|
@ -62,6 +63,7 @@ go_test(
|
|||
"//internal/storage/backends",
|
||||
"//internal/storage/domain",
|
||||
"//internal/storage/embeddeddolt",
|
||||
"//internal/storage/issueops",
|
||||
"//internal/testutil/bazeltest",
|
||||
],
|
||||
)
|
||||
|
|
|
|||
|
|
@ -142,6 +142,8 @@ func corpusServeConfig(t *testing.T, s storage.DoltStorage) httpapi.Config {
|
|||
must("BatchGetter")
|
||||
cfg.LeaseReclaimer, err = s.LeaseReclaimer()
|
||||
must("LeaseReclaimer")
|
||||
cfg.Heartbeater, err = s.Heartbeater()
|
||||
must("Heartbeater")
|
||||
cfg.Relations, err = s.IssueRelations()
|
||||
must("IssueRelations")
|
||||
cfg.Commenter, err = s.Commenter()
|
||||
|
|
|
|||
32
backend/http/corpus/testdata/corpus.yaml
vendored
32
backend/http/corpus/testdata/corpus.yaml
vendored
|
|
@ -96,6 +96,9 @@ files:
|
|||
{"key": "t-ifa", "id": "e2e-t-ifa", "title": "target ifa", "type": "task", "assignee": "worker", "metadata": {"seeded": "keep"}},
|
||||
{"key": "t-death", "id": "e2e-t-death", "title": "target death", "type": "task", "assignee": "worker"},
|
||||
{"key": "t-rc", "id": "e2e-t-rc", "title": "target rc", "type": "task", "assignee": "worker"},
|
||||
{"key": "t-hb", "id": "e2e-t-hb", "title": "target hb", "type": "task"},
|
||||
{"key": "t-hb2", "id": "e2e-t-hb2", "title": "target hb2", "type": "task"},
|
||||
{"key": "t-hb3", "id": "e2e-t-hb3", "title": "target hb3", "type": "task"},
|
||||
{"key": "t-boot", "id": "e2e-t-boot", "title": "target boot", "type": "task"},
|
||||
{"key": "t-ro", "id": "e2e-t-ro", "title": "target reopen", "type": "task"},
|
||||
{"key": "t-dr", "id": "e2e-t-dr", "title": "target dep remove", "type": "task", "deps": [{"target": "t-db"}]},
|
||||
|
|
@ -710,6 +713,35 @@ shapes:
|
|||
compare: exit
|
||||
verify: [[show, --json, e2e-t-rc]]
|
||||
pending: {slice: "SRV-1", reason: "the reference server does not advertise issues.reclaim (B3 #12), so the client refuses at its capability preflight"}
|
||||
- id: gc-heartbeat
|
||||
source: gc formulas/mol-do-work.toml:47 (gc bd heartbeat <id> -> bd heartbeat)
|
||||
setup: [[update, e2e-t-hb, --claim]]
|
||||
args: [heartbeat, e2e-t-hb, --json]
|
||||
verify: [[show, --json, e2e-t-hb]]
|
||||
pending: {slice: "reference-server-heartbeat", reason: "the reference server does not advertise issues.heartbeat, so the client refuses at its capability preflight; pinned passing by backend/http TestE2E_HeartbeatOverHTTP; remove the mark when reference_server.json is upgraded"}
|
||||
- id: gc-heartbeat-non-holder
|
||||
source: gc formulas/mol-do-work.toml:47 (a worker heartbeating a bead another actor holds)
|
||||
setup: [[update, e2e-t-hb2, --claim, --actor, alice]]
|
||||
args: [heartbeat, e2e-t-hb2]
|
||||
expect_exit: 1
|
||||
expect_stderr: "(?i)claimed by|alice"
|
||||
compare: exit
|
||||
pending: {slice: "reference-server-heartbeat", reason: "the reference server does not advertise issues.heartbeat, so the client refuses at its capability preflight; pinned passing by backend/http TestE2E_HeartbeatOverHTTP; remove the mark when reference_server.json is upgraded"}
|
||||
- id: gc-heartbeat-closed
|
||||
source: gc formulas/mol-do-work.toml:47 (a worker heartbeating a bead that was closed under it)
|
||||
setup: [[update, e2e-t-hb3, --claim], [close, e2e-t-hb3]]
|
||||
args: [heartbeat, e2e-t-hb3]
|
||||
expect_exit: 1
|
||||
expect_stderr: "(?i)status closed|is closed"
|
||||
compare: exit
|
||||
pending: {slice: "reference-server-heartbeat", reason: "the reference server does not advertise issues.heartbeat, so the client refuses at its capability preflight; pinned passing by backend/http TestE2E_HeartbeatOverHTTP; remove the mark when reference_server.json is upgraded"}
|
||||
- id: gc-heartbeat-wisp
|
||||
source: gc formulas/mol-do-work.toml:47 (a heartbeat on a wisp, which is never leased)
|
||||
args: [heartbeat, e2e-wisp-2]
|
||||
expect_exit: 1
|
||||
expect_stderr: "(?i)ephemeral"
|
||||
compare: exit
|
||||
pending: {slice: "reference-server-heartbeat", reason: "the reference server does not advertise issues.heartbeat, so the client refuses at its capability preflight; pinned passing by backend/http TestE2E_HeartbeatOverHTTP; remove the mark when reference_server.json is upgraded"}
|
||||
- id: mc-promote
|
||||
source: mc mol-normalized-issue-pr-v1 (promote --reason)
|
||||
args: [promote, e2e-wisp-p, --reason, "keep it"]
|
||||
|
|
|
|||
|
|
@ -733,6 +733,15 @@ func startE2EServer(t *testing.T) string {
|
|||
// the one a prior `bd connect` already pinned, to prove the pin is honored
|
||||
// rather than silently replaced.
|
||||
func startE2EServerWithIdentity(t *testing.T, projectID, database string) string {
|
||||
t.Helper()
|
||||
addr, _ := startE2EServerWithStore(t, projectID, database)
|
||||
return addr
|
||||
}
|
||||
|
||||
// startE2EServerWithStore is startE2EServerWithIdentity that also hands back
|
||||
// the reference store the server serves, for a test that must plant or age
|
||||
// state no wire verb can reach (an aged lease row).
|
||||
func startE2EServerWithStore(t *testing.T, projectID, database string) (string, *embeddeddolt.EmbeddedDoltStore) {
|
||||
t.Helper()
|
||||
beadsDir := t.TempDir()
|
||||
ctx := context.Background()
|
||||
|
|
@ -761,7 +770,7 @@ func startE2EServerWithIdentity(t *testing.T, projectID, database string) string
|
|||
t.Error("the in-process server did not shut down")
|
||||
}
|
||||
})
|
||||
return srv.Addr()
|
||||
return srv.Addr(), reference
|
||||
}
|
||||
|
||||
func e2eServeConfig(t *testing.T, s storage.DoltStorage, projectID, database string) httpapi.Config {
|
||||
|
|
@ -803,6 +812,8 @@ func e2eServeConfig(t *testing.T, s storage.DoltStorage, projectID, database str
|
|||
fail("BatchGetter")
|
||||
cfg.LeaseReclaimer, err = s.LeaseReclaimer()
|
||||
fail("LeaseReclaimer")
|
||||
cfg.Heartbeater, err = s.Heartbeater()
|
||||
fail("Heartbeater")
|
||||
cfg.Relations, err = s.IssueRelations()
|
||||
fail("IssueRelations")
|
||||
cfg.Commenter, err = s.Commenter()
|
||||
|
|
|
|||
185
backend/http/e2e_heartbeat_test.go
Normal file
185
backend/http/e2e_heartbeat_test.go
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
//go:build cgo
|
||||
|
||||
package bdhttp_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
)
|
||||
|
||||
// TestE2E_HeartbeatOverHTTP runs `bd heartbeat <id>` — the keepalive gc's
|
||||
// mol-do-work formula issues every ~90s (`gc bd heartbeat`, corpus shapes
|
||||
// gc-heartbeat*) — through the real `bd` binary against an in-process `bd
|
||||
// serve`.
|
||||
//
|
||||
// The CLI takes the store's Heartbeater, which over HTTP dials POST
|
||||
// /v0/beads/issues/{id}:heartbeat. What a worker relies on is pinned here end
|
||||
// to end: exit 0 and the {"id","status":"heartbeat","owner"} shape for the
|
||||
// holder, with the row's revision and updated_at UNCHANGED (the heartbeat
|
||||
// writes only the lease, so the holder's guarded writes keep their token);
|
||||
// exit 1 for a foreign holder (whose lease stays untouched), a closed issue, a
|
||||
// wisp and an unclaimed issue, each saying which. Against a server that does
|
||||
// not advertise issues.heartbeat (an older server's handshake shape) the
|
||||
// client refuses BEFORE the dial, naming the token, and never falls back.
|
||||
func TestE2E_HeartbeatOverHTTP(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
bin := buildBD(t)
|
||||
addr, reference := startE2EServerWithStore(t, e2eProjectID, e2eDatabase)
|
||||
leaseRow := func(id string) string {
|
||||
t.Helper()
|
||||
var row string
|
||||
if err := reference.RunEventsMaintenanceTx(context.Background(), func(ctx context.Context, tx issueops.DBTX) error {
|
||||
return tx.QueryRowContext(ctx,
|
||||
"SELECT CONCAT(holder, '|', CAST(lease_expires_at AS CHAR), '|', CAST(heartbeat_at AS CHAR)) FROM leases WHERE issue_id = ?", id,
|
||||
).Scan(&row)
|
||||
}); err != nil {
|
||||
t.Fatalf("read the lease row on %s: %v", id, err)
|
||||
}
|
||||
return row
|
||||
}
|
||||
ageLease := func(id string) {
|
||||
t.Helper()
|
||||
aged := time.Now().UTC().Add(-time.Minute).Truncate(time.Second)
|
||||
if err := reference.RunEventsMaintenanceTx(context.Background(), func(ctx context.Context, tx issueops.DBTX) error {
|
||||
_, err := tx.ExecContext(ctx, "UPDATE leases SET lease_expires_at = ?, heartbeat_at = ? WHERE issue_id = ?", aged, aged, id)
|
||||
return err
|
||||
}); err != nil {
|
||||
t.Fatalf("age the lease on %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
|
||||
heartbeatPosts := func(paths []string) int {
|
||||
n := 0
|
||||
for _, p := range paths {
|
||||
if strings.HasPrefix(p, "POST ") && strings.HasSuffix(p, ":heartbeat") {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
t.Run("advertised", func(t *testing.T) {
|
||||
proxyAddr, counter := startCountingProxy(t, addr)
|
||||
f := newHTTPFixture(t, bin, proxyAddr)
|
||||
|
||||
id := f.create("heartbeat target")
|
||||
f.mustRun(nil, "update", id, "--claim", "--actor", "worker-7")
|
||||
before := f.mustRun(nil, "show", id, "--json").stdout
|
||||
revision := firstJSONField(t, before, "revision")
|
||||
if revision == "" {
|
||||
t.Fatalf("bd show --json reported no revision: %s", before)
|
||||
}
|
||||
updatedAt := firstJSONField(t, before, "updated_at")
|
||||
|
||||
counter.reset()
|
||||
r := f.mustRun(nil, "heartbeat", id, "--actor", "worker-7", "--json")
|
||||
var out map[string]any
|
||||
if err := json.Unmarshal([]byte(strings.TrimSpace(r.stdout)), &out); err != nil {
|
||||
t.Fatalf("parse bd heartbeat --json: %v\n%s", err, r.stdout)
|
||||
}
|
||||
if out["id"] != id || out["status"] != "heartbeat" || out["owner"] != "worker-7" {
|
||||
t.Errorf("bd heartbeat --json = %v, want {id: %s, status: heartbeat, owner: worker-7}", out, id)
|
||||
}
|
||||
if n := heartbeatPosts(counter.snapshot()); n != 1 {
|
||||
t.Errorf("bd heartbeat sent %d :heartbeat requests, want exactly one: %v", n, counter.snapshot())
|
||||
}
|
||||
for _, p := range counter.snapshot() {
|
||||
if strings.HasPrefix(p, "PATCH ") || (strings.HasPrefix(p, "POST ") && !strings.HasSuffix(p, ":heartbeat")) {
|
||||
t.Errorf("bd heartbeat sent %s; a heartbeat writes nothing but the lease", p)
|
||||
}
|
||||
}
|
||||
|
||||
after := f.mustRun(nil, "show", id, "--json").stdout
|
||||
if got := firstJSONField(t, after, "revision"); got != revision {
|
||||
t.Errorf("revision moved across a heartbeat: %s -> %s; a guarded write against the pre-heartbeat token would now fail", revision, got)
|
||||
}
|
||||
if got := firstJSONField(t, after, "updated_at"); got != updatedAt {
|
||||
t.Errorf("updated_at moved across a heartbeat: %s -> %s", updatedAt, got)
|
||||
}
|
||||
// The holder's guarded write with the pre-heartbeat token still lands.
|
||||
f.mustRun(nil, "update", id, "--notes", "still mine", "--if-revision", revision, "--actor", "worker-7")
|
||||
|
||||
// A second heartbeat is fine: it only moves the expiry later.
|
||||
f.mustRun(nil, "heartbeat", id, "--actor", "worker-7")
|
||||
})
|
||||
|
||||
t.Run("refusals", func(t *testing.T) {
|
||||
f := newHTTPFixture(t, bin, addr)
|
||||
|
||||
foreign := f.create("held by alice")
|
||||
f.mustRun(nil, "update", foreign, "--claim", "--actor", "alice")
|
||||
// Aged first, so a heartbeat that ignored the holder fence would
|
||||
// really change the row rather than rewrite the claim's own second.
|
||||
ageLease(foreign)
|
||||
held := leaseRow(foreign)
|
||||
r := f.run(nil, "heartbeat", foreign, "--actor", "bob")
|
||||
if r.code != 1 || !strings.Contains(r.stderr, "claimed by alice") {
|
||||
t.Errorf("heartbeat by a non-holder: exit %d stderr=%s, want exit 1 naming the holder", r.code, r.stderr)
|
||||
}
|
||||
if got := leaseRow(foreign); got != held {
|
||||
t.Errorf("a refused heartbeat by a non-holder moved alice's lease: %s -> %s", held, got)
|
||||
}
|
||||
|
||||
// Each lease_not_held names the one state it found, and no other.
|
||||
says := func(what, stderr, want string, not ...string) {
|
||||
t.Helper()
|
||||
if !strings.Contains(stderr, want) {
|
||||
t.Errorf("heartbeat on %s: stderr=%s, want it to say %q", what, stderr, want)
|
||||
}
|
||||
for _, n := range not {
|
||||
if strings.Contains(stderr, n) {
|
||||
t.Errorf("heartbeat on %s: stderr=%s says %q, which is not the state it found", what, stderr, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
closed := f.create("closed under its worker")
|
||||
f.mustRun(nil, "update", closed, "--claim", "--actor", "worker-7")
|
||||
f.mustRun(nil, "close", closed, "--actor", "worker-7")
|
||||
r = f.run(nil, "heartbeat", closed, "--actor", "worker-7")
|
||||
if r.code != 1 {
|
||||
t.Errorf("heartbeat on a closed issue exited %d, want 1: %s", r.code, r.stderr)
|
||||
}
|
||||
says("a closed issue", r.stderr, "is closed", "ephemeral", "not claimed")
|
||||
|
||||
wisp := f.create("a wisp is never leased", "--ephemeral")
|
||||
r = f.run(nil, "heartbeat", wisp, "--actor", "worker-7")
|
||||
if r.code != 1 {
|
||||
t.Errorf("heartbeat on a wisp exited %d, want 1: %s", r.code, r.stderr)
|
||||
}
|
||||
says("a wisp", r.stderr, "ephemeral wisp", "closed", "not claimed")
|
||||
|
||||
unclaimed := f.create("never claimed")
|
||||
r = f.run(nil, "heartbeat", unclaimed, "--actor", "worker-7")
|
||||
if r.code != 1 {
|
||||
t.Errorf("heartbeat on an unclaimed issue exited %d, want 1: %s", r.code, r.stderr)
|
||||
}
|
||||
says("an unclaimed issue", r.stderr, "not claimed", "closed", "ephemeral")
|
||||
})
|
||||
|
||||
t.Run("older server mask", func(t *testing.T) {
|
||||
proxyAddr, counter := startMaskingProxy(t, addr, mcContextMask)
|
||||
f := newHTTPFixture(t, bin, proxyAddr)
|
||||
id := f.create("an old server cannot refresh this lease")
|
||||
f.mustRun(nil, "update", id, "--claim", "--actor", "worker-7")
|
||||
|
||||
counter.reset()
|
||||
r := f.run(nil, "heartbeat", id, "--actor", "worker-7")
|
||||
if r.code != 1 {
|
||||
t.Fatalf("bd heartbeat against a server without issues.heartbeat exited %d, want 1: %s", r.code, r.stdout)
|
||||
}
|
||||
if !strings.Contains(r.stderr, "issues.heartbeat") {
|
||||
t.Errorf("refusal does not name the missing capability: %s", r.stderr)
|
||||
}
|
||||
for _, p := range counter.snapshot() {
|
||||
if !strings.HasPrefix(p, "GET ") {
|
||||
t.Errorf("a refused heartbeat sent %s; the refusal must come before any write", p)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
@ -118,6 +118,7 @@ const (
|
|||
EventLabelAdded = types.EventLabelAdded
|
||||
EventLabelRemoved = types.EventLabelRemoved
|
||||
EventLeaseReclaimed = types.EventLeaseReclaimed
|
||||
EventLeaseRearmed = types.EventLeaseRearmed
|
||||
EventCompacted = types.EventCompacted
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import (
|
|||
|
||||
"github.com/steveyegge/beads/internal/metrics"
|
||||
"github.com/steveyegge/beads/internal/ui"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
var heartbeatCmd = &cobra.Command{
|
||||
|
|
@ -24,10 +25,13 @@ it up. Heartbeat pushes lease_expires_at forward and stamps heartbeat_at = now.
|
|||
Only the current owner may heartbeat. If the lease has already been reclaimed or
|
||||
the issue closed, heartbeat fails so the worker learns to stop.
|
||||
|
||||
Leases live in an ephemeral, node-local table: heartbeats write no Dolt commit
|
||||
and no history, so any cadence comfortably below the TTL is fine. Leases are
|
||||
only enforceable on the node that granted them; cross-machine claim visibility
|
||||
rides the issue's status and assignee, which do commit.
|
||||
Leases live in an ephemeral, node-local table: a heartbeat that refreshes your
|
||||
lease writes no Dolt commit and no history, so any cadence comfortably below
|
||||
the TTL is fine. Only a re-arm (the first heartbeat of a claim that has no
|
||||
lease, such as one set by a plain update) is recorded as a change to the issue
|
||||
and committed, once. Leases are only enforceable on the node that granted them;
|
||||
cross-machine claim visibility rides the issue's status and assignee, which do
|
||||
commit.
|
||||
|
||||
Examples:
|
||||
bd heartbeat bd-123
|
||||
|
|
@ -48,8 +52,15 @@ Examples:
|
|||
ctx := rootCtx
|
||||
id := args[0]
|
||||
|
||||
// One role on every route: the store's own
|
||||
// Heartbeater accessor on the direct routes — dolt, embedded dolt and
|
||||
// a registered HTTP backend, where it dials issues.heartbeat and an
|
||||
// older server is refused before the dial naming the token — and the
|
||||
// proxied-server provider's on the proxied one. On every leg a refresh
|
||||
// writes ONLY the lease row (no revision remint, no history, no Dolt
|
||||
// commit); a re-arm is recorded as a mutation and committed once.
|
||||
if usesProxiedServer() {
|
||||
return runHeartbeatProxiedServer(ctx, id)
|
||||
return runHeartbeatProxied(ctx, id)
|
||||
}
|
||||
|
||||
result, err := resolveAndGetIssueForMutation(ctx, store, id)
|
||||
|
|
@ -68,7 +79,14 @@ Examples:
|
|||
defer result.Close()
|
||||
|
||||
issueStore := result.Store
|
||||
if err := issueStore.HeartbeatIssue(ctx, result.ResolvedID, currentActor()); err != nil {
|
||||
heartbeater, err := issueStore.Heartbeater()
|
||||
if err != nil {
|
||||
return HandleErrorRespectJSON("heartbeat %s: %v", result.ResolvedID, err)
|
||||
}
|
||||
if _, err := heartbeater.Heartbeat(ctx, issueops.HeartbeatRequest{
|
||||
Actor: currentActor(),
|
||||
IssueID: result.ResolvedID,
|
||||
}); err != nil {
|
||||
return HandleErrorRespectJSON("heartbeat %s: %v", result.ResolvedID, err)
|
||||
}
|
||||
|
||||
|
|
@ -85,9 +103,9 @@ Examples:
|
|||
},
|
||||
}
|
||||
|
||||
// renderHeartbeatSuccess prints the success shape both storage routes share —
|
||||
// the classic path and the proxied-server path (heartbeat_proxied_server.go)
|
||||
// call exactly this code, so the --json contract workers parse
|
||||
// renderHeartbeatSuccess prints the success shape every storage route shares —
|
||||
// the direct path and the proxied-server path (runHeartbeatProxied) call
|
||||
// exactly this code, so the --json contract workers parse
|
||||
// ({"id","status":"heartbeat","owner"}) cannot drift between modes.
|
||||
func renderHeartbeatSuccess(id, title string) error {
|
||||
if jsonOutput {
|
||||
|
|
|
|||
|
|
@ -16,10 +16,11 @@ import (
|
|||
// plane (bd-aq0ql). The worker contract is the exit code (workers call
|
||||
// heartbeat every ~90s and only check rc), the --json success shape is
|
||||
// {"id","status":"heartbeat","owner"}, and — the bd-lrgn1 invariant — a
|
||||
// heartbeat mints exactly ZERO Dolt commits: the lease write lands in the
|
||||
// refresh mints exactly ZERO Dolt commits: the lease write lands in the
|
||||
// dolt_ignored leases table via the SQL-only ephemeral commit, with the same
|
||||
// timestamps/format classic `bd reclaim` staleness semantics consume (the
|
||||
// SQL body is literally issueops.HeartbeatIssueInTx in both modes).
|
||||
// SQL body is literally issueops.HeartbeatIssueInTx in both modes). A re-arm
|
||||
// is recorded as a mutation and mints exactly one (rearms_hand_doled_claim).
|
||||
func TestProxiedServerHeartbeat(t *testing.T) {
|
||||
requireSharedProxiedServer(t)
|
||||
t.Parallel()
|
||||
|
|
@ -209,9 +210,9 @@ func TestProxiedServerHeartbeat(t *testing.T) {
|
|||
t.Run("rearms_hand_doled_claim", func(t *testing.T) {
|
||||
// A claim hand-doled through a generic update never arms a lease
|
||||
// (bd-9hpgf); the holder's first heartbeat opts into lease semantics.
|
||||
// This is the INSERT path of the lease write, so it doubles as the
|
||||
// zero-Dolt-commit proof for UpsertLeaseInTx under the ephemeral
|
||||
// commit.
|
||||
// That re-arm is recorded as a mutation (row_lock reminted so it
|
||||
// collides with a racing release or close), so it mints exactly one
|
||||
// Dolt commit; the refresh after it mints none.
|
||||
issue := bdProxiedCreate(t, bd, p.dir, "Hand-doled claim", "--type", "task")
|
||||
bdProxiedUpdate(t, bd, p.dir, issue.ID, "--assignee", "hand-worker", "--status", "in_progress")
|
||||
if _, _, _, ok := leaseRow(t, issue.ID); ok {
|
||||
|
|
@ -223,8 +224,9 @@ func TestProxiedServerHeartbeat(t *testing.T) {
|
|||
if err != nil {
|
||||
t.Fatalf("re-arm heartbeat: %v\nstdout:\n%s\nstderr:\n%s", err, out, stderr)
|
||||
}
|
||||
if commitsAfter := doltCommits(t); commitsAfter != commitsBefore {
|
||||
t.Errorf("re-arm heartbeat minted Dolt commits: %d -> %d", commitsBefore, commitsAfter)
|
||||
commitsAfter := doltCommits(t)
|
||||
if commitsAfter != commitsBefore+1 {
|
||||
t.Errorf("re-arm heartbeat Dolt commits: %d -> %d, want exactly one (a re-arm is recorded as a mutation)", commitsBefore, commitsAfter)
|
||||
}
|
||||
holder, expires, _, ok := leaseRow(t, issue.ID)
|
||||
if !ok || holder != "hand-worker" {
|
||||
|
|
@ -233,5 +235,12 @@ func TestProxiedServerHeartbeat(t *testing.T) {
|
|||
if !expires.After(time.Now().UTC()) {
|
||||
t.Errorf("re-armed lease_expires_at should be in the future, got %v", expires)
|
||||
}
|
||||
refreshOut, refreshErrOut, refreshErr := bdProxiedRunBuffers(t, bd, p.dir, "--actor", "hand-worker", "heartbeat", issue.ID)
|
||||
if refreshErr != nil {
|
||||
t.Fatalf("refresh heartbeat after the re-arm: %v\nstdout:\n%s\nstderr:\n%s", refreshErr, refreshOut, refreshErrOut)
|
||||
}
|
||||
if commitsRefresh := doltCommits(t); commitsRefresh != commitsAfter {
|
||||
t.Errorf("refresh heartbeat minted Dolt commits: %d -> %d, want none", commitsAfter, commitsRefresh)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,49 +7,68 @@ import (
|
|||
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/storage/uow"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
"github.com/steveyegge/beads/internal/workapi"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// heartbeatProxiedOutcome carries what the render layer needs out of the
|
||||
// transaction: the fully-resolved ID and the title for the feedback line.
|
||||
type heartbeatProxiedOutcome struct {
|
||||
id string
|
||||
title string
|
||||
}
|
||||
|
||||
// runHeartbeatProxiedServer routes bd heartbeat through the proxied-server
|
||||
// plane. The lease write is EPHEMERAL (the dolt_ignored leases table,
|
||||
// bd-lrgn1): the transaction commits via uow.RunTxEphemeral's SQL-only form,
|
||||
// so a heartbeat mints exactly ZERO Dolt commits per invocation — the same
|
||||
// commit discipline as the classic DoltStore.HeartbeatIssue, and deliberately
|
||||
// nothing here sets commandDidWrite or creates a Dolt commit. The exit code
|
||||
// is the worker contract (workers call this every ~90s and only check rc):
|
||||
// 0 = lease refreshed; nonzero = the lease is gone (wrong owner, not
|
||||
// in_progress, closed, reclaimed) and the worker should stop.
|
||||
func runHeartbeatProxiedServer(ctx context.Context, id string) error {
|
||||
if uowProvider == nil {
|
||||
return HandleErrorRespectJSON("proxied-server UOW provider not initialized")
|
||||
}
|
||||
|
||||
res, err := uow.RunTxEphemeral(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (heartbeatProxiedOutcome, error) {
|
||||
issue, _, rerr := workapi.GetIssueOrWisp(ctx, workapi.NewUOWDetailSource(uw), id)
|
||||
if errors.Is(rerr, storage.ErrNotFound) {
|
||||
return heartbeatProxiedOutcome{}, fmt.Errorf("issue %s not found", id)
|
||||
}
|
||||
if rerr != nil {
|
||||
return heartbeatProxiedOutcome{}, fmt.Errorf("resolving %s: %w", id, rerr)
|
||||
}
|
||||
// Wisps resolve here too and are refused below: the repo verb
|
||||
// classifies them ErrNotClaimable ("is ephemeral"), same as classic.
|
||||
if herr := uw.IssueUseCase().Heartbeat(ctx, issue.ID, currentActor()); herr != nil {
|
||||
return heartbeatProxiedOutcome{}, fmt.Errorf("heartbeat %s: %w", issue.ID, herr)
|
||||
}
|
||||
return heartbeatProxiedOutcome{id: issue.ID, title: issue.Title}, nil
|
||||
})
|
||||
// runHeartbeatProxied is bd heartbeat on the proxied-server plane: the id is
|
||||
// resolved exactly, on either plane, in a read unit of work of its own, and
|
||||
// the refresh goes through the provider's OWN Heartbeater accessor (the
|
||||
// capability the uow leg commits with RunTxScoped: a refresh names no tables
|
||||
// and takes its SQL-only form, so it mints exactly ZERO Dolt commits,
|
||||
// bd-lrgn1; a re-arm mints at most one, none when a concurrent writer's Dolt
|
||||
// commit already took its row). Nothing here sets commandDidWrite.
|
||||
// The exit code is the worker contract: 0 = lease refreshed; nonzero = the
|
||||
// lease is gone and the worker should stop.
|
||||
func runHeartbeatProxied(ctx context.Context, id string) error {
|
||||
issue, err := proxiedHeartbeatTarget(ctx, id)
|
||||
if err != nil {
|
||||
return HandleErrorRespectJSON("%v", err)
|
||||
return err
|
||||
}
|
||||
heartbeater, err := proxiedHeartbeater()
|
||||
if err != nil {
|
||||
return HandleErrorRespectJSON("heartbeat %s: %v", issue.ID, err)
|
||||
}
|
||||
if _, err := heartbeater.Heartbeat(ctx, issueops.HeartbeatRequest{
|
||||
Actor: currentActor(),
|
||||
IssueID: issue.ID,
|
||||
}); err != nil {
|
||||
return HandleErrorRespectJSON("heartbeat %s: %v", issue.ID, err)
|
||||
}
|
||||
SetLastTouchedID(issue.ID)
|
||||
return renderHeartbeatSuccess(issue.ID, issue.Title)
|
||||
}
|
||||
|
||||
SetLastTouchedID(res.id)
|
||||
return renderHeartbeatSuccess(res.id, res.title)
|
||||
// proxiedHeartbeatTarget reads the row a proxied heartbeat names, on either
|
||||
// plane (a wisp resolves here and the role refuses it as ephemeral), for its
|
||||
// canonical id and the title the feedback line prints.
|
||||
func proxiedHeartbeatTarget(ctx context.Context, id string) (*types.Issue, error) {
|
||||
uw, err := proxiedOpenReadUOW(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer uw.Close(ctx)
|
||||
issue, _, err := workapi.GetIssueOrWisp(ctx, workapi.NewUOWDetailSource(uw), id)
|
||||
if errors.Is(err, storage.ErrNotFound) {
|
||||
return nil, HandleErrorRespectJSON("issue %s not found", id)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, HandleErrorRespectJSON("resolving %s: %v", id, err)
|
||||
}
|
||||
return issue, nil
|
||||
}
|
||||
|
||||
// proxiedHeartbeater takes the lease refresh through the proxied-server
|
||||
// provider's own capability accessor, proxiedLeaseReclaimer's two-step and for
|
||||
// its reason: the accessor is where each layer is added.
|
||||
func proxiedHeartbeater() (issueops.Heartbeater, error) {
|
||||
if uowProvider == nil {
|
||||
return nil, errors.New("proxied-server UOW provider not initialized")
|
||||
}
|
||||
src, ok := uowProvider.(uow.HeartbeaterSource)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("proxied-server provider %T does not offer the lease-refresh surface", uowProvider)
|
||||
}
|
||||
return src.Heartbeater()
|
||||
}
|
||||
|
|
|
|||
|
|
@ -320,6 +320,7 @@ func runServe() error {
|
|||
GraphCounter: roles.edgeCounter,
|
||||
BatchGetter: roles.batchGetter,
|
||||
LeaseReclaimer: roles.leaseReclaimer,
|
||||
Heartbeater: roles.heartbeater,
|
||||
Relations: roles.relations,
|
||||
Commenter: roles.commenter,
|
||||
BlockingAnnotator: roles.blocking,
|
||||
|
|
@ -682,6 +683,7 @@ type serveRoleSource interface {
|
|||
GraphCounter() (issueops.GraphCounter, error)
|
||||
BatchGetter() (issueops.BatchGetter, error)
|
||||
LeaseReclaimer() (issueops.LeaseReclaimer, error)
|
||||
Heartbeater() (issueops.Heartbeater, error)
|
||||
IssueRelations() (issueops.Relations, error)
|
||||
Commenter() (issueops.Commenter, error)
|
||||
BlockingAnnotator() (issueops.BlockingAnnotator, error)
|
||||
|
|
@ -750,6 +752,7 @@ func serveIssueRoles(src serveRoleSource, journalEnabled bool) (serveRoles, erro
|
|||
{"graph counter", func() (err error) { roles.edgeCounter, err = src.GraphCounter(); return }},
|
||||
{"batch getter", func() (err error) { roles.batchGetter, err = src.BatchGetter(); return }},
|
||||
{"lease reclaimer", func() (err error) { roles.leaseReclaimer, err = src.LeaseReclaimer(); return }},
|
||||
{"heartbeater", func() (err error) { roles.heartbeater, err = src.Heartbeater(); return }},
|
||||
{"issue relations", func() (err error) { roles.relations, err = src.IssueRelations(); return }},
|
||||
{"commenter", func() (err error) { roles.commenter, err = src.Commenter(); return }},
|
||||
{"blocking annotator", func() (err error) { roles.blocking, err = src.BlockingAnnotator(); return }},
|
||||
|
|
@ -866,8 +869,12 @@ type serveRoles struct {
|
|||
// leaseReclaimer is the stale-lease sweep behind
|
||||
// POST /v0/beads/issues:reclaim. Its hook decorator WRAPS (a reclaim is an
|
||||
// update per reverted row), which is why it comes off the peeled store with
|
||||
// the releaser: bd serve runs no hooks.
|
||||
// the releaser: bd serve runs no hooks. heartbeater, the lease refresh
|
||||
// behind POST /v0/beads/issues/{id}:heartbeat, comes off the same store
|
||||
// for the same reason: its decorator WRAPS too, firing on_update for a
|
||||
// re-arm (a refresh fires nothing).
|
||||
leaseReclaimer issueops.LeaseReclaimer
|
||||
heartbeater issueops.Heartbeater
|
||||
relations issueops.Relations
|
||||
commenter issueops.Commenter
|
||||
blocking issueops.BlockingAnnotator
|
||||
|
|
|
|||
|
|
@ -101,6 +101,7 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) {
|
|||
readyClaimer: &serveStubReadyClaimer{},
|
||||
releaser: &serveStubReleaser{},
|
||||
reclaimer: &serveStubLeaseReclaimer{},
|
||||
heartbeater: &serveStubHeartbeater{},
|
||||
lifecycle: &serveStubLifecycle{},
|
||||
dependencies: &serveStubDependencyEditor{},
|
||||
batchApplier: &serveStubBatchApplier{},
|
||||
|
|
@ -267,6 +268,9 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) {
|
|||
if storage.RoleFiresHooks(roles.leaseReclaimer) || roles.leaseReclaimer != issueops.LeaseReclaimer(middle.reclaimer) {
|
||||
t.Errorf("lease reclaimer came from %p, want the layer directly beneath the hooks (%p)", roles.leaseReclaimer, middle.reclaimer)
|
||||
}
|
||||
if storage.RoleFiresHooks(roles.heartbeater) || roles.heartbeater != issueops.Heartbeater(middle.heartbeater) {
|
||||
t.Errorf("heartbeater came from %p, want the layer directly beneath the hooks (%p)", roles.heartbeater, middle.heartbeater)
|
||||
}
|
||||
reader, claimer := roles.reader, roles.claimer
|
||||
// The same predicate httpapi.Listen refuses on, so a regression here is a
|
||||
// server that refuses to boot rather than one that runs hooks silently.
|
||||
|
|
@ -420,6 +424,7 @@ type serveRolesStore struct {
|
|||
readyClaimer *serveStubReadyClaimer
|
||||
releaser *serveStubReleaser
|
||||
reclaimer *serveStubLeaseReclaimer
|
||||
heartbeater *serveStubHeartbeater
|
||||
lifecycle *serveStubLifecycle
|
||||
dependencies *serveStubDependencyEditor
|
||||
batchApplier *serveStubBatchApplier
|
||||
|
|
@ -480,6 +485,13 @@ func (s *serveRolesStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) {
|
|||
return s.reclaimer, nil
|
||||
}
|
||||
|
||||
// Heartbeater carries an identifiable value so the test can assert which layer
|
||||
// bd serve took it from: the layer beneath the hooks, since the hook decorator
|
||||
// wraps it (a re-arm fires on_update).
|
||||
func (s *serveRolesStore) Heartbeater() (issueops.Heartbeater, error) {
|
||||
return s.heartbeater, nil
|
||||
}
|
||||
|
||||
// IssueLifecycle carries an identifiable value for the same reason the reader
|
||||
// and the claimer do: it is one of the roles the hook decorator wraps, so a peel
|
||||
// of the wrong depth would hand bd serve a lifecycle that runs the workspace's
|
||||
|
|
@ -676,6 +688,12 @@ func (*serveStubReleaser) Release(context.Context, issueops.ReleaseRequest) (iss
|
|||
return issueops.ReleaseResult{}, errors.ErrUnsupported
|
||||
}
|
||||
|
||||
type serveStubHeartbeater struct{}
|
||||
|
||||
func (*serveStubHeartbeater) Heartbeat(context.Context, issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
return issueops.HeartbeatResult{}, errors.ErrUnsupported
|
||||
}
|
||||
|
||||
type serveStubLeaseReclaimer struct{}
|
||||
|
||||
func (*serveStubLeaseReclaimer) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) {
|
||||
|
|
|
|||
|
|
@ -217,6 +217,12 @@ func (*serveIdentityStore) Releaser() (issueops.Releaser, error) { return serveI
|
|||
func (*serveIdentityStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) {
|
||||
return serveIdentityRole{}, nil
|
||||
}
|
||||
|
||||
// Heartbeater is declared at depth 1 for the same reason: its hook decorator
|
||||
// recurses, and the recursion would otherwise land on a nil-embedded promotion.
|
||||
func (*serveIdentityStore) Heartbeater() (issueops.Heartbeater, error) {
|
||||
return serveIdentityRole{}, nil
|
||||
}
|
||||
func (*serveIdentityStore) IssueLifecycle() (issueops.Lifecycle, error) {
|
||||
return serveIdentityRole{}, nil
|
||||
}
|
||||
|
|
@ -305,6 +311,7 @@ type serveIdentityRole struct {
|
|||
issueops.Lifecycle
|
||||
issueops.Releaser
|
||||
issueops.LeaseReclaimer
|
||||
issueops.Heartbeater
|
||||
issueops.ReadyClaimer
|
||||
issueops.BatchCloser
|
||||
issueops.MetadataCAS
|
||||
|
|
|
|||
|
|
@ -273,9 +273,10 @@ local heartbeat is keeping it alive:
|
|||
row from the snapshot, granting replica included — so this node's own stale
|
||||
lease can come back labelled with a remote one.
|
||||
|
||||
(The one path that moves a lease the other way is a heartbeat whose holder is a
|
||||
different *spelling* of the lease row's holder: it re-arms through the upsert,
|
||||
which stamps this node.)
|
||||
(No heartbeat moves an existing lease, including one whose holder is a
|
||||
different *spelling* of the lease row's holder: that row is refreshed in place
|
||||
and keeps its granting replica. Only a re-arm, which grants a lease to a live
|
||||
claim that had none, stamps this node.)
|
||||
|
||||
Recover a stranded lease with `bd reclaim --any-replica`, once you have
|
||||
confirmed the granting replica is not still reaping — that confirmation is the
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ go_library(
|
|||
"events.go",
|
||||
"events_watch.go",
|
||||
"handlers.go",
|
||||
"heartbeat.go",
|
||||
"memories.go",
|
||||
"metadata_cas.go",
|
||||
"pinning.go",
|
||||
|
|
@ -90,6 +91,7 @@ go_test(
|
|||
"events_watch_test.go",
|
||||
"gate_fields_test.go",
|
||||
"get_issue_test.go",
|
||||
"heartbeat_test.go",
|
||||
"large_apply_client_budget_test.go",
|
||||
"large_batch_apply_test.go",
|
||||
"list_comments_omitted_test.go",
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -352,6 +352,7 @@ var (
|
|||
_ uow.GraphCounterSource = timedProvider{}
|
||||
_ uow.BatchGetterSource = timedProvider{}
|
||||
_ uow.LeaseReclaimerSource = timedProvider{}
|
||||
_ uow.HeartbeaterSource = timedProvider{}
|
||||
_ uow.RelationsSource = timedProvider{}
|
||||
_ uow.CommenterSource = timedProvider{}
|
||||
_ uow.BlockingAnnotatorSource = timedProvider{}
|
||||
|
|
@ -482,6 +483,12 @@ func (p timedProvider) LeaseReclaimer() (issueops.LeaseReclaimer, error) {
|
|||
return uow.NewLeaseReclaimer(p)
|
||||
}
|
||||
|
||||
// Heartbeater builds the lease refresh OVER THIS WRAPPER, for the same reason
|
||||
// and with the same hazard as IssueReader.
|
||||
func (p timedProvider) Heartbeater() (issueops.Heartbeater, error) {
|
||||
return uow.NewHeartbeater(p)
|
||||
}
|
||||
|
||||
// MoleculeStepper builds the advance-a-molecule role OVER THIS WRAPPER, so
|
||||
// every read and the claim it composes go through this wrapper's roles.
|
||||
func (p timedProvider) MoleculeStepper() (issueops.MoleculeStepper, error) {
|
||||
|
|
|
|||
178
internal/httpapi/heartbeat.go
Normal file
178
internal/httpapi/heartbeat.go
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"github.com/steveyegge/beads/internal/httpapi/apigen"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// heartbeatRequestMembers is the document's member list for
|
||||
// HeartbeatIssueRequest. The schema is additionalProperties: false, so anything
|
||||
// else is refused BY NAME, the posture every body on this surface takes.
|
||||
var heartbeatRequestMembers = []string{claimActorMember}
|
||||
|
||||
// handleHeartbeat refreshes the lease on one issue the actor holds: the verb
|
||||
// `bd heartbeat` spells, behind issueops.Heartbeater.
|
||||
//
|
||||
// A REFRESH WRITES ONLY THE LEASE ROW, and that is the role's promise rather
|
||||
// than this handler's: no revision is reminted (so the holder's own guarded
|
||||
// writes keep their token across heartbeats), no version is recorded,
|
||||
// updated_at does not move, and no event or journal entry is written. A
|
||||
// RE-ARM (`rearmed: true`: a live claim that had no lease row) is recorded as
|
||||
// a mutation and remints the revision; see issueops.Heartbeater. The response
|
||||
// carries no `revision` member either way: it reports the lease, not the
|
||||
// issue, so a refresh publishes no unchanged token a reader could take for a
|
||||
// write, and `rearmed: true` tells the caller to re-read the issue before a
|
||||
// guarded write.
|
||||
//
|
||||
// THE ACTOR IS THE FENCE'S SUBJECT, the release's posture: the heartbeat lands
|
||||
// only while the actor holds the claim. It is caller-asserted provenance, not
|
||||
// authenticated identity, exactly as on the claim.
|
||||
//
|
||||
// PLANES: the id resolves across both, so a wisp is answered with the role's
|
||||
// typed refusal (`lease_not_held`) rather than a 404 — wisps are never leased.
|
||||
func (s *Server) handleHeartbeat(w http.ResponseWriter, r *http.Request) {
|
||||
// The custom-method dispatcher split the id off the segment and bounded it
|
||||
// before this handler was chosen at all; see customMethodTarget.
|
||||
id := r.PathValue(customMethodIDValue)
|
||||
if !s.requireNoQuery(w, r) {
|
||||
return
|
||||
}
|
||||
if !s.requireJSONContent(w, r) {
|
||||
return
|
||||
}
|
||||
actor, ok := s.heartbeatActor(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
heartbeater, err := s.heartbeater(r)
|
||||
if err != nil {
|
||||
s.failHeartbeat(w, r, err)
|
||||
return
|
||||
}
|
||||
result, err := heartbeater.Heartbeat(r.Context(), issueops.HeartbeatRequest{Actor: actor, IssueID: id})
|
||||
if err != nil {
|
||||
s.failHeartbeat(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, apigen.HeartbeatIssueResponse{
|
||||
Id: result.IssueID,
|
||||
Holder: result.Holder,
|
||||
HeartbeatAt: result.HeartbeatAt.UTC(),
|
||||
LeaseExpiresAt: result.LeaseExpiresAt.UTC(),
|
||||
Rearmed: result.Rearmed,
|
||||
})
|
||||
}
|
||||
|
||||
// heartbeatActor decodes the body and validates its one member by the claim's
|
||||
// rules (validateActor, through bodyActor), and reports whether the request may
|
||||
// proceed. Every refusal here happens BEFORE any database work.
|
||||
func (s *Server) heartbeatActor(w http.ResponseWriter, r *http.Request) (string, bool) {
|
||||
members, res := decodeJSONObjectBody(w, r)
|
||||
if res != nil {
|
||||
s.fail(w, r, *res)
|
||||
return "", false
|
||||
}
|
||||
if offender, unknown := unknownMember(members, heartbeatRequestMembers); unknown {
|
||||
s.failUnknownMember(w, r, offender, heartbeatRequestMembers)
|
||||
return "", false
|
||||
}
|
||||
return s.bodyActor(w, r, members)
|
||||
}
|
||||
|
||||
// failHeartbeat answers a failed heartbeat, mapping the role's TYPED refusals
|
||||
// onto the frozen codes.
|
||||
//
|
||||
// The 409 arms are matched first, from *issueops.LeaseConflictError's typed
|
||||
// fields — never from the sentinel's prose — and BEFORE the generic arms:
|
||||
// ErrLeaseNotHeld also matches ErrNotClaimable, which ClassifyError would
|
||||
// answer `not_claimable`, the code CodeLeaseNotHeld's doc says this must not
|
||||
// be.
|
||||
//
|
||||
// A FOREIGN HOLDER is the claim's own conflict shape: `already_claimed`
|
||||
// naming the holder in `assignee`, with `issue_status`. The role reads both
|
||||
// inside the refusing transaction.
|
||||
func (s *Server) failHeartbeat(w http.ResponseWriter, r *http.Request, err error) {
|
||||
// The 4xx path does not log by default, so the refusals whose real reason
|
||||
// the response replaces with the server's own words are recorded here.
|
||||
refused := func() {
|
||||
s.event("request_refused", "request_id", requestInfo(r.Context()).id, "error", err.Error())
|
||||
}
|
||||
|
||||
var conflict *issueops.LeaseConflictError
|
||||
var field *issueops.HeartbeatFieldError
|
||||
switch {
|
||||
case errors.As(err, &conflict) && errors.Is(err, issueops.ErrAlreadyClaimed):
|
||||
res := newResult(CodeAlreadyClaimed,
|
||||
"another actor holds this issue's claim; nothing on the issue was written")
|
||||
if conflict.Holder != "" {
|
||||
res = res.WithAssignee(conflict.Holder)
|
||||
}
|
||||
if conflict.Status != "" {
|
||||
res = res.WithIssueStatus(string(conflict.Status))
|
||||
}
|
||||
s.fail(w, r, res)
|
||||
|
||||
case errors.As(err, &conflict) && errors.Is(err, issueops.ErrLeaseNotHeld):
|
||||
refused()
|
||||
// The detail names the state the refusing transaction found, from the
|
||||
// typed fields (never the role's prose): a wisp, an unclaimed issue, or
|
||||
// the status the claim was ended into. A wisp also carries
|
||||
// `reason: ephemeral`, the typed form a client reads.
|
||||
res := newResult(CodeLeaseNotHeld, leaseNotHeldDetail(conflict))
|
||||
if conflict.Status != "" {
|
||||
res = res.WithIssueStatus(string(conflict.Status))
|
||||
}
|
||||
if conflict.Ephemeral {
|
||||
res = res.WithReason(ReasonEphemeral)
|
||||
}
|
||||
s.fail(w, r, res)
|
||||
|
||||
case errors.Is(err, storage.ErrNotFound):
|
||||
s.fail(w, r, NotFound())
|
||||
|
||||
// The role's request rules. The actor's are refused at the edge by the
|
||||
// claim's stricter rules, and the dispatcher guarantees a non-empty id, so
|
||||
// this is defensive — but it names the member when it fires, rather than
|
||||
// a generic refusal.
|
||||
case errors.As(err, &field) && field.Field == issueops.HeartbeatFieldActor:
|
||||
refused()
|
||||
s.fail(w, r, InvalidArgument(claimActorMember, ReasonInvalidValue, field.Detail))
|
||||
|
||||
case !errors.Is(err, storage.ErrValidation):
|
||||
s.failErr(w, r, err)
|
||||
|
||||
default:
|
||||
refused()
|
||||
s.fail(w, r, InvalidArgument("", ReasonInvalidValue,
|
||||
"the request was refused by this workspace's own validation; nothing was written"))
|
||||
}
|
||||
}
|
||||
|
||||
// leaseNotHeldDetail says WHICH state refused a heartbeat with no live claim,
|
||||
// from *issueops.LeaseConflictError's typed fields. Each sentence names one
|
||||
// state only, so a reader of `bd heartbeat` output (and a pattern over it)
|
||||
// can tell a closed issue from a released one and from a wisp.
|
||||
func leaseNotHeldDetail(conflict *issueops.LeaseConflictError) string {
|
||||
// The tail is "on the issue": the one write a refusal can make is deleting
|
||||
// a stale lease row this actor still held (see the 409's description).
|
||||
const tail = "; nothing on the issue was written"
|
||||
switch {
|
||||
case conflict.Ephemeral:
|
||||
return "this issue is an ephemeral wisp, which is never leased" + tail
|
||||
case conflict.Status == "":
|
||||
return "this actor holds no live claim on this issue" + tail
|
||||
case conflict.Status == types.StatusInProgress:
|
||||
return "this actor holds no live claim on this issue: it is in_progress but not claimed by this actor" + tail
|
||||
case conflict.Status == types.StatusOpen:
|
||||
return "this actor holds no live claim on this issue: it is open and not claimed (released, reclaimed or never claimed)" +
|
||||
tail + ". Claim it again rather than heartbeating"
|
||||
default:
|
||||
return "this actor holds no live claim on this issue: it is " + string(conflict.Status) + tail
|
||||
}
|
||||
}
|
||||
283
internal/httpapi/heartbeat_test.go
Normal file
283
internal/httpapi/heartbeat_test.go
Normal file
|
|
@ -0,0 +1,283 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// These cover the transport half of POST /v0/beads/issues/{id}:heartbeat: the
|
||||
// body shape, the refusals the handler raises itself, how the role's typed
|
||||
// refusals reach the wire, and the projection of the role's result. Everything
|
||||
// below the wire (the lease-only write, the in_progress fence, the re-arm) is
|
||||
// issueops.Heartbeater's, pinned by backend/conformance/heartbeater_contract.go
|
||||
// on every leg, the served one included.
|
||||
|
||||
const heartbeatPath = "/v0/beads/issues/bd-1:heartbeat"
|
||||
|
||||
func newHeartbeatServer(t *testing.T, heartbeater *roleHeartbeater) *testServer {
|
||||
t.Helper()
|
||||
return newTestServer(t, rolesConfig(Config{Heartbeater: heartbeater}))
|
||||
}
|
||||
|
||||
// TestHeartbeatMapsTheRequestAndProjectsTheLease pins both halves of a
|
||||
// success: the path id and the trimmed actor reach the role, and the role's
|
||||
// result reaches the wire member for member — with NO `revision` and no issue
|
||||
// body, because the response reports the lease, not the issue: even after a
|
||||
// re-arm, which reminted the revision, `rearmed` is how the caller learns to
|
||||
// re-read the issue.
|
||||
func TestHeartbeatMapsTheRequestAndProjectsTheLease(t *testing.T) {
|
||||
at := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
|
||||
heartbeater := &roleHeartbeater{result: issueops.HeartbeatResult{
|
||||
IssueID: "bd-1",
|
||||
Holder: "worker",
|
||||
HeartbeatAt: at,
|
||||
LeaseExpiresAt: at.Add(5 * time.Minute),
|
||||
Rearmed: true,
|
||||
}}
|
||||
ts := newHeartbeatServer(t, heartbeater)
|
||||
|
||||
resp := ts.claim(t, heartbeatPath, `{"actor":" worker "}`)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp))
|
||||
}
|
||||
calls := heartbeater.heartbeatRequests()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("role calls = %d, want 1", len(calls))
|
||||
}
|
||||
if want := (issueops.HeartbeatRequest{Actor: "worker", IssueID: "bd-1"}); calls[0] != want {
|
||||
t.Fatalf("role request = %+v, want %+v", calls[0], want)
|
||||
}
|
||||
|
||||
body := decodeBody(t, resp)
|
||||
want := map[string]any{
|
||||
"id": "bd-1",
|
||||
"holder": "worker",
|
||||
"heartbeat_at": "2026-10-16T12:00:00Z",
|
||||
"lease_expires_at": "2026-10-16T12:05:00Z",
|
||||
"rearmed": true,
|
||||
}
|
||||
if len(body) != len(want) {
|
||||
t.Errorf("response members = %v, want exactly %v", body, want)
|
||||
}
|
||||
for member, value := range want {
|
||||
if body[member] != value {
|
||||
t.Errorf("%s = %v, want %v", member, body[member], value)
|
||||
}
|
||||
}
|
||||
for _, absent := range []string{"revision", "issue", "changed"} {
|
||||
if _, present := body[absent]; present {
|
||||
t.Errorf("response carries %q; a heartbeat answers the lease, not the issue", absent)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatRefusesABadRequestBeforeTheRole pins the edge's own refusals:
|
||||
// each is a 400 naming the offender, and none reaches the role.
|
||||
func TestHeartbeatRefusesABadRequestBeforeTheRole(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
path string
|
||||
body string
|
||||
param string
|
||||
}{
|
||||
{"a missing actor", heartbeatPath, `{}`, "actor"},
|
||||
{"a blank actor", heartbeatPath, `{"actor":" "}`, "actor"},
|
||||
{"a null actor", heartbeatPath, `{"actor":null}`, "actor"},
|
||||
{"a non-string actor", heartbeatPath, `{"actor":7}`, "actor"},
|
||||
{"an actor with a newline", heartbeatPath, `{"actor":"a\nb"}`, "actor"},
|
||||
{"an actor one character over the column", heartbeatPath, `{"actor":"` + strings.Repeat("a", types.MaxFieldLen+1) + `"}`, "actor"},
|
||||
{"an unknown member", heartbeatPath, `{"actor":"worker","ttl_seconds":60}`, "ttl_seconds"},
|
||||
{"an expected_version member", heartbeatPath, `{"actor":"worker","expected_version":"3"}`, "expected_version"},
|
||||
{"a query parameter", heartbeatPath + "?force=true", `{"actor":"worker"}`, "force"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
heartbeater := &roleHeartbeater{}
|
||||
ts := newHeartbeatServer(t, heartbeater)
|
||||
resp := ts.claim(t, tc.path, tc.body)
|
||||
if resp.StatusCode != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp))
|
||||
}
|
||||
body := decodeBody(t, resp)
|
||||
if body["code"] != string(CodeInvalidArgument) || body["param"] != tc.param {
|
||||
t.Fatalf("problem = %v, want invalid_argument naming %q", body, tc.param)
|
||||
}
|
||||
if calls := heartbeater.heartbeatRequests(); len(calls) != 0 {
|
||||
t.Fatalf("the role was called %d times for a request the edge refuses", len(calls))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatAnswersTheRefusalsTheRoleRaises walks the role's typed refusals.
|
||||
//
|
||||
// The 409 arms are matched BEFORE the generic ones, and two rows here exist to
|
||||
// pin that: ErrLeaseNotHeld also matches ErrNotClaimable, which the generic
|
||||
// classifier would answer `not_claimable` — the code this operation must not
|
||||
// answer — and a foreign holder must arrive in the claim's own conflict shape,
|
||||
// naming the holder.
|
||||
func TestHeartbeatAnswersTheRefusalsTheRoleRaises(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
err error
|
||||
status int
|
||||
code Code
|
||||
param string
|
||||
members map[string]any
|
||||
absent []string
|
||||
// detail must appear in a lease_not_held detail, and none of
|
||||
// notDetail may: each refusal names its own state only.
|
||||
detail string
|
||||
notDetail []string
|
||||
}{
|
||||
{
|
||||
name: "a foreign holder",
|
||||
err: &issueops.LeaseConflictError{
|
||||
IssueID: "bd-1", Holder: "alice", Status: types.StatusInProgress,
|
||||
Err: fmt.Errorf("%w: bd-1 is held by alice", issueops.ErrAlreadyClaimed),
|
||||
},
|
||||
status: http.StatusConflict,
|
||||
code: CodeAlreadyClaimed,
|
||||
members: map[string]any{"assignee": "alice", "issue_status": "in_progress"},
|
||||
},
|
||||
{
|
||||
name: "no live claim (closed)",
|
||||
err: &issueops.LeaseConflictError{
|
||||
IssueID: "bd-1", Status: types.StatusClosed, Err: issueops.ErrLeaseNotHeld,
|
||||
},
|
||||
status: http.StatusConflict,
|
||||
code: CodeLeaseNotHeld,
|
||||
members: map[string]any{"issue_status": "closed"},
|
||||
absent: []string{"assignee", "reason"},
|
||||
detail: "it is closed",
|
||||
notDetail: []string{"ephemeral", "open"},
|
||||
},
|
||||
{
|
||||
name: "no live claim (released: open, unassigned)",
|
||||
err: &issueops.LeaseConflictError{
|
||||
IssueID: "bd-1", Status: types.StatusOpen, Err: issueops.ErrLeaseNotHeld,
|
||||
},
|
||||
status: http.StatusConflict,
|
||||
code: CodeLeaseNotHeld,
|
||||
members: map[string]any{"issue_status": "open"},
|
||||
absent: []string{"assignee", "reason"},
|
||||
detail: "not claimed",
|
||||
notDetail: []string{"ephemeral", "closed"},
|
||||
},
|
||||
{
|
||||
name: "a wisp",
|
||||
err: &issueops.LeaseConflictError{
|
||||
IssueID: "bd-1", Status: types.StatusInProgress, Ephemeral: true,
|
||||
Err: fmt.Errorf("%w: bd-1 is ephemeral", issueops.ErrLeaseNotHeld),
|
||||
},
|
||||
status: http.StatusConflict,
|
||||
code: CodeLeaseNotHeld,
|
||||
members: map[string]any{"issue_status": "in_progress", "reason": string(ReasonEphemeral)},
|
||||
absent: []string{"assignee"},
|
||||
detail: "ephemeral wisp",
|
||||
notDetail: []string{"closed", "open"},
|
||||
},
|
||||
{
|
||||
name: "an absent id",
|
||||
err: fmt.Errorf("%w: bd-1", issueops.ErrNotFound),
|
||||
status: http.StatusNotFound,
|
||||
code: CodeNotFound,
|
||||
},
|
||||
{
|
||||
name: "the role's own actor rule",
|
||||
err: &issueops.HeartbeatFieldError{Field: issueops.HeartbeatFieldActor, Detail: "validation failed: actor is too long"},
|
||||
status: http.StatusBadRequest,
|
||||
code: CodeInvalidArgument,
|
||||
param: "actor",
|
||||
},
|
||||
{
|
||||
name: "some other validation refusal",
|
||||
err: fmt.Errorf("%w: refused", issueops.ErrValidation),
|
||||
status: http.StatusBadRequest,
|
||||
code: CodeInvalidArgument,
|
||||
},
|
||||
{
|
||||
name: "a storage fault",
|
||||
err: errors.New("dial tcp 10.0.0.1:3306: connection refused"),
|
||||
status: http.StatusInternalServerError,
|
||||
code: CodeInternal,
|
||||
},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
ts := newHeartbeatServer(t, &roleHeartbeater{err: tc.err})
|
||||
resp := ts.claim(t, heartbeatPath, `{"actor":"worker"}`)
|
||||
if resp.StatusCode != tc.status {
|
||||
t.Fatalf("status = %d, want %d: %s", resp.StatusCode, tc.status, readAll(t, resp))
|
||||
}
|
||||
body := decodeBody(t, resp)
|
||||
if body["code"] != string(tc.code) {
|
||||
t.Fatalf("code = %v, want %s (%v)", body["code"], tc.code, body)
|
||||
}
|
||||
if tc.param != "" && body["param"] != tc.param {
|
||||
t.Errorf("param = %v, want %q", body["param"], tc.param)
|
||||
}
|
||||
for member, value := range tc.members {
|
||||
if body[member] != value {
|
||||
t.Errorf("%s = %v, want %v", member, body[member], value)
|
||||
}
|
||||
}
|
||||
for _, member := range tc.absent {
|
||||
if _, present := body[member]; present {
|
||||
t.Errorf("%s carries %q; it is documented only beside already_claimed", tc.code, member)
|
||||
}
|
||||
}
|
||||
detail, _ := body["detail"].(string)
|
||||
if strings.Contains(detail, "10.0.0.1") {
|
||||
t.Errorf("detail leaks the storage error: %q", detail)
|
||||
}
|
||||
// The lease_not_held rows share the code, so the detail names the
|
||||
// one state the role observed, and nothing else.
|
||||
if tc.detail != "" && !strings.Contains(detail, tc.detail) {
|
||||
t.Errorf("detail = %q, want it to say %q", detail, tc.detail)
|
||||
}
|
||||
for _, not := range tc.notDetail {
|
||||
if strings.Contains(detail, not) {
|
||||
t.Errorf("detail = %q names %q; it must name only the state the role observed", detail, not)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatDocumentsItsOwnConflictVocabulary pins the operation's code
|
||||
// row from the direction that would make it wrong: `lease_not_held` is minted
|
||||
// for the unheld case, so the claim's `not_claimable` must not be documented
|
||||
// here, and the fence is the holder, so there is no precondition_failed.
|
||||
func TestHeartbeatDocumentsItsOwnConflictVocabulary(t *testing.T) {
|
||||
codes := operationCodes[OpHeartbeatIssue]
|
||||
for _, want := range []Code{CodeAlreadyClaimed, CodeLeaseNotHeld, CodeNotFound, CodeInvalidArgument} {
|
||||
if !containsCode(codes, want) {
|
||||
t.Errorf("heartbeatIssue does not document %s", want)
|
||||
}
|
||||
}
|
||||
for _, refused := range []Code{CodeNotClaimable, CodePreconditionFailed} {
|
||||
if containsCode(codes, refused) {
|
||||
t.Errorf("heartbeatIssue documents %s; see CodeLeaseNotHeld", refused)
|
||||
}
|
||||
}
|
||||
if got := CodeLeaseNotHeld.Status(); got != http.StatusConflict {
|
||||
t.Errorf("lease_not_held status = %d, want 409", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatIsAdvertisedAsItsOwnCapability pins the token a client
|
||||
// preflights on.
|
||||
func TestHeartbeatIsAdvertisedAsItsOwnCapability(t *testing.T) {
|
||||
for _, token := range Capabilities() {
|
||||
if token == "issues.heartbeat" {
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatalf("Capabilities() = %v, want it to advertise issues.heartbeat", Capabilities())
|
||||
}
|
||||
|
|
@ -113,6 +113,18 @@ const (
|
|||
// distinction before then reads the row, which the operation description
|
||||
// tells it to do for a safety reason rather than a taste one.
|
||||
CodeNotReleasable Code = "not_releasable"
|
||||
// CodeLeaseNotHeld is a heartbeat refused because the actor holds no live
|
||||
// claim on the row: it is open, closed, released, reclaimed, moved off
|
||||
// in_progress, or a wisp (wisps are never leased). The `issue_status`
|
||||
// extension carries the status the refusing transaction observed, from
|
||||
// *issueops.LeaseConflictError's typed field.
|
||||
//
|
||||
// It is MINTED rather than folded into CodeNotClaimable for
|
||||
// CodeNotReleasable's reason: an open, unassigned row is the most claimable
|
||||
// row a workspace has, and answering `not_claimable` about it would send a
|
||||
// worker somewhere there is nothing to find. A FOREIGN holder is not this
|
||||
// code: that is the claim's own `already_claimed`, naming the holder.
|
||||
CodeLeaseNotHeld Code = "lease_not_held"
|
||||
// CodeNotClosable is close policy refusing an unforced close: open
|
||||
// children, or a live blocker. The open-children refusal carries the count
|
||||
// in the `open_children` extension member, read inside the refusing
|
||||
|
|
@ -293,6 +305,7 @@ var codeStatus = map[Code]int{
|
|||
CodeIssuePinned: http.StatusConflict,
|
||||
CodeNotAssignee: http.StatusConflict,
|
||||
CodeNotReleasable: http.StatusConflict,
|
||||
CodeLeaseNotHeld: http.StatusConflict,
|
||||
CodeDependencyCycle: http.StatusConflict,
|
||||
CodeDependencyExists: http.StatusConflict,
|
||||
CodeAlreadyExists: http.StatusConflict,
|
||||
|
|
@ -345,6 +358,12 @@ const (
|
|||
// client built for a revision it no longer supports, never to retry the same
|
||||
// request.
|
||||
ReasonWireRevisionUnsupported Reason = "wire_revision_unsupported"
|
||||
// ReasonEphemeral qualifies a 409 CodeLeaseNotHeld: the id names a wisp,
|
||||
// which is never leased, so no claim on it can ever be heartbeaten. It is
|
||||
// the typed form of issueops.LeaseConflictError.Ephemeral, so a client
|
||||
// tells a wisp from an unclaimed issue without parsing `detail` (a wisp's
|
||||
// `issue_status` can be in_progress).
|
||||
ReasonEphemeral Reason = "ephemeral"
|
||||
)
|
||||
|
||||
// staticDetail is the set of codes whose `detail` is FIXED, whatever the
|
||||
|
|
@ -505,6 +524,13 @@ const (
|
|||
// row, and this one sweeps MANY leases on a clock's say-so and reports an id
|
||||
// that was not a stale lease by leaving it out of `reclaimed`.
|
||||
OpReclaimIssues = "reclaimIssues"
|
||||
// OpHeartbeatIssue refreshes the lease on one issue the actor holds, behind
|
||||
// issueops.Heartbeater: what `bd heartbeat` spells. It is NOT claimIssue
|
||||
// repeated: a re-claim is idempotent for the holder WITHOUT extending the
|
||||
// lease, and this extends the lease. A refresh writes nothing else — no
|
||||
// revision, no version, no updated_at, no event; a re-arm (`rearmed`) is
|
||||
// recorded as a mutation and writes each of them.
|
||||
OpHeartbeatIssue = "heartbeatIssue"
|
||||
// OpListRelatedIssues reads ONE issue's neighbors in a named direction,
|
||||
// behind issueops.Relations. It is NOT listDependencies narrowed to one
|
||||
// anchor: that operation answers the stored edge ROWS with their targets
|
||||
|
|
@ -764,6 +790,17 @@ var operationCodes = map[string][]Code{
|
|||
// named a version of. Its only 400s are the role's own refusals and the
|
||||
// handler's shape checks.
|
||||
OpReclaimIssues: {CodeInvalidArgument, CodeUnauthenticated, CodeBusy, CodeDBUnavailable, CodeInternal},
|
||||
// Two 409s, and they answer different questions. `already_claimed` is a
|
||||
// live FOREIGN holder, in the claim's own shape: `assignee` names the
|
||||
// holder and `issue_status` the status, both from the refusing
|
||||
// transaction's *issueops.LeaseConflictError. `lease_not_held` is the
|
||||
// actor holding no live claim at all, with `issue_status`. No
|
||||
// precondition_failed: the fence is the holder, not a revision.
|
||||
OpHeartbeatIssue: {
|
||||
CodeInvalidArgument, CodeUnauthenticated, CodeNotFound,
|
||||
CodeAlreadyClaimed, CodeLeaseNotHeld,
|
||||
CodeBusy, CodeDBUnavailable, CodeInternal,
|
||||
},
|
||||
// The same vocabulary as the stored-edge read beside it, and no not_found
|
||||
// for a stronger version of the same reason: this operation probes no id's
|
||||
// existence at all, so there is nothing it could 404 on.
|
||||
|
|
@ -1148,6 +1185,15 @@ func (r Result) WithIssueStatus(status string) Result {
|
|||
return r
|
||||
}
|
||||
|
||||
// WithReason attaches the `reason` member to a refusal other than the 400
|
||||
// (InvalidArgument sets its own). Same rule as WithAssignee: it comes from the
|
||||
// typed error's own field, never from parsing the refusal's prose.
|
||||
func (r Result) WithReason(reason Reason) Result {
|
||||
v := string(reason)
|
||||
r.Problem.Reason = &v
|
||||
return r
|
||||
}
|
||||
|
||||
// WithOpenChildren attaches the `open_children` extension member (how many open
|
||||
// children the transaction that refused a close observed). Same rule as
|
||||
// WithAssignee: it comes from the typed error's own field, read inside that
|
||||
|
|
|
|||
|
|
@ -190,6 +190,34 @@ func (l *roleLeaseReclaimer) reclaimRequests() []issueops.ReclaimRequest {
|
|||
return append([]issueops.ReclaimRequest(nil), l.calls...)
|
||||
}
|
||||
|
||||
// roleHeartbeater is the lease refresh of the store-shaped source, its own fake
|
||||
// beside roleClaimer for roleBatchGetter's reason: a separate role on a
|
||||
// separate accessor, so one fake answering both would let a test pass on a
|
||||
// server that had wired the heartbeat handler to the claim.
|
||||
type roleHeartbeater struct {
|
||||
result issueops.HeartbeatResult
|
||||
err error
|
||||
|
||||
mu sync.Mutex
|
||||
calls []issueops.HeartbeatRequest
|
||||
}
|
||||
|
||||
func (h *roleHeartbeater) Heartbeat(_ context.Context, req issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
h.mu.Lock()
|
||||
h.calls = append(h.calls, req)
|
||||
h.mu.Unlock()
|
||||
if h.err != nil {
|
||||
return issueops.HeartbeatResult{}, h.err
|
||||
}
|
||||
return h.result, nil
|
||||
}
|
||||
|
||||
func (h *roleHeartbeater) heartbeatRequests() []issueops.HeartbeatRequest {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
return append([]issueops.HeartbeatRequest(nil), h.calls...)
|
||||
}
|
||||
|
||||
// roleRelations is the single-anchor NEIGHBOR role of the store-shaped source,
|
||||
// its own fake beside roleEdgeReader for roleGraphCounter's reason: the two sit
|
||||
// on adjacent accessors and answer about the same edges, so one fake serving
|
||||
|
|
@ -991,6 +1019,9 @@ func rolesConfig(cfg Config) Config {
|
|||
if cfg.LeaseReclaimer == nil {
|
||||
cfg.LeaseReclaimer = &roleLeaseReclaimer{}
|
||||
}
|
||||
if cfg.Heartbeater == nil {
|
||||
cfg.Heartbeater = &roleHeartbeater{}
|
||||
}
|
||||
if cfg.Relations == nil {
|
||||
cfg.Relations = &roleRelations{}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -835,6 +835,18 @@ var routeTable = []route{
|
|||
implemented: true,
|
||||
handler: (*Server).handleRelease,
|
||||
},
|
||||
{
|
||||
op: OpHeartbeatIssue,
|
||||
method: http.MethodPost,
|
||||
// The lease refresh, on the dispatcher the close built. Everything the
|
||||
// claim row says about the wildcard's width holds here unchanged.
|
||||
pattern: customMethodPattern,
|
||||
specPath: "/v0/beads/issues/{id}:heartbeat",
|
||||
customMethod: ":heartbeat",
|
||||
capability: "issues.heartbeat",
|
||||
implemented: true,
|
||||
handler: (*Server).handleHeartbeat,
|
||||
},
|
||||
{
|
||||
op: OpCloseIssue,
|
||||
method: http.MethodPost,
|
||||
|
|
|
|||
|
|
@ -293,6 +293,13 @@ type Config struct {
|
|||
// claim on its holder's say-so, and this reverts MANY on a clock's.
|
||||
// Required on the same terms as every field here.
|
||||
LeaseReclaimer issueops.LeaseReclaimer
|
||||
// Heartbeater is the lease refresh behind
|
||||
// POST /v0/beads/issues/{id}:heartbeat. It is a SEPARATE field from
|
||||
// Claimer for the reason issueops.Heartbeater's own doc gives: a claim
|
||||
// takes ownership and never extends the lease, a heartbeat proves the
|
||||
// holder is alive and writes only the lease row. Required on the same
|
||||
// terms as every field here.
|
||||
Heartbeater issueops.Heartbeater
|
||||
// Relations is the single-anchor neighbor read behind
|
||||
// GET /v0/beads/issues/{id}/related. It is a SEPARATE field from EdgeReader
|
||||
// for the reason issueops.EdgeReader's own doc gives at length: that role
|
||||
|
|
@ -442,6 +449,7 @@ type Server struct {
|
|||
issueEdgeCounter issueops.GraphCounter
|
||||
issueBatchGetter issueops.BatchGetter
|
||||
issueReclaimer issueops.LeaseReclaimer
|
||||
issueHeartbeater issueops.Heartbeater
|
||||
issueRelations issueops.Relations
|
||||
issueCommenter issueops.Commenter
|
||||
issueBlocking issueops.BlockingAnnotator
|
||||
|
|
@ -655,6 +663,7 @@ func Listen(cfg Config) (*Server, error) {
|
|||
issueEdgeCounter: cfg.GraphCounter,
|
||||
issueBatchGetter: cfg.BatchGetter,
|
||||
issueReclaimer: cfg.LeaseReclaimer,
|
||||
issueHeartbeater: cfg.Heartbeater,
|
||||
issueRelations: cfg.Relations,
|
||||
issueCommenter: cfg.Commenter,
|
||||
issueBlocking: cfg.BlockingAnnotator,
|
||||
|
|
@ -777,12 +786,12 @@ func Listen(cfg Config) (*Server, error) {
|
|||
// "all or nothing" would turn an honest condition into a special case inside
|
||||
// three functions. It is checked once, on its own, below.
|
||||
func sourceRoles(cfg Config) []any {
|
||||
return []any{cfg.Reader, cfg.Claimer, cfg.ReadyClaimer, cfg.Releaser, cfg.Lifecycle, cfg.BatchCloser, cfg.Settings, cfg.Stats, cfg.CycleDetector, cfg.EdgeReader, cfg.GraphCounter, cfg.BatchGetter, cfg.LeaseReclaimer, cfg.Relations, cfg.Commenter, cfg.BlockingAnnotator, cfg.TreeWalker, cfg.ReadyCounter, cfg.Counter, cfg.Querier, cfg.Sweeper, cfg.Deleter, cfg.BatchCreator, cfg.DependencyEditor, cfg.BatchApplier, cfg.Memories, cfg.MetadataCAS}
|
||||
return []any{cfg.Reader, cfg.Claimer, cfg.ReadyClaimer, cfg.Releaser, cfg.Lifecycle, cfg.BatchCloser, cfg.Settings, cfg.Stats, cfg.CycleDetector, cfg.EdgeReader, cfg.GraphCounter, cfg.BatchGetter, cfg.LeaseReclaimer, cfg.Heartbeater, cfg.Relations, cfg.Commenter, cfg.BlockingAnnotator, cfg.TreeWalker, cfg.ReadyCounter, cfg.Counter, cfg.Querier, cfg.Sweeper, cfg.Deleter, cfg.BatchCreator, cfg.DependencyEditor, cfg.BatchApplier, cfg.Memories, cfg.MetadataCAS}
|
||||
}
|
||||
|
||||
// roleSourceNames spells sourceRoles for the refusal message, in the same
|
||||
// order, so a caller reading the error learns the whole set it must pass.
|
||||
const roleSourceNames = "Reader, Claimer, ReadyClaimer, Releaser, Lifecycle, BatchCloser, Settings, Stats, CycleDetector, EdgeReader, GraphCounter, BatchGetter, LeaseReclaimer, Relations, Commenter, BlockingAnnotator, TreeWalker, ReadyCounter, Counter, Querier, Sweeper, Deleter, BatchCreator, DependencyEditor, BatchApplier, Memories and MetadataCAS"
|
||||
const roleSourceNames = "Reader, Claimer, ReadyClaimer, Releaser, Lifecycle, BatchCloser, Settings, Stats, CycleDetector, EdgeReader, GraphCounter, BatchGetter, LeaseReclaimer, Heartbeater, Relations, Commenter, BlockingAnnotator, TreeWalker, ReadyCounter, Counter, Querier, Sweeper, Deleter, BatchCreator, DependencyEditor, BatchApplier, Memories and MetadataCAS"
|
||||
|
||||
func anyRoleSet(cfg Config) bool {
|
||||
return slices.ContainsFunc(sourceRoles(cfg), func(r any) bool { return r != nil })
|
||||
|
|
@ -1184,6 +1193,18 @@ func (s *Server) leaseReclaimer(r *http.Request) (issueops.LeaseReclaimer, error
|
|||
return src.LeaseReclaimer()
|
||||
}
|
||||
|
||||
// heartbeater returns the lease-refresh surface for one request, on the same
|
||||
// terms as every role above and held by INTERFACE so uow.HeartbeaterSource is
|
||||
// load-bearing rather than decorative. HeartbeatResult is a plain value, so it
|
||||
// goes out unwrapped.
|
||||
func (s *Server) heartbeater(r *http.Request) (issueops.Heartbeater, error) {
|
||||
if s.provider == nil {
|
||||
return s.issueHeartbeater, nil
|
||||
}
|
||||
var src uow.HeartbeaterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
||||
return src.Heartbeater()
|
||||
}
|
||||
|
||||
// relations returns the single-anchor neighbor surface for one request, built
|
||||
// the same two ways as edgeReader above and held by INTERFACE so
|
||||
// uow.RelationsSource is load-bearing rather than decorative.
|
||||
|
|
|
|||
|
|
@ -3256,6 +3256,106 @@ paths:
|
|||
'503':
|
||||
$ref: '#/components/responses/Unavailable'
|
||||
|
||||
/v0/beads/issues/{id}:heartbeat:
|
||||
post:
|
||||
operationId: heartbeatIssue
|
||||
summary: Refresh the claim lease on an issue the actor holds
|
||||
description: >-
|
||||
What `bd heartbeat` spells. It pushes the lease forward by the
|
||||
server's lease TTL and stamps `heartbeat_at`, and a refresh writes ONLY
|
||||
the lease. The issue row is untouched: `revision` is NOT reminted (a
|
||||
following `expected_version` guard written against the pre-heartbeat
|
||||
revision still wins), the versioned-history revision does not advance,
|
||||
`updated_at` does not move, and no event or events-journal entry is
|
||||
written (`events.list` / `events.watch` see nothing).
|
||||
|
||||
|
||||
FENCED ON THE HOLDER AND THE LIVE CLAIM: `actor` must hold the claim,
|
||||
and the issue must be `in_progress` under it when the lease is
|
||||
refreshed. A holder that is `in_progress` with no lease row (a claim
|
||||
hand-doled through `updateIssue`) is re-armed and reported
|
||||
`rearmed: true`; a refresh of an existing lease row never is. A re-arm
|
||||
is recorded as a mutation, a refresh is not: the re-arm remints
|
||||
`revision`, moves `updated_at` and records an event (once, so that it
|
||||
collides with a release or close racing it rather than leaving a
|
||||
lease on an issue that is no longer claimed); re-read the issue
|
||||
before a guarded write after `rearmed: true`.
|
||||
`heartbeat_at` and `lease_expires_at` are whole seconds, the values
|
||||
stored. Wisps hold no
|
||||
lease and are refused. Safe to repeat: a repeat only moves the expiry
|
||||
forward. The request carries no TTL, as the claim carries none: lease
|
||||
policy is the server's, and `lease_expires_at` reports the result so a
|
||||
worker can see its margin.
|
||||
|
||||
|
||||
There is no `expected_version` member, deliberately: the fence is the
|
||||
holder, and the revision changes on unrelated edits by the holder
|
||||
itself, so a revision-fenced heartbeat would fail spuriously.
|
||||
|
||||
|
||||
Available when the server advertises the `issues.heartbeat` capability.
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/IssueID'
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/HeartbeatIssueRequest'
|
||||
security:
|
||||
- bearerToken: []
|
||||
responses:
|
||||
'200':
|
||||
description: >-
|
||||
The lease is refreshed. The body reports the lease, not the issue,
|
||||
so there is no row and no `revision` in it. A refresh wrote nothing
|
||||
to the issue row; after `rearmed: true` the issue's `revision` was
|
||||
reminted, so re-read the issue before a guarded write.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/HeartbeatIssueResponse'
|
||||
'400':
|
||||
description: >-
|
||||
Invalid request: an unknown query parameter, an unparseable or
|
||||
oversized body, an unknown body member, or an `actor` refused by the
|
||||
rules `ClaimRequest.actor` states.
|
||||
x-bd-codes: [invalid_argument]
|
||||
content:
|
||||
application/problem+json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/Problem'
|
||||
'401':
|
||||
$ref: '#/components/responses/Unauthenticated'
|
||||
'404':
|
||||
$ref: '#/components/responses/NotFound'
|
||||
'409':
|
||||
description: >-
|
||||
`already_claimed` is a live FOREIGN holder, in the claim's own
|
||||
shape: the `assignee` extension member names the holder and
|
||||
`issue_status` carries the status, both read inside the refusing
|
||||
transaction.
|
||||
|
||||
|
||||
`lease_not_held` is the actor holding no live claim at all: the
|
||||
issue is not `in_progress` under this actor (open, closed,
|
||||
released, reclaimed, or moved to another status), or it is a wisp.
|
||||
It carries `issue_status`, and `reason: ephemeral` for a wisp;
|
||||
`detail` names the one state the refusing transaction found. The
|
||||
recovery is a claim, not another heartbeat. Nothing on the issue
|
||||
is written on either refusal; the one write a refusal can make is
|
||||
deleting a stale lease row this actor still held on an issue it no
|
||||
longer holds a live claim on (never another holder's lease).
|
||||
x-bd-codes: [already_claimed, lease_not_held]
|
||||
content:
|
||||
application/problem+json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/Problem'
|
||||
'500':
|
||||
$ref: '#/components/responses/InternalError'
|
||||
'503':
|
||||
$ref: '#/components/responses/Unavailable'
|
||||
|
||||
/v0/beads/issues/{id}:close:
|
||||
post:
|
||||
operationId: closeIssue
|
||||
|
|
@ -8262,6 +8362,7 @@ components:
|
|||
`issues.related`, `issues.create`, `issues.addComment`,
|
||||
`issues.batchClose`,
|
||||
`issues.claim`, `issues.claimNext`, `issues.release`,
|
||||
`issues.heartbeat`,
|
||||
`issues.close`, `issues.reopen`, `issues.update`,
|
||||
`issues.sweep`, `issues.delete`, `issues.batchGet`,
|
||||
`issues.reclaim`, `issues.batchCreate`,
|
||||
|
|
@ -11186,6 +11287,55 @@ components:
|
|||
nothing — the idempotent re-claim. A claim held by a DIFFERENT
|
||||
actor is a 409, not a 200 with this flag.
|
||||
|
||||
HeartbeatIssueRequest:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [actor]
|
||||
properties:
|
||||
actor:
|
||||
type: string
|
||||
minLength: 1
|
||||
maxLength: 256
|
||||
pattern: '^[^\u0000-\u001F\u007F-\u009F\u2028\u2029]+$'
|
||||
description: >-
|
||||
The lease holder refreshing its lease, and the fence's subject.
|
||||
`ClaimRequest.actor`'s rules exactly: the server trims it, then
|
||||
refuses an empty result, anything longer than 256 BYTES (or 255
|
||||
characters, the column's bound), and any control character
|
||||
including newline.
|
||||
|
||||
HeartbeatIssueResponse:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [id, holder, heartbeat_at, lease_expires_at, rearmed]
|
||||
properties:
|
||||
id:
|
||||
type: string
|
||||
description: The issue whose lease was refreshed.
|
||||
holder:
|
||||
type: string
|
||||
description: The lease holder after the refresh (the request's `actor`).
|
||||
heartbeat_at:
|
||||
type: string
|
||||
format: date-time
|
||||
description: The server clock, in UTC, the refresh stamped on the lease.
|
||||
lease_expires_at:
|
||||
type: string
|
||||
format: date-time
|
||||
description: >-
|
||||
`heartbeat_at` plus the server's lease TTL: the deadline the next
|
||||
heartbeat must beat before a reclaim may revert the claim.
|
||||
rearmed:
|
||||
type: boolean
|
||||
description: >-
|
||||
True when the actor held the claim but no lease row existed for it
|
||||
(a claim hand-doled through `updateIssue`), so the heartbeat granted
|
||||
a fresh lease instead of refreshing one. A re-arm is recorded as a
|
||||
mutation of the issue: its `revision` is reminted, `updated_at`
|
||||
moves and an event is recorded, so re-read the issue before a
|
||||
guarded write. False for every refresh, which writes only the
|
||||
lease.
|
||||
|
||||
ReleaseIssueRequest:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
|
|
@ -12731,6 +12881,7 @@ components:
|
|||
configured with a token file), `not_found` (404),
|
||||
`already_claimed` (409), `not_claimable` (409),
|
||||
`not_closable` (409), `not_releasable` (409),
|
||||
`lease_not_held` (409),
|
||||
`template_read_only` (409), `issue_pinned` (409),
|
||||
`not_assignee` (409),
|
||||
`dependency_cycle` (409), `dependency_exists` (409),
|
||||
|
|
@ -12786,7 +12937,9 @@ components:
|
|||
and the one that carries `min_wire_revision`; see the
|
||||
document-level rule and `ContextResponse.wire_revision`). Either
|
||||
way the recovery is to send something different, never to retry
|
||||
the same request. The set may grow; default-branch on unknown
|
||||
the same request. With `lease_not_held`: `ephemeral` (the id
|
||||
names a wisp, which is never leased; its `issue_status` may be
|
||||
`in_progress`). The set may grow; default-branch on unknown
|
||||
values.
|
||||
assignee:
|
||||
type: string
|
||||
|
|
|
|||
47
internal/httpapi/wireshape/testdata/golden.json
vendored
47
internal/httpapi/wireshape/testdata/golden.json
vendored
|
|
@ -2039,6 +2039,44 @@
|
|||
],
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"schema": "HeartbeatIssueRequest",
|
||||
"member": "actor",
|
||||
"type": "string",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"schema": "HeartbeatIssueResponse",
|
||||
"member": "heartbeat_at",
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"schema": "HeartbeatIssueResponse",
|
||||
"member": "holder",
|
||||
"type": "string",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"schema": "HeartbeatIssueResponse",
|
||||
"member": "id",
|
||||
"type": "string",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"schema": "HeartbeatIssueResponse",
|
||||
"member": "lease_expires_at",
|
||||
"type": "string",
|
||||
"format": "date-time",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"schema": "HeartbeatIssueResponse",
|
||||
"member": "rearmed",
|
||||
"type": "boolean",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"schema": "Issue",
|
||||
"member": "acceptance_criteria",
|
||||
|
|
@ -5751,6 +5789,13 @@
|
|||
"explode": true,
|
||||
"default": "false"
|
||||
},
|
||||
{
|
||||
"schema": "param:heartbeatIssue",
|
||||
"member": "path:id",
|
||||
"type": "string",
|
||||
"required": true,
|
||||
"style": "simple"
|
||||
},
|
||||
{
|
||||
"schema": "param:listBlockingAnnotations",
|
||||
"member": "query:is_blocked",
|
||||
|
|
@ -6404,6 +6449,8 @@
|
|||
"EventRecord.issue": "response",
|
||||
"EventsPage": "response",
|
||||
"Health": "response",
|
||||
"HeartbeatIssueRequest": "request",
|
||||
"HeartbeatIssueResponse": "response",
|
||||
"Issue": "response",
|
||||
"Issue.labels[]": "response",
|
||||
"Issue.metadata": "response",
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ go_library(
|
|||
"dial.go",
|
||||
"dispatch.go",
|
||||
"errors.go",
|
||||
"heartbeater.go",
|
||||
"insecure_credential_guard.go",
|
||||
"journal.go",
|
||||
"leasereclaimer.go",
|
||||
|
|
@ -119,6 +120,7 @@ go_test(
|
|||
"expected_revision_contract_test.go",
|
||||
"gate_fields_test.go",
|
||||
"getissue_miss_test.go",
|
||||
"heartbeater_test.go",
|
||||
"helpers_test.go",
|
||||
"insecure_credential_guard_test.go",
|
||||
"lifecycle_create_test.go",
|
||||
|
|
@ -154,6 +156,7 @@ go_test(
|
|||
"served_graph_aux_test.go",
|
||||
"served_graph_counter_test.go",
|
||||
"served_harness_test.go",
|
||||
"served_heartbeater_test.go",
|
||||
"served_hooks_test.go",
|
||||
"served_issue_operations_test.go",
|
||||
"served_journal_test.go",
|
||||
|
|
|
|||
|
|
@ -393,3 +393,12 @@ func (s *Store) BatchGetter() (issueops.BatchGetter, error) {
|
|||
func (s *Store) LeaseReclaimer() (issueops.LeaseReclaimer, error) {
|
||||
return &httpLeaseReclaimer{store: s}, nil
|
||||
}
|
||||
|
||||
// Heartbeater serves heartbeatIssue (POST /v0/beads/issues/{id}:heartbeat) —
|
||||
// the lease refresh behind `bd heartbeat`. Like
|
||||
// LeaseReclaimer it dials through the store directly rather than through
|
||||
// roleWire: Store.dispatch preflights issues.heartbeat, so an older server is
|
||||
// refused before the dial naming the token. See heartbeater.go.
|
||||
func (s *Store) Heartbeater() (issueops.Heartbeater, error) {
|
||||
return &httpHeartbeater{store: s}, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -153,9 +153,10 @@ var legitimatelyUnsupported = map[string]string{
|
|||
// now, kept unwired for CreateIssue's rule rather than for the absence of an
|
||||
// operation. What has not moved is the LEASE row the release deletes: the
|
||||
// role owns that as part of its post-state, and neither of these front doors
|
||||
// can say anything about it, which is why the heartbeat and the reaper above
|
||||
// keep the original reason.
|
||||
"HeartbeatIssue": "lease: no wire vocabulary",
|
||||
// can say anything about it, which is why the raw reaper keeps the original
|
||||
// reason. The HEARTBEAT has left this list: issues.heartbeat publishes the
|
||||
// lease refresh and both Heartbeater and the legacy
|
||||
// HeartbeatIssue dial it (heartbeater.go).
|
||||
"ReclaimExpiredLeases": "lease: no wire vocabulary",
|
||||
"UnclaimIssue": "write: raw release, force flag and all (role: Releaser). Re-routing it would put a second spelling of the transition beside the role's, with a bare error where the role answers the post-state row and its reminted version",
|
||||
"UnclaimIssueIfAssignee": "write: raw conditional release (role: Releaser, via ReleaseRequest.ExpectedAssignee). Its expectation is a STRING where the role's is a POINTER, so it cannot express the one distinction the role's guard is built on — absent means 'do not check' and empty is a refusal, and this signature spells both as \"\"",
|
||||
|
|
|
|||
|
|
@ -38,10 +38,12 @@ const (
|
|||
//
|
||||
// The bounded 503 retry lives here too: the wire layer bounds Retry-After and
|
||||
// hands it back but never sleeps on it. Whether a 503 is safe to come back to
|
||||
// is the question wire.Request.Replayable answers, and only a read's answer
|
||||
// is yes. (The wire does resend a replayable request once, at once, when its
|
||||
// first attempt stalled before any headers or had its connection reset; that
|
||||
// retry answers a transport-level failure, a 503 is the server's answer.)
|
||||
// is the question wire.Request.Replayable answers: yes for a read, and for the
|
||||
// one write marked Idempotent, the lease heartbeat, whose replay moves the
|
||||
// expiry and applies nothing twice. (The wire does resend a replayable request
|
||||
// once, at once, when its first attempt stalled before any headers or had its
|
||||
// connection reset; that retry answers a transport-level failure, a 503 is the
|
||||
// server's answer.)
|
||||
//
|
||||
// A store with no transport is a build-wiring fault (ErrNoTransport), not the
|
||||
// user's: the workspace and the server may both be fine.
|
||||
|
|
@ -59,8 +61,9 @@ func (s *Store) dispatch(ctx context.Context, req wire.Request, out any) error {
|
|||
}
|
||||
|
||||
err := s.wire.Do(ctx, req, out)
|
||||
// Only a replayable read is retried: a GET, or a read on POST that says so
|
||||
// (the batch get). A 503 can interrupt a non-idempotent write whose first
|
||||
// Only a replayable request is retried: a GET, a read on POST that says so
|
||||
// (the batch get), or the heartbeat, the one write that says so (see
|
||||
// httpHeartbeater). A 503 can interrupt a non-idempotent write whose first
|
||||
// attempt already committed before the server ran out of slots, so
|
||||
// resending a POST/PATCH/DELETE could double-apply it — DELETE included,
|
||||
// since forgetMemory is a mutation the wire happens to spell as one.
|
||||
|
|
|
|||
|
|
@ -72,6 +72,7 @@ func startOracle(t *testing.T) *oracle {
|
|||
GraphCounter: stubGraphCounter{},
|
||||
BatchGetter: stubBatchGetter{},
|
||||
LeaseReclaimer: stubLeaseReclaimer{},
|
||||
Heartbeater: stubHeartbeater{},
|
||||
Relations: stubRelations{},
|
||||
Commenter: stubCommenter{},
|
||||
BlockingAnnotator: stubBlocking{},
|
||||
|
|
@ -272,6 +273,7 @@ type (
|
|||
stubGraphCounter struct{ issueops.GraphCounter }
|
||||
stubBatchGetter struct{ issueops.BatchGetter }
|
||||
stubLeaseReclaimer struct{ issueops.LeaseReclaimer }
|
||||
stubHeartbeater struct{ issueops.Heartbeater }
|
||||
stubRelations struct{ issueops.Relations }
|
||||
stubCommenter struct{ issueops.Commenter }
|
||||
stubBlocking struct{ issueops.BlockingAnnotator }
|
||||
|
|
|
|||
159
internal/httpclient/heartbeater.go
Normal file
159
internal/httpclient/heartbeater.go
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
package httpclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/steveyegge/beads/internal/httpapi/apigen"
|
||||
"github.com/steveyegge/beads/internal/httpclient/wire"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// httpHeartbeater is issueops.Heartbeater over heartbeatIssue (POST
|
||||
// /v0/beads/issues/{id}:heartbeat), the lease refresh behind `bd heartbeat`.
|
||||
//
|
||||
// THE MAPPING IS TOTAL: the id rides in the path and the actor is the body's one
|
||||
// member, so nothing here refuses on shape and the ledger carries no row for this
|
||||
// operation. The answer carries no revision, deliberately: it reports the
|
||||
// lease, not the issue. A refresh writes only the lease, so there is no token
|
||||
// to stitch back; a re-arm remints it, which Rearmed reports so the caller
|
||||
// re-reads the issue before a guarded write.
|
||||
//
|
||||
// THE REQUEST RULES ARE CHECKED BEFORE THE DIAL, in the order
|
||||
// internal/storage/issueops.ValidateHeartbeatRequest applies them and with its
|
||||
// *issueops.HeartbeatFieldError, for leasereclaimer.go's reason: the role's
|
||||
// contract promises the typed field error, and a round trip would spend a write
|
||||
// slot to be told what the request already said. The rules are restated rather
|
||||
// than imported (that package is the storage engine's shared body);
|
||||
// TestHeartbeatValidationMatchesTheSharedValidator pins the two together.
|
||||
//
|
||||
// AN OLDER SERVER IS REFUSED BEFORE THE DIAL. Store.dispatch preflights
|
||||
// issues.heartbeat, so a server that does not advertise it answers
|
||||
// *wire.CapabilityError naming the token (taxonomy case 2), never a fallback
|
||||
// to a re-claim or a metadata stamp: a re-claim by the holder does not extend
|
||||
// the lease, and a stamp would remint the revision the holder's guarded
|
||||
// writes depend on.
|
||||
//
|
||||
// IT IS REPLAYABLE (wire.Request.Idempotent), the one write this client marks
|
||||
// so. A repeat only moves the expiry later and writes nothing else (a repeat
|
||||
// of a re-arm finds the lease row the first one granted and refreshes it), and
|
||||
// a replay that lands after a reclaim fails the holder fence, which is the
|
||||
// answer the worker needs anyway. So a 503 is retried within dispatch's bound
|
||||
// rather than reported to a worker that heartbeats every ~90s and stops on any
|
||||
// nonzero exit.
|
||||
//
|
||||
// THE COST: A REPLAY CAN UNDER-REPORT Rearmed. When a re-arm commits but its
|
||||
// answer is lost (the wire resends after a stalled or reset attempt, dispatch
|
||||
// after a 503), the replay finds the lease row the re-arm granted and answers
|
||||
// a refresh, Rearmed false, though the revision token was reminted. The
|
||||
// backstop is the guarded write itself: one composed on the old revision fails
|
||||
// its revision check once and re-reads (see issueops.HeartbeatResult.Rearmed).
|
||||
type httpHeartbeater struct{ store *Store }
|
||||
|
||||
var _ issueops.Heartbeater = (*httpHeartbeater)(nil)
|
||||
|
||||
// Heartbeat dials POST /v0/beads/issues/{id}:heartbeat.
|
||||
//
|
||||
// THE REFUSALS reach the caller typed as the role lists them: not_found is
|
||||
// issueops.ErrNotFound, lease_not_held is *issueops.LeaseConflictError
|
||||
// wrapping ErrLeaseNotHeld (the wire's code table builds it), and
|
||||
// already_claimed — which the wire's table rebuilds as the CLAIM's conflict,
|
||||
// the code's meaning on every other operation — is re-typed here as the
|
||||
// heartbeat's *LeaseConflictError naming the holder (heartbeatError).
|
||||
func (h *httpHeartbeater) Heartbeat(ctx context.Context, req issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
if err := validateHeartbeatRequest(req); err != nil {
|
||||
return issueops.HeartbeatResult{}, err
|
||||
}
|
||||
path, err := wire.IssueMethodPath(req.IssueID, wire.MethodHeartbeat)
|
||||
if err != nil {
|
||||
return issueops.HeartbeatResult{}, err
|
||||
}
|
||||
|
||||
var out apigen.HeartbeatIssueResponse
|
||||
if err := h.store.dispatch(ctx, wire.Request{
|
||||
Op: wire.OpHeartbeatIssue,
|
||||
Method: http.MethodPost,
|
||||
Path: path,
|
||||
Body: apigen.HeartbeatIssueRequest{Actor: req.Actor},
|
||||
IssueID: req.IssueID,
|
||||
Actor: req.Actor,
|
||||
Idempotent: true,
|
||||
}, &out); err != nil {
|
||||
return issueops.HeartbeatResult{}, heartbeatError(err)
|
||||
}
|
||||
return issueops.HeartbeatResult{
|
||||
IssueID: out.Id,
|
||||
Holder: out.Holder,
|
||||
HeartbeatAt: out.HeartbeatAt.UTC(),
|
||||
LeaseExpiresAt: out.LeaseExpiresAt.UTC(),
|
||||
Rearmed: out.Rearmed,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// heartbeatError re-types the two refusals the shared problem table cannot
|
||||
// spell as the heartbeat's.
|
||||
//
|
||||
// - already_claimed arrives as *issueops.ClaimConflictError (the claim's
|
||||
// shape, from the same code). On this operation it is a foreign holder,
|
||||
// so it becomes *issueops.LeaseConflictError naming that holder and
|
||||
// wrapping ErrAlreadyClaimed, read in the native heartbeat's own words
|
||||
// ("issue already claimed by <holder>").
|
||||
// - invalid_argument on `actor` is the server's actor rule (the claim's,
|
||||
// which also refuses control characters) and becomes the role's
|
||||
// *HeartbeatFieldError on that field.
|
||||
//
|
||||
// Everything else — lease_not_held, not_found, a capability refusal, a
|
||||
// transport failure — is already the role's answer and travels unchanged.
|
||||
func heartbeatError(err error) error {
|
||||
var claim *issueops.ClaimConflictError
|
||||
if errors.As(err, &claim) && errors.Is(err, issueops.ErrAlreadyClaimed) {
|
||||
refusal := issueops.ErrAlreadyClaimed
|
||||
if claim.Assignee != "" {
|
||||
refusal = fmt.Errorf("%w by %s", issueops.ErrAlreadyClaimed, claim.Assignee)
|
||||
}
|
||||
return &issueops.LeaseConflictError{
|
||||
IssueID: claim.IssueID, Holder: claim.Assignee, Status: claim.Status, Err: refusal,
|
||||
}
|
||||
}
|
||||
var problem *wire.ProblemError
|
||||
if errors.As(err, &problem) && problem.Code == "invalid_argument" && problem.Param == "actor" {
|
||||
return &issueops.HeartbeatFieldError{Field: issueops.HeartbeatFieldActor, Detail: problem.Detail}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// validateHeartbeatRequest restates the role's request rules, in its order:
|
||||
// the actor present (blank once trimmed is absent), the actor within its
|
||||
// column AS SENT (it is never trimmed: it is compared with the lease holder
|
||||
// verbatim), then a non-empty id.
|
||||
func validateHeartbeatRequest(req issueops.HeartbeatRequest) error {
|
||||
field := func(name, detail string) error {
|
||||
return &issueops.HeartbeatFieldError{Field: name, Detail: issueops.ErrValidation.Error() + ": " + detail}
|
||||
}
|
||||
if strings.TrimSpace(req.Actor) == "" {
|
||||
return field(issueops.HeartbeatFieldActor, "heartbeat actor is required")
|
||||
}
|
||||
if err := types.CheckFieldLen("actor", req.Actor); err != nil {
|
||||
return field(issueops.HeartbeatFieldActor, err.Error())
|
||||
}
|
||||
if req.IssueID == "" {
|
||||
return field(issueops.HeartbeatFieldIssueID, "heartbeat issue id is required")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// HeartbeatIssue is the legacy lease refresh, served over the role so an
|
||||
// in-process library caller holding a storage.DoltStorage reaches the same
|
||||
// operation `bd heartbeat` does, with the same refusals.
|
||||
func (s *Store) HeartbeatIssue(ctx context.Context, id, actor string) error {
|
||||
role, err := s.Heartbeater()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = role.Heartbeat(ctx, issueops.HeartbeatRequest{Actor: actor, IssueID: id})
|
||||
return err
|
||||
}
|
||||
372
internal/httpclient/heartbeater_test.go
Normal file
372
internal/httpclient/heartbeater_test.go
Normal file
|
|
@ -0,0 +1,372 @@
|
|||
package httpclient
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/httpclient/wire"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
storeops "github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// The unit half of the Heartbeater client leg: the
|
||||
// preflight, the request rules, the problem mapping and the replay, over the
|
||||
// real wire client against a doctored server. The contract itself runs against
|
||||
// a real bd serve in served_heartbeater_test.go.
|
||||
|
||||
// heartbeatStub is a doctored bd serve: a handshake advertising caps, and a
|
||||
// :heartbeat route that answers each POST with the next canned reply.
|
||||
type heartbeatStub struct {
|
||||
mu sync.Mutex
|
||||
posts []string // the raw bodies, in arrival order
|
||||
paths []string
|
||||
replies []func(http.ResponseWriter)
|
||||
}
|
||||
|
||||
func (h *heartbeatStub) store(t *testing.T, caps ...string) *Store {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == wire.PathContext {
|
||||
ctx := v0Context("proj-hb")
|
||||
ctx.Capabilities = caps
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(ctx)
|
||||
return
|
||||
}
|
||||
if r.Method != http.MethodPost || !strings.HasSuffix(r.URL.Path, wire.MethodHeartbeat) {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
h.mu.Lock()
|
||||
h.posts = append(h.posts, string(body))
|
||||
h.paths = append(h.paths, r.URL.Path)
|
||||
var reply func(http.ResponseWriter)
|
||||
if len(h.replies) > 0 {
|
||||
reply, h.replies = h.replies[0], h.replies[1:]
|
||||
}
|
||||
h.mu.Unlock()
|
||||
if reply == nil {
|
||||
t.Errorf("an unexpected heartbeat POST reached the server: %s", body)
|
||||
w.WriteHeader(http.StatusTeapot)
|
||||
return
|
||||
}
|
||||
reply(w)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
base, err := url.Parse(srv.URL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
client, err := wire.New(base, nil, wire.Options{HTTPClient: srv.Client()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return New(Target{BaseURL: base}, client, nil)
|
||||
}
|
||||
|
||||
func (h *heartbeatStub) postCount() int {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
return len(h.posts)
|
||||
}
|
||||
|
||||
func heartbeatOK(id, holder string, rearmed bool) func(http.ResponseWriter) {
|
||||
return func(w http.ResponseWriter) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"id": id, "holder": holder, "rearmed": rearmed,
|
||||
"heartbeat_at": "2026-10-16T01:00:00Z",
|
||||
"lease_expires_at": "2026-10-16T01:05:00Z",
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func heartbeatProblem(status int, body string) func(http.ResponseWriter) {
|
||||
return func(w http.ResponseWriter) {
|
||||
w.Header().Set("Content-Type", "application/problem+json; charset=utf-8")
|
||||
w.WriteHeader(status)
|
||||
_, _ = io.WriteString(w, body)
|
||||
}
|
||||
}
|
||||
|
||||
func heartbeaterOf(t *testing.T, s *Store) issueops.Heartbeater {
|
||||
t.Helper()
|
||||
role, err := s.Heartbeater()
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeater(): %v", err)
|
||||
}
|
||||
return role
|
||||
}
|
||||
|
||||
// TestHeartbeatAgainstAServerWithoutTheTokenRefusesBeforeTheDial is the skew
|
||||
// row for a new client on an old server: no issues.heartbeat in the handshake
|
||||
// means a case-2 refusal NAMING the token, with nothing sent — never a
|
||||
// fallback to a re-claim or a metadata stamp.
|
||||
func TestHeartbeatAgainstAServerWithoutTheTokenRefusesBeforeTheDial(t *testing.T) {
|
||||
stub := &heartbeatStub{}
|
||||
s := stub.store(t, "issues.get", "issues.claim", "issues.release", "issues.reclaim")
|
||||
|
||||
_, err := heartbeaterOf(t, s).Heartbeat(t.Context(), issueops.HeartbeatRequest{Actor: "worker", IssueID: "bd-1"})
|
||||
var capErr *wire.CapabilityError
|
||||
if !errors.As(err, &capErr) {
|
||||
t.Fatalf("Heartbeat on an old server = %v, want *wire.CapabilityError", err)
|
||||
}
|
||||
if capErr.Capability != "issues.heartbeat" {
|
||||
t.Errorf("CapabilityError.Capability = %q, want issues.heartbeat", capErr.Capability)
|
||||
}
|
||||
var unsupported *storage.ErrUnsupported
|
||||
if !errors.As(err, &unsupported) || unsupported.Capability != "issues.heartbeat" {
|
||||
t.Errorf("Heartbeat on an old server = %v, want it to unwrap to *storage.ErrUnsupported{Capability: issues.heartbeat}", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "issues.heartbeat") {
|
||||
t.Errorf("the refusal %q does not name the token", err)
|
||||
}
|
||||
if n := stub.postCount(); n != 0 {
|
||||
t.Errorf("%d heartbeat POSTs reached an old server, want 0", n)
|
||||
}
|
||||
|
||||
// The legacy front door rides the same role, so it refuses the same way.
|
||||
if err := s.HeartbeatIssue(t.Context(), "bd-1", "worker"); !errors.As(err, &capErr) {
|
||||
t.Errorf("HeartbeatIssue on an old server = %v, want *wire.CapabilityError", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatSendsTheActorVerbatimAndProjectsTheAnswer pins the request the
|
||||
// client builds (the id in the path, the actor as the one body member, sent
|
||||
// untrimmed: it is the fence's subject) and the result it reads back.
|
||||
func TestHeartbeatSendsTheActorVerbatimAndProjectsTheAnswer(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){heartbeatOK("bd-7", " worker ", true)}}
|
||||
s := stub.store(t, "issues.heartbeat")
|
||||
req := issueops.HeartbeatRequest{Actor: " worker ", IssueID: "bd-7"}
|
||||
|
||||
got, err := heartbeaterOf(t, s).Heartbeat(t.Context(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeat: %v", err)
|
||||
}
|
||||
if req.Actor != " worker " || req.IssueID != "bd-7" {
|
||||
t.Errorf("the request was mutated: %+v", req)
|
||||
}
|
||||
if stub.paths[0] != wire.PathIssues+"/bd-7"+wire.MethodHeartbeat {
|
||||
t.Errorf("dialed %q, want the issue's :heartbeat", stub.paths[0])
|
||||
}
|
||||
var body map[string]any
|
||||
if err := json.Unmarshal([]byte(stub.posts[0]), &body); err != nil {
|
||||
t.Fatalf("decode the sent body %q: %v", stub.posts[0], err)
|
||||
}
|
||||
if len(body) != 1 || body["actor"] != " worker " {
|
||||
t.Errorf("sent body %s, want exactly {\"actor\":\" worker \"}", stub.posts[0])
|
||||
}
|
||||
want := issueops.HeartbeatResult{
|
||||
IssueID: "bd-7", Holder: " worker ", Rearmed: true,
|
||||
HeartbeatAt: time.Date(2026, 10, 16, 1, 0, 0, 0, time.UTC),
|
||||
LeaseExpiresAt: time.Date(2026, 10, 16, 1, 5, 0, 0, time.UTC),
|
||||
}
|
||||
if got != want {
|
||||
t.Errorf("Heartbeat = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatMapsTheRefusalsOntoTheRolesTypes pins each documented refusal
|
||||
// to the role's typed answer. already_claimed is the sharp one: the shared
|
||||
// problem table rebuilds it as the CLAIM's conflict, and on this operation it
|
||||
// must read as the heartbeat's, naming the holder in the native words.
|
||||
func TestHeartbeatMapsTheRefusalsOntoTheRolesTypes(t *testing.T) {
|
||||
t.Run("already_claimed is a LeaseConflictError naming the holder", func(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){heartbeatProblem(http.StatusConflict,
|
||||
`{"status":409,"code":"already_claimed","title":"conflict","detail":"another actor holds this issue's claim; nothing on the issue was written","assignee":"alice","issue_status":"in_progress"}`)}}
|
||||
_, err := heartbeaterOf(t, stub.store(t, "issues.heartbeat")).Heartbeat(t.Context(),
|
||||
issueops.HeartbeatRequest{Actor: "bob", IssueID: "bd-1"})
|
||||
var conflict *issueops.LeaseConflictError
|
||||
if !errors.As(err, &conflict) {
|
||||
t.Fatalf("Heartbeat = %v (%T), want *issueops.LeaseConflictError", err, err)
|
||||
}
|
||||
if conflict.Holder != "alice" || conflict.IssueID != "bd-1" || conflict.Status != "in_progress" {
|
||||
t.Errorf("LeaseConflictError = %+v, want Holder alice, IssueID bd-1, Status in_progress", conflict)
|
||||
}
|
||||
if !errors.Is(err, issueops.ErrAlreadyClaimed) || errors.Is(err, issueops.ErrLeaseNotHeld) {
|
||||
t.Errorf("Heartbeat = %v: want ErrAlreadyClaimed and not ErrLeaseNotHeld", err)
|
||||
}
|
||||
var claim *issueops.ClaimConflictError
|
||||
if errors.As(err, &claim) {
|
||||
t.Errorf("Heartbeat = %v still carries the claim's ClaimConflictError", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "claimed by alice") {
|
||||
t.Errorf("Heartbeat error %q does not read as the native %q", err, "issue already claimed by alice")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("lease_not_held wraps ErrLeaseNotHeld and ErrNotClaimable", func(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){heartbeatProblem(http.StatusConflict,
|
||||
`{"status":409,"code":"lease_not_held","title":"conflict","detail":"this actor holds no live claim on this issue: it is closed; nothing on the issue was written","issue_status":"closed"}`)}}
|
||||
_, err := heartbeaterOf(t, stub.store(t, "issues.heartbeat")).Heartbeat(t.Context(),
|
||||
issueops.HeartbeatRequest{Actor: "worker", IssueID: "bd-2"})
|
||||
var conflict *issueops.LeaseConflictError
|
||||
if !errors.As(err, &conflict) || conflict.Status != "closed" || conflict.Holder != "" || conflict.Ephemeral {
|
||||
t.Fatalf("Heartbeat = %v, want *issueops.LeaseConflictError{Status: closed, Holder: \"\", Ephemeral: false}", err)
|
||||
}
|
||||
if !errors.Is(err, issueops.ErrLeaseNotHeld) || !errors.Is(err, issueops.ErrNotClaimable) || errors.Is(err, issueops.ErrAlreadyClaimed) {
|
||||
t.Errorf("Heartbeat = %v: want ErrLeaseNotHeld and ErrNotClaimable, not ErrAlreadyClaimed", err)
|
||||
}
|
||||
if !strings.Contains(conflict.Error(), "bd-2 status closed") || strings.Contains(err.Error(), "ephemeral") {
|
||||
t.Errorf("Heartbeat error %q (typed %q): want the native %q and no mention of a wisp", err, conflict, "bd-2 status closed")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "it is closed") {
|
||||
t.Errorf("Heartbeat error %q drops the server's detail", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("lease_not_held with reason ephemeral is a wisp", func(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){heartbeatProblem(http.StatusConflict,
|
||||
`{"status":409,"code":"lease_not_held","title":"conflict","detail":"this issue is an ephemeral wisp, which is never leased; nothing on the issue was written","issue_status":"in_progress","reason":"ephemeral"}`)}}
|
||||
_, err := heartbeaterOf(t, stub.store(t, "issues.heartbeat")).Heartbeat(t.Context(),
|
||||
issueops.HeartbeatRequest{Actor: "worker", IssueID: "bd-w"})
|
||||
var conflict *issueops.LeaseConflictError
|
||||
if !errors.As(err, &conflict) || !conflict.Ephemeral || conflict.Status != "in_progress" {
|
||||
t.Fatalf("Heartbeat = %v, want *issueops.LeaseConflictError{Ephemeral: true, Status: in_progress}", err)
|
||||
}
|
||||
if !errors.Is(err, issueops.ErrLeaseNotHeld) || !strings.Contains(conflict.Error(), "bd-w is ephemeral") {
|
||||
t.Errorf("Heartbeat error %q (typed %q): want ErrLeaseNotHeld in the native %q", err, conflict, "bd-w is ephemeral")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("not_found is ErrNotFound", func(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){heartbeatProblem(http.StatusNotFound,
|
||||
`{"status":404,"code":"not_found","title":"not found"}`)}}
|
||||
_, err := heartbeaterOf(t, stub.store(t, "issues.heartbeat")).Heartbeat(t.Context(),
|
||||
issueops.HeartbeatRequest{Actor: "worker", IssueID: "bd-404"})
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Errorf("Heartbeat = %v, want storage.ErrNotFound", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("an invalid actor is the role's field error", func(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){heartbeatProblem(http.StatusBadRequest,
|
||||
`{"status":400,"code":"invalid_argument","title":"bad","param":"actor","reason":"invalid_value","detail":"actor must not contain control characters"}`)}}
|
||||
_, err := heartbeaterOf(t, stub.store(t, "issues.heartbeat")).Heartbeat(t.Context(),
|
||||
issueops.HeartbeatRequest{Actor: "work\u0007er", IssueID: "bd-3"})
|
||||
var field *issueops.HeartbeatFieldError
|
||||
if !errors.As(err, &field) || field.Field != issueops.HeartbeatFieldActor || !errors.Is(err, issueops.ErrValidation) {
|
||||
t.Errorf("Heartbeat = %v, want *issueops.HeartbeatFieldError{Field: actor} matching ErrValidation", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestHeartbeatReplaysABusyAnswer pins the replayable flag: the one write this
|
||||
// client marks idempotent, because a repeat only moves the expiry later. A 503
|
||||
// is retried, not handed to a worker that stops on any nonzero exit.
|
||||
func TestHeartbeatReplaysABusyAnswer(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){
|
||||
func(w http.ResponseWriter) {
|
||||
w.Header().Set("Retry-After", "0")
|
||||
heartbeatProblem(http.StatusServiceUnavailable, `{"status":503,"code":"busy","title":"busy"}`)(w)
|
||||
},
|
||||
heartbeatOK("bd-9", "worker", false),
|
||||
}}
|
||||
got, err := heartbeaterOf(t, stub.store(t, "issues.heartbeat")).Heartbeat(t.Context(),
|
||||
issueops.HeartbeatRequest{Actor: "worker", IssueID: "bd-9"})
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeat after one busy answer = %v, want the replay's success", err)
|
||||
}
|
||||
if got.IssueID != "bd-9" {
|
||||
t.Errorf("Heartbeat = %+v, want bd-9", got)
|
||||
}
|
||||
if n := stub.postCount(); n != 2 {
|
||||
t.Errorf("%d heartbeat POSTs, want 2 (the busy one and its replay)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatRefusesBeforeTheDial pins the request rules: each is the role's
|
||||
// *HeartbeatFieldError, and none reaches the server.
|
||||
func TestHeartbeatRefusesBeforeTheDial(t *testing.T) {
|
||||
stub := &heartbeatStub{}
|
||||
role := heartbeaterOf(t, stub.store(t, "issues.heartbeat"))
|
||||
for name, tc := range map[string]struct {
|
||||
req issueops.HeartbeatRequest
|
||||
field string
|
||||
}{
|
||||
"blank actor": {issueops.HeartbeatRequest{Actor: " \t", IssueID: "bd-1"}, issueops.HeartbeatFieldActor},
|
||||
"over-long actor": {issueops.HeartbeatRequest{Actor: strings.Repeat("a", 256), IssueID: "bd-1"}, issueops.HeartbeatFieldActor},
|
||||
"empty id": {issueops.HeartbeatRequest{Actor: "worker"}, issueops.HeartbeatFieldIssueID},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
_, err := role.Heartbeat(t.Context(), tc.req)
|
||||
var field *issueops.HeartbeatFieldError
|
||||
if !errors.As(err, &field) || field.Field != tc.field || !errors.Is(err, issueops.ErrValidation) {
|
||||
t.Fatalf("Heartbeat(%s) = %v, want *issueops.HeartbeatFieldError{Field: %s}", name, err, tc.field)
|
||||
}
|
||||
})
|
||||
}
|
||||
if n := stub.postCount(); n != 0 {
|
||||
t.Errorf("%d invalid heartbeats reached the server, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatValidationMatchesTheSharedValidator binds heartbeater.go's
|
||||
// restated rules to internal/storage/issueops.ValidateHeartbeatRequest, for
|
||||
// TestReclaimValidationMatchesTheSharedValidator's reason.
|
||||
func TestHeartbeatValidationMatchesTheSharedValidator(t *testing.T) {
|
||||
for name, req := range map[string]issueops.HeartbeatRequest{
|
||||
"valid": {Actor: "worker", IssueID: "bd-1"},
|
||||
"padded actor is valid": {Actor: " worker ", IssueID: "bd-1"},
|
||||
"blank actor": {Actor: " ", IssueID: "bd-1"},
|
||||
"absent actor": {IssueID: "bd-1"},
|
||||
"at the column": {Actor: strings.Repeat("a", 255), IssueID: "bd-1"},
|
||||
"over the column": {Actor: strings.Repeat("a", 256), IssueID: "bd-1"},
|
||||
"empty id": {Actor: "worker"},
|
||||
"blank actor and id": {},
|
||||
"padded id is an id": {Actor: "worker", IssueID: " bd-1 "},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
mine := validateHeartbeatRequest(req)
|
||||
shared := storeops.ValidateHeartbeatRequest(req)
|
||||
if (mine == nil) != (shared == nil) {
|
||||
t.Fatalf("validateHeartbeatRequest = %v, storeops.ValidateHeartbeatRequest = %v: the client's restatement has drifted", mine, shared)
|
||||
}
|
||||
if mine == nil {
|
||||
return
|
||||
}
|
||||
var mineField, sharedField *issueops.HeartbeatFieldError
|
||||
if !errors.As(mine, &mineField) || !errors.As(shared, &sharedField) {
|
||||
t.Fatalf("validateHeartbeatRequest = %v, shared = %v: want both *issueops.HeartbeatFieldError", mine, shared)
|
||||
}
|
||||
if *mineField != *sharedField {
|
||||
t.Errorf("validateHeartbeatRequest = %+v, shared = %+v", *mineField, *sharedField)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLegacyHeartbeatIssueRidesTheRole pins the raw front door onto the role:
|
||||
// one POST with the same body, and the role's typed refusal back.
|
||||
func TestLegacyHeartbeatIssueRidesTheRole(t *testing.T) {
|
||||
stub := &heartbeatStub{replies: []func(http.ResponseWriter){
|
||||
heartbeatOK("bd-5", "worker", false),
|
||||
heartbeatProblem(http.StatusConflict,
|
||||
`{"status":409,"code":"already_claimed","title":"conflict","assignee":"alice","issue_status":"in_progress"}`),
|
||||
}}
|
||||
s := stub.store(t, "issues.heartbeat")
|
||||
if err := s.HeartbeatIssue(t.Context(), "bd-5", "worker"); err != nil {
|
||||
t.Fatalf("HeartbeatIssue: %v", err)
|
||||
}
|
||||
err := s.HeartbeatIssue(t.Context(), "bd-5", "worker")
|
||||
var conflict *issueops.LeaseConflictError
|
||||
if !errors.As(err, &conflict) || conflict.Holder != "alice" || !errors.Is(err, storage.ErrAlreadyClaimed) {
|
||||
t.Errorf("HeartbeatIssue against a foreign holder = %v, want *issueops.LeaseConflictError{Holder: alice}", err)
|
||||
}
|
||||
if err := s.HeartbeatIssue(t.Context(), "bd-5", ""); !errors.Is(err, issueops.ErrValidation) {
|
||||
t.Errorf("HeartbeatIssue with no actor = %v, want ErrValidation before the dial", err)
|
||||
}
|
||||
if n := stub.postCount(); n != 2 {
|
||||
t.Errorf("%d heartbeat POSTs, want 2", n)
|
||||
}
|
||||
}
|
||||
|
|
@ -303,6 +303,8 @@ func serveConfig(t *testing.T, s storage.DoltStorage) httpapi.Config {
|
|||
fail("BatchGetter")
|
||||
cfg.LeaseReclaimer, err = s.LeaseReclaimer()
|
||||
fail("LeaseReclaimer")
|
||||
cfg.Heartbeater, err = s.Heartbeater()
|
||||
fail("Heartbeater")
|
||||
|
||||
// THE JOURNAL IS NOT AN ACCESSOR, alone among the roles above: it is reached
|
||||
// by TYPE ASSERTION, because it is not on storage.DoltStorage and a backend
|
||||
|
|
|
|||
147
internal/httpclient/served_heartbeater_test.go
Normal file
147
internal/httpclient/served_heartbeater_test.go
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
//go:build cgo
|
||||
|
||||
package httpclient
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/backend/conformance"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
)
|
||||
|
||||
// The Heartbeater contract, run through the http client's accessor against a
|
||||
// real bd serve: POST /v0/beads/issues/{id}:heartbeat.
|
||||
// Nothing is parked: the id rides in the path and the actor is the body's one
|
||||
// member, and every request rule the role lists is raised before the dial
|
||||
// (heartbeater.go).
|
||||
//
|
||||
// Seeding and every precondition (the claim, the close, the release, the
|
||||
// reclaim, the status move) go through the REFERENCE store's own roles, never
|
||||
// the wire, per the harness's binding rule; the post-state is read raw.
|
||||
//
|
||||
// THE HOOK CASE runs the heartbeater off a HookFiringStore wrapped around the
|
||||
// client, the chain cmd/bd composes for a connected workspace: bd serve runs
|
||||
// no hooks, so it is the CLIENT's decorator that must fire none for a
|
||||
// refresh — the native refresh fires none on any route (a re-arm, recorded as
|
||||
// a mutation, fires on_update).
|
||||
func newServedHeartbeaterFixture(t *testing.T, prefix string) conformance.HeartbeaterFixture {
|
||||
t.Helper()
|
||||
env := newServedEnv(t, prefix)
|
||||
ref := env.reference
|
||||
heartbeater, err := env.subject.Heartbeater()
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeater(): %v", err)
|
||||
}
|
||||
claimer, err := ref.IssueClaimer()
|
||||
if err != nil {
|
||||
t.Fatalf("IssueClaimer(): %v", err)
|
||||
}
|
||||
lifecycle, err := ref.IssueLifecycle()
|
||||
if err != nil {
|
||||
t.Fatalf("IssueLifecycle(): %v", err)
|
||||
}
|
||||
releaser, err := ref.Releaser()
|
||||
if err != nil {
|
||||
t.Fatalf("Releaser(): %v", err)
|
||||
}
|
||||
reclaimer, err := ref.LeaseReclaimer()
|
||||
if err != nil {
|
||||
t.Fatalf("LeaseReclaimer(): %v", err)
|
||||
}
|
||||
runner, fired := conformance.NewUpdateHookLog(t)
|
||||
hooked := heartbeater
|
||||
if runner != nil {
|
||||
if hooked, err = storage.NewHookFiringStore(env.subject, runner).Heartbeater(); err != nil {
|
||||
t.Fatalf("hooked Heartbeater(): %v", err)
|
||||
}
|
||||
}
|
||||
return conformance.HeartbeaterFixture{
|
||||
IssuePrefix: env.prefix,
|
||||
Heartbeater: heartbeater,
|
||||
CreateIssue: env.createIssue,
|
||||
CreateWisp: env.createWisp,
|
||||
Claimer: claimer,
|
||||
Lifecycle: lifecycle,
|
||||
Releaser: releaser,
|
||||
LeaseReclaimer: reclaimer,
|
||||
Exec: env.exec,
|
||||
QueryScalar: env.queryScalar,
|
||||
CountHistory: env.countHistory,
|
||||
SetJournalEnabled: ref.SetEventsJournalEnabled,
|
||||
SetVersionedHistoryEnabled: ref.SetVersionedHistoryEnabled,
|
||||
HookedHeartbeater: hooked,
|
||||
FiredUpdates: fired,
|
||||
}
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefreshesTheHoldersLease(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefreshesTheHoldersLease(t, t.Context(), newServedHeartbeaterFixture(t, "hhb00"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterChangesNothingOnTheRow(t *testing.T) {
|
||||
conformance.RunHeartbeaterChangesNothingOnTheRow(t, t.Context(), newServedHeartbeaterFixture(t, "hhb01"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterAGuardedWriteWithThePreHeartbeatRevisionStillWins(t *testing.T) {
|
||||
conformance.RunHeartbeaterAGuardedWriteWithThePreHeartbeatRevisionStillWins(t, t.Context(), newServedHeartbeaterFixture(t, "hhb02"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAForeignHolder(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAForeignHolder(t, t.Context(), newServedHeartbeaterFixture(t, "hhb03"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAfterClose(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterClose(t, t.Context(), newServedHeartbeaterFixture(t, "hhb04"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAfterRelease(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterRelease(t, t.Context(), newServedHeartbeaterFixture(t, "hhb05"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAfterReclaim(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterReclaim(t, t.Context(), newServedHeartbeaterFixture(t, "hhb06"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAfterStatusMovedOffInProgress(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterStatusMovedOffInProgress(t, t.Context(), newServedHeartbeaterFixture(t, "hhb07"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAWisp(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAWisp(t, t.Context(), newServedHeartbeaterFixture(t, "hhb08"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRearmsAHandDoledClaim(t *testing.T) {
|
||||
conformance.RunHeartbeaterRearmsAHandDoledClaim(t, t.Context(), newServedHeartbeaterFixture(t, "hhb09"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterAnswersNotFoundForAnAbsentID(t *testing.T) {
|
||||
conformance.RunHeartbeaterAnswersNotFoundForAnAbsentID(t, t.Context(), newServedHeartbeaterFixture(t, "hhb10"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesABlankOrOverLongActor(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesABlankOrOverLongActor(t, t.Context(), newServedHeartbeaterFixture(t, "hhb11"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterDoesNotMutateTheRequest(t *testing.T) {
|
||||
conformance.RunHeartbeaterDoesNotMutateTheRequest(t, t.Context(), newServedHeartbeaterFixture(t, "hhb12"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterASameSecondHeartbeatIsNotARearm(t *testing.T) {
|
||||
conformance.RunHeartbeaterASameSecondHeartbeatIsNotARearm(t, t.Context(), newServedHeartbeaterFixture(t, "hhb13"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAnOrphanedLeaseOnAReleasedIssue(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAReleasedIssue(t, t.Context(), newServedHeartbeaterFixture(t, "hhb14"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAnOrphanedLeaseOnAClosedIssue(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAClosedIssue(t, t.Context(), newServedHeartbeaterFixture(t, "hhb15"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterRefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds(t, t.Context(), newServedHeartbeaterFixture(t, "hhb16"))
|
||||
}
|
||||
|
||||
func TestServedHeartbeaterCommitsOnlyItsOwnTables(t *testing.T) {
|
||||
conformance.RunHeartbeaterCommitsOnlyItsOwnTables(t, t.Context(), newServedHeartbeaterFixture(t, "hhb17"))
|
||||
}
|
||||
|
|
@ -243,6 +243,9 @@ func serveRoles(store storage.DoltStorage) (httpapi.Config, error) {
|
|||
if cfg.LeaseReclaimer, err = store.LeaseReclaimer(); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
if cfg.Heartbeater, err = store.Heartbeater(); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -402,11 +402,6 @@ func (unsupportedDoltStorage) HasRemote(_ context.Context, _ string) (_ bool, er
|
|||
return
|
||||
}
|
||||
|
||||
func (unsupportedDoltStorage) HeartbeatIssue(_ context.Context, _ string, _ string) (err error) {
|
||||
err = errUnsupported("HeartbeatIssue")
|
||||
return
|
||||
}
|
||||
|
||||
func (unsupportedDoltStorage) History(_ context.Context, _ string) (_ []*storage.HistoryEntry, err error) {
|
||||
err = errUnsupported("History")
|
||||
return
|
||||
|
|
@ -727,4 +722,4 @@ func (unsupportedDoltStorage) VersionReconciler() (_ issueops.VersionReconciler,
|
|||
return
|
||||
}
|
||||
|
||||
// NOTE: partial shell (142 of 203 methods generated; 61 left to this package's hand-written set).
|
||||
// NOTE: partial shell (141 of 204 methods generated; 63 left to this package's hand-written set).
|
||||
|
|
|
|||
|
|
@ -112,10 +112,10 @@ type Options struct {
|
|||
// arrived is never cut off by it. Such a request that hits it, or whose
|
||||
// connection is reset, is retried once on another connection, with no
|
||||
// header timer and only what is left of the overall Timeout, counted from
|
||||
// when the first attempt was sent. A write is never
|
||||
// retried and is bounded by the overall Timeout alone (bd serve answers a
|
||||
// write after it commits). A written request whose headers do not come
|
||||
// back in time fails with *ResponseHeaderTimeoutError.
|
||||
// when the first attempt was sent. A write that is not Request.Idempotent
|
||||
// is never retried and is bounded by the overall Timeout alone (bd serve
|
||||
// answers a write after it commits). A written request whose headers do
|
||||
// not come back in time fails with *ResponseHeaderTimeoutError.
|
||||
ResponseHeaderTimeout time.Duration
|
||||
// OnStallRetry, when set, is told about every such retry. The process-wide
|
||||
// count is StallRetryCount either way.
|
||||
|
|
@ -276,7 +276,9 @@ type Request struct {
|
|||
Actor string
|
||||
// Idempotent marks a read carried on a non-GET method (a POST batch get)
|
||||
// as safe to replay, the way a GET is (see Replayable). Never set it on a
|
||||
// write.
|
||||
// write whose replay could double-apply. The one write that sets it is the
|
||||
// lease heartbeat (heartbeatIssue): a repeat only moves the expiry later,
|
||||
// and a replay after the lease is gone fails the holder fence.
|
||||
Idempotent bool
|
||||
// Timeout, when positive, RAISES the client's per-request ceiling for this
|
||||
// one request — never lowers it, and never bounds a client whose
|
||||
|
|
|
|||
|
|
@ -107,6 +107,10 @@ var opCapability = map[string]string{
|
|||
// with a typed capability error instead of dialing a path the older server
|
||||
// would answer as a claim of an issue called ":reclaim".
|
||||
OpReclaimIssues: "issues.reclaim",
|
||||
// The lease refresh, dialed by httpHeartbeater. A server that predates it
|
||||
// does not advertise the token, so Preflight refuses locally with a typed
|
||||
// capability error rather than dialing a route the older server lacks.
|
||||
OpHeartbeatIssue: "issues.heartbeat",
|
||||
// The advance-a-molecule role (molecule.go) dials it when served and
|
||||
// refuses before dialing when not; it never composes a client-side copy.
|
||||
OpAdvanceMolecule: "issues.advanceMolecule",
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ const (
|
|||
OpForgetMemory = "forgetMemory"
|
||||
OpBatchGetIssues = "batchGetIssues"
|
||||
OpReclaimIssues = "reclaimIssues"
|
||||
OpHeartbeatIssue = "heartbeatIssue"
|
||||
)
|
||||
|
||||
// The paths that carry no caller-supplied segment. A path with one is built by
|
||||
|
|
@ -105,6 +106,7 @@ const (
|
|||
MethodAdvanceMolecule = ":advanceMolecule"
|
||||
MethodRelease = ":release"
|
||||
MethodCASMetadata = ":casMetadata"
|
||||
MethodHeartbeat = ":heartbeat"
|
||||
)
|
||||
|
||||
// The sub-resource collections that hang off the issue-detail path. Unlike a
|
||||
|
|
@ -130,7 +132,7 @@ func IssuePath(id string) (string, error) {
|
|||
// final segment: /v0/beads/issues/{id}:claim and its two siblings.
|
||||
func IssueMethodPath(id, method string) (string, error) {
|
||||
switch method {
|
||||
case MethodClaim, MethodClose, MethodReopen, MethodRelease, MethodCASMetadata, MethodAdvanceMolecule:
|
||||
case MethodClaim, MethodClose, MethodReopen, MethodRelease, MethodCASMetadata, MethodAdvanceMolecule, MethodHeartbeat:
|
||||
default:
|
||||
return "", fmt.Errorf("no custom method %q on the issue resource", method)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -133,6 +133,7 @@ func TestIssueMethodPathBuildsTheSuffixIntoTheFinalSegment(t *testing.T) {
|
|||
{MethodClose, "/v0/beads/issues/ga-1:close"},
|
||||
{MethodReopen, "/v0/beads/issues/ga-1:reopen"},
|
||||
{MethodRelease, "/v0/beads/issues/ga-1:release"},
|
||||
{MethodHeartbeat, "/v0/beads/issues/ga-1:heartbeat"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
got, err := IssueMethodPath("ga-1", tc.method)
|
||||
|
|
|
|||
|
|
@ -299,6 +299,11 @@ const ReasonProjectMismatch = "project_mismatch"
|
|||
// tells this refusal apart from an ordinary malformed argument.
|
||||
const ReasonWireRevisionUnsupported = "wire_revision_unsupported"
|
||||
|
||||
// ReasonEphemeral is the Problem `reason` that qualifies a lease_not_held: the
|
||||
// id names a wisp, which is never leased. Spelled exactly as the server's
|
||||
// httpapi.ReasonEphemeral (held to it by the vocabulary test).
|
||||
const ReasonEphemeral = "ephemeral"
|
||||
|
||||
// ErrWireRevisionUnsupported reports that bd serve refused this client's
|
||||
// declared Bd-Wire-Revision header as older than the revision it now
|
||||
// requires. See WireRevisionUnsupportedError.
|
||||
|
|
@ -633,6 +638,31 @@ var codeSentinel = map[string]func(*ProblemError, target) error{
|
|||
// and the one whose message does not assert a fact the wire never sent.
|
||||
"not_releasable": func(*ProblemError, target) error { return issueops.ErrNotReleasable },
|
||||
|
||||
// lease_not_held is the HEARTBEAT's refusal: the actor holds no live claim
|
||||
// (open, closed, released, reclaimed, moved off in_progress, or a wisp).
|
||||
// httpclient's Heartbeater reaches it (heartbeater.go). The typed error is
|
||||
// rebuilt whole from `issue_status` and `reason: ephemeral` (a wisp), and
|
||||
// it wraps ErrLeaseNotHeld, which also matches ErrNotClaimable, in the
|
||||
// native heartbeat's own words ("<id> status <status>", "<id> is
|
||||
// ephemeral").
|
||||
"lease_not_held": func(e *ProblemError, t target) error {
|
||||
id := refusedIssueID(e, t)
|
||||
status := deref(e.IssueStatus)
|
||||
conflict := &issueops.LeaseConflictError{
|
||||
IssueID: id,
|
||||
Status: issueops.Status(status),
|
||||
Ephemeral: e.Reason == ReasonEphemeral,
|
||||
Err: issueops.ErrLeaseNotHeld,
|
||||
}
|
||||
switch {
|
||||
case conflict.Ephemeral:
|
||||
conflict.Err = fmt.Errorf("%w: %s is ephemeral", issueops.ErrLeaseNotHeld, id)
|
||||
case status != "":
|
||||
conflict.Err = fmt.Errorf("%w: %s status %s", issueops.ErrLeaseNotHeld, id, status)
|
||||
}
|
||||
return conflict
|
||||
},
|
||||
|
||||
// [Added: OSS publishes notes_overwrite_refused (internal/httpapi/problem.go
|
||||
// CodeNotesOverwrite, status 409) for an update whose Patch.Notes would
|
||||
// replace existing notes without an explicit overwrite flag. This code had
|
||||
|
|
|
|||
|
|
@ -110,8 +110,8 @@ func TestAgainstAFirstSliceServerEveryPostBaselineOperationRefusesWithCaseTwoDat
|
|||
t.Fatalf("post-baseline ops on a first-slice token = %v, want exactly [claimIssue]", compat)
|
||||
}
|
||||
// The dimension is pinned so the matrix cannot shrink silently: the first
|
||||
// slice served six operations and the tip surface has forty-four, so
|
||||
// thirty-eight of them are things a first-slice server has never heard of. The
|
||||
// slice served six operations and the tip surface has forty-five, so
|
||||
// thirty-nine of them are things a first-slice server has never heard of. The
|
||||
// wire wave moved this from twenty-seven to thirty — claimNext, release and
|
||||
// count — wave 2 to thirty-two, adding the dependency-edge count and the
|
||||
// related read, the mini-sync to thirty-five: the comment write and the two
|
||||
|
|
@ -119,9 +119,10 @@ func TestAgainstAFirstSliceServerEveryPostBaselineOperationRefusesWithCaseTwoDat
|
|||
// stale-lease sweep to thirty-seven. Each is
|
||||
// an operation a first-slice server never served and this client does not
|
||||
// yet dial, which is exactly the shape the refusal matrix is about. The
|
||||
// molecule advance (S16) makes thirty-eight.
|
||||
if len(refused) != 38 {
|
||||
t.Fatalf("the post-baseline set has %d operations a first-slice server never served, want 38", len(refused))
|
||||
// molecule advance (S16) makes thirty-eight, and the lease refresh
|
||||
// (issues.heartbeat) thirty-nine.
|
||||
if len(refused) != 39 {
|
||||
t.Fatalf("the post-baseline set has %d operations a first-slice server never served, want 39", len(refused))
|
||||
}
|
||||
|
||||
// The compat op forces the handshake yet preflights clean: issues.claim is on
|
||||
|
|
|
|||
|
|
@ -35,9 +35,9 @@ import (
|
|||
// slow body that keeps streaming is governed by DefaultTimeout (and the
|
||||
// caller's context) alone.
|
||||
//
|
||||
// The timer applies to replayable requests (see Request.Replayable). A write is
|
||||
// bounded by the client's overall timeout instead, because bd serve answers a
|
||||
// write only after it commits; see roundTrip.
|
||||
// The timer applies to replayable requests (see Request.Replayable). Any other
|
||||
// write is bounded by the client's overall timeout instead, because bd serve
|
||||
// answers a write only after it commits; see roundTrip.
|
||||
const DefaultResponseHeaderTimeout = 15 * time.Second
|
||||
|
||||
// ErrResponseHeaderTimeout is the sentinel a *ResponseHeaderTimeoutError
|
||||
|
|
|
|||
|
|
@ -69,6 +69,7 @@ func TestOperationIdsMatchTheServerRouteTable(t *testing.T) {
|
|||
{OpWatchEvents, httpapi.OpWatchEvents},
|
||||
{OpBatchGetIssues, httpapi.OpBatchGetIssues},
|
||||
{OpReclaimIssues, httpapi.OpReclaimIssues},
|
||||
{OpHeartbeatIssue, httpapi.OpHeartbeatIssue},
|
||||
{OpAdvanceMolecule, httpapi.OpAdvanceMolecule},
|
||||
}
|
||||
|
||||
|
|
@ -187,6 +188,9 @@ func TestTheProjectIdentityVocabularyMatchesTheServer(t *testing.T) {
|
|||
if ReasonProjectMismatch != string(httpapi.ReasonProjectMismatch) {
|
||||
t.Errorf("ReasonProjectMismatch = %q, server says %q", ReasonProjectMismatch, httpapi.ReasonProjectMismatch)
|
||||
}
|
||||
if ReasonEphemeral != string(httpapi.ReasonEphemeral) {
|
||||
t.Errorf("ReasonEphemeral = %q, server says %q", ReasonEphemeral, httpapi.ReasonEphemeral)
|
||||
}
|
||||
if WireRevisionHeader != httpapi.WireRevisionHeader {
|
||||
t.Errorf("WireRevisionHeader = %q, server says %q", WireRevisionHeader, httpapi.WireRevisionHeader)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@ go_library(
|
|||
"hook_dependency_editor.go",
|
||||
"hook_edgereader.go",
|
||||
"hook_graph_counter.go",
|
||||
"hook_heartbeater.go",
|
||||
"hook_init_verifier.go",
|
||||
"hook_issue_claimer.go",
|
||||
"hook_issue_operations.go",
|
||||
|
|
@ -92,6 +93,7 @@ go_test(
|
|||
"hook_decorator_internal_test.go",
|
||||
"hook_decorator_test.go",
|
||||
"hook_dependency_editor_test.go",
|
||||
"hook_heartbeater_test.go",
|
||||
"hook_issue_claimer_test.go",
|
||||
"hook_issue_operations_test.go",
|
||||
"hook_issue_reader_test.go",
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ go_library(
|
|||
"filters.go",
|
||||
"gateway_credential.go",
|
||||
"graph_counter.go",
|
||||
"heartbeater.go",
|
||||
"history.go",
|
||||
"init_verifier.go",
|
||||
"issue_claimer.go",
|
||||
|
|
@ -237,6 +238,7 @@ go_test(
|
|||
"git_remote_test.go",
|
||||
"gm2g3g5r_leak_repro_test.go",
|
||||
"graph_counter_contract_test.go",
|
||||
"heartbeater_contract_test.go",
|
||||
"history_presence_contract_test.go",
|
||||
"history_test.go",
|
||||
"identity_mismatch_message_test.go",
|
||||
|
|
|
|||
59
internal/storage/dolt/heartbeater.go
Normal file
59
internal/storage/dolt/heartbeater.go
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
package dolt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
storeops "github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// Heartbeater returns the lease-refresh surface for this store.
|
||||
func (s *DoltStore) Heartbeater() (issueops.Heartbeater, error) {
|
||||
if s == nil {
|
||||
return nil, &storage.ErrUnsupported{Op: "Heartbeater", Backend: "nil"}
|
||||
}
|
||||
return &heartbeater{store: s}, nil
|
||||
}
|
||||
|
||||
// heartbeater answers a Heartbeat from one retrying transaction around the
|
||||
// shared body, storeops.ExecuteHeartbeatInTx, through the issue-operation
|
||||
// funnel. A REFRESH stages nothing (the leases table is dolt_ignored), so it
|
||||
// mints no commit and no history (bd-lrgn1); a RE-ARM is recorded as a
|
||||
// mutation and gets the version-control commit LeaseRearmVersionCommit names.
|
||||
// The retry replays a heartbeat that loses Dolt's optimistic merge to a
|
||||
// concurrent reclaim, release or close.
|
||||
type heartbeater struct{ store *DoltStore }
|
||||
|
||||
var _ issueops.Heartbeater = (*heartbeater)(nil)
|
||||
|
||||
func (h *heartbeater) Heartbeat(ctx context.Context, request issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
if err := storeops.ValidateHeartbeatRequest(request); err != nil {
|
||||
return issueops.HeartbeatResult{}, err
|
||||
}
|
||||
var result issueops.HeartbeatResult
|
||||
var refusal error
|
||||
if err := h.store.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) {
|
||||
var err error
|
||||
result, err = storeops.ExecuteHeartbeatInTx(ctx, tx, request)
|
||||
// A lease refusal commits (it may have deleted an orphaned lease
|
||||
// row) and is reported after the commit; see HeartbeatRefusalCommits.
|
||||
refusal = nil
|
||||
if storeops.HeartbeatRefusalCommits(err) {
|
||||
refusal = err
|
||||
return nil, "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
tables, msg := storeops.LeaseRearmVersionCommit(request.IssueID, result.Rearmed)
|
||||
return tables, msg, nil
|
||||
}); err != nil {
|
||||
return issueops.HeartbeatResult{}, err
|
||||
}
|
||||
if refusal != nil {
|
||||
return issueops.HeartbeatResult{}, refusal
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
142
internal/storage/dolt/heartbeater_contract_test.go
Normal file
142
internal/storage/dolt/heartbeater_contract_test.go
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
package dolt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/backend/conformance"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
)
|
||||
|
||||
// TestHeartbeaterContract runs the Heartbeater contract against the
|
||||
// server-backed store, which wraps internal/storage/issueops.ExecuteHeartbeatInTx
|
||||
// in a retrying transaction that makes no Dolt commit for a refresh and one
|
||||
// for a re-arm.
|
||||
func TestHeartbeaterContract(t *testing.T) {
|
||||
fixture, ctx, cleanup := newDoltHeartbeaterFixture(t, "hbc")
|
||||
defer cleanup()
|
||||
t.Run("RefreshesTheHoldersLease", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefreshesTheHoldersLease(t, ctx, fixture)
|
||||
})
|
||||
t.Run("ChangesNothingOnTheRow", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterChangesNothingOnTheRow(t, ctx, fixture)
|
||||
})
|
||||
t.Run("AGuardedWriteWithThePreHeartbeatRevisionStillWins", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterAGuardedWriteWithThePreHeartbeatRevisionStillWins(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAForeignHolder", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAForeignHolder(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterClose", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterClose(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterRelease", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterRelease(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterReclaim", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterReclaim(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterStatusMovedOffInProgress", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterStatusMovedOffInProgress(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAWisp", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAWisp(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RearmsAHandDoledClaim", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRearmsAHandDoledClaim(t, ctx, fixture)
|
||||
})
|
||||
t.Run("AnswersNotFoundForAnAbsentID", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterAnswersNotFoundForAnAbsentID(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesABlankOrOverLongActor", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesABlankOrOverLongActor(t, ctx, fixture)
|
||||
})
|
||||
t.Run("DoesNotMutateTheRequest", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterDoesNotMutateTheRequest(t, ctx, fixture)
|
||||
})
|
||||
t.Run("CommitsOnlyItsOwnTables", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterCommitsOnlyItsOwnTables(t, ctx, fixture)
|
||||
})
|
||||
t.Run("ASameSecondHeartbeatIsNotARearm", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterASameSecondHeartbeatIsNotARearm(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAReleasedIssue", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAReleasedIssue(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAClosedIssue", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAClosedIssue(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds(t, ctx, fixture)
|
||||
})
|
||||
}
|
||||
|
||||
// newDoltHeartbeaterFixture composes the frozen role kit with this backend's
|
||||
// accessors, plus the heartbeater taken off a HookFiringStore so the quiet case
|
||||
// observes the decorator a command actually holds.
|
||||
func newDoltHeartbeaterFixture(t *testing.T, prefix string) (conformance.HeartbeaterFixture, context.Context, func()) {
|
||||
t.Helper()
|
||||
store, storeCleanup := setupTestStore(t)
|
||||
ctx, cancel := testContext(t)
|
||||
stop := func() {
|
||||
cancel()
|
||||
storeCleanup()
|
||||
}
|
||||
fail := func(what string, err error) {
|
||||
stop()
|
||||
t.Fatalf("%s: %v", what, err)
|
||||
}
|
||||
heartbeater, err := store.Heartbeater()
|
||||
if err != nil {
|
||||
fail("Heartbeater()", err)
|
||||
}
|
||||
claimer, err := store.IssueClaimer()
|
||||
if err != nil {
|
||||
fail("IssueClaimer()", err)
|
||||
}
|
||||
lifecycle, err := store.IssueLifecycle()
|
||||
if err != nil {
|
||||
fail("IssueLifecycle()", err)
|
||||
}
|
||||
releaser, err := store.Releaser()
|
||||
if err != nil {
|
||||
fail("Releaser()", err)
|
||||
}
|
||||
reclaimer, err := store.LeaseReclaimer()
|
||||
if err != nil {
|
||||
fail("LeaseReclaimer()", err)
|
||||
}
|
||||
runner, fired := conformance.NewUpdateHookLog(t)
|
||||
var hooked = heartbeater
|
||||
if runner != nil {
|
||||
if hooked, err = storage.NewHookFiringStore(store, runner).Heartbeater(); err != nil {
|
||||
fail("hooked Heartbeater()", err)
|
||||
}
|
||||
}
|
||||
kit := newDoltRoleFixtureKit(store, prefix)
|
||||
return conformance.HeartbeaterFixture{
|
||||
IssuePrefix: kit.IssuePrefix,
|
||||
Heartbeater: heartbeater,
|
||||
CreateIssue: kit.CreateIssue,
|
||||
CreateWisp: kit.CreateWisp,
|
||||
Claimer: claimer,
|
||||
Lifecycle: lifecycle,
|
||||
Releaser: releaser,
|
||||
LeaseReclaimer: reclaimer,
|
||||
QueryScalar: kit.QueryScalar,
|
||||
CountHistory: kit.CountHistory,
|
||||
Exec: func(ctx context.Context, statements []conformance.SQLStatement) error {
|
||||
for _, stmt := range statements {
|
||||
if _, err := store.db.ExecContext(ctx, stmt.Query, stmt.Args...); err != nil {
|
||||
return fmt.Errorf("%s: %w", stmt.Query, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
SetJournalEnabled: store.SetEventsJournalEnabled,
|
||||
SetVersionedHistoryEnabled: store.SetVersionedHistoryEnabled,
|
||||
HookedHeartbeater: hooked,
|
||||
FiredUpdates: fired,
|
||||
}, ctx, stop
|
||||
}
|
||||
|
|
@ -445,20 +445,41 @@ func (s *DoltStore) verifiedReadyClaim(ctx context.Context, actor string, write
|
|||
|
||||
// HeartbeatIssue refreshes the lease on an issue actor holds in_progress,
|
||||
// pushing lease_expires_at forward on its row in the ephemeral leases table
|
||||
// (see issueops.lease). Deliberately NO DOLT_ADD/DOLT_COMMIT: the leases
|
||||
// table is dolt_ignored, so a heartbeat mints no commit and no history — this
|
||||
// is the whole point of bd-lrgn1 (fleet heartbeats were the dominant source
|
||||
// of unbounded reachable history). Wrapped in withRetryTx so a heartbeat that
|
||||
// loses Dolt's optimistic merge to a concurrent reclaim/close on the same
|
||||
// lease row is replayed against a fresh snapshot rather than surfaced.
|
||||
// (see issueops.lease). A REFRESH stages nothing: the leases table is
|
||||
// dolt_ignored, so it mints no Dolt commit and no history — the whole point of
|
||||
// bd-lrgn1 (fleet heartbeats were the dominant source of unbounded reachable
|
||||
// history). A RE-ARM (a live claim with no lease row the caller holds) is
|
||||
// recorded as a mutation and gets the one version-control commit
|
||||
// issueops.LeaseRearmVersionCommit names. Wrapped in withRetryTx so a
|
||||
// heartbeat that loses Dolt's optimistic merge to a concurrent reclaim/close
|
||||
// on the same lease row is replayed against a fresh snapshot rather than
|
||||
// surfaced.
|
||||
func (s *DoltStore) HeartbeatIssue(ctx context.Context, id, actor string) error {
|
||||
if s.isActiveWisp(ctx, id) {
|
||||
// Wisps are ephemeral and never leased; nothing to heartbeat.
|
||||
return fmt.Errorf("%w: %s is ephemeral", storage.ErrNotClaimable, id)
|
||||
}
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
return issueops.HeartbeatIssueInTx(ctx, tx, id, actor)
|
||||
})
|
||||
// A refusal that read the issue row COMMITS (it may have deleted the
|
||||
// caller's orphaned lease row) and is returned after; one whose issue read
|
||||
// failed is returned from the callback, so its transaction rolls back
|
||||
// (issueops.LeaseRenewalRefusalCommits).
|
||||
var refusal error
|
||||
if err := s.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (issueops.ChangedTables, string, error) {
|
||||
renewal, err := issueops.RenewLeaseInTx(ctx, tx, id, actor)
|
||||
refusal = nil
|
||||
if issueops.LeaseRenewalRefusalCommits(err) {
|
||||
refusal = err
|
||||
return nil, "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
tables, msg := issueops.LeaseRearmVersionCommit(id, renewal.Rearmed)
|
||||
return tables, msg, nil
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
return refusal
|
||||
}
|
||||
|
||||
// ReclaimExpiredLeases reverts in_progress issues whose lease expired more than
|
||||
|
|
|
|||
|
|
@ -848,6 +848,106 @@ func TestHeartbeatRevivesExpiredLease(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatIssueCommitsARefusalOnlyWhenItReadTheIssue pins the legacy
|
||||
// door's commit rule for a refusal (issueops.LeaseRenewalRefusalCommits, the
|
||||
// role's HeartbeatRefusalCommits rule). A refusal that read the issue row
|
||||
// commits, so the orphaned lease row it deletes stays deleted. A refusal whose
|
||||
// issue read failed rolls back: a lease row with no issue row takes the fast
|
||||
// path's UPDATE, which extends its expiry, then fails the live-claim check, and
|
||||
// committing that would keep the extension while answering an error. It also
|
||||
// pins the delete's holder fence: the caller's orphan held under another
|
||||
// spelling of its identity is deleted, another identity's orphan survives.
|
||||
func TestHeartbeatIssueCommitsARefusalOnlyWhenItReadTheIssue(t *testing.T) {
|
||||
store, cleanup := setupConcurrentTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
plantLease := func(id, holder string) {
|
||||
t.Helper()
|
||||
granted := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
|
||||
if _, err := store.db.ExecContext(ctx, `
|
||||
INSERT INTO leases (issue_id, holder, granted_at, lease_expires_at, heartbeat_at, granted_node)
|
||||
VALUES (?, ?, ?, ?, ?, '')
|
||||
`, id, holder, granted, granted.Add(time.Minute), granted); err != nil {
|
||||
t.Fatalf("plant a lease row on %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
seedOpenIssue := func(id string) {
|
||||
t.Helper()
|
||||
if err := store.CreateIssue(ctx, &types.Issue{
|
||||
ID: id, Title: "lease " + id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask,
|
||||
}, "seeder"); err != nil {
|
||||
t.Fatalf("seed %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
leaseRows := func(id string) int {
|
||||
t.Helper()
|
||||
var rows int
|
||||
if err := store.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM leases WHERE issue_id = ?`, id).Scan(&rows); err != nil {
|
||||
t.Fatalf("count lease rows on %s: %v", id, err)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
hbCtx := issueops.WithLeaseTTL(ctx, time.Hour)
|
||||
|
||||
// No issue row: the refusal read nothing, so the fast path's extension
|
||||
// must roll back.
|
||||
plantLease("lease-noissue", "alice")
|
||||
var before string
|
||||
if err := store.db.QueryRowContext(ctx,
|
||||
`SELECT CAST(lease_expires_at AS CHAR) FROM leases WHERE issue_id = ?`, "lease-noissue",
|
||||
).Scan(&before); err != nil {
|
||||
t.Fatalf("read planted expiry: %v", err)
|
||||
}
|
||||
if err := store.HeartbeatIssue(hbCtx, "lease-noissue", "alice"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat on a lease row with no issue row err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
var after string
|
||||
if err := store.db.QueryRowContext(ctx,
|
||||
`SELECT CAST(lease_expires_at AS CHAR) FROM leases WHERE issue_id = ?`, "lease-noissue",
|
||||
).Scan(&after); err != nil {
|
||||
t.Fatalf("read expiry after the refusal: %v", err)
|
||||
}
|
||||
if after != before {
|
||||
t.Errorf("refused heartbeat committed the fast path's extension: lease_expires_at %s -> %s, want unchanged", before, after)
|
||||
}
|
||||
|
||||
// An open issue: the refusal read it and deleted the caller's orphaned
|
||||
// row, and that delete must commit.
|
||||
seedOpenIssue("lease-orphan")
|
||||
plantLease("lease-orphan", "alice")
|
||||
if err := store.HeartbeatIssue(hbCtx, "lease-orphan", "alice"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat over an orphaned lease on an open issue err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
if rows := leaseRows("lease-orphan"); rows != 0 {
|
||||
t.Errorf("lease rows on lease-orphan after the refusal = %d, want 0: the refusal's orphan delete must commit", rows)
|
||||
}
|
||||
|
||||
// The caller's orphan held under another spelling of its identity
|
||||
// (ga-v2k49): the delete's holder fence is judged under actorMatches, as
|
||||
// the in-place refresh's is, so the row is the caller's and is deleted.
|
||||
seedOpenIssue("lease-orphan-spelling")
|
||||
plantLease("lease-orphan-spelling", "gastown.mayor")
|
||||
if err := store.HeartbeatIssue(hbCtx, "lease-orphan-spelling", "gastown__mayor"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("cross-spelling heartbeat over an orphaned lease err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
if rows := leaseRows("lease-orphan-spelling"); rows != 0 {
|
||||
t.Errorf("lease rows on lease-orphan-spelling after a cross-spelling refusal = %d, want 0: the orphan is the caller's under another spelling", rows)
|
||||
}
|
||||
|
||||
// Control: a genuinely different identity's orphan is not the caller's,
|
||||
// so the refusal leaves it alone.
|
||||
seedOpenIssue("lease-orphan-other")
|
||||
plantLease("lease-orphan-other", "gastown.dog-3")
|
||||
if err := store.HeartbeatIssue(hbCtx, "lease-orphan-other", "gastown__mayor"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat over another identity's orphaned lease err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
if rows := leaseRows("lease-orphan-other"); rows != 1 {
|
||||
t.Errorf("lease rows on lease-orphan-other after the refusal = %d, want 1: a refusal deletes only the caller's own row", rows)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConcurrentReapersAreBenign pins L5.5: two reapers racing the same stale
|
||||
// set neither error nor double-report a reclaim, and do not disturb live work.
|
||||
func TestConcurrentReapersAreBenign(t *testing.T) {
|
||||
|
|
|
|||
|
|
@ -1287,9 +1287,14 @@ func (r *issueSQLRepositoryImpl) UnclaimIssueIfAssignee(ctx context.Context, id,
|
|||
// (time.Now().UTC()), same TTL resolution (issueops.LeaseTTL), and the same
|
||||
// only-current-owner classification (storage.ErrAlreadyClaimed /
|
||||
// ErrNotClaimable) — so classic `bd reclaim` staleness semantics see proxied
|
||||
// heartbeats identically. Deliberately NO Dolt commit: the leases table is
|
||||
// dolt_ignored (bd-lrgn1), and the cmd layer commits this transaction with
|
||||
// uow.RunTxEphemeral (plain SQL COMMIT, nothing in dolt_log).
|
||||
// heartbeats identically. Deliberately NO Dolt commit here: a refresh writes
|
||||
// only the dolt_ignored leases table (bd-lrgn1). A re-arm is recorded as a
|
||||
// mutation (versioned rows), so a caller must commit this unit of work with
|
||||
// a message, as the Heartbeater role's uow leg does (uow.RunTxScoped). A
|
||||
// refusal is returned as an error, so the caller's unit of work rolls back
|
||||
// and the orphaned-lease delete the refusal made is not kept on this door
|
||||
// (no production caller reaches it; the role's leg commits refusals, and the
|
||||
// live-claim predicate keeps an orphan from ever answering a refresh).
|
||||
func (r *issueSQLRepositoryImpl) HeartbeatIssue(ctx context.Context, id, actor string) error {
|
||||
if issueops.IsActiveWispInTx(ctx, r.runner, id) {
|
||||
return fmt.Errorf("db: IssueSQLRepository.HeartbeatIssue: %w: %s is ephemeral", storage.ErrNotClaimable, id)
|
||||
|
|
@ -1324,6 +1329,13 @@ func (r *issueSQLRepositoryImpl) ReclaimExpiredLeases(ctx context.Context, older
|
|||
return out, nil
|
||||
}
|
||||
|
||||
// RenewLease runs the shared heartbeat body directly on r.runner, Reclaim's
|
||||
// bare-passthrough shape: validation, the wisp refusal and the typed
|
||||
// refusals all happen inside issueops.ExecuteHeartbeatInTx.
|
||||
func (r *issueSQLRepositoryImpl) RenewLease(ctx context.Context, request publicops.HeartbeatRequest) (publicops.HeartbeatResult, error) {
|
||||
return issueops.ExecuteHeartbeatInTx(ctx, r.runner, request)
|
||||
}
|
||||
|
||||
// Reclaim runs the shared sweep body directly on r.runner, matching GetMany's
|
||||
// bare-passthrough shape: validation and error-wrapping both happen inside
|
||||
// issueops.ExecuteReclaimInTx.
|
||||
|
|
|
|||
|
|
@ -122,6 +122,12 @@ type IssueSQLRepository interface {
|
|||
// the same function the two store backends wrap. It is the publicops-typed
|
||||
// counterpart to ReclaimExpiredLeases above, the way GetMany is to GetByIDs.
|
||||
Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error)
|
||||
// RenewLease runs the SHARED heartbeat body (issueops.ExecuteHeartbeatInTx)
|
||||
// on this repository's transaction, which is how the unit-of-work provider
|
||||
// reaches the same function the two store backends wrap. It is the
|
||||
// publicops-typed counterpart to HeartbeatIssue above, the way Reclaim is
|
||||
// to ReclaimExpiredLeases.
|
||||
RenewLease(ctx context.Context, request publicops.HeartbeatRequest) (publicops.HeartbeatResult, error)
|
||||
WakeExpiredDefers(ctx context.Context) (issues, wisps int, err error)
|
||||
}
|
||||
|
||||
|
|
@ -353,6 +359,14 @@ type IssueUseCase interface {
|
|||
// IssueRepository.Reclaim for why it is a separate method from
|
||||
// ReclaimExpiredLeases rather than a replacement for it.
|
||||
Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error)
|
||||
// RenewLease is the shape issueops.Heartbeater publishes; see
|
||||
// IssueSQLRepository.RenewLease for why it is a separate method from
|
||||
// Heartbeat rather than a replacement for it. A refresh writes only the
|
||||
// ephemeral leases table, but a re-arm (HeartbeatResult.Rearmed) writes
|
||||
// versioned rows, so the caller commits with uow.RunTxScoped and the
|
||||
// tables issueops.LeaseRearmVersionCommit names, as the uow Heartbeater
|
||||
// does — never with uow.RunTxEphemeral.
|
||||
RenewLease(ctx context.Context, request publicops.HeartbeatRequest) (publicops.HeartbeatResult, error)
|
||||
WakeExpiredDefers(ctx context.Context) (issues, wisps int, err error)
|
||||
|
||||
CreateIssue(ctx context.Context, params CreateIssueParams, actor string) (CreateIssueResult, error)
|
||||
|
|
@ -1954,10 +1968,13 @@ func (u *issueUseCaseImpl) UnclaimIfAssignee(ctx context.Context, id, actor, exp
|
|||
return nil
|
||||
}
|
||||
|
||||
// Heartbeat refreshes the lease on an issue actor holds in_progress. The
|
||||
// write touches ONLY the ephemeral leases table (bd-lrgn1), so the caller
|
||||
// must run it under uow.RunTxEphemeral's no-Dolt-commit form — a heartbeat
|
||||
// mints no Dolt commit and no history in any mode (bd-aq0ql).
|
||||
// Heartbeat refreshes the lease on an issue actor holds in_progress. A
|
||||
// refresh touches ONLY the ephemeral leases table (bd-lrgn1) and needs no Dolt
|
||||
// commit. A re-arm (a live claim with no lease row the caller holds) is
|
||||
// recorded as a mutation in versioned tables, and this method cannot report
|
||||
// one, so never commit it with uow.RunTxEphemeral: take RenewLease, which
|
||||
// reports the re-arm, and commit as the Heartbeater role does
|
||||
// (uow.RunTxScoped with issueops.LeaseRearmVersionCommit).
|
||||
func (u *issueUseCaseImpl) Heartbeat(ctx context.Context, id, actor string) error {
|
||||
if id == "" {
|
||||
return fmt.Errorf("Heartbeat: id must not be empty")
|
||||
|
|
@ -1997,3 +2014,11 @@ func (u *issueUseCaseImpl) ReclaimExpiredLeases(ctx context.Context, olderThan t
|
|||
func (u *issueUseCaseImpl) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) {
|
||||
return u.issueRepo.Reclaim(ctx, request)
|
||||
}
|
||||
|
||||
// RenewLease passes straight through to the repository with no pre-check and
|
||||
// no error-wrapping, Reclaim's reasoning: the request is fully validated inside
|
||||
// the shared body (issueops.ExecuteHeartbeatInTx), and the role's typed
|
||||
// refusals must reach the caller as the body built them.
|
||||
func (u *issueUseCaseImpl) RenewLease(ctx context.Context, request publicops.HeartbeatRequest) (publicops.HeartbeatResult, error) {
|
||||
return u.issueRepo.RenewLease(ctx, request)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ go_library(
|
|||
"federation.go",
|
||||
"get_issue.go",
|
||||
"graph_counter.go",
|
||||
"heartbeater.go",
|
||||
"init_verifier.go",
|
||||
"issue_claimer.go",
|
||||
"issue_operations.go",
|
||||
|
|
@ -153,6 +154,7 @@ go_test(
|
|||
"git_ref_pin_test.go",
|
||||
"graph_counter_contract_test.go",
|
||||
"graph_walk_plan_test.go",
|
||||
"heartbeater_contract_test.go",
|
||||
"history_presence_contract_test.go",
|
||||
"history_test.go",
|
||||
"is_blocked_batch_test.go",
|
||||
|
|
|
|||
60
internal/storage/embeddeddolt/heartbeater.go
Normal file
60
internal/storage/embeddeddolt/heartbeater.go
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
//go:build cgo
|
||||
|
||||
package embeddeddolt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
storeops "github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// Heartbeater returns the lease-refresh surface for this store.
|
||||
func (s *EmbeddedDoltStore) Heartbeater() (issueops.Heartbeater, error) {
|
||||
if s == nil {
|
||||
return nil, &storage.ErrUnsupported{Op: "Heartbeater", Backend: "nil"}
|
||||
}
|
||||
return &heartbeater{store: s}, nil
|
||||
}
|
||||
|
||||
// heartbeater answers a Heartbeat from one connection's transaction around the
|
||||
// shared body, storeops.ExecuteHeartbeatInTx, the embedded twin of the
|
||||
// server-backed store's heartbeater. A REFRESH commits as SQL only: the
|
||||
// leases table is dolt_ignored, so there is nothing versioned to record and
|
||||
// no version-control entry follows it. A RE-ARM is recorded as a mutation and
|
||||
// gets the version-control entry LeaseRearmVersionCommit names.
|
||||
type heartbeater struct{ store *EmbeddedDoltStore }
|
||||
|
||||
var _ issueops.Heartbeater = (*heartbeater)(nil)
|
||||
|
||||
func (h *heartbeater) Heartbeat(ctx context.Context, request issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
if err := storeops.ValidateHeartbeatRequest(request); err != nil {
|
||||
return issueops.HeartbeatResult{}, err
|
||||
}
|
||||
var result issueops.HeartbeatResult
|
||||
var refusal error
|
||||
if err := h.store.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) {
|
||||
var err error
|
||||
result, err = storeops.ExecuteHeartbeatInTx(ctx, tx, request)
|
||||
// A lease refusal commits (it may have deleted an orphaned lease
|
||||
// row) and is reported after the commit; see HeartbeatRefusalCommits.
|
||||
refusal = nil
|
||||
if storeops.HeartbeatRefusalCommits(err) {
|
||||
refusal = err
|
||||
return nil, "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
tables, msg := storeops.LeaseRearmVersionCommit(request.IssueID, result.Rearmed)
|
||||
return tables, msg, nil
|
||||
}); err != nil {
|
||||
return issueops.HeartbeatResult{}, err
|
||||
}
|
||||
if refusal != nil {
|
||||
return issueops.HeartbeatResult{}, refusal
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
139
internal/storage/embeddeddolt/heartbeater_contract_test.go
Normal file
139
internal/storage/embeddeddolt/heartbeater_contract_test.go
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
//go:build cgo
|
||||
|
||||
package embeddeddolt_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/backend/conformance"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/storage/embeddeddolt"
|
||||
)
|
||||
|
||||
// TestHeartbeaterContract runs the Heartbeater contract against the embedded
|
||||
// store, which wraps internal/storage/issueops.ExecuteHeartbeatInTx in one
|
||||
// connection's transaction and records no version entry after a refresh, one
|
||||
// after a re-arm.
|
||||
func TestHeartbeaterContract(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
te := newTestEnv(t, "hbc")
|
||||
ctx := t.Context()
|
||||
fixture := newEmbeddedHeartbeaterFixture(t, te, "hbc")
|
||||
t.Run("RefreshesTheHoldersLease", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefreshesTheHoldersLease(t, ctx, fixture)
|
||||
})
|
||||
t.Run("ChangesNothingOnTheRow", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterChangesNothingOnTheRow(t, ctx, fixture)
|
||||
})
|
||||
t.Run("AGuardedWriteWithThePreHeartbeatRevisionStillWins", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterAGuardedWriteWithThePreHeartbeatRevisionStillWins(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAForeignHolder", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAForeignHolder(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterClose", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterClose(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterRelease", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterRelease(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterReclaim", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterReclaim(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterStatusMovedOffInProgress", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterStatusMovedOffInProgress(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAWisp", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAWisp(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RearmsAHandDoledClaim", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRearmsAHandDoledClaim(t, ctx, fixture)
|
||||
})
|
||||
t.Run("AnswersNotFoundForAnAbsentID", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterAnswersNotFoundForAnAbsentID(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesABlankOrOverLongActor", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesABlankOrOverLongActor(t, ctx, fixture)
|
||||
})
|
||||
t.Run("DoesNotMutateTheRequest", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterDoesNotMutateTheRequest(t, ctx, fixture)
|
||||
})
|
||||
t.Run("CommitsOnlyItsOwnTables", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterCommitsOnlyItsOwnTables(t, ctx, fixture)
|
||||
})
|
||||
t.Run("ASameSecondHeartbeatIsNotARearm", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterASameSecondHeartbeatIsNotARearm(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAReleasedIssue", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAReleasedIssue(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAClosedIssue", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAClosedIssue(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds(t, ctx, fixture)
|
||||
})
|
||||
}
|
||||
|
||||
func newEmbeddedHeartbeaterFixture(t *testing.T, te *testEnv, prefix string) conformance.HeartbeaterFixture {
|
||||
t.Helper()
|
||||
heartbeater, err := te.store.Heartbeater()
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeater(): %v", err)
|
||||
}
|
||||
claimer, err := te.store.IssueClaimer()
|
||||
if err != nil {
|
||||
t.Fatalf("IssueClaimer(): %v", err)
|
||||
}
|
||||
lifecycle, err := te.store.IssueLifecycle()
|
||||
if err != nil {
|
||||
t.Fatalf("IssueLifecycle(): %v", err)
|
||||
}
|
||||
releaser, err := te.store.Releaser()
|
||||
if err != nil {
|
||||
t.Fatalf("Releaser(): %v", err)
|
||||
}
|
||||
reclaimer, err := te.store.LeaseReclaimer()
|
||||
if err != nil {
|
||||
t.Fatalf("LeaseReclaimer(): %v", err)
|
||||
}
|
||||
runner, fired := conformance.NewUpdateHookLog(t)
|
||||
hooked := heartbeater
|
||||
if runner != nil {
|
||||
if hooked, err = storage.NewHookFiringStore(te.store, runner).Heartbeater(); err != nil {
|
||||
t.Fatalf("hooked Heartbeater(): %v", err)
|
||||
}
|
||||
}
|
||||
kit := newEmbeddedRoleFixtureKit(te, prefix)
|
||||
return conformance.HeartbeaterFixture{
|
||||
IssuePrefix: kit.IssuePrefix,
|
||||
Heartbeater: heartbeater,
|
||||
CreateIssue: kit.CreateIssue,
|
||||
CreateWisp: kit.CreateWisp,
|
||||
Claimer: claimer,
|
||||
Lifecycle: lifecycle,
|
||||
Releaser: releaser,
|
||||
LeaseReclaimer: reclaimer,
|
||||
QueryScalar: kit.QueryScalar,
|
||||
CountHistory: kit.CountHistory,
|
||||
Exec: func(ctx context.Context, statements []conformance.SQLStatement) error {
|
||||
db, cleanup, err := embeddeddolt.OpenSQL(ctx, te.dataDir, te.database, "main")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = cleanup() }()
|
||||
for _, stmt := range statements {
|
||||
if _, err := db.ExecContext(ctx, stmt.Query, stmt.Args...); err != nil {
|
||||
return fmt.Errorf("%s: %w", stmt.Query, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
SetJournalEnabled: te.store.SetEventsJournalEnabled,
|
||||
SetVersionedHistoryEnabled: te.store.SetVersionedHistoryEnabled,
|
||||
HookedHeartbeater: hooked,
|
||||
FiredUpdates: fired,
|
||||
}
|
||||
}
|
||||
|
|
@ -113,15 +113,36 @@ func (s *EmbeddedDoltStore) UpdateIssueChecked(ctx context.Context, id string, u
|
|||
}
|
||||
|
||||
// HeartbeatIssue refreshes the lease on an issue actor holds in_progress.
|
||||
// Delegates SQL work to issueops; EmbeddedDolt auto-commits the transaction.
|
||||
// Delegates SQL work to issueops. A REFRESH writes only the dolt_ignored
|
||||
// leases row, so it stages nothing and mints no Dolt commit; a RE-ARM is
|
||||
// recorded as a mutation and gets the one version-control commit
|
||||
// issueops.LeaseRearmVersionCommit names.
|
||||
func (s *EmbeddedDoltStore) HeartbeatIssue(ctx context.Context, id, actor string) error {
|
||||
return s.withConn(ctx, true, func(tx *sql.Tx) error {
|
||||
// A refusal that read the issue row COMMITS (it may have deleted the
|
||||
// caller's orphaned lease row) and is returned after; one whose issue read
|
||||
// failed is returned from the callback, so its transaction rolls back
|
||||
// (issueops.LeaseRenewalRefusalCommits).
|
||||
var refusal error
|
||||
if err := s.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (issueops.ChangedTables, string, error) {
|
||||
refusal = nil
|
||||
if issueops.IsActiveWispInTx(ctx, tx, id) {
|
||||
// Wisps are ephemeral and never leased; nothing to heartbeat.
|
||||
return fmt.Errorf("%w: %s is ephemeral", storage.ErrNotClaimable, id)
|
||||
return nil, "", fmt.Errorf("%w: %s is ephemeral", storage.ErrNotClaimable, id)
|
||||
}
|
||||
return issueops.HeartbeatIssueInTx(ctx, tx, id, actor)
|
||||
})
|
||||
renewal, err := issueops.RenewLeaseInTx(ctx, tx, id, actor)
|
||||
if issueops.LeaseRenewalRefusalCommits(err) {
|
||||
refusal = err
|
||||
return nil, "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
tables, msg := issueops.LeaseRearmVersionCommit(id, renewal.Rearmed)
|
||||
return tables, msg, nil
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
return refusal
|
||||
}
|
||||
|
||||
// ReclaimExpiredLeases reverts in_progress issues whose lease expired more than
|
||||
|
|
|
|||
|
|
@ -190,8 +190,9 @@ func TestReclaimExpiredLeaseSurvivesRestartEmbedded(t *testing.T) {
|
|||
// claimed under, and still refuses a genuinely different identity. The
|
||||
// primary UPDATE's own holder=? predicate stays verbatim SQL by design (see
|
||||
// lease.go); this test exercises the fallback path that makes the caller's
|
||||
// experience correct anyway, including that it re-arms the lease row under
|
||||
// actor's current spelling so the NEXT heartbeat hits the fast path.
|
||||
// experience correct anyway, including that it refreshes the lease row in
|
||||
// place under actor's current spelling, keeping its grant (granted_at and
|
||||
// granted_node), rather than re-granting it.
|
||||
func TestHeartbeatMatchesActorAcrossSpellingEmbedded(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
|
||||
|
|
@ -211,6 +212,12 @@ func TestHeartbeatMatchesActorAcrossSpellingEmbedded(t *testing.T) {
|
|||
if err := te.store.ClaimIssue(ctx, "lease-spelling-1", "gastown.mayor"); err != nil {
|
||||
t.Fatalf("ClaimIssue: %v", err)
|
||||
}
|
||||
// Give the grant a provenance a re-grant would visibly overwrite (it
|
||||
// stamps granted_at = now and this node, which is unnamed here).
|
||||
te.exec(t, ctx, "UPDATE leases SET granted_at = DATE_SUB(granted_at, INTERVAL 1 HOUR), granted_node = 'replica-b' WHERE issue_id = ?", "lease-spelling-1")
|
||||
const grantQuery = "SELECT CONCAT(CAST(granted_at AS CHAR), '|', COALESCE(granted_node, '')) FROM leases WHERE issue_id = ?"
|
||||
var grantBefore string
|
||||
te.queryScalar(t, ctx, grantQuery, []any{"lease-spelling-1"}, &grantBefore)
|
||||
|
||||
// Same identity, dot-separator form sanitized to double-underscore (the
|
||||
// exact ga-wzl83 repro shape) — must succeed, not ErrAlreadyClaimed.
|
||||
|
|
@ -218,13 +225,18 @@ func TestHeartbeatMatchesActorAcrossSpellingEmbedded(t *testing.T) {
|
|||
t.Fatalf("cross-spelling HeartbeatIssue: %v, want success (same identity, different separator)", err)
|
||||
}
|
||||
|
||||
// The re-arm re-wrote holder under the caller's current spelling, so the
|
||||
// NEXT heartbeat under that same spelling hits the primary UPDATE's fast
|
||||
// path directly (0 rows would mean it fell to the fallback again).
|
||||
var holder string
|
||||
// The in-place refresh re-wrote holder under the caller's current
|
||||
// spelling and kept the grant. The assignee keeps the claim's spelling, so
|
||||
// every later heartbeat under this one still fails the fast path's
|
||||
// verbatim live-claim check and is refreshed in place the same way.
|
||||
var holder, grantAfter string
|
||||
te.queryScalar(t, ctx, "SELECT holder FROM leases WHERE issue_id = ?", []any{"lease-spelling-1"}, &holder)
|
||||
if holder != "gastown__mayor" {
|
||||
t.Errorf("lease holder after cross-spelling heartbeat = %q, want re-armed to %q", holder, "gastown__mayor")
|
||||
t.Errorf("lease holder after cross-spelling heartbeat = %q, want refreshed in place under %q", holder, "gastown__mayor")
|
||||
}
|
||||
te.queryScalar(t, ctx, grantQuery, []any{"lease-spelling-1"}, &grantAfter)
|
||||
if grantAfter != grantBefore {
|
||||
t.Errorf("granted_at|granted_node moved from %s to %s across a cross-spelling heartbeat; a refresh never re-grants", grantBefore, grantAfter)
|
||||
}
|
||||
|
||||
// Control: a genuinely different identity — canonicalActor's own doc
|
||||
|
|
@ -257,3 +269,87 @@ func TestHeartbeatRejectsWispEmbedded(t *testing.T) {
|
|||
t.Fatalf("wisp heartbeat err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatIssueCommitsARefusalOnlyWhenItReadTheIssueEmbedded is the
|
||||
// embedded twin of the server-backed store's test: a refusal whose issue read
|
||||
// failed rolls back the fast path's extension of a lease row with no issue
|
||||
// row, and a refusal that read the issue row commits its orphan delete
|
||||
// (issueops.LeaseRenewalRefusalCommits), which takes the caller's orphan under
|
||||
// any spelling of its identity and never another identity's.
|
||||
func TestHeartbeatIssueCommitsARefusalOnlyWhenItReadTheIssueEmbedded(t *testing.T) {
|
||||
skipUnlessEmbeddedDolt(t)
|
||||
|
||||
te := newTestEnv(t, "lease_refusal")
|
||||
ctx := t.Context()
|
||||
|
||||
plantLease := func(id, holder string) {
|
||||
t.Helper()
|
||||
granted := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
|
||||
te.exec(t, ctx, `
|
||||
INSERT INTO leases (issue_id, holder, granted_at, lease_expires_at, heartbeat_at, granted_node)
|
||||
VALUES (?, ?, ?, ?, ?, '')
|
||||
`, id, holder, granted, granted.Add(time.Minute), granted)
|
||||
}
|
||||
seedOpenIssue := func(id string) {
|
||||
t.Helper()
|
||||
if err := te.store.CreateIssue(ctx, &types.Issue{
|
||||
ID: id, Title: "orphaned lease", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask,
|
||||
}, "seeder"); err != nil {
|
||||
t.Fatalf("CreateIssue %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
leaseRows := func(id string) int {
|
||||
t.Helper()
|
||||
var rows int
|
||||
te.queryScalar(t, ctx, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", []any{id}, &rows)
|
||||
return rows
|
||||
}
|
||||
hbCtx := issueops.WithLeaseTTL(ctx, time.Hour)
|
||||
|
||||
// No issue row: the refusal read nothing, so the fast path's extension
|
||||
// must roll back.
|
||||
plantLease("lease-noissue-1", "alice")
|
||||
var before, after string
|
||||
te.queryScalar(t, ctx, "SELECT CAST(lease_expires_at AS CHAR) FROM leases WHERE issue_id = ?", []any{"lease-noissue-1"}, &before)
|
||||
if err := te.store.HeartbeatIssue(hbCtx, "lease-noissue-1", "alice"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat on a lease row with no issue row err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
te.queryScalar(t, ctx, "SELECT CAST(lease_expires_at AS CHAR) FROM leases WHERE issue_id = ?", []any{"lease-noissue-1"}, &after)
|
||||
if after != before {
|
||||
t.Errorf("refused heartbeat committed the fast path's extension: lease_expires_at %s -> %s, want unchanged", before, after)
|
||||
}
|
||||
|
||||
// An open issue: the refusal read it and deleted the caller's orphaned
|
||||
// row, and that delete must commit.
|
||||
seedOpenIssue("lease-orphan-1")
|
||||
plantLease("lease-orphan-1", "alice")
|
||||
if err := te.store.HeartbeatIssue(hbCtx, "lease-orphan-1", "alice"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat over an orphaned lease on an open issue err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
if rows := leaseRows("lease-orphan-1"); rows != 0 {
|
||||
t.Errorf("lease rows on lease-orphan-1 after the refusal = %d, want 0: the refusal's orphan delete must commit", rows)
|
||||
}
|
||||
|
||||
// The caller's orphan held under another spelling of its identity
|
||||
// (ga-v2k49): the delete's holder fence is judged under actorMatches, as
|
||||
// the in-place refresh's is, so the row is the caller's and is deleted.
|
||||
seedOpenIssue("lease-orphan-spelling-1")
|
||||
plantLease("lease-orphan-spelling-1", "gastown.mayor")
|
||||
if err := te.store.HeartbeatIssue(hbCtx, "lease-orphan-spelling-1", "gastown__mayor"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("cross-spelling heartbeat over an orphaned lease err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
if rows := leaseRows("lease-orphan-spelling-1"); rows != 0 {
|
||||
t.Errorf("lease rows on lease-orphan-spelling-1 after a cross-spelling refusal = %d, want 0: the orphan is the caller's under another spelling", rows)
|
||||
}
|
||||
|
||||
// Control: a genuinely different identity's orphan is not the caller's,
|
||||
// so the refusal leaves it alone.
|
||||
seedOpenIssue("lease-orphan-other-1")
|
||||
plantLease("lease-orphan-other-1", "gastown.dog-3")
|
||||
if err := te.store.HeartbeatIssue(hbCtx, "lease-orphan-other-1", "gastown__mayor"); !errors.Is(err, storage.ErrNotClaimable) {
|
||||
t.Fatalf("heartbeat over another identity's orphaned lease err = %v, want ErrNotClaimable", err)
|
||||
}
|
||||
if rows := leaseRows("lease-orphan-other-1"); rows != 1 {
|
||||
t.Errorf("lease rows on lease-orphan-other-1 after the refusal = %d, want 1: a refusal deletes only the caller's own row", rows)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -144,7 +144,11 @@ func (p *uowProvider) Releaser() (publicops.Releaser, error) { return uow.
|
|||
func (p *uowProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) {
|
||||
return uow.NewLeaseReclaimer(p)
|
||||
}
|
||||
func (p *uowProvider) Memories() (memoryops.Memories, error) { return uow.NewMemories(p) }
|
||||
|
||||
// Heartbeater passes through: the external-dependency policy governs what a
|
||||
// write may point at, and a heartbeat on an existing claim points at nothing.
|
||||
func (p *uowProvider) Heartbeater() (publicops.Heartbeater, error) { return uow.NewHeartbeater(p) }
|
||||
func (p *uowProvider) Memories() (memoryops.Memories, error) { return uow.NewMemories(p) }
|
||||
func (p *uowProvider) EventsJournalCursor() (storage.EventsJournalCursor, error) {
|
||||
return uow.NewEventsJournalCursor(p)
|
||||
}
|
||||
|
|
@ -222,6 +226,7 @@ var (
|
|||
_ uow.MetadataCASSource = (*uowProvider)(nil)
|
||||
_ uow.ReleaserSource = (*uowProvider)(nil)
|
||||
_ uow.LeaseReclaimerSource = (*uowProvider)(nil)
|
||||
_ uow.HeartbeaterSource = (*uowProvider)(nil)
|
||||
_ uow.MemoriesSource = (*uowProvider)(nil)
|
||||
_ uow.EventsJournalCursorSource = (*uowProvider)(nil)
|
||||
)
|
||||
|
|
|
|||
|
|
@ -42,6 +42,9 @@ func (r *recordingBatchApplyHooks) CompleteIssueOperationRelease(_ context.Conte
|
|||
func (r *recordingBatchApplyHooks) CompleteIssueOperationReclaim(_ context.Context, issueID string) {
|
||||
r.completions = append(r.completions, "reclaim:"+issueID)
|
||||
}
|
||||
func (r *recordingBatchApplyHooks) CompleteIssueOperationUpdateByID(_ context.Context, issueID string) {
|
||||
r.completions = append(r.completions, "update-by-id:"+issueID)
|
||||
}
|
||||
|
||||
func issueIDOrNil(issue *types.Issue) string {
|
||||
if issue == nil {
|
||||
|
|
@ -234,3 +237,4 @@ func (r *recordingBatchApplyIssues) CompleteIssueOperationComment(context.Contex
|
|||
func (r *recordingBatchApplyIssues) CompleteIssueOperationMetadata(context.Context, string) {}
|
||||
func (r *recordingBatchApplyIssues) CompleteIssueOperationRelease(context.Context, string) {}
|
||||
func (r *recordingBatchApplyIssues) CompleteIssueOperationReclaim(context.Context, string) {}
|
||||
func (r *recordingBatchApplyIssues) CompleteIssueOperationUpdateByID(context.Context, string) {}
|
||||
|
|
|
|||
|
|
@ -160,6 +160,11 @@ func RoleFiresHooks(role any) bool {
|
|||
// or a request.
|
||||
case *hookLeaseReclaimer:
|
||||
return true
|
||||
// The heartbeater fires the update hook for a RE-ARM only
|
||||
// (hook_heartbeater.go): rare, but a write the workspace's script runs
|
||||
// for, so a server handed an unpeeled one is refused like the others.
|
||||
case *hookHeartbeater:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
|
@ -482,6 +487,13 @@ func (h *HookFiringStore) CompleteIssueOperationReclaim(ctx context.Context, iss
|
|||
h.fireHookByID(ctx, hooks.EventUpdate, issueID)
|
||||
}
|
||||
|
||||
// CompleteIssueOperationUpdateByID fires on_update for a write that names
|
||||
// its row by id (a heartbeat's re-arm), reading the row back to hand it to
|
||||
// the script.
|
||||
func (h *HookFiringStore) CompleteIssueOperationUpdateByID(ctx context.Context, issueID string) {
|
||||
h.fireHookByID(ctx, hooks.EventUpdate, issueID)
|
||||
}
|
||||
|
||||
func (h *HookFiringStore) fireHookByID(ctx context.Context, event, id string) {
|
||||
if !h.hookConfigured(event) {
|
||||
return
|
||||
|
|
|
|||
47
internal/storage/hook_heartbeater.go
Normal file
47
internal/storage/hook_heartbeater.go
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// Heartbeater returns the inner store's lease-refresh surface with this
|
||||
// decorator's completion hook layered over it.
|
||||
//
|
||||
// A REFRESH FIRES NOTHING. It writes only the ephemeral lease row: it changes
|
||||
// no field a hook script could read back, it is the hottest write in a fleet,
|
||||
// and the raw HeartbeatIssue has never fired a hook. Firing on_update for it
|
||||
// would run a user's subprocess every ninety seconds per worker for a row
|
||||
// that did not change.
|
||||
//
|
||||
// A RE-ARM FIRES on_update, once. It is recorded as a mutation (row_lock,
|
||||
// updated_at, an event, a journal entry, a version), the row it names is
|
||||
// still there to hand a script, and the result's Rearmed is the fact that the
|
||||
// row was written — the Releaser's Changed rule.
|
||||
//
|
||||
// It recurses instead of delegating, so a blind delegation would hand back
|
||||
// the inner store's surface and silently stop firing it.
|
||||
func (h *HookFiringStore) Heartbeater() (issueops.Heartbeater, error) {
|
||||
inner, err := h.inner.Heartbeater()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &hookHeartbeater{inner: inner, hooks: h}, nil
|
||||
}
|
||||
|
||||
type hookHeartbeater struct {
|
||||
inner issueops.Heartbeater
|
||||
hooks issueOperationHooks
|
||||
}
|
||||
|
||||
// Heartbeat fires the update hook for a re-arm and nothing for a refresh or
|
||||
// a refusal.
|
||||
func (b *hookHeartbeater) Heartbeat(ctx context.Context, request issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
result, err := b.inner.Heartbeat(ctx, request)
|
||||
if err != nil || !result.Rearmed {
|
||||
return result, err
|
||||
}
|
||||
b.hooks.CompleteIssueOperationUpdateByID(ctx, result.IssueID)
|
||||
return result, nil
|
||||
}
|
||||
42
internal/storage/hook_heartbeater_test.go
Normal file
42
internal/storage/hook_heartbeater_test.go
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
type stubHeartbeater struct{ result issueops.HeartbeatResult }
|
||||
|
||||
func (s stubHeartbeater) Heartbeat(context.Context, issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
return s.result, nil
|
||||
}
|
||||
|
||||
// TestHookHeartbeaterFiresOnlyForARearm pins hook_heartbeater.go: a refresh
|
||||
// fires nothing, a re-arm (a mutation) fires on_update once.
|
||||
func TestHookHeartbeaterFiresOnlyForARearm(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
rearmed bool
|
||||
want []string
|
||||
}{
|
||||
{"a refresh fires nothing", false, nil},
|
||||
{"a re-arm fires on_update", true, []string{"update-by-id:bd-1"}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
recorder := &recordingIssueOperationHooks{}
|
||||
hooked := &hookHeartbeater{
|
||||
inner: stubHeartbeater{result: issueops.HeartbeatResult{IssueID: "bd-1", Rearmed: test.rearmed}},
|
||||
hooks: recorder,
|
||||
}
|
||||
if _, err := hooked.Heartbeat(t.Context(), issueops.HeartbeatRequest{Actor: "w", IssueID: "bd-1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(recorder.completions, test.want) {
|
||||
t.Fatalf("completions = %v, want %v", recorder.completions, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -36,6 +36,7 @@ type issueOperationHooks interface {
|
|||
CompleteIssueOperationMetadata(ctx context.Context, issueID string)
|
||||
CompleteIssueOperationRelease(ctx context.Context, issueID string)
|
||||
CompleteIssueOperationReclaim(ctx context.Context, issueID string)
|
||||
CompleteIssueOperationUpdateByID(ctx context.Context, issueID string)
|
||||
}
|
||||
|
||||
var _ issueOperationHooks = (*HookFiringStore)(nil)
|
||||
|
|
|
|||
|
|
@ -66,6 +66,9 @@ func (r *recordingIssueOperationHooks) CompleteIssueOperationRelease(_ context.C
|
|||
func (r *recordingIssueOperationHooks) CompleteIssueOperationReclaim(_ context.Context, issueID string) {
|
||||
r.completions = append(r.completions, "reclaim:"+issueID)
|
||||
}
|
||||
func (r *recordingIssueOperationHooks) CompleteIssueOperationUpdateByID(_ context.Context, issueID string) {
|
||||
r.completions = append(r.completions, "update-by-id:"+issueID)
|
||||
}
|
||||
|
||||
// lifecycleStore is a DoltStorage whose only real method is IssueLifecycle.
|
||||
type lifecycleStore struct {
|
||||
|
|
|
|||
|
|
@ -58,6 +58,7 @@ go_library(
|
|||
"filters.go",
|
||||
"gate_fields.go",
|
||||
"get_issue.go",
|
||||
"heartbeat_role.go",
|
||||
"helpers.go",
|
||||
"history.go",
|
||||
"history_presence.go",
|
||||
|
|
|
|||
|
|
@ -25,6 +25,19 @@ func (tables ChangedTables) Add(names ...string) {
|
|||
}
|
||||
}
|
||||
|
||||
// Sorted returns the recorded tables in name order (nil when empty).
|
||||
func (tables ChangedTables) Sorted() []string {
|
||||
if len(tables) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(tables))
|
||||
for table := range tables {
|
||||
out = append(out, table)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Merge records every table in other.
|
||||
func (tables ChangedTables) Merge(other map[string]bool) {
|
||||
for table := range other {
|
||||
|
|
|
|||
135
internal/storage/issueops/heartbeat_role.go
Normal file
135
internal/storage/issueops/heartbeat_role.go
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
package issueops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
publicops "github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// ValidateHeartbeatRequest applies the request rules every Heartbeater
|
||||
// implementation shares. It is the ONE definition of a valid heartbeat: the
|
||||
// library legs run it before they open a transaction, and the shared body runs
|
||||
// it again, so a leg that skipped it still answers the same contract.
|
||||
//
|
||||
// THE ACTOR IS NOT TRIMMED, Releaser.Actor's rule: it is the fence's subject
|
||||
// and is compared with the lease row's holder as sent. Only a blank one is
|
||||
// refused, and one longer than the lease row's holder column can hold.
|
||||
func ValidateHeartbeatRequest(request publicops.HeartbeatRequest) error {
|
||||
if strings.TrimSpace(request.Actor) == "" {
|
||||
return heartbeatFieldError(publicops.HeartbeatFieldActor, "heartbeat actor is required")
|
||||
}
|
||||
if err := types.CheckFieldLen("actor", request.Actor); err != nil {
|
||||
return heartbeatFieldError(publicops.HeartbeatFieldActor, err.Error())
|
||||
}
|
||||
if request.IssueID == "" {
|
||||
return heartbeatFieldError(publicops.HeartbeatFieldIssueID, "heartbeat issue id is required")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func heartbeatFieldError(field, detail string) error {
|
||||
return &publicops.HeartbeatFieldError{Field: field, Detail: storage.ErrValidation.Error() + ": " + detail}
|
||||
}
|
||||
|
||||
// ExecuteHeartbeatInTx refreshes the lease request.Actor holds on
|
||||
// request.IssueID, in tx. It is the body behind the Heartbeater accessor on
|
||||
// ALL THREE library legs: dolt, embedded dolt and the unit-of-work provider.
|
||||
//
|
||||
// It is RenewLeaseInTx — the native heartbeat, SQL text unchanged — with the
|
||||
// role's request rules in front and its typed refusals behind:
|
||||
//
|
||||
// - a wisp is refused before anything is written, as every native leg
|
||||
// refuses it ("is ephemeral"): wisps are never leased;
|
||||
// - an id that names no row on either plane is storage.ErrNotFound, where
|
||||
// the native slow path answered ErrNotClaimable;
|
||||
// - a foreign holder and a missing live claim are *LeaseConflictError,
|
||||
// classified from the issue row the slow path already read.
|
||||
//
|
||||
// A REFUSAL MAY WRITE ONE THING: it deletes the caller's own lease row when
|
||||
// that row survives on an issue the caller no longer holds a live claim on (an
|
||||
// orphan; see RenewLeaseInTx). The legs therefore COMMIT the transaction of a
|
||||
// refusal HeartbeatRefusalCommits recognizes and report the refusal after the
|
||||
// commit, so the cleanup is not rolled back with it.
|
||||
//
|
||||
// WHAT A REFRESH MUST NEVER DO is everything RenewLeaseInTx's own doc rules
|
||||
// out, and the conformance contract asserts each one: remint row_lock, record
|
||||
// a version (current_revision, issue_versions), stamp updated_at, write an
|
||||
// event or a journal entry. The legs add the other two: no version-control
|
||||
// commit and no hook. A RE-ARM (HeartbeatResult.Rearmed) does every one of
|
||||
// them, because it is recorded as a mutation: the legs commit it once, as
|
||||
// LeaseRearmVersionCommit names, and the hook decorator fires on_update.
|
||||
func ExecuteHeartbeatInTx(ctx context.Context, tx DBTX, request publicops.HeartbeatRequest) (publicops.HeartbeatResult, error) {
|
||||
if err := ValidateHeartbeatRequest(request); err != nil {
|
||||
return publicops.HeartbeatResult{}, err
|
||||
}
|
||||
id := request.IssueID
|
||||
|
||||
var wispStatus string
|
||||
if err := tx.QueryRowContext(ctx, "SELECT status FROM wisps WHERE id = ? LIMIT 1", id).Scan(&wispStatus); err == nil {
|
||||
return publicops.HeartbeatResult{}, &publicops.LeaseConflictError{
|
||||
IssueID: id,
|
||||
Status: types.Status(wispStatus),
|
||||
Ephemeral: true,
|
||||
Err: fmt.Errorf("%w: %s is ephemeral", publicops.ErrLeaseNotHeld, id),
|
||||
}
|
||||
}
|
||||
|
||||
renewal, err := RenewLeaseInTx(ctx, tx, id, request.Actor)
|
||||
if err != nil {
|
||||
var refusal *LeaseRenewalRefusal
|
||||
if !errors.As(err, &refusal) {
|
||||
return publicops.HeartbeatResult{}, err
|
||||
}
|
||||
return publicops.HeartbeatResult{}, classifyLeaseRenewalRefusal(refusal, request.Actor)
|
||||
}
|
||||
return publicops.HeartbeatResult{
|
||||
IssueID: id,
|
||||
Holder: request.Actor,
|
||||
HeartbeatAt: renewal.HeartbeatAt,
|
||||
LeaseExpiresAt: renewal.ExpiresAt,
|
||||
Rearmed: renewal.Rearmed,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// classifyLeaseRenewalRefusal types a native heartbeat refusal for the role.
|
||||
func classifyLeaseRenewalRefusal(refusal *LeaseRenewalRefusal, actor string) error {
|
||||
id := refusal.IssueID
|
||||
if !refusal.Found {
|
||||
if errors.Is(refusal.ReadErr, sql.ErrNoRows) {
|
||||
return fmt.Errorf("%w: issue %s", storage.ErrNotFound, id)
|
||||
}
|
||||
return fmt.Errorf("heartbeat %s: read issue: %w", id, refusal.ReadErr)
|
||||
}
|
||||
status := types.Status(refusal.Status)
|
||||
if refusal.Wisp {
|
||||
return &publicops.LeaseConflictError{
|
||||
IssueID: id, Status: status, Ephemeral: true,
|
||||
Err: fmt.Errorf("%w: %s is ephemeral", publicops.ErrLeaseNotHeld, id),
|
||||
}
|
||||
}
|
||||
if refusal.Assignee != "" && !actorMatches(refusal.Assignee, actor) {
|
||||
return &publicops.LeaseConflictError{
|
||||
IssueID: id, Holder: refusal.Assignee, Status: status, Err: refusal.Err,
|
||||
}
|
||||
}
|
||||
return &publicops.LeaseConflictError{
|
||||
IssueID: id, Status: status,
|
||||
Err: fmt.Errorf("%w: %s status %s", publicops.ErrLeaseNotHeld, id, refusal.Status),
|
||||
}
|
||||
}
|
||||
|
||||
// HeartbeatRefusalCommits reports whether err is a heartbeat refusal whose
|
||||
// transaction a leg must COMMIT rather than roll back: a typed
|
||||
// *LeaseConflictError, the one refusal that can carry a write (the orphaned
|
||||
// lease row ExecuteHeartbeatInTx deletes). Committing a refusal that wrote
|
||||
// nothing is a no-op. Every other error rolls back as usual.
|
||||
func HeartbeatRefusalCommits(err error) bool {
|
||||
var conflict *publicops.LeaseConflictError
|
||||
return errors.As(err, &conflict)
|
||||
}
|
||||
|
|
@ -98,7 +98,16 @@ var mutationEntryPoints = []string{
|
|||
"ClaimReadyIssueInTx",
|
||||
"UnclaimIssueInTx",
|
||||
"UnclaimIssueIfAssigneeInTx",
|
||||
// HeartbeatIssueInTx is deliberately absent — see journalExemptMutations.
|
||||
// The heartbeat family journals on its RE-ARM path only (a live claim
|
||||
// with no lease row the caller holds: row_lock reminted, a lease granted,
|
||||
// recorded as a mutation). Its REFRESH path writes only the clone-local
|
||||
// leases table and still records nothing; the fast-path return in
|
||||
// RenewLeaseInTx says why. The ruling that exempted the whole family
|
||||
// changed when the re-arm became a mutation (it must collide with a racing
|
||||
// release or close on row_lock), so the family is listed here.
|
||||
"RenewLeaseInTx",
|
||||
"HeartbeatIssueInTx",
|
||||
"ExecuteHeartbeatInTx",
|
||||
"ReclaimExpiredLeasesInTx",
|
||||
// plane moves, graph edges, labels, comments, renames
|
||||
"PromoteFromEphemeralInTx",
|
||||
|
|
@ -209,13 +218,7 @@ var beadDMLExemptions = map[string]string{
|
|||
// never are. TestExemptMutationsDoNotJournal pins the decision from the other
|
||||
// side — each must exist and must NOT emit — so re-adding an emit fails loudly
|
||||
// instead of silently reversing the ruling.
|
||||
var journalExemptMutations = map[string]string{
|
||||
"HeartbeatIssueInTx": "high-frequency lease keepalive: writes only the clone-local `leases` table " +
|
||||
"(lease-liveness state), never a durable bead field. Journaling it would put a full-snapshot " +
|
||||
"write plus the shared seq-counter serialization on the hottest write in the fleet, and a replay " +
|
||||
"consumer gains nothing — lease state is working-set-plane and expires on its own. Lease RECLAIM " +
|
||||
"is the opposite case and does journal: it clears assignee and reverts status, which is durable.",
|
||||
}
|
||||
var journalExemptMutations = map[string]string{}
|
||||
|
||||
// TestEveryMutationFunctionJournals parses this package's source, builds the
|
||||
// intra-package call graph, and asserts every mutation entry point either
|
||||
|
|
|
|||
|
|
@ -3,7 +3,9 @@ package issueops
|
|||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"database/sql"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
|
@ -62,10 +64,14 @@ func leaseTTL(ctx context.Context) time.Duration {
|
|||
// races care about (claim, close, updateIssueInTx, reclaim, unclaim all do).
|
||||
// Paths that touch only orthogonal cells (is_blocked, compaction_level,
|
||||
// dependency metadata, rename, or reopen — which acts on closed rows) are safe
|
||||
// to merge with a reclaim and intentionally do NOT rewrite it. Heartbeats no
|
||||
// longer touch the issues row at all (bd-lrgn1): the lease lives in the
|
||||
// to merge with a reclaim and intentionally do NOT rewrite it. A heartbeat
|
||||
// REFRESH does not touch the issues row (bd-lrgn1): the lease lives in the
|
||||
// ephemeral leases table, where a racing heartbeat and reclaim contend on the
|
||||
// SAME lease row and conflict without any help. Adding a new path that sets
|
||||
// SAME lease row and conflict without any help. A heartbeat RE-ARM (a lease
|
||||
// granted to a live claim that has none) is the one deliberate remint that
|
||||
// changes no lifecycle field, and it is load-bearing: its lease INSERT shares
|
||||
// no row with a concurrent release or close, so the remint is what makes the
|
||||
// two collide (see RenewLeaseInTx). Adding a new path that sets
|
||||
// status/assignee outside updateIssueInTx without rewriting row_lock would
|
||||
// silently reintroduce the zombie-merge bug.
|
||||
//
|
||||
|
|
@ -78,10 +84,11 @@ func leaseTTL(ctx context.Context) time.Duration {
|
|||
// that already goes out of its way to preserve updated_at
|
||||
// (blocked_state.go), and bumping the token there would clobber a concurrent
|
||||
// whole-row CAS over a cell derived from OTHER rows. (The PR's second
|
||||
// exemption of this kind, the lease heartbeat, no longer arises here — since
|
||||
// bd-lrgn1 heartbeats never touch the issues row at all.) So the invariant
|
||||
// cuts both ways: status/ownership writes MUST remint, aux-marker writes MUST
|
||||
// NOT. The remint direction is enforced at build time by
|
||||
// exemption of this kind, the lease heartbeat, no longer arises here: since
|
||||
// bd-lrgn1 a refresh never touches the issues row, and a re-arm remints on
|
||||
// purpose, accepting that a guarded write composed before it fails once and
|
||||
// re-reads.) So the invariant cuts both ways: status/ownership writes MUST
|
||||
// remint, aux-marker writes MUST NOT. The remint direction is enforced at build time by
|
||||
// TestAllIssueRowWritesStampRowLock (row_lock_guard_test.go), whose exemption
|
||||
// markers are the machine-readable form of this list — widen the stamping
|
||||
// policy there first, not by ad-hoc stamps.
|
||||
|
|
@ -293,14 +300,115 @@ func RestoreLeaseOnImportInTx(ctx context.Context, tx DBTX, issue *types.Issue,
|
|||
// storage.ErrNotClaimable / ErrAlreadyClaimed so the caller learns its lease
|
||||
// is gone.
|
||||
//
|
||||
// The write touches ONLY the leases table (ephemeral, dolt_ignored): a
|
||||
// A refresh touches ONLY the leases table (ephemeral, dolt_ignored): a
|
||||
// heartbeat mints no Dolt commit and no history, and deliberately does NOT
|
||||
// stamp issues.updated_at — updated_at keeps its merge/LWW meaning and bd
|
||||
// stale consults leases.heartbeat_at for in_progress rows instead (bd-lrgn1).
|
||||
//
|
||||
//nolint:gosec // G201: table names come from WispTableRouting (hardcoded constants)
|
||||
// A RE-ARM (a live claim with no lease row the caller holds, which the
|
||||
// heartbeat grants) is the exception: it is recorded as a mutation (row_lock,
|
||||
// updated_at, event, journal, version); see RenewLeaseInTx. A caller that runs
|
||||
// this in a transaction must make the version-control commit
|
||||
// LeaseRearmVersionCommit names when it re-arms.
|
||||
//
|
||||
// It is RenewLeaseInTx with the renewal's report dropped: the legacy
|
||||
// HeartbeatIssue legs answer only whether the lease was refreshed.
|
||||
func HeartbeatIssueInTx(ctx context.Context, tx DBTX, id, actor string) error {
|
||||
_, err := RenewLeaseInTx(ctx, tx, id, actor)
|
||||
return err
|
||||
}
|
||||
|
||||
// LeaseRenewal reports one lease RenewLeaseInTx refreshed or re-armed. Both
|
||||
// times are computed in Go from the same clock reading and TTL the statement
|
||||
// binds, rounded to whole seconds as the lease columns round them on store,
|
||||
// so they are what the lease row now holds.
|
||||
type LeaseRenewal struct {
|
||||
HeartbeatAt time.Time
|
||||
ExpiresAt time.Time
|
||||
// Rearmed is true only when the slow path CREATED the caller's lease row:
|
||||
// the caller held a live claim with no lease row for it (a hand-doled
|
||||
// claim), or the issue's lease row named a holder that is not the caller.
|
||||
// A refresh of the caller's own row is never a re-arm, even when it
|
||||
// changed no value (two heartbeats in one second).
|
||||
Rearmed bool
|
||||
}
|
||||
|
||||
// LeaseRearmVersionCommit is the version-control commit a leg makes after a
|
||||
// heartbeat: none for a refresh or a refusal (the leases table is
|
||||
// dolt_ignored), and the release's staged set, issues and events, for a
|
||||
// re-arm, which is recorded as a mutation. One helper for every leg so the
|
||||
// message cannot drift.
|
||||
func LeaseRearmVersionCommit(id string, rearmed bool) (ChangedTables, string) {
|
||||
if !rearmed {
|
||||
return nil, ""
|
||||
}
|
||||
tables := ChangedTables{}
|
||||
tables.Add("issues", "events")
|
||||
return tables, fmt.Sprintf("bd: re-arm lease %s", id)
|
||||
}
|
||||
|
||||
// LeaseRenewalRefusal is the refusal RenewLeaseInTx answers when no lease row
|
||||
// matched the caller and the slow path could not re-arm one. It carries the
|
||||
// issue row the slow path read so a typed front door (ExecuteHeartbeatInTx)
|
||||
// can classify the refusal without a second read, and it reads and unwraps as
|
||||
// Err — the exact error HeartbeatIssueInTx has always returned — so the
|
||||
// legacy legs' text and errors.Is behavior are unchanged.
|
||||
type LeaseRenewalRefusal struct {
|
||||
IssueID string
|
||||
// Found is false when the issue row could not be read.
|
||||
Found bool
|
||||
// ReadErr is the error the issue-row read failed with when Found is false.
|
||||
ReadErr error
|
||||
Wisp bool
|
||||
Assignee string
|
||||
Status string
|
||||
// Err is the wrapped refusal: storage.ErrAlreadyClaimed or
|
||||
// storage.ErrNotClaimable.
|
||||
Err error
|
||||
// OrphanDeleted is true when the refusal deleted a lease row the caller
|
||||
// held on an issue it no longer holds a live claim on (see
|
||||
// RenewLeaseInTx). The deletion is part of the refusing transaction: a
|
||||
// caller that wants it kept must COMMIT that transaction and report the
|
||||
// refusal afterwards (HeartbeatRefusalCommits).
|
||||
OrphanDeleted bool
|
||||
}
|
||||
|
||||
func (e *LeaseRenewalRefusal) Error() string { return e.Err.Error() }
|
||||
|
||||
// Unwrap makes LeaseRenewalRefusal match the refusal it carries.
|
||||
func (e *LeaseRenewalRefusal) Unwrap() error { return e.Err }
|
||||
|
||||
// LeaseRenewalRefusalCommits reports whether err is a RenewLeaseInTx refusal
|
||||
// whose transaction a legacy HeartbeatIssue leg must COMMIT rather than roll
|
||||
// back: one that read the issue row (Found), the only refusal that can carry a
|
||||
// write, the orphaned lease row it deletes. A refusal whose issue read failed
|
||||
// deletes nothing, and may follow a fast-path UPDATE that already extended an
|
||||
// orphan row's expiry, so its transaction rolls back — the rule the role's
|
||||
// HeartbeatRefusalCommits applies to the same refusal.
|
||||
func LeaseRenewalRefusalCommits(err error) bool {
|
||||
var refused *LeaseRenewalRefusal
|
||||
return errors.As(err, &refused) && refused.Found
|
||||
}
|
||||
|
||||
// RenewLeaseInTx is the body of HeartbeatIssueInTx, reporting the renewal:
|
||||
// see that function for what it writes and, above all, what it must never
|
||||
// write. Its SQL text is the heartbeat's own and must stay so: the Postgres
|
||||
// dialect rewrites the one IF() in it keyed on its `update leases set`
|
||||
// prefix.
|
||||
//
|
||||
//nolint:gosec // G201: table names come from WispTableRouting (hardcoded constants)
|
||||
func RenewLeaseInTx(ctx context.Context, tx DBTX, id, actor string) (LeaseRenewal, error) {
|
||||
// THE BOUND CLOCK IS UNROUNDED; THE REPORTED ONE IS ROUNDED. The lease
|
||||
// columns are DATETIME (whole seconds), which round a sub-second value on
|
||||
// store, so the result reports now rounded to the second: exactly what
|
||||
// the row now holds. The binds stay unrounded on purpose: Dolt counts a
|
||||
// value-identical UPDATE as 0 rows affected even under CLIENT_FOUND_ROWS,
|
||||
// and binding the rounded value would make every same-second refresh look
|
||||
// like a miss and take the slow path (several extra reads on the hottest
|
||||
// write in the fleet).
|
||||
now := time.Now().UTC()
|
||||
reported := now.Round(time.Second)
|
||||
ttl := leaseTTL(ctx)
|
||||
// granted_node is backfilled, never overwritten: a lease row whose
|
||||
// provenance is unknown (granted before ignored migration 0016, or before
|
||||
// this replica was named) is being kept alive through THIS node's store,
|
||||
|
|
@ -340,28 +448,69 @@ func HeartbeatIssueInTx(ctx context.Context, tx DBTX, id, actor string) error {
|
|||
// `bd unclaim --force <id>` for a single stranded unit once that replica is
|
||||
// gone for good.
|
||||
//
|
||||
// SCOPE, precisely: the no-re-home invariant holds on THIS fast path only.
|
||||
// The disambiguation fallback below re-homes, by design and unavoidably —
|
||||
// when `holder = ?` misses because the lease row spells the same identity
|
||||
// differently (ga-v2k49/ga-wzl83), it re-arms through UpsertLeaseInTx, whose
|
||||
// ON DUPLICATE KEY UPDATE stamps the LOCAL node over whatever was there.
|
||||
// So "a heartbeat never re-homes a foreign lease" is false for a
|
||||
// spelling-mismatched holder, and RestoreLeaseOnImportInTx can relabel in
|
||||
// the other direction (it takes granted_node from the snapshot when the
|
||||
// local row is expired). Neither is changed here: both are pre-existing
|
||||
// paths with their own rationale, and the point of this note is that the
|
||||
// invariant is a property of the common path, not of the lease row.
|
||||
// SCOPE, precisely: the no-re-home invariant holds on every REFRESH — this
|
||||
// fast path and the slow path's in-place refresh of a row the caller
|
||||
// already holds (same second, or the same identity spelled differently,
|
||||
// ga-v2k49/ga-wzl83), which keeps granted_at and granted_node. Only a
|
||||
// RE-ARM re-homes: when the caller holds a live claim and no lease row the
|
||||
// caller holds exists, UpsertLeaseInTx mints one and stamps the LOCAL node.
|
||||
// RestoreLeaseOnImportInTx can relabel in the other direction (it takes
|
||||
// granted_node from the snapshot when the local row is expired); that is a
|
||||
// pre-existing path with its own rationale.
|
||||
//
|
||||
// THE LIVE-CLAIM PREDICATE. The fast path refreshes the caller's row only
|
||||
// while the issue is still in_progress under the caller (verbatim, the
|
||||
// holder predicate's own rule). "A lease row exists only while its claim
|
||||
// is live" is an invariant every claim-ending path maintains by deleting
|
||||
// the row, and the slow path's re-arm keeps it against a concurrent ender
|
||||
// by reminting row_lock (see below). The predicate is the second line: a
|
||||
// lease row orphaned any other way (a binary without that remint, a
|
||||
// hand-edited table) fails the fast path, and the slow path refuses AND
|
||||
// deletes it, instead of every later heartbeat refreshing it and answering
|
||||
// success for a claim that no longer exists. The `update leases set`
|
||||
// prefix and the one IF() the Postgres dialect rewrites are unchanged.
|
||||
//
|
||||
// THE COMPARISON IS VERBATIM (assignee = the holder as sent), deliberately,
|
||||
// as the holder predicate is: a holder whose lease row and assignee are
|
||||
// spelled differently (ga-v2k49's separator case) misses here on every
|
||||
// beat and takes the slow path, which judges under actorMatches and
|
||||
// refreshes the row in place. Correct, a few reads slower, and only for
|
||||
// that spelling mismatch; the common case pays one indexed count read.
|
||||
//
|
||||
// FOR A POSTGRES PORT: the refresh now READS the issues row. Under
|
||||
// SERIALIZABLE that is a read-write dependency on every concurrent write
|
||||
// to the same issue (the holder's own edits racing its heartbeat), so
|
||||
// expect more serialization failures on this statement, and keep it inside
|
||||
// the retrying write transaction.
|
||||
result, err := tx.ExecContext(ctx, `
|
||||
UPDATE leases SET lease_expires_at = ?, heartbeat_at = ?,
|
||||
granted_node = IF(COALESCE(granted_node, '') = '', ?, granted_node)
|
||||
WHERE issue_id = ? AND holder = ?
|
||||
`, now.Add(leaseTTL(ctx)), now, NodeID(ctx), id, actor)
|
||||
`, now.Add(ttl), now, NodeID(ctx), id, actor)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to heartbeat issue: %w", err)
|
||||
return LeaseRenewal{}, fmt.Errorf("failed to heartbeat issue: %w", err)
|
||||
}
|
||||
rows, err := result.RowsAffected()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get rows affected: %w", err)
|
||||
return LeaseRenewal{}, fmt.Errorf("failed to get rows affected: %w", err)
|
||||
}
|
||||
if rows > 0 {
|
||||
// The live-claim check, in the same transaction as the UPDATE (so on
|
||||
// the same snapshot, and atomically with it: a refusal below commits
|
||||
// nothing the caller keeps but the orphan delete). It is a separate
|
||||
// read rather than an EXISTS in the UPDATE because Dolt reports a
|
||||
// value-identical UPDATE that carries a subquery as 0 rows affected
|
||||
// even under CLIENT_FOUND_ROWS, which would send every same-second
|
||||
// refresh down the slow path.
|
||||
var live int
|
||||
if err := tx.QueryRowContext(ctx,
|
||||
`SELECT COUNT(*) FROM issues WHERE id = ? AND status = 'in_progress' AND assignee = ?`, id, actor,
|
||||
).Scan(&live); err != nil {
|
||||
return LeaseRenewal{}, fmt.Errorf("failed to heartbeat issue: %w", err)
|
||||
}
|
||||
if live == 0 {
|
||||
rows = 0 // an orphan or a spelling mismatch: the slow path decides
|
||||
}
|
||||
}
|
||||
if rows == 0 {
|
||||
// No lease row. Disambiguate from the issue row: gone
|
||||
|
|
@ -374,7 +523,47 @@ func HeartbeatIssueInTx(ctx context.Context, tx DBTX, id, actor string) error {
|
|||
fmt.Sprintf("SELECT COALESCE(assignee, ''), status FROM %s WHERE id = ?", issueTable), id,
|
||||
).Scan(&assignee, &status)
|
||||
if qerr != nil {
|
||||
return fmt.Errorf("%w: %s", storage.ErrNotClaimable, id)
|
||||
return LeaseRenewal{}, &LeaseRenewalRefusal{
|
||||
IssueID: id, ReadErr: qerr, Wisp: isWisp,
|
||||
Err: fmt.Errorf("%w: %s", storage.ErrNotClaimable, id),
|
||||
}
|
||||
}
|
||||
// A refusal deletes the caller's own lease row, if one survives on an
|
||||
// issue the caller no longer holds a live claim on: that row is an
|
||||
// orphan (the re-arm race above), and leaving it would keep a stale
|
||||
// lease on an issue that is open, closed or someone else's. The holder
|
||||
// fence keeps it to the caller's own row; a live holder's lease is
|
||||
// never touched. The fence is judged under actorMatches, as the
|
||||
// in-place refresh below judges it: a row held under another spelling
|
||||
// of the caller (ga-v2k49) is the caller's orphan too, so its holder
|
||||
// is read and bound back, the refresh's own lookup, on this cold path
|
||||
// only. The delete is judged in this transaction's snapshot, which is
|
||||
// the snapshot that just found no live claim for the caller.
|
||||
refused := func(err error) error {
|
||||
refusal := &LeaseRenewalRefusal{
|
||||
IssueID: id, Found: true, Wisp: isWisp, Assignee: assignee, Status: status, Err: err,
|
||||
}
|
||||
if isWisp {
|
||||
return refusal
|
||||
}
|
||||
var leaseHolder string
|
||||
lerr := tx.QueryRowContext(ctx, `SELECT holder FROM leases WHERE issue_id = ?`, id).Scan(&leaseHolder)
|
||||
switch {
|
||||
case errors.Is(lerr, sql.ErrNoRows):
|
||||
return refusal
|
||||
case lerr != nil:
|
||||
return fmt.Errorf("read lease for %s: %w", id, lerr)
|
||||
case !actorMatches(leaseHolder, actor):
|
||||
return refusal
|
||||
}
|
||||
res, derr := tx.ExecContext(ctx, `DELETE FROM leases WHERE issue_id = ? AND holder = ?`, id, leaseHolder)
|
||||
if derr != nil {
|
||||
return fmt.Errorf("delete orphaned lease for %s: %w", id, derr)
|
||||
}
|
||||
if n, nerr := res.RowsAffected(); nerr == nil && n > 0 {
|
||||
refusal.OrphanDeleted = true
|
||||
}
|
||||
return refusal
|
||||
}
|
||||
// Judged under actorMatches, not verbatim (ga-v2k49): the primary
|
||||
// UPDATE's own `holder = ?` predicate above stays a verbatim SQL
|
||||
|
|
@ -383,11 +572,11 @@ func HeartbeatIssueInTx(ctx context.Context, tx DBTX, id, actor string) error {
|
|||
// above) just to bind back its own holder value. A spelling
|
||||
// difference across layers (ga-wzl83) already makes that predicate
|
||||
// affect 0 rows and fall here; fixing the disambiguation instead
|
||||
// means the caller still self-heals correctly (re-arms below, under
|
||||
// its current spelling) at the cost of the slow path on a spelling
|
||||
// mismatch, never on the byte-identical common case.
|
||||
// means the caller still self-heals correctly (its row is refreshed in
|
||||
// place below, under its current spelling) at the cost of the slow
|
||||
// path on a spelling mismatch, never on the byte-identical common case.
|
||||
if assignee != "" && !actorMatches(assignee, actor) {
|
||||
return fmt.Errorf("%w by %s", storage.ErrAlreadyClaimed, assignee)
|
||||
return LeaseRenewal{}, refused(fmt.Errorf("%w by %s", storage.ErrAlreadyClaimed, assignee))
|
||||
}
|
||||
if !isWisp && actorMatches(assignee, actor) && status == string(types.StatusInProgress) {
|
||||
// The caller genuinely holds the claim but has no lease row — e.g.
|
||||
|
|
@ -396,13 +585,80 @@ func HeartbeatIssueInTx(ctx context.Context, tx DBTX, id, actor string) error {
|
|||
// semantics, or (ga-v2k49) the existing lease row's holder is a
|
||||
// different spelling of the same identity (ga-wzl83) and the
|
||||
// primary UPDATE's verbatim predicate above missed it. A real
|
||||
// worker's heartbeat re-arms recovery — under actor's current
|
||||
// spelling, which is what the next heartbeat's fast path will see.
|
||||
return UpsertLeaseInTx(ctx, tx, id, actor, now, leaseTTL(ctx))
|
||||
// worker's heartbeat recovers under actor's current spelling: a
|
||||
// lease row the caller already holds is refreshed in place (its
|
||||
// holder rewritten to that spelling, which the next heartbeat's
|
||||
// UPDATE matches), and a missing one is re-armed. An ASSIGNEE
|
||||
// spelled differently from actor still fails the verbatim
|
||||
// live-claim check, so that holder takes this path on every beat.
|
||||
//
|
||||
// FIRST, is there a lease row for this issue the caller already
|
||||
// holds? Then this is a refresh the fast path missed without
|
||||
// anything being wrong: on embedded Dolt an UPDATE that changes no
|
||||
// value (a second heartbeat in the same second) reports 0 rows, and
|
||||
// a holder spelled differently from the assignee misses the
|
||||
// verbatim live-claim predicate. Refresh that row in place, keeping
|
||||
// its granted_at and granted_node (the fast path's no-re-home rule,
|
||||
// with the same backfill), and report no re-arm.
|
||||
var leaseHolder string
|
||||
lerr := tx.QueryRowContext(ctx, `SELECT holder FROM leases WHERE issue_id = ?`, id).Scan(&leaseHolder)
|
||||
switch {
|
||||
case lerr == nil && actorMatches(leaseHolder, actor):
|
||||
if _, err := tx.ExecContext(ctx, `
|
||||
UPDATE leases SET lease_expires_at = ?, heartbeat_at = ?, holder = ?,
|
||||
granted_node = IF(COALESCE(granted_node, '') = '', ?, granted_node)
|
||||
WHERE issue_id = ? AND holder = ?
|
||||
`, now.Add(ttl), now, actor, NodeID(ctx), id, leaseHolder); err != nil {
|
||||
return LeaseRenewal{}, fmt.Errorf("failed to heartbeat issue: %w", err)
|
||||
}
|
||||
return LeaseRenewal{HeartbeatAt: reported, ExpiresAt: reported.Add(ttl)}, nil
|
||||
case lerr != nil && !errors.Is(lerr, sql.ErrNoRows):
|
||||
return LeaseRenewal{}, fmt.Errorf("read lease for %s: %w", id, lerr)
|
||||
}
|
||||
// No lease row the caller holds: grant one. THIS IS A MUTATION,
|
||||
// recorded as every bead mutator records one: row_lock reminted,
|
||||
// updated_at stamped, a lease_rearmed event, a journal entry and a
|
||||
// version row (the legs then make the version-control commit and
|
||||
// fire the update hook, keyed on LeaseRenewal.Rearmed). Why the
|
||||
// remint is load-bearing: The live claim this branch acts on was read in
|
||||
// THIS transaction's snapshot, but a snapshot cannot see a release
|
||||
// or close committing concurrently, and a lease INSERT shares no
|
||||
// row with such an ender (its DELETE FROM leases matched nothing in
|
||||
// its own snapshot): Dolt would commit both and leave an orphaned
|
||||
// lease on an open or closed issue. Every claim-ending path
|
||||
// rewrites row_lock (the freshRowLock invariant), so reminting it
|
||||
// here makes the grant COLLIDE with any concurrent ender, and the
|
||||
// loser is replayed against the winner's state: either the ender
|
||||
// sees and deletes this lease, or this heartbeat sees the ended
|
||||
// claim and is refused. Concurrent re-arms collide with each other
|
||||
// the same way, so exactly one reports Rearmed.
|
||||
//
|
||||
// THE COST, accepted: a re-arm (a hand-doled claim's first
|
||||
// heartbeat, once) remints the holder's revision token, so a
|
||||
// guarded write composed before it fails once and re-reads. A
|
||||
// REFRESH never touches the issue row and records nothing.
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE issues SET row_lock = ?, updated_at = ? WHERE id = ?`,
|
||||
freshRowLock(), now, id); err != nil {
|
||||
return LeaseRenewal{}, fmt.Errorf("remint row_lock for %s: %w", id, err)
|
||||
}
|
||||
if err := UpsertLeaseInTx(ctx, tx, id, actor, now, ttl); err != nil {
|
||||
return LeaseRenewal{}, err
|
||||
}
|
||||
if err := RecordFullEventInTable(ctx, tx, "events", id, types.EventLeaseRearmed, actor, "", actor); err != nil {
|
||||
return LeaseRenewal{}, fmt.Errorf("record lease re-arm event for %s: %w", id, err)
|
||||
}
|
||||
if err := RecordEventInTx(ctx, tx, EventUpdate, id, actor); err != nil {
|
||||
return LeaseRenewal{}, err
|
||||
}
|
||||
if err := RecordVersionInTx(ctx, tx, id, actor); err != nil {
|
||||
return LeaseRenewal{}, err
|
||||
}
|
||||
return LeaseRenewal{HeartbeatAt: reported, ExpiresAt: reported.Add(ttl), Rearmed: true}, nil
|
||||
}
|
||||
return fmt.Errorf("%w: %s status %s", storage.ErrNotClaimable, id, status)
|
||||
return LeaseRenewal{}, refused(fmt.Errorf("%w: %s status %s", storage.ErrNotClaimable, id, status))
|
||||
}
|
||||
// DELIBERATELY NOT JOURNALED — do not add an emit here. A heartbeat is a
|
||||
// A REFRESH IS DELIBERATELY NOT JOURNALED — do not add an emit here (the
|
||||
// re-arm above is the one heartbeat path recorded as a mutation). A heartbeat is a
|
||||
// high-frequency lease keepalive: it writes only the clone-local `leases`
|
||||
// table (lease-liveness state), never a durable bead field. Journaling it
|
||||
// would put a full-snapshot write plus the shared seq-counter serialization
|
||||
|
|
@ -410,8 +666,8 @@ func HeartbeatIssueInTx(ctx context.Context, tx DBTX, id, actor string) error {
|
|||
// lease state lives on the working-set plane and expires on its own. Lease
|
||||
// RECLAIM is the opposite case and does journal: it clears assignee and
|
||||
// reverts status, which is durable bead state. The decision is pinned by
|
||||
// journalExemptMutations in journal_completeness_test.go.
|
||||
return nil
|
||||
// mutationEntryPoints' heartbeat note in journal_completeness_test.go.
|
||||
return LeaseRenewal{HeartbeatAt: reported, ExpiresAt: reported.Add(ttl)}, nil
|
||||
}
|
||||
|
||||
// warnReplica writes a replica-guard audit line to STDERR (never stdout —
|
||||
|
|
|
|||
|
|
@ -68,8 +68,9 @@ var sqlWriteKeywordRe = regexp.MustCompile(`(?i)\b(?:SET|VALUES|SELECT)\b`)
|
|||
// the token would clobber a concurrent whole-row CAS (ExpectedVersion) for
|
||||
// no content change. Analysis absorbed from gastownhall/beads#4682.
|
||||
// (The PR's other exemption, the lease heartbeat, is moot on main: since
|
||||
// bd-lrgn1 heartbeats live entirely in the ephemeral leases table and never
|
||||
// touch the issues row.)
|
||||
// bd-lrgn1 a heartbeat refresh lives entirely in the ephemeral leases
|
||||
// table and never touches the issues row, and a re-arm remints row_lock on
|
||||
// purpose; see the freshRowLock invariant.)
|
||||
// - "compaction_level = ": compaction apply/restore rewrites bookkeeping and
|
||||
// compacted body text only — cells the reclaim/close races don't care
|
||||
// about, exempted by name in the freshRowLock invariant. The assignment
|
||||
|
|
|
|||
|
|
@ -89,6 +89,12 @@ var versionCompositeMutations = []string{
|
|||
// does. Every write plumbing bottoms out in one of these; the structural
|
||||
// cross-check (TestEveryBeadMutatorMintsOrIsExempt) keeps the list complete.
|
||||
var versionedEntryPoints = []string{
|
||||
// the heartbeat family mints on its RE-ARM path only (recorded as a
|
||||
// mutation); a refresh writes only the clone-local leases table and mints
|
||||
// nothing.
|
||||
"RenewLeaseInTx",
|
||||
"HeartbeatIssueInTx",
|
||||
"ExecuteHeartbeatInTx",
|
||||
// create — the singular path mints in place; the batch path defers the
|
||||
// mint past PersistDependenciesWithOptionsResult so the first version
|
||||
// carries the creation-time edge set.
|
||||
|
|
@ -244,7 +250,6 @@ var versionNeverMints = map[string]string{
|
|||
"UpdateIssueIDInTx": "rename: out of contract by design",
|
||||
"ApplyCompactionInTx": "compaction bookkeeping: out of contract by design",
|
||||
"RestoreFromSnapshotInTx": "restore: Phase 3",
|
||||
"HeartbeatIssueInTx": "lease keepalive: writes only the clone-local leases table, never a durable bead field",
|
||||
"BootstrapInTx": "bootstrap: config and metadata tables only, no issue-plane row",
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -98,6 +98,14 @@ func versionedHistoryEnabled(tx DBTX) bool {
|
|||
return on
|
||||
}
|
||||
|
||||
// VersionedHistoryEnabledFor reports whether versioned history is active on
|
||||
// tx, the transaction ScopeVersionedHistoryTransaction bound. A commit path
|
||||
// that stages a NAMED set of tables uses it to add
|
||||
// VersionedHistoryStagedTables only when the transaction could have minted.
|
||||
func VersionedHistoryEnabledFor(tx DBTX) bool {
|
||||
return versionedHistoryEnabled(tx)
|
||||
}
|
||||
|
||||
// VersionedHistoryStagedTables names every table RecordVersionInTx writes, so
|
||||
// a staging path can add them to whatever it was already going to stage.
|
||||
//
|
||||
|
|
|
|||
|
|
@ -68,11 +68,11 @@ func roleAccessorNamesOf(surface reflect.Type) (names, unclassified []string) {
|
|||
}
|
||||
|
||||
// TestEveryStoreRoleAccessorIsClassified fails when DoltStorage hands out an
|
||||
// interface the census cannot place. Every one of the twenty-nine today is a
|
||||
// interface the census cannot place. Every one of the thirty today is a
|
||||
// facade role, so this costs nothing and closes the path where a role surface
|
||||
// grows a package and the census quietly stops covering it. (roleAccessorNames
|
||||
// is derived, so the count is prose and only this sentence goes stale — but it
|
||||
// does go stale, and the last three sweeps found it saying twenty-eight.)
|
||||
// does go stale, and the last four sweeps found it behind.)
|
||||
func TestEveryStoreRoleAccessorIsClassified(t *testing.T) {
|
||||
_, unclassified := roleAccessorNamesOf(reflect.TypeOf((*DoltStorage)(nil)).Elem())
|
||||
for _, accessor := range unclassified {
|
||||
|
|
@ -151,7 +151,7 @@ func assertRoleAccessorsAreDeclared(t *testing.T, decorator reflect.Type) {
|
|||
}
|
||||
|
||||
// roleAccessorStore is a DoltStorage whose only real methods are the
|
||||
// twenty-nine role accessors, each answering with a distinguishable sentinel
|
||||
// thirty role accessors, each answering with a distinguishable sentinel
|
||||
// so a test can tell a decorated surface from a passed-through one.
|
||||
type roleAccessorStore struct {
|
||||
DoltStorage
|
||||
|
|
@ -184,6 +184,7 @@ type roleAccessorStore struct {
|
|||
metadataCAS issueops.MetadataCAS
|
||||
releaser issueops.Releaser
|
||||
leaseReclaimer issueops.LeaseReclaimer
|
||||
heartbeater issueops.Heartbeater
|
||||
err error
|
||||
}
|
||||
|
||||
|
|
@ -218,6 +219,7 @@ func newRoleAccessorStore() *roleAccessorStore {
|
|||
metadataCAS: sentinel,
|
||||
releaser: sentinel,
|
||||
leaseReclaimer: sentinel,
|
||||
heartbeater: sentinel,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -291,8 +293,11 @@ func (s *roleAccessorStore) Releaser() (issueops.Releaser, error) {
|
|||
func (s *roleAccessorStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) {
|
||||
return s.leaseReclaimer, s.err
|
||||
}
|
||||
func (s *roleAccessorStore) Heartbeater() (issueops.Heartbeater, error) {
|
||||
return s.heartbeater, s.err
|
||||
}
|
||||
|
||||
// roleAccessorSentinel implements twenty-eight of the twenty-nine roles at
|
||||
// roleAccessorSentinel implements twenty-nine of the thirty roles at
|
||||
// once — every one but memoryops.Memories, whose List collides with
|
||||
// issueops.Reader.List and needs the second sentinel below.
|
||||
// Nothing calls its methods; identity is the whole point.
|
||||
|
|
@ -420,6 +425,9 @@ func (*roleAccessorSentinel) Release(context.Context, issueops.ReleaseRequest) (
|
|||
func (*roleAccessorSentinel) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) {
|
||||
return issueops.ReclaimResult{}, nil
|
||||
}
|
||||
func (*roleAccessorSentinel) Heartbeat(context.Context, issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
return issueops.HeartbeatResult{}, nil
|
||||
}
|
||||
|
||||
// memoryRoleSentinel is the memory role's sentinel, and the one role
|
||||
// memoryRoleSentinel is the remaining role's sentinel, and the one role
|
||||
|
|
@ -455,7 +463,7 @@ func (*memoryRoleSentinel) List(context.Context, memoryops.ListRequest) (memoryo
|
|||
// ReadyCounter, Querier and InitVerifier deliberately return the inner surface
|
||||
// unwrapped,
|
||||
// each in its own hook_*.go. The ones in that column that are NOT reads are
|
||||
// WorkspaceConfig, Memories, VersionReconciler, Bootstrapper, Sweeper and
|
||||
// WorkspaceConfig, Memories, VersionReconciler, Bootstrapper, Sweeper,
|
||||
// Deleter: a settings write changes the workspace rather than a bead, a
|
||||
// remembered insight names no bead and there is no on_remember to fire, a
|
||||
// version marker names no bead at all, a bootstrap lands on a workspace whose
|
||||
|
|
@ -502,6 +510,7 @@ func TestHookFiringStoreWrapsTheWriteRolesAndPassesTheReadsThrough(t *testing.T)
|
|||
{"Deleter", func() (any, error) { return store.Deleter() }, inner.deleter, false},
|
||||
{"Bootstrapper", func() (any, error) { return store.Bootstrapper() }, inner.bootstrapper, false},
|
||||
{"InitVerifier", func() (any, error) { return store.InitVerifier() }, inner.verifier, false},
|
||||
{"Heartbeater", func() (any, error) { return store.Heartbeater() }, inner.heartbeater, true},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
surface, err := test.got()
|
||||
|
|
@ -561,6 +570,7 @@ func TestHookFiringStoreRoleAccessorsPropagateInnerErrors(t *testing.T) {
|
|||
{"Deleter", func() (any, error) { return store.Deleter() }},
|
||||
{"Bootstrapper", func() (any, error) { return store.Bootstrapper() }},
|
||||
{"InitVerifier", func() (any, error) { return store.InitVerifier() }},
|
||||
{"Heartbeater", func() (any, error) { return store.Heartbeater() }},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := test.got(); !errors.Is(err, want) {
|
||||
|
|
|
|||
|
|
@ -387,6 +387,16 @@ type Storage interface {
|
|||
// assignee and status per reverted row, which is on_update, the same
|
||||
// event Releaser fires. See hook_lease_reclaimer.go.
|
||||
LeaseReclaimer() (issueops.LeaseReclaimer, error)
|
||||
// Heartbeater returns the lease-refresh surface for this store: push the
|
||||
// lease the caller holds on one issue forward by the store's TTL.
|
||||
//
|
||||
// It is a WRITE role and its hook decorator WRAPS: a refresh writes only
|
||||
// the ephemeral lease row and fires nothing, but a re-arm (a live claim
|
||||
// with no lease row, HeartbeatResult.Rearmed) is recorded as a mutation
|
||||
// and fires on_update once. That is why RoleFiresHooks answers true for
|
||||
// it, and a server must take it off a peeled store. See
|
||||
// hook_heartbeater.go.
|
||||
Heartbeater() (issueops.Heartbeater, error)
|
||||
|
||||
// MoleculeStepper returns the advance-a-molecule surface for this store:
|
||||
// find a molecule's next ready step after one closed and, on request,
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ go_library(
|
|||
"events_journal.go",
|
||||
"external_doltserver_provider.go",
|
||||
"graph_counter.go",
|
||||
"heartbeater.go",
|
||||
"importer.go",
|
||||
"init_verifier.go",
|
||||
"issue_claimer.go",
|
||||
|
|
@ -120,6 +121,7 @@ go_test(
|
|||
"expected_revision_contract_test.go",
|
||||
"external_doltserver_provider_test.go",
|
||||
"graph_counter_contract_test.go",
|
||||
"heartbeater_contract_test.go",
|
||||
"importer_contract_test.go",
|
||||
"importer_uow_test.go",
|
||||
"issue_claimer_test.go",
|
||||
|
|
|
|||
|
|
@ -5,9 +5,16 @@ import (
|
|||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
"github.com/cenkalti/backoff/v4"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/domain/db"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/internal/storage/schema"
|
||||
)
|
||||
|
||||
type doltServerTx struct {
|
||||
|
|
@ -80,6 +87,156 @@ func (t *doltServerTx) Commit(ctx context.Context, message string) error {
|
|||
return t.closeOpenTxAfterFailure(ctx, err)
|
||||
}
|
||||
|
||||
const (
|
||||
// scopedCommitMaxElapsed bounds the retries of CommitScoped's Dolt commit.
|
||||
// It is short for the reason the DoltStore's postTxCommitMaxElapsed is:
|
||||
// the failure is logged rather than returned, so a long fight for a commit
|
||||
// the next one on the branch carries anyway buys nothing.
|
||||
scopedCommitMaxElapsed = 5 * time.Second
|
||||
// scopedCommitGrace pads the detached context past the retry budget. The
|
||||
// budget only stops scheduling attempts; the deadline is what cancels one
|
||||
// already hung in flight.
|
||||
scopedCommitGrace = 2 * time.Second
|
||||
)
|
||||
|
||||
// CommitScoped commits the transaction and then makes a Dolt commit of
|
||||
// exactly tables (plus the versioned-history tables when this transaction
|
||||
// could have minted) named message: DOLT_ADD per table, then
|
||||
// DOLT_COMMIT('-m') — never '-Am', so another writer's uncommitted change
|
||||
// elsewhere in the working set is neither committed nor attributed to this
|
||||
// message. With nothing staged, or under a deferred version commit
|
||||
// (dolt.auto-commit batch/off), there is no Dolt commit.
|
||||
//
|
||||
// THE DOLT COMMIT RUNS AFTER THE SQL COMMIT, never inside the transaction:
|
||||
// the ordering contract of the DoltStore funnel (runIssueOperationTxWithMessage,
|
||||
// LatentLabsSpace/NEXUS#92). DOLT_ADD stages whole tables from the session's
|
||||
// transaction root, so staging inside the open transaction builds the Dolt
|
||||
// commit from its BEGIN-time snapshot and can write rows a concurrent writer
|
||||
// committed in the meantime back to their old values. Committed first, this
|
||||
// transaction's writes go through Dolt's commit-time merge with those
|
||||
// writers, and the DOLT_ADD on the same pinned session, now outside any
|
||||
// transaction, stages the merged state.
|
||||
//
|
||||
// A serialization failure of the SQL COMMIT is returned for the caller to redo
|
||||
// the whole unit of work, as Commit's is. Once the SQL COMMIT succeeds the
|
||||
// write is durable, so the Dolt commit after it never fails CommitScoped (see
|
||||
// commitStagedTables).
|
||||
func (t *doltServerTx) CommitScoped(ctx context.Context, tables []string, message string) error {
|
||||
if t.done {
|
||||
return errors.New("uow: commit: already done")
|
||||
}
|
||||
if message == "" || len(tables) == 0 || issueops.VersionCommitDeferred(ctx) {
|
||||
return t.Commit(ctx, "")
|
||||
}
|
||||
// Resolved before the commit, while the versioned-history scope is bound
|
||||
// to the pinned session.
|
||||
staged := append([]string(nil), tables...)
|
||||
if issueops.VersionedHistoryEnabledFor(t.conn) {
|
||||
for _, table := range issueops.VersionedHistoryStagedTables() {
|
||||
if !slices.Contains(staged, table) {
|
||||
staged = append(staged, table)
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := t.conn.ExecContext(ctx, "COMMIT;"); err != nil {
|
||||
if isSerializationError(err) {
|
||||
return err
|
||||
}
|
||||
return t.closeOpenTxAfterFailure(ctx, err)
|
||||
}
|
||||
t.done = true
|
||||
clean := t.commitStagedTables(ctx, staged, message)
|
||||
if t.recheckBlockedAfterCommit(ctx, issueops.TakeBlockedRecheck(t.conn)) && clean {
|
||||
t.releaseConn()
|
||||
} else {
|
||||
t.poisonConn()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// commitStagedTables is CommitScoped's Dolt commit, made on the pinned session
|
||||
// after the transaction committed. It runs detached from the caller's
|
||||
// cancellation on its own short budget, so a deadline landing after the SQL
|
||||
// COMMIT does not skip it, and replays only the conflicts that leave nothing
|
||||
// behind (serialization failures and Dolt's @@autocommit rollback, which is
|
||||
// how a conflict surfaces outside a transaction), so re-staging and
|
||||
// committing again is safe. That is doltAddAndCommitPostTx's retry.
|
||||
//
|
||||
// A failure is logged, never returned: the write is durable, and an error
|
||||
// would make RunTxScoped replay a committed unit of work, which for a
|
||||
// heartbeat re-arm then reports a refresh. The change rides the next Dolt
|
||||
// commit on the branch, as a failed post-transaction commit does in the
|
||||
// DoltStore funnel. Unlike the funnel, this leg counts nothing: uow records
|
||||
// no metrics, so bd.db.post_tx_commit_dropped, the funnel's counter for the
|
||||
// same drop, never sees one made here, and the log line is its only signal.
|
||||
// It reports whether the session may go back to the pool; after a failure its
|
||||
// state is unknown (a statement cut off mid-flight, or under @@autocommit=0
|
||||
// the implicit transaction a DOLT_ADD opened), so it may not.
|
||||
func (t *doltServerTx) commitStagedTables(ctx context.Context, tables []string, message string) bool {
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), scopedCommitMaxElapsed+scopedCommitGrace)
|
||||
defer cancel()
|
||||
|
||||
bo := backoff.NewExponentialBackOff()
|
||||
bo.InitialInterval = txRetryInitialInterval
|
||||
bo.MaxElapsedTime = scopedCommitMaxElapsed
|
||||
var lastErr error
|
||||
err := backoff.Retry(func() error {
|
||||
err := t.stageAndCommitOnce(ctx, tables, message)
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
lastErr = err
|
||||
if isSerializationError(err) || isDoltAutocommitRollbackError(err) {
|
||||
return err
|
||||
}
|
||||
return backoff.Permanent(err)
|
||||
}, backoff.WithContext(bo, ctx))
|
||||
if err != nil && lastErr != nil && !errors.Is(err, lastErr) {
|
||||
// The deadline expiring between attempts returns the bare context
|
||||
// error; keep the last real failure, the only diagnostic of a dropped
|
||||
// commit.
|
||||
err = fmt.Errorf("%w (last attempt: %w)", err, lastErr)
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf("warning: uow: dolt commit %q failed after its SQL commit (data already committed; change rides the next dolt commit): %v", message, err)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// stageAndCommitOnce is one attempt at commitStagedTables: stage each table,
|
||||
// then commit the staged set alone.
|
||||
func (t *doltServerTx) stageAndCommitOnce(ctx context.Context, tables []string, message string) error {
|
||||
for _, table := range tables {
|
||||
if err := schema.DrainCall(ctx, t.conn, "CALL DOLT_ADD(?)", table); err != nil {
|
||||
return fmt.Errorf("dolt add %s: %w", table, err)
|
||||
}
|
||||
}
|
||||
// Sessions on one branch share the working set, so a concurrent writer's
|
||||
// Dolt commit can already have taken this transaction's rows under its
|
||||
// own message, leaving nothing staged here: skip the empty commit Dolt
|
||||
// would refuse. The same race between this check and the commit surfaces
|
||||
// as "nothing to commit", which here means exactly that and not a lost
|
||||
// write (the SQL COMMIT has landed), so it is no failure either; it is
|
||||
// logged, as the DoltStore funnel logs it, so the missing audit line is
|
||||
// explicable.
|
||||
staged, err := issueops.HasStagedChanges(ctx, t.conn)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !staged {
|
||||
return nil
|
||||
}
|
||||
err = schema.DrainCall(ctx, t.conn, "CALL DOLT_COMMIT('-m', ?)", message)
|
||||
if err != nil {
|
||||
if !issueops.IsNothingToCommitError(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
log.Printf("uow: dolt commit %q made no dolt commit (nothing to commit): the staged change was most likely absorbed into a concurrent writer's commit (data already committed)", message)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// commitStatement picks the statement that ends the open transaction on the
|
||||
// pinned session for a commit named message.
|
||||
func (t *doltServerTx) commitStatement(ctx context.Context, message string) (string, []interface{}, error) {
|
||||
|
|
@ -95,12 +252,15 @@ func (t *doltServerTx) commitStatement(ctx context.Context, message string) (str
|
|||
}
|
||||
// An empty message selects the EPHEMERAL commit form (bd-aq0ql): a plain
|
||||
// SQL COMMIT persists the transaction's writes into the working set
|
||||
// without minting a Dolt commit or history. It has two entry points:
|
||||
// without minting a Dolt commit or history. It has these entry points:
|
||||
//
|
||||
// - uow.RunTxEphemeral, for work that touches ONLY dolt_ignored state —
|
||||
// today the leases table (bd-lrgn1), whose heartbeats must never create
|
||||
// commits. RunTx/RunTxResult treat an empty commitMsg as "nothing to
|
||||
// commit" and never call Commit at all, so they cannot reach it that way.
|
||||
// - uow.RunTxEphemeral, for work that touches ONLY dolt_ignored state.
|
||||
// RunTx/RunTxResult treat an empty commitMsg as "nothing to commit" and
|
||||
// never call Commit at all, so they cannot reach it that way.
|
||||
// - uow.RunTxScoped, for an attempt that names no tables — a lease
|
||||
// refresh (bd-lrgn1), which writes only the dolt_ignored leases table
|
||||
// and must never create a commit. A heartbeat that re-arms names its
|
||||
// tables instead and takes the scoped commit.
|
||||
// - RunTx/RunTxResult under a deferred-version-commit context, via the
|
||||
// blanking just above (GH#4995). There the writes are ordinary versioned
|
||||
// rows and the ONLY-dolt_ignored-state rationale does not apply: the
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ import (
|
|||
"testing"
|
||||
|
||||
"github.com/DATA-DOG/go-sqlmock"
|
||||
mysql "github.com/go-sql-driver/mysql"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
|
@ -285,3 +286,134 @@ func TestDoltServerTxRunTxWithDeferredContextSkipsDoltCommit(t *testing.T) {
|
|||
require.NoError(t, mock.ExpectationsWereMet())
|
||||
assert.Equal(t, 1, p.db.Stats().OpenConnections)
|
||||
}
|
||||
|
||||
// matchHasStaged matches the staged-set check issueops.HasStagedChanges runs
|
||||
// before CommitScoped's DOLT_COMMIT('-m').
|
||||
const matchHasStaged = `SELECT COUNT\(\*\) FROM dolt_status WHERE staged = 1`
|
||||
|
||||
// expectScopedDoltCommit arms one attempt of CommitScoped's Dolt commit: a
|
||||
// DOLT_ADD per table, the staged-set check reporting staged rows, and, when
|
||||
// something is staged, a DOLT_COMMIT('-m') answering commitErr.
|
||||
func expectScopedDoltCommit(mock sqlmock.Sqlmock, tables []string, staged int, message string, commitErr error) {
|
||||
for _, table := range tables {
|
||||
mock.ExpectQuery(`CALL DOLT_ADD\(\?\)`).WithArgs(table).
|
||||
WillReturnRows(sqlmock.NewRows([]string{"status"}).AddRow(0))
|
||||
}
|
||||
mock.ExpectQuery(matchHasStaged).WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(staged))
|
||||
if staged == 0 {
|
||||
return
|
||||
}
|
||||
commit := mock.ExpectQuery(`CALL DOLT_COMMIT\('-m', \?\)`).WithArgs(message)
|
||||
if commitErr != nil {
|
||||
commit.WillReturnError(commitErr)
|
||||
} else {
|
||||
commit.WillReturnRows(sqlmock.NewRows([]string{"hash"}).AddRow("c0ffee"))
|
||||
}
|
||||
}
|
||||
|
||||
// TestDoltServerTxCommitScopedCommitsSQLBeforeDoltCommit pins the
|
||||
// LatentLabsSpace/NEXUS#92 ordering on the uow leg: CommitScoped ends the SQL
|
||||
// transaction with a plain COMMIT and only then stages and Dolt-commits its
|
||||
// tables, on the same pinned session. Staged inside the open transaction,
|
||||
// DOLT_ADD builds the Dolt commit from the transaction's start snapshot and
|
||||
// writes a concurrent writer's committed rows back to their old values.
|
||||
func TestDoltServerTxCommitScopedCommitsSQLBeforeDoltCommit(t *testing.T) {
|
||||
p, mock := newMockTxProvider(t)
|
||||
mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
mock.ExpectExec(matchPlainCommit).WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
expectScopedDoltCommit(mock, []string{"events", "issues"}, 2, "bd: re-arm lease x", nil)
|
||||
|
||||
tx, err := p.BeginTx(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, tx.(ScopedCommitter).CommitScoped(context.Background(), []string{"events", "issues"}, "bd: re-arm lease x"))
|
||||
require.NoError(t, mock.ExpectationsWereMet(), "the SQL COMMIT must come first, then DOLT_ADD per table and DOLT_COMMIT('-m')")
|
||||
assert.Equal(t, 1, p.db.Stats().OpenConnections, "a cleanly committed session may return to the pool")
|
||||
require.ErrorContains(t, tx.Commit(context.Background(), ""), "already done")
|
||||
}
|
||||
|
||||
// TestDoltServerTxCommitScopedSwallowsAFailedDoltCommit: once the SQL COMMIT
|
||||
// has landed the write is durable, so a Dolt commit that fails after it must
|
||||
// not fail CommitScoped (RunTxScoped would replay a committed unit of work).
|
||||
// The session, whose state the failed statement leaves unknown, is discarded.
|
||||
func TestDoltServerTxCommitScopedSwallowsAFailedDoltCommit(t *testing.T) {
|
||||
p, mock := newMockTxProvider(t)
|
||||
mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
mock.ExpectExec(matchPlainCommit).WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
mock.ExpectQuery(`CALL DOLT_ADD\(\?\)`).WithArgs("issues").WillReturnError(errors.New("dolt add exploded"))
|
||||
|
||||
tx, err := p.BeginTx(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, tx.(ScopedCommitter).CommitScoped(context.Background(), []string{"issues"}, "bd: re-arm lease x"),
|
||||
"a committed write must not be reported as failed")
|
||||
require.NoError(t, mock.ExpectationsWereMet(), "a failure other than a conflict is not retried")
|
||||
assert.Equal(t, 0, p.db.Stats().OpenConnections, "the session must be discarded, not pooled")
|
||||
require.ErrorContains(t, tx.Commit(context.Background(), ""), "already done")
|
||||
}
|
||||
|
||||
// TestDoltServerTxCommitScopedRetriesAConflictedDoltCommit: outside a
|
||||
// transaction, a Dolt commit that conflicts with a concurrent writer is rolled
|
||||
// back as an @@autocommit merge conflict (1105), not 1213, and is retried like
|
||||
// one.
|
||||
func TestDoltServerTxCommitScopedRetriesAConflictedDoltCommit(t *testing.T) {
|
||||
p, mock := newMockTxProvider(t)
|
||||
mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
mock.ExpectExec(matchPlainCommit).WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
conflict := &mysql.MySQLError{Number: 1105, Message: "Merge conflict detected, @autocommit transaction rolled back"}
|
||||
expectScopedDoltCommit(mock, []string{"issues"}, 1, "bd: re-arm lease x", conflict)
|
||||
expectScopedDoltCommit(mock, []string{"issues"}, 1, "bd: re-arm lease x", nil)
|
||||
|
||||
tx, err := p.BeginTx(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, tx.(ScopedCommitter).CommitScoped(context.Background(), []string{"issues"}, "bd: re-arm lease x"))
|
||||
require.NoError(t, mock.ExpectationsWereMet(), "the conflicted attempt must be staged and committed again")
|
||||
assert.Equal(t, 1, p.db.Stats().OpenConnections, "a session whose retry committed may return to the pool")
|
||||
}
|
||||
|
||||
// TestDoltServerTxCommitScopedReturnsASerializationFailure: a serialization
|
||||
// failure of the SQL COMMIT is returned for RunTxScoped to redo the whole unit
|
||||
// of work, and no Dolt commit follows a COMMIT that did not land.
|
||||
func TestDoltServerTxCommitScopedReturnsASerializationFailure(t *testing.T) {
|
||||
p, mock := newMockTxProvider(t)
|
||||
mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
mock.ExpectExec(matchPlainCommit).WillReturnError(&mysql.MySQLError{Number: 1213, Message: "serialization failure"})
|
||||
|
||||
tx, err := p.BeginTx(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
err = tx.(ScopedCommitter).CommitScoped(context.Background(), []string{"issues"}, "bd: re-arm lease x")
|
||||
require.True(t, isSerializationError(err), "err = %v, want the serialization failure", err)
|
||||
require.NoError(t, mock.ExpectationsWereMet(), "nothing may be staged or committed after a failed COMMIT")
|
||||
}
|
||||
|
||||
// TestDoltServerTxCommitScopedMakesNoEmptyDoltCommit: a concurrent writer's
|
||||
// Dolt commit can take this transaction's rows before its own Dolt commit
|
||||
// runs, seen either before the commit (nothing staged) or by it ("nothing to
|
||||
// commit"). Neither is a failure: unlike Commit's in-transaction commit, the
|
||||
// rows are already committed.
|
||||
func TestDoltServerTxCommitScopedMakesNoEmptyDoltCommit(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
staged int
|
||||
commitErr error
|
||||
}{
|
||||
{name: "nothing staged", staged: 0},
|
||||
{name: "nothing to commit", staged: 1, commitErr: errors.New("nothing to commit")},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
p, mock := newMockTxProvider(t)
|
||||
mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
mock.ExpectExec(matchPlainCommit).WillReturnResult(sqlmock.NewResult(0, 0))
|
||||
expectScopedDoltCommit(mock, []string{"issues"}, tc.staged, "bd: re-arm lease x", tc.commitErr)
|
||||
|
||||
tx, err := p.BeginTx(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, tx.(ScopedCommitter).CommitScoped(context.Background(), []string{"issues"}, "bd: re-arm lease x"))
|
||||
require.NoError(t, mock.ExpectationsWereMet())
|
||||
assert.Equal(t, 1, p.db.Stats().OpenConnections, "the session is clean and may return to the pool")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,6 +33,21 @@ func isSerializationError(err error) bool {
|
|||
return false
|
||||
}
|
||||
|
||||
// isDoltAutocommitRollbackError reports whether err is Dolt rolling back a
|
||||
// statement it ran under @@autocommit because the statement's commit-time
|
||||
// merge conflicted with a concurrent writer (MySQL 1105, "Merge conflict
|
||||
// detected, @autocommit transaction rolled back"). Outside an explicit
|
||||
// transaction this, not 1213, is how such a conflict surfaces, and like a
|
||||
// serialization failure it guarantees the statement left nothing behind. It
|
||||
// mirrors the DoltStore's classifier of the same name.
|
||||
func isDoltAutocommitRollbackError(err error) bool {
|
||||
var mysqlErr *mysql.MySQLError
|
||||
return errors.As(err, &mysqlErr) &&
|
||||
mysqlErr.Number == 1105 &&
|
||||
(mysqlErr.Message == "Merge conflict detected, @autocommit transaction rolled back" ||
|
||||
strings.HasPrefix(mysqlErr.Message, "Merge conflict detected, @autocommit transaction rolled back."))
|
||||
}
|
||||
|
||||
// isDatabaseExistsError reports whether err is the server refusing a bare
|
||||
// CREATE DATABASE because the database already exists (MySQL 1007,
|
||||
// ER_DB_CREATE_EXISTS). The message fallback matches how the rest of the
|
||||
|
|
|
|||
|
|
@ -59,6 +59,37 @@ func TestIsDatabaseNotFoundError(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// The DoltStore's table for its twin classifier, plus the prefix boundary:
|
||||
// commitStagedTables replays exactly what this accepts, so a false positive
|
||||
// re-runs a Dolt commit that failed for another reason.
|
||||
func TestIsDoltAutocommitRollbackError(t *testing.T) {
|
||||
exact := &mysql.MySQLError{
|
||||
Number: 1105,
|
||||
Message: "Merge conflict detected, @autocommit transaction rolled back",
|
||||
}
|
||||
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
err error
|
||||
want bool
|
||||
}{
|
||||
{name: "nil", err: nil, want: false},
|
||||
{name: "exact typed Dolt rollback", err: fmt.Errorf("dolt commit: %w", exact), want: true},
|
||||
{name: "typed Dolt rollback with recovery guidance", err: &mysql.MySQLError{Number: 1105, Message: exact.Message + ". @autocommit must be disabled"}, want: true},
|
||||
{name: "typed text running on past the message", err: &mysql.MySQLError{Number: 1105, Message: exact.Message + "ward"}, want: false},
|
||||
{name: "untyped matching text", err: errors.New(exact.Error()), want: false},
|
||||
{name: "other typed 1105", err: &mysql.MySQLError{Number: 1105, Message: "Merge conflict detected"}, want: false},
|
||||
{name: "typed connection-like 1105", err: &mysql.MySQLError{Number: 1105, Message: "connection lost while validating commit"}, want: false},
|
||||
{name: "same text wrong code", err: &mysql.MySQLError{Number: 1213, Message: exact.Message}, want: false},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := isDoltAutocommitRollbackError(tt.err); got != tt.want {
|
||||
t.Fatalf("isDoltAutocommitRollbackError(%v) = %t, want %t", tt.err, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyUseDatabaseError(t *testing.T) {
|
||||
// The exact server errors observed from dolt 2.1.8 against a credential
|
||||
// granted only one database.
|
||||
|
|
|
|||
84
internal/storage/uow/heartbeater.go
Normal file
84
internal/storage/uow/heartbeater.go
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
package uow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
storageissueops "github.com/steveyegge/beads/internal/storage/issueops"
|
||||
publicops "github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// HeartbeaterSource is the capability accessor a unit-of-work provider offers
|
||||
// for the lease-refresh role, the sibling of LeaseReclaimerSource.
|
||||
type HeartbeaterSource interface {
|
||||
Heartbeater() (publicops.Heartbeater, error)
|
||||
}
|
||||
|
||||
// heartbeater refreshes a held lease through a unit of work.
|
||||
type heartbeater struct {
|
||||
provider UnitOfWorkProvider
|
||||
}
|
||||
|
||||
// Heartbeater returns the lease-refresh surface for this provider.
|
||||
func (p *doltSQLProvider) Heartbeater() (publicops.Heartbeater, error) {
|
||||
return NewHeartbeater(p)
|
||||
}
|
||||
|
||||
// NewHeartbeater constructs a public Heartbeater backed by provider.
|
||||
func NewHeartbeater(provider UnitOfWorkProvider) (publicops.Heartbeater, error) {
|
||||
if isNilUnitOfWorkProvider(provider) {
|
||||
return nil, fmt.Errorf("new heartbeater: unit-of-work provider must not be nil")
|
||||
}
|
||||
return &heartbeater{provider: provider}, nil
|
||||
}
|
||||
|
||||
var _ publicops.Heartbeater = (*heartbeater)(nil)
|
||||
|
||||
// Heartbeat reaches domain.IssueUseCase.RenewLease inside ONE unit of work. A
|
||||
// refresh touches only the dolt_ignored leases table, so it mints no Dolt
|
||||
// commit and records nothing in dolt_log; a re-arm is recorded as a mutation
|
||||
// and commits like one. A serialization loser (a heartbeat racing a reclaim,
|
||||
// release or close) is replayed against a fresh unit of work.
|
||||
//
|
||||
// VALIDATION HAPPENS BEFORE THE UNIT OF WORK OPENS, as on the dolt and
|
||||
// embedded-dolt legs: a refusal then costs no connection. The shared body
|
||||
// validates again inside the transaction.
|
||||
func (h *heartbeater) Heartbeat(ctx context.Context, request publicops.HeartbeatRequest) (publicops.HeartbeatResult, error) {
|
||||
if err := storageissueops.ValidateHeartbeatRequest(request); err != nil {
|
||||
return publicops.HeartbeatResult{}, err
|
||||
}
|
||||
// A lease refusal commits (it may have deleted an orphaned lease row) and
|
||||
// is reported after the commit; see HeartbeatRefusalCommits.
|
||||
//
|
||||
// THE COMMIT IS SCOPED (RunTxScoped). A refresh or a refusal writes only
|
||||
// the dolt_ignored leases table and commits with the plain SQL COMMIT: no
|
||||
// working-set scan, no Dolt commit, and so nothing another writer left
|
||||
// uncommitted (a batch-mode edit) is swept into a heartbeat commit. A
|
||||
// re-arm is recorded as a mutation and commits exactly the tables
|
||||
// LeaseRearmVersionCommit names, never DOLT_COMMIT('-Am'), in a Dolt
|
||||
// commit made after the SQL COMMIT (ScopedCommitter): built inside the
|
||||
// transaction, it would write an issues row a concurrent writer changed
|
||||
// meanwhile back to its value at this heartbeat's start.
|
||||
type outcome struct {
|
||||
result publicops.HeartbeatResult
|
||||
refusal error
|
||||
}
|
||||
out, err := RunTxScoped(ctx, h.provider, func(ctx context.Context, uw UnitOfWork) (outcome, []string, string, error) {
|
||||
result, err := uw.IssueUseCase().RenewLease(ctx, request)
|
||||
if storageissueops.HeartbeatRefusalCommits(err) {
|
||||
return outcome{refusal: err}, nil, "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return outcome{}, nil, "", err
|
||||
}
|
||||
tables, msg := storageissueops.LeaseRearmVersionCommit(request.IssueID, result.Rearmed)
|
||||
return outcome{result: result}, tables.Sorted(), msg, nil
|
||||
})
|
||||
if err != nil {
|
||||
return publicops.HeartbeatResult{}, err
|
||||
}
|
||||
if out.refusal != nil {
|
||||
return publicops.HeartbeatResult{}, out.refusal
|
||||
}
|
||||
return out.result, nil
|
||||
}
|
||||
339
internal/storage/uow/heartbeater_contract_test.go
Normal file
339
internal/storage/uow/heartbeater_contract_test.go
Normal file
|
|
@ -0,0 +1,339 @@
|
|||
package uow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/backend/conformance"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
publicops "github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// TestHeartbeaterContract runs the Heartbeater contract against the
|
||||
// unit-of-work provider, which reaches internal/storage/issueops.
|
||||
// ExecuteHeartbeatInTx through domain.IssueUseCase.RenewLease inside one unit
|
||||
// of work committed with RunTxScoped: the SQL-only form for a refresh, the
|
||||
// scoped commit for a re-arm.
|
||||
func TestHeartbeaterContract(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
fixture := newUOWHeartbeaterFixture(t, ctx, "hbc")
|
||||
t.Run("RefreshesTheHoldersLease", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefreshesTheHoldersLease(t, ctx, fixture)
|
||||
})
|
||||
t.Run("ChangesNothingOnTheRow", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterChangesNothingOnTheRow(t, ctx, fixture)
|
||||
})
|
||||
t.Run("AGuardedWriteWithThePreHeartbeatRevisionStillWins", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterAGuardedWriteWithThePreHeartbeatRevisionStillWins(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAForeignHolder", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAForeignHolder(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterClose", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterClose(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterRelease", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterRelease(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterReclaim", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterReclaim(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAfterStatusMovedOffInProgress", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAfterStatusMovedOffInProgress(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAWisp", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAWisp(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RearmsAHandDoledClaim", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRearmsAHandDoledClaim(t, ctx, fixture)
|
||||
})
|
||||
t.Run("AnswersNotFoundForAnAbsentID", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterAnswersNotFoundForAnAbsentID(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesABlankOrOverLongActor", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesABlankOrOverLongActor(t, ctx, fixture)
|
||||
})
|
||||
t.Run("DoesNotMutateTheRequest", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterDoesNotMutateTheRequest(t, ctx, fixture)
|
||||
})
|
||||
t.Run("CommitsOnlyItsOwnTables", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterCommitsOnlyItsOwnTables(t, ctx, fixture)
|
||||
})
|
||||
t.Run("ASameSecondHeartbeatIsNotARearm", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterASameSecondHeartbeatIsNotARearm(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAReleasedIssue", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAReleasedIssue(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAClosedIssue", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAClosedIssue(t, ctx, fixture)
|
||||
})
|
||||
t.Run("RefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds", func(t *testing.T) {
|
||||
conformance.RunHeartbeaterRefusesAnOrphanedLeaseOnAnIssueSomeoneElseHolds(t, ctx, fixture)
|
||||
})
|
||||
}
|
||||
|
||||
// TestHeartbeaterRearmKeepsAConcurrentWrite is the LatentLabsSpace/NEXUS#92
|
||||
// regression for the re-arm's scoped commit: another session commits a change
|
||||
// to a different issues row while the re-arm's transaction is open, and the
|
||||
// Dolt commit the re-arm makes must stage issues as committed, with that
|
||||
// change in it, not as the re-arm's start snapshot. DOLT_ADD stages a whole
|
||||
// table from the session's transaction root, so staging inside the open
|
||||
// transaction did the latter: HEAD then disagreed with the rows every reader
|
||||
// sees, for a write still only in the working set (batch mode, or the window
|
||||
// before the writer's own Dolt commit). ScopedCommitter commits the SQL
|
||||
// transaction first.
|
||||
func TestHeartbeaterRearmKeepsAConcurrentWrite(t *testing.T) {
|
||||
ctx := t.Context()
|
||||
const prefix = "hbw"
|
||||
provider := newUOWRoleFixtureProvider(t, ctx, prefix)
|
||||
kit := newUOWRoleFixtureKit(provider, prefix)
|
||||
lifecycle, err := provider.(IssueLifecycleSource).IssueLifecycle()
|
||||
if err != nil {
|
||||
t.Fatalf("IssueLifecycle(): %v", err)
|
||||
}
|
||||
const holder = "hb-holder"
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
write func(ctx context.Context, id, title string) error
|
||||
}{
|
||||
{"a write with its own dolt commit", func(ctx context.Context, id, title string) error {
|
||||
return RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) {
|
||||
_, err := uw.RawSQLUseCase().Exec(ctx, "UPDATE issues SET title = ? WHERE id = ?", title, id)
|
||||
return "bd: retitle " + id, err
|
||||
})
|
||||
}},
|
||||
{"a write only in the working set", func(ctx context.Context, id, title string) error {
|
||||
_, err := RunTxEphemeral(ctx, provider, func(ctx context.Context, uw UnitOfWork) (struct{}, error) {
|
||||
_, err := uw.RawSQLUseCase().Exec(ctx, "UPDATE issues SET title = ? WHERE id = ?", title, id)
|
||||
return struct{}{}, err
|
||||
})
|
||||
return err
|
||||
}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
suffix := strings.ReplaceAll(tc.name, " ", "-")
|
||||
doled, bystander := prefix+"-doled-"+suffix, prefix+"-bystander-"+suffix
|
||||
for _, id := range []string{doled, bystander} {
|
||||
issue := &types.Issue{ID: id, Title: "seed " + id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}
|
||||
if err := kit.CreateIssue(ctx, issue, "seeder"); err != nil {
|
||||
t.Fatalf("seed %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
if _, err := lifecycle.Update(ctx, publicops.UpdateRequest{
|
||||
Actor: holder, IssueID: doled,
|
||||
Patch: publicops.IssuePatch{
|
||||
Status: publicops.Field[publicops.Status]{Set: true, Value: types.StatusInProgress},
|
||||
Assignee: publicops.Field[string]{Set: true, Value: holder},
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("hand-dole %s: %v", doled, err)
|
||||
}
|
||||
|
||||
const retitled = "retitled while the re-arm was open"
|
||||
wrote := false
|
||||
heartbeater, err := NewHeartbeater(&beforeScopedCommitProvider{
|
||||
UnitOfWorkProvider: provider,
|
||||
before: func(ctx context.Context) error {
|
||||
if wrote {
|
||||
return nil
|
||||
}
|
||||
wrote = true
|
||||
return tc.write(ctx, bystander, retitled)
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewHeartbeater: %v", err)
|
||||
}
|
||||
result, err := heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: holder, IssueID: doled})
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeat: %v", err)
|
||||
}
|
||||
if !result.Rearmed || !wrote {
|
||||
t.Fatalf("result = %+v, concurrent write ran = %v: the case needs a re-arm with a write inside its window", result, wrote)
|
||||
}
|
||||
|
||||
for _, at := range []string{"the working set", "HEAD"} {
|
||||
query := "SELECT title FROM issues WHERE id = ?"
|
||||
if at == "HEAD" {
|
||||
query = "SELECT title FROM issues AS OF 'HEAD' WHERE id = ?"
|
||||
}
|
||||
var title string
|
||||
if err := kit.QueryScalar(ctx, query, []any{bystander}, &title); err != nil {
|
||||
t.Fatalf("read %s title at %s: %v", bystander, at, err)
|
||||
}
|
||||
if title != retitled {
|
||||
t.Errorf("%s title at %s = %q, want %q: the re-arm's Dolt commit staged issues from its start snapshot", bystander, at, title, retitled)
|
||||
}
|
||||
}
|
||||
// The re-arm itself is committed: its reminted row_lock is at HEAD
|
||||
// (events is dolt_ignored, so the issues row is what its commit
|
||||
// carries).
|
||||
var working, head int64
|
||||
if err := kit.QueryScalar(ctx, "SELECT row_lock FROM issues WHERE id = ?", []any{doled}, &working); err != nil {
|
||||
t.Fatalf("read %s row_lock: %v", doled, err)
|
||||
}
|
||||
if err := kit.QueryScalar(ctx, "SELECT row_lock FROM issues AS OF 'HEAD' WHERE id = ?", []any{doled}, &head); err != nil {
|
||||
t.Fatalf("read %s row_lock at HEAD: %v", doled, err)
|
||||
}
|
||||
var commits int
|
||||
if err := kit.QueryScalar(ctx, "SELECT COUNT(*) FROM dolt_log WHERE message = ?", []any{"bd: re-arm lease " + doled}, &commits); err != nil {
|
||||
t.Fatalf("count re-arm commits: %v", err)
|
||||
}
|
||||
if head != working || commits != 1 {
|
||||
t.Errorf("%s row_lock at HEAD = %d (working set %d), re-arm commits = %d, want the re-arm in one Dolt commit", doled, head, working, commits)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// beforeScopedCommitProvider runs before ahead of each scoped commit of the
|
||||
// units of work it opens: a way into the window between a transaction's
|
||||
// writes and its commit.
|
||||
type beforeScopedCommitProvider struct {
|
||||
UnitOfWorkProvider
|
||||
before func(context.Context) error
|
||||
}
|
||||
|
||||
func (p *beforeScopedCommitProvider) NewUOW(ctx context.Context) (UnitOfWork, error) {
|
||||
uw, err := p.UnitOfWorkProvider.NewUOW(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &beforeScopedCommitUOW{UnitOfWork: uw, before: p.before}, nil
|
||||
}
|
||||
|
||||
type beforeScopedCommitUOW struct {
|
||||
UnitOfWork
|
||||
before func(context.Context) error
|
||||
}
|
||||
|
||||
func (u *beforeScopedCommitUOW) Unwrap() UnitOfWork { return u.UnitOfWork }
|
||||
|
||||
func (u *beforeScopedCommitUOW) CommitScoped(ctx context.Context, tables []string, message string) error {
|
||||
if err := u.before(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
return commitScoped(ctx, u.UnitOfWork, tables, message)
|
||||
}
|
||||
|
||||
// TestHeartbeaterRefusesBeforeOpeningAUOW: a malformed heartbeat is a
|
||||
// deterministic validation failure and must not cost a database connection to
|
||||
// discover, as on the dolt and embedded-dolt legs.
|
||||
func TestHeartbeaterRefusesBeforeOpeningAUOW(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
request publicops.HeartbeatRequest
|
||||
}{
|
||||
{"no actor", publicops.HeartbeatRequest{IssueID: "x"}},
|
||||
{"no id", publicops.HeartbeatRequest{Actor: "worker"}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
provider := &mockUnitOfWorkProvider{newUOWErr: errors.New("unexpected unit-of-work open")}
|
||||
heartbeater, err := NewHeartbeater(provider)
|
||||
if err != nil {
|
||||
t.Fatalf("NewHeartbeater: %v", err)
|
||||
}
|
||||
if _, err := heartbeater.Heartbeat(context.Background(), tc.request); !errors.Is(err, publicops.ErrValidation) {
|
||||
t.Fatalf("err = %v, want ErrValidation", err)
|
||||
}
|
||||
if provider.newUOWCalls != 0 {
|
||||
t.Errorf("an invalid request opened %d units of work", provider.newUOWCalls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// newUOWHeartbeaterFixture takes every role through the provider's own
|
||||
// capability accessors, and the hooked heartbeater off the notifying provider
|
||||
// the proxied CLI route wraps around its provider.
|
||||
func newUOWHeartbeaterFixture(t *testing.T, ctx context.Context, prefix string) conformance.HeartbeaterFixture {
|
||||
t.Helper()
|
||||
provider := newUOWRoleFixtureProvider(t, ctx, prefix)
|
||||
source, ok := provider.(HeartbeaterSource)
|
||||
if !ok {
|
||||
t.Fatalf("provider %T does not offer the Heartbeater accessor", provider)
|
||||
}
|
||||
heartbeater, err := source.Heartbeater()
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeater(): %v", err)
|
||||
}
|
||||
claimer, err := provider.(IssueClaimerSource).IssueClaimer()
|
||||
if err != nil {
|
||||
t.Fatalf("IssueClaimer(): %v", err)
|
||||
}
|
||||
lifecycle, err := provider.(IssueLifecycleSource).IssueLifecycle()
|
||||
if err != nil {
|
||||
t.Fatalf("IssueLifecycle(): %v", err)
|
||||
}
|
||||
releaser, err := provider.(ReleaserSource).Releaser()
|
||||
if err != nil {
|
||||
t.Fatalf("Releaser(): %v", err)
|
||||
}
|
||||
reclaimer, err := provider.(LeaseReclaimerSource).LeaseReclaimer()
|
||||
if err != nil {
|
||||
t.Fatalf("LeaseReclaimer(): %v", err)
|
||||
}
|
||||
configurer, ok := provider.(storage.EventsJournalConfigurer)
|
||||
if !ok {
|
||||
t.Fatalf("provider %T does not implement storage.EventsJournalConfigurer", provider)
|
||||
}
|
||||
versionConfigurer, ok := provider.(storage.VersionedHistoryConfigurer)
|
||||
if !ok {
|
||||
t.Fatalf("provider %T does not implement storage.VersionedHistoryConfigurer", provider)
|
||||
}
|
||||
runner, fired := conformance.NewUpdateHookLog(t)
|
||||
hooked := heartbeater
|
||||
if runner != nil {
|
||||
notifying, ok := NewNotifyingProvider(provider, Sinks{Hook: runner}).(HeartbeaterSource)
|
||||
if !ok {
|
||||
t.Fatal("the notifying provider does not offer the Heartbeater accessor")
|
||||
}
|
||||
if hooked, err = notifying.Heartbeater(); err != nil {
|
||||
t.Fatalf("hooked Heartbeater(): %v", err)
|
||||
}
|
||||
}
|
||||
kit := newUOWRoleFixtureKit(provider, prefix)
|
||||
return conformance.HeartbeaterFixture{
|
||||
IssuePrefix: kit.IssuePrefix,
|
||||
Heartbeater: heartbeater,
|
||||
CreateIssue: kit.CreateIssue,
|
||||
CreateWisp: kit.CreateWisp,
|
||||
Claimer: claimer,
|
||||
Lifecycle: lifecycle,
|
||||
Releaser: releaser,
|
||||
LeaseReclaimer: reclaimer,
|
||||
QueryScalar: kit.QueryScalar,
|
||||
CountHistory: kit.CountHistory,
|
||||
ExecUncommitted: func(ctx context.Context, statements []conformance.SQLStatement) error {
|
||||
_, err := RunTxEphemeral(ctx, provider, func(ctx context.Context, uw UnitOfWork) (struct{}, error) {
|
||||
for _, stmt := range statements {
|
||||
if _, err := uw.RawSQLUseCase().Exec(ctx, stmt.Query, stmt.Args...); err != nil {
|
||||
return struct{}{}, fmt.Errorf("%s: %w", stmt.Query, err)
|
||||
}
|
||||
}
|
||||
return struct{}{}, nil
|
||||
})
|
||||
return err
|
||||
},
|
||||
Exec: func(ctx context.Context, statements []conformance.SQLStatement) error {
|
||||
return RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) {
|
||||
for _, stmt := range statements {
|
||||
if _, err := uw.RawSQLUseCase().Exec(ctx, stmt.Query, stmt.Args...); err != nil {
|
||||
return "", fmt.Errorf("%s: %w", stmt.Query, err)
|
||||
}
|
||||
}
|
||||
return "seed heartbeater rows", nil
|
||||
})
|
||||
},
|
||||
SetJournalEnabled: configurer.SetEventsJournalEnabled,
|
||||
SetVersionedHistoryEnabled: versionConfigurer.SetVersionedHistoryEnabled,
|
||||
HookedHeartbeater: hooked,
|
||||
FiredUpdates: fired,
|
||||
}
|
||||
}
|
||||
|
|
@ -308,6 +308,14 @@ func (p *notifyingProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) {
|
|||
return NewLeaseReclaimer(p)
|
||||
}
|
||||
|
||||
// Heartbeater builds the lease refresh over THIS provider. recordingIssueUC
|
||||
// declares RenewLease: a refresh records nothing (it writes only the
|
||||
// ephemeral lease row), and a re-arm records an update, the hook
|
||||
// storage.HookFiringStore.Heartbeater fires for it on the DoltStorage chain.
|
||||
func (p *notifyingProvider) Heartbeater() (publicops.Heartbeater, error) {
|
||||
return NewHeartbeater(p)
|
||||
}
|
||||
|
||||
func (p *notifyingProvider) Memories() (memoryops.Memories, error) { return NewMemories(p) }
|
||||
|
||||
// EventsJournalCursor builds on THIS provider, like every role above it, so a
|
||||
|
|
@ -424,6 +432,7 @@ var (
|
|||
_ VersionReconcilerSource = (*notifyingProvider)(nil)
|
||||
_ MemoriesSource = (*notifyingProvider)(nil)
|
||||
_ LeaseReclaimerSource = (*notifyingProvider)(nil)
|
||||
_ HeartbeaterSource = (*notifyingProvider)(nil)
|
||||
_ EventsJournalCursorSource = (*notifyingProvider)(nil)
|
||||
)
|
||||
|
||||
|
|
@ -488,6 +497,22 @@ func (u *notifyingUOW) Commit(ctx context.Context, message string) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// CommitScoped is Commit for a scoped version commit (see ScopedCommitter):
|
||||
// commit through the chain beneath, then fire the buffered hooks.
|
||||
func (u *notifyingUOW) CommitScoped(ctx context.Context, tables []string, message string) error {
|
||||
if err := commitScoped(ctx, u.UnitOfWork, tables, message); err != nil {
|
||||
return err
|
||||
}
|
||||
entries := u.rec.drain()
|
||||
if u.sinks.Hook == nil {
|
||||
return nil
|
||||
}
|
||||
for _, entry := range entries {
|
||||
u.sinks.Hook.Run(entry.event, entry.issue)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close rolls the unit of work back unless it committed, so whatever the
|
||||
// attempt buffered is dropped rather than fired.
|
||||
func (u *notifyingUOW) Close(ctx context.Context) {
|
||||
|
|
@ -1009,6 +1034,20 @@ func (u *recordingIssueUC) ReleaseIssue(ctx context.Context, req publicops.Relea
|
|||
return result, wrote, nil
|
||||
}
|
||||
|
||||
// RenewLease records an update for a heartbeat that re-armed a lease (a
|
||||
// mutation: row_lock, updated_at, event, journal, version) and nothing for a
|
||||
// refresh, which writes only the ephemeral lease row. It is DECLARED for
|
||||
// ReleaseIssue's reason: an inherited accessor would compile and record
|
||||
// nothing, silently dropping the re-arm's hook.
|
||||
func (u *recordingIssueUC) RenewLease(ctx context.Context, request publicops.HeartbeatRequest) (publicops.HeartbeatResult, error) {
|
||||
result, err := u.IssueUseCase.RenewLease(ctx, request)
|
||||
if err != nil || !result.Rearmed {
|
||||
return result, err
|
||||
}
|
||||
u.rec.record(opUpdate, u.snap.issue(ctx, result.IssueID))
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// Reclaim records an update for every lease the sweep reverted.
|
||||
//
|
||||
// It is DECLARED for the same reason ReleaseIssue above is: an inherited
|
||||
|
|
|
|||
|
|
@ -130,7 +130,9 @@ func TestRecordingIssueUseCaseCoversItsSurface(t *testing.T) {
|
|||
"has no name for a deletion and the row is gone (hook_deleter.go, hookTrackingTransaction.DeleteIssue)"
|
||||
leases := "lease and claim bookkeeping the DoltStorage plumbing fires no hook for: " +
|
||||
"HookFiringStore overrides neither UnclaimIssue, UnclaimIssueIfAssignee, HeartbeatIssue nor " +
|
||||
"ReclaimExpiredLeases, so `bd unclaim` and `bd heartbeat` run no script on either plumbing"
|
||||
"ReclaimExpiredLeases, so `bd unclaim` runs no script on either plumbing, and neither does a " +
|
||||
"heartbeat through these doors (`bd heartbeat` takes the Heartbeater role, whose re-arm records " +
|
||||
"an update through RenewLease)"
|
||||
|
||||
assertPartition(t, "IssueUseCase", interfaceMethods(reflect.TypeOf((*domain.IssueUseCase)(nil)).Elem()),
|
||||
reflect.TypeOf((*recordingIssueUC)(nil)), map[string]string{
|
||||
|
|
|
|||
|
|
@ -181,19 +181,20 @@ func RunTxResultWithin[T any](ctx context.Context, p UnitOfWorkProvider, maxElap
|
|||
}
|
||||
|
||||
// RunTxEphemeral is RunTxResult for work whose writes touch ONLY ephemeral
|
||||
// (dolt_ignored) state — today the leases table, whose writes must mint no
|
||||
// Dolt commit and no history (bd-lrgn1; ported to proxied mode by bd-aq0ql).
|
||||
// On success the attempt is committed with the SQL-only form (Tx.Commit with
|
||||
// an empty message ⇒ plain COMMIT, no DOLT_COMMIT), which persists the
|
||||
// working set but records nothing in dolt_log — the proxied analog of the
|
||||
// classic DoltStore.withRetryTx commit discipline that HeartbeatIssue relies
|
||||
// on. A serialization loser (heartbeat and reclaim/close contend on the same
|
||||
// lease row by design) is replayed against a fresh unit of work, exactly like
|
||||
// RunTxResult.
|
||||
// (dolt_ignored) state, which must mint no Dolt commit and no history
|
||||
// (bd-aq0ql) — today local metadata and the events-journal prune. On success
|
||||
// the attempt is committed with the SQL-only form (Tx.Commit with an empty
|
||||
// message ⇒ plain COMMIT, no DOLT_COMMIT), which persists the working set but
|
||||
// records nothing in dolt_log. A serialization loser is replayed against a
|
||||
// fresh unit of work, exactly like RunTxResult.
|
||||
//
|
||||
// Do NOT use this for work that writes versioned tables: those writes would
|
||||
// persist while silently bypassing Dolt history. Every versioned write goes
|
||||
// through RunTx/RunTxResult with a real commit message.
|
||||
// through RunTx/RunTxResult with a real commit message. A heartbeat is NOT
|
||||
// ephemeral work: a lease refresh writes only the dolt_ignored leases table
|
||||
// (bd-lrgn1), but a re-arm writes versioned rows, so the Heartbeater commits
|
||||
// through RunTxScoped, whose attempt that names no tables takes this same
|
||||
// plain COMMIT.
|
||||
func RunTxEphemeral[T any](ctx context.Context, p UnitOfWorkProvider, work TxReadFunc[T]) (T, error) {
|
||||
var result T
|
||||
bo := backoff.NewExponentialBackOff()
|
||||
|
|
@ -232,6 +233,99 @@ func RunTxEphemeral[T any](ctx context.Context, p UnitOfWorkProvider, work TxRea
|
|||
return result, err
|
||||
}
|
||||
|
||||
// ScopedCommitter is the optional commit a unit of work offers for a write
|
||||
// whose versioned tables are KNOWN: it commits the SQL transaction, then
|
||||
// stages exactly tables (plus the versioned-history tables when the
|
||||
// transaction could have minted) and makes a Dolt commit named message from
|
||||
// that staged set alone, leaving another writer's uncommitted change to any
|
||||
// other table in the working set where it was. A staged table is committed
|
||||
// whole, so another writer's uncommitted change to one of those tables rides
|
||||
// in this commit, as it does in the DoltStore funnel. With nothing staged, or
|
||||
// under a deferred version commit, there is no Dolt commit. It is the uow
|
||||
// twin of the DoltStore funnel's post-transaction DOLT_ADD-per-table commit,
|
||||
// in the same order — the Dolt commit after the SQL COMMIT, never inside the
|
||||
// transaction, where it would be built from the transaction's start snapshot
|
||||
// (LatentLabsSpace/NEXUS#92) — and what a mutation that mostly touches
|
||||
// dolt_ignored state (a heartbeat re-arm) commits through instead of
|
||||
// DOLT_COMMIT('-Am'). An error means the SQL transaction did not commit; a
|
||||
// Dolt commit that fails after it is logged, never returned.
|
||||
type ScopedCommitter interface {
|
||||
CommitScoped(ctx context.Context, tables []string, message string) error
|
||||
}
|
||||
|
||||
// commitScoped commits uw through the first ScopedCommitter on its decorator
|
||||
// chain. A backend whose unit of work has none (a non-Dolt engine) gets the
|
||||
// plain SQL COMMIT: there is no version control to scope.
|
||||
func commitScoped(ctx context.Context, uw UnitOfWork, tables []string, message string) error {
|
||||
for u := uw; u != nil; {
|
||||
if scoped, ok := u.(ScopedCommitter); ok {
|
||||
return scoped.CommitScoped(ctx, tables, message)
|
||||
}
|
||||
unwrapper, ok := u.(unitOfWorkUnwrapper)
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
u = unwrapper.Unwrap()
|
||||
}
|
||||
return uw.Commit(ctx, "")
|
||||
}
|
||||
|
||||
// TxScopedFunc is the work RunTxScoped runs: its result, and the versioned
|
||||
// tables the attempt changed with the commit message naming them. No tables
|
||||
// means the attempt wrote only dolt_ignored state (or nothing).
|
||||
type TxScopedFunc[T any] func(ctx context.Context, uw UnitOfWork) (result T, tables []string, commitMsg string, err error)
|
||||
|
||||
// RunTxScoped is RunTxEphemeral for work that is USUALLY ephemeral but
|
||||
// sometimes writes versioned rows it can name: an attempt that reports no
|
||||
// tables commits with the plain SQL COMMIT (no working-set scan, no Dolt
|
||||
// commit, nothing swept); one that reports tables commits through
|
||||
// ScopedCommitter, staging exactly those. Never DOLT_COMMIT('-Am'): that
|
||||
// would sweep other writers' uncommitted (batch-mode) changes to other tables
|
||||
// into this commit (a staged table is committed whole). Serialization losers
|
||||
// are replayed like RunTxEphemeral's.
|
||||
func RunTxScoped[T any](ctx context.Context, p UnitOfWorkProvider, work TxScopedFunc[T]) (T, error) {
|
||||
var result T
|
||||
bo := backoff.NewExponentialBackOff()
|
||||
bo.InitialInterval = txRetryInitialInterval
|
||||
bo.MaxElapsedTime = DefaultTxRetryMaxElapsed
|
||||
|
||||
err := backoff.Retry(func() error {
|
||||
uw, err := p.NewUOW(ctx)
|
||||
if err != nil {
|
||||
if isSerializationError(err) {
|
||||
return err
|
||||
}
|
||||
return backoff.Permanent(err)
|
||||
}
|
||||
defer closeAttempt(ctx, uw)
|
||||
|
||||
r, tables, msg, err := work(ctx, uw)
|
||||
if err != nil {
|
||||
if isSerializationError(err) {
|
||||
return err
|
||||
}
|
||||
return backoff.Permanent(err)
|
||||
}
|
||||
|
||||
if len(tables) == 0 || msg == "" {
|
||||
err = uw.Commit(ctx, "")
|
||||
} else {
|
||||
err = commitScoped(ctx, uw, tables, msg)
|
||||
}
|
||||
if err != nil {
|
||||
if isSerializationError(err) {
|
||||
return err
|
||||
}
|
||||
return backoff.Permanent(err)
|
||||
}
|
||||
|
||||
result = r
|
||||
return nil
|
||||
}, backoff.WithContext(bo, ctx))
|
||||
|
||||
return result, err
|
||||
}
|
||||
|
||||
func RunTxRead[T any](ctx context.Context, p UnitOfWorkProvider, work TxReadFunc[T]) (T, error) {
|
||||
var zero T
|
||||
uw, err := p.NewUOW(ctx)
|
||||
|
|
|
|||
|
|
@ -55,6 +55,20 @@ func (u *baseUOW) Commit(ctx context.Context, message string) error {
|
|||
return u.tx.Commit(ctx, message)
|
||||
}
|
||||
|
||||
// CommitScoped commits through the transaction's scoped commit when it has
|
||||
// one (the Dolt server transaction), else with the plain SQL COMMIT.
|
||||
func (u *baseUOW) CommitScoped(ctx context.Context, tables []string, message string) error {
|
||||
if scoped, ok := u.tx.(scopedCommitTx); ok {
|
||||
return scoped.CommitScoped(ctx, tables, message)
|
||||
}
|
||||
return u.tx.Commit(ctx, "")
|
||||
}
|
||||
|
||||
// scopedCommitTx is the optional Tx half of ScopedCommitter.
|
||||
type scopedCommitTx interface {
|
||||
CommitScoped(ctx context.Context, tables []string, message string) error
|
||||
}
|
||||
|
||||
func (u *baseUOW) Close(ctx context.Context) {
|
||||
u.tx.RollbackUnlessCommitted(ctx)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ go_library(
|
|||
"dependency_editor.go",
|
||||
"edgereader.go",
|
||||
"graph_counter.go",
|
||||
"heartbeater.go",
|
||||
"init_verifier.go",
|
||||
"issue_claimer.go",
|
||||
"issue_operations.go",
|
||||
|
|
|
|||
36
internal/telemetry/heartbeater.go
Normal file
36
internal/telemetry/heartbeater.go
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
package telemetry
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// Heartbeater returns the inner store's lease-refresh surface wrapped in this
|
||||
// layer's instrumentation. It recurses instead of delegating: a blind
|
||||
// delegation would return the inner surface unspanned and untimed.
|
||||
func (s *InstrumentedStorage) Heartbeater() (issueops.Heartbeater, error) {
|
||||
inner, err := s.Unwrap().Heartbeater()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.WrapHeartbeater(inner), nil
|
||||
}
|
||||
|
||||
// WrapHeartbeater instruments the lease refresh with this storage layer's
|
||||
// existing telemetry meter and tracer.
|
||||
func (s *InstrumentedStorage) WrapHeartbeater(inner issueops.Heartbeater) issueops.Heartbeater {
|
||||
return &instrumentedHeartbeater{storage: s, inner: inner}
|
||||
}
|
||||
|
||||
type instrumentedHeartbeater struct {
|
||||
storage *InstrumentedStorage
|
||||
inner issueops.Heartbeater
|
||||
}
|
||||
|
||||
func (c *instrumentedHeartbeater) Heartbeat(ctx context.Context, request issueops.HeartbeatRequest) (result issueops.HeartbeatResult, err error) {
|
||||
ctx, span, started := c.storage.op(ctx, "Heartbeater.Heartbeat")
|
||||
result, err = c.inner.Heartbeat(ctx, request)
|
||||
c.storage.done(ctx, span, started, err)
|
||||
return result, err
|
||||
}
|
||||
|
|
@ -94,7 +94,7 @@ func TestInstrumentedStorageDeclaresEveryRoleAccessor(t *testing.T) {
|
|||
}
|
||||
|
||||
// roleAccessorStore is a DoltStorage whose only real methods are the
|
||||
// twenty-nine role accessors, each answering with a distinguishable sentinel
|
||||
// thirty role accessors, each answering with a distinguishable sentinel
|
||||
// so a test can tell an instrumented surface from a passed-through one.
|
||||
//
|
||||
// TWO sentinels rather than one: memoryops.Memories.List and issueops.Reader.List
|
||||
|
|
@ -178,8 +178,11 @@ func (s *roleAccessorStore) Releaser() (issueops.Releaser, error) {
|
|||
func (s *roleAccessorStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) {
|
||||
return s.surface, s.err
|
||||
}
|
||||
func (s *roleAccessorStore) Heartbeater() (issueops.Heartbeater, error) {
|
||||
return s.surface, s.err
|
||||
}
|
||||
|
||||
// roleAccessorSentinel implements twenty-eight of the twenty-nine roles at
|
||||
// roleAccessorSentinel implements twenty-nine of the thirty roles at
|
||||
// once — every one but memoryops.Memories, whose List collides with
|
||||
// issueops.Reader.List and needs the second sentinel below.
|
||||
// Nothing calls its methods; identity is the whole point.
|
||||
|
|
@ -312,6 +315,9 @@ func (*roleAccessorSentinel) Release(context.Context, issueops.ReleaseRequest) (
|
|||
func (*roleAccessorSentinel) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) {
|
||||
return issueops.ReclaimResult{}, nil
|
||||
}
|
||||
func (*roleAccessorSentinel) Heartbeat(context.Context, issueops.HeartbeatRequest) (issueops.HeartbeatResult, error) {
|
||||
return issueops.HeartbeatResult{}, nil
|
||||
}
|
||||
|
||||
// memoryRoleSentinel is the memory role's sentinel — see
|
||||
// memoryRoleSentinel is the remaining role's sentinel — see
|
||||
|
|
@ -389,6 +395,7 @@ func TestInstrumentedStorageInstrumentsEveryRoleAccessor(t *testing.T) {
|
|||
{"BatchApplier", func() (any, error) { return wrapped.BatchApplier() }, sentinel},
|
||||
{"Releaser", func() (any, error) { return wrapped.Releaser() }, sentinel},
|
||||
{"LeaseReclaimer", func() (any, error) { return wrapped.LeaseReclaimer() }, sentinel},
|
||||
{"Heartbeater", func() (any, error) { return wrapped.Heartbeater() }, sentinel},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
surface, err := test.got()
|
||||
|
|
@ -476,6 +483,7 @@ func TestInstrumentedStorageRoleAccessorsPropagateInnerErrors(t *testing.T) {
|
|||
{"BatchApplier", func() (any, error) { return wrapped.BatchApplier() }},
|
||||
{"Releaser", func() (any, error) { return wrapped.Releaser() }},
|
||||
{"LeaseReclaimer", func() (any, error) { return wrapped.LeaseReclaimer() }},
|
||||
{"Heartbeater", func() (any, error) { return wrapped.Heartbeater() }},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
surface, err := test.got()
|
||||
|
|
|
|||
|
|
@ -1849,6 +1849,11 @@ const (
|
|||
// EventLeaseReclaimed records that a stale lease was reverted to ready by
|
||||
// bd reclaim (dead-worker recovery). old_value is the previous owner.
|
||||
EventLeaseReclaimed EventType = "lease_reclaimed"
|
||||
// EventLeaseRearmed records that a heartbeat re-armed a claim that held no
|
||||
// lease row (a claim hand-doled through a generic update): the lease was
|
||||
// granted and row_lock reminted. new_value is the holder. A plain lease
|
||||
// refresh records no event.
|
||||
EventLeaseRearmed EventType = "lease_rearmed"
|
||||
)
|
||||
|
||||
// ProvenanceEvent is one entry in the append-only provenance log: a typed
|
||||
|
|
|
|||
|
|
@ -846,3 +846,35 @@ func TestDeleterExposesTypedUnsupportedError(t *testing.T) {
|
|||
t.Fatalf("Deleter() error = %v, want *beads.ErrUnsupported", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeaterLayersHooksOutsideTelemetry is the same pin for the lease
|
||||
// refresh. It gives the release's answer: a refresh fires nothing, but a
|
||||
// re-arm is recorded as a mutation and fires on_update
|
||||
// (internal/storage/hook_heartbeater.go), so this decorator wraps rather than
|
||||
// recursing the way Sweeper and Deleter do.
|
||||
func TestHeartbeaterLayersHooksOutsideTelemetry(t *testing.T) {
|
||||
t.Setenv("BD_OTEL_STDOUT", "true")
|
||||
instrumented, ok := telemetry.WrapStorage(&dolt.DoltStore{}).(*telemetry.InstrumentedStorage)
|
||||
if !ok {
|
||||
t.Fatal("WrapStorage() did not create InstrumentedStorage")
|
||||
}
|
||||
|
||||
heartbeater, err := storage.NewHookFiringStore(instrumented, nil).Heartbeater()
|
||||
if err != nil {
|
||||
t.Fatalf("Heartbeater() error = %v", err)
|
||||
}
|
||||
if got := reflect.TypeOf(heartbeater).String(); got != "*storage.hookHeartbeater" {
|
||||
t.Fatalf("outer layer = %s, want the hook wrapper", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHeartbeaterExposesTypedUnsupportedError(t *testing.T) {
|
||||
heartbeater, err := (*dolt.DoltStore)(nil).Heartbeater()
|
||||
if heartbeater != nil {
|
||||
t.Fatalf("Heartbeater() heartbeater = %T, want nil", heartbeater)
|
||||
}
|
||||
var unsupported *beads.ErrUnsupported
|
||||
if !errors.As(err, &unsupported) {
|
||||
t.Fatalf("Heartbeater() error = %v, want *beads.ErrUnsupported", err)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ go_library(
|
|||
"export.go",
|
||||
"gates.go",
|
||||
"graphcounter.go",
|
||||
"heartbeater.go",
|
||||
"importer.go",
|
||||
"initverifier.go",
|
||||
"issueops.go",
|
||||
|
|
@ -61,6 +62,7 @@ go_test(
|
|||
"current_test.go",
|
||||
"export_test.go",
|
||||
"gates_test.go",
|
||||
"heartbeater_test.go",
|
||||
"issueops_external_test.go",
|
||||
"molecule_internal_test.go",
|
||||
"molecule_test.go",
|
||||
|
|
|
|||
182
issueops/heartbeater.go
Normal file
182
issueops/heartbeater.go
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
package issueops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrLeaseNotHeld is the refusal a heartbeat answers when the caller holds no
|
||||
// live claim on the issue: it is open, closed, released, reclaimed, moved off
|
||||
// in_progress, or a wisp (wisps are never leased).
|
||||
//
|
||||
// IT ALSO MATCHES ErrNotClaimable, which is what the raw heartbeat has always
|
||||
// answered for the same states, so a caller written against that sentinel
|
||||
// keeps working when it moves onto this role.
|
||||
var ErrLeaseNotHeld = fmt.Errorf("lease not held: %w", ErrNotClaimable)
|
||||
|
||||
// LeaseConflictError reports the state that refused a heartbeat, read inside
|
||||
// the same transaction that found no lease row for the caller. It wraps the
|
||||
// refusal rather than replacing it, the way ClaimConflictError does, so the
|
||||
// sentinel still matches and a front door can classify the conflict from
|
||||
// typed fields instead of parsing prose.
|
||||
type LeaseConflictError struct {
|
||||
// IssueID names the issue the heartbeat was refused on.
|
||||
IssueID string
|
||||
// Holder is the foreign assignee that holds the claim. It is empty when
|
||||
// the refusal is about the status (no live claim) rather than a foreign
|
||||
// holder.
|
||||
Holder string
|
||||
// Status is the status the refusing transaction observed.
|
||||
Status Status
|
||||
// Ephemeral is true when the id names a wisp, which is never leased.
|
||||
// Status is then the wisp's own status, which may well be in_progress.
|
||||
Ephemeral bool
|
||||
// Err is the wrapped refusal. It matches ErrAlreadyClaimed (a foreign
|
||||
// holder) or ErrLeaseNotHeld (no live claim by the caller).
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *LeaseConflictError) Error() string { return e.Err.Error() }
|
||||
|
||||
// Unwrap makes LeaseConflictError match the refusal it carries.
|
||||
func (e *LeaseConflictError) Unwrap() error { return e.Err }
|
||||
|
||||
// The HeartbeatRequest fields a HeartbeatFieldError can name.
|
||||
const (
|
||||
HeartbeatFieldActor = "actor"
|
||||
HeartbeatFieldIssueID = "issue_id"
|
||||
)
|
||||
|
||||
// HeartbeatFieldError reports a HeartbeatRequest refusal and the one field it
|
||||
// is about, so a front door can name that field without parsing the message.
|
||||
// It wraps ErrValidation.
|
||||
type HeartbeatFieldError struct {
|
||||
// Field is one of the HeartbeatField* constants.
|
||||
Field string
|
||||
// Detail is the sentence the error reads as.
|
||||
Detail string
|
||||
}
|
||||
|
||||
func (e *HeartbeatFieldError) Error() string { return e.Detail }
|
||||
|
||||
// Unwrap makes HeartbeatFieldError match ErrValidation.
|
||||
func (e *HeartbeatFieldError) Unwrap() error { return ErrValidation }
|
||||
|
||||
// HeartbeatRequest refreshes the lease on one issue the actor holds — the
|
||||
// shape behind `bd heartbeat`.
|
||||
//
|
||||
// Implementations never mutate caller-owned request values.
|
||||
type HeartbeatRequest struct {
|
||||
// Actor is the lease holder refreshing its lease, and the fence's
|
||||
// subject: only the current holder may heartbeat. It is REQUIRED — a
|
||||
// blank Actor is ErrValidation — and so is one longer than
|
||||
// types.MaxFieldLen characters, which no lease row can hold.
|
||||
//
|
||||
// THE ROLE DOES NOT TRIM OR CASE-FOLD IT, Releaser.Actor's rule: the
|
||||
// value is compared with the lease row's holder as sent. A front door may
|
||||
// normalize before it calls the role: bd serve trims the body's actor by
|
||||
// the claim's rule (so a served claim and a served heartbeat agree on the
|
||||
// spelling), and the CLI passes its resolved actor through. A holder
|
||||
// spelled differently across layers (separator-insensitive, see
|
||||
// ReleaseRequest.ExpectedAssignee) still heartbeats: it misses the fast
|
||||
// path, and the slow path refreshes the lease row in place under this
|
||||
// spelling.
|
||||
Actor string
|
||||
// IssueID is the exact canonical id, on either plane, and must not be
|
||||
// empty. There is no fuzzy or prefix resolution: that is a front-door
|
||||
// convenience, for the reason GetRequest.ID gives.
|
||||
IssueID string
|
||||
}
|
||||
|
||||
// HeartbeatResult reports one refreshed lease.
|
||||
type HeartbeatResult struct {
|
||||
// IssueID is the issue whose lease was refreshed.
|
||||
IssueID string
|
||||
// Holder is the lease holder after the refresh: the request's Actor, in
|
||||
// the request's spelling.
|
||||
Holder string
|
||||
// HeartbeatAt is the server clock, in UTC and rounded to whole seconds
|
||||
// as the lease row's column rounds it: the stored value.
|
||||
HeartbeatAt time.Time
|
||||
// LeaseExpiresAt is HeartbeatAt plus the store's lease TTL: the deadline
|
||||
// the next heartbeat must beat before a reclaim may revert the claim.
|
||||
LeaseExpiresAt time.Time
|
||||
// Rearmed is true only when the heartbeat CREATED the caller's lease row:
|
||||
// the caller held a live claim but no lease row the caller holds existed
|
||||
// (a claim hand-doled through a generic update), so the heartbeat granted
|
||||
// a fresh one instead of refreshing it. A re-arm is recorded as a
|
||||
// mutation and remints the issue's revision token (see Heartbeater);
|
||||
// re-read before a guarded write. A refresh of the caller's own row
|
||||
// is never a re-arm, including one that changed no value (two heartbeats
|
||||
// in one second) and one under a differently spelled holder.
|
||||
//
|
||||
// false does not prove that nothing reminted the token: a transport that
|
||||
// resends a heartbeat whose answer was lost (the HTTP client does) reports
|
||||
// the resend's answer, a refresh of the lease row a committed re-arm
|
||||
// granted. A guarded write composed on the old revision then fails its
|
||||
// revision check once and re-reads.
|
||||
Rearmed bool
|
||||
}
|
||||
|
||||
// Heartbeater refreshes a held lease — the capability behind `bd heartbeat` —
|
||||
// and, like every other capability here, a role with its own accessor.
|
||||
//
|
||||
// IT IS A DIFFERENT QUESTION FROM Claimer. A claim takes ownership and is
|
||||
// idempotent for the holder WITHOUT extending the lease (a re-claim returns
|
||||
// before the lease is touched); a heartbeat proves the holder is alive and
|
||||
// does nothing else. Folding one into the other would make every keepalive a
|
||||
// claim attempt, with a claim's refusals and a claim's history.
|
||||
//
|
||||
// A REFRESH WRITES ONLY THE LEASE ROW. A heartbeat touches the ephemeral
|
||||
// leases table and nothing else: it does not remint row_lock (so the holder's
|
||||
// own revision token survives it and its guarded writes keep working), it
|
||||
// records no version (current_revision and issue_versions stay put), it does
|
||||
// not stamp updated_at, it records no event and no events-journal entry, it
|
||||
// makes no version-control commit, and it fires no hook. It is the hottest
|
||||
// write in a fleet, and every one of those would make it expensive or noisy.
|
||||
//
|
||||
// A RE-ARM IS RECORDED AS A MUTATION; A REFRESH IS NOT. A re-arm (Rearmed:
|
||||
// the caller held a live claim with no lease row, a claim hand-doled through
|
||||
// a generic update) is recorded the way every bead mutator records a write:
|
||||
// it remints row_lock, stamps updated_at, records a lease_rearmed event, an
|
||||
// events-journal entry and a version, makes a version-control commit, and
|
||||
// fires on_update. The remint is what makes the grant collide with a release
|
||||
// or close racing it, instead of committing beside it and leaving a lease on
|
||||
// an issue that is no longer claimed. Re-arms are rare (a hand-doled claim's
|
||||
// first heartbeat), so the cost lands once per such claim.
|
||||
//
|
||||
// THE LIVE-CLAIM FENCE IS AN INVARIANT AND A PREDICATE. Every path that ends
|
||||
// or transfers a claim deletes the lease row, and a re-arm racing such a path
|
||||
// collides with it on row_lock, so a lease row exists only while its claim is
|
||||
// live. The refresh also matches the lease row by holder AND requires, in the
|
||||
// same transaction, that the issue is in_progress with that holder as assignee,
|
||||
// so a lease row orphaned any other way fails the refresh, and the refusal
|
||||
// deletes it. A refresh that lands is a refresh of a live claim.
|
||||
//
|
||||
// EVERY LEG RUNS ONE BODY: dolt, embedded dolt and the unit-of-work provider
|
||||
// wrap internal/storage/issueops.ExecuteHeartbeatInTx.
|
||||
//
|
||||
// Deterministic request-validation failures match ErrValidation. Result
|
||||
// values are unspecified when error is non-nil. Every refusal below leaves
|
||||
// persistent state unchanged, with one exception: a *LeaseConflictError
|
||||
// refusal deletes the caller's own lease row if one survives on an issue the
|
||||
// caller no longer holds a live claim on (an orphan; never a live holder's
|
||||
// lease).
|
||||
type Heartbeater interface {
|
||||
// Heartbeat refreshes the lease on request.IssueID held by request.Actor.
|
||||
//
|
||||
// REFUSALS:
|
||||
//
|
||||
// - a blank Actor, one longer than types.MaxFieldLen characters, or an
|
||||
// empty IssueID: a *HeartbeatFieldError (ErrValidation), before
|
||||
// anything is read;
|
||||
// - an IssueID that names no row on either plane: ErrNotFound;
|
||||
// - an issue another actor holds: a *LeaseConflictError naming the
|
||||
// Holder, wrapping ErrAlreadyClaimed;
|
||||
// - a wisp, or an issue the actor holds no live claim on (open, closed,
|
||||
// released, reclaimed, or any status but in_progress): a
|
||||
// *LeaseConflictError wrapping ErrLeaseNotHeld, which also matches
|
||||
// ErrNotClaimable; Ephemeral is set for a wisp.
|
||||
Heartbeat(ctx context.Context, request HeartbeatRequest) (HeartbeatResult, error)
|
||||
}
|
||||
30
issueops/heartbeater_test.go
Normal file
30
issueops/heartbeater_test.go
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
package issueops_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// TestErrLeaseNotHeldStillMatchesErrNotClaimable pins the compatibility
|
||||
// promise ErrLeaseNotHeld's doc makes: a caller written against the raw
|
||||
// heartbeat's ErrNotClaimable keeps classifying the role's refusal.
|
||||
func TestErrLeaseNotHeldStillMatchesErrNotClaimable(t *testing.T) {
|
||||
err := &issueops.LeaseConflictError{IssueID: "x", Err: fmt.Errorf("%w: x status closed", issueops.ErrLeaseNotHeld)}
|
||||
if !errors.Is(err, issueops.ErrLeaseNotHeld) || !errors.Is(err, issueops.ErrNotClaimable) {
|
||||
t.Fatalf("LeaseConflictError(ErrLeaseNotHeld) = %v: want it to match ErrLeaseNotHeld and ErrNotClaimable", err)
|
||||
}
|
||||
if errors.Is(err, issueops.ErrAlreadyClaimed) {
|
||||
t.Fatal("ErrLeaseNotHeld matches ErrAlreadyClaimed; the two refusals must stay distinguishable")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHeartbeatFieldErrorMatchesErrValidation pins the field error's sentinel.
|
||||
func TestHeartbeatFieldErrorMatchesErrValidation(t *testing.T) {
|
||||
var err error = &issueops.HeartbeatFieldError{Field: issueops.HeartbeatFieldActor, Detail: "validation failed: heartbeat actor is required"}
|
||||
if !errors.Is(err, issueops.ErrValidation) {
|
||||
t.Fatalf("HeartbeatFieldError = %v: want it to match ErrValidation", err)
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue