perf(http): answer a command's repeated reads from one response; bd show --json makes 1 request (port of bd-enterprise e626072b8) (#7588)

* perf(http): answer a command's repeated reads from one response (port of bd-enterprise e626072b8)

Over an http workspace, bd show <id> --json made three identical
GET /v0/beads/issues/<id> requests: the id resolver's exact probe,
routed.go's GetIssue and show's detail read all build the same URL. Text
show made six. Over a gateway each one is a sequential round trip, so the
request count is the command's latency floor (mc-5dkzp).

This ports bd-enterprise e626072b8 (bts-xpw.3.14.6, #217) verbatim, except
cmd/bd/main.go, where OSS still opens the identity preview inline, and
bd-enterprise's httpgw dialer test, which has no OSS counterpart:

- httpclient.WithSharedConnections: the identity-gate preview and the real
  open share one transport, so a write makes one handshake.
- httpclient.WithReadMemo and a wire.Client read memo, for one-shot
  commands only (not bd serve, --watch or --follow): identical getIssue,
  getSetting and listSettings reads are answered from the response already
  received. An entry ages from the server's answer (5 s), any write clears
  the memo, a read that overlapped a write is not stored, and expired
  entries are swept.
- bd list's three vocabulary reads come from one memoized listSettings.
- bd update --status / --if-status skips the custom-status read for a
  built-in status.
- The hook decorator skips its post-write re-reads when no executable hook
  exists for the event.
- bd dep add reads its confirmation titles before the write, from the memo.

Against a loopback bd serve: show --json goes from 3 requests to 1, text
show 6 to 3, update 5 to 3, close 6 to 4. Dolt and the other registered
backends receive the same ctx and ignore it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(http): BD_HTTP_READ_MEMO=0 turns the per-command read memo off

An operator who suspects a stale read can rule the memo out without a
rebuild. Any other value, or none, leaves the default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(http): pin bd show to one fetch per issue (mc-5dkzp)

TestE2E_PerCommandRequestBudget allowed show two requests, and a read
that drifts to a URL of its own (a mutant whose reader spells
brief_deps=false) fits that ceiling on both handshake shapes. These guards
pin the single fetch instead:

- TestE2E_ShowFetchesTheIssueOnce: the exact requests, path and query, of
  show --json, show --json --id, show --short, show of two ids and text
  show, against both handshake shapes, plus a BD_HTTP_READ_MEMO=0 step
  that must read the row more than once.
- TestDefaultIssueReadsShareOneMemoKey: the resolver's exact probe,
  GetIssue, GetDependenciesWithMetadata, GetLabels and IssueReader().Get
  build one URL, checked without a server.
- TestE2E_CloseRereadsItsOwnWrite: against the MC handshake, closing a
  lone molecule re-reads its own closed row from the server after the
  write, and prints no false "completed molecule" note.
- Corpus shapes can pin `requests:`, counted on the served run alone (not
  setup, verify or {rev:ID} reads). The seven gc and mc show shapes pin
  1, 4, 1, 1, 3, 1 and 3.

Also: TestReadMemoOptInReadsTheRealPollingFlags holds the opt-in to the
real show/list --watch and -w, events tail --follow and bd serve; the
show budget drops from 2 to 1; requestCounter records each request's
query; and ledger row L-read-memo records the memo's bounded difference
from local reads.

Mutation runs against this tree: brief_deps=true on GetIssue and
brief_deps=false on the reader's Get each fail the e2e list and the unit
test; show opting out of the memo fails the e2e list; writes that leave
the memo standing fail the read-your-own-write e2e; a memo keyed by path
alone fails the e2e list; dropping the kill switch fails its unit test
and the e2e step.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(config): list BD_HTTP_READ_MEMO

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(http): memoize a read's answer only once it has decoded

wire.Client.Do stored a memoable read's 2xx before it checked the body,
so an empty or undecodable 200 would have answered every identical read
in the command for the memo's whole TTL with the same failure, instead
of asking the server again. The answer is now stored only after it
decodes; a read that decodes nothing (out == nil) is not stored, and
every memoable read in this package decodes its answer.

TestReadMemoKeepsOnlyAnswersThatDecoded pins it (empty body, truncated
body, undecoded answer). Ledger row L-read-memo says so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(http): pin the read memo's 5xx, write-method, per-client, in-flight-write and hook-gate behavior

Review follow-ups for the per-command read memo (mc-5dkzp):

- TestReadMemoNeverKeepsA5xx: a 503, 500 or 502 is not memoized and the
  2xx after it is. TestDispatchRetryOfABusyMemoizedReadAsksTheServer
  holds the same through dispatch's retry loop over a real wire.Client:
  the retry reaches the server instead of replaying the 503.
- TestReadMemoIsDroppedByAnyWrite covers POST, PUT and DELETE, not just
  PATCH.
- TestReadMemoIsPerClient: in either order, one client's read is never
  answered from another's memo, and opting one client in opts in no
  other. (A shared memo was otherwise caught only by test-order
  pollution.)
- TestReadMemoDropsAnAnswerReadWhileAWriteWasInFlight pins the clear on
  the write's way back: a read that starts and ends while the write is
  on the wire is not answered again once the write has returned.
- TestHookFiringStoreReReadsOnlyForAnEventWithAHook pins hookConfigured's
  probe: the decorator asks the runner about the event it fires, and
  re-reads the row to build a payload only when a hook for that event
  exists, through a probing double and through a real hooks.Runner.

Each was run against a mutant of the code it pins and fails there:
5xx stored before the status check; only PATCH clearing; one memo
shared by every client; no clear after the write returns;
hookConfigured answering true, false, or about the wrong event.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(changelog): note the per-command http read memo

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-10-16 04:26:37 +00:00 • committed by GitHub
commit b303d494df
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
29 changed files with 1560 additions and 16 deletions

View file

@ -316,6 +316,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **Over an http workspace, a bd command answers its repeated reads of a row
or setting from one response.** `bd show <id> --json` makes one request
instead of three, text `bd show` three instead of six, and `bd update`,
`bd close`, `bd label add`, `bd comments` and `bd dep add` one to seven
fewer (port of bd-enterprise e626072b8). A one-shot command's transport
answers an identical `getIssue`, `getSetting` or `listSettings` GET from the
answer it already received, for at most 5 s from that answer. Any write the
command makes clears it, and only a 2xx that decoded is kept. `bd serve`,
`--watch` and `--follow` never memoize, and neither do library embedders.
Within one command a re-read no longer sees another writer's change made in
the meantime (ledger row `L-read-memo`); `BD_HTTP_READ_MEMO=0` turns the
memo off. The identity-gate preview and the real open now share one
transport, so a write makes one handshake. `bd list` reads its vocabulary
with one `listSettings`, and the hook decorator skips its post-write re-reads
when no hook for the event is installed.
- **`created_by` defaults to the actor in the library, for every create
shape and every backend.** `PreparePublicCreateRequest`, which
`Lifecycle.Create`, `BatchCreator` items and batch-apply create items all

View file

@ -36,10 +36,13 @@ go_test(
"e2e_list_filters_test.go",
"e2e_mol_test.go",
"e2e_promote_test.go",
"e2e_read_your_own_write_test.go",
"e2e_recompute_blocked_test.go",
"e2e_remote_roles_test.go",
"e2e_request_budget_test.go",
"e2e_search_test.go",
"e2e_show_current_test.go",
"e2e_show_single_fetch_test.go",
"open_test.go",
"per_open_credential_test.go",
"register_test.go",

View file

@ -117,6 +117,7 @@ Each shape in `corpus.yaml` has these fields:
| `ignore_fields` | JSON keys dropped, at any depth, before a `json` comparison and before `verify` output is compared; for example comment ids, which each workspace mints itself |
| `expect_exit`, `expect_stderr` | expected outcome on both sides (exit 0 by default). An allowlisted shape must state `expect_exit`; only its embedded run is held to these |
| `pending` | a known failure |
| `requests` | exactly how many requests the served run of `args` makes through the reference proxy, not counting `setup`, `verify` or `{rev:ID}` reads. Each request is a sequential round trip, so the count is the shape's latency floor over a gateway. A change that makes the shape cheaper updates the pin |
`pending: {slice, reason}` marks a known failure and names the slice or
slices that fix it, for example `S6c+S15`. A pending shape **still runs**:

View file

@ -85,6 +85,12 @@ type shape struct {
// Verify are reads run on both sides after the shape and compared as json:
// how a write is checked for having done the same thing on both backends.
Verify [][]string `yaml:"verify"`
// Requests, when set, is exactly how many requests the served run of Args
// makes through the reference proxy (not its setup or verify reads). Each
// is a sequential round trip, so the count is the shape's latency floor
// over a gateway: a change that adds one fails here, and one that saves
// one updates the pin.
Requests *int `yaml:"requests"`
// Pending marks a known failure and names the slice that fixes it.
Pending *pending `yaml:"pending"`
}
@ -199,6 +205,9 @@ func TestCorpusFileIsWellFormed(t *testing.T) {
if modeOf(s) != "exit" || len(s.Verify) > 0 {
t.Errorf("shape %s is allowlisted, so only its exit code and stderr are checked; give it compare: exit and no verify", s.ID)
}
if s.Requests != nil {
t.Errorf("shape %s is allowlisted, so its request count is not checked; drop requests", s.ID)
}
}
switch s.Compare {
case "", "json", "ids", "count", "text", "exit":
@ -211,6 +220,9 @@ func TestCorpusFileIsWellFormed(t *testing.T) {
if _, err := regexp.Compile(s.ExpectStderr); err != nil {
t.Errorf("shape %s: expect_stderr: %v", s.ID, err)
}
if s.Requests != nil && *s.Requests < 0 {
t.Errorf("shape %s: requests %d is negative", s.ID, *s.Requests)
}
}
for id, entry := range allow {
if !seen[id] {
@ -442,7 +454,10 @@ func runShape(t *testing.T, bin string, s shape, served, local *side, allow map[
}
}
proxy.takeRefusals()
sr := runBD(t, bin, served.dir, served.env, s.Stdin, expand(context.Background(), t, bin, served, s.Args)...)
args := expand(context.Background(), t, bin, served, s.Args)
before := proxy.requests.Load()
sr := runBD(t, bin, served.dir, served.env, s.Stdin, args...)
servedRequests := proxy.requests.Load() - before
wireRefusals := proxy.takeRefusals()
lr := runBD(t, bin, local.dir, local.env, s.Stdin, expand(context.Background(), t, bin, local, s.Args)...)
t.Logf("bd %s\nserved: %s\nembedded: exit=%d", strings.Join(s.Args, " "), sr, lr.code)
@ -487,6 +502,9 @@ func runShape(t *testing.T, bin string, s shape, served, local *side, allow map[
if s.ExpectStderr != "" && !wantStderr.MatchString(sr.stderr) {
failf("served stderr does not match %q (embedded does): %s", s.ExpectStderr, sr)
}
if s.Requests != nil && servedRequests != int64(*s.Requests) {
failf("the served run made %d requests, want exactly %d (requests); a change that saves one updates the pin", servedRequests, *s.Requests)
}
if len(fails) > 0 {
return fails
}

View file

@ -144,18 +144,25 @@ seed:
shapes:
# ---------------------------------------------------------------- show
# requests pins each show's round trips (mc-5dkzp): resolving the id,
# fetching the row and reading it for output are one request per id, which
# the per-command read memo answers once.
- id: gc-show-json
source: gc bdstore.go BdStore.Get (show --json <id>); counts must not read 0
args: [show, --json, e2e-blk]
requests: 1
- id: gc-show-json-batch
source: gc bdstore_exact_batch.go GetExactBatch
args: [show, --json, e2e-r1, e2e-a1, e2e-blk, e2e-mol]
requests: 4
- id: gc-show-json-wisp
source: gc workquery.go blocked_by enrichment; wisps live in the ready tier
args: [show, e2e-wisp-1, --json]
requests: 1
- id: gc-show-id-flag
source: gc cmd_bd_show_watch.go bdWatchRunner (show --json --id=<id>)
args: [show, --json, --id=e2e-r1]
requests: 1
# The worker holds eight in-progress issues; e2e-ip1 is the only P0 among
# them, so "first in the store's default order" (priority first) names it on
# both sides. Among equal priorities the order rests on created_at, and a
@ -163,16 +170,21 @@ shapes:
- id: gc-show-current
source: gc cmd_bd_show_watch.go bdWatchRunner (show --current --json)
args: [show, --current, --json]
requests: 3
- id: gc-show-missing
source: gc BdStore.Get not-found classification (routing fallbacks)
args: [show, --json, e2e-nonexistent]
expect_exit: 1
expect_stderr: "(?i)not found"
compare: exit
requests: 1
# Text show reads the row, then its dependents and its comments: three
# different URLs the memo cannot collapse.
- id: mc-show-text
source: mc prompts (bd show <id>, ~80)
args: [show, e2e-r1]
compare: text
requests: 3
# ---------------------------------------------------------------- ready
# e2e-wr and e2e-wisp-wr are the only open rows assigned to the worker when

View file

@ -0,0 +1,51 @@
//go:build cgo
package bdhttp_test
import (
"slices"
"strings"
"testing"
)
// TestE2E_CloseRereadsItsOwnWrite (mc-5dkzp): the per-command read memo must
// never answer a read with a row from before this command's own write. Against
// the MC server's handshake (no issues.close.autoCloseMolecule) bd close runs
// the molecule auto-close check client side after the close, and that check
// re-reads the closed row. Two halves hold it:
//
// - The wire: the re-read reaches the server after the close's write. A memo
// that survived the write answers it from the pre-close response, and no
// request is made.
// - What the user sees: a lone molecule closed by the command is its own
// root. Read fresh it is closed and nothing is said; where the check reads
// the root's status, a memoized pre-close row reads as a completed,
// still-open molecule and the close prints a false "completed molecule"
// note.
//
// A hook-payload test cannot guard this: the http write roles fire hooks from
// the write's own response, never from a re-read.
func TestE2E_CloseRereadsItsOwnWrite(t *testing.T) {
skipUnlessEmbeddedDolt(t)
bin := buildBD(t)
addr := startE2EServer(t)
proxyAddr, counter := startMaskingProxy(t, addr, mcContextMask)
f := newHTTPFixture(t, bin, proxyAddr)
mol := f.create("lonely molecule", "-t", "molecule")
// bd close resolves and reads the row before it writes, so a memo that
// survived the write would still hold the open row.
counter.reset()
res := f.mustRun(nil, "close", mol)
uris := counter.uriSnapshot()
t.Logf("bd close: %s", strings.Join(uris, ", "))
if strings.Contains(res.stderr, "completed molecule") {
t.Fatalf("bd close re-read its own closed row from before the write and named it a completed, still-open molecule: %s", res.stderr)
}
write := slices.Index(uris, "POST /v0/beads/issues:batchClose")
if write < 0 {
t.Fatalf("bd close %s sent no batchClose: %q", mol, uris)
}
if !slices.Contains(uris[write+1:], "GET /v0/beads/issues/"+mol) {
t.Fatalf("bd close %s did not re-read the row from the server after closing it (%q): a read after the command's own write was answered from before it", mol, uris)
}
}

View file

@ -160,16 +160,20 @@ func newHTTPFixture(t *testing.T, bin, addr string) *remoteRolesFixture {
}
// requestCounter is a loopback reverse proxy in front of the in-process
// server that counts the requests a command makes, by path.
// server that counts the requests a command makes, by path. uris records the
// same requests with their query, for a test that tells a bare read from one
// that asks for more of the row.
type requestCounter struct {
mu sync.Mutex
paths []string
uris []string
}
func (c *requestCounter) reset() {
c.mu.Lock()
defer c.mu.Unlock()
c.paths = nil
c.uris = nil
}
func (c *requestCounter) snapshot() []string {
@ -178,6 +182,13 @@ func (c *requestCounter) snapshot() []string {
return append([]string(nil), c.paths...)
}
// uriSnapshot is snapshot with each request's query: "GET /path?query".
func (c *requestCounter) uriSnapshot() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.uris...)
}
func startCountingProxy(t *testing.T, upstream string) (string, *requestCounter) {
t.Helper()
return startMaskingProxy(t, upstream, nil)
@ -252,6 +263,7 @@ func startMaskingProxy(t *testing.T, upstream string, mask func(map[string]any))
srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
counter.mu.Lock()
counter.paths = append(counter.paths, r.Method+" "+r.URL.Path)
counter.uris = append(counter.uris, r.Method+" "+r.URL.RequestURI())
counter.mu.Unlock()
proxy.ServeHTTP(w, r)
})}

View file

@ -0,0 +1,91 @@
//go:build cgo
package bdhttp_test
import (
"strings"
"testing"
)
// TestE2E_PerCommandRequestBudget pins how many sequential gateway requests
// each everyday bd command makes over the http backend (bts-xpw.3.14.6,
// beads-team-server docs/v0-gateway-latency.md option F). On the hosted /v0
// path every request costs ~0.33 s warm, and a command makes them strictly one
// after another, so the request count IS the command's latency floor.
//
// The budgets are ceilings, not exact counts: a change that makes a command
// cheaper passes, a change that adds a round trip fails here first. Both the
// current server's handshake and the MC server's (fewer capabilities, no
// wire_revision) are held to the same ceilings.
func TestE2E_PerCommandRequestBudget(t *testing.T) {
skipUnlessEmbeddedDolt(t)
bin := buildBD(t)
addr := startE2EServer(t)
for _, tc := range []struct {
name string
mask func(map[string]any)
// over raises a step's budget for this handshake shape, where the
// server lacks a capability the step otherwise relies on.
over map[string]int
}{
// The in-process server composes no external-dependency policy, so
// its handshake lacks policy.external_dependencies and the client
// applies the policy itself with extra reads. A deployed bd serve
// (hosted included) enforces it and says so; advertise it here so
// the budget measures that shape.
{"current server", advertiseExternalPolicy, nil},
// The MC server predates issues.close.autoCloseMolecule, so a close
// there also runs the molecule auto-close check client side (an edge
// read and a re-read of the closed row).
{"MC server mask", mcContextMask, map[string]int{"close": 5}},
} {
t.Run(tc.name, func(t *testing.T) {
proxyAddr, counter := startMaskingProxy(t, addr, tc.mask)
f := newHTTPFixture(t, bin, proxyAddr)
target := f.create("budget target", "-t", "task")
other := f.create("budget other", "-t", "task")
closing := f.create("budget closing", "-t", "task")
f.mustRun(nil, "comments", "add", target, "seed comment")
for _, step := range []struct {
name string
budget int
args []string
}{
{"ready", 2, []string{"ready", "--json"}},
{"list", 4, []string{"list", "--json"}},
{"show", 1, []string{"show", target, "--json"}},
{"create", 2, []string{"create", "budget created", "-t", "task", "--json"}},
{"update --append-notes", 3, []string{"update", target, "--append-notes", "more", "--json"}},
{"update --status", 3, []string{"update", target, "--status", "in_progress", "--json"}},
{"close", 3, []string{"close", closing, "--json"}},
{"comments add", 3, []string{"comments", "add", target, "a budget comment"}},
{"dep add", 5, []string{"dep", "add", target, other}},
} {
counter.reset()
f.mustRun(nil, step.args...)
paths := counter.snapshot()
t.Logf("bd %s: %d requests: %s", step.name, len(paths), strings.Join(paths, ", "))
budget := step.budget
if over, ok := tc.over[step.name]; ok {
budget = over
}
if len(paths) > budget {
t.Errorf("bd %s made %d gateway requests, budget %d: %v", step.name, len(paths), budget, paths)
}
}
})
}
}
// advertiseExternalPolicy adds policy.external_dependencies to the handshake's
// capabilities, the shape of a deployed bd serve that enforces the policy.
func advertiseExternalPolicy(body map[string]any) {
caps, _ := body["capabilities"].([]any)
for _, c := range caps {
if c == "policy.external_dependencies" {
return
}
}
body["capabilities"] = append(caps, "policy.external_dependencies")
}

View file

@ -0,0 +1,89 @@
//go:build cgo
package bdhttp_test
import (
"slices"
"strings"
"testing"
)
// TestE2E_ShowFetchesTheIssueOnce (mc-5dkzp) pins the exact requests, path AND
// query, each bd show shape makes over the http backend. bd show resolves the
// id, fetches the row, then reads it for output: three reads of one URL that
// the per-command read memo answers with one request. A read that drifts to a
// URL of its own (a new parameter, or a default spelled out) still fits a
// request ceiling of two, so this is an exact list, not a budget: a change that
// makes show cheaper updates the list here too.
//
// Text show's dependents and comments sections are separate reads of the same
// row with include_dependents and include_comments, which the memo cannot
// collapse; a later change that reads them in one request updates that step.
func TestE2E_ShowFetchesTheIssueOnce(t *testing.T) {
skipUnlessEmbeddedDolt(t)
bin := buildBD(t)
addr := startE2EServer(t)
for _, tc := range []struct {
name string
mask func(map[string]any)
}{
{"current server", advertiseExternalPolicy},
{"MC server mask", mcContextMask},
} {
t.Run(tc.name, func(t *testing.T) {
proxyAddr, counter := startMaskingProxy(t, addr, tc.mask)
f := newHTTPFixture(t, bin, proxyAddr)
target := f.create("once target", "-t", "task")
other := f.create("once other", "-t", "task")
f.mustRun(nil, "dep", "add", target, other)
f.mustRun(nil, "comments", "add", target, "seed comment")
get := func(id, query string) string {
uri := "GET /v0/beads/issues/" + id
if query != "" {
uri += "?" + query
}
return uri
}
for _, step := range []struct {
args []string
want []string
}{
{[]string{"show", target, "--json"}, []string{get(target, "")}},
{[]string{"show", "--json", "--id=" + target}, []string{get(target, "")}},
{[]string{"show", target, "--short"}, []string{get(target, "")}},
{[]string{"show", target, other, "--json"}, []string{get(target, ""), get(other, "")}},
{[]string{"show", target}, []string{
get(target, ""),
get(target, "include_dependents=true"),
get(target, "include_comments=true"),
}},
} {
counter.reset()
f.mustRun(nil, step.args...)
if got := counter.uriSnapshot(); !slices.Equal(got, step.want) {
t.Errorf("bd %s:\n got %q\n want %q", strings.Join(step.args, " "), got, step.want)
}
}
// BD_HTTP_READ_MEMO=0 turns the memo off: the same show asks
// the server again for every read of the row.
counter.reset()
f.mustRun([]string{"BD_HTTP_READ_MEMO=0"}, "show", target, "--json")
got := counter.uriSnapshot()
if bare := countOf(got, get(target, "")); bare < 2 {
t.Errorf("BD_HTTP_READ_MEMO=0 bd show %s --json read the row %d times (%q); the kill switch must turn the memo off", target, bare, got)
}
})
}
}
func countOf(list []string, want string) int {
n := 0
for _, s := range list {
if s == want {
n++
}
}
return n
}

View file

@ -470,6 +470,15 @@ Examples:
if err != nil {
return HandleErrorRespectJSON("%v", err)
}
// The confirmation line names both issues by title. Adding an edge
// changes neither title, so read them now, while the resolution's
// reads above are still fresh: over http those two lookups are then
// answered without a round trip, where after the write they would be
// two more (bts-xpw.3.14.6).
var fromTitle, toTitle string
if !jsonOutput {
fromTitle, toTitle = lookupTitle(fromID), lookupTitle(toID)
}
edge := issueops.DependencyEdge{IssueID: fromID, DependsOnID: toID, Type: dt}
if err := addDependencyEdgesDirect(opsCtx, fromStore, []issueops.DependencyEdge{edge}, false); err != nil {
return HandleErrorRespectJSON("%v", err)
@ -500,7 +509,7 @@ Examples:
}
fmt.Printf("%s Added dependency: %s %s %s (%s)\n",
ui.RenderPass("✓"), formatFeedbackIDParen(fromID, lookupTitle(fromID)), depRelationFor(dt).phrase, formatFeedbackIDParen(toID, lookupTitle(toID)), dt)
ui.RenderPass("✓"), formatFeedbackIDParen(fromID, fromTitle), depRelationFor(dt).phrase, formatFeedbackIDParen(toID, toTitle), dt)
return nil
},
}

View file

@ -251,6 +251,36 @@ func runsPostCommandMaintenance(cmdName string, strictReadonly bool) bool {
return effectiveRootStorePolicy(cmdName, strictReadonly).runMaintenance
}
// readMemoEnvVar turns the per-command http read memo off when set to "0": an
// operator who suspects a stale read can rule the memo out without a rebuild.
const readMemoEnvVar = "BD_HTTP_READ_MEMO"
// commandMemoizesReads reports whether this invocation's http transport may
// answer an identical row or settings read from a response it already received
// (httpclient.WithReadMemo, bts-xpw.3.14.6). A one-shot command may: it reads,
// writes and exits within seconds, and each write it makes drops the memo.
//
// A process that outlives its reads may not, and bd serve is excluded for the
// reason runsPostCommandMaintenance excludes it — a server is not a command.
// Over an http workspace it answers other clients, who must see what other
// writers changed. A --watch or --follow poll exists to see exactly those
// changes, and a memo would answer its polls from a snapshot instead.
// BD_HTTP_READ_MEMO=0 turns the memo off for every command.
func commandMemoizesReads(cmd *cobra.Command) bool {
if os.Getenv(readMemoEnvVar) == "0" {
return false
}
if cmd.Name() == serveCmdName {
return false
}
for _, name := range []string{"watch", "follow"} {
if f := cmd.Flags().Lookup(name); f != nil && f.Value.String() == "true" {
return false
}
}
return true
}
// resolveDoltServerConnection fills in how to reach the workspace's Dolt SQL
// server — host, port, socket, user, password, TLS — on doltCfg.
//
@ -2128,8 +2158,20 @@ var rootCmd = &cobra.Command{
// swallowed the same way, which let a preview failure wave the
// real, mutating open through unexamined — the one thing this check
// exists to prevent (be-3bt2e). Those cases now refuse instead.
//
// openCtx scopes ONE http transport to the gate's preview open and the
// real open below (bts-xpw.3.14.6): over a gateway each dial pays its
// own handshake and, with a gasworks credential, its own sts.Context,
// so sharing it saves both on every write. A one-shot command's
// transport also answers its repeated row and settings reads from the
// response it already has (commandMemoizesReads). Other backends
// ignore both.
openCtx := httpclient.WithSharedConnections(rootCtx)
if commandMemoizesReads(cmd) {
openCtx = httpclient.WithReadMemo(openCtx)
}
if !useReadOnly && !globalFlag && os.Getenv("BEADS_SKIP_IDENTITY_CHECK") != "1" {
idStore, previewErr := newPreviewStoreFromConfig(rootCtx, beadsDir)
idStore, previewErr := newPreviewStoreFromConfig(openCtx, beadsDir)
switch {
case previewErr == nil:
checkErr := validateWorkspaceIdentity(rootCtx, idStore, beadsDir)
@ -2150,7 +2192,7 @@ var rootCmd = &cobra.Command{
// Dolt manages these files itself; external interference is never safe.
if _, ok := backends.Lookup(cfg.GetBackend()); ok {
store, err = newRegisteredBackendStore(rootCtx, cfg.GetBackend(), beadsDir, useReadOnly)
store, err = newRegisteredBackendStore(openCtx, cfg.GetBackend(), beadsDir, useReadOnly)
} else {
store, err = newDoltStore(rootCtx, doltCfg)
}

View file

@ -15,6 +15,7 @@ import (
"testing"
"time"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/steveyegge/beads/internal/configfile"
@ -137,6 +138,98 @@ func TestServeSkipsPostCommandMaintenance(t *testing.T) {
}
}
// TestOnlyOneShotCommandsMemoizeRemoteReads (bts-xpw.3.14.6): an http
// transport's read memo answers a repeated row read from a response already
// received, which is only right for a command that exits within seconds. bd
// serve answering other clients, and a --watch or --follow poll, would show a
// snapshot instead of the changes they exist to show.
func TestOnlyOneShotCommandsMemoizeRemoteReads(t *testing.T) {
command := func(name string, args ...string) *cobra.Command {
t.Helper()
cmd := &cobra.Command{Use: name}
cmd.Flags().BoolP("watch", "w", false, "")
cmd.Flags().Bool("follow", false, "")
if err := cmd.ParseFlags(args); err != nil {
t.Fatalf("%s %v: %v", name, args, err)
}
return cmd
}
for _, tc := range []struct {
cmd *cobra.Command
args string
want bool
}{
{command("show"), "show", true},
{command("list"), "list", true},
{command("update"), "update", true},
{command("show", "--watch"), "show --watch", false},
{command("list", "-w"), "list -w", false},
{command("show", "--watch=false"), "show --watch=false", true},
{command("tail", "--follow"), "events tail --follow", false},
{command(serveCmdName), "serve", false},
{&cobra.Command{Use: "close"}, "close (no watch flag at all)", true},
} {
if got := commandMemoizesReads(tc.cmd); got != tc.want {
t.Errorf("bd %s: commandMemoizesReads = %v, want %v", tc.args, got, tc.want)
}
}
}
// TestReadMemoOptInReadsTheRealPollingFlags (mc-5dkzp): the synthetic commands
// above pin the denylist's logic; this pins it against the commands that really
// poll. Renaming a polling flag, or giving it another spelling, must not quietly
// memoize a poll.
func TestReadMemoOptInReadsTheRealPollingFlags(t *testing.T) {
for _, tc := range []struct {
name string
cmd *cobra.Command
args []string
}{
{"show --watch", showCmd, []string{"--watch"}},
{"show -w", showCmd, []string{"-w"}},
{"list --watch", listCmd, []string{"--watch"}},
{"list -w", listCmd, []string{"-w"}},
{"events tail --follow", eventsTailCmd, []string{"--follow"}},
} {
t.Run(tc.name, func(t *testing.T) {
resetCommandFlags(tc.cmd)
t.Cleanup(func() { resetCommandFlags(tc.cmd) })
if !commandMemoizesReads(tc.cmd) {
t.Fatalf("bd %s without the flag: commandMemoizesReads = false, want true (a one-shot read)", tc.cmd.Name())
}
if err := tc.cmd.ParseFlags(tc.args); err != nil {
t.Fatalf("parse %v: %v", tc.args, err)
}
if commandMemoizesReads(tc.cmd) {
t.Errorf("bd %s: commandMemoizesReads = true, want false (a poll must see other writers)", tc.name)
}
})
}
if commandMemoizesReads(serveCmd) {
t.Errorf("bd serve: commandMemoizesReads = true, want false (a server answers other clients)")
}
}
// TestReadMemoKillSwitchTurnsTheMemoOff (mc-5dkzp): BD_HTTP_READ_MEMO=0 takes
// the memo away from every command, so an operator who suspects a stale read
// can rule it out without a rebuild. Any other value leaves the default.
func TestReadMemoKillSwitchTurnsTheMemoOff(t *testing.T) {
oneShot := &cobra.Command{Use: "show"}
for _, tc := range []struct {
value string
want bool
}{
{"", true},
{"1", true},
{"0", false},
} {
t.Setenv(readMemoEnvVar, tc.value)
if got := commandMemoizesReads(oneShot); got != tc.want {
t.Errorf("%s=%q: commandMemoizesReads(show) = %v, want %v", readMemoEnvVar, tc.value, got, tc.want)
}
}
}
// TestServeRefusalsPromiseNothing is the honesty gate on the mode gate. The
// refusal is typed so a caller can dispatch on it, and it must promise nothing
// — claiming otherwise sends an operator to do the wrong work.

View file

@ -153,7 +153,10 @@ pointless).`,
if cmd.Flags().Changed("status") {
status, _ := cmd.Flags().GetString("status")
var customStatuses []string
if store != nil {
// A built-in status is valid whatever the custom vocabulary says,
// so only a non-built-in one pays for reading it: over http that
// read is a gateway round trip (bts-xpw.3.14.6).
if store != nil && !types.Status(status).IsValid() {
cs, err := store.GetCustomStatuses(rootCtx)
if err != nil {
if !jsonOutput {
@ -1029,7 +1032,7 @@ func updateGuardsFromFlags(cmd *cobra.Command, claimFlag bool, updates map[strin
if cmd.Flags().Changed("if-status") {
v, _ := cmd.Flags().GetString("if-status")
var customStatuses []string
if store != nil {
if store != nil && !types.Status(v).IsValid() {
if cs, csErr := store.GetCustomStatuses(rootCtx); csErr == nil {
customStatuses = cs
}

View file

@ -504,6 +504,7 @@ Selected commonly-used variables:
| `BD_NO_PAGER`, `BD_PAGER` | Pager behavior |
| `BD_NON_INTERACTIVE` | Disable prompts |
| `BD_DEBUG` | Enable debug logging |
| `BD_HTTP_READ_MEMO` | `0` turns off the per-command read memo on an http workspace, so each repeated row or settings read in a command goes to the server; for ruling out a stale read |
| `BD_MIGRATION_FREEZE_FILE` | Check this exact path for the freeze marker instead of walking ancestor directories; authoritative when set (see [Migration Freeze](#migration-freeze)) |
| `BEADS_DIR` | Force the active beads workspace directory |
| `BEADS_CEILING_DIRECTORIES` | Directories (separated like `PATH`) that `.beads` and `config.yaml` discovery never looks at or above, like git's `GIT_CEILING_DIRECTORIES`; the starting directory is always examined. For sandboxes such as `bazel test` that must not reach the user's own `~/.beads` |

View file

@ -47,7 +47,7 @@ the behavior.
## Summary
261 rows: 201 refuse, 34 degrade, 26 retired.
262 rows: 201 refuse, 35 degrade, 26 retired.
## Whole-behavior divergences (D9 ledger)
@ -83,6 +83,7 @@ The L-rows the design's D9 table names: whole behaviors that differ over HTTP, e
| L-batchcreate-target | degrade | — | a batch create whose edge target names no issue refuses as issueops.ErrValidation alone, not as ErrValidation WRAPPING ErrNotFound | the wire answers a dangling edge target with a 400 invalid_argument naming `items` (internal/httpapi's failBatchCreate), and deliberately does not quote the role's own message, which arrives as a driver error naming tables and constraints. The refusal still fails the whole batch and creates nothing — the promise the row is about — and only the second sentinel is lost. Upstream ask: a distinguishing code for the absent-target refusal | D8 row 14 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline |
| L-batchcreate-inbatch | refuse | — | an edge onto an EARLIER ITEM OF THE SAME BATCH cannot be expressed at all over http | the capability needs the earlier item to have named an id for itself, and apigen.BatchCreateItem publishes no id member — an explicit id is refused with the rest of W-BatchCreateItem.Issue. So the target of an in-batch edge can only be a row the workspace already holds. It is a REFUSE row rather than a degrade because the request that asks for it fails: the client refuses the explicit id before the dial, and there is no second spelling that would land the edge silently. The wire's own `target_id` description names this case, so the gap is in the SCHEMA rather than in the handler — the upstream ask is an item id member, which retires this row | D8 row 14 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline |
| L18 | degrade | — | DOWN-LEVEL ONLY: against a server that does not advertise issues.batchClose, a single-id `bd close` still composes onto closeIssue and records the SERVER's own close commit message rather than the batch's id-naming entry; the multi-id and --claim-next shapes refuse there | the composed leg is one closeIssue call, so the history entry it leaves is the one that operation writes for itself — the batch's entry names the ids it closed and this one cannot, because no batch ran. Where issues.batchClose IS advertised the whole request is one call and one server-side transaction, so this divergence does not arise at all; the down-level leg survives on purpose, because `bd close` worked against pre-batchClose servers before this port and refusing it now would be a regression dressed as progress. It is the same posture L14 takes toward issues.claimNext. It is a degrade rather than a refuse because the close LANDS and the answer is neither narrower nor wider than the caller asked for — only the record of it is the operation's rather than the batch's. Retirement is a fleet fact rather than an upstream ask: the row goes when no server this client may meet is older than issues.batchClose | D9 L18, D8 row 17 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline |
| L-read-memo | degrade | — | inside one bd command, a repeated getIssue, getSetting or listSettings read of the same URL within ReadMemoTTL (5 s) of the server's answer is answered from that answer unless the command wrote in between; locally every re-read sees other writers at once | a command resolves an id, fetches the row and reads it again for output, and over a gateway each read is a sequential round trip (bd show --json made three requests for one row). The memo (wire.Client.EnableReadMemo, httpclient.WithReadMemo) answers the repeats from the first response, so the command sees the row as of that one read, which is what a single fetch would show. IT IS BOUNDED, NOT A CACHE: it lives on one process's transport and dies with it; only one-shot commands opt in (never bd serve, --watch or --follow); any non-GET through the transport clears it before it is sent and after it returns, and a read that overlapped a write is not stored, so a command always re-reads what its own write could have changed; only a 2xx answer that decoded is kept; an entry ages from the server's answer and a hit never renews it. Other writers' changes reach a later read of the same command only after the TTL, which is the residue this row records. A guarded write built from a memoized row (an expected revision, a CAS) fails safely on a version mismatch rather than losing an update. BD_HTTP_READ_MEMO=0 turns the memo off. Further pins outside this package's scan: internal/httpclient/wire TestReadMemoIsDroppedByAnyWrite, TestReadMemoKeepsOnlySuccessesAndExpires and TestReadMemoAgesFromTheServersAnswerNotTheLastHit; backend/http TestE2E_ShowFetchesTheIssueOnce and TestE2E_CloseRereadsItsOwnWrite | mc-5dkzp, bts-xpw.3.14.6 | TestOnlyOneShotCommandsMemoizeRemoteReads |
| L-create-notfound | degrade | — | a create whose dependency or waits-for target names no row refuses as issueops.ErrValidation ALONE — not as ErrValidation wrapping ErrNotFound — and does not name the target | the wire answers a dangling target with a 400 invalid_argument naming `dependencies` and a FIXED detail (internal/httpapi's failCreateIssue), which deliberately does not quote the role's own message: that message arrives as a driver error naming tables and constraints, and 4xx details on this surface reflect the caller's own input back rather than server internals. The refusal still fails the whole request and creates NOTHING — the promise the row is about — and only the second sentinel and the target's name are lost. It is batchCreateIssues' L-batchcreate-notfound on the single create, and it retires the same way: an upstream code that distinguishes the absent-target refusal and carries the target. A missing --parent target stays on this row beside dependency and waits-for targets, not ahead of them: OpCreateIssue's problem-code table documents NO 404 for ANY edge target on purpose (internal/httpapi/problem.go) — there is no id in the path for parent_id to have missed any more than dependencies[i].target_id or waits_for.spawner_id has, so carving a parent-only not_found out of this row would contradict that design rather than follow it. A prior draft of this row and its pin (TestServedCreateRefusesAnAbsentParentAsNotFound) described that carve-out as already shipped; it never was, and the S3 review that found the drift retired the draft rather than the design | D8 row 16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline |
| L-create-prefix | degrade | — | an unforced explicit id outside the workspace's configured prefix refuses as issueops.ErrValidation rather than as storage.ErrPrefixMismatch | the wire spells that refusal as a 400 invalid_argument with `param: "id"` and `reason: invalid_value` (internal/httpapi's failCreateIssue), which is the SAME pair a malformed or oversized `id` earns — so nothing on the wire tells the two apart, and reconstructing the typed sentinel from the pair would misclassify every other refusal of that member. The guard itself is untouched: the id is refused, nothing is created, and the detail names `force_id_prefix` as the bypass, which is the recovery a caller needs. What is lost is the errors.Is arm both local front doors use to decide whether to re-offer the create with --force. Upstream ask: a prefix_mismatch code, or a reason that distinguishes it, which retires this row | D8 row 16 | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline |
| L-apply-snapshot | degrade | — | every ItemResult of a batch apply comes back with Issue nil, where a local backend hydrates a post-item snapshot | the WIRE result is lean by the document's own decision and the role's leaf says why: ApplyItemResult carries ids, `changed` and `revision` and no issue, because the Go contract carries the snapshot for its COMPLETION HOOKS — which hand a script the row they are telling it about — and hooks never fire on the http surface at all. A hundred hydrated issues with their labels and edges would be a response an order of magnitude larger than the request that produced it. What a caller loses is a read it can make: the ids ARE published, per item and through Keys, so the rows are one getIssue away and the plan's next step is composed from ids rather than from snapshots. It is a degradation rather than a refusal because nothing about the WRITE differs — every item landed exactly as asked — and refusing would make the whole operation unavailable over http to keep a member no consumer on this surface can use | D8 refuse-not-drop (result half) | TODO(S3): store/dial-seam and cmd/bd conformance this package's own gates do not reach; see the in-repo divergence ledger, engdocs/design/http-divergence-ledger.md, for the discipline |

View file

@ -18,6 +18,7 @@ go_library(
"ca_permissions_unix.go",
"claimer.go",
"commenter.go",
"conn_share.go",
"counter.go",
"credential.go",
"deleter.go",
@ -105,6 +106,7 @@ go_test(
"ca_wrong_ca_test.go",
"completeness_test.go",
"composition_test.go",
"conn_share_test.go",
"counts_test.go",
"create_priority_test.go",
"credential_source_test.go",
@ -123,6 +125,8 @@ go_test(
"list_scope_test.go",
"list_search_test.go",
"list_sort_test.go",
"memo_key_test.go",
"memo_retry_test.go",
"metadatacas_test.go",
"molecule_test.go",
"open_with_test.go",

View file

@ -0,0 +1,122 @@
package httpclient
import (
"context"
"fmt"
"path/filepath"
"sync"
)
// connShareKey is the context key WithSharedConnections stores its memo under.
type connShareKey struct{}
// connShare memoizes the transports the registry opens dial within one scope.
type connShare struct {
mu sync.Mutex
conns map[string]WireClient
}
// WithSharedConnections returns a context under which every registry open
// (NewFromConfig, NewReadOnlyFromConfig) of the same workspace and target reuses
// ONE dialed transport, instead of dialing a fresh one per open
// (bts-xpw.3.14.6, beads-team-server docs/v0-gateway-latency.md option F).
//
// cmd/bd opens an http workspace twice per write: the pre-open identity gate
// previews it, then the command opens it for real. Two transports meant two
// handshakes (GET /v0/beads/context) and, on a gasworks gateway, two credential
// providers — each paying its own sts.Context discovery before its first mint.
// On the hosted /v0 path each of those is ~0.2-0.33 s, sequential. The
// transport carries the handshake cache and the credential provider, so sharing
// it makes the second open free: same server, same identity pin, same
// credential, already verified.
//
// The scope is explicit and per call site, never process-wide: only an open
// whose context carries this memo shares, so a multi-tenant embedder (OpenWith,
// which never reads it) and every test that opens with a plain context keep the
// one-dial-per-open behavior. A shared transport is keyed on the workspace AND
// every target field the dial reads (URL, identity pin, CA file, plaintext
// grant), so a sidecar rewritten between two opens dials afresh.
//
// Sharing alone changes no answer: a shared transport still sends every read.
// WithReadMemo is the separate, narrower opt-in that lets it answer a repeated
// read from the response it already has.
func WithSharedConnections(ctx context.Context) context.Context {
if ctx == nil {
ctx = context.Background()
}
if _, ok := ctx.Value(connShareKey{}).(*connShare); ok {
return ctx
}
return context.WithValue(ctx, connShareKey{}, &connShare{conns: map[string]WireClient{}})
}
// dialShared dials target through d, reusing a transport an earlier open under
// the same WithSharedConnections scope already dialed for this workspace and
// target. A failed dial is not memoized.
func dialShared(ctx context.Context, d WireDialer, beadsDir string, target Target) (WireClient, error) {
share, _ := ctx.Value(connShareKey{}).(*connShare)
if share == nil {
return d(ctx, target)
}
key := connShareKeyFor(beadsDir, target)
share.mu.Lock()
defer share.mu.Unlock()
if conn, ok := share.conns[key]; ok {
return conn, nil
}
conn, err := d(ctx, target)
if err != nil {
return nil, err
}
// A transport shared for a one-shot command may also collapse the
// identical row and settings reads that command repeats (WithReadMemo);
// any write through it drops them.
if memoize, _ := ctx.Value(readMemoKey{}).(bool); memoize {
if m, ok := conn.(readMemoizer); ok {
m.EnableReadMemo()
}
}
share.conns[key] = conn
return conn, nil
}
// readMemoizer is the optional transport half WithReadMemo enables;
// *wire.Client (and so *Conn) implements it.
type readMemoizer interface{ EnableReadMemo() }
// readMemoKey is the context key WithReadMemo stores its opt-in under.
type readMemoKey struct{}
// WithReadMemo marks ctx so that a transport a WithSharedConnections scope
// dials under it also memoizes identical single-row and settings reads for the
// rest of its life (wire.Client.EnableReadMemo, wire.ReadMemoTTL;
// bts-xpw.3.14.6). It is for a ONE-SHOT command: one that reads, writes and
// exits within seconds, where an identical re-read is the same question asked
// twice and every write it makes drops the memo first.
//
// A process that outlives its reads must not ask for it. bd serve over an http
// workspace answers other clients, who must see what other writers changed,
// and a --watch poll exists to see exactly those changes; either one answered
// from a memo shows a snapshot up to a TTL old.
//
// It takes effect only inside a WithSharedConnections scope, because sharing is
// what puts every open of the workspace on the one transport whose writes clear
// the memo its reads are answered from. The first open of a shared transport
// decides; the order the two wrappers are applied in does not matter.
func WithReadMemo(ctx context.Context) context.Context {
if ctx == nil {
ctx = context.Background()
}
return context.WithValue(ctx, readMemoKey{}, true)
}
func connShareKeyFor(beadsDir string, target Target) string {
if abs, err := filepath.Abs(beadsDir); err == nil {
beadsDir = abs
}
base := ""
if target.BaseURL != nil {
base = target.BaseURL.String()
}
return fmt.Sprintf("%q|%q|%q|%q|%t", beadsDir, base, target.ExpectProjectID, target.CAFile, target.AllowInsecureCredential)
}

View file

@ -0,0 +1,107 @@
package httpclient
import (
"context"
"testing"
"github.com/steveyegge/beads/internal/httpapi/apigen"
)
// TestSharedConnectionsDialOncePerWorkspaceTarget pins WithSharedConnections'
// key (bts-xpw.3.14.6): within one scope, the registry's two opens of the same
// workspace share one transport; a rewritten sidecar (another server, or another
// identity pin) dials afresh; and outside any scope every open dials its own.
func TestSharedConnectionsDialOncePerWorkspaceTarget(t *testing.T) {
dir := t.TempDir()
target := pinnedProjectTarget(t)
if err := SaveTarget(dir, target); err != nil {
t.Fatalf("SaveTarget: %v", err)
}
dials := 0
swapDialer(t, func(context.Context, Target) (WireClient, error) {
dials++
return &fakeWire{res: &apigen.ContextResponse{ProjectId: "proj-1"}}, nil
})
ctx := WithSharedConnections(context.Background())
if WithSharedConnections(ctx) != ctx {
t.Error("re-scoping an already scoped context started a second memo")
}
a, err := NewReadOnlyFromConfig(ctx, dir)
if err != nil {
t.Fatal(err)
}
b, err := NewFromConfig(ctx, dir)
if err != nil {
t.Fatal(err)
}
if dials != 1 || a.(*Store).wire != b.(*Store).wire {
t.Fatalf("two scoped opens of one workspace dialed %d times; want one shared transport", dials)
}
target.ExpectProjectID = "proj-2"
if err := SaveTarget(dir, target); err != nil {
t.Fatalf("SaveTarget: %v", err)
}
if _, err := NewFromConfig(ctx, dir); err != nil {
t.Fatal(err)
}
if dials != 2 {
t.Fatalf("a re-pinned sidecar reused the old transport (%d dials)", dials)
}
for range 2 {
if _, err := NewFromConfig(context.Background(), dir); err != nil {
t.Fatal(err)
}
}
if dials != 4 {
t.Fatalf("unscoped opens shared a transport (%d dials, want 4)", dials)
}
}
// memoWire is a fakeWire that records whether an open turned its read memo on.
type memoWire struct {
fakeWire
memoized bool
}
func (w *memoWire) EnableReadMemo() { w.memoized = true }
// TestOnlyAReadMemoScopeMemoizesSharedTransports pins the memo as its own
// opt-in (bts-xpw.3.14.6): cmd/bd shares the gate's transport with every
// command, bd serve and --watch polls included, but only a one-shot command
// asks for the memo, so a shared transport memoizes nothing unless WithReadMemo
// says so — and WithReadMemo outside a sharing scope memoizes nothing either,
// since an unshared open's writes would not clear its sibling open's memo.
func TestOnlyAReadMemoScopeMemoizesSharedTransports(t *testing.T) {
dir := t.TempDir()
if err := SaveTarget(dir, pinnedProjectTarget(t)); err != nil {
t.Fatalf("SaveTarget: %v", err)
}
var last *memoWire
swapDialer(t, func(context.Context, Target) (WireClient, error) {
last = &memoWire{fakeWire: fakeWire{res: &apigen.ContextResponse{ProjectId: "proj-1"}}}
return last, nil
})
for _, tc := range []struct {
name string
ctx context.Context
want bool
}{
{"shared only", WithSharedConnections(context.Background()), false},
{"shared with a read memo", WithReadMemo(WithSharedConnections(context.Background())), true},
{"read memo wrapped first", WithSharedConnections(WithReadMemo(context.Background())), true},
{"read memo without a sharing scope", WithReadMemo(context.Background()), false},
{"neither", context.Background(), false},
} {
last = nil
if _, err := NewFromConfig(tc.ctx, dir); err != nil {
t.Fatalf("%s: %v", tc.name, err)
}
if last == nil || last.memoized != tc.want {
t.Errorf("%s: the open's transport memoized reads = %v, want %v", tc.name, last != nil && last.memoized, tc.want)
}
}
}

View file

@ -521,6 +521,17 @@ func designRows() []Row {
// refuses without dialing.
PinnedBy: pinnedByS3Conformance,
},
{
ID: "L-read-memo", Kind: KindDegrade,
What: "inside one bd command, a repeated getIssue, getSetting or listSettings read of the same URL within ReadMemoTTL (5 s) of the server's answer is answered from that answer unless the command wrote in between; locally every re-read sees other writers at once",
Why: "a command resolves an id, fetches the row and reads it again for output, and over a gateway each read is a sequential round trip (bd show --json made three requests for one row). The memo (wire.Client.EnableReadMemo, httpclient.WithReadMemo) answers the repeats from the first response, so the command sees the row as of that one read, which is what a single fetch would show. " +
"IT IS BOUNDED, NOT A CACHE: it lives on one process's transport and dies with it; only one-shot commands opt in (never bd serve, --watch or --follow); any non-GET through the transport clears it before it is sent and after it returns, and a read that overlapped a write is not stored, so a command always re-reads what its own write could have changed; only a 2xx answer that decoded is kept; an entry ages from the server's answer and a hit never renews it. " +
"Other writers' changes reach a later read of the same command only after the TTL, which is the residue this row records. A guarded write built from a memoized row (an expected revision, a CAS) fails safely on a version mismatch rather than losing an update. BD_HTTP_READ_MEMO=0 turns the memo off. " +
"Further pins outside this package's scan: internal/httpclient/wire TestReadMemoIsDroppedByAnyWrite, TestReadMemoKeepsOnlySuccessesAndExpires and TestReadMemoAgesFromTheServersAnswerNotTheLastHit; backend/http TestE2E_ShowFetchesTheIssueOnce and TestE2E_CloseRereadsItsOwnWrite",
SpecRow: "mc-5dkzp, bts-xpw.3.14.6",
// The bound the row rests on: only a one-shot command memoizes.
PinnedBy: "TestOnlyOneShotCommandsMemoizeRemoteReads",
},
}
}

View file

@ -0,0 +1,56 @@
package httpclient
import (
"context"
"net/http"
"testing"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/issueops"
)
// TestDefaultIssueReadsShareOneMemoKey (mc-5dkzp): every default-shaped read
// of one issue that bd show makes — and every resolve-then-get command with it
// — must be the SAME request. The per-command read memo is keyed by the full
// URL, so a read that grows a parameter of its own (even one spelling its
// default, like brief_deps=false) silently brings the extra round trip back.
// This pins the five reads to one memo key without a server.
func TestDefaultIssueReadsShareOneMemoKey(t *testing.T) {
s, w := recordingStore(t)
ctx := context.Background()
const id = "bd-1"
// The id resolver's exact probe (utils.ResolvePartialID over an
// exact-only store).
if _, err := s.SearchIssues(ctx, "", types.IssueFilter{IDs: []string{id}}); err != nil {
t.Fatal(err)
}
// cmd/bd/routed.go's fetch of the resolved id.
if _, err := s.GetIssue(ctx, id); err != nil {
t.Fatal(err)
}
// Text show's dependency and label sections.
if _, err := s.GetDependenciesWithMetadata(ctx, id); err != nil {
t.Fatal(err)
}
if _, err := s.GetLabels(ctx, id); err != nil {
t.Fatal(err)
}
// bd show --json's detail read with default flags.
rd, err := s.IssueReader()
if err != nil {
t.Fatal(err)
}
if _, err := rd.Get(ctx, issueops.GetRequest{ID: id}); err != nil {
t.Fatal(err)
}
if len(w.requests) != 5 {
t.Fatalf("dispatched %d requests (%v), want 5", len(w.requests), w.dispatched)
}
want := "getIssue " + http.MethodGet + " /v0/beads/issues/" + id + "?"
for i, r := range w.requests {
if got := r.Op + " " + r.Method + " " + r.Path + "?" + r.Query.Encode(); got != want {
t.Errorf("read %d dispatched %q; every default issue read must be %q (one memo key)", i, got, want)
}
}
}

View file

@ -0,0 +1,67 @@
package httpclient
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync/atomic"
"testing"
"github.com/steveyegge/beads/internal/httpclient/wire"
)
// TestDispatchRetryOfABusyMemoizedReadAsksTheServer (mc-5dkzp): dispatch comes
// back to a busy read through wire.Client.Do again (dispatch.go's retry loop).
// With the per-command read memo on, the retry must reach the server rather
// than be answered with the 503 the first attempt got, and the 200 it gets is
// what the command's later identical reads are answered from.
func TestDispatchRetryOfABusyMemoizedReadAsksTheServer(t *testing.T) {
var gets atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == wire.PathContext:
ctx := v0Context("proj-busy")
ctx.Capabilities = []string{"issues.get"}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(ctx)
case strings.HasPrefix(r.URL.Path, wire.PathIssues+"/"):
if gets.Add(1) == 1 {
w.Header().Set("Content-Type", "application/problem+json; charset=utf-8")
w.WriteHeader(http.StatusServiceUnavailable)
_, _ = io.WriteString(w, `{"status":503,"code":"busy","title":"busy"}`)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"id":"bd-1","title":"answered after a 503","revision":"7"}`)
default:
http.NotFound(w, r)
}
}))
t.Cleanup(srv.Close)
base, err := url.Parse(srv.URL)
if err != nil {
t.Fatal(err)
}
client, err := wire.New(base, nil, wire.Options{HTTPClient: srv.Client()})
if err != nil {
t.Fatal(err)
}
client.EnableReadMemo()
store := New(Target{BaseURL: base}, client, nil)
issue, err := store.GetIssue(t.Context(), "bd-1")
if err != nil {
t.Fatalf("GetIssue after one 503: %v (getIssue requests = %d); the retry must reach the server", err, gets.Load())
}
if issue.Title != "answered after a 503" || gets.Load() != 2 {
t.Fatalf("GetIssue after one 503 answered %q in %d getIssue requests; want the retry's answer in 2", issue.Title, gets.Load())
}
again, err := store.GetIssue(t.Context(), "bd-1")
if err != nil || again.Title != issue.Title || gets.Load() != 2 {
t.Fatalf("the repeat read answered (%v, %v) in %d getIssue requests; want the retry's 200 from the memo (2 requests)", again, err, gets.Load())
}
}

View file

@ -181,7 +181,7 @@ func open(ctx context.Context, beadsDir string) (storage.DoltStorage, error) {
if dialer == nil {
return nil, fmt.Errorf("%w: cannot dial %s", ErrNoTransport, target)
}
wire, err := dialer(ctx, target)
wire, err := dialShared(ctx, dialer, beadsDir, target)
if err != nil {
return nil, err
}

View file

@ -192,7 +192,20 @@ const infraTypesKey = "types.infra"
// settingOrEmpty reads one setting, treating an unset key as the empty string —
// which is what the role already promises — and reporting only a real failure.
//
// Under a read memo (a one-shot CLI command's transport, WithReadMemo) it
// reads the whole settings page instead, once: `bd list` asks for three
// vocabulary keys back to back, and one memoized listSettings answers all three
// where three getSettings cost three sequential round trips (bts-xpw.3.14.6).
// The two reads agree on every key this serves: an unset key and a redacted one
// both read as "" either way. A failed page read (a server without config.list)
// falls back to the per-key read.
func (s *Store) settingOrEmpty(ctx context.Context, key string) (string, error) {
if s.readsMemoized() {
if all, err := s.GetAllConfig(ctx); err == nil {
return all[key], nil
}
}
value, err := s.GetConfig(ctx, key)
if errors.Is(err, ErrSettingRedacted) {
// A vocabulary key an operator marked credential-bearing is a
@ -287,3 +300,11 @@ type vocabularyCache struct {
// isNotFound reports the wire's not_found, which several roles turn back into a
// result rather than an error.
func isNotFound(err error) bool { return errors.Is(err, issueops.ErrNotFound) }
// readsMemoized reports whether this store's transport memoizes repeated reads
// (wire.Client.EnableReadMemo), which only a one-shot command's open turns on
// (WithReadMemo).
func (s *Store) readsMemoized() bool {
m, ok := s.wire.(interface{ ReadMemoEnabled() bool })
return ok && m.ReadMemoEnabled()
}

View file

@ -9,6 +9,7 @@ go_library(
"handshake.go",
"paths.go",
"problem.go",
"read_memo.go",
"revision_tolerance.go",
"stall_retry.go",
"stream.go",
@ -37,6 +38,7 @@ go_test(
"problem_test.go",
"problem_wire_revision_test.go",
"project_stamp_test.go",
"read_memo_test.go",
"revision_tolerance_test.go",
"skew_test.go",
"stall_retry_test.go",

View file

@ -139,6 +139,11 @@ type Client struct {
handshake handshakeCache
expectID string
// memo, when EnableReadMemo turned it on, answers a repeated identical
// single-row or settings read from the response already received
// (read_memo.go). Nil is off, the default.
memo *readMemo
}
// New builds a client for base.
@ -321,9 +326,35 @@ func (c *Client) Do(ctx context.Context, r Request, out any) error {
return fmt.Errorf("%s: %w", r.Op, err)
}
res, err := c.roundTrip(ctx, r, u, body)
if err != nil {
return err
var (
res *response
// memoKey is set only for a memoable read that MISSED, whose answer is
// then stored under it at memoGen. A hit is never stored again: that
// would renew its age (read_memo.go, put).
memoKey string
memoGen uint64
)
switch {
case c.memo == nil:
case memoable(r):
key := u.String()
if res, memoGen = c.memo.lookup(key); res == nil {
memoKey = key
}
case r.Method != http.MethodGet:
// A write may change anything a memoized read answered: drop them all
// before it is sent, and again once it returns. Each clear starts a new
// generation, so a read that overlapped the write cannot store a
// pre-write answer behind it either.
c.memo.clear()
defer c.memo.clear()
}
if res == nil {
var err error
res, err = c.roundTrip(ctx, r, u, body)
if err != nil {
return err
}
}
// The rotation window, and the whole of it. The server re-reads its token
@ -355,6 +386,8 @@ func (c *Client) Do(ctx context.Context, r Request, out any) error {
return prob
}
if out == nil {
// Nothing decoded the answer, so nothing vouches for it: it is not
// memoized (every memoable read in this package decodes its answer).
return nil
}
if len(bytes.TrimSpace(res.body)) == 0 {
@ -372,6 +405,12 @@ func (c *Client) Do(ctx context.Context, r Request, out any) error {
if err := json.Unmarshal(decodeBody, out); err != nil {
return fmt.Errorf("%s: decoding the response from bd serve at %s: %w", r.Op, c.base.Redacted(), err)
}
if memoKey != "" {
// Only an answer that decoded is memoized. A 2xx with an empty or
// undecodable body is the server misbehaving once; storing it would
// fail every identical read for a whole TTL instead of asking again.
c.memo.put(memoKey, res, memoGen)
}
return nil
}

View file

@ -0,0 +1,124 @@
package wire
import (
"net/http"
"sync"
"time"
)
// ReadMemoTTL bounds how long a memoized read answers for, counted from when
// the server answered it: a hit never renews it. The memo exists to collapse
// the identical reads ONE bd command makes back to back (an id resolution,
// then the command's own fetch of the same row, then its output render), all
// within a second or two; it is not a cache of the workspace. An entry older
// than this is re-read, so even a command that stays alive (an editor session
// between two reads) never answers from a view older than this.
const ReadMemoTTL = 5 * time.Second
// memoableOps are the reads the memo serves: single-row and settings reads,
// which a command repeats verbatim. Listings, counts and the ready queue are
// not memoized — a command reads each once, and they are where a concurrent
// writer's change matters most.
var memoableOps = map[string]bool{
OpGetIssue: true,
OpGetSetting: true,
OpListSettings: true,
}
// readMemo holds whole 2xx responses to memoable GETs, keyed by their full URL
// (path AND query: getIssue with include_comments is a different answer from a
// bare getIssue). Responses are stored as the raw bytes the server sent, and
// every hit is decoded afresh by Do, so no two callers share a decoded value
// and the decode path (revision tolerance included) is the same as a miss's.
type readMemo struct {
mu sync.Mutex
now func() time.Time
// gen counts the clears. A read takes it when it misses and stores its
// answer only if no clear has happened since: a read whose request was in
// flight while a write went through may carry the pre-write row even when
// it returns after the write's own clears, and storing that row would hand
// it to every later read for a whole TTL.
gen uint64
entries map[string]memoEntry
}
type memoEntry struct {
res *response
at time.Time
}
// EnableReadMemo turns on per-client memoization of identical single-row and
// settings reads (bts-xpw.3.14.6). It is opt-in and meant for a client whose
// lifetime is one CLI command: cmd/bd enables it through httpclient.WithReadMemo,
// for one-shot commands only. Any non-GET request through the client drops
// every memoized answer, before it is sent and again after it returns, and an
// answer to a read that overlapped such a request is not stored, so a command
// always re-reads what its own write could have changed.
//
// Call it before the client is shared; it is idempotent.
func (c *Client) EnableReadMemo() {
if c.memo != nil {
return
}
c.memo = &readMemo{now: time.Now, entries: map[string]memoEntry{}}
}
// ReadMemoEnabled reports whether EnableReadMemo has been called. A caller
// with a choice of reads (one listSettings, or one getSetting per key) uses it
// to prefer the read a memo can answer again.
func (c *Client) ReadMemoEnabled() bool { return c.memo != nil }
// lookup returns the live answer memoized under key, or nil, together with the
// memo's generation as of the lookup, which a miss hands back to put.
func (m *readMemo) lookup(key string) (*response, uint64) {
m.mu.Lock()
defer m.mu.Unlock()
e, ok := m.entries[key]
if !ok {
return nil, m.gen
}
if m.now().Sub(e.at) > ReadMemoTTL {
delete(m.entries, key)
return nil, m.gen
}
return e.res, m.gen
}
// put stores the server's answer to a read that missed at generation gen. Only
// a miss's answer is ever stored, never a hit's: an entry's age is its
// answer's, and re-storing a hit would let a row read at least once per TTL
// answer from its first read forever. A non-2xx answer is not stored (a
// missing row may be created next), nor is one whose read overlapped a clear.
// Expired entries are swept on the way in, so however long the client lives
// the memo holds at most one TTL's worth of answers.
func (m *readMemo) put(key string, res *response, gen uint64) {
if res == nil || res.status < 200 || res.status >= 300 {
return
}
m.mu.Lock()
defer m.mu.Unlock()
if gen != m.gen {
return
}
now := m.now()
for k, e := range m.entries {
if now.Sub(e.at) > ReadMemoTTL {
delete(m.entries, k)
}
}
m.entries[key] = memoEntry{res: res, at: now}
}
// clear drops every memoized answer and starts a new generation, so a read
// already in flight cannot store its answer either.
func (m *readMemo) clear() {
m.mu.Lock()
defer m.mu.Unlock()
m.gen++
clear(m.entries)
}
// memoable reports whether r is a read the memo may answer.
func memoable(r Request) bool {
return r.Method == http.MethodGet && memoableOps[r.Op]
}

View file

@ -0,0 +1,450 @@
package wire
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"sync"
"testing"
"time"
)
// memoServer answers getIssue for bd-1 (a fresh title per request, so a hit is
// told apart from a re-read), a 404 for anything else, and 200 to any write.
func memoServer(t *testing.T) (*Client, *recorder) {
t.Helper()
served := 0
return newTestClient(t, Options{}, nil, func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if r.Method != http.MethodGet {
_, _ = w.Write([]byte(`{}`))
return
}
if r.URL.Path != "/v0/beads/issues/bd-1" {
w.Header().Set("Content-Type", "application/problem+json")
w.WriteHeader(http.StatusNotFound)
_, _ = w.Write([]byte(`{"type":"about:blank","title":"Not Found","status":404,"code":"not_found","detail":"no such issue"}`))
return
}
served++
_ = json.NewEncoder(w).Encode(map[string]any{"id": "bd-1", "title": "read " + string(rune('0'+served))})
})
}
func memoGetIssue(t *testing.T, c *Client, id string, q url.Values) (string, error) {
t.Helper()
path, err := IssuePath(id)
if err != nil {
t.Fatal(err)
}
var out struct {
Title string `json:"title"`
}
err = c.Do(context.Background(), Request{Op: OpGetIssue, Method: http.MethodGet, Path: path, Query: q, IssueID: id}, &out)
return out.Title, err
}
// TestReadMemoCollapsesRepeatedRowReads is bts-xpw.3.14.6's per-command memo:
// once enabled, an identical getIssue is answered from the response already
// received, a different query is a different read, and every answer is decoded
// afresh (the second caller gets its own value, equal to the first).
func TestReadMemoCollapsesRepeatedRowReads(t *testing.T) {
c, rec := memoServer(t)
c.EnableReadMemo()
first, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil {
t.Fatal(err)
}
second, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil {
t.Fatal(err)
}
if rec.count() != 1 || first != second {
t.Fatalf("two identical reads made %d requests and answered %q then %q; want one request, one answer", rec.count(), first, second)
}
withComments := url.Values{"include_comments": {"true"}}
if _, err := memoGetIssue(t, c, "bd-1", withComments); err != nil {
t.Fatal(err)
}
if rec.count() != 2 {
t.Fatalf("a read with a different query was answered from the memo (%d requests)", rec.count())
}
}
// TestReadMemoIsDroppedByAnyWrite: a write through the client may change what a
// memoized read said, so the next read goes back to the server — whatever the
// write's method, since the wire spells writes as all four.
func TestReadMemoIsDroppedByAnyWrite(t *testing.T) {
issuePath, _ := IssuePath("bd-1")
settingPath, _ := SettingPath("k")
for _, write := range []Request{
{Op: OpUpdateIssue, Method: http.MethodPatch, Path: issuePath, Body: map[string]any{}},
{Op: OpCreateIssue, Method: http.MethodPost, Path: PathIssues, Body: map[string]any{}},
{Op: OpSetSetting, Method: http.MethodPut, Path: settingPath, Body: map[string]any{}},
{Op: OpUnsetSetting, Method: http.MethodDelete, Path: settingPath},
} {
t.Run(write.Method, func(t *testing.T) {
c, rec := memoServer(t)
c.EnableReadMemo()
before, _ := memoGetIssue(t, c, "bd-1", nil)
if err := c.Do(context.Background(), write, nil); err != nil {
t.Fatal(err)
}
after, _ := memoGetIssue(t, c, "bd-1", nil)
if rec.count() != 3 || before == after {
t.Fatalf("read after a %s: %d requests, %q then %q; want the row re-read", write.Method, rec.count(), before, after)
}
})
}
}
// TestReadMemoKeepsOnlySuccessesAndExpires: a miss is re-asked (it may have been
// created since), and an entry older than ReadMemoTTL is re-read.
func TestReadMemoKeepsOnlySuccessesAndExpires(t *testing.T) {
c, rec := memoServer(t)
c.EnableReadMemo()
now := time.Unix(1_000_000, 0)
c.memo.now = func() time.Time { return now }
for range 2 {
if _, err := memoGetIssue(t, c, "bd-404", nil); err == nil {
t.Fatal("a missing row read without an error")
}
}
if rec.count() != 2 {
t.Fatalf("a 404 was memoized: %d requests for two reads", rec.count())
}
_, _ = memoGetIssue(t, c, "bd-1", nil)
now = now.Add(ReadMemoTTL + time.Second)
_, _ = memoGetIssue(t, c, "bd-1", nil)
if rec.count() != 4 {
t.Fatalf("an expired entry answered: %d requests, want 4", rec.count())
}
}
// TestReadMemoIsOffByDefault: a client nobody opted in re-reads every time —
// the memo is for one command's transport, never a long-lived one's.
func TestReadMemoIsOffByDefault(t *testing.T) {
c, rec := memoServer(t)
_, _ = memoGetIssue(t, c, "bd-1", nil)
_, _ = memoGetIssue(t, c, "bd-1", nil)
if rec.count() != 2 || c.ReadMemoEnabled() {
t.Fatalf("an un-enabled client memoized (%d requests)", rec.count())
}
}
// TestReadMemoAgesFromTheServersAnswerNotTheLastHit: an entry's age is the age
// of the answer the server gave, and a hit does not renew it. A row read more
// often than ReadMemoTTL (bd show --watch polls every 2 s) is still re-read
// once its answer is a TTL old; were each hit to re-stamp the entry, it would
// answer from the first read for as long as the polling kept up.
func TestReadMemoAgesFromTheServersAnswerNotTheLastHit(t *testing.T) {
c, rec := memoServer(t)
c.EnableReadMemo()
now := time.Unix(1_000_000, 0)
c.memo.now = func() time.Time { return now }
var titles []string
for range 4 { // t = 0 s (miss), 2 s (hit), 4 s (hit), 6 s (a TTL old: re-read)
title, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil {
t.Fatal(err)
}
titles = append(titles, title)
now = now.Add(2 * time.Second)
}
if rec.count() != 2 || titles[3] == titles[0] {
t.Fatalf("a row polled every 2 s made %d requests and answered %q; want it re-read at 6 s (2 requests)", rec.count(), titles)
}
}
// TestReadMemoDropsAnAnswerThatOverlappedAWrite: a read whose request was in
// flight while a write went through the client may carry the pre-write row,
// even though it returned after the write did. Storing it would answer every
// later read with that row for a whole TTL — the command's own write unseen.
func TestReadMemoDropsAnAnswerThatOverlappedAWrite(t *testing.T) {
arrived, release := make(chan struct{}), make(chan struct{})
var mu sync.Mutex
served := 0
c, rec := newTestClient(t, Options{ResponseHeaderTimeout: -1}, nil, func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if r.Method != http.MethodGet {
_, _ = w.Write([]byte(`{}`))
return
}
mu.Lock()
served++
n := served
mu.Unlock()
if n == 1 {
close(arrived)
<-release
}
_ = json.NewEncoder(w).Encode(map[string]any{"id": "bd-1", "title": fmt.Sprintf("read %d", n)})
})
c.EnableReadMemo()
path, err := IssuePath("bd-1")
if err != nil {
t.Fatal(err)
}
type answer struct {
title string
err error
}
done := make(chan answer, 1)
go func() {
var out struct {
Title string `json:"title"`
}
err := c.Do(context.Background(), Request{Op: OpGetIssue, Method: http.MethodGet, Path: path, IssueID: "bd-1"}, &out)
done <- answer{out.Title, err}
}()
<-arrived // the read is on the wire; the write starts and finishes while it waits
if err := c.Do(context.Background(), Request{Op: OpUpdateIssue, Method: http.MethodPatch, Path: path, Body: map[string]any{}}, nil); err != nil {
close(release)
t.Fatal(err)
}
close(release)
overlapped := <-done
if overlapped.err != nil {
t.Fatal(overlapped.err)
}
after, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil {
t.Fatal(err)
}
if rec.count() != 3 || after == overlapped.title {
t.Fatalf("the read after the write answered %q (the overlapped read's %q) in %d requests; want it re-read (3 requests)", after, overlapped.title, rec.count())
}
}
// TestReadMemoHoldsAtMostOneTTLOfAnswers: expired entries are swept when a new
// answer is stored, so a client that lives on (and keeps reading new rows)
// does not keep every answer it ever received.
func TestReadMemoHoldsAtMostOneTTLOfAnswers(t *testing.T) {
c, _ := memoServer(t)
c.EnableReadMemo()
now := time.Unix(1_000_000, 0)
c.memo.now = func() time.Time { return now }
if _, err := memoGetIssue(t, c, "bd-1", nil); err != nil {
t.Fatal(err)
}
now = now.Add(ReadMemoTTL + time.Second)
if _, err := memoGetIssue(t, c, "bd-1", url.Values{"include_comments": {"true"}}); err != nil {
t.Fatal(err)
}
c.memo.mu.Lock()
held := len(c.memo.entries)
c.memo.mu.Unlock()
if held != 1 {
t.Fatalf("the memo holds %d answers after one expired and one was stored; want 1", held)
}
}
// TestReadMemoNeverKeepsA5xx: a 5xx is not an answer to remember. A 503 above
// all — dispatch comes back to a busy read through Do again (dispatch.go's
// retry loop), and a memoized 503 would answer every retry with the same
// refusal. The 2xx that follows it is the read's answer, and that one is kept.
func TestReadMemoNeverKeepsA5xx(t *testing.T) {
for _, status := range []int{http.StatusServiceUnavailable, http.StatusInternalServerError, http.StatusBadGateway} {
t.Run(strconv.Itoa(status), func(t *testing.T) {
var mu sync.Mutex
served := 0
c, rec := newTestClient(t, Options{}, nil, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
served++
n := served
mu.Unlock()
if n == 1 {
problemJSON(w, status, fmt.Sprintf(`{"type":"about:blank","title":"failing","status":%d,"detail":"try again"}`, status))
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"id": "bd-1", "title": fmt.Sprintf("read %d", n)})
})
c.EnableReadMemo()
_, err := memoGetIssue(t, c, "bd-1", nil)
var prob *ProblemError
if !errors.As(err, &prob) || prob.Status != status {
t.Fatalf("first read: %v, want the server's %d", err, status)
}
again, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil || rec.count() != 2 {
t.Fatalf("the read after a %d answered (%q, %v) in %d requests; want the server asked again (2 requests)", status, again, err, rec.count())
}
third, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil || third != again || rec.count() != 2 {
t.Fatalf("the 2xx after a %d was not memoized: %q then %q in %d requests", status, again, third, rec.count())
}
})
}
}
// TestReadMemoKeepsOnlyAnswersThatDecoded: a 2xx is memoized only once it has
// decoded. An empty or undecodable body is the server misbehaving once, and
// keeping it would fail every identical read for a whole TTL rather than ask
// again. A read that decoded nothing (out == nil) vouches for nothing either.
func TestReadMemoKeepsOnlyAnswersThatDecoded(t *testing.T) {
for _, tc := range []struct {
name string
body string
noOut bool
}{
{name: "empty body", body: ""},
{name: "undecodable body", body: `{"id":"bd-1","title":`},
{name: "answer nobody decoded", body: `{"id":"bd-1","title":"undecoded"}`, noOut: true},
} {
t.Run(tc.name, func(t *testing.T) {
var mu sync.Mutex
served := 0
c, rec := newTestClient(t, Options{}, nil, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
served++
n := served
mu.Unlock()
w.Header().Set("Content-Type", "application/json")
if n == 1 {
_, _ = w.Write([]byte(tc.body))
return
}
_ = json.NewEncoder(w).Encode(map[string]any{"id": "bd-1", "title": fmt.Sprintf("read %d", n)})
})
c.EnableReadMemo()
path, err := IssuePath("bd-1")
if err != nil {
t.Fatal(err)
}
if tc.noOut {
if err := c.Do(context.Background(), Request{Op: OpGetIssue, Method: http.MethodGet, Path: path, IssueID: "bd-1"}, nil); err != nil {
t.Fatal(err)
}
} else if _, err := memoGetIssue(t, c, "bd-1", nil); err == nil {
t.Fatalf("a 200 with body %q decoded without an error", tc.body)
}
title, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil || title != "read 2" || rec.count() != 2 {
t.Fatalf("the read after a 200 with body %q answered (%q, %v) in %d requests; want the server asked again (\"read 2\", 2 requests)", tc.body, title, err, rec.count())
}
})
}
}
// TestReadMemoIsPerClient: each client holds its own memo. In either order, a
// read through one client is never answered from what another client read,
// and turning the memo on for one client turns it on for no other.
func TestReadMemoIsPerClient(t *testing.T) {
for _, first := range []int{0, 1} {
t.Run(fmt.Sprintf("client %d reads first", first), func(t *testing.T) {
var mu sync.Mutex
served := 0
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
served++
n := served
mu.Unlock()
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"id": "bd-1", "title": fmt.Sprintf("read %d", n)})
}))
t.Cleanup(srv.Close)
base, err := url.Parse(srv.URL)
if err != nil {
t.Fatal(err)
}
var clients [3]*Client
for i := range clients {
if clients[i], err = New(base, nil, Options{HTTPClient: srv.Client()}); err != nil {
t.Fatal(err)
}
}
clients[0].EnableReadMemo()
clients[1].EnableReadMemo()
// clients[2] never opts in.
requests := func() int {
mu.Lock()
defer mu.Unlock()
return served
}
a, b := clients[first], clients[1-first]
a1, _ := memoGetIssue(t, a, "bd-1", nil)
b1, _ := memoGetIssue(t, b, "bd-1", nil)
if requests() != 2 || a1 == b1 {
t.Fatalf("two clients' first reads made %d requests and answered %q and %q; want each to ask the server", requests(), a1, b1)
}
a2, _ := memoGetIssue(t, a, "bd-1", nil)
b2, _ := memoGetIssue(t, b, "bd-1", nil)
if requests() != 2 || a2 != a1 || b2 != b1 {
t.Fatalf("each client's repeat read: %d requests, %q/%q then %q/%q; want each answered from its own memo", requests(), a1, b1, a2, b2)
}
for range 2 {
_, _ = memoGetIssue(t, clients[2], "bd-1", nil)
}
if requests() != 4 || clients[2].ReadMemoEnabled() {
t.Fatalf("a client that never opted in memoized (%d requests, enabled=%v)", requests(), clients[2].ReadMemoEnabled())
}
})
}
}
// TestReadMemoDropsAnAnswerReadWhileAWriteWasInFlight: a read that starts and
// finishes while a write is on the wire (after the write's own pre-send clear)
// may carry the pre-write row. The write's clear on the way back drops it, so
// a read that starts once the write has returned asks the server again.
func TestReadMemoDropsAnAnswerReadWhileAWriteWasInFlight(t *testing.T) {
writeArrived, releaseWrite := make(chan struct{}), make(chan struct{})
var mu sync.Mutex
served := 0
c, rec := newTestClient(t, Options{ResponseHeaderTimeout: -1}, nil, func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if r.Method != http.MethodGet {
close(writeArrived)
<-releaseWrite
_, _ = w.Write([]byte(`{}`))
return
}
mu.Lock()
served++
n := served
mu.Unlock()
_ = json.NewEncoder(w).Encode(map[string]any{"id": "bd-1", "title": fmt.Sprintf("read %d", n)})
})
c.EnableReadMemo()
path, err := IssuePath("bd-1")
if err != nil {
t.Fatal(err)
}
wrote := make(chan error, 1)
go func() {
wrote <- c.Do(context.Background(), Request{Op: OpUpdateIssue, Method: http.MethodPatch, Path: path, Body: map[string]any{}}, nil)
}()
<-writeArrived // the write's pre-send clear has run, and the server holds it
during, err := memoGetIssue(t, c, "bd-1", nil)
close(releaseWrite)
if err != nil {
t.Fatal(err)
}
if err := <-wrote; err != nil {
t.Fatal(err)
}
after, err := memoGetIssue(t, c, "bd-1", nil)
if err != nil {
t.Fatal(err)
}
if rec.count() != 3 || after == during {
t.Fatalf("the read after the write answered %q (the in-flight read's %q) in %d requests; want it re-read (3 requests)", after, during, rec.count())
}
}

View file

@ -203,7 +203,7 @@ func (h *HookFiringStore) CreateIssues(ctx context.Context, issues []*types.Issu
h.fireHook(p.event, p.issue)
}
}
if h.runner != nil {
if h.hookConfigured(hooks.EventUpdate) {
for _, p := range dependencyHookEvents(ctx, issues, h.inner.GetIssue, h.inner.GetDependencyRecords) {
h.fireHook(p.event, p.issue)
}
@ -417,7 +417,7 @@ func (h *HookFiringStore) CompleteIssueOperationCreate(ctx context.Context, issu
for _, pending := range createHookEvents(issue) {
h.fireHook(pending.event, pending.issue)
}
if h.runner == nil || len(dependencies) == 0 {
if !h.hookConfigured(hooks.EventUpdate) || len(dependencies) == 0 {
return
}
request := &types.Issue{Dependencies: cloneDependenciesForHook(dependencies)}
@ -483,7 +483,7 @@ func (h *HookFiringStore) CompleteIssueOperationReclaim(ctx context.Context, iss
}
func (h *HookFiringStore) fireHookByID(ctx context.Context, event, id string) {
if h.runner == nil {
if !h.hookConfigured(event) {
return
}
issue, err := h.inner.GetIssue(ctx, id)
@ -494,7 +494,7 @@ func (h *HookFiringStore) fireHookByID(ctx context.Context, event, id string) {
}
func (h *HookFiringStore) fireDependencyHookByID(ctx context.Context, id string) {
if h.runner == nil {
if !h.hookConfigured(hooks.EventUpdate) {
return
}
issue, err := dependencySnapshot(ctx, id, h.inner.GetIssue, h.inner.GetDependencyRecords)
@ -504,6 +504,22 @@ func (h *HookFiringStore) fireDependencyHookByID(ctx context.Context, id string)
h.runner.Run(hooks.EventUpdate, issue)
}
// hookConfigured reports whether firing event could run anything. The by-id
// paths re-read the issue (and its edges) to build the hook's payload, and over
// a remote backend each of those reads is a gateway round trip a command pays
// after its write (bts-xpw.3.14.6) — for nothing, when no executable hook for
// the event exists and Run would skip it anyway. A runner that cannot say
// (a test double with Run alone) is assumed to have one.
func (h *HookFiringStore) hookConfigured(event string) bool {
if h.runner == nil {
return false
}
if probe, ok := h.runner.(interface{ HookExists(string) bool }); ok {
return probe.HookExists(event)
}
return true
}
// ── Hook tracking transaction ───────────────────────────────────────
// pendingHook records a hook to fire after transaction commit.

View file

@ -586,3 +586,87 @@ func TestHookFiringStoreIssueLifecycleTransactionReopenForwardsCommitOnly(t *tes
}
})
}
// probingHookRunner is a recordingHookRunner that also answers the probe
// hookConfigured asks a real hooks.Runner, and records each event it was asked
// about.
type probingHookRunner struct {
recordingHookRunner
exists map[string]bool
probed []string
}
func (r *probingHookRunner) HookExists(event string) bool {
r.probed = append(r.probed, event)
return r.exists[event]
}
// countingHookStore counts the re-reads the decorator makes to build a payload.
type countingHookStore struct {
fakeHookStore
gets *int
}
func (s countingHookStore) GetIssue(ctx context.Context, id string) (*types.Issue, error) {
*s.gets++
return s.fakeHookStore.GetIssue(ctx, id)
}
// The real runner answers the probe hookConfigured asks. A runner that cannot
// is assumed to hold a hook for every event, and the decorator would re-read
// after every write.
var _ interface{ HookExists(string) bool } = (*hooks.Runner)(nil)
// TestHookFiringStoreReReadsOnlyForAnEventWithAHook (mc-5dkzp): the by-id
// paths re-read the issue to build a hook's payload, which over an http
// workspace is a round trip the command pays after its write. The decorator
// asks the runner about the event it is about to fire — that event, not
// another — and re-reads and fires only when a hook for it exists.
func TestHookFiringStoreReReadsOnlyForAnEventWithAHook(t *testing.T) {
for _, tc := range []struct {
name string
exists map[string]bool
fires bool
}{
{"on_update exists", map[string]bool{hooks.EventUpdate: true}, true},
{"every hook but on_update", map[string]bool{hooks.EventCreate: true, hooks.EventClose: true}, false},
{"no hooks", nil, false},
} {
t.Run(tc.name, func(t *testing.T) {
runner := &probingHookRunner{exists: tc.exists}
gets := 0
inner := countingHookStore{
fakeHookStore: fakeHookStore{issues: map[string]*types.Issue{"bd-1": {ID: "bd-1", Title: "hooked"}}},
gets: &gets,
}
store := &HookFiringStore{DoltStorage: inner, inner: inner, runner: runner}
store.CompleteIssueOperationComment(context.Background(), "bd-1") // re-reads the row
store.CompleteIssueOperationDependency(context.Background(), "bd-1") // re-reads the row and its edges
if want := []string{hooks.EventUpdate, hooks.EventUpdate}; !reflect.DeepEqual(runner.probed, want) {
t.Fatalf("probed %v, want %v: the decorator must ask about the event it fires", runner.probed, want)
}
wantGets, wantEvents := 0, []string(nil)
if tc.fires {
wantGets, wantEvents = 2, []string{hooks.EventUpdate, hooks.EventUpdate}
}
if gets != wantGets || !reflect.DeepEqual(runner.events, wantEvents) {
t.Fatalf("re-read %d times and fired %v; want %d re-reads and %v", gets, runner.events, wantGets, wantEvents)
}
})
}
// The same through a real runner over a hooks directory with nothing in it.
gets := 0
inner := countingHookStore{
fakeHookStore: fakeHookStore{issues: map[string]*types.Issue{"bd-1": {ID: "bd-1"}}},
gets: &gets,
}
store := NewHookFiringStore(inner, hooks.NewRunner(t.TempDir()))
store.CompleteIssueOperationComment(context.Background(), "bd-1")
store.CompleteIssueOperationDependency(context.Background(), "bd-1")
if gets != 0 {
t.Fatalf("with no hook installed the decorator re-read the row %d times; want none", gets)
}
}