beads/cmd/bd/events_proxied_server_test.go
Julian Knutsen 6e9a15507e
ci(bazel): proxied-server tier 15 → 30 duration-balanced shards; un-hide 12 proxied tests (F2) (#7129)
* ci(bazel): raise proxied-server tier from 15 to 30 shards (F2)

bazel.yml's bazel-proxied lane (//cmd/bd:bd_proxied_test, run only with
remote execution) was sharing PR Risk's legacy 15-shard, frozen
bd-init-cost-proxy manifest block, so its longest shard was the long
pole of the Bazel-only lanes. Give it its own 30-shard block in
.github/scripts/proxied-cmd-test-shards.txt, bin-packed
longest-processing-time-first by measured wall-time
(scripts/ci/proxied_test_durations.json, captured from two bazel-proxied
job failures' testlogs) instead of the bd-init count proxy, via
scripts/ci/gen_proxied_shard_manifest.py's new --weights=duration mode.
Coverage is proven equal to the full discovered test set by two
independent methods (sorted-name diff and an actual 30-shard invocation
of proxied-test-shard.sh in list-only mode), both exact matches with no
duplicates.

cmd/bd:bd_proxied_test's shard_count moves to 30; PR Risk's and
main.yml's own test-proxied-cmd jobs keep their 15-shard matrix and
frozen manifest block untouched, per D2's retirement design (the Bazel
lane only needs to run the same tests, not the same split). Updated the
two coupled policy tests (scripts/ci_workflow_test.go's
TestBazelDoltServerTiersMirrorPRRisk, scripts/pr_risk_bazel_coverage_test.go's
TestBazelRetiredLanesCheckListedTestsRan) to allow the Bazel target's
shard_count to differ from its legacy job's own matrix size, and
engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md / engdocs/TESTING.md to document
the split.

Verified: go test ./scripts/...; bazel --nohome_rc test
//scripts:scripts_test; actionlint on bazel.yml; make bazel-sync-check;
make ci-pr-policy; 3 representative shards (1, 15, 30 of 30) run
directly against Bazel-built race bd_test/bd_for_tests binaries and the
pinned dolt 2.2.0 release, all green, 55-81s wall time each (the
duration data's absolute seconds are from a contention-era sample and
run far faster here; the LPT balance itself, not the absolute SLA, is
the property that matters).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): close fork-PR gap in proxied 30-shard manifest validation (review B1/S1)

TestShardScriptsListOnlyRealTests only ran proxied-test-shard.sh at PR
Risk's legacy 15-shard matrix, never at the Bazel-only bazel-proxied
lane's 30-shard manifest block. bazel-proxied only runs in mode=remote
(skipped on fork PRs), so a fork PR could rename, duplicate, or corrupt
a TestProxiedServer*/TestServerMode* manifest entry in the 30-block and
merge green via go test ./scripts/... (which runs on every PR), only
breaking the required lane afterward on main and every same-repo PR.

Fix: the test now computes every shard total it must validate per
script (this job's own PR Risk matrix size, every distinct total found
in that script's own committed manifest, and - for proxied-test-shard.sh
- cmd/bd:bd_proxied_test's live shard_count from BUILD.bazel) and runs
the full per-shard validation loop at each one, relying on the script's
own duplicate/stale-name guards. It also sums each total's "manifest: N"
counts across all shards and fails if they sum to zero, catching a
silently-deleted block that would otherwise fall back to 100% hashing
undetected.

Also fold in S1: replace the independently hard-coded `30` overrides in
ci_workflow_test.go and pr_risk_bazel_coverage_test.go with
bazelProxiedShardCount(t), a new shared helper that reads
cmd/bd:bd_proxied_test's shard_count live from cmd/bd/BUILD.bazel. This
turns the check_shard_coverage.py argument comparison in
pr_risk_bazel_coverage_test.go into a genuine cross-file pin (BUILD.bazel
vs bazel.yml) instead of two literals that could drift together
unnoticed. ci_workflow_test.go's tiers loop no longer asserts
BUILD.bazel's shard_count against a value read from the same file
(tautological); it instead asserts pr-risk.yml's and main.yml's legacy
test-proxied-cmd matrices equal each other, restoring the pin S1 found
removed on this branch relative to base.

Verified against the reviewer's mutations (duplicate entry, bogus
entry, whole-block deletion, stale rename, matrix divergence, BUILD
shard_count drift, bazel.yml arg drift): each now fails go test
./scripts/..., where it previously passed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): add generator --write/--check and freeze legacy manifest block (review S3/N1/N4/N5)

gen_proxied_shard_manifest.py:
- Add --write (rewrite only the target total_shards block in place,
  leaving every other block's header/body text untouched) and --check
  (regenerate in memory and diff against the committed block, exiting
  1 with the exact --write command needed if stale). Both operate via
  new split_blocks()/write_block()/check_block() helpers that round-trip
  the real manifest file losslessly.
- N1: fix the duration-weights header's false claim about the
  fallback ratio.
- N5: warn on stderr when proxied_test_durations.json has duration
  captures for tests that no longer exist in cmd/bd/*_test.go (renamed,
  split, or deleted since the last capture).

proxied-cmd-test-shards.txt:
- N4: mark the legacy 15-shard block's header as frozen for
  pr-risk.yml's/main.yml's legacy fork/push jobs: it is deliberately
  not kept in sync with newly-added tests (those hash-fallback
  instead), so regenerating it is not routine maintenance. Only the
  header text changed; all 149 explicit assignment lines and the
  "known blind spots" callouts are untouched (verified: a fresh
  --write would pack all 164 currently-discovered tests explicitly,
  silently changing which shard runs which test for this block, which
  is exactly what "frozen" means to avoid).
- Regenerate the Bazel-only 30-shard block's header via
  `gen_proxied_shard_manifest.py 30 --weights=duration --write` to
  describe --write/--check instead of the old manual-copy-paste
  instructions. proxied_test_durations.json has not changed yet (S2
  is still in progress), so this run only touched header text; the 164
  assignment lines are unchanged.

pr_risk_bazel_coverage_test.go:
- Add TestProxiedShardManifestGeneratorNotStale: runs
  gen_proxied_shard_manifest.py 30 --weights=duration --check under
  plain `go test` (skipped under `bazel test`, whose runfiles hold
  neither the generator's sources nor cmd/bd) so a stale 30-block
  (e.g. committed by hand, or left behind after adding tests without
  regenerating) fails scripts-go-checks on every PR, including forks.

tools/bazel/equivalence_allowlist.txt: allowlist the new test's guarded
Bazel skip.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): rename 14 undiscovered proxied-gated tests; N2/N3/N6 doc fixes

cmd/bd/*_test.go: 14 tests call requireProxiedServerEnv/
requireSharedProxiedServer but their names did not match the shard
scripts' discovery regex ^Test(ProxiedServer|ServerMode), so no CI lane
ever set BEADS_TEST_PROXIED_SERVER=1 for them and they silently skipped
everywhere (pre-existing gap, not caused by F2; list was in
~/beads-bazel-plan/f2/review/undiscovered_gated.txt).

Fixed by renaming rather than widening discovery: 162 of the 164
already-discovered tests call one of the two gate functions in the first
few lines of their body, so name and gate already correlate almost
perfectly, but the 14 tests' names are each styled differently with no
single regex term that would cover all of them without a hand-maintained
list. Renaming lets the existing discovery mechanism, hash-fallback
behavior, and both shard-coverage policy tests pick them up with zero
other changes. Checked each of the 14 names for cross-references first;
each appeared only in its own file plus at most one doc-comment in a
sibling file, both updated alongside the rename. No collisions with
existing test names.

Discovery went from 164 to 178 tests. Regenerated the Bazel-only 30-shard
block (gen_proxied_shard_manifest.py 30 --weights=duration --write) to
pack the 14 new tests; this is the first --write this round to actually
change assignments, not just header text. The frozen legacy 15-shard
block (N4) was deliberately left alone: the 14 renamed tests now
hash-fallback into it, exactly like every other test added since it was
last generated.

cmd/bd/BUILD.bazel (N6): restore the bd_proxied_test comment sentence
bazel.yml's mirror comment kept but BUILD.bazel's had dropped: PR Risk
builds its bd-cmd-test with -race, as here; main.yml's test-proxied-cmd
twin uses build-artifacts' non-race bd-cmd-test, so this lane is the
stricter of the two.

.github/workflows/bazel.yml, .bazelrc (N2/N3): remove stale timing
language ("30 shards of ~5-10 minutes" / ".bazelrc's leftover "15 shards
of ~10 minutes") never reconciled with this task's own local
representative-shard measurements (55-81s/shard); replaced with the
concurrency argument (30 simultaneous dolt sql-servers don't fit a
GitHub-hosted runner regardless of any single shard's wall time). Also
removes the "within 5s of each other" balance claim (S2): the underlying
duration weights are contaminated by -test.parallel overlap in the
top-level <testsuite time> figures they were derived from; a corrected
re-measurement is in progress (separate commit once it lands).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): re-derive proxied duration weights without overlap double-counting (review S2)

scripts/ci/proxied_test_durations.json's 178 entries are now each a
clean, single-job (no concurrent heavy bazel/dolt job on the box),
-test.parallel=1 measurement: a fresh manifest-aligned 30-shard sweep's
own <testsuite time> per top-level test where errors="0" (including
failing-but-complete suites, whose elapsed time is still trustworthy),
plus, for the handful too slow to finish inside Bazel's default
per-target timeout, a `bazel run` leaf-subtest-only sum that never adds
a parallel group's own PASS/FAIL line on top of its already-overlapping
children's lines. An earlier pass at this data summed every PASS/FAIL
line indiscriminately, double-counting parallel subtest groups.

A first attempt at re-measuring the slow stragglers overlapped in
wall-clock time with a second, unrelated heavy sweep and came out 2-3x
inflated versus the same tests measured cleanly; that contaminated data
was discarded in favor of isolated re-runs. TestProxiedServerDep2
initially looked hung during its isolated re-run (15+ minutes with no
PASS/FAIL output and no CPU growth in the top-level test binary) but
was not: each of its subtests does real `bd init --proxied-server`
work in a short-lived child process, and Go buffers a parallel
subtest group's output until the group's last member finishes. Letting
it run to completion (2357.72s) rather than guessing avoided reporting
bad data.

seconds_per_init_fallback is recomputed as the weighted average
seconds-per-init across all 178 measured tests (sum of durations / sum
of inits), for use only when a brand-new test has no entry yet.

Re-packing the 30-shard manifest with this corrected data changes the
shard-balance picture: TestProxiedServerDep2 and TestProxiedServerClose
are each large enough, alone, to dominate their own shard (~2358s and
~1105s locally) while every other shard clusters around 741-876s. The
"within 5s" claim this replaces was never earned by the underlying
data; .github/workflows/bazel.yml now documents the real, uneven
spread and flags that the two outlier shards' local numbers already
exceed the job's 25m step timeout on their own merits (unverified
against RBE's actual per-shard throughput, since this job has only
ever run remote).

Verified: go vet ./scripts/..., go test ./scripts/... (includes the
manifest/coverage policy tests), actionlint, make bazel-sync-check,
make ci-pr-policy all pass. Exact-once coverage re-confirmed: 178
manifest assignments for 178 discovered tests, zero duplicates, zero
gaps.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(cmd/bd): apply re-review candidate fixes; revert 2 unready renames (R-B1)

Re-review found the prior round's 14 renames would turn the required
Bazel + legacy 15-shard lanes red once discovered:

- TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix/external-unix and
  TestProxiedServerOutageReconnectAcceptanceMatrix/external-unix-socket
  built a dolt.sock path under t.TempDir(), which nests Bazel's hermetic
  tmp-root plus the (64-byte-truncated) test name deep enough to exceed
  sockaddr_un's 108-byte sun_path limit. Both now use a new shortSocketPath
  helper (a short os.MkdirTemp("", "bds") directory) instead.

- runProxiedDeadline raced a deadline-watcher goroutine against
  exec.Cmd.Start()/Run()'s write to cmd.Process, caught by -race. Replaced
  with cmd.Cancel, which exec.CommandContext only invokes after Start, so
  the read is ordered after Start's write.

With those two fixes applied, TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix
passes outright. TestProxiedServerOutageReconnectAcceptanceMatrix's race is
fixed, but both its subtests still fail a real "exceeded 3s bound" assertion
against a stopped external upstream -- a genuine latency issue, not a test
bug -- so it is reverted back to its original name
(TestProxiedOutageReconnectAcceptanceMatrix) with a comment explaining why,
pending a bead against the underlying latency.

TestServerModeSharedServeGlobalReportsTheServedDatabase also reverted back
to TestSharedServerModeServeGlobalReportsTheServedDatabase: `bd serve
--global` refuses to auto-apply 4 pending schema migrations (v65->v69) to a
shared/proxied server database and exits 1 with a consent-required usage
message instead of starting, per the #5920 migration-consent guard. Also a
genuine product/test-fixture question, not fixed here; reverted with a
comment, pending a bead.

The other 12 of the original 14 renames are kept; all 12 verified passing
in both lanes (see commits that follow for the manifest update and full
verification).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): re-derive proxied weights from parallel=4 CI-identical sweep (R-S1)

The prior round's duration data (committed in 6be9605d9f) was captured at
-test.parallel=1 (serial). That fixed a real double-counting bug, but
serial timings don't model this lane, which always runs -test.parallel=4
(--config=doltserver-proxied's own setting). Re-review measured near-zero
correlation (r=0.08) between the serial-derived predicted shard load and
actual parallel=4 wall time -- e.g. TestProxiedServerRecomputeBlocked was
weighted 812s serially but measured 33s actual at parallel=4.

Replaced with a full fresh capture: `bazel --nohome_rc test
--config=doltserver-proxied //cmd/bd:bd_proxied_test` (local execution
only, the exact config bazel.yml's proxied-server tier uses), 30/30 shards
passing, 0 failures, 0 races. Every one of the 176 discovered tests'
top-level <testsuite time> was captured directly from this run's
bazel-testlogs test.xml files -- no leaf-summation workaround was needed
this time, since none of the 176 hit Bazel's per-target default timeout
running alongside the rest of its shard.

Re-ran `gen_proxied_shard_manifest.py 30 --weights=duration --repack
--write` with these corrected weights: a deliberate full reweight, not an
incremental add, since every weight changed. The new packing clusters all
30 shards within a few seconds of ~355-360s each (176 tests summing to
10712s / 30), a dramatic improvement over the serial data's wrongly
implied two-outlier-shard spread (TestProxiedServerDep2 alone at ~2358s,
TestProxiedServerClose alone at ~1105s -- both artifacts of serial
timing, not real per-shard cost). Updated bazel.yml's paragraph to report
this honestly instead of the prior, false "exceeds 25-minute timeout"
claim, which was itself downstream of the serial/parallel mismatch.

Verified: the 12 tests kept renamed in the previous commit, plus
TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix's socket-path fix,
all pass under the new 30-shard Bazel manifest (this capture run) and
under the legacy 15-shard hash-fallback script (ran the 7 shards their
hash lands in: 2, 3, 4, 5, 8, 10, 11 -- all pass).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): incremental --write, --repack, name-coverage --check (R-S2)

The prior --check compared the committed 30-shard block byte-for-byte
against a fresh from-scratch LPT pack, and --write always did a full
repack. That meant any weight drift (even from proxied_test_durations.json
alone, with no test added/renamed/removed) failed --check, and two
unrelated PRs each adding one proxied test would both force a full
repack and conflict on unrelated shard lines -- re-review demonstrated
one new test moving 73 assignment lines under the old full-repack
behavior.

Redesigned:

- --check now verifies exact-once NAME coverage of the currently
  discovered TestProxiedServer*/TestServerMode* set against the
  committed block -- missing, stale, or duplicated names fail with an
  actionable message -- but does NOT compare shard assignments, so a
  weights-only change no longer forces --check to fail.
- --write is now incremental by default: every already-assigned test
  keeps its current shard, stale (no longer discovered) names are
  dropped, and only newly-discovered names are LPT-placed onto whichever
  shard is currently lightest. A full from-scratch LPT pack now requires
  the explicit --repack flag (used in the preceding commit for the
  deliberate full reweight; ordinary new-test additions should use the
  incremental default).

Mutation-tested directly (not just reviewed): verified --check correctly
flags a stale name as "stale/unknown" and a newly added test as "missing"
with the exact --write command to fix it; verified incremental --write
touches only the lines for the added/removed names; verified --repack
--write produces a materially different (full rebalance) diff from the
incremental default.

Updated TestProxiedShardManifestGeneratorNotStale's comment in
pr_risk_bazel_coverage_test.go to describe the new name-coverage-only
--check semantics (it previously described the old byte-identical-packing
behavior; the test's own assertion -- exit code only -- was already
correct and unaffected).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 09:36:05 -07:00

165 lines
5.9 KiB
Go

//go:build cgo && unix
package main
import (
"encoding/json"
"os/exec"
"strconv"
"strings"
"testing"
"github.com/steveyegge/beads/internal/eventsjournal"
"github.com/steveyegge/beads/internal/storage"
)
// TestProxiedServerEventsJournal is the end-to-end guard for bd's SECOND write
// plumbing. In proxied-server mode no store is ever opened: commands dispatch
// through the unit-of-work provider, whose repositories reach the same issueops
// emit helpers over their own pinned transactions.
//
// Emission at that seam is only half of coverage — the provider must also bind
// journal activation to the transaction the mutation runs in, and until this
// slice it had no activation path at all. That failure is invisible from the
// outside: every command succeeds, every write lands, and the journal is simply
// empty. Only an end-to-end read can see it, which is why this test drives the
// real binary against a real proxied server rather than asserting on wiring.
func TestProxiedServerEventsJournal(t *testing.T) {
requireProxiedServerEnv(t)
t.Parallel()
bd := buildEmbeddedBD(t)
p := bdProxiedInit(t, bd, "pjr")
journalEnv := []string{"BD_EVENTS_JOURNAL=1"}
run := func(args ...string) string {
t.Helper()
stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, journalEnv, args...)
if err != nil {
t.Fatalf("bd %s: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), err, stdout, stderr)
}
return stdout
}
run("create", "first", "--silent")
run("create", "second", "--silent")
records := decodeEventRecords(t, run("events", "export"))
if len(records) < 2 {
t.Fatalf("proxied mutations journaled %d records, want at least 2 — the uow plumbing is not scoping journal activation\n%v",
len(records), records)
}
var prev int64
for i, r := range records {
if r.Seq <= prev {
t.Errorf("record %d seq %d not strictly greater than prev %d", i, r.Seq, prev)
}
prev = r.Seq
if r.TS == "" {
t.Errorf("record %d carries no timestamp", i)
}
}
creates := 0
for _, r := range records {
if r.Op == "create" {
creates++
}
}
if creates != 2 {
t.Errorf("journaled %d create records, want 2: %+v", creates, records)
}
// tail --since resumes from a checkpoint, the operation a consumer actually
// performs on every poll.
tailed := decodeEventRecords(t, run("events", "tail", "--since", itoa(records[0].Seq)))
if len(tailed) != len(records)-1 || tailed[0].Seq != records[0].Seq+1 {
t.Fatalf("tail --since %d = %+v, want the records after it", records[0].Seq, tailed)
}
// prune runs through the provider's own transaction (an ephemeral commit,
// since the journal table is dolt_ignored) and honors the floors.
head := records[len(records)-1].Seq
pruneEnv := append(append([]string{}, journalEnv...),
"BD_EVENTS_JOURNAL_RETAIN_DAYS=0", "BD_EVENTS_JOURNAL_RETAIN_ROWS=0")
stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, pruneEnv,
"events", "prune", "--before", itoa(head), "--json")
if err != nil {
t.Fatalf("bd events prune: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr)
}
var pruned struct {
Pruned int64 `json:"pruned"`
}
if decErr := json.Unmarshal([]byte(firstJSONObject(stdout)), &pruned); decErr != nil {
t.Fatalf("prune output is not a JSON object: %v\n%s", decErr, stdout)
}
if want := int64(len(records) - 1); pruned.Pruned != want {
t.Errorf("prune deleted %d records below seq %d, want %d", pruned.Pruned, head, want)
}
// And the truncation contract reaches the proxied read path too: a
// checkpoint below the surviving floor fails typed rather than presenting
// the remaining suffix as a whole history.
stdout, stderr, err = bdProxiedRunBuffersWithEnv(t, bd, p.dir, journalEnv, "events", "export", "--json")
if err == nil {
t.Fatalf("export from a pruned-past checkpoint succeeded; want a truncation failure\nstdout:\n%s", stdout)
}
var trunc struct {
Code string `json:"code"`
}
if decErr := json.Unmarshal([]byte(firstJSONObject(stdout+stderr)), &trunc); decErr != nil {
t.Fatalf("truncation output is not a JSON object: %v\nstdout:\n%s\nstderr:\n%s", decErr, stdout, stderr)
}
if trunc.Code != storage.EventsJournalTruncatedCode {
t.Errorf("code = %q, want %q\nstdout:\n%s\nstderr:\n%s", trunc.Code, storage.EventsJournalTruncatedCode, stdout, stderr)
}
}
// TestProxiedServerEventsJournalOffByDefault pins the default: a proxied
// workspace that never asked for the journal writes no rows, so the feature
// costs an ordinary team nothing.
func TestProxiedServerEventsJournalOffByDefault(t *testing.T) {
requireProxiedServerEnv(t)
t.Parallel()
bd := buildEmbeddedBD(t)
p := bdProxiedInit(t, bd, "pjo")
// Explicitly unset, not merely absent: a developer with BD_EVENTS_JOURNAL
// exported would otherwise turn this assertion into a false green.
clean := envWithout(bdProxiedEnv(p.dir), "BD_EVENTS_JOURNAL")
mustRun := func(args ...string) string {
t.Helper()
cmd := exec.Command(bd, args...)
cmd.Dir = p.dir
cmd.Env = clean
stdout, stderr, err := runCommandBuffers(t, cmd)
if err != nil {
t.Fatalf("bd %s: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), err, stdout.String(), stderr.String())
}
return stdout.String()
}
mustRun("create", "untracked work", "--silent")
if got := strings.TrimSpace(mustRun("events", "export")); got != "" {
t.Errorf("journal recorded %q with events-journal unset; it must be off by default", got)
}
}
func decodeEventRecords(t *testing.T, out string) []eventsjournal.Record {
t.Helper()
var records []eventsjournal.Record
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, "{") {
continue
}
var rec eventsjournal.Record
if err := json.Unmarshal([]byte(line), &rec); err != nil {
t.Fatalf("decode journal record %q: %v", line, err)
}
records = append(records, rec)
}
return records
}
func itoa(v int64) string { return strconv.FormatInt(v, 10) }