beads/cmd/bd/events_journal.go
Julian Knutsen 8988340994
ci(embedded): run every BEADS_TEST_EMBEDDED_DOLT-gated cmd/bd test in the embedded lane (#7708)
* test(cmd/bd): repair the embedded-gated tests that rotted while no lane ran them

These tests skip unless BEADS_TEST_EMBEDDED_DOLT=1 but no CI lane selected
them, so they rotted unseen. Running them shows:

- TestEventsTailFollowReportsTruncationOnTheStream raced on a plain
  bytes.Buffer that os/exec's copy goroutine writes while the test polls it.
  Use the package's syncBuffer.
- TestCorruptMetadataDiagnosticsRunAndDataFailsLoud still asserted that
  doctor and init run on a corrupt metadata.json. #4857 deliberately made
  every workspace command refuse instead (pinned for doctor by
  TestDoctorRejectsCorruptMetadataBeforeSharedServerChecks). Rewrite it to
  the shipped contract as TestEmbeddedCorruptMetadataFailsLoudUntilRestored:
  version warns, doctor/list/init refuse naming the file without touching
  it, and restoring the file brings the data back.
- dolt_remote_origin_guard_embedded_test.go was tagged
  `embeddeddolt && cgo`, a tag no lane sets, so it never compiled in CI
  (gazelle left it out of bd_test srcs). Retag it `cgo`, as #4153 did for
  its dolt_local_only sibling, and add it to srcs. Drop its four bd doctor
  git-origin tests: bd doctor runs no checks on an embedded workspace
  (GH#3794), so two could never pass and two passed vacuously on empty
  output. Rename the surviving tests into the TestEmbedded* prefix.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(embedded): run every BEADS_TEST_EMBEDDED_DOLT-gated cmd/bd test in the embedded lane

The embedded-Dolt cmd lane selected tests with
grep '^func TestEmbedded' cmd/bd/*_embedded_test.go, and only its two
Bazel targets set BEADS_TEST_EMBEDDED_DOLT. A test that skips without the
variable but missed the name prefix or the file suffix skipped in every
lane and reported green: 92 of them, including #7203's --if-revision CLI
coverage (TestIfRevisionMatchAndMismatch, TestReassignFenceCLI, ...).

- embedded-test-shard.sh and gen_embedded_cmd_shard_manifest.py select
  TestEmbedded* by name across cmd/bd/*_test.go, as proxied-test-shard.sh
  does, leaving authors one rule.
- Rename the gated tests into the TestEmbedded* prefix
  (TestDoctorEmbedded<X> -> TestEmbeddedDoctor<X>) and update every
  reference.
- Place the new names with gen_embedded_cmd_shard_manifest.py 100
  --weights=duration --write (incremental).
- Add TestEmbeddedGatedTestsRunInTheEmbeddedLane: it walks cmd/bd's tests
  for any that reach os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" and fails
  if a shard of the lane never lists one. The proxied guard's helpers now
  take the variable and the script; a fixture test pins the walk.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 22:02:22 +00:00

256 lines
12 KiB
Go

package main
import (
"context"
"sync"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/debug"
"github.com/steveyegge/beads/internal/eventsjournal"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/issueops"
"github.com/steveyegge/beads/internal/storage/uow"
)
// Events-journal activation, applied in ONE place: the factories that construct
// a store or a unit-of-work provider (store_factory.go, store_factory_nocgo.go,
// uow_factory.go, and bd doctor's repair handlers through the same package).
// Every plumbing bd opens goes through one of those, so a new command cannot
// acquire a store that silently records nothing.
//
// It did not start there, and the three ways it was wrong are the argument for
// the guard that now holds it. First a process-global switch. Then two root
// pre-run call sites, which covered the CLI's own store and its proxied
// provider and missed `bd serve` (which builds its own provider for a
// server-mode workspace), routed creates and remote-cache hydration (which open
// a SECOND store for another workspace), the pluggable backend registry arm,
// and the personal-migration planning store. Then a blanket exemption for bd
// doctor, whose stated reason — "workspace repairs, not bead mutations" — was
// false for three of its repair handlers. Each miss ran with the journal off
// while every command reported success, because an empty journal is
// indistinguishable from a quiet one.
// TestEveryStoreConstructionActivatesTheEventsJournal keeps it centralized.
//
// The policy itself lives in internal/eventsjournal so bd doctor's fix package,
// which cannot import package main, applies the identical rule.
// eventsJournalEnabled reports the activation the PROCESS resolved, for the
// paths that have no particular workspace in hand.
func eventsJournalEnabled() bool {
return config.GetBool(eventsjournal.ConfigKey)
}
// activateEventsJournalStore is eventsjournal.ActivateStore under the name the
// construction guard matches. Kept as a wrapper rather than called directly so
// cmd/bd has one spelling of the idiom and the guard has one name to look for.
func activateEventsJournalStore(beadsDir string, s storage.DoltStorage, err error) (storage.DoltStorage, error) {
return eventsjournal.ActivateStore(beadsDir, s, err)
}
// eventsJournalMaintenanceRunner returns the plumbing auto-prune should run
// against, or nil when this process has none.
//
// The provider is checked FIRST because a proxied-server command has both: a
// unit-of-work provider it actually writes through, and (in some topologies) a
// store opened alongside it. Maintenance belongs on the plumbing that wrote the
// records — same connection settings, same database, same transaction
// discipline.
//
// A backend that cannot journal cannot need pruning, so a nil result is a
// silent no-op rather than a diagnostic.
func eventsJournalMaintenanceRunner() issueops.EventsMaintenanceRunner {
if uowProvider != nil {
if runner := eventsJournalMaintenanceRunnerFor(uowProvider); runner != nil {
return runner
}
}
return eventsJournalMaintenanceRunnerFor(store)
}
// eventsJournalMaintenanceRunnerFor resolves ONE plumbing value — a store or a
// unit-of-work provider — to its maintenance capability.
//
// BOTH decorator chains are peeled, and the provider half is not symmetry for
// its own sake. bd wraps the unit-of-work provider in a hook-firing decorator
// whenever a hook runner exists, which is by default; a bare type assertion
// against that wrapper matches nothing, and auto-prune becomes a silent no-op
// in exactly the topology — proxied server — that journals fastest. Nothing
// reports it: retention that never runs looks identical to retention with
// nothing to do.
//
// The wrapper also forwards the capability itself (notifyingProvider.
// RunEventsMaintenanceTx), so this is the second of two defenses rather than
// the only one. That is deliberate: the forwarder keeps the wrapper's SURFACE
// honest, which the parity guards enforce, and this peel keeps RESOLUTION
// honest against a future decorator that forgets one. A capability that is
// invisible when it goes missing gets both.
func eventsJournalMaintenanceRunnerFor(plumbing any) issueops.EventsMaintenanceRunner {
if plumbing == nil {
return nil
}
switch p := plumbing.(type) {
case storage.DoltStorage:
if p == nil {
return nil
}
plumbing = storage.UnwrapStore(p)
case uow.UnitOfWorkProvider:
if p == nil {
return nil
}
plumbing = uow.UnwrapProvider(p)
}
runner, _ := plumbing.(issueops.EventsMaintenanceRunner)
return runner
}
// shouldAutoPruneEventsJournal reports whether this command pays for retention
// maintenance. The first two gates are the SAME pair auto-export and auto-push
// use one block above, deliberately rather than a new classification:
//
// - runsPostCommandMaintenance excludes strict --readonly (a store opened to
// refuse writes) and `bd serve`, which drives the same pass from its own
// ticker rather than from a command boundary it does not have.
// - isReadOnlyCommand excludes the classified reads, and it is load-bearing
// rather than belt-and-braces: a read-only classification opens the store
// with OpenForReadOnlyCommand, which is "otherwise a normal writable
// store", so nothing underneath refuses a maintenance delete. Drop this
// check and `bd list` prunes the journal — which
// TestEmbeddedAutoPruneDoesNotRunForReadOnlyCommands catches.
//
// Two more gates are specific to this trigger:
//
// - A PREVIEW command (--dry-run, --inspect) promised not to mutate, and the
// root pre-run holds it to that with a write-refusing open. The trigger has
// to make the same judgement the store-open mode did, or maintenance
// becomes the one write a preview performs — logged as a failure at best,
// and a real delete on any plumbing that does not refuse it.
// - The `bd events` family is excluded whole. Reading a feed must never be
// what prunes it: `bd events tail --follow` is a consumer, and a consumer
// that trims its own backlog on the way out is a surprise nobody asked
// for. `bd events prune` is excluded for the opposite reason — it just
// resolved the retention bound by hand, so an automatic pass immediately
// behind it is work with nothing to do.
//
// commandDidWrite — the flag Dolt auto-commit gates on — is deliberately NOT
// consulted. It marks "this write still needs a Dolt commit", which the
// embedded backend never sets because it commits inside the store; gating on it
// would silently exclude the entire embedded topology from retention. The
// classification is coarser (an unclassified command that wrote nothing can
// reach the throttle read) and that is the right way round: the cost of a
// false positive is one indexed query, and the cost of a false negative is a
// journal nothing ever bounds.
func shouldAutoPruneEventsJournal(cmd *cobra.Command) bool {
if cmd == nil {
return false
}
if !runsPostCommandMaintenance(cmd.Name(), readonlyMode) || isReadOnlyCommand(cmd.Name()) {
return false
}
if isPreviewCommand(cmd) || isEventsJournalCommand(cmd) {
return false
}
return true
}
// isEventsJournalCommand reports whether cmd is `bd events` or one of its
// subcommands. It walks the ancestor chain rather than matching leaf names:
// `tail`, `export` and `prune` are all names the top level uses for something
// else, and a gate that matched on the leaf would be exempting the wrong
// commands the day one of them moves.
func isEventsJournalCommand(cmd *cobra.Command) bool {
for c := cmd; c != nil; c = c.Parent() {
if c == eventsCmd {
return true
}
}
return false
}
// autoPruneOnce holds the once-per-process guarantee the trigger promises.
// PersistentPostRunE runs once per bd invocation anyway; the guard is what
// keeps that true for the in-process callers that run Execute() repeatedly (the
// cmd/bd test binary, library embedders), where an unguarded trigger would turn
// a maintenance pass into a per-command tax.
var autoPruneOnce sync.Once
// maybeAutoPruneEventsJournal is the CLI's writer-pays retention trigger: after
// a mutating command has committed, bound the journal to its configured floors.
//
// WHERE IT IS CALLED FROM is the whole design. It runs in the root
// PersistentPostRunE, in the post-command maintenance region that already
// carries Dolt auto-commit, auto-backup, auto-export and auto-push — after the
// user's data is durably committed and pushed, before the store is closed. That
// is the narrowest existing seam with all four properties this needs: the
// command SUCCEEDED (an error return never reaches it), the mutation
// transaction is closed (so maintenance cannot extend or roll back a user's
// write), a store or provider is still open, and read-only commands are already
// excluded by the surrounding classification. `bd serve` is excluded there by
// name and drives the same pass from its own ticker instead.
//
// The alternatives were all worse in the same way — they fire somewhere that
// does not know a command finished. A post-write hook in the store decorator
// chain fires per write op, which on a batch command means dozens of passes,
// each one racing the mutation transaction that has not been committed yet. The
// issueops emit leaf is inside that transaction. Close() is reached from tests,
// doctor handlers and half-failed opens, where a delete is a surprise. And a
// PostRunE on each mutating command is the hand-maintained list that this
// feature's own review found broken twice.
//
// gating, in order of cost: config (free), plumbing (free), then the throttle
// (one indexed query). A workspace without the journal pays nothing at all.
func maybeAutoPruneEventsJournal(ctx context.Context, beadsDir string) {
autoPruneOnce.Do(func() {
if !eventsjournal.EnabledFor(beadsDir) || !eventsjournal.AutoPruneEnabledFor(beadsDir) {
return
}
runner := eventsJournalMaintenanceRunner()
if runner == nil {
return
}
n, err := eventsjournal.AutoPrune(ctx, runner, eventsJournalAutoPruneOptions())
reportEventsJournalAutoPrune(n, err)
})
}
// eventsJournalAutoPruneOptions reads the retention floors the pass applies.
// Same keys and same accessor as `bd events prune`, so the automatic bound and
// the manual one cannot be computed from different numbers.
func eventsJournalAutoPruneOptions() eventsjournal.AutoPruneOptions {
return eventsjournal.AutoPruneOptions{
RetainDays: config.GetInt("events-journal-retain-days"),
RetainRows: config.GetInt("events-journal-retain-rows"),
}
}
// reportEventsJournalAutoPrune logs a pass. Debug-level and never returned:
// retention maintenance must not change what a user's command printed or what
// it exited with. A workspace whose journal cannot be pruned is a workspace
// with a large journal, not a broken CLI.
func reportEventsJournalAutoPrune(n int64, err error) {
if err != nil {
debug.Logf("events journal: auto-prune failed after %d row(s): %v\n", n, err)
return
}
if n > 0 {
debug.Logf("events journal: auto-prune deleted %d row(s)\n", n)
}
}
// activateEventsJournalProvider is the same for a unit-of-work provider — bd's
// second write plumbing, with its own transactions and its own activation.
func activateEventsJournalProvider(ctx context.Context, beadsDir string, p uow.UnitOfWorkProvider, err error) (uow.UnitOfWorkProvider, error) {
if err != nil || p == nil {
return p, err
}
configurer, _ := p.(storage.EventsJournalConfigurer)
if cfgErr := eventsjournal.Apply(configurer, eventsjournal.EnabledFor(beadsDir)); cfgErr != nil {
closeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), providerCloseTimeout)
defer cancel()
_ = p.Close(closeCtx)
return nil, cfgErr
}
return p, nil
}