beads/internal/storage/sqlkit/portable.go
Eddie the Engineer c79bb32bca fix(backends): maintainer fixup — CI repair + adoption review fixes
Maintainer fixup for the choose-your-own-storage-backend PR, amended across the
pre-review CI-repair and the review-loop apply-fixes iterations so contributor
commits stay intact under one maintainer commit.

CI repair (pre-review):
- default.nix: bump vendorHash for the added Postgres/MySQL driver deps.
- conformance.yml: provision the Postgres service DB with a C-locale cluster so
  text ORDER BY sorts by code point, matching Dolt's utf8mb4_0900_bin oracle.

Review fixes — blockers:
- uow/doltserver_tx: on a non-serialization DOLT_COMMIT failure, ROLLBACK on the
  pinned session before releasing it, and poison (pool-discard via driver.ErrBadConn)
  the connection when the rollback also fails, so a later borrower can't inherit an
  open transaction and implicitly commit orphaned writes. Satisfies the existing
  doltserver_tx_test rollback/poison cases.
- sqlitedialect: translate GREATEST/LEAST to SQLite scalar max/min so the
  ReconcileChildCounters upsert (GREATEST(last_child, ?)) no longer aborts creation
  or import of hierarchical dotted issue IDs (e.g. x-1.2) on the SQLite backend.

Review fixes — majors:
- conformance: add a portable dotted hierarchical child-ID scenario (x-1, x-1.1,
  x-1.2, x-1.2.1) exercising the ReconcileChildCounters GREATEST path on every
  backend — the coverage hole that let the SQLite blocker ship green. Verified
  against the embedded-Dolt oracle and the real SQLite driver.
- postgres: warn at provision/open when the database collation is not code-point
  ordered (C/POSIX/C.*), since text ORDER BY otherwise diverges from Dolt silently.
- sqlite: pin the pool to one connection and add busy_timeout + WAL to the DSN so
  overlapping operations wait instead of surfacing a raw "database is locked"
  (matches the Dolt backend's single-connection posture). Adds a concurrency guard.
- cmd/bd: gate explicit `bd dolt commit` / `bd vc commit` with a typed
  storage.ErrUnsupported on non-Dolt backends instead of the no-op Commit printing
  "Committed."; internal opportunistic auto-commit is untouched.

Review fixes — attempt 2 (stale init backend-flag test contract):
- cmd/bd/init_test, init_embedded_test: the init backend-flag tests still asserted
  the pre-#3151 one-backend contract (--backend=sqlite must fail "SQLite backend
  has been removed"; --backend=postgres is an "unknown backend"), so
  `go test ./cmd/bd -run '^TestInitBackendFlag$'` failed in a clean environment now
  that sqlite/postgres/mysql are supported pluggable backends. Rewrite
  TestInitBackendFlag and the cgo TestEmbeddedInit sibling (rejected_backends →
  sql_backend_flags) for the new contract: --backend=sqlite initializes and
  persists backend=sqlite; --backend=postgres is recognized and fails only on
  missing connection config (not "unknown backend"); a genuinely unknown backend
  is still rejected. Pin BEADS_DIR and scrub BEADS_*/BD_* so the subtests are
  hermetic against an ambient .beads or Dolt server.

Cleanups (non-gating):
- cmd/bd/init: extract checkExistingSQLBackend so the added SQL-backend branches
  don't inflate checkExistingBeadsDataAt's complexity, and load config once.
- configfile: correct the stale "Deprecated: always dolt" Backend field comment.

Review fixes -- attempt 3 (SQLite datetime parity blocker + conformance/rewrite
hardening of the storage-parity gate):
- sqlite/dsn: add _time_format=datetime. Without it modernc binds time.Time via
  t.String() ("2026-07-09 12:34:56.123 +0000 UTC"), a shape SQLite's date functions
  cannot parse, so the counts mega-query's DATE_FORMAT(created_at,...) (strftime on
  SQLite) returns NULL and deps_json exposes a zero created_at while Dolt emits the
  real timestamp. datetime stores whole-second "2026-07-09 12:34:56", matching
  Dolt/MySQL datetime(0). Verified end-to-end against the real modernc driver.
- conformance: add ReadyWorkDepCreatedAtParity -- imports a dependency with a known
  created_at (the bd import / CreateIssuesWithFullOptions bound-time.Time path) and
  asserts the counts-rendered deps_json created_at equals it, closing the blind spot
  that let the SQLite blocker ship green (the counts tests only checked DependencyCount).
  Proven to fail on the pre-fix DSN and pass on both the fixed SQLite backend and the
  embedded-Dolt oracle.
- conformance: add ClaimReadyIssueConcurrentExclusivity -- races 24 workers over 8
  ready issues and asserts each is claimed exactly once, exercising the SQL backends'
  multi-writer claim property that was previously only asserted, never raced. Gated to
  the multi-connection dialects (postgres/mysql) via DialectName; SQLite (single-conn
  pool) and embedded-Dolt skip.
- conformance.sh: run the embedded-Dolt reference conformance with
  BEADS_TEST_EMBEDDED_DOLT=1 and fail loudly if it skips. It self-skips without the env
  var, so the parity gate was reporting success while never running the reference oracle.
- mysqldialect: drive the is_blocked-recompute error-1093 rewrite test from the real
  issueops templates (new issueops.SelfReferentialRecomputeTemplates) instead of a
  hand-copied fixture, so template drift fails a non-gated test rather than only the
  env-gated MySQL conformance suite. All four mark/unmark x issue/wisp templates
  verified rewritten into the materialized derived-table form.

Note: the review re-flagged TestInitBackendFlag as failing, but it passes in a clean run
(plain and gms_pure_go, matching the scorecard's own rerun); the only cmd/bd failure in
this sandbox is an unrelated pre-existing embedded-Dolt/CGO test not in this PR's diff.

Review fixes -- attempt 4 (conformance-gate self-brick + all-wisps inline-dependency
drop blockers; Postgres error-log password-leak major):
- conformance.sh: anchor the embedded-Dolt reference skip/pass guards to top-level
  result lines (^--- SKIP/PASS: TestConformance). The attempt-3 loud-fail guard used
  an unanchored grep, so the indented, legitimate subtest skip of
  ClaimReadyIssueConcurrentExclusivity (embedded-Dolt is single-writer) matched it and
  false-failed the entire storage-parity gate on every run even when the top-level
  suite passed. Whole-suite-skip protection is preserved.
- sqlkit/portable + embeddeddolt/create_issue: the all-wisps arm looped the
  single-issue issueops.CreateIssueInTx, which runs no batch dependency-persist pass,
  so an all-wisps batch carrying inline dependencies (e.g. importing no-history beads
  with inter-bead edges -- export DOES include no-history beads, GH#2619) silently
  dropped every edge (not even reported via OnSkippedDependency) and, one transaction
  per issue, forfeited batch atomicity. Both now mark ephemeral, then run the whole
  batch in ONE transaction through issueops.CreateIssuesInTx, matching the server-mode
  DoltStore all-wisps arm. The embedded-Dolt reference oracle had the same latent drop,
  so the oracle and the new SQL backends were corrected together to keep conformance
  parity (the finding compared sqlkit against the server-mode DoltStore, which already
  persisted these edges).
- conformance: add CreateAllWispsInlineDependencies -- an all-wisp batch with one valid
  inter-wisp edge and one unresolvable edge; asserts the valid edge persists and the
  unresolvable one is reported via OnSkippedDependency. Fails on the pre-fix per-issue
  loop, passes on the batch path; verified on SQLite and the embedded-Dolt oracle, with
  a full embedded-Dolt conformance re-run clean.
- pgdialect: scrub query-param/sslpassword secrets from pgx ParseConfig errors in Open
  and OpenRaw (new ScrubDSNError). pgx's ParseConfigError redacts only URL userinfo, so
  a postgres://u@h/db?password=SECRET DSN (the shape bd init connects with) leaked the
  cleartext secret through the wrapped parse/TLS-config error into logs. Connect-time
  errors were verified not to carry the password. Adds redact_error_test coverage for
  the query-param, sslpassword, userinfo, and libpq-keyword shapes plus the real
  Open/OpenRaw parse-failure path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 17:44:37 +00:00

218 lines
7.2 KiB
Go

package sqlkit
import (
"context"
"database/sql"
"time"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/issueops"
"github.com/steveyegge/beads/internal/types"
)
// This file wires the shared "portable" non-version-control methods onto *Store. Each is
// the same thin issueops delegation the embedded-Dolt reference uses, so behavior is the
// reference's by construction; the dialect layer translates the underlying SQL. Reads run
// in a read tx, writes in a write tx.
// --- Molecule rollups ---
func (s *Store) GetMoleculeProgress(ctx context.Context, moleculeID string) (*types.MoleculeProgressStats, error) {
var out *types.MoleculeProgressStats
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
var e error
out, e = issueops.GetMoleculeProgressInTx(ctx, tx, moleculeID)
return e
})
return out, err
}
func (s *Store) GetMoleculeLastActivity(ctx context.Context, moleculeID string) (*types.MoleculeLastActivity, error) {
var out *types.MoleculeLastActivity
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
var e error
out, e = issueops.GetMoleculeLastActivityInTx(ctx, tx, moleculeID)
return e
})
return out, err
}
// --- Repo-mtime cache ---
func (s *Store) GetRepoMtime(ctx context.Context, repoPath string) (int64, error) {
var out int64
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
var e error
out, e = issueops.GetRepoMtimeInTx(ctx, tx, repoPath)
return e
})
return out, err
}
func (s *Store) SetRepoMtime(ctx context.Context, repoPath, jsonlPath string, mtimeNs int64) error {
return s.withWriteTx(ctx, func(tx *sql.Tx) error {
return issueops.SetRepoMtimeInTx(ctx, tx, repoPath, jsonlPath, mtimeNs)
})
}
func (s *Store) ClearRepoMtime(ctx context.Context, repoPath string) error {
return s.withWriteTx(ctx, func(tx *sql.Tx) error {
return issueops.ClearRepoMtimeInTx(ctx, tx, repoPath)
})
}
// --- Event / dependency streams ---
func (s *Store) GetAllEventsSince(ctx context.Context, since time.Time) ([]*types.Event, error) {
var out []*types.Event
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
var e error
out, e = issueops.GetAllEventsSinceInTx(ctx, tx, since)
return e
})
return out, err
}
func (s *Store) IterAllEventsSince(ctx context.Context, since time.Time) (storage.Iter[types.Event], error) {
events, err := s.GetAllEventsSince(ctx, since)
if err != nil {
return nil, err
}
return storage.NewSliceIter(events), nil
}
func (s *Store) GetAllDependencyRecords(ctx context.Context) (map[string][]*types.Dependency, error) {
var out map[string][]*types.Dependency
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
var e error
out, e = issueops.GetAllDependencyRecordsInTx(ctx, tx)
return e
})
return out, err
}
func (s *Store) IterAllDependencyRecords(ctx context.Context) (storage.Iter[types.Dependency], error) {
all, err := s.GetAllDependencyRecords(ctx)
if err != nil {
return nil, err
}
var flat []*types.Dependency
for _, deps := range all {
flat = append(flat, deps...)
}
return storage.NewSliceIter(flat), nil
}
// --- Dependent counts ---
// CountDependentsByStatus counts issues that depend on issueID whose source issue is in
// the given status, summed across the durable and wisp dependency tables. Mirrors the
// embedded-Dolt reference (counts.go): COALESCE resolves the edge target, and both
// classes are counted so a durable issue with a wisp dependent is not under-counted.
func (s *Store) CountDependentsByStatus(ctx context.Context, issueID string, status types.Status) (int64, error) {
var total int64
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
var perm, wisp int64
if err := tx.QueryRowContext(ctx, `
SELECT COUNT(*) FROM dependencies d
JOIN issues i ON i.id = d.issue_id
WHERE COALESCE(d.depends_on_issue_id, d.depends_on_wisp_id, d.depends_on_external) = ? AND i.status = ?
`, issueID, string(status)).Scan(&perm); err != nil {
return err
}
if err := tx.QueryRowContext(ctx, `
SELECT COUNT(*) FROM wisp_dependencies d
JOIN wisps w ON w.id = d.issue_id
WHERE COALESCE(d.depends_on_issue_id, d.depends_on_wisp_id, d.depends_on_external) = ? AND w.status = ?
`, issueID, string(status)).Scan(&wisp); err != nil {
return err
}
total = perm + wisp
return nil
})
return total, err
}
func (s *Store) FindWispDependentsRecursive(ctx context.Context, ids []string) (map[string]bool, error) {
var out map[string]bool
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
var e error
out, e = issueops.FindWispDependentsRecursiveInTx(ctx, tx, ids)
return e
})
return out, err
}
// --- Comment / audit writes ---
func (s *Store) AddComment(ctx context.Context, issueID, actor, comment string) error {
return s.withWriteTx(ctx, func(tx *sql.Tx) error {
return issueops.AddCommentEventInTx(ctx, tx, issueID, actor, comment)
})
}
func (s *Store) ImportIssueComment(ctx context.Context, issueID, author, text string, createdAt time.Time) (*types.Comment, error) {
var out *types.Comment
err := s.withWriteTx(ctx, func(tx *sql.Tx) error {
var e error
out, e = issueops.ImportIssueCommentInTx(ctx, tx, issueID, author, text, createdAt)
return e
})
return out, err
}
// --- Id rekey / wisp promote / source-repo purge ---
func (s *Store) UpdateIssueID(ctx context.Context, oldID, newID string, issue *types.Issue, actor string) error {
return s.withWriteTx(ctx, func(tx *sql.Tx) error {
return issueops.UpdateIssueIDInTx(ctx, tx, oldID, newID, issue, actor)
})
}
func (s *Store) PromoteFromEphemeral(ctx context.Context, id, actor string) error {
return s.withWriteTx(ctx, func(tx *sql.Tx) error {
return issueops.PromoteFromEphemeralInTx(ctx, tx, id, actor)
})
}
func (s *Store) DeleteIssuesBySourceRepo(ctx context.Context, sourceRepo string) (int, error) {
var count int
err := s.withWriteTx(ctx, func(tx *sql.Tx) error {
var e error
count, e = issueops.DeleteIssuesBySourceRepoInTx(ctx, tx, sourceRepo)
return e
})
return count, err
}
// --- Batch create ---
// CreateIssuesWithFullOptions mirrors the embedded-Dolt reference (dolt/issues.go):
// an all-wisps batch marks every issue ephemeral (unless it is explicitly
// no-history) so the rows route to the wisp tables, and EVERY batch — all-wisps or
// mixed/durable — is created in a SINGLE transaction through
// issueops.CreateIssuesInTx. That batch path is what threads opts, persists inline
// dependencies, applies conflict/stale policy, reconciles child counters, and
// recomputes is_blocked. The reference keeps two arms only to skip Dolt versioning
// for the all-wisps case; sqlkit has no Dolt versioning, so the arms differ solely
// in the ephemeral marking.
//
// Do NOT loop the single-issue issueops.CreateIssueInTx here: it never runs the
// dependency-persist pass, so an all-wisps batch carrying inline dependencies would
// silently drop every edge, and one transaction per issue would forfeit batch
// atomicity.
func (s *Store) CreateIssuesWithFullOptions(ctx context.Context, issues []*types.Issue, actor string, opts storage.BatchCreateOptions) error {
if len(issues) == 0 {
return nil
}
if issueops.AllWisps(issues) {
for _, issue := range issues {
if !issue.NoHistory {
issue.Ephemeral = true
}
}
}
return s.withWriteTx(ctx, func(tx *sql.Tx) error {
return issueops.CreateIssuesInTx(ctx, tx, issues, actor, opts)
})
}