Commit graph

8 commits

Author SHA1 Message Date
ecuthiell
4bcea42d8a
ci: give conformance explicit timeout budgets (#5073)
The shared conformance entrypoint inherited Go's 10-minute default even though the embedded oracle regularly needs longer and dedicated jobs already budget at least 30 minutes. Give Tier 1 the maintained 30-minute budget, make Tier 2's existing 10-minute behavior explicit, and bound the outer workflow at 45 minutes.

Test the exact Go argv, failure propagation from both tiers, and the outer workflow budget without sleeping through the real deadlines.

Agent-Signature: codex-tui-gpt-5.6-sol-ultra on behalf of Ewen Cuthiell
2026-07-28 09:01:39 -07:00
Test User
6fd9dbda7d refactor(storage)!: remove the SQLite backend and dialect
Extend #4847's rollback to the last alternative backend: bd now ships a
single storage engine (Dolt, embedded and server mode).

- Delete internal/storage/{sqlite,sqlitedialect} and the packages the
  removal orphans: sqlkit (sole importer was the sqlite store),
  unsupportedgen (sole go:generate consumer), the NonCommitGraphBackend
  marker and its two dead consumer checks, and the domain-db parity test
  that used sqlite as its harness. Drop modernc.org/sqlite from go.mod
  and regenerate the nix vendorHash.
- Tombstone backend="sqlite" exactly like postgres/mysql: recognized in
  metadata (never silently aliased to a fresh Dolt DB), rejected before
  any open with its own rationale and the export -> re-init -> import
  guidance. The fail-closed guarantee now also covers never creating the
  beads.db file the removed backend would have provisioned.
- bd init: --backend=sqlite reaches removed-backend guidance;
  --sqlite-path becomes a hidden parser tombstone like the pg/mysql
  flags; help text and supported-backend enumerations say Dolt only.
- Doctor: sqlite workspaces fail closed like the other tombstones; the
  modern sqlite stub payloads are gone. The legacy pre-Dolt surface
  (artifact scans, migration validation, migrate-sqlite-to-current.sh,
  the migration-test harness) is untouched; genuine pre-Dolt workspaces
  carry no backend field and never hit the tombstone.
- Conformance: drop the sqlite profile; the CLI differential runner
  idles (skip, not fail) until another backend profile is registered,
  and the export/import round-trip keeps running against the embedded
  Dolt reference. Tier-1 oracle legs are unchanged.

Backend selection has never shipped in a release (v1.1.0 predates it),
so released workflows are unaffected and the pinned CLI docs already
describe the resulting behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 02:43:56 +00:00
Test User
0d7cf0022d Reapply "Simplify the supported backends: roll back the direct PostgreSQL and MySQL adapters" (#4857)
This reverts commit 0cf3b3ca09.
2026-07-18 02:43:56 +00:00
Julian Knutsen
0cf3b3ca09
Revert "Simplify the supported backends: roll back the direct PostgreSQL and MySQL adapters" (#4857) 2026-07-16 16:35:51 -07:00
Test User
cafb6888bd refactor(storage): remove direct PostgreSQL and MySQL adapters (bd-sadcd)
Roll back the direct adapters, dialect layers, credential handling, dependencies, CI services, and conformance profiles for general-purpose server databases. That support adds operational and testing complexity at odds with our goal of keeping Beads as simple as possible and consuming as few resources as possible.

Keep the Storage and DoltStorage interfaces unchanged. Embedded Dolt, Dolt server, and SQLite remain supported; the MySQL wire driver remains because Dolt server uses it.

Retain legacy metadata identifiers and hidden init flags only as compatibility tombstones. Removed, unknown, or corrupt backend metadata now fails safely across CLI, SDK, bootstrap, doctor, factories, and Dolt administration before storage is opened or modified, with guidance to export using a supporting build and reinitialize via bd help init-safety.

Agent-Signature: codex-gpt-5-unknown-reasoning on behalf of Test User
2026-07-16 03:35:18 +00:00
Julian Knutsen
ddca3e3ef1
fix(security,ci): redact all DSN password forms in telemetry; require pg/mysql conformance (post-merge #4601) (#4712)
* fix(security,ci): redact all DSN password forms in telemetry; require pg/mysql conformance

Post-merge remediation for #4601 (choose-your-own-backend), reviewed landed
range 812ad6d..3f0474a. Addresses the two gating findings from the post-merge
review.

Security (telemetry DSN leak): scrubArgsForTelemetry only redacted
user:PASS@host userinfo, so `bd init --pg-url` URL `?password=`/`?sslpassword=`
query params and libpq `password=`/`sslpassword=` keyword/value forms leaked
into the bd.args telemetry span before init reached the parser-backed
redaction path. Route --pg-url/--mysql-url values (both `--flag=value` and
`--flag value`) through the parser-backed pgdialect scrubber via a new exported
pgdialect.ScrubDSNString; every other arg keeps the narrow userinfo scrub, so
ordinary text is never over-redacted.

CI (conformance anti-skip): scripts/conformance.sh ran the pg/mysql Tier-1
suites with no skip/PASS detector, so a missing BEADS_*_TEST_URL self-skipped
to green and silently narrowed backend coverage. Add require_backend +
run_sql_conformance mirroring the embedded-Dolt reference guard, and set
BEADS_CONFORMANCE_REQUIRE=postgres,mysql in conformance.yml (which also arms
the existing Tier-2 e2e hard-fail).

Tests: pgdialect TestScrubDSNString; cmd/bd TestScrubArgsForTelemetry (all DSN
password forms x both spellings) and TestScrubArgsForTelemetryLeavesOrdinaryArgs
(no over-redaction + userinfo defense-in-depth). Bash guards unit- and
integration-tested against real `go test` self-skip/required paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): redact federation add-peer --password/-p from telemetry args

The bd.args telemetry root span scrubbed --pg-url/--mysql-url DSN passwords
but not federation add-peer's --password/-p, so `bd federation add-peer
<name> <url> --user u --password <secret>` leaked the plaintext SQL
password verbatim into the exported bd.args attribute and trace logs — the
same sink and harm class the DSN scrub already closed.

Redact by flag identity: secretFlagTokens resolves the secret flag's actual
--long/-short tokens from the running command, so -p is redacted only on the
command that binds it to --password (federation add-peer) and never on the
many commands that bind -p to --priority/--prefix/--parallel. The value is
redacted wholesale across every pflag spelling: --password <v>,
--password=<v>, -p <v>, -p=<v>, and the concatenated -p<v> shorthand.

Tests cover all five spellings, the command-aware token resolution, and the
no-over-redaction guarantee for the overloaded -p on non-secret commands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): redact raw encoded URL-userinfo passwords in DSN scrubber

Post-merge remediation follow-up review (attempt-4) for #4601: the shared
pgdialect DSN redactor missed percent-encoded URL-userinfo passwords when
called directly — ScrubDSNError in driver.go, and any telemetry/log sink that
scrubs a raw DSN through ScrubDSNString.

dsnPasswordValues added only url.User.Password(), which url.Parse returns
percent-DECODED, so ScrubDSNString("postgres://u:SUPER%2ASECRET@h/db", same)
left the raw "SUPER%2ASECRET" in error text and logs. The query-param branch
already adds both the raw and decoded forms; the userinfo branch now does too,
extracting the raw as-written password structurally (rawURLUserinfoPassword)
so it matches regardless of encoding form (mixed case, over-encoding).

Tests: add encoded-userinfo cases to TestScrubDSNError and TestScrubDSNString
(the latter now checks every leak form, matching TestScrubDSNError). Both new
cases fail on the pre-fix redactor and pass after.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): close five telemetry DSN-redaction bypasses found in review

Test-first fixes, each reproduced against pgx.ParseConfig before fixing:

- percent-encoded query KEYS (pass%77ord=, %70assword=) are live password
  params after url.Query() decoding; compare unescaped keys too
- pflag boolean-shorthand clusters (-qpSECRET parses as -q -p SECRET) leaked
  the raw token; walk the leading letter run to the first secret shorthand
- backslash-escaped whitespace in unquoted libpq values (password=SUPER\ SECRET)
  leaked the tail; value pattern now consumes escaped sequences
- pgx treats \v as DSN whitespace but Go regexp \s does not; spell the
  whitespace class out explicitly
- overlapping secrets (one a prefix of another) leaked the longer tail;
  replace longest-first in ScrubDSNString

Found by paired cross-vendor review (claude fable-5 + codex gpt-5.6-sol) on
PR 4712.

Agent-Signature: claude-fable-5-high on behalf of maphew

---------

Co-authored-by: Eddie the Engineer <test@test.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-12 13:44:35 -07:00
Eddie the Engineer
c79bb32bca fix(backends): maintainer fixup — CI repair + adoption review fixes
Maintainer fixup for the choose-your-own-storage-backend PR, amended across the
pre-review CI-repair and the review-loop apply-fixes iterations so contributor
commits stay intact under one maintainer commit.

CI repair (pre-review):
- default.nix: bump vendorHash for the added Postgres/MySQL driver deps.
- conformance.yml: provision the Postgres service DB with a C-locale cluster so
  text ORDER BY sorts by code point, matching Dolt's utf8mb4_0900_bin oracle.

Review fixes — blockers:
- uow/doltserver_tx: on a non-serialization DOLT_COMMIT failure, ROLLBACK on the
  pinned session before releasing it, and poison (pool-discard via driver.ErrBadConn)
  the connection when the rollback also fails, so a later borrower can't inherit an
  open transaction and implicitly commit orphaned writes. Satisfies the existing
  doltserver_tx_test rollback/poison cases.
- sqlitedialect: translate GREATEST/LEAST to SQLite scalar max/min so the
  ReconcileChildCounters upsert (GREATEST(last_child, ?)) no longer aborts creation
  or import of hierarchical dotted issue IDs (e.g. x-1.2) on the SQLite backend.

Review fixes — majors:
- conformance: add a portable dotted hierarchical child-ID scenario (x-1, x-1.1,
  x-1.2, x-1.2.1) exercising the ReconcileChildCounters GREATEST path on every
  backend — the coverage hole that let the SQLite blocker ship green. Verified
  against the embedded-Dolt oracle and the real SQLite driver.
- postgres: warn at provision/open when the database collation is not code-point
  ordered (C/POSIX/C.*), since text ORDER BY otherwise diverges from Dolt silently.
- sqlite: pin the pool to one connection and add busy_timeout + WAL to the DSN so
  overlapping operations wait instead of surfacing a raw "database is locked"
  (matches the Dolt backend's single-connection posture). Adds a concurrency guard.
- cmd/bd: gate explicit `bd dolt commit` / `bd vc commit` with a typed
  storage.ErrUnsupported on non-Dolt backends instead of the no-op Commit printing
  "Committed."; internal opportunistic auto-commit is untouched.

Review fixes — attempt 2 (stale init backend-flag test contract):
- cmd/bd/init_test, init_embedded_test: the init backend-flag tests still asserted
  the pre-#3151 one-backend contract (--backend=sqlite must fail "SQLite backend
  has been removed"; --backend=postgres is an "unknown backend"), so
  `go test ./cmd/bd -run '^TestInitBackendFlag$'` failed in a clean environment now
  that sqlite/postgres/mysql are supported pluggable backends. Rewrite
  TestInitBackendFlag and the cgo TestEmbeddedInit sibling (rejected_backends →
  sql_backend_flags) for the new contract: --backend=sqlite initializes and
  persists backend=sqlite; --backend=postgres is recognized and fails only on
  missing connection config (not "unknown backend"); a genuinely unknown backend
  is still rejected. Pin BEADS_DIR and scrub BEADS_*/BD_* so the subtests are
  hermetic against an ambient .beads or Dolt server.

Cleanups (non-gating):
- cmd/bd/init: extract checkExistingSQLBackend so the added SQL-backend branches
  don't inflate checkExistingBeadsDataAt's complexity, and load config once.
- configfile: correct the stale "Deprecated: always dolt" Backend field comment.

Review fixes -- attempt 3 (SQLite datetime parity blocker + conformance/rewrite
hardening of the storage-parity gate):
- sqlite/dsn: add _time_format=datetime. Without it modernc binds time.Time via
  t.String() ("2026-07-09 12:34:56.123 +0000 UTC"), a shape SQLite's date functions
  cannot parse, so the counts mega-query's DATE_FORMAT(created_at,...) (strftime on
  SQLite) returns NULL and deps_json exposes a zero created_at while Dolt emits the
  real timestamp. datetime stores whole-second "2026-07-09 12:34:56", matching
  Dolt/MySQL datetime(0). Verified end-to-end against the real modernc driver.
- conformance: add ReadyWorkDepCreatedAtParity -- imports a dependency with a known
  created_at (the bd import / CreateIssuesWithFullOptions bound-time.Time path) and
  asserts the counts-rendered deps_json created_at equals it, closing the blind spot
  that let the SQLite blocker ship green (the counts tests only checked DependencyCount).
  Proven to fail on the pre-fix DSN and pass on both the fixed SQLite backend and the
  embedded-Dolt oracle.
- conformance: add ClaimReadyIssueConcurrentExclusivity -- races 24 workers over 8
  ready issues and asserts each is claimed exactly once, exercising the SQL backends'
  multi-writer claim property that was previously only asserted, never raced. Gated to
  the multi-connection dialects (postgres/mysql) via DialectName; SQLite (single-conn
  pool) and embedded-Dolt skip.
- conformance.sh: run the embedded-Dolt reference conformance with
  BEADS_TEST_EMBEDDED_DOLT=1 and fail loudly if it skips. It self-skips without the env
  var, so the parity gate was reporting success while never running the reference oracle.
- mysqldialect: drive the is_blocked-recompute error-1093 rewrite test from the real
  issueops templates (new issueops.SelfReferentialRecomputeTemplates) instead of a
  hand-copied fixture, so template drift fails a non-gated test rather than only the
  env-gated MySQL conformance suite. All four mark/unmark x issue/wisp templates
  verified rewritten into the materialized derived-table form.

Note: the review re-flagged TestInitBackendFlag as failing, but it passes in a clean run
(plain and gms_pure_go, matching the scorecard's own rerun); the only cmd/bd failure in
this sandbox is an unrelated pre-existing embedded-Dolt/CGO test not in this PR's diff.

Review fixes -- attempt 4 (conformance-gate self-brick + all-wisps inline-dependency
drop blockers; Postgres error-log password-leak major):
- conformance.sh: anchor the embedded-Dolt reference skip/pass guards to top-level
  result lines (^--- SKIP/PASS: TestConformance). The attempt-3 loud-fail guard used
  an unanchored grep, so the indented, legitimate subtest skip of
  ClaimReadyIssueConcurrentExclusivity (embedded-Dolt is single-writer) matched it and
  false-failed the entire storage-parity gate on every run even when the top-level
  suite passed. Whole-suite-skip protection is preserved.
- sqlkit/portable + embeddeddolt/create_issue: the all-wisps arm looped the
  single-issue issueops.CreateIssueInTx, which runs no batch dependency-persist pass,
  so an all-wisps batch carrying inline dependencies (e.g. importing no-history beads
  with inter-bead edges -- export DOES include no-history beads, GH#2619) silently
  dropped every edge (not even reported via OnSkippedDependency) and, one transaction
  per issue, forfeited batch atomicity. Both now mark ephemeral, then run the whole
  batch in ONE transaction through issueops.CreateIssuesInTx, matching the server-mode
  DoltStore all-wisps arm. The embedded-Dolt reference oracle had the same latent drop,
  so the oracle and the new SQL backends were corrected together to keep conformance
  parity (the finding compared sqlkit against the server-mode DoltStore, which already
  persisted these edges).
- conformance: add CreateAllWispsInlineDependencies -- an all-wisp batch with one valid
  inter-wisp edge and one unresolvable edge; asserts the valid edge persists and the
  unresolvable one is reported via OnSkippedDependency. Fails on the pre-fix per-issue
  loop, passes on the batch path; verified on SQLite and the embedded-Dolt oracle, with
  a full embedded-Dolt conformance re-run clean.
- pgdialect: scrub query-param/sslpassword secrets from pgx ParseConfig errors in Open
  and OpenRaw (new ScrubDSNError). pgx's ParseConfigError redacts only URL userinfo, so
  a postgres://u@h/db?password=SECRET DSN (the shape bd init connects with) leaked the
  cleartext secret through the wrapped parse/TLS-config error into logs. Connect-time
  errors were verified not to carry the password. Adds redact_error_test coverage for
  the query-param, sslpassword, userinfo, and libpq-keyword shapes plus the real
  Open/OpenRaw parse-failure path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 17:44:37 +00:00
Test User
1fc38ba778 feat: choose your own storage backend (Postgres / MySQL / SQLite) — rebased on main
Rebase of PR #4601 (feat/choose-your-own-backend) onto current origin/main,
squashed to a single commit because the original 73-commit history built and then
removed the uowStore derisk spike incrementally, which does not replay cleanly
over the 125 commits main has since advanced (notably #4679, the ready-work
counts fix, which merged to main and is now inherited rather than re-applied).
The full pre-rebase history remains on the original branch tip for reference.

Contents (multi-backend feature + the six review follow-ups):
- bd init --backend=postgres|mysql|sqlite provisioning behind the Dolt-parity
  seam; the three SQL backends share *sqlkit.Store over pluggable dialects and
  carry a generated typed-unsupported shell for the non-core surface.
- Passwords in init DSNs are redacted before persisting to metadata.json (driver
  parser + re-parse verification), with a negative-test matrix proving no secret
  survives (pgdialect + mysql).
- The uowStore derisk spike and its report artifacts are removed; the shell
  generator now lives in internal/storage/unsupportedgen (neutral name).
- Compile-time guard `var _ storage.DoltStorage = (*Store)(nil)` on each backend
  plus a regen-idempotence test keep the generated skip lists honest.
- sqlkit exposes CountReadyWork so every SQL backend satisfies
  storage.ReadyWorkCounter and inherits the shared page-pushed ready-counts path.
- Commercial/product names scrubbed from the feature's own code for vendor
  neutrality.

Rebase reconciliations: adopted main's metadata-load hardening in the store
factories alongside the backend dispatch; combined the config interface
(GetMetadata + ReconcileVersion) and the conformance registrations (ReadyCounts +
Claim/lease); took main's versions of the shared-infra concurrency (uow
doltserver_tx) and delete-cascade (domain/issue_delete) evolutions; implemented
the interface's new UnclaimIssue on sqlkit (delegating to issueops) and added it
to the skip lists.

Verified: go build (cgo + CGO_ENABLED=0) and go vet clean; unit tests
(sqlbuild/issueops/pgdialect/mysql/unsupportedgen) and the SQLite conformance
suite green (RunAll, incl. ReadyCounts equivalence and Claim/lease on the real
sqlkit backend).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:44:37 +00:00