beads/scripts/conformance.sh
Julian Knutsen ddca3e3ef1
fix(security,ci): redact all DSN password forms in telemetry; require pg/mysql conformance (post-merge #4601) (#4712)
* fix(security,ci): redact all DSN password forms in telemetry; require pg/mysql conformance

Post-merge remediation for #4601 (choose-your-own-backend), reviewed landed
range 812ad6d..3f0474a. Addresses the two gating findings from the post-merge
review.

Security (telemetry DSN leak): scrubArgsForTelemetry only redacted
user:PASS@host userinfo, so `bd init --pg-url` URL `?password=`/`?sslpassword=`
query params and libpq `password=`/`sslpassword=` keyword/value forms leaked
into the bd.args telemetry span before init reached the parser-backed
redaction path. Route --pg-url/--mysql-url values (both `--flag=value` and
`--flag value`) through the parser-backed pgdialect scrubber via a new exported
pgdialect.ScrubDSNString; every other arg keeps the narrow userinfo scrub, so
ordinary text is never over-redacted.

CI (conformance anti-skip): scripts/conformance.sh ran the pg/mysql Tier-1
suites with no skip/PASS detector, so a missing BEADS_*_TEST_URL self-skipped
to green and silently narrowed backend coverage. Add require_backend +
run_sql_conformance mirroring the embedded-Dolt reference guard, and set
BEADS_CONFORMANCE_REQUIRE=postgres,mysql in conformance.yml (which also arms
the existing Tier-2 e2e hard-fail).

Tests: pgdialect TestScrubDSNString; cmd/bd TestScrubArgsForTelemetry (all DSN
password forms x both spellings) and TestScrubArgsForTelemetryLeavesOrdinaryArgs
(no over-redaction + userinfo defense-in-depth). Bash guards unit- and
integration-tested against real `go test` self-skip/required paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): redact federation add-peer --password/-p from telemetry args

The bd.args telemetry root span scrubbed --pg-url/--mysql-url DSN passwords
but not federation add-peer's --password/-p, so `bd federation add-peer
<name> <url> --user u --password <secret>` leaked the plaintext SQL
password verbatim into the exported bd.args attribute and trace logs — the
same sink and harm class the DSN scrub already closed.

Redact by flag identity: secretFlagTokens resolves the secret flag's actual
--long/-short tokens from the running command, so -p is redacted only on the
command that binds it to --password (federation add-peer) and never on the
many commands that bind -p to --priority/--prefix/--parallel. The value is
redacted wholesale across every pflag spelling: --password <v>,
--password=<v>, -p <v>, -p=<v>, and the concatenated -p<v> shorthand.

Tests cover all five spellings, the command-aware token resolution, and the
no-over-redaction guarantee for the overloaded -p on non-secret commands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): redact raw encoded URL-userinfo passwords in DSN scrubber

Post-merge remediation follow-up review (attempt-4) for #4601: the shared
pgdialect DSN redactor missed percent-encoded URL-userinfo passwords when
called directly — ScrubDSNError in driver.go, and any telemetry/log sink that
scrubs a raw DSN through ScrubDSNString.

dsnPasswordValues added only url.User.Password(), which url.Parse returns
percent-DECODED, so ScrubDSNString("postgres://u:SUPER%2ASECRET@h/db", same)
left the raw "SUPER%2ASECRET" in error text and logs. The query-param branch
already adds both the raw and decoded forms; the userinfo branch now does too,
extracting the raw as-written password structurally (rawURLUserinfoPassword)
so it matches regardless of encoding form (mixed case, over-encoding).

Tests: add encoded-userinfo cases to TestScrubDSNError and TestScrubDSNString
(the latter now checks every leak form, matching TestScrubDSNError). Both new
cases fail on the pre-fix redactor and pass after.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): close five telemetry DSN-redaction bypasses found in review

Test-first fixes, each reproduced against pgx.ParseConfig before fixing:

- percent-encoded query KEYS (pass%77ord=, %70assword=) are live password
  params after url.Query() decoding; compare unescaped keys too
- pflag boolean-shorthand clusters (-qpSECRET parses as -q -p SECRET) leaked
  the raw token; walk the leading letter run to the first secret shorthand
- backslash-escaped whitespace in unquoted libpq values (password=SUPER\ SECRET)
  leaked the tail; value pattern now consumes escaped sequences
- pgx treats \v as DSN whitespace but Go regexp \s does not; spell the
  whitespace class out explicitly
- overlapping secrets (one a prefix of another) leaked the longer tail;
  replace longest-first in ScrubDSNString

Found by paired cross-vendor review (claude fable-5 + codex gpt-5.6-sol) on
PR 4712.

Agent-Signature: claude-fable-5-high on behalf of maphew

---------

Co-authored-by: Eddie the Engineer <test@test.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-12 13:44:35 -07:00

118 lines
6 KiB
Bash
Executable file

#!/usr/bin/env bash
#
# Single conformance entrypoint. CI runs this verbatim; run it locally the same way:
#
# ./scripts/conformance.sh
#
# Two tiers, both reading the backend registry (test/conformance/profiles.go for the
# E2E tier; the per-backend conformance_test.go files for the in-process tier). Add a
# backend = add a profile + a factory; both tiers pick it up.
#
# Backends that need an external service auto-skip when their env is unset:
# BEADS_PG_TEST_URL postgres://user:pass@host:port/db (enables the postgres backend)
# BEADS_PG_PASSWORD optional, if the password is not in the URL
#
# Optional deep gate (bts-rs 523-scenario differential oracle; needs the bts-rs
# checkout + ~50 min, so it is off by default and not part of the per-PR loop):
# CONFORMANCE_DEEP=1 BTS_RS_DIR=/path/to/bts-rs ./scripts/conformance.sh
#
set -euo pipefail
cd "$(dirname "$0")/.."
TAGS="gms_pure_go"
# assert_conformance_passed LOGFILE LABEL
# Fail the gate unless the top-level TestConformance in LOGFILE ran and PASSED. The
# checks MUST anchor to column-0 result lines: `go test -v` indents subtest results, and
# RunAll legitimately skips backend-inapplicable subtests (e.g. Claim concurrency on the
# single-writer embedded-Dolt reference), so an unanchored `--- SKIP: TestConformance`
# grep would match an indented subtest and false-fail the gate even though the top-level
# suite passed. Deletes LOGFILE either way.
assert_conformance_passed() {
local log="$1" label="$2"
if grep -qE '^--- SKIP: TestConformance' "$log" || ! grep -qE '^--- PASS: TestConformance' "$log"; then
rm -f "$log"
echo "FATAL: $label conformance (top-level TestConformance) skipped or did not run; storage-parity gate is not enforced" >&2
exit 1
fi
rm -f "$log"
}
# require_backend NAME -> success if NAME is listed (comma-separated) in
# BEADS_CONFORMANCE_REQUIRE. CI sets BEADS_CONFORMANCE_REQUIRE=postgres,mysql so a
# backend that self-skips because its BEADS_*_TEST_URL is missing becomes a hard
# failure instead of a silent green; unset locally, the backend self-skips as before.
require_backend() {
local want="$1" tok
local IFS=,
for tok in ${BEADS_CONFORMANCE_REQUIRE:-}; do
[ "$(printf '%s' "$tok" | tr -d '[:space:]')" = "$want" ] && return 0
done
return 1
}
# run_sql_conformance NAME PKG RUN URLVAR
# Run one SQL backend's in-process conformance + wedge gates. When NAME is required
# (see require_backend), run verbose and assert the TestConformance parity oracle
# actually ran and passed — mirroring the embedded-Dolt reference guard so a missing
# BEADS_*_TEST_URL can't silently narrow coverage. Otherwise run plain and let the
# suite self-skip. URLVAR names the env var to set, surfaced in the failure hint.
run_sql_conformance() {
local name="$1" pkg="$2" run="$3" urlvar="$4"
if require_backend "$name"; then
local log
log="$(mktemp)"
CGO_ENABLED=1 go test -tags "$TAGS" -v "$pkg" -run "$run" | tee "$log"
assert_conformance_passed "$log" "$name backend (set $urlvar)"
else
CGO_ENABLED=1 go test -tags "$TAGS" "$pkg" -run "$run"
fi
}
echo "==> Tier 1: in-process store conformance + wedge gates"
# The embedded-Dolt backend runs the full backend-agnostic suite (conformance.RunAll) as
# the reference oracle every SQL backend is compared against. BEADS_TEST_EMBEDDED_DOLT=1
# is REQUIRED, not optional: without it that suite self-skips
# (internal/storage/embeddeddolt/conformance_test.go), which would let this whole
# storage-parity gate report success while never running the oracle. Run it with -v and
# fail loudly if the top-level TestConformance skips or reports no pass, so the gate can
# never go green on a silent skip.
# (assert_conformance_passed explains why the skip/pass checks anchor to column-0 lines.)
embedded_dolt_log="$(mktemp)"
BEADS_TEST_EMBEDDED_DOLT=1 CGO_ENABLED=1 go test -tags "$TAGS" -v \
./internal/storage/embeddeddolt/ -run TestConformance | tee "$embedded_dolt_log"
assert_conformance_passed "$embedded_dolt_log" "embedded-Dolt reference (need BEADS_TEST_EMBEDDED_DOLT=1)"
# Each SQL backend now passes the FULL in-process store conformance suite (TestConformance
# = conformance.RunAll, the same ~40 behavior subtests the Dolt reference runs) — the only
# methods it does NOT implement are the genuinely-Dolt-only ones (version-control/remote/
# sync/…), which RunAll does not exercise. Alongside RunAll: live smoke, the
# interface-completeness audit (shell == deferral allowlist — no SILENT unsupported) plus
# its behavioral complement (every allowlisted method returns typed ErrUnsupported), the
# seed-once regression, and the dialect corpus-PREPARE + password-redaction gates. All
# self-skip without BEADS_PG_TEST_URL — or hard-fail when BEADS_CONFORMANCE_REQUIRE
# lists postgres (see run_sql_conformance), so CI can't go green with the wedge absent.
run_sql_conformance postgres ./internal/storage/postgres/ \
'TestPGSmoke|TestInterfaceCompleteness|TestUnsupportedContract|TestConformance|TestSeedOnlyOnFirstProvision' \
BEADS_PG_TEST_URL
CGO_ENABLED=1 go test -tags "$TAGS" ./internal/storage/pgdialect/
# MySQL wedge gates (self-skip without BEADS_MYSQL_TEST_URL, or hard-fail when
# BEADS_CONFORMANCE_REQUIRE lists mysql); the dialect rewrite test (the is_blocked 1093
# workaround) always runs.
run_sql_conformance mysql ./internal/storage/mysql/ \
'TestInterfaceCompleteness|TestUnsupportedContract|TestConformance|TestSeedOnlyOnFirstProvision' \
BEADS_MYSQL_TEST_URL
CGO_ENABLED=1 go test -tags "$TAGS" ./internal/storage/mysqldialect/
# SQLite is embedded (pure-Go), always runs.
CGO_ENABLED=1 go test -tags "$TAGS" ./internal/storage/sqlite/ \
-run 'TestInterfaceCompleteness|TestUnsupportedContract|TestConformance|TestSeedOnlyOnFirstProvision'
CGO_ENABLED=1 go test -tags "$TAGS" ./internal/storage/sqlitedialect/
echo "==> Tier 2: end-to-end 'bd init' + CLI conformance (differential vs Dolt)"
CGO_ENABLED=1 go test -tags "$TAGS e2e" ./test/conformance/
if [[ "${CONFORMANCE_DEEP:-0}" == "1" ]]; then
echo "==> Deep: bts-rs 523-scenario differential oracle"
./scripts/run-oracle-p.sh
fi
echo "==> conformance OK"