beads/scripts/migrate-legacy-to-current.sh
Julian Knutsen 92ad0295f1
Guard and qualify historical upgrades to v1.2 (#4907)
* Add sealed-copy SQLite migration bridge (bd-mvh)

Create a fresh current workspace from a fingerprinted copy of an authenticated historical SQLite workspace. Keep every historical executable inside an isolated destination and reject malformed or semantically lossy cutovers without touching the source.

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Test User <test@test.com>
Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

* Refuse implicit historical workspace upgrades (bd-mvh)

Detect historical SQLite and legacy Dolt layouts before config migration, version tracking, init, bootstrap, doctor, or store construction. Require the sealed-copy bridge while preserving current shared-server admission against the explicitly selected workspace.

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Test User <test@test.com>
Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

* Qualify authenticated historical upgrades (bd-mvh)

Replace generic recipe discovery with a fixed, checksum-pinned corpus spanning the reviewed SQLite, server-Dolt, and embedded-Dolt eras. Assert source isolation, representative data fidelity, schema idempotence, and rollback retention while keeping historical executables inside bounded temporary environments.

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Test User <test@test.com>
Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

* Add authenticated legacy SQLite reader (bd-tku)

Let the current binary read the exact v0.49.6/v0.50.3 SQLite contract from a sealed private DB/WAL copy and emit canonical JSONL for the normal fresh-Dolt import path. Reject schema drift and values that current import would lose, default, truncate, or deduplicate.

Qualify both authentic releases end to end, including current DATETIME(0) canonicalization and reader-to-fresh-import timestamp equality.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

* Fix empty embedded-root upgrade bypass (bd-e6m)

Require an actual embedded Dolt repository marker before current layout precedence can suppress historical SQLite or legacy Dolt refusal.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

* fix(migrate): maintainer review fixups for legacy SQLite upgrade PR

- decompose the legacy-workspace guard and extract the dependency-row loader
  (behavior-preserving), and align the legacy SQLite reader projection with
  issueops.ScanIssueFrom arity after the 0060/0061 rebase, guarded by a
  projection arity test.

- assert the historical-upgrade harness schema no-op by shape
  ("Schema already at v<N>") instead of a hardcoded v59, so the migration-test
  matrix stops reddening on every schema bump (candidate now prints v61).

* Fix historical upgrade review blockers (bd-mvh)

Route direct SQLite lanes through the current authenticated reader, guard no-store commands before legacy workspace mutation, and move coarse embedded repository detection behind the storage adapter. Restore the legacy reader projection contract and verify source-preserving behavior across the repaired paths.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

* Fix historical upgrade edge cases (bd-mvh)

Refuse ambiguous local server-Dolt roots before storage opens and normalize nullable legacy SQLite sentinel fields during authenticated export.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

* Fix final historical upgrade review blockers (bd-mvh)

Make explicit server selection outrank stale embedded repository markers, so ambiguous local server roots still fail closed.

Permit zero-record cutovers only through the authenticated direct SQLite reader, verify the empty round trip against a fresh current export, and align the upgrade guidance with the enforced admission matrix.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

---------

Co-authored-by: CI Bot <ci@beads.test>
Co-authored-by: Test User <test@test.com>
2026-08-06 18:45:04 -07:00

402 lines
16 KiB
Bash
Executable file

#!/usr/bin/env bash
set -euo pipefail
# Build a fresh current workspace from a sealed copy of a historical SQLite one.
# The original SOURCE is fingerprinted before and after every migration step and
# is never renamed, removed, or opened by either migration binary.
usage() {
cat <<'EOF'
Usage: migrate-legacy-to-current.sh \
--source DIR --destination DIR --source-version VERSION \
--old-bd PATH --new-bd PATH --prefix PREFIX \
[--canonicalizer-bd PATH]
SOURCE must contain one historical SQLite database under .beads. DESTINATION
must not exist and must resolve outside SOURCE.
Versions before v0.49.6 require a verified v0.49.6 binary supplied through
--canonicalizer-bd. The script opens only a sealed copy, exports canonical
JSONL, imports it into DESTINATION/cutover, and compares the candidate export
semantically before reporting success.
EOF
}
source=""
destination=""
source_version=""
old_bd=""
new_bd=""
canonicalizer_bd=""
prefix=""
while (($#)); do
case "$1" in
--source) source=${2:?}; shift 2 ;;
--destination) destination=${2:?}; shift 2 ;;
--source-version) source_version=${2:?}; shift 2 ;;
--old-bd) old_bd=${2:?}; shift 2 ;;
--new-bd) new_bd=${2:?}; shift 2 ;;
--canonicalizer-bd) canonicalizer_bd=${2:?}; shift 2 ;;
--prefix) prefix=${2:?}; shift 2 ;;
-h|--help) usage; exit 0 ;;
*) printf 'unknown argument: %s\n' "$1" >&2; usage >&2; exit 2 ;;
esac
done
for value in source destination source_version old_bd new_bd prefix; do
[ -n "${!value}" ] || { usage >&2; exit 2; }
done
die() {
printf 'legacy migration: %s\n' "$*" >&2
exit 1
}
require_regular_executable() {
[ -f "$1" ] && [ ! -L "$1" ] && [ -x "$1" ] ||
die "required executable must be a non-symlink regular file: $1"
}
canonical_file() {
local path=$1 parent base
parent=$(dirname "$path")
base=$(basename "$path")
parent=$(cd -P "$parent" && pwd -P) || return 1
printf '%s/%s\n' "$parent" "$base"
}
canonical_new_path() {
local path=${1%/} parent base
[ -n "$path" ] || return 1
base=${path##*/}
[ "$base" != "." ] && [ "$base" != ".." ] && [ -n "$base" ] || return 1
parent=${path%/*}
[ "$parent" != "$path" ] || parent=.
parent=$(cd -P "$parent" && pwd -P) || return 1
printf '%s/%s\n' "$parent" "$base"
}
sha256_file() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | awk '{print $1}'
elif command -v shasum >/dev/null 2>&1; then
shasum -a 256 "$1" | awk '{print $1}'
else
die 'no SHA-256 utility is available (need sha256sum or shasum)'
fi
}
sha256_stream() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum | awk '{print $1}'
elif command -v shasum >/dev/null 2>&1; then
shasum -a 256 | awk '{print $1}'
else
die 'no SHA-256 utility is available (need sha256sum or shasum)'
fi
}
fingerprint() {
(
cd "$1"
find . -type f -print | LC_ALL=C sort |
while IFS= read -r file; do
printf '%s\n' "$file"
sha256_file "$file"
done
) | sha256_stream
}
classify_source_version() {
case "$1" in
v0.9.1) printf 'materialize-and-canonicalize\n' ;;
v0.17.0) printf 'canonicalize\n' ;;
v0.49.6|v0.50.3) printf 'direct\n' ;;
*) die "source version is not one of the authenticated SQLite releases: $1" ;;
esac
}
verify_binary_version() {
local expected=$1 binary=$2 output bare
output=$(run_at "$destination/probe" "$binary" version 2>&1) || die "$binary does not run"
bare=${expected#v}
if [ "$expected" = v0.9.1 ]; then
grep -Fq 'bd version 0.9.0 (dev)' <<< "$output" ||
die "v0.9.1 source binary has unexpected version output: $output"
else
grep -Eq "(^|[[:space:]])${bare//./\\.}([[:space:]]|\\(|$)" <<< "$output" ||
die "source binary reports an unexpected version: $output"
fi
}
normalize_export() {
jq -S -s '
def stamp:
if type != "string" or (test("\\.[0-9]+Z$") | not) then .
else
. as $raw
| (
(sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601)
+ (if ($raw | test("\\.[5-9][0-9]*Z$")) then 1 else 0 end)
)
| todateiso8601
end;
def normalize_times:
reduce ["created_at", "updated_at", "started_at", "closed_at",
"due_at", "defer_until", "compacted_at"][] as $key (.;
if has($key) and .[$key] != null
then .[$key] |= stamp
else .
end);
def dependency:
{
issue_id: (.issue_id // ""),
depends_on_id: (.depends_on_id // .id // ""),
type: (.type // .dependency_type // "blocks"),
created_at: ((.created_at // "") | stamp),
created_by: (.created_by // "unknown"),
metadata: (.metadata // "{}")
};
def comment:
{
issue_id: (.issue_id // ""),
author: (.author // ""),
text: (.text // ""),
created_at: ((.created_at // "") | stamp)
};
map(
if (._type // "issue") == "memory" then
{kind: "memory", key: .key, value: .value}
else
del(._type, .comment_count, .dependency_count, .dependent_count)
| normalize_times
| .labels = ((.labels // []) | sort)
| .dependencies = ((.dependencies // []) | map(dependency) |
sort_by(.issue_id, .depends_on_id, .type))
| .comments = ((.comments // []) | map(comment) |
sort_by(.issue_id, .author, .text))
| .kind = "issue"
end
)
| sort_by(.kind, (.id // .key))
' "$1"
}
validate_sqlite_descriptor() {
local descriptor=$1 name=$2
[ ! -e "$descriptor" ] && [ ! -L "$descriptor" ] && return 0
[ -f "$descriptor" ] && [ ! -L "$descriptor" ] ||
die "source $name must be a non-symlink regular file"
jq -e '
type == "object" and
((.backend // "sqlite") == "sqlite") and
([keys[] | ascii_downcase |
select(
startswith("postgres_") or
startswith("mysql_") or
startswith("dolt_") or
contains("provider") or
contains("dsn") or
contains("url")
)
] | length == 0)
' "$descriptor" >/dev/null ||
die "source $name must describe SQLite without alternate storage settings"
}
validate_sqlite_database_reference() {
local descriptor=$1 name=$2 database_name=$3
[ ! -e "$descriptor" ] && return 0
jq -e --arg database_name "$database_name" '
def local_database:
. == null or . == "" or . == $database_name;
(.database | local_database) and
(.sqlite_path | local_database)
' "$descriptor" >/dev/null ||
die "source $name points outside the sole local SQLite database"
}
verify_current_cutover() {
local beads_dir=$1 metadata active_path
metadata="$beads_dir/metadata.json"
[ -f "$metadata" ] && [ ! -L "$metadata" ] ||
die 'candidate did not write a non-symlink metadata.json'
jq -e '.backend == "dolt" and .dolt_mode == "embedded"' "$metadata" >/dev/null ||
die 'candidate did not select embedded Dolt'
[ -d "$beads_dir/embeddeddolt" ] && [ ! -L "$beads_dir/embeddeddolt" ] ||
die 'candidate did not create a non-symlink embedded Dolt directory'
if ! (set +o pipefail; find "$beads_dir/embeddeddolt" -type f -print | grep -q .); then
die 'candidate embedded Dolt directory contains no storage files'
fi
[ ! -e "$beads_dir/dolt" ] && [ ! -L "$beads_dir/dolt" ] ||
die 'candidate left an active legacy Dolt directory'
for active_path in "$beads_dir/"*.db; do
[ ! -e "$active_path" ] && [ ! -L "$active_path" ] ||
die 'candidate left an active SQLite database'
done
for active_path in "$beads_dir"/legacy-dolt-*; do
[ ! -e "$active_path" ] && [ ! -L "$active_path" ] ||
die 'candidate left a legacy Dolt directory'
done
}
require_regular_executable "$old_bd"
require_regular_executable "$new_bd"
command -v jq >/dev/null 2>&1 || die 'jq is required'
command -v git >/dev/null 2>&1 || die 'git is required'
source=$(cd -P "$source" && pwd -P) || die "source directory is unavailable: $source"
destination=$(canonical_new_path "$destination") ||
die 'destination parent must already exist and resolve to a physical directory'
old_bd=$(canonical_file "$old_bd") || die "cannot resolve old bd binary: $old_bd"
new_bd=$(canonical_file "$new_bd") || die "cannot resolve new bd binary: $new_bd"
source_version="v${source_version#v}"
strategy=$(classify_source_version "$source_version")
if [ "$strategy" != direct ]; then
[ -n "$canonicalizer_bd" ] ||
die "$source_version requires --canonicalizer-bd with a verified v0.49.6 binary"
require_regular_executable "$canonicalizer_bd"
canonicalizer_bd=$(canonical_file "$canonicalizer_bd") ||
die "cannot resolve canonicalizer binary: $canonicalizer_bd"
fi
[ -d "$source/.beads" ] && [ ! -L "$source/.beads" ] ||
die "source has no non-symlink .beads directory: $source"
[ ! -e "$source/.beads/dolt" ] && [ ! -L "$source/.beads/dolt" ] &&
[ ! -e "$source/.beads/embeddeddolt" ] && [ ! -L "$source/.beads/embeddeddolt" ] ||
die 'source contains Dolt data; this helper accepts historical SQLite only'
if (set +o pipefail; find "$source/.beads" -type l -print | grep -q .); then
die 'source .beads contains a symlink; refusing to follow it'
fi
validate_sqlite_descriptor "$source/.beads/metadata.json" metadata.json
validate_sqlite_descriptor "$source/.beads/config.json" config.json
sqlite_database=""
sqlite_database_count=0
for candidate_database in "$source/.beads/"*.db; do
[ -f "$candidate_database" ] || continue
sqlite_database=$candidate_database
sqlite_database_count=$((sqlite_database_count + 1))
done
[ "$sqlite_database_count" -eq 1 ] ||
die 'source must contain exactly one historical SQLite *.db file'
cmp -s <(dd if="$sqlite_database" bs=16 count=1 2>/dev/null) \
<(printf 'SQLite format 3\0') ||
die "source database does not have a SQLite header: $sqlite_database"
sqlite_database_name=${sqlite_database##*/}
validate_sqlite_database_reference "$source/.beads/metadata.json" metadata.json "$sqlite_database_name"
validate_sqlite_database_reference "$source/.beads/config.json" config.json "$sqlite_database_name"
case "$destination/" in
"$source/"*) die "destination must resolve outside source: $destination" ;;
esac
[ ! -e "$destination" ] && [ ! -L "$destination" ] ||
die "destination already exists; refusing to overwrite: $destination"
source_before=$(fingerprint "$source/.beads")
mkdir -p "$destination/sealed-source" "$destination/cutover/.beads" \
"$destination/home/config" "$destination/home/cache" "$destination/probe"
cp -Rp "$source/.beads" "$destination/sealed-source/.beads"
sealed_fingerprint=$(fingerprint "$destination/sealed-source/.beads")
[ "$sealed_fingerprint" = "$source_before" ] ||
die 'sealed copy fingerprint does not match source'
[ "$(fingerprint "$source/.beads")" = "$source_before" ] ||
die 'source changed while its sealed copy was made; stop and retry'
for repo in "$destination/sealed-source" "$destination/cutover"; do
git -c core.hooksPath=.git/hooks -C "$repo" init --quiet
git -C "$repo" config core.hooksPath .git/hooks
git -C "$repo" config user.name legacy-migration
git -C "$repo" config user.email legacy-migration@invalid
done
run_at() {
local directory=$1
shift
(
cd "$directory"
env -i PATH="$PATH" HOME="$destination/home" \
XDG_CONFIG_HOME="$destination/home/config" \
XDG_CACHE_HOME="$destination/home/cache" \
GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null \
GIT_TERMINAL_PROMPT=0 BD_DISABLE_METRICS=1 \
BD_DISABLE_EVENT_FLUSH=1 BD_NON_INTERACTIVE=1 \
BEADS_NO_DAEMON=1 BEADS_DOLT_AUTO_START=0 \
BRIDGE_TEST_MARKER="${BRIDGE_TEST_MARKER:-}" "$@"
)
}
verify_binary_version "$source_version" "$old_bd"
if [ "$strategy" != direct ]; then
verify_binary_version v0.49.6 "$canonicalizer_bd"
fi
export_jsonl="$destination/export.jsonl"
case "$strategy" in
materialize-and-canonicalize)
run_at "$destination/sealed-source" "$old_bd" --no-auto-import \
export --format jsonl --output .beads/issues.jsonl >/dev/null
run_at "$destination/sealed-source" "$canonicalizer_bd" \
--no-daemon --no-auto-import init --force --prefix "$prefix" >/dev/null
run_at "$destination/sealed-source" "$canonicalizer_bd" \
--no-daemon --no-auto-import export --format jsonl > "$export_jsonl"
;;
canonicalize)
run_at "$destination/sealed-source" "$canonicalizer_bd" \
--no-daemon --no-auto-import init --force --prefix "$prefix" >/dev/null
run_at "$destination/sealed-source" "$canonicalizer_bd" \
--no-daemon --no-auto-import export --format jsonl > "$export_jsonl"
;;
direct)
run_at "$destination/probe" "$new_bd" \
migrate legacy-sqlite \
--source-db "$destination/sealed-source/.beads/$sqlite_database_name" \
--output "$export_jsonl"
;;
esac
[ -f "$export_jsonl" ] && [ ! -L "$export_jsonl" ] ||
die 'historical export was not written as a non-symlink regular file'
# Only the direct strategy uses the candidate's authenticated SQLite reader.
# Older exporter failures must not be allowed to masquerade as an empty source.
if [ "$strategy" != direct ] && [ ! -s "$export_jsonl" ]; then
die 'historical export was empty'
fi
jq -s -e '
all(.[];
type == "object" and
(if (._type // "") == "memory"
then (.key | type) == "string" and (.value | type) == "string"
else (.id | type) == "string" and (.id | length) > 0
end))
' "$export_jsonl" >/dev/null ||
die 'historical export is not a valid JSONL bridge payload'
[ "$(fingerprint "$source/.beads")" = "$source_before" ] ||
die 'historical source changed during canonical export'
cp -f "$export_jsonl" "$destination/cutover/.beads/issues.jsonl"
run_at "$destination/cutover" "$new_bd" init --from-jsonl --quiet \
--non-interactive --skip-hooks --skip-agents --prefix "$prefix"
verify_current_cutover "$destination/cutover/.beads"
candidate_export="$destination/candidate-export.jsonl"
run_at "$destination/cutover" "$new_bd" export --all -o "$candidate_export" >/dev/null
[ -f "$candidate_export" ] && [ ! -L "$candidate_export" ] ||
die 'candidate export was not written as a non-symlink regular file'
# A zero-record candidate is valid only for a zero-record expected export.
if [ -s "$export_jsonl" ] && [ ! -s "$candidate_export" ]; then
die 'candidate export was empty after import'
fi
normalize_export "$export_jsonl" > "$destination/expected-normalized.json"
normalize_export "$candidate_export" > "$destination/candidate-normalized.json"
if ! cmp -s "$destination/expected-normalized.json" "$destination/candidate-normalized.json"; then
diff -u "$destination/expected-normalized.json" \
"$destination/candidate-normalized.json" >&2 || true
die 'candidate data does not semantically match the canonical historical export'
fi
[ "$(fingerprint "$source/.beads")" = "$source_before" ] ||
die 'historical source changed during candidate verification'
printf 'cutover created at %s/cutover\n' "$destination"
printf 'sealed source retained at %s/sealed-source; review before activating manually.\n' "$destination"