517 lines
13 KiB
Markdown
517 lines
13 KiB
Markdown
# Dolt Storage Backend
|
|
|
|
Beads uses [Dolt](https://www.dolthub.com/) as its default storage backend. Dolt provides Git-like version control for your database, enabling advanced workflows like branch-based development, time travel queries, and distributed sync.
|
|
|
|
> **Note:** Dolt is the only supported backend. The legacy SQLite backend has been removed.
|
|
> To migrate from SQLite, see [Migration from SQLite](#migration-from-sqlite-legacy) below.
|
|
|
|
## Overview
|
|
|
|
| Feature | Dolt |
|
|
|---------|------|
|
|
| Storage | Directory-based |
|
|
| Version control | Native (cell-level) |
|
|
| Branching | Yes |
|
|
| Time travel | Yes |
|
|
| Merge conflicts | SQL-based (cell-level merge) |
|
|
| Multi-user concurrent | Server mode |
|
|
| Sync | Native push/pull to Dolt remotes |
|
|
|
|
## Quick Start
|
|
|
|
### 1. Install Dolt
|
|
|
|
```bash
|
|
# macOS
|
|
brew install dolt
|
|
|
|
# Linux
|
|
curl -L https://github.com/dolthub/dolt/releases/latest/download/install.sh | bash
|
|
|
|
# Verify installation
|
|
dolt version
|
|
```
|
|
|
|
### 2. Initialize
|
|
|
|
```bash
|
|
# New project (Dolt is the default backend)
|
|
bd init
|
|
|
|
# For legacy SQLite installations, see Migration from SQLite below
|
|
```
|
|
|
|
### 3. Configure Sync Mode
|
|
|
|
```yaml
|
|
# .beads/config.yaml
|
|
sync:
|
|
mode: dolt-native # Default: use Dolt remotes
|
|
```
|
|
|
|
## Embedded Mode (Default)
|
|
|
|
Embedded mode runs the Dolt engine in-process — no separate server needed. This is
|
|
the default for all `bd init` installations. Just `bd init` and go.
|
|
|
|
- Zero-config: no server, no ports, no PID files
|
|
- Single-writer (one process at a time, enforced via file lock)
|
|
- Data lives in `.beads/embeddeddolt/` alongside your code
|
|
|
|
## Server Mode (Opt-In)
|
|
|
|
Server mode connects to a running `dolt sql-server` for multi-client access. Use
|
|
server mode when you need concurrent writers (multiple agents, orchestrator setups).
|
|
|
|
### Enable Server Mode
|
|
|
|
```bash
|
|
# During init
|
|
bd init --server
|
|
|
|
# Or via environment variable
|
|
export BEADS_DOLT_SERVER_MODE=1
|
|
bd init
|
|
```
|
|
|
|
For an existing embedded project, see [Migrating Between Backends](#migrating-between-backends) below.
|
|
|
|
### Server Configuration
|
|
|
|
| Environment Variable | Default | Description |
|
|
|---------------------|---------|-------------|
|
|
| `BEADS_DOLT_SERVER_MODE` | (empty) | Set to `1` to enable server mode |
|
|
| `BEADS_DOLT_SERVER_HOST` | `127.0.0.1` | Server bind address |
|
|
| `BEADS_DOLT_SERVER_PORT` | `3307` | Server port (MySQL protocol) |
|
|
| `BEADS_DOLT_SERVER_USER` | `root` | MySQL username |
|
|
| `BEADS_DOLT_SERVER_PASS` | (empty) | MySQL password |
|
|
| `BEADS_DOLT_SHARED_SERVER` | (empty) | Shared server mode: `1` or `true` to enable |
|
|
|
|
### Server Lifecycle
|
|
|
|
These commands are only available in server mode:
|
|
|
|
```bash
|
|
# Check server status
|
|
bd doctor
|
|
|
|
# Start/stop/status
|
|
bd dolt start
|
|
bd dolt stop
|
|
bd dolt status
|
|
|
|
# Server-only config commands (error in embedded mode)
|
|
bd dolt show
|
|
bd dolt set
|
|
bd dolt test
|
|
```
|
|
|
|
### Shared Server Mode
|
|
|
|
On multi-project machines, enable shared server mode to use a single Dolt server
|
|
for all projects (instead of one server per project):
|
|
|
|
```bash
|
|
# Enable via config
|
|
bd dolt set shared-server true
|
|
|
|
# Or via environment variable (machine-wide)
|
|
export BEADS_DOLT_SHARED_SERVER=1
|
|
```
|
|
|
|
Shared server state lives in `~/.beads/shared-server/` and uses port 3308 by default
|
|
(avoiding conflict with the orchestrator on 3307). Each project's data remains isolated in its
|
|
own database (named by project prefix). See [DOLT.md](DOLT.md) for details.
|
|
|
|
## Central Dolt Server (macOS)
|
|
|
|
If you plan to use an orchestrator or manage multiple beads projects from a single
|
|
machine, you can run a central persistent Dolt server instead of per-project
|
|
embedded instances.
|
|
|
|
### Embedded vs Central Server
|
|
|
|
| | Embedded (default) | Central Server |
|
|
|---|---|---|
|
|
| **Setup** | Zero-config — `bd init` handles everything | One-time server setup required |
|
|
| **Data location** | `.beads/embeddeddolt/` per project | Central directory (e.g. `/opt/homebrew/var/dolt`) |
|
|
| **Concurrency** | Single writer per project | Multi-writer via MySQL protocol |
|
|
| **Use case** | Solo development, single agent | Orchestrator, multiple projects, multiple agents |
|
|
|
|
Embedded mode is the default and requires no setup. Switch to a central server
|
|
when you need an orchestrator or concurrent access from multiple agents.
|
|
|
|
### Why Not `brew services start dolt`?
|
|
|
|
After installing Dolt with `brew install dolt`, the natural next step is
|
|
`brew services start dolt`. However, this **silently ignores your config file**.
|
|
|
|
The Homebrew formula runs `dolt sql-server` without the `--config` flag. Dolt
|
|
does **not** auto-discover `config.yaml` from its working directory — the config
|
|
file must be passed explicitly via `--config <file>`. Any edits to
|
|
`/opt/homebrew/var/dolt/config.yaml` (port, host, etc.) have no effect when
|
|
started through `brew services`.
|
|
|
|
### Setup with a Custom LaunchAgent
|
|
|
|
Instead of `brew services`, create a custom LaunchAgent that passes the config
|
|
file explicitly.
|
|
|
|
**1. Install Dolt and initialize its data directory:**
|
|
|
|
```bash
|
|
brew install dolt
|
|
|
|
# Initialize the dolt data directory (if not already done)
|
|
cd /opt/homebrew/var/dolt && dolt init
|
|
```
|
|
|
|
**2. Configure Dolt for port 3307:**
|
|
|
|
```yaml
|
|
# /opt/homebrew/var/dolt/config.yaml
|
|
log_level: info
|
|
|
|
listener:
|
|
host: 127.0.0.1
|
|
port: 3307
|
|
max_connections: 100
|
|
|
|
behavior:
|
|
autocommit: true
|
|
```
|
|
|
|
**3. Create the LaunchAgent plist:**
|
|
|
|
```bash
|
|
cat > ~/Library/LaunchAgents/com.local.dolt-server.plist << 'EOF'
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
|
|
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
|
<plist version="1.0">
|
|
<dict>
|
|
<key>Label</key>
|
|
<string>com.local.dolt-server</string>
|
|
<key>ProgramArguments</key>
|
|
<array>
|
|
<string>/opt/homebrew/opt/dolt/bin/dolt</string>
|
|
<string>sql-server</string>
|
|
<string>--config</string>
|
|
<string>config.yaml</string>
|
|
</array>
|
|
<key>WorkingDirectory</key>
|
|
<string>/opt/homebrew/var/dolt</string>
|
|
<key>RunAtLoad</key>
|
|
<true/>
|
|
<key>KeepAlive</key>
|
|
<true/>
|
|
<key>StandardOutPath</key>
|
|
<string>/opt/homebrew/var/log/dolt.log</string>
|
|
<key>StandardErrorPath</key>
|
|
<string>/opt/homebrew/var/log/dolt.error.log</string>
|
|
</dict>
|
|
</plist>
|
|
EOF
|
|
```
|
|
|
|
**4. Load the service:**
|
|
|
|
```bash
|
|
launchctl load ~/Library/LaunchAgents/com.local.dolt-server.plist
|
|
|
|
# Verify it's running
|
|
mysql -h 127.0.0.1 -P 3307 -u root -e "SELECT 1"
|
|
```
|
|
|
|
**5. Point beads at the central server** — add to `~/.zshrc` (or `~/.bashrc`):
|
|
|
|
```bash
|
|
export BEADS_DOLT_SERVER_HOST="127.0.0.1"
|
|
export BEADS_DOLT_SERVER_PORT="3307"
|
|
export BEADS_DOLT_SERVER_MODE=1
|
|
```
|
|
|
|
Now `bd init` in any project will connect to the central server instead of
|
|
spawning an embedded instance.
|
|
|
|
### Managing the Service
|
|
|
|
```bash
|
|
# Stop
|
|
launchctl unload ~/Library/LaunchAgents/com.local.dolt-server.plist
|
|
|
|
# Restart (unload + load)
|
|
launchctl unload ~/Library/LaunchAgents/com.local.dolt-server.plist
|
|
launchctl load ~/Library/LaunchAgents/com.local.dolt-server.plist
|
|
|
|
# Check logs
|
|
tail -f /opt/homebrew/var/log/dolt.log
|
|
```
|
|
|
|
## Sync Modes
|
|
|
|
Dolt supports multiple sync strategies:
|
|
|
|
### Sync Mode
|
|
|
|
Beads uses `dolt-native` sync mode exclusively:
|
|
|
|
- Uses Dolt remotes (DoltHub, S3, GCS, etc.)
|
|
- Native database-level sync with cell-level merge
|
|
- Supports branching and merging
|
|
- `bd export` available for issue portability; `bd backup init` / `bd backup sync` / `bd backup restore` for Dolt-native backups
|
|
|
|
## Dolt Remotes
|
|
|
|
### Configure a Remote
|
|
|
|
Use `bd dolt remote add` to configure remotes. This ensures the running Dolt SQL
|
|
server sees the remote immediately. Remotes added via the `dolt` CLI directly
|
|
are written to the filesystem but are not visible to the server until restart.
|
|
|
|
```bash
|
|
# DoltHub (public or private)
|
|
bd dolt remote add origin https://doltremoteapi.dolthub.com/org/beads
|
|
|
|
# S3
|
|
bd dolt remote add origin aws://[bucket]/path/to/repo
|
|
|
|
# GCS
|
|
bd dolt remote add origin gs://[bucket]/path/to/repo
|
|
|
|
# Git SSH (GitHub, GitLab, etc.)
|
|
bd dolt remote add origin git+ssh://git@github.com/org/repo.git
|
|
|
|
# Local file system
|
|
bd dolt remote add origin file:///path/to/remote
|
|
```
|
|
|
|
### Push/Pull
|
|
|
|
```bash
|
|
bd dolt push
|
|
bd dolt pull
|
|
```
|
|
|
|
For SSH remotes, `bd dolt push` and `bd dolt pull` automatically use the `dolt`
|
|
CLI instead of the SQL server to avoid MySQL connection timeouts during transfer.
|
|
|
|
`bd dolt remote add` registers the remote on both the SQL server and the
|
|
filesystem (CLI) config. This ensures `dolt push`/`dolt pull` via CLI can find
|
|
the remote. If either surface already has a remote with that name, you'll be
|
|
prompted before overwriting.
|
|
|
|
> **Also supports sharing a Git repo**: Dolt stores data under `refs/dolt/data`,
|
|
> separate from standard Git refs (`refs/heads/`, `refs/tags/`). You can safely
|
|
> point a `git+ssh://` remote at the same repository as your project source code.
|
|
> See [Dolt Git Remotes](https://docs.dolthub.com/concepts/dolt/git/remotes) for details.
|
|
|
|
### List/Remove Remotes
|
|
|
|
```bash
|
|
bd dolt remote list # Shows remotes from both SQL server and CLI, flags discrepancies
|
|
bd dolt remote remove origin # Removes from both surfaces
|
|
```
|
|
|
|
Use `bd doctor --fix` to resolve any discrepancies between SQL and CLI remote configs.
|
|
|
|
## Migration from SQLite (Legacy)
|
|
|
|
If upgrading from an older version that used SQLite:
|
|
|
|
### Option 1: Migration Script
|
|
|
|
> **Note:** The `bd migrate --to-dolt` command was removed in v0.58.0.
|
|
> For pre-0.50 installations with JSONL data, use the migration script:
|
|
>
|
|
> ```bash
|
|
> scripts/migrate-jsonl-to-dolt.sh
|
|
> ```
|
|
>
|
|
> See [Troubleshooting](TROUBLESHOOTING.md#circuit-breaker-server-appears-down-failing-fast) if you encounter connection errors after migration.
|
|
|
|
### Option 2: Fresh Start
|
|
|
|
```bash
|
|
# Export current state
|
|
bd export -o backup.jsonl
|
|
|
|
# Archive existing beads
|
|
mv .beads .beads-sqlite-backup
|
|
|
|
# Initialize fresh from backup
|
|
bd init --from-jsonl
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Already Committed dolt/ to Git
|
|
|
|
If you committed `.beads/dolt/` before this fix:
|
|
|
|
1. Update gitignore: `bd doctor --fix`
|
|
2. Remove from git tracking: `git rm --cached -r .beads/dolt/`
|
|
3. Commit the removal: `git commit -m "fix: remove accidentally committed dolt data"`
|
|
4. To purge from history (optional): use [BFG Repo-Cleaner](https://rtyley.github.io/bfg-repo-cleaner/) or `git filter-repo`
|
|
|
|
### Server Won't Start
|
|
|
|
```bash
|
|
# Check if port is in use
|
|
lsof -i :3306
|
|
|
|
# Check server logs
|
|
cat .beads/dolt/sql-server.log
|
|
|
|
# Verify dolt installation
|
|
dolt version
|
|
|
|
# Try manual start
|
|
cd .beads/dolt && dolt sql-server --host 127.0.0.1 --port 3306
|
|
```
|
|
|
|
### Connection Issues
|
|
|
|
```bash
|
|
# Test connection
|
|
mysql -h 127.0.0.1 -P 3306 -u root beads
|
|
|
|
# Check server is running
|
|
bd doctor
|
|
|
|
# Force restart
|
|
kill $(cat .beads/dolt/sql-server.pid) 2>/dev/null
|
|
bd list # Triggers auto-start
|
|
```
|
|
|
|
### Performance Issues
|
|
|
|
1. Run `bd doctor` for diagnostics
|
|
2. If using server mode, check server logs for errors
|
|
3. Consider `dolt gc` for database maintenance:
|
|
```bash
|
|
cd .beads/dolt && dolt gc
|
|
```
|
|
|
|
## Advanced Usage
|
|
|
|
### Branching
|
|
|
|
```bash
|
|
cd .beads/dolt
|
|
|
|
# Create feature branch
|
|
dolt checkout -b feature/experiment
|
|
|
|
# Make changes via bd commands
|
|
bd create "experimental issue"
|
|
|
|
# Merge back
|
|
dolt checkout main
|
|
dolt merge feature/experiment
|
|
```
|
|
|
|
### Time Travel
|
|
|
|
```bash
|
|
cd .beads/dolt
|
|
|
|
# List commits
|
|
dolt log --oneline
|
|
|
|
# Query at specific commit
|
|
dolt sql -q "SELECT * FROM issues AS OF 'abc123'"
|
|
|
|
# Checkout historical state
|
|
dolt checkout abc123
|
|
```
|
|
|
|
### Diff and Blame
|
|
|
|
```bash
|
|
cd .beads/dolt
|
|
|
|
# See changes since last commit
|
|
dolt diff
|
|
|
|
# Diff between commits
|
|
dolt diff HEAD~5 HEAD -- issues
|
|
|
|
# Blame (who changed what)
|
|
dolt blame issues
|
|
```
|
|
|
|
## Configuration Reference
|
|
|
|
### Full Config Example
|
|
|
|
```yaml
|
|
# .beads/config.yaml
|
|
backend: dolt
|
|
|
|
sync:
|
|
mode: dolt-native
|
|
auto_dolt_commit: true # Auto-commit after sync (default: true)
|
|
auto_dolt_push: false # Auto-push after sync (default: false)
|
|
|
|
dolt:
|
|
server_mode: false # Use sql-server (default: false — embedded mode)
|
|
server_host: "127.0.0.1" # Only used when server_mode: true
|
|
server_port: 3307
|
|
server_user: "root"
|
|
server_pass: ""
|
|
|
|
# Lock settings
|
|
lock_retries: 30
|
|
lock_retry_delay: "100ms"
|
|
idle_timeout: "30s"
|
|
|
|
federation:
|
|
remote: "dolthub://myorg/beads"
|
|
sovereignty: "T3" # T1-T4
|
|
```
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `BEADS_DOLT_SERVER_MODE` | Server mode: `1` or `0` |
|
|
| `BEADS_DOLT_SERVER_HOST` | Server host |
|
|
| `BEADS_DOLT_SERVER_PORT` | Server port |
|
|
| `BEADS_DOLT_SERVER_USER` | Server user |
|
|
| `BEADS_DOLT_SERVER_PASS` | Server password |
|
|
|
|
## Migrating Between Backends
|
|
|
|
Use `bd backup` to migrate data between embedded and server mode. Both
|
|
directions preserve full Dolt commit history. See [DOLT.md](DOLT.md#migrating-between-backends)
|
|
for step-by-step instructions.
|
|
|
|
**Quick reference:**
|
|
|
|
```bash
|
|
# 1. Backup the source project
|
|
bd backup init /path/to/backup-dir
|
|
bd backup sync
|
|
|
|
# 2. Create target project and restore
|
|
mkdir target && cd target
|
|
bd init # or bd init --server
|
|
bd backup restore --force /path/to/backup-dir
|
|
|
|
# 3. Verify
|
|
bd list
|
|
```
|
|
|
|
**Key details:**
|
|
- Embedded data lives in `.beads/embeddeddolt/`, server data in `.beads/dolt/`
|
|
- `--force` is required when restoring into an initialized project (overwrites the database)
|
|
- The restore auto-registers the backup dir for future syncs and updates the project identity
|
|
- Embedded mode uses file locking (`flock`) — only one writer at a time
|
|
- Alternative: migrate via Dolt remotes (`bd dolt push` / `bd dolt pull`) if both projects share a remote
|
|
|
|
## See Also
|
|
|
|
- [DOLT.md](DOLT.md) - Dolt backend overview, federation, and mode details
|
|
- [Troubleshooting](TROUBLESHOOTING.md) - General troubleshooting
|
|
- [Dolt Documentation](https://docs.dolthub.com/) - Official Dolt docs
|