beads/engdocs/ICU-POLICY.md
Julian Knutsen bb9b9e5020
ci(bazel): run the upgrade smoke tests under Bazel; retire cross-version-smoke.yml (#7331)
cross-version-smoke.yml resolved the newest 5 GitHub releases (30 on a tag
push) with `gh release list`, then built the candidate and downloaded each
release on a GitHub runner for scripts/upgrade-smoke-test.sh.

//tests/upgrade_smoke has one target per release in the newest 30 the
release catalog pins a linux/amd64 asset for (@bd_releases//:versions.bzl):
upgrade_smoke_<release>_test runs the script against the non-race
bd_for_tests with that release pinned as PREV_BIN (sha256-verified when
Bazel fetches it; the script checks it reports the release and downloads
nothing). A release added to the catalog joins the window. Tagged
pr-core-only, so bazel-test (not the integration lane) runs them remotely
on every PR, merge group and
push, cached until the candidate, the script or the release changes.

The tested set is now reviewed rather than resolved at run time: stable
releases from the catalog, where the workflow took GitHub releases
including prereleases. cross-version-smoke.yml and the F7c checks of its
chunk fold, path filter, runners and binary cache are removed (the
path-filter and binary-cache checks had no other workflow left).

Refs: ga-96smfk.19

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 02:49:39 +00:00

9.1 KiB

ICU Regex Policy

The Rule

bd never ships with an ICU runtime dependency. All release binaries use Go's stdlib regexp via the gms_pure_go build tag.

This is non-negotiable. Do not remove, conditionally skip, or override gms_pure_go in any build target, release workflow, or install script.

Background

ICU (International Components for Unicode) is a C library that provides MySQL-compatible regex via go-icu-regex. It enters our dependency tree through go-mysql-server (the embedded Dolt SQL engine). bd does not use SQL REGEXP functions, so ICU provides zero functional value while creating significant portability problems:

Platform Problem without gms_pure_go
Linux Binaries dynamically link a specific libicui18n.so.NN version; crash on distros with a different ICU version
macOS ICU is keg-only in Homebrew; go install fails without manual CGO_CFLAGS/CGO_LDFLAGS
Windows ICU C headers (unicode/uregex.h) not available; go install and CGO builds fail
go install The module cannot make plain go install pkg@latest use -tags gms_pure_go automatically

How It Works

go-mysql-server
  ├── (default)      → go-icu-regex → links libicu (BAD)
  └── gms_pure_go    → Go stdlib regexp (GOOD)

The gms_pure_go build tag tells go-mysql-server to use Go's regexp package instead of go-icu-regex. This eliminates the ICU shared-library dependency at the binary level.

CGO stays enabled. CGO is required for the embedded Dolt database (file locking, SQL engine). CGO and ICU are independent concerns:

  • CGO_ENABLED=1 + gms_pure_go = Dolt works, no ICU (what we ship)
  • CGO_ENABLED=1 without gms_pure_go = Dolt works, ICU linked (test-only)
  • CGO_ENABLED=0 = no Dolt backend at all

Where gms_pure_go Must Be Set

Every build path that produces a binary for users must include -tags gms_pure_go:

Location File
Local builds Makefile (BUILD_TAGS := gms_pure_go)
Release builds .goreleaser.yml (all build targets)
Install script scripts/install.sh
Windows installer install.ps1
PR CI .github/workflows/pr.yml, .github/workflows/pr-risk.yml
Main CI test matrix .github/workflows/main.yml (Linux, macOS, Windows)
macOS release .github/workflows/release.yml
Migration tests //tests/migration/... (Bazel, bazel.yml bazel-test)
Nightly tests .github/workflows/nightly.yml
Cross-version smoke //tests/upgrade_smoke/... (Bazel, bazel.yml bazel-test)
Regression tests //tests/regression:regression_test (Bazel, bazel.yml bazel-cmd-dolt)

Canonical pattern: source .buildflags

The preferred way for a shell script to comply is to source .buildflags at the top. That sets CGO_ENABLED=1 and puts -tags=gms_pure_go into GOFLAGS, so every subsequent bare go invocation in the script picks it up automatically:

#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
# shellcheck source=../.buildflags
source "$REPO_ROOT/.buildflags"

go build -o bd ./cmd/bd     # -tags=gms_pure_go applied via GOFLAGS

Makefile targets use -tags "$(BUILD_TAGS)" directly, since make already defines BUILD_TAGS := gms_pure_go at the top of the file. Workflow YAML passes the tag inline (-tags gms_pure_go).

Source-time guard: scripts/check-build-tags.sh

CI runs scripts/check-build-tags.sh on every PR (see the Check build-tag policy job in .github/workflows/pr.yml). It fails if any tracked shell script, git hook, or the Makefile contains a go build|test|run|generate|install invocation that:

  • does not carry -tags=...gms_pure_go, AND
  • is not in a file that sources .buildflags, AND
  • does not reference a file-level variable (e.g. $(BUILD_TAGS)) whose value contains gms_pure_go, AND
  • is not a third-party tool install (go install X@version / go run X@version).

The required policy wrapper separately parses GitHub Actions YAML and checks the first recognizable direct first-party Go command on each logical line in jobs.*.steps[*].run. Each such command declares the literal gms_pure_go tag in its own -tags argument; sources and variables elsewhere in a workflow do not establish that source convention. This is deliberately not shell control-flow analysis: dynamic payloads, later compound-command segments, command substitutions, and sourced-script internals are outside the check's contract. The pinned-tool exception covers the repository's simple go install path@version and go run path@version source forms.

These source-time guards complement scripts/verify-cgo.sh, which checks the release binary at runtime.

To intentionally opt a file out (e.g. because it tests the ICU path), add # build-tags: allow-bare within the first five lines of the file. scripts/test-cgo.sh and scripts/test-icu-path.sh are exempt by name.

Where gms_pure_go Is Intentionally Omitted

scripts/test-icu-path.sh omits gms_pure_go as an explicit, opt-in local developer tool for exercising the ICU code path in go-mysql-server on demand. CI no longer does this: upstream confirmed (dolthub/go-mysql-server#3506) that -tags=gms_pure_go is the sanctioned escape hatch, so we test the configuration we ship.

The older name scripts/test-cgo.sh is retained only as a deprecated shim that warns and forwards to scripts/test-icu-path.sh.

Post-Build Verification

Release builds are verified to be ICU-free:

  • Linux: readelf -d and ldd check for libicu (must not appear)
  • macOS: otool -L check for libicu (must not appear)
  • Script: scripts/verify-cgo.sh runs these checks as a goreleaser post-hook

If ICU linkage is detected, the release build fails.

The Upstream Fork (historical)

Beads used to carry a replace github.com/dolthub/go-mysql-server => github.com/maphew/go-mysql-server ... directive in go.mod, added in PR #3112 to try to make go install work on Windows without ICU headers. It was removed in PR #3306 (see GH#3303) after empirical testing confirmed that replace directives are not honored by go install pkg@version — the mechanism never worked for its stated purpose, and having the directive actively broke go install on every platform with a confusing error.

Upstream PR (closed, declined): https://github.com/dolthub/go-mysql-server/pull/3504 Upstream issue (closed, declined): https://github.com/dolthub/go-mysql-server/issues/3506

The dolthub maintainers have made clear the upstream default will not flip: "We want our software to work as intended with the default settings. If users want to circumvent certain features with build tags or other build-time or run-time configuration, that's fine. Changing the default is not aligned with what we are actually trying to do."

How go install is handled now

Two supported modes, documented in INSTALLING.md:

  1. CGO_ENABLED=0 go install github.com/steveyegge/beads/cmd/bd@latest produces a server-mode-only binary. Works on any Go-capable box with no C compiler. Users must run an external dolt sql-server and use bd init --server.

  2. CGO_ENABLED=1 GOFLAGS=-tags=gms_pure_go go install ... produces an embedded-capable binary. Requires a C compiler but NOT libicu.

No fork, no replace directive, no upstream patch required. The tradeoff is that go install users who want embedded mode have to pass an explicit GOFLAGS; those who don't care can use the shorter nocgo form.

Common Mistakes to Avoid

  1. Adding ICU flags to .buildflags or Makefile -- these were removed in PR #3066. The gms_pure_go tag makes them unnecessary.

  2. Removing gms_pure_go from a build target -- this re-introduces ICU linkage. The post-build checks will catch it, but don't do it.

  3. Installing libicu-dev in release or CI test workflows -- only needed for local, on-demand developer testing via scripts/test-icu-path.sh. Neither release builds nor the CI test matrix link ICU; both must not depend on ICU being installed.

  4. Confusing CGO with ICU -- CGO is required for embedded Dolt mode (NBS chunk compression via gozstd). ICU is independent. CGO_ENABLED=1 does not imply ICU linkage as long as -tags gms_pure_go is present. beads also supports CGO_ENABLED=0 builds via nocgo stubs: the binary runs in server-mode only (no embedded Dolt backend), which is the blessed go install path for users without a C toolchain.

Trade-offs

  • Go's regexp uses RE2 syntax, which is slightly less MySQL-compatible than ICU regex (no backreferences, no lookahead/lookbehind)
  • bd does not use SQL REGEXP functions, so this has zero practical impact
  • If a future feature needs SQL REGEXP, revisit this policy then

See Also