Commit graph

19 commits

Author SHA1 Message Date
CI Bot
6560ca064f ci(proxied): name the serve suites' shard instead of leaving it to the hash
The audit read TestProxiedServerServeReadParity as unenforced on PRs because it
does not appear in pr.yml. It is enforced, in the other PR-time workflow:
pr-risk.yml's test-proxied-cmd job runs `on: pull_request`, discovers every
`^func TestProxiedServer` in cmd/bd/*_test.go, and feeds TEST_PROXIED_CMD into
that workflow's "CI Gate / Required". Its concurrency group is keyed on the PR
number, so it is not the perpetually-cancelled Main workflow. Verified by running
the lane's own discovery across all fifteen shards: the oracle appears in exactly
one.

What was true is that it appeared there by HASH FALLBACK. The manifest header
asks for a regenerate after adding a TestProxiedServer* function, and neither
`bd serve` suite was ever added, so the assignment was incidental rather than
reviewed and nothing in the file said the parity oracle was in the lane at all.

Both suites are listed at the shard the hash was already choosing (8), which is
zero behaviour change and avoids reshuffling the other 100 entries — a full
regenerate rewrites 202 lines across tests this branch never touched. Verified:
shard 8 now reports `manifest: 8, fallback: 0`.

Two further tests remain unlisted (TestProxiedServerCleanDatabasesPurgeDropped,
TestProxiedServerReassignFence). They are not from this branch and still
distribute by hash, which is the supported path; a cost-rebalancing regenerate
is the change that should pick them up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 20:04:43 +00:00
CI Bot
87c64ec8f1 feat(serve): bd serve skeleton — lifecycle, mode gate, /healthz, /v0/beads/context (bd-60z)
The first slice that runs a server. Proxied-server workspaces go live; the
read and claim operations are registered as transitional 501 stubs so route
and spec parity can be proved all at once instead of growing a hole per slice.

`bd serve --addr 127.0.0.1:0 [--allow-non-loopback]`, and no other flags in
v0. The bound address is stdout's first line, because under the ephemeral
default it is the only way to find the port. Shutdown rides the signal context
the root command already installs. No lock file, pid file or discovery file:
serve is operator-invoked and the TCP bind IS the mutual exclusion — dbproxy
needs that machinery only because clients race to auto-spawn it.

The mode gate says two different things, deliberately. Embedded is a PERMANENT
typed storage.ErrUnsupported: there is no unit-of-work provider for that
backend. Dolt server mode is STAGED — the provider construction exists and is
tested, it just is not wired in yet — so the text names what works today and
says this does not, rather than reading as an architectural limit that would
send an operator off to change their topology for nothing.

One unit of work per request, closed through a DETACHED context. The
transaction layer poisons the pinned connection when a ROLLBACK send fails
(doltserver_tx.go), so closing with the request's own canceled context would
burn a session on every client disconnect.

Front-tier hardening lands here, not later, because without it a hung Dolt
becomes a silently growing black hole: slots fill with immortal queries,
/healthz stays green because it never touches the database, and recovery
produces a thundering herd. So: a Host-header allowlist (DNS-rebinding defense
for an unauthenticated, browser-reachable loopback service), a
netutil.LimitListener connection cap (Go spawns a goroutine per connection, so
the database semaphore alone does not bound what is accepted), a database
semaphore with a BOUNDED 10s wait mapping to the already-documented 503 busy,
a 60s per-request deadline, and explicit pool limits on the provider's *sql.DB
via a new uow.PoolTuner knob — openDB set none, and database/sql's default is
unlimited.

/v0/beads/context serializes an explicit ten-field allowlist and nothing else.
SyncRemote is hard-excluded in this and every future version: it is populated
unconditionally from config sync.remote, and remote URLs routinely embed
credentials. The database bind endpoint and the remaining absolute host paths
are excluded too. `capabilities` is derived from IMPLEMENTED handlers only, so
a release cut between this slice and the endpoint slices advertises only what
actually works.

The observability floor is an acceptance criterion, not a convention: one
structured stderr line per request (method, path, status, problem code,
duration, semaphore wait, unit-of-work acquire), startup and shutdown lines, a
semaphore-saturation event, and a request_id on every problem response. That
last one required documenting request_id on the spec's Problem schema — the
only spec edit here, additive, with types regenerated.

Decisions implemented as given: hooks do not fire (a user-controlled
subprocess per mutation is an unbounded latency multiplier and an orphaned
child at shutdown); the per-command auto-commit globals are never touched
(nothing to exclude — proxied PersistentPostRunE only closes the provider);
and actor on an HTTP request stays caller-asserted provenance, which the
context and capabilities wording does not contradict.

Tests: TestSpecRouteParity asserts set equality between the route table and
the document's paths, and builds the real mux so a pattern ServeMux would
refuse fails here rather than on the first request.
TestSpecStatusCodesMatchHandlerTable carries a commented, enumerated 501
exemption that is self-deleting — it fails if the route table and the list
disagree, and fails again if the list outlives the last stub. The context
allowlist is enforced from the handler out, over a body built from a snapshot
with every field populated and a credential-bearing sync remote.
TestWithUOWClosesWithADetachedContext records the close context's state AT
close time, which is the only moment it says anything.

Verified: build, vet and golangci-lint clean (the two gosec findings in
cmd/bd/main.go pre-date this change); internal/httpapi green under -race;
protocol corpus byte-identical with BEADS_PROTOCOL_REQUIRE_DOLT=1; and a real
server started against a proxied workspace, answered both endpoints, refused
a foreign Host, an unknown query key and a second bind, and drained to exit 0
on SIGTERM.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 10:07:14 +00:00
matt wilkie
65084057f2
ci: shard Embedded Dolt Storage into 5 manifest-driven shards (#5117)
Test (Embedded Dolt Storage) ran the ~84-function internal/storage/embeddeddolt
suite (minus TestConformance) as a single unsharded job, ~24.3min and the
longest pole in every Main/PR-Risk run per engdocs/CI_CLEANUP_PLAN.md and
mybd-t8l8; it timeout-red'd upstream main at its 30m suite budget twice on
2026-07-28 (mybd-jobt).

Extends the existing test-embedded-cmd manifest-driven scheme
(embedded-test-shard.sh + embedded-cmd-test-shards.txt, already 20-way) to the
storage job: new embedded-storage-test-shard.sh discovers top-level Test*
functions in internal/storage/embeddeddolt/*_test.go (excluding
TestConformance, which stays in its own job), consults a committed manifest
(embedded-storage-test-shards.txt) for balance, and hash-falls-back for
newly-added tests. No per-test timing data exists for this suite, so the
manifest assigns the 83 discovered tests round-robin over the alphabetically
sorted list for 5 shards (17/17/17/16/16), which spreads related families
(the 7 TestEmbeddedMergeAndSettle*, ~11 TestEmbeddedMigrat(e|ion)*, 4
TestSearchIssues_MaxRows_*) across shards instead of clustering them.

test-embedded-storage becomes a 5-way matrix job in both main.yml and
pr-risk.yml, per-shard timeout-minutes: 15 and -test.timeout=15m (half the
old 30m budget, matching that each shard carries roughly 1/5 of the tests and
~2-3 of the ~11 migration-chain tests that motivated the original 20m/30m
budget per bd-lrgn1). Job id test-embedded-storage and its BEADS_TEST_EMBEDDED_DOLT
env stay the same, so pr-risk.yml's ci-gate needs/CI_GATE_REQUIRED wiring
(TEST_EMBEDDED_STORAGE) is unaffected by the matrix split; only the job's
display name changes shape, from "Test (Embedded Dolt Storage)" to
"Test (Embedded Dolt Storage N/5)". Updated the two required-check-name
references to that in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md and
engdocs/CI_TEST_SURFACE_AUDIT.md.

Conformance (core/audit, ~14min each, mybd-ae0n) is left alone: TestConformance
is sharded by sub-test path via -test.run '^TestConformance$/^Audit$', a
different mechanism from this manifest's exact top-level function-name
matching, so it does not extend naturally in this commit.

Agent-Signature: claude-fable-5-high on behalf of maphew
2026-07-28 22:16:48 -07:00
coffeegoddd☕️✨
2425194fa3 ci(proxied): regenerate shard manifest for new bd mol test functions
Adds the 14 new TestProxiedServer{MolShow,MolCurrent,MolProgress,
MolLastActivity,MolStale,MolReadyGated,MolDistill,Pour,MolBond,
MolSquash,MolBurn,WispCreate,WispList,WispGC} entry points to the
committed 15-shard manifest via scripts/ci/gen_proxied_shard_manifest.py,
so they run with pinned, cost-balanced shard assignments in the
test-proxied-cmd CI job (main.yml and pr-risk.yml) instead of falling
back to hash-based distribution. Verified via
`proxied-test-shard.sh <N> 15` with BEADS_TEST_SHARD_LIST_ONLY=1 across
all 15 shards: every test resolves via the manifest with zero fallback
and no validation errors.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-27 13:06:47 -07:00
coffeegoddd☕️✨
6389546c56 test(proxied): split the six heaviest suites so shards balance under the timeout
The proxied cmd lane is init-bound: each subtest's bd init runs the full
migration chain, so a shard's wall-time tracks its total init count. The six
heaviest suites (Update 56 inits, Show 47, Create/Close 41, Dep 36, Ready 34)
were each a single top-level function, and a function is the atomic unit of
sharding — so no shard split could pull the worst shard below Update's ~169s
(local), which is what kept brushing the 15m budget on CI.

Split each of the six into 2-3 top-level functions (Update/Show ->3, the rest
->2), balanced by init count, by inserting function boundaries between the
already-independent per-subtest blocks — no test logic moves, every t.Run set
is byte-identical to before. Regenerate the 15-shard manifest by greedily
bin-packing every TestProxiedServer* function by init count
(scripts/ci/gen_proxied_shard_manifest.py); the heaviest shard drops from ~56
to ~31 estimated inits, roughly halving the long pole.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 15:55:17 -07:00
coffeegoddd☕️✨
51bd6cf4ed ci(proxied): shard proxied-server cmd tests across 15 shards (was 8)
The proxied suites are init-bound (each per-subtest project runs the full
migration chain), so the heaviest shards were brushing the 15m budget and
occasionally timing out. Widen the matrix from 8 to 15 shards in both the
main and pr-risk workflows and rebalance the committed manifest, greedily
bin-packing by estimated cost so the six heaviest suites (Update, Show,
Create, Close, Dep, Ready) each own a shard and the rest pack evenly.
Estimated worst shard ~600s, comfortably under the 900s timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 15:55:17 -07:00
coffeegoddd☕️✨
7caa8cd554 test(proxied): cut shard-4 timeout — revert per-subtest project splits, move Ready to shard 8
The per-subtest project split multiplied bd init calls (each runs the full
migration chain on a fresh DB): epic 2->8, set-state 1->4, unclaim 2->9,
defer 1->7. That is added, not saved, wall-clock on already-saturated
shards and pushed shard 4/8 over the 15m budget. Revert those five suites
to one shared project per test function (the read-suite pattern), and move
the pre-existing heaviest suite (Ready: 36 subtests / 34 inits) off the
overloaded shard 4 onto the near-idle shard 8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 15:55:17 -07:00
coffeegoddd☕️✨
ea7b95ff49 ci 2026-07-20 14:41:07 -07:00
coffeegoddd☕️✨
e20691832c /.github/{scripts,workflows}: shard it 2026-07-15 18:19:36 -07:00
Chris Sells
789044b83e chore: purge deleted docs-pipeline references from release tooling, CI, and repo plumbing
- beads-release formula: drop the snapshot-docs step (ran the deleted
  scripts/snapshot-release-docs.sh on the critical needs-chain); stamp-
  changelog now needs update-vendorhash
- update-versions.sh: remove the mangled --skip-docs/docs-snapshot logic
  and garbled usage text left by mid-line deletions
- pre-push hook: drop the snapshot-script remediation hint
- release.yml: collapse the prerelease branch; BEADS_REQUIRE_RELEASE_DOCS
  has no consumer since check-docs-version.sh was deleted
- RELEASING.md / scripts/docs.md: describe the current pipeline (no
  --versioned, no website/ outputs, no check-docs-version.sh)
- docs-mintlify.yml: paths filter now includes engdocs/** and the curated
  root markdown files the docsync guard validates
- docs-render-check.sh: pin mint@4.2.687, fail closed when mint dies
  without producing a broken-links report, parse report lines only, drop
  dead extract_page_links(), fix the stale jq requirement; mint.sh pins
  the same version
- nightly.yml: remove the deleted website measurement suite option
- CODEOWNERS: init-safety ADR gate follows the file to engdocs/adr/
- beads-docs skill: align with settled decision 6 — no pointer stubs,
  moved pages get docs.json redirects; bd's printed paths are fixed at
  the source
- CHANGELOG (edited line), FEDERATION-SETUP, README, examples, oracle-a,
  .buildflags, gh-issue-to-pr formula: retarget moved-file references;
  anchor the build/ gitignore pattern
2026-07-12 11:07:26 -07:00
Chris Sells
3198395a32 docs(ci): add Mintlify preview, docsync guard, and broken-links workflow
- mint.sh + make docs-dev: local preview of the docs/ Mintlify site
  (Node 22 handling modeled on the Gas City wrapper).
- test/docsync: Go guard keeping docs/ and docs.json in exact
  correspondence — every nav entry has a file, every page is in the nav
  (tiny stub allowlist for paths released bd binaries print), published
  pages use root-relative extensionless links that resolve, and engdocs/
  plus curated root markdown keep exact GitHub-style paths. Wired into
  make check-docs.
- .github/workflows/docs-mintlify.yml: docsync + baseline-aware
  mint broken-links check on PRs touching docs/**.
- Retire scripts/sync-website-docs.sh and its mirror-drift gate: the six
  mirrored files now live once in the Mintlify site; the committed
  Docusaurus mirrors stay frozen so the parallel-run site keeps building.
- Repoint inbound links to moved docs across README, guides, examples,
  npm-package, plugin resources, and Go comments; add docs/SYNC_CONCEPTS.md
  pointer stub (bd's agent templates print that path). Runtime strings in
  cmd/bd (prime.go, store_factory_nocgo.go, init_git_hooks.go) still print
  old paths — covered by stubs, flagged in the PR.
2026-07-12 10:37:07 -07:00
Julian Knutsen
9e8d411f71
Add PR CI wrapper command surface (#4211)
* ci: add PR wrapper command surface

* ci: collect wrapper timing in PR checks

* docs: record initial CI wrapper measurements

* ci: add manual measurement workflow

* ci: allow branch measurement dispatch

* ci: keep measuring independent commands after failures

* docs: record branch-dispatched CI measurement

* ci: keep measuring package suites after failures

* docs: record CI measurement batch

* fix: repair CI-measured test failures

* test: make local Go gates hermetic

* ci: add sharded integration measurement lane

* test: fix internal beads integration TestMain

* docs: record fixed integration shard measurement

* ci: add hybrid integration sharding measurement

* ci: speed up cmd/bd test shard discovery

* ci: add 16-way integration sharding measurement

* ci: measure prebuilt cmd bd sharding

* docs: record prebuilt sharding measurement

* docs: design CI build artifact stage

* docs: record repeat prebuilt integration measurement

* ci: add build artifact stage

* ci: reuse build artifact in linux test lane

* ci: promote main linux integration shards

* ci: gate legacy platform tests to main

* ci: satisfy workflow shell lint

* ci: ignore archived migrations in duplicate check

* ci: add package gate wrappers

* test: clean up testing short boundaries

* ci: allow unreleased version docs lag

* ci: gate releases on package checks

* ci: split workflow by tier

* ci: add aggregate PR gates
2026-05-29 18:52:53 -07:00
matt wilkie
cc31c6046e
Rebalance embedded cmd CI shards (bd-8zg) (#3499) 2026-04-25 17:58:51 -07:00
matt wilkie
04ec3f52b9
Tier embedded Dolt CI for PRs (bd-0gz) (#3459) 2026-04-24 07:53:23 -07:00
matt wilkie
2ade22e84c
chore(build): codify the gms_pure_go build-tag policy (GH#3303)
Converges the sprinkle of -tags=gms_pure_go fixes into a single source plus a source-time CI guard. .buildflags is now canonical (exports GOFLAGS, CGO_ENABLED), scripts source it, scripts/check-build-tags.sh runs in CI to catch bare go commands. Fixes 5 latent bugs the guard caught (regression.yml, Makefile bench/test-regression, devcontainer, embedded-test-shard fallback, smoke-test scripts).
2026-04-16 12:28:11 -07:00
coffeegoddd☕️✨
a35edfd28e /{internal,cmd,.goreleaser,.github}: embedded mode by default 2026-03-28 13:47:20 -07:00
coffeegoddd☕️✨
7e41ced834 /.github/{scripts,workflows}: precompile test binary 2026-03-25 16:29:39 -07:00
coffeegoddd☕️✨
4789cf17f8 /.github: fix workflow 2026-03-25 13:14:57 -07:00
coffeegoddd☕️✨
a2f5639b14 /{.github,cmd,internal}: sync and data integration tests 2026-03-24 16:49:44 -07:00