beads/scripts/ci/proxied_test_durations.json
Julian Knutsen 683993660f
fix(proxy): fail fast with a clear MySQL error when an external upstream is unreachable (#7181)
* fix(proxy): fail fast with a MySQL error when the external upstream is unreachable

A proxied-server command against a stopped external Dolt upstream stalled
~20-30s before failing with a bare "invalid connection". The local db proxy
stays up, so on a refused backend dial (or a front such as stunnel/socat
that accepts and closes because its own upstream is gone) it simply closed
the client connection. The MySQL driver reports that as invalid
connection / unexpected EOF, which the uow bootstrap ping (#6003) classifies
as a transient drop and retries with exponential backoff for its whole 30s
budget. Retrying cannot help: the proxy already proved the upstream at its
own startup, and the outage is the upstream's.

The proxy now answers such a connection itself with a MySQL ERR packet
(2003, HY000) in place of the server greeting, naming the upstream and the
cause, exactly as a real server refuses a connection before the handshake.
The driver surfaces it as a *mysql.MySQLError, which the bootstrap retry
already treats as permanent, so the command fails in ~0.1s with an
actionable message. Only shapes that prove the upstream is not serving
qualify: ECONNREFUSED/ENOENT/EHOSTUNREACH/ENETUNREACH on dial, or a backend
that reaches EOF having sent zero bytes (a MySQL server always speaks
first). A dial timeout or a drop after the greeting keeps the plain close
and the client's transient retry.

External backends only: a managed (local sidecar) backend is owned by the
proxy and its dial can be refused transiently while bd drives it (the
full proxied lane caught TestProxiedServerInitExplicitBeadsDirIsAuthoritative
hitting exactly that mid-init when this applied to every backend), so
managed backends keep the bare close and the client's bootstrap retry.
TestExternalProxyOutageAndRestoreReDialsTCPAndUnix asserted the old
silent close during an outage; it now asserts the ERR packet.

With the latency fixed, TestProxiedOutageReconnectAcceptanceMatrix is
renamed into the proxied discovery pattern as
TestProxiedServerOutageReconnectAcceptanceMatrix and placed in the Bazel
30-shard block by gen_proxied_shard_manifest.py's incremental --write
(shard 8). The frozen legacy 15-shard block is untouched; the test
hash-distributes there like every other test added since it was frozen.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(proxy,uow): retry the proxy's upstream-outage report for ~1s; hedge the zero-byte case

Review follow-up to the fail-fast change.

A zero-byte close from the upstream is not proof it is down: Dolt closes
queued connections without a byte when max_connections/back_log are full,
when a queued connection times out, and while it shuts down for a
restart; an L4 front does the same while its target is replaced. Its
report now says "closed the connection before the MySQL greeting (down,
restarting, or at its connection limit)" instead of "unreachable", and
the comment no longer claims proof.

Refusals and endpoint churn also lost #6003's retry entirely, so an
external Dolt restart or a Service with no ready endpoints failed a
command at once. The client now retries the proxy's report
(proxy.IsUpstreamOutageError: 2003 with the "beads db proxy: " prefix)
for upstreamOutageRetryWindow (1s) from the first time it sees it, with
the backoff clamped so the last attempt lands at the window's end; other
transient errors keep the 30s budget. A real outage still fails in about
a second, inside the acceptance test's 3s bound.

Also: the dial report drops Go call-site wrapping ("dial tcp H:P:
connection refused"), truncation keeps UTF-8 intact, the comment states
how the packet differs from a real server's pre-handshake ERR (keeps the
'#'+SQLSTATE marker for vitess; 2003 is a client code chosen on purpose),
a Stats counter for reported outages, and the isTransientPingError
paragraph is reflowed and corrected.

Tests: pingWithRetry retries the report inside the window and gives up
at its end without using the full budget; openDB against a front that
plays the proxy (its real packet) and comes back 300ms after the first
connection succeeds in one call, and fails in about the window when it
never does; NewProxyServer enables the report for external backends
only; the dial message carries no Go type names; rune-safe truncation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(proxied): fail, not skip, socat-bridged subtests in the Bazel lane

The external-unix front-door matrix subtest and both outage/reconnect
matrix subtests bridge an external endpoint with socat and skipped when
it was missing, so they could stop running without anyone noticing (the
parent test still passes). requireSocat now fails them when
BEADS_TEST_REQUIRE_SOCAT=1, which //cmd/bd:bd_proxied_test sets (its
remote workers have socat; a recent main run's external-unix subtest
passed there). The legacy GitHub 15-shard jobs have no socat and leave
it unset, so the subtests keep skipping there. The check now runs before
the upstream Dolt server is started, so a skip no longer pays for one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(proxied): measured weight for the outage matrix; pin it to legacy shard 1

Give TestProxiedServerOutageReconnectAcceptanceMatrix a duration entry
(80s: local runs measured 55-79s alone, 60-70s in its shard, and review
68-109s under load) and re-place it with the generator's incremental
--write: it moves from 30/8 to 30/11, one line, nothing else moved. The
heaviest estimated shard is now 435s against a ~357s median; a repack
would even that out but moves most assignments, so it is left to a
separate CI-only change that refreshes all durations together.

Legacy 15-shard block: the test would hash onto shard 14, already ~13.5
of the 15 min timeout on main. It is pinned to shard 1 (~6.5 min in
recent main runs) under the block's reviewed-pin rule; it skips there
today (no socat), and the pin keeps shard 14 safe if that changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 02:37:58 -07:00

241 lines
12 KiB
JSON

{
"_comment": [
"Measured locally with `bazel --nohome_rc test --config=doltserver-proxied",
"//cmd/bd:bd_proxied_test` -- the exact config bazel.yml's proxied-server tier",
"uses, including `-test.parallel=4` (local sandboxed execution only; no remote",
"executor or cache flags were configured for this measurement run; Bazel's own",
"--remote_download_regex etc. in that config do not enable remote execution",
"without --config=remote-exec, which was not passed).",
"",
"This replaces an earlier capture that used `-test.parallel=1` (serial). That",
"methodology avoided double-counting a parallel subtest group's own duration on",
"top of its children's, which was a genuine bug in an even earlier pass -- but",
"serial timings do not model this lane, which always runs at parallel=4, and an",
"independent re-review measured near-zero correlation (r=0.08) between the",
"serial-derived predicted shard load and actual parallel=4 wall time. These",
"values are each a clean, single-job (no other concurrent heavy bazel/dolt job",
"intentionally started on the box; see below), parallel=4 top-level test's own",
"<testsuite time=...> attribute from a single fresh, manifest-aligned 30-shard",
"sweep where every shard ran its exact assigned test set once (no duplicates,",
"no gaps -- see scripts/ci/gen_proxied_shard_manifest.py's exact-once coverage",
"check). A top-level Go test's <testsuite time> already reflects, not adds to,",
"its own parallel children's wall-clock time, so no leaf-summation step was",
"needed this time: every one of the 176 discovered tests completed inside a",
"single shard run with errors=\"0\" (none needed Bazel's per-target default",
"timeout worked around via a separate `bazel run`, unlike the prior capture).",
"",
"`seconds_per_init_fallback` is the weighted average seconds-per-init over",
"every measured test in this file (sum of all durations / sum of all inits),",
"used only for a brand-new test with no entry yet (see",
"scripts/ci/gen_proxied_shard_manifest.py's discover_inits_cost()). It dropped",
"from the serial capture's 39.7275 to this capture's lower value because",
"parallel=4 genuinely shortens most tests' wall time relative to inits count;",
"this is expected, not a data error.",
"",
"This file still does not claim shard totals land within a few seconds of each",
"other -- see .github/workflows/bazel.yml for the current, honest expected",
"spread for the 30-shard proxied lane, now also derived from this same sweep.",
"",
"The box this was measured on also runs other engineers'/agents' unrelated",
"Bazel jobs; this capture was not run with exclusive use of the machine, only",
"with no other *intentional, task-related* heavy bazel/dolt job started",
"concurrently by this task. Absolute seconds may be inflated by ambient",
"contention; relative ordering between tests, which is what LPT packing",
"actually depends on, should still be far more representative than the",
"discarded serial data (confirmed: 30/30 shards passed, 0 failures, 0 races,",
"with the two blocker fixes -- shortSocketPath and cmd.Cancel -- applied).",
"",
"Regenerate with a local `bazel --nohome_rc test --config=doltserver-proxied",
"//cmd/bd:bd_proxied_test --build_event_json_file=<path>` sweep, then pull",
"each shard's bazel-testlogs test.xml <testsuite time> per top-level test --",
"not from downloaded CI artifacts -- so the parallel=4, CI-representative",
"methodology above is applied consistently.",
"",
"Added after that sweep: TestProxiedServerOutageReconnectAcceptanceMatrix (80.0,",
"renamed into the discovery pattern once its outage bound was fixed). Not a",
"single sweep's value: local --config=doltserver-proxied runs measured it at",
"55-79 s alone, 60-70 s inside its shard, and a reviewer's 68-109 s under load;",
"80 s is a rounded middle of that spread. seconds_per_init_fallback was",
"recomputed to include it."
],
"seconds_per_init_fallback": 17.2952,
"tests": {
"TestProxiedServerAssign": 38.67,
"TestProxiedServerAutoPrune": 67.5,
"TestProxiedServerBackupRefusedOnExternalTopology": 5.96,
"TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce": 9.18,
"TestProxiedServerBatchParity": 60.04,
"TestProxiedServerBlocked": 138.63,
"TestProxiedServerCascadeParity": 78.23,
"TestProxiedServerChildren": 11.28,
"TestProxiedServerCleanDatabases": 7.14,
"TestProxiedServerCleanDatabasesPurgeDropped": 7.76,
"TestProxiedServerClose": 174.6,
"TestProxiedServerClose2": 218.83,
"TestProxiedServerCloseConcurrent": 38.25,
"TestProxiedServerCloseSiblingBlockersFanIn": 99.71,
"TestProxiedServerComment": 64.04,
"TestProxiedServerCompactDolt": 11.55,
"TestProxiedServerCompactHistory": 12.38,
"TestProxiedServerConfig": 188.91,
"TestProxiedServerConfigConcurrent": 48.59,
"TestProxiedServerConfigSetMany": 196.57,
"TestProxiedServerContext": 72.27,
"TestProxiedServerCount": 60.65,
"TestProxiedServerCountIncludeInfra": 34.69,
"TestProxiedServerCreate": 46.42,
"TestProxiedServerCreate2": 165.07,
"TestProxiedServerCreateInfraTypeRoutesToWisps": 142.17,
"TestProxiedServerDeferAutoWake": 12.95,
"TestProxiedServerDeferUndefer": 51.14,
"TestProxiedServerDelete": 44.82,
"TestProxiedServerDeleteBlockedJSONSingleDoc": 8.09,
"TestProxiedServerDeleteConcurrent": 10.57,
"TestProxiedServerDeleteForceOrphans": 9.16,
"TestProxiedServerDeleteWisp": 7.06,
"TestProxiedServerDep": 121.62,
"TestProxiedServerDep2": 227.28,
"TestProxiedServerDepConcurrent": 30.55,
"TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge": 10.13,
"TestProxiedServerDoltLifecycle": 6.18,
"TestProxiedServerDoltRemoteRemove": 72.67,
"TestProxiedServerDuplicates": 44.77,
"TestProxiedServerEdit": 10.16,
"TestProxiedServerEpic": 109.49,
"TestProxiedServerEventsJournal": 6.48,
"TestProxiedServerEventsJournalOffByDefault": 65.83,
"TestProxiedServerExport": 26.44,
"TestProxiedServerExternalCreate": 5.73,
"TestProxiedServerExternalMutationPolicy": 25.0,
"TestProxiedServerExternalPolicyDirectParity": 18.12,
"TestProxiedServerFindDuplicates": 51.66,
"TestProxiedServerGC": 12.73,
"TestProxiedServerGateCheck": 140.99,
"TestProxiedServerGateCreateInheritsRepoMetadata": 47.1,
"TestProxiedServerGateFrontDoorDirectServerProxyParity": 17.36,
"TestProxiedServerGateLifecycle": 49.79,
"TestProxiedServerGateList": 41.19,
"TestProxiedServerGraph": 15.43,
"TestProxiedServerHeartbeat": 57.23,
"TestProxiedServerHistory": 21.81,
"TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix": 59.61,
"TestProxiedServerHistoryRemoteSupportedFrontDoorParity": 5.97,
"TestProxiedServerHuman": 263.77,
"TestProxiedServerImport": 160.85,
"TestProxiedServerInfo": 28.32,
"TestProxiedServerInitExplicitBeadsDirIsAuthoritative": 11.99,
"TestProxiedServerInitIfMissing": 6.69,
"TestProxiedServerKV": 94.24,
"TestProxiedServerLabel": 179.25,
"TestProxiedServerLink": 51.49,
"TestProxiedServerLint": 15.51,
"TestProxiedServerList": 112.68,
"TestProxiedServerListComments": 31.13,
"TestProxiedServerListSortRetiresTheWalk": 46.29,
"TestProxiedServerListWatchCountsTheOffsetIntoMaxRows": 10.54,
"TestProxiedServerMemory": 58.87,
"TestProxiedServerMolBond": 79.34,
"TestProxiedServerMolBurn": 253.65,
"TestProxiedServerMolCurrent": 85.1,
"TestProxiedServerMolDistill": 78.03,
"TestProxiedServerMolLastActivity": 111.33,
"TestProxiedServerMolProgress": 182.51,
"TestProxiedServerMolReadyGated": 66.34,
"TestProxiedServerMolShow": 36.15,
"TestProxiedServerMolSquash": 118.23,
"TestProxiedServerMolStale": 53.95,
"TestProxiedServerMoleculeFrontDoorDirectServerProxyParity": 22.98,
"TestProxiedServerMoleculeFrontDoorWriteParity": 19.52,
"TestProxiedServerMultiStatementSQL": 6.76,
"TestProxiedServerNote": 11.21,
"TestProxiedServerOrphans": 12.11,
"TestProxiedServerOutageReconnectAcceptanceMatrix": 80.0,
"TestProxiedServerPathHelpers": 0.0,
"TestProxiedServerPing": 11.73,
"TestProxiedServerPour": 253.65,
"TestProxiedServerPourRejectsEnumViolatingVar": 21.66,
"TestProxiedServerPrime": 94.24,
"TestProxiedServerPriority": 41.35,
"TestProxiedServerPromote": 255.67,
"TestProxiedServerPrune": 119.06,
"TestProxiedServerPurgeWispsPlaneRetention": 24.55,
"TestProxiedServerQuery": 42.94,
"TestProxiedServerQuick": 22.7,
"TestProxiedServerReady": 238.66,
"TestProxiedServerReady2": 71.96,
"TestProxiedServerReadyClaimConcurrent": 20.86,
"TestProxiedServerReadyConcurrent": 11.34,
"TestProxiedServerReadyWisp": 255.67,
"TestProxiedServerReassignFence": 57.62,
"TestProxiedServerReclaim": 11.62,
"TestProxiedServerRecomputeBlocked": 19.71,
"TestProxiedServerRelate": 160.07,
"TestProxiedServerReopen": 125.1,
"TestProxiedServerReopenConcurrent": 66.34,
"TestProxiedServerSQL": 13.91,
"TestProxiedServerSQLDatabaseFlag": 12.28,
"TestProxiedServerSQLStatementRouting": 18.36,
"TestProxiedServerSearch": 78.35,
"TestProxiedServerServeActivatesTheEventsJournal": 9.87,
"TestProxiedServerServeAddComment": 25.29,
"TestProxiedServerServeAddDependencies": 15.59,
"TestProxiedServerServeBatchClose": 16.3,
"TestProxiedServerServeClaim": 16.32,
"TestProxiedServerServeClaimNext": 17.04,
"TestProxiedServerServeClose": 38.42,
"TestProxiedServerServeCount": 21.49,
"TestProxiedServerServeCreate": 11.84,
"TestProxiedServerServeDelete": 13.83,
"TestProxiedServerServeEdgeCounts": 20.14,
"TestProxiedServerServeLifecycle": 21.66,
"TestProxiedServerServeModeGate": 31.59,
"TestProxiedServerServeReadParity": 15.44,
"TestProxiedServerServeReadRevision": 10.4,
"TestProxiedServerServeReadsTheJournalItJustWrote": 38.39,
"TestProxiedServerServeRefusesAStaleCheckpointWithGone": 22.47,
"TestProxiedServerServeRefusesAStreamOnADisabledJournal": 42.35,
"TestProxiedServerServeRefusesTheJournalWhenItIsDisabled": 19.4,
"TestProxiedServerServeRelatedIssues": 28.93,
"TestProxiedServerServeRelease": 11.29,
"TestProxiedServerServeRemoveDependency": 12.52,
"TestProxiedServerServeSettingsWrites": 10.68,
"TestProxiedServerServeStreamsTheJournalItIsWriting": 11.89,
"TestProxiedServerServeUpdate": 17.56,
"TestProxiedServerServeUpdateClaim": 13.57,
"TestProxiedServerSetState": 15.28,
"TestProxiedServerShow": 216.16,
"TestProxiedServerShow2": 232.1,
"TestProxiedServerShow3": 157.69,
"TestProxiedServerShowWatch": 109.69,
"TestProxiedServerStale": 15.61,
"TestProxiedServerState": 110.26,
"TestProxiedServerStatus": 28.61,
"TestProxiedServerStatuses": 27.0,
"TestProxiedServerStorageClass": 150.78,
"TestProxiedServerSwarm": 29.89,
"TestProxiedServerTag": 33.46,
"TestProxiedServerTodo": 43.18,
"TestProxiedServerTypes": 12.51,
"TestProxiedServerTypesAgreeWithCreate": 14.34,
"TestProxiedServerUnclaim": 16.57,
"TestProxiedServerUnclaimIfAssigneeParity": 25.33,
"TestProxiedServerUpdate": 238.19,
"TestProxiedServerUpdate2": 170.45,
"TestProxiedServerUpdate3": 184.1,
"TestProxiedServerUpdateClosePolicy": 37.4,
"TestProxiedServerUpdateConcurrentAppendNotes": 6.6,
"TestProxiedServerUpdateConcurrentClaim": 17.68,
"TestProxiedServerUpdateConcurrentMetadata": 8.68,
"TestProxiedServerUpdateHooks": 20.14,
"TestProxiedServerUpdateIfGuards": 141.97,
"TestProxiedServerUpdateWisp": 82.23,
"TestProxiedServerWispCreate": 74.62,
"TestProxiedServerWispCreateRejectsEnumViolatingVar": 25.49,
"TestProxiedServerWispGC": 52.72,
"TestProxiedServerWispList": 118.23,
"TestServerModeDeferAutoWake": 32.77,
"TestServerModeServe": 19.28,
"TestServerModeServeSkipsPostRunMaintenance": 10.52,
"TestServerModeTypesAgreeWithCreate": 16.0
}
}