* fix(proxy): fail fast with a MySQL error when the external upstream is unreachable A proxied-server command against a stopped external Dolt upstream stalled ~20-30s before failing with a bare "invalid connection". The local db proxy stays up, so on a refused backend dial (or a front such as stunnel/socat that accepts and closes because its own upstream is gone) it simply closed the client connection. The MySQL driver reports that as invalid connection / unexpected EOF, which the uow bootstrap ping (#6003) classifies as a transient drop and retries with exponential backoff for its whole 30s budget. Retrying cannot help: the proxy already proved the upstream at its own startup, and the outage is the upstream's. The proxy now answers such a connection itself with a MySQL ERR packet (2003, HY000) in place of the server greeting, naming the upstream and the cause, exactly as a real server refuses a connection before the handshake. The driver surfaces it as a *mysql.MySQLError, which the bootstrap retry already treats as permanent, so the command fails in ~0.1s with an actionable message. Only shapes that prove the upstream is not serving qualify: ECONNREFUSED/ENOENT/EHOSTUNREACH/ENETUNREACH on dial, or a backend that reaches EOF having sent zero bytes (a MySQL server always speaks first). A dial timeout or a drop after the greeting keeps the plain close and the client's transient retry. External backends only: a managed (local sidecar) backend is owned by the proxy and its dial can be refused transiently while bd drives it (the full proxied lane caught TestProxiedServerInitExplicitBeadsDirIsAuthoritative hitting exactly that mid-init when this applied to every backend), so managed backends keep the bare close and the client's bootstrap retry. TestExternalProxyOutageAndRestoreReDialsTCPAndUnix asserted the old silent close during an outage; it now asserts the ERR packet. With the latency fixed, TestProxiedOutageReconnectAcceptanceMatrix is renamed into the proxied discovery pattern as TestProxiedServerOutageReconnectAcceptanceMatrix and placed in the Bazel 30-shard block by gen_proxied_shard_manifest.py's incremental --write (shard 8). The frozen legacy 15-shard block is untouched; the test hash-distributes there like every other test added since it was frozen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(proxy,uow): retry the proxy's upstream-outage report for ~1s; hedge the zero-byte case Review follow-up to the fail-fast change. A zero-byte close from the upstream is not proof it is down: Dolt closes queued connections without a byte when max_connections/back_log are full, when a queued connection times out, and while it shuts down for a restart; an L4 front does the same while its target is replaced. Its report now says "closed the connection before the MySQL greeting (down, restarting, or at its connection limit)" instead of "unreachable", and the comment no longer claims proof. Refusals and endpoint churn also lost #6003's retry entirely, so an external Dolt restart or a Service with no ready endpoints failed a command at once. The client now retries the proxy's report (proxy.IsUpstreamOutageError: 2003 with the "beads db proxy: " prefix) for upstreamOutageRetryWindow (1s) from the first time it sees it, with the backoff clamped so the last attempt lands at the window's end; other transient errors keep the 30s budget. A real outage still fails in about a second, inside the acceptance test's 3s bound. Also: the dial report drops Go call-site wrapping ("dial tcp H:P: connection refused"), truncation keeps UTF-8 intact, the comment states how the packet differs from a real server's pre-handshake ERR (keeps the '#'+SQLSTATE marker for vitess; 2003 is a client code chosen on purpose), a Stats counter for reported outages, and the isTransientPingError paragraph is reflowed and corrected. Tests: pingWithRetry retries the report inside the window and gives up at its end without using the full budget; openDB against a front that plays the proxy (its real packet) and comes back 300ms after the first connection succeeds in one call, and fails in about the window when it never does; NewProxyServer enables the report for external backends only; the dial message carries no Go type names; rune-safe truncation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(proxied): fail, not skip, socat-bridged subtests in the Bazel lane The external-unix front-door matrix subtest and both outage/reconnect matrix subtests bridge an external endpoint with socat and skipped when it was missing, so they could stop running without anyone noticing (the parent test still passes). requireSocat now fails them when BEADS_TEST_REQUIRE_SOCAT=1, which //cmd/bd:bd_proxied_test sets (its remote workers have socat; a recent main run's external-unix subtest passed there). The legacy GitHub 15-shard jobs have no socat and leave it unset, so the subtests keep skipping there. The check now runs before the upstream Dolt server is started, so a skip no longer pays for one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(proxied): measured weight for the outage matrix; pin it to legacy shard 1 Give TestProxiedServerOutageReconnectAcceptanceMatrix a duration entry (80s: local runs measured 55-79s alone, 60-70s in its shard, and review 68-109s under load) and re-place it with the generator's incremental --write: it moves from 30/8 to 30/11, one line, nothing else moved. The heaviest estimated shard is now 435s against a ~357s median; a repack would even that out but moves most assignments, so it is left to a separate CI-only change that refreshes all durations together. Legacy 15-shard block: the test would hash onto shard 14, already ~13.5 of the 15 min timeout on main. It is pinned to shard 1 (~6.5 min in recent main runs) under the block's reviewed-pin rule; it skips there today (no socat), and the pin keeps shard 14 safe if that changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
241 lines
12 KiB
JSON
241 lines
12 KiB
JSON
{
|
|
"_comment": [
|
|
"Measured locally with `bazel --nohome_rc test --config=doltserver-proxied",
|
|
"//cmd/bd:bd_proxied_test` -- the exact config bazel.yml's proxied-server tier",
|
|
"uses, including `-test.parallel=4` (local sandboxed execution only; no remote",
|
|
"executor or cache flags were configured for this measurement run; Bazel's own",
|
|
"--remote_download_regex etc. in that config do not enable remote execution",
|
|
"without --config=remote-exec, which was not passed).",
|
|
"",
|
|
"This replaces an earlier capture that used `-test.parallel=1` (serial). That",
|
|
"methodology avoided double-counting a parallel subtest group's own duration on",
|
|
"top of its children's, which was a genuine bug in an even earlier pass -- but",
|
|
"serial timings do not model this lane, which always runs at parallel=4, and an",
|
|
"independent re-review measured near-zero correlation (r=0.08) between the",
|
|
"serial-derived predicted shard load and actual parallel=4 wall time. These",
|
|
"values are each a clean, single-job (no other concurrent heavy bazel/dolt job",
|
|
"intentionally started on the box; see below), parallel=4 top-level test's own",
|
|
"<testsuite time=...> attribute from a single fresh, manifest-aligned 30-shard",
|
|
"sweep where every shard ran its exact assigned test set once (no duplicates,",
|
|
"no gaps -- see scripts/ci/gen_proxied_shard_manifest.py's exact-once coverage",
|
|
"check). A top-level Go test's <testsuite time> already reflects, not adds to,",
|
|
"its own parallel children's wall-clock time, so no leaf-summation step was",
|
|
"needed this time: every one of the 176 discovered tests completed inside a",
|
|
"single shard run with errors=\"0\" (none needed Bazel's per-target default",
|
|
"timeout worked around via a separate `bazel run`, unlike the prior capture).",
|
|
"",
|
|
"`seconds_per_init_fallback` is the weighted average seconds-per-init over",
|
|
"every measured test in this file (sum of all durations / sum of all inits),",
|
|
"used only for a brand-new test with no entry yet (see",
|
|
"scripts/ci/gen_proxied_shard_manifest.py's discover_inits_cost()). It dropped",
|
|
"from the serial capture's 39.7275 to this capture's lower value because",
|
|
"parallel=4 genuinely shortens most tests' wall time relative to inits count;",
|
|
"this is expected, not a data error.",
|
|
"",
|
|
"This file still does not claim shard totals land within a few seconds of each",
|
|
"other -- see .github/workflows/bazel.yml for the current, honest expected",
|
|
"spread for the 30-shard proxied lane, now also derived from this same sweep.",
|
|
"",
|
|
"The box this was measured on also runs other engineers'/agents' unrelated",
|
|
"Bazel jobs; this capture was not run with exclusive use of the machine, only",
|
|
"with no other *intentional, task-related* heavy bazel/dolt job started",
|
|
"concurrently by this task. Absolute seconds may be inflated by ambient",
|
|
"contention; relative ordering between tests, which is what LPT packing",
|
|
"actually depends on, should still be far more representative than the",
|
|
"discarded serial data (confirmed: 30/30 shards passed, 0 failures, 0 races,",
|
|
"with the two blocker fixes -- shortSocketPath and cmd.Cancel -- applied).",
|
|
"",
|
|
"Regenerate with a local `bazel --nohome_rc test --config=doltserver-proxied",
|
|
"//cmd/bd:bd_proxied_test --build_event_json_file=<path>` sweep, then pull",
|
|
"each shard's bazel-testlogs test.xml <testsuite time> per top-level test --",
|
|
"not from downloaded CI artifacts -- so the parallel=4, CI-representative",
|
|
"methodology above is applied consistently.",
|
|
"",
|
|
"Added after that sweep: TestProxiedServerOutageReconnectAcceptanceMatrix (80.0,",
|
|
"renamed into the discovery pattern once its outage bound was fixed). Not a",
|
|
"single sweep's value: local --config=doltserver-proxied runs measured it at",
|
|
"55-79 s alone, 60-70 s inside its shard, and a reviewer's 68-109 s under load;",
|
|
"80 s is a rounded middle of that spread. seconds_per_init_fallback was",
|
|
"recomputed to include it."
|
|
],
|
|
"seconds_per_init_fallback": 17.2952,
|
|
"tests": {
|
|
"TestProxiedServerAssign": 38.67,
|
|
"TestProxiedServerAutoPrune": 67.5,
|
|
"TestProxiedServerBackupRefusedOnExternalTopology": 5.96,
|
|
"TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce": 9.18,
|
|
"TestProxiedServerBatchParity": 60.04,
|
|
"TestProxiedServerBlocked": 138.63,
|
|
"TestProxiedServerCascadeParity": 78.23,
|
|
"TestProxiedServerChildren": 11.28,
|
|
"TestProxiedServerCleanDatabases": 7.14,
|
|
"TestProxiedServerCleanDatabasesPurgeDropped": 7.76,
|
|
"TestProxiedServerClose": 174.6,
|
|
"TestProxiedServerClose2": 218.83,
|
|
"TestProxiedServerCloseConcurrent": 38.25,
|
|
"TestProxiedServerCloseSiblingBlockersFanIn": 99.71,
|
|
"TestProxiedServerComment": 64.04,
|
|
"TestProxiedServerCompactDolt": 11.55,
|
|
"TestProxiedServerCompactHistory": 12.38,
|
|
"TestProxiedServerConfig": 188.91,
|
|
"TestProxiedServerConfigConcurrent": 48.59,
|
|
"TestProxiedServerConfigSetMany": 196.57,
|
|
"TestProxiedServerContext": 72.27,
|
|
"TestProxiedServerCount": 60.65,
|
|
"TestProxiedServerCountIncludeInfra": 34.69,
|
|
"TestProxiedServerCreate": 46.42,
|
|
"TestProxiedServerCreate2": 165.07,
|
|
"TestProxiedServerCreateInfraTypeRoutesToWisps": 142.17,
|
|
"TestProxiedServerDeferAutoWake": 12.95,
|
|
"TestProxiedServerDeferUndefer": 51.14,
|
|
"TestProxiedServerDelete": 44.82,
|
|
"TestProxiedServerDeleteBlockedJSONSingleDoc": 8.09,
|
|
"TestProxiedServerDeleteConcurrent": 10.57,
|
|
"TestProxiedServerDeleteForceOrphans": 9.16,
|
|
"TestProxiedServerDeleteWisp": 7.06,
|
|
"TestProxiedServerDep": 121.62,
|
|
"TestProxiedServerDep2": 227.28,
|
|
"TestProxiedServerDepConcurrent": 30.55,
|
|
"TestProxiedServerDepListSingleIDWarnsOnDroppedExternalEdge": 10.13,
|
|
"TestProxiedServerDoltLifecycle": 6.18,
|
|
"TestProxiedServerDoltRemoteRemove": 72.67,
|
|
"TestProxiedServerDuplicates": 44.77,
|
|
"TestProxiedServerEdit": 10.16,
|
|
"TestProxiedServerEpic": 109.49,
|
|
"TestProxiedServerEventsJournal": 6.48,
|
|
"TestProxiedServerEventsJournalOffByDefault": 65.83,
|
|
"TestProxiedServerExport": 26.44,
|
|
"TestProxiedServerExternalCreate": 5.73,
|
|
"TestProxiedServerExternalMutationPolicy": 25.0,
|
|
"TestProxiedServerExternalPolicyDirectParity": 18.12,
|
|
"TestProxiedServerFindDuplicates": 51.66,
|
|
"TestProxiedServerGC": 12.73,
|
|
"TestProxiedServerGateCheck": 140.99,
|
|
"TestProxiedServerGateCreateInheritsRepoMetadata": 47.1,
|
|
"TestProxiedServerGateFrontDoorDirectServerProxyParity": 17.36,
|
|
"TestProxiedServerGateLifecycle": 49.79,
|
|
"TestProxiedServerGateList": 41.19,
|
|
"TestProxiedServerGraph": 15.43,
|
|
"TestProxiedServerHeartbeat": 57.23,
|
|
"TestProxiedServerHistory": 21.81,
|
|
"TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix": 59.61,
|
|
"TestProxiedServerHistoryRemoteSupportedFrontDoorParity": 5.97,
|
|
"TestProxiedServerHuman": 263.77,
|
|
"TestProxiedServerImport": 160.85,
|
|
"TestProxiedServerInfo": 28.32,
|
|
"TestProxiedServerInitExplicitBeadsDirIsAuthoritative": 11.99,
|
|
"TestProxiedServerInitIfMissing": 6.69,
|
|
"TestProxiedServerKV": 94.24,
|
|
"TestProxiedServerLabel": 179.25,
|
|
"TestProxiedServerLink": 51.49,
|
|
"TestProxiedServerLint": 15.51,
|
|
"TestProxiedServerList": 112.68,
|
|
"TestProxiedServerListComments": 31.13,
|
|
"TestProxiedServerListSortRetiresTheWalk": 46.29,
|
|
"TestProxiedServerListWatchCountsTheOffsetIntoMaxRows": 10.54,
|
|
"TestProxiedServerMemory": 58.87,
|
|
"TestProxiedServerMolBond": 79.34,
|
|
"TestProxiedServerMolBurn": 253.65,
|
|
"TestProxiedServerMolCurrent": 85.1,
|
|
"TestProxiedServerMolDistill": 78.03,
|
|
"TestProxiedServerMolLastActivity": 111.33,
|
|
"TestProxiedServerMolProgress": 182.51,
|
|
"TestProxiedServerMolReadyGated": 66.34,
|
|
"TestProxiedServerMolShow": 36.15,
|
|
"TestProxiedServerMolSquash": 118.23,
|
|
"TestProxiedServerMolStale": 53.95,
|
|
"TestProxiedServerMoleculeFrontDoorDirectServerProxyParity": 22.98,
|
|
"TestProxiedServerMoleculeFrontDoorWriteParity": 19.52,
|
|
"TestProxiedServerMultiStatementSQL": 6.76,
|
|
"TestProxiedServerNote": 11.21,
|
|
"TestProxiedServerOrphans": 12.11,
|
|
"TestProxiedServerOutageReconnectAcceptanceMatrix": 80.0,
|
|
"TestProxiedServerPathHelpers": 0.0,
|
|
"TestProxiedServerPing": 11.73,
|
|
"TestProxiedServerPour": 253.65,
|
|
"TestProxiedServerPourRejectsEnumViolatingVar": 21.66,
|
|
"TestProxiedServerPrime": 94.24,
|
|
"TestProxiedServerPriority": 41.35,
|
|
"TestProxiedServerPromote": 255.67,
|
|
"TestProxiedServerPrune": 119.06,
|
|
"TestProxiedServerPurgeWispsPlaneRetention": 24.55,
|
|
"TestProxiedServerQuery": 42.94,
|
|
"TestProxiedServerQuick": 22.7,
|
|
"TestProxiedServerReady": 238.66,
|
|
"TestProxiedServerReady2": 71.96,
|
|
"TestProxiedServerReadyClaimConcurrent": 20.86,
|
|
"TestProxiedServerReadyConcurrent": 11.34,
|
|
"TestProxiedServerReadyWisp": 255.67,
|
|
"TestProxiedServerReassignFence": 57.62,
|
|
"TestProxiedServerReclaim": 11.62,
|
|
"TestProxiedServerRecomputeBlocked": 19.71,
|
|
"TestProxiedServerRelate": 160.07,
|
|
"TestProxiedServerReopen": 125.1,
|
|
"TestProxiedServerReopenConcurrent": 66.34,
|
|
"TestProxiedServerSQL": 13.91,
|
|
"TestProxiedServerSQLDatabaseFlag": 12.28,
|
|
"TestProxiedServerSQLStatementRouting": 18.36,
|
|
"TestProxiedServerSearch": 78.35,
|
|
"TestProxiedServerServeActivatesTheEventsJournal": 9.87,
|
|
"TestProxiedServerServeAddComment": 25.29,
|
|
"TestProxiedServerServeAddDependencies": 15.59,
|
|
"TestProxiedServerServeBatchClose": 16.3,
|
|
"TestProxiedServerServeClaim": 16.32,
|
|
"TestProxiedServerServeClaimNext": 17.04,
|
|
"TestProxiedServerServeClose": 38.42,
|
|
"TestProxiedServerServeCount": 21.49,
|
|
"TestProxiedServerServeCreate": 11.84,
|
|
"TestProxiedServerServeDelete": 13.83,
|
|
"TestProxiedServerServeEdgeCounts": 20.14,
|
|
"TestProxiedServerServeLifecycle": 21.66,
|
|
"TestProxiedServerServeModeGate": 31.59,
|
|
"TestProxiedServerServeReadParity": 15.44,
|
|
"TestProxiedServerServeReadRevision": 10.4,
|
|
"TestProxiedServerServeReadsTheJournalItJustWrote": 38.39,
|
|
"TestProxiedServerServeRefusesAStaleCheckpointWithGone": 22.47,
|
|
"TestProxiedServerServeRefusesAStreamOnADisabledJournal": 42.35,
|
|
"TestProxiedServerServeRefusesTheJournalWhenItIsDisabled": 19.4,
|
|
"TestProxiedServerServeRelatedIssues": 28.93,
|
|
"TestProxiedServerServeRelease": 11.29,
|
|
"TestProxiedServerServeRemoveDependency": 12.52,
|
|
"TestProxiedServerServeSettingsWrites": 10.68,
|
|
"TestProxiedServerServeStreamsTheJournalItIsWriting": 11.89,
|
|
"TestProxiedServerServeUpdate": 17.56,
|
|
"TestProxiedServerServeUpdateClaim": 13.57,
|
|
"TestProxiedServerSetState": 15.28,
|
|
"TestProxiedServerShow": 216.16,
|
|
"TestProxiedServerShow2": 232.1,
|
|
"TestProxiedServerShow3": 157.69,
|
|
"TestProxiedServerShowWatch": 109.69,
|
|
"TestProxiedServerStale": 15.61,
|
|
"TestProxiedServerState": 110.26,
|
|
"TestProxiedServerStatus": 28.61,
|
|
"TestProxiedServerStatuses": 27.0,
|
|
"TestProxiedServerStorageClass": 150.78,
|
|
"TestProxiedServerSwarm": 29.89,
|
|
"TestProxiedServerTag": 33.46,
|
|
"TestProxiedServerTodo": 43.18,
|
|
"TestProxiedServerTypes": 12.51,
|
|
"TestProxiedServerTypesAgreeWithCreate": 14.34,
|
|
"TestProxiedServerUnclaim": 16.57,
|
|
"TestProxiedServerUnclaimIfAssigneeParity": 25.33,
|
|
"TestProxiedServerUpdate": 238.19,
|
|
"TestProxiedServerUpdate2": 170.45,
|
|
"TestProxiedServerUpdate3": 184.1,
|
|
"TestProxiedServerUpdateClosePolicy": 37.4,
|
|
"TestProxiedServerUpdateConcurrentAppendNotes": 6.6,
|
|
"TestProxiedServerUpdateConcurrentClaim": 17.68,
|
|
"TestProxiedServerUpdateConcurrentMetadata": 8.68,
|
|
"TestProxiedServerUpdateHooks": 20.14,
|
|
"TestProxiedServerUpdateIfGuards": 141.97,
|
|
"TestProxiedServerUpdateWisp": 82.23,
|
|
"TestProxiedServerWispCreate": 74.62,
|
|
"TestProxiedServerWispCreateRejectsEnumViolatingVar": 25.49,
|
|
"TestProxiedServerWispGC": 52.72,
|
|
"TestProxiedServerWispList": 118.23,
|
|
"TestServerModeDeferAutoWake": 32.77,
|
|
"TestServerModeServe": 19.28,
|
|
"TestServerModeServeSkipsPostRunMaintenance": 10.52,
|
|
"TestServerModeTypesAgreeWithCreate": 16.0
|
|
}
|
|
}
|