Keep connections alive through brief daemon slowdowns (#1581)

* Keep connections alive through brief daemon slowdowns

The active connection's liveness check was a side effect of the
candidate-probe loop: a single 5s latency measurement doubled as the
death timeout, so a daemon that was alive but briefly stalled (e.g. a
busy event loop) was misread as dead and the connection was torn down.

Give DaemonClient its own self-scheduling heartbeat that solely owns
liveness (15s timeout, 2 consecutive misses), and make the probe
read-only on the active connection — it reads the heartbeat's last RTT
instead of pinging. Split the shared ping into a pure measurement path
(measureLatency) and a private liveness path, so nothing outside the
client can drive teardown.

* Don't let a candidate latency timeout count as a liveness failure

The heartbeat (livenessPing) and candidate ranking (measureLatency) share
one in-flight ping slot. If a heartbeat tick lands while a candidate
measurement is still in flight, the heartbeat dedupes onto it — and a
measurement timeout was then recorded as a liveness failure, nudging the
connection one step closer to a spurious teardown.

Tag each ping with whether its timeout may drive a liveness failure
(only the heartbeat sets it) and gate recordLivenessFailure on that flag,
so an adopted measurement probe can never contribute to teardown.
This commit is contained in:
Mohamed Boudra
2026-06-18 21:10:08 +08:00
committed by GitHub
parent eb628765d7
commit ea2f768d3f
4 changed files with 667 additions and 343 deletions
+17 -2
View File
@@ -674,7 +674,7 @@ export class HostRuntimeController {
async activateConnection(input: {
connectionId: string;
existingClient: DaemonClient;
existingClient?: DaemonClient;
}): Promise<void> {
await this.switchToConnection(input);
}
@@ -913,7 +913,22 @@ export class HostRuntimeController {
const activated = await maybeActivateFirstAvailable(connection.id, connectedClient);
shouldCloseClient = shouldCloseClient && !activated;
const { rttMs } = await connectedClient.checkLiveness({ timeoutMs: 5000 });
if (activeClient) {
const rttMs = activeClient.getLastLivenessRttMs();
if (!this.isCurrentProbeRequest(requestVersion)) {
return;
}
if (rttMs !== null) {
probeByConnectionId.set(connection.id, {
status: "available",
latencyMs: rttMs,
});
publishProbeState();
}
return;
}
const rttMs = await connectedClient.measureLatency({ timeoutMs: 5000 });
if (!this.isCurrentProbeRequest(requestVersion)) {
return;
}