Fix relay E2EE reconnect races (#973)
* Fix relay E2EE reconnect races
* fix(relay): use WebSocket protocol ping for daemon control keepalive
The daemon's control socket pinged the relay every 10s with a JSON
`{type:"ping"}` message. Every JSON message wakes a hibernated Cloudflare
Durable Object via `webSocketMessage`, so an idle daemon was billing
relay DO CPU 24/7 just to echo `pong`.
Switch to `socket.ping()` (RFC 6455 protocol ping frames). Cloudflare's
runtime auto-responds at the edge without waking hibernated DOs, so an
idle daemon-to-relay connection now costs zero DO CPU.
Liveness still works: the `ws` library emits a `pong` event on the
daemon when the edge replies, which refreshes `controlLastSeenAt`.
`markControlReady` stays tied to DO-originated messages (sync,
connected, disconnected), so we still detect a broken DO at connect.
The DO keeps its JSON ping handler tagged `COMPAT(relay-json-ping)` so
old daemons (< v0.1.76) continue to work.
* test(relay): assert protocol-ping pongs arrive in e2e idle test
Adds a `relay_control_pong_received` debug log to the daemon's pong
handler, then asserts in the existing idle-relay e2e test that:
- at least one pong arrived during the 12s idle wait (proving the
Cloudflare edge auto-responds to RFC 6455 protocol pings)
- no stale terminations fired (proving the keepalive path is healthy)
If the daemon ever regresses to JSON-level keepalive pings (which wake
the hibernated relay Durable Object and incur CPU billing), this test
will catch it: the `pong` event would never fire because the DO's reply
would arrive via `message`, not as a protocol pong frame.
The test's capturing logger is bumped to `debug` level so the assertion
can read the new debug log; production daemons stay at `info`.
Confirmed locally against wrangler-hosted relay: 2 pongs / 0 stale per
12s window.
* test(relay): catch dual-ping regression via DO-side legacy ping log
The existing idle-test assertion catches "revert to JSON ping only" via
the missing pong event, but misses the dual-ping regression where
someone adds `socket.send(JSON.stringify({type:"ping"}))` alongside
`socket.ping()`. Both paths would work, the pong assertion still
passes, and DO CPU billing silently returns.
Add a console.log in the DO's handleControlKeepalive (which only runs
when a JSON ping reaches the DO), then assert in the idle e2e test
that no such log appeared during the 12s idle window.
Verified the assertion catches the regression: temporarily injected a
`socket.send(JSON.stringify({type:"ping"}))` next to socket.ping(),
test failed with `expected 1 to be 0` and the relay logged
`legacy_json_ping_received`. Removed the injection.
Log noise during the COMPAT window doubles as useful telemetry: it
counts how many old daemons are still pinging, which informs the
removal date for the COMPAT shim.
This commit is contained in:
@@ -602,8 +602,20 @@ export class HostRuntimeController {
|
||||
}
|
||||
|
||||
async updateHost(host: HostProfile): Promise<void> {
|
||||
const activeConnectionId = this.snapshot.activeConnectionId;
|
||||
const previousActiveConnection = findConnectionById(this.host, activeConnectionId);
|
||||
this.host = host;
|
||||
this.trackConnectionFirstSeen();
|
||||
const nextActiveConnection = findConnectionById(this.host, activeConnectionId);
|
||||
if (
|
||||
activeConnectionId &&
|
||||
previousActiveConnection &&
|
||||
nextActiveConnection &&
|
||||
!equal(previousActiveConnection, nextActiveConnection)
|
||||
) {
|
||||
this.connectionLastProbedAt.delete(activeConnectionId);
|
||||
await this.switchToConnection({ connectionId: activeConnectionId });
|
||||
}
|
||||
await this.runProbeCycleNow();
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user