Fix relay E2EE reconnect races (#973)

* Fix relay E2EE reconnect races

* fix(relay): use WebSocket protocol ping for daemon control keepalive

The daemon's control socket pinged the relay every 10s with a JSON
`{type:"ping"}` message. Every JSON message wakes a hibernated Cloudflare
Durable Object via `webSocketMessage`, so an idle daemon was billing
relay DO CPU 24/7 just to echo `pong`.

Switch to `socket.ping()` (RFC 6455 protocol ping frames). Cloudflare's
runtime auto-responds at the edge without waking hibernated DOs, so an
idle daemon-to-relay connection now costs zero DO CPU.

Liveness still works: the `ws` library emits a `pong` event on the
daemon when the edge replies, which refreshes `controlLastSeenAt`.
`markControlReady` stays tied to DO-originated messages (sync,
connected, disconnected), so we still detect a broken DO at connect.

The DO keeps its JSON ping handler tagged `COMPAT(relay-json-ping)` so
old daemons (< v0.1.76) continue to work.

* test(relay): assert protocol-ping pongs arrive in e2e idle test

Adds a `relay_control_pong_received` debug log to the daemon's pong
handler, then asserts in the existing idle-relay e2e test that:

  - at least one pong arrived during the 12s idle wait (proving the
    Cloudflare edge auto-responds to RFC 6455 protocol pings)
  - no stale terminations fired (proving the keepalive path is healthy)

If the daemon ever regresses to JSON-level keepalive pings (which wake
the hibernated relay Durable Object and incur CPU billing), this test
will catch it: the `pong` event would never fire because the DO's reply
would arrive via `message`, not as a protocol pong frame.

The test's capturing logger is bumped to `debug` level so the assertion
can read the new debug log; production daemons stay at `info`.

Confirmed locally against wrangler-hosted relay: 2 pongs / 0 stale per
12s window.

* test(relay): catch dual-ping regression via DO-side legacy ping log

The existing idle-test assertion catches "revert to JSON ping only" via
the missing pong event, but misses the dual-ping regression where
someone adds `socket.send(JSON.stringify({type:"ping"}))` alongside
`socket.ping()`. Both paths would work, the pong assertion still
passes, and DO CPU billing silently returns.

Add a console.log in the DO's handleControlKeepalive (which only runs
when a JSON ping reaches the DO), then assert in the idle e2e test
that no such log appeared during the 12s idle window.

Verified the assertion catches the regression: temporarily injected a
`socket.send(JSON.stringify({type:"ping"}))` next to socket.ping(),
test failed with `expected 1 to be 0` and the relay logged
`legacy_json_ping_received`. Removed the injection.

Log noise during the COMPAT window doubles as useful telemetry: it
counts how many old daemons are still pinging, which informs the
removal date for the COMPAT shim.
This commit is contained in:
Mohamed Boudra
2026-05-13 13:44:11 +08:00
committed by GitHub
parent 15cab6014c
commit d24087c10a
6 changed files with 301 additions and 11 deletions
@@ -302,6 +302,58 @@ function makeProbeMap(
}
describe("HostRuntimeController", () => {
it("replaces the active relay client when re-pairing changes the daemon public key", async () => {
const oldRelay: HostConnection = {
id: "relay:wss:relay.paseo.sh:443",
type: "relay",
relayEndpoint: "relay.paseo.sh:443",
useTls: true,
daemonPublicKeyB64: "pk_old",
};
const newRelay: HostConnection = {
...oldRelay,
daemonPublicKeyB64: "pk_new",
};
const createdClients: Array<{ client: FakeDaemonClient; connection: HostConnection }> = [];
const controller = new HostRuntimeController({
host: makeHost({
connections: [oldRelay],
preferredConnectionId: oldRelay.id,
}),
deps: {
createClient: ({ connection }) => {
const client = new FakeDaemonClient();
createdClients.push({ client, connection });
return client as unknown as DaemonClient;
},
connectToDaemon: async ({ host, connection }) => ({
client: makeConnectedProbeClient(5) as unknown as DaemonClient,
serverId: host.serverId,
hostname: connection.id,
}),
getClientId: async () => "cid_test_runtime",
},
});
await (
controller as unknown as {
switchToConnection: (input: { connectionId: string }) => Promise<void>;
}
).switchToConnection({ connectionId: oldRelay.id });
expect(controller.getSnapshot().client).toBe(createdClients[0]?.client);
await controller.updateHost(
makeHost({
connections: [newRelay],
preferredConnectionId: newRelay.id,
}),
);
expect(createdClients.map((entry) => entry.connection)).toEqual([oldRelay, newRelay]);
expect(createdClients[0]?.client.closeCalls).toBe(1);
expect(controller.getSnapshot().client).toBe(createdClients[1]?.client);
});
it("keeps known hosts in connecting when a created client reports idle during connect", async () => {
const host = makeHost({
connections: [
+12
View File
@@ -602,8 +602,20 @@ export class HostRuntimeController {
}
async updateHost(host: HostProfile): Promise<void> {
const activeConnectionId = this.snapshot.activeConnectionId;
const previousActiveConnection = findConnectionById(this.host, activeConnectionId);
this.host = host;
this.trackConnectionFirstSeen();
const nextActiveConnection = findConnectionById(this.host, activeConnectionId);
if (
activeConnectionId &&
previousActiveConnection &&
nextActiveConnection &&
!equal(previousActiveConnection, nextActiveConnection)
) {
this.connectionLastProbedAt.delete(activeConnectionId);
await this.switchToConnection({ connectionId: activeConnectionId });
}
await this.runProbeCycleNow();
}