Fix relay E2EE reconnect races (#973)
* Fix relay E2EE reconnect races
* fix(relay): use WebSocket protocol ping for daemon control keepalive
The daemon's control socket pinged the relay every 10s with a JSON
`{type:"ping"}` message. Every JSON message wakes a hibernated Cloudflare
Durable Object via `webSocketMessage`, so an idle daemon was billing
relay DO CPU 24/7 just to echo `pong`.
Switch to `socket.ping()` (RFC 6455 protocol ping frames). Cloudflare's
runtime auto-responds at the edge without waking hibernated DOs, so an
idle daemon-to-relay connection now costs zero DO CPU.
Liveness still works: the `ws` library emits a `pong` event on the
daemon when the edge replies, which refreshes `controlLastSeenAt`.
`markControlReady` stays tied to DO-originated messages (sync,
connected, disconnected), so we still detect a broken DO at connect.
The DO keeps its JSON ping handler tagged `COMPAT(relay-json-ping)` so
old daemons (< v0.1.76) continue to work.
* test(relay): assert protocol-ping pongs arrive in e2e idle test
Adds a `relay_control_pong_received` debug log to the daemon's pong
handler, then asserts in the existing idle-relay e2e test that:
- at least one pong arrived during the 12s idle wait (proving the
Cloudflare edge auto-responds to RFC 6455 protocol pings)
- no stale terminations fired (proving the keepalive path is healthy)
If the daemon ever regresses to JSON-level keepalive pings (which wake
the hibernated relay Durable Object and incur CPU billing), this test
will catch it: the `pong` event would never fire because the DO's reply
would arrive via `message`, not as a protocol pong frame.
The test's capturing logger is bumped to `debug` level so the assertion
can read the new debug log; production daemons stay at `info`.
Confirmed locally against wrangler-hosted relay: 2 pongs / 0 stale per
12s window.
* test(relay): catch dual-ping regression via DO-side legacy ping log
The existing idle-test assertion catches "revert to JSON ping only" via
the missing pong event, but misses the dual-ping regression where
someone adds `socket.send(JSON.stringify({type:"ping"}))` alongside
`socket.ping()`. Both paths would work, the pong assertion still
passes, and DO CPU billing silently returns.
Add a console.log in the DO's handleControlKeepalive (which only runs
when a JSON ping reaches the DO), then assert in the idle e2e test
that no such log appeared during the 12s idle window.
Verified the assertion catches the regression: temporarily injected a
`socket.send(JSON.stringify({type:"ping"}))` next to socket.ping(),
test failed with `expected 1 to be 0` and the relay logged
`legacy_json_ping_received`. Removed the injection.
Log noise during the COMPAT window doubles as useful telemetry: it
counts how many old daemons are still pinging, which informs the
removal date for the COMPAT shim.
This commit is contained in:
@@ -302,6 +302,58 @@ function makeProbeMap(
|
||||
}
|
||||
|
||||
describe("HostRuntimeController", () => {
|
||||
it("replaces the active relay client when re-pairing changes the daemon public key", async () => {
|
||||
const oldRelay: HostConnection = {
|
||||
id: "relay:wss:relay.paseo.sh:443",
|
||||
type: "relay",
|
||||
relayEndpoint: "relay.paseo.sh:443",
|
||||
useTls: true,
|
||||
daemonPublicKeyB64: "pk_old",
|
||||
};
|
||||
const newRelay: HostConnection = {
|
||||
...oldRelay,
|
||||
daemonPublicKeyB64: "pk_new",
|
||||
};
|
||||
const createdClients: Array<{ client: FakeDaemonClient; connection: HostConnection }> = [];
|
||||
const controller = new HostRuntimeController({
|
||||
host: makeHost({
|
||||
connections: [oldRelay],
|
||||
preferredConnectionId: oldRelay.id,
|
||||
}),
|
||||
deps: {
|
||||
createClient: ({ connection }) => {
|
||||
const client = new FakeDaemonClient();
|
||||
createdClients.push({ client, connection });
|
||||
return client as unknown as DaemonClient;
|
||||
},
|
||||
connectToDaemon: async ({ host, connection }) => ({
|
||||
client: makeConnectedProbeClient(5) as unknown as DaemonClient,
|
||||
serverId: host.serverId,
|
||||
hostname: connection.id,
|
||||
}),
|
||||
getClientId: async () => "cid_test_runtime",
|
||||
},
|
||||
});
|
||||
|
||||
await (
|
||||
controller as unknown as {
|
||||
switchToConnection: (input: { connectionId: string }) => Promise<void>;
|
||||
}
|
||||
).switchToConnection({ connectionId: oldRelay.id });
|
||||
expect(controller.getSnapshot().client).toBe(createdClients[0]?.client);
|
||||
|
||||
await controller.updateHost(
|
||||
makeHost({
|
||||
connections: [newRelay],
|
||||
preferredConnectionId: newRelay.id,
|
||||
}),
|
||||
);
|
||||
|
||||
expect(createdClients.map((entry) => entry.connection)).toEqual([oldRelay, newRelay]);
|
||||
expect(createdClients[0]?.client.closeCalls).toBe(1);
|
||||
expect(controller.getSnapshot().client).toBe(createdClients[1]?.client);
|
||||
});
|
||||
|
||||
it("keeps known hosts in connecting when a created client reports idle during connect", async () => {
|
||||
const host = makeHost({
|
||||
connections: [
|
||||
|
||||
Reference in New Issue
Block a user