notes/PhenixRTS/frontend-draining/Potential causes.md

• Close path can loop forever if peer never finishes closing. When close() is called we just invoke socket.close() and wait for a future disconnectDelegate If the FIN/ACK never arrives, we keep the entry and never fall back to terminate() (despite adding that method). Consider scheduling a timeout so that a close automatically escalates to terminate() and disconnectDelegate cleanup.

-- Check for send errors during drain
SELECT 
  timestamp,
  message
  FROM
  `phenix-pcast.pcast_logs_us.syslog`
WHERE 
Timestamp > TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -10 DAY)
  AND Facility = 'platform'
  AND HostName = "frontend-australia-southeast-1-k3ms"
  AND (
    message LIKE '%Failed to send%'
    OR message LIKE '%not opened%'
    OR message LIKE '%not found%'
  )
ORDER BY timestamp DESC
LIMIT 100;

id	Timestamp	Message	Hostname	Thread
0	2025-11-22 14:09:32.453398 UTC	[australia-southeast#WOkAeFYOsT3gWMwVbTLpcHYqIEYttN9k] Rejecting message [chat.RoomEvent] for closed client: Client MQ adapter for australia-southeast#WOkAeFYOsT3gWMwVbTLpcHYqIEYttN9k not found	frontend-australia-southeast-1-k3ms	PID:35
1	2025-11-22 14:04:56.692624 UTC	[australia-southeast#f0AOMhXqggfsFmog5FYfDOfHzl4guVTp] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#f0AOMhXqggfsFmog5FYfDOfHzl4guVTp not found	frontend-australia-southeast-1-k3ms	PID:35
2	2025-11-22 14:04:55.976509 UTC	[australia-southeast#01let42yTTGJJViCmDNG5YkfPAhr1Fk5] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#01let42yTTGJJViCmDNG5YkfPAhr1Fk5 not found	frontend-australia-southeast-1-k3ms	PID:35
3	2025-11-22 14:03:48.854516 UTC	[australia-southeast#x1o42d2VicP5Cfq9pYH1k9eW6sKi5djC] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#x1o42d2VicP5Cfq9pYH1k9eW6sKi5djC not found	frontend-australia-southeast-1-k3ms	PID:35
4	2025-11-22 10:24:52.469294 UTC	[australia-southeast#TVhflpoNzOK6dSloQCFqoYcY1ttbiOiM] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#TVhflpoNzOK6dSloQCFqoYcY1ttbiOiM not found	frontend-australia-southeast-1-k3ms	PID:35
5	2025-11-22 10:00:42.335501 UTC	[australia-southeast#Kb5C372wLecwyTihj1yO1jZT5t6ghNmU] Rejecting message [chat.RoomEvent] for closed client: Client MQ adapter for australia-southeast#Kb5C372wLecwyTihj1yO1jZT5t6ghNmU not found	frontend-australia-southeast-1-k3ms	PID:35
6	2025-11-22 06:46:44.278841 UTC	[australia-southeast#f4cOuMd2wMGQ7v8DWLVYbCu6andAXeKO] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#f4cOuMd2wMGQ7v8DWLVYbCu6andAXeKO not found	frontend-australia-southeast-1-k3ms	PID:35
7	2025-11-22 06:37:02.677132 UTC	[australia-southeast#vKYJCpFY1rXi4VT0Rn1A5ufFvbHuUNX4] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#vKYJCpFY1rXi4VT0Rn1A5ufFvbHuUNX4 not found	frontend-australia-southeast-1-k3ms	PID:35
8	2025-11-22 06:36:11.151866 UTC	[australia-southeast#S8vqusQFBG4s7zK4IS1hlqIg5ADM4KwB] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#S8vqusQFBG4s7zK4IS1hlqIg5ADM4KwB not found	frontend-australia-southeast-1-k3ms	PID:35