Files
notes/PhenixRTS/frontend-draining/Potential causes.md
2025-11-25 21:38:17 -05:00

38 lines
3.9 KiB
Markdown
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

- Close path can loop forever if peer never finishes closing. When `close()` is called we just invoke `socket.close()` and wait for a future `disconnectDelegate` If the FIN/ACK never arrives, we keep the entry and never fall back to terminate() (despite adding that method). Consider scheduling a timeout so that a close automatically escalates to terminate() and disconnectDelegate cleanup.
```sql
-- Check for send errors during drain
SELECT
timestamp,
message
FROM
`phenix-pcast.pcast_logs_us.syslog`
WHERE
Timestamp > TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -10 DAY)
AND Facility = 'platform'
AND HostName = "frontend-australia-southeast-1-k3ms"
AND (
message LIKE '%Failed to send%'
OR message LIKE '%not opened%'
OR message LIKE '%not found%'
)
ORDER BY timestamp DESC
LIMIT 100;
```
| id | Timestamp | Message | Hostname | Thread |
| :-- | :----------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------- | :----- |
| 0 | 2025-11-22 14:09:32.453398 UTC | [australia-southeast#WOkAeFYOsT3gWMwVbTLpcHYqIEYttN9k] Rejecting message [chat.RoomEvent] for closed client: Client MQ adapter for australia-southeast#WOkAeFYOsT3gWMwVbTLpcHYqIEYttN9k not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 1 | 2025-11-22 14:04:56.692624 UTC | [australia-southeast#f0AOMhXqggfsFmog5FYfDOfHzl4guVTp] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#f0AOMhXqggfsFmog5FYfDOfHzl4guVTp not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 2 | 2025-11-22 14:04:55.976509 UTC | [australia-southeast#01let42yTTGJJViCmDNG5YkfPAhr1Fk5] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#01let42yTTGJJViCmDNG5YkfPAhr1Fk5 not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 3 | 2025-11-22 14:03:48.854516 UTC | [australia-southeast#x1o42d2VicP5Cfq9pYH1k9eW6sKi5djC] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#x1o42d2VicP5Cfq9pYH1k9eW6sKi5djC not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 4 | 2025-11-22 10:24:52.469294 UTC | [australia-southeast#TVhflpoNzOK6dSloQCFqoYcY1ttbiOiM] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#TVhflpoNzOK6dSloQCFqoYcY1ttbiOiM not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 5 | 2025-11-22 10:00:42.335501 UTC | [australia-southeast#Kb5C372wLecwyTihj1yO1jZT5t6ghNmU] Rejecting message [chat.RoomEvent] for closed client: Client MQ adapter for australia-southeast#Kb5C372wLecwyTihj1yO1jZT5t6ghNmU not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 6 | 2025-11-22 06:46:44.278841 UTC | [australia-southeast#f4cOuMd2wMGQ7v8DWLVYbCu6andAXeKO] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#f4cOuMd2wMGQ7v8DWLVYbCu6andAXeKO not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 7 | 2025-11-22 06:37:02.677132 UTC | [australia-southeast#vKYJCpFY1rXi4VT0Rn1A5ufFvbHuUNX4] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#vKYJCpFY1rXi4VT0Rn1A5ufFvbHuUNX4 not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 8 | 2025-11-22 06:36:11.151866 UTC | [australia-southeast#S8vqusQFBG4s7zK4IS1hlqIg5ADM4KwB] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#S8vqusQFBG4s7zK4IS1hlqIg5ADM4KwB not found | frontend-australia-southeast-1-k3ms | PID:35 |