Files
notes/PhenixRTS/frontend-draining/Potential causes.md
2025-11-25 21:38:17 -05:00

38 lines
3.9 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

- Close path can loop forever if peer never finishes closing. When `close()` is called we just invoke `socket.close()` and wait for a future `disconnectDelegate` If the FIN/ACK never arrives, we keep the entry and never fall back to terminate() (despite adding that method). Consider scheduling a timeout so that a close automatically escalates to terminate() and disconnectDelegate cleanup.
```sql
-- Check for send errors during drain
SELECT
timestamp,
message
FROM
`phenix-pcast.pcast_logs_us.syslog`
WHERE
Timestamp > TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -10 DAY)
AND Facility = 'platform'
AND HostName = "frontend-australia-southeast-1-k3ms"
AND (
message LIKE '%Failed to send%'
OR message LIKE '%not opened%'
OR message LIKE '%not found%'
)
ORDER BY timestamp DESC
LIMIT 100;
```
| id | Timestamp | Message | Hostname | Thread |
| :-- | :----------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------- | :----- |
| 0 | 2025-11-22 14:09:32.453398 UTC | [australia-southeast#WOkAeFYOsT3gWMwVbTLpcHYqIEYttN9k] Rejecting message [chat.RoomEvent] for closed client: Client MQ adapter for australia-southeast#WOkAeFYOsT3gWMwVbTLpcHYqIEYttN9k not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 1 | 2025-11-22 14:04:56.692624 UTC | [australia-southeast#f0AOMhXqggfsFmog5FYfDOfHzl4guVTp] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#f0AOMhXqggfsFmog5FYfDOfHzl4guVTp not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 2 | 2025-11-22 14:04:55.976509 UTC | [australia-southeast#01let42yTTGJJViCmDNG5YkfPAhr1Fk5] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#01let42yTTGJJViCmDNG5YkfPAhr1Fk5 not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 3 | 2025-11-22 14:03:48.854516 UTC | [australia-southeast#x1o42d2VicP5Cfq9pYH1k9eW6sKi5djC] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#x1o42d2VicP5Cfq9pYH1k9eW6sKi5djC not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 4 | 2025-11-22 10:24:52.469294 UTC | [australia-southeast#TVhflpoNzOK6dSloQCFqoYcY1ttbiOiM] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#TVhflpoNzOK6dSloQCFqoYcY1ttbiOiM not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 5 | 2025-11-22 10:00:42.335501 UTC | [australia-southeast#Kb5C372wLecwyTihj1yO1jZT5t6ghNmU] Rejecting message [chat.RoomEvent] for closed client: Client MQ adapter for australia-southeast#Kb5C372wLecwyTihj1yO1jZT5t6ghNmU not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 6 | 2025-11-22 06:46:44.278841 UTC | [australia-southeast#f4cOuMd2wMGQ7v8DWLVYbCu6andAXeKO] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#f4cOuMd2wMGQ7v8DWLVYbCu6andAXeKO not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 7 | 2025-11-22 06:37:02.677132 UTC | [australia-southeast#vKYJCpFY1rXi4VT0Rn1A5ufFvbHuUNX4] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#vKYJCpFY1rXi4VT0Rn1A5ufFvbHuUNX4 not found | frontend-australia-southeast-1-k3ms | PID:35 |
| 8 | 2025-11-22 06:36:11.151866 UTC | [australia-southeast#S8vqusQFBG4s7zK4IS1hlqIg5ADM4KwB] Rejecting message [pcast.StreamEnded] for closed client: Client MQ adapter for australia-southeast#S8vqusQFBG4s7zK4IS1hlqIg5ADM4KwB not found | frontend-australia-southeast-1-k3ms | PID:35 |