This issue is resolved. From 08:09 UTC on 13 October, a single malformed record blocked one of the queues we use to process thread events, which delayed a small share of thread updates across workspaces.
Messages were received as normal throughout. We've deployed a fix and processed the delayed events, and everything is back to normal. If a thread still looks out of place, please get in touch and we'll take a look.
Resolved
This issue is resolved. From 08:09 UTC on 13 October, a single malformed record blocked one of the queues we use to process thread events, which delayed a small share of thread updates across workspaces.
Messages were received as normal throughout. We've deployed a fix and processed the delayed events, and everything is back to normal. If a thread still looks out of place, please get in touch and we'll take a look.
Monitoring
We've deployed a fix, and delayed thread events are now being processed. Affected threads are catching up, so you may still see some catch-up activity for a short while.
Messages were received as normal throughout. We're monitoring closely and will post another update once everything has fully caught up.
Identified
We've identified the cause. A single malformed record was blocking one of the queues we use to process thread events, which affects a small share of thread events (around 2%) across workspaces. Messages are still being received as normal. We're preparing to clear the backlog shortly, and when we do you may see a short burst of catch-up activity.
We'll post another update once the backlog has been processed.
Investigating
We're continuing to investigate delayed processing of some thread events. Affected threads may not update as expected. For example, a thread may not be assigned or reopened, an SLA timer may not stop when a reply is sent, or a snoozed thread may not move back to needing a response when a customer replies. Messages are still being received. We've found the part of our system that's blocked and are carefully working on a recovery, so that catching up doesn't trigger a burst of delayed actions. We'll share another update as soon as we have more information.
Investigating
We are investigating an issue where some thread events are being processed late. Affected threads may not update as expected, for example a thread may not be assigned or reopened, or an SLA timer may not stop when a reply is sent. Messages are still being received. We are working to identify the cause and will share an update as soon as we have more information.