All of our metrics, mainly SMTP connection delay and SMTP round trip delivery time, have returned to normal as of 18:00 UTC. This means that if there are any emails backed up in the server queues, they should start delivering. Customers may continue to receive complaints from users about delivery delays because messages may still be leaving the queue as per the exponential back-off rules of the SMTP servers. However, new messages introduced into the service will be delivered immediately. One option is to flush the queues.
The root cause has been identified as a failure of certain auto-scaling functionality within the service that failed to adapt service scale to a surge of incoming email. We will provide a more detailed analysis when this incident is fully resolved.
Posted Apr 22, 2024 - 11:04 PDT
Update
Legitimate emails backed up in our queues have been delivered. We are still seeing increased latency at the ingress side. We are continuing to investigate.
Posted Apr 22, 2024 - 08:16 PDT
Update
Message delivery is slow for a small percentage of traffic. We are trying to clear out all the messages in our queues as fast as possible.
Posted Apr 22, 2024 - 05:39 PDT
Update
We are continuing to investigate the issue.
Posted Apr 22, 2024 - 05:25 PDT
Update
We are continuing to investigate the issue.
Posted Apr 22, 2024 - 02:29 PDT
Investigating
Severity Level: 3
We are currently investigating higher than normal connection latency that might have an impact on email delivery services.
Posted Apr 21, 2024 - 23:44 PDT
This incident affected: Outbound Filtering (SMTP Service).