At 6:00 AM Pacific Time, a bad query caused unexpected, extreme load on a critical database system. This led to almost all messages being sent to the inbound filtering system to receive the a ‘451’ temporary failure response. By 2:12 PM Pacific Time, all messages had been delivered from our queues and the system was operating normally.
Email delivery latency is back within normal operating parameters.
Posted Sep 15, 2020 - 14:29 PDT
Update
All the mails have now been drained out from our internal queues
Posted Sep 15, 2020 - 14:12 PDT
Update
We are continuing to monitor for any further issues.
Posted Sep 15, 2020 - 13:47 PDT
Update
Messages are now draining from our internal queues and customers should see messages that were delayed hitting their inboxes now.
Posted Sep 15, 2020 - 13:02 PDT
Update
Delays continue as we work on rolling out a fix. Customer logs will show that we are taking in email to our queues; however, messages are not being delayed quickly to customer mail servers. We are working diligently to get that email flowing so that queues can drain and customers can receive email.
Posted Sep 15, 2020 - 12:38 PDT
Update
We continue to monitor the situation. Mail is now being taken into the service at normal rates. Customers should expect continued delays as the internet re-delivers backlogged messages into the service over the coming hours.
Posted Sep 15, 2020 - 11:49 PDT
Monitoring
We have deployed a change while increasing our capacity by many times to relieve backpressure from queues. Inbound email is flowing again. We will continue to monitor the situation until we are certain that parameters are within normal operating ranges.
Posted Sep 15, 2020 - 10:40 PDT
Update
Please note that a temporary configuration change resulted in some inbound emails being rejected with 5xx error responses.
Posted Sep 15, 2020 - 10:20 PDT
Update
Please note that we will conduct a full post-mortem analysis in the coming day or two and will make changes to our configurations and software as needed to prevent this situation from recurring. We apologize to our valued customers for the delays to email and related issues accessing our console.
Posted Sep 15, 2020 - 10:09 PDT
Update
We are continuing to investigate this issue and have deployed additional technical resources and capacity. We appreciate your patience while we continue to do whatever is necessary to resolve the delays and get email flowing again.
Posted Sep 15, 2020 - 09:26 PDT
Identified
An unexpected increase in database load is causing some inbound messages to be temporarily deferred. We have identified the issue and are working to resolve it.
Posted Sep 15, 2020 - 07:40 PDT
This incident affected: Inbound Filtering (SMTP Service, Domain Console).