Inbound filtering delays

Incident Report for MailChannels

Postmortem

This morning, at around 5:00 AM Pacific Time, we experienced an unusually large number of connections to the database which stores the list of domains which are provisioned to use our inbound filtering service. This database is queried whenever a sender tries to deliver a message to our inbound mail servers. It became overloaded, and could not respond quickly to queries, which resulted in senders receiving a 451 SMTP error response.

We responded to the incident by adding additional database capacity, and restricting the number of concurrent connections that each client application could make. By 7:30 AM Pacific Time, mail traffic was flowing normally.

We are continuing to investigate the root cause of this issue. We will update this post-mortem when we have more information.

Posted Sep 09, 2020 - 15:31 PDT

Resolved

The issue is now resolved. We will post a post-mortem shortly.

Posted Sep 09, 2020 - 07:54 PDT

Update

Listing domains within the web console is now fully functional. Email is being processed quickly again and queues are draining.

Posted Sep 09, 2020 - 07:43 PDT

Update

We are continuing to monitor for any further issues.

Posted Sep 09, 2020 - 07:42 PDT

Monitoring

We've added sufficient capacity to deal with the surge. Delivery metrics are now stabilizing.

Posted Sep 09, 2020 - 07:34 PDT

Update

Please note that no email is being lost. While email may be delayed, senders will queue and retry later. We appreciate how frustrating any delay in email delivery is.

Posted Sep 09, 2020 - 07:23 PDT

Update

An extreme surge in load on our internal licensing service caused a slowdown in domain routing lookups. We are adding capacity while we investigate the cause of the increased load.

Posted Sep 09, 2020 - 07:22 PDT

Investigating

Some customers are reporting delays in receiving email via the MailChannels Inbound service. Fetching the list of customer domains within the web console is also delayed or down.

Posted Sep 09, 2020 - 07:05 PDT

This incident affected: Inbound Filtering (SMTP Service, Domain Console).