Between September 30th and October 2nd, our Outbound Filtering service periodically became overloaded because of a coordinated attack on our infrastructure by automated spam bots. The overloading condition caused delays for some customers. We have now identified the root cause for these delays and have made changes to prevent the same situation from recurring.
A flood of invalid authentication attempts from automated spam bots placed unexpected, sudden load on our SMTP service. One of our internal components responded to the additional load by artificially slowing down its response rate to incoming connections. This “load shedding” behaviour led to client queues growing as incoming SMTP connections piled up waiting for service.
Note that MailChannels is designed to scale automatically and, in most cases, a sudden increase in requests will result in additional capacity being brought online within seconds.
What have we done to mitigate this?
We have written new policies to detect and thwart the bot attacks before they can overwhelm our authentication capacity. We have deployed more resources (database, MTA) so that we can handle these additional spurious requests without delaying legitimate authentication requests. Previously, we were permanently rejecting the connections in case of the authentication failure. We have now made these temporary rejections so that the mail server can retry sending the emails.
We have also improved our monitoring around the authentication issue and the number of connections. Our infrastructure auto-scales resources based on a number of parameters. We have added additional parameters so that the back-end MTA’s will auto-scale based on the number of connections before they go into stress mode.
In the near future, we will add automated updates to the status page if we identify any issues with the authentication infrastructure.
We would like to reassure our customers that we are constantly working on new capabilities so that these issues do not happen again.
If you have any concerns, please do not hesitate to reach out to us.