On June 20th, between 00:40 UTC and 06:02 UTC, some customers reported issues in which they were having HTTP 503 responses when downloading assets from the CDN in our EU region. Around 40 minutes after the first report, we fixed the issue by restarting Varnish. Approximately 7.83% of CDN requests in the EU region failed during this period.

What happened

Our CDN at this time used nginx as a proxy, with Varnish for caching, and S3 as the backend. Around 00:40 UTC, one of our Varnish instances lost its connection to the S3 backend and started returning an error to nginx, which then propagated this error to customers as an HTTP 503 response.

Due to a monitoring failure, we didn't detect this automatically; after our support team received a report, the incident response team was called to investigate. Upon investigation, we found the faulty node and proceeded to fix it by restarting Varnish, which resolved the incident.

Timeline

• 00:40 UTC: First error occurred
• 05:55 UTC: Incident response team was paged and began research
• 06:02 UTC: Restarted Varnish, which stopped the errors and resolved the incident

What are we doing about it?

• We improved monitoring of our CDN resources.
• We replaced our custom CDN with an Amazon CloudFront alternative. More details below.

Summary

The existing Auth0 CDN service was one of our older services. It had been built and maintained internally since the early days of the company. To improve its scaling and availability, we changed providers to use Amazon CloudFront on July 12 for customers in Europe and Australia.

We are sorry about this issue. This change, and the addition of better monitoring, can help us prevent similar situations in the future.

Thank you for your understanding and your continued support of Auth0.