AU region expired certificate causing outage

Root Cause Analysis - Australian Certificate Renewal Failure - April 15, 2019

Root Cause Analysis

Customers in our Australian region started to observe failures on their requests due to an expired certificate, starting 2019-04-15T23:59:59UTC.

While all production environments have auto-renewal certificates on the ACM service from AWS, the Australian production environment was still using an imported Certificate from our previous DNS provider, whose use was deprecated in 2018.

While our other regions had already migrated to ACM, Auth0 recognised potential impact to a subset of customers who had yet to migrate away from unsupported use cases should the Australia certificates be migrated at the time. Therefore, the change was delayed, and deprecation of the legacy certificate manager resulted in Auth0 not being alerted when the certificate was due to expire.

A new ACM certificate was deployed to all AUS Load-Balancers which resolved the issue.

Why this won’t happen again

The Australian region has been updated to use our new Certificate Service. Though Auth0 believes all certificates are centrally managed and set to auto-renew, it is in the process of auditing all existing certificates to ensure 100% compliance, and expects the project to be complete by April 27th.

Timeline

00:00 AM UTC - Auth0 *.au.auth0.com domain certificate expired.

00:10 AM UTC - Auth0 Technical Support was alerted of the issue, engaged the operations team to resolve, and posted to the Auth0 Status Page.

00:20 AM UTC - Certificates were renewed and replaced in load balancers.

00:23 AM UTC - Service was restored.