History

Increased errors in Auth0

Increased errors in Auth0

Current Status: Resolved | Last updated at July 1st 2021, 21:11 UTC

Affected environments: US-1 Production, US-1 Preview

We have posted the final update to our detailed root-cause analysis for our service disruption on April 20, 2021.

This update includes:

– Completion of the remaining six action items

https://cdn.auth0.com/blog/Detailed_Root_Cause_Analysis_(RCA)_4-2021.pdf


History for this incident

April 20, 202123:53 UTC

Resolved

We have now restored the affected regions of @auth0 for US-1, Support Center, and the Auth0 Dashboard. The root cause has been identified and is related to specific queries that created resource contention and impacted database performance. We will also provide our customers with a detailed Root Cause Analysis (RCA) within the next 12 days or sooner. We sincerely apologize to all affected customers and will post additional follow-ups here.

April 20, 202123:05 UTC

Monitoring

We are continuing to closely watch performance in US-1.

April 20, 202121:39 UTC

Monitoring

Based on the sustained performance for Auth0 US-1, Support Center, and the Auth0 Dashboard, we have now updated our status to monitoring. We are continuing to closely watch performance in US-1. We will now be moving to hourly updates.

April 20, 202121:23 UTC

Investigating

We have now restored the affected regions of @auth0 for US-1, Support Center, and the management dashboard. Our User Search v3 service has been enabled and the data is catching up. We will update here once all data has been brought up to current.

April 20, 202120:52 UTC

Investigating

We continue to see performance improvements. We are working to fully restore all services to customers in our US-1 region.

April 20, 202120:38 UTC

Investigating

We continue to see additional customers that have moved to degraded performance. Our User Search v3 service is currently disabled, which can generate stale data when using `/api/v2/users` endpoints. Once the service is enabled again, all data will be brought up to current.

April 20, 202120:18 UTC

Investigating

We continue to see customers that have moved to degraded performance. Systems are recovering and access to the Auth0 Dashboard has been restored. We are continuing to dedicate our full team's efforts on restoring services for all customers impacted by this incident.

April 20, 202120:05 UTC

Investigating

We are seeing customers in US-1 Production moving to degraded performance. We are continuing with all efforts to fully restore services for all customers.

April 20, 202119:44 UTC

Investigating

We are continuing to work on restoring services for our outage. We can communicate that users who are logged in are not impacted by this incident. We are executing all remediation steps for our incident protocol. Our entire technical and engineering teams are taking this as an all hands on deck situation to find resolution.

April 20, 202119:25 UTC

Investigating

We are continuing to work on restoring services as quickly as possible. As soon as we have an ETA for the restoration of services, we will update our status.

April 20, 202119:08 UTC

Investigating

We are continuing to work on restoring services as quickly as possible. We understand that this is causing a significant impact on your business.

April 20, 202118:54 UTC

Investigating

We are continuing to work on restoring services as quickly as possible. We understand that this is causing a significant impact on your business.

April 20, 202118:22 UTC

Investigating

We are continuing to investigate the root cause and resolution of the issue at hand.

April 20, 202118:06 UTC

Investigating

We continue to look into the root cause of this particular issue. Our Engineers are taking this as an all hands on deck situation.

April 20, 202117:50 UTC

Investigating

We have identified that this issue is coming from our database. However, we are still tracking down the root cause of the issue. We understand that this is an issue impacting the entirety of the environment, and the full Auth0 team is engaged in resolving this.

April 20, 202117:31 UTC

Investigating

We are continuing to work on this incident and we are also looking into ways to reduce the impact this is generating. At this time, we are still unclear as to the root cause of the issue. We understand that this is causing a significant impact on your business, and we thank you for your patience. We are doing everything we can to restore service.

April 20, 202117:14 UTC

Investigating

Our Engineers have identified some patterns in our databases that we believe could be related to this issue. We are attempting to leverage them to identify the issue

April 20, 202116:41 UTC

Investigating

A set of our customer base has reported being unable to access our Support Center as a result of this issue. If you need to file a ticket, we have enabled a temporary support route, which can be accessed by emailing support-backup@auth0.com

April 20, 202116:20 UTC

Investigating

We have received some reports of status.auth0.com being inaccessible by a small subset of customers. As a reminder, you can also follow the status updates on twitter.com/auth0status.

April 20, 202115:59 UTC

Investigating

Our Engineers and on-call teams continue to investigate this set of issues.

April 20, 202115:43 UTC

Investigating

We are currently experiencing an increased error rate in Auth0 services. Our Engineering team is investigating, and we will provide you with further updates as we have them.