Increased errors in Auth0
Current Status: Resolved | Last updated at July 1, 2021, 21:11 UTC
Affected environments: US-1 Preview, US-1 Production
We have posted the final update to our detailed root-cause analysis for our service disruption on April 20, 2021.
This update includes:
– Completion of the remaining six action items
https://cdn.auth0.com/blog/Detailed_Root_Cause_Analysis_(RCA)_4-2021.pdf
History for this incident
Resolved
We have now restored the affected regions of @auth0 for US-1, Support Center, and the Auth0 Dashboard. The root cause has been identified and is related to specific queries that created resource contention and impacted database performance. We will also provide our customers with a detailed Root Cause Analysis (RCA) within the next 12 days or sooner. We sincerely apologize to all affected customers and will post additional follow-ups here.
Monitoring
We are continuing to closely watch performance in US-1.
Monitoring
Based on the sustained performance for Auth0 US-1, Support Center, and the Auth0 Dashboard, we have now updated our status to monitoring. We are continuing to closely watch performance in US-1. We will now be moving to hourly updates.
Investigating
We have now restored the affected regions of @auth0 for US-1, Support Center, and the management dashboard. Our User Search v3 service has been enabled and the data is catching up. We will update here once all data has been brought up to current.
Investigating
We continue to see performance improvements. We are working to fully restore all services to customers in our US-1 region.
Investigating
We continue to see additional customers that have moved to degraded performance. Our User Search v3 service is currently disabled, which can generate stale data when using `/api/v2/users` endpoints. Once the service is enabled again, all data will be brought up to current.
Investigating
We continue to see customers that have moved to degraded performance. Systems are recovering and access to the Auth0 Dashboard has been restored. We are continuing to dedicate our full team's efforts on restoring services for all customers impacted by this incident.
Investigating
We are seeing customers in US-1 Production moving to degraded performance. We are continuing with all efforts to fully restore services for all customers.
Investigating
We are continuing to work on restoring services for our outage. We can communicate that users who are logged in are not impacted by this incident. We are executing all remediation steps for our incident protocol. Our entire technical and engineering teams are taking this as an all hands on deck situation to find resolution.
Investigating
We are continuing to work on restoring services as quickly as possible. As soon as we have an ETA for the restoration of services, we will update our status.
Investigating
We are continuing to work on restoring services as quickly as possible. We understand that this is causing a significant impact on your business.
Investigating
We are continuing to work on restoring services as quickly as possible. We understand that this is causing a significant impact on your business.
Investigating
We are continuing to investigate the root cause and resolution of the issue at hand.
Investigating
We continue to look into the root cause of this particular issue. Our Engineers are taking this as an all hands on deck situation.
Investigating
We have identified that this issue is coming from our database. However, we are still tracking down the root cause of the issue. We understand that this is an issue impacting the entirety of the environment, and the full Auth0 team is engaged in resolving this.
Investigating
We are continuing to work on this incident and we are also looking into ways to reduce the impact this is generating. At this time, we are still unclear as to the root cause of the issue. We understand that this is causing a significant impact on your business, and we thank you for your patience. We are doing everything we can to restore service.
Investigating
Our Engineers have identified some patterns in our databases that we believe could be related to this issue. We are attempting to leverage them to identify the issue
Investigating
A set of our customer base has reported being unable to access our Support Center as a result of this issue. If you need to file a ticket, we have enabled a temporary support route, which can be accessed by emailing support-backup@auth0.com
Investigating
We have received some reports of status.auth0.com being inaccessible by a small subset of customers. As a reminder, you can also follow the status updates on twitter.com/auth0status.
Investigating
Our Engineers and on-call teams continue to investigate this set of issues.
Investigating
We are currently experiencing an increased error rate in Auth0 services. Our Engineering team is investigating, and we will provide you with further updates as we have them.