Elevated error rate in logins
Current Status: Resolved | Last updated at October 8, 2021, 22:34 UTC
Affected environments: US-1 Preview, EU-1 Production, US-1 Production, EU-1 Preview
A root-cause analysis for this issue has been performed and is now available at https://cdn.auth0.com/blog/2021-09-24&2021-09-28_Extensibility_Service_Disruption_in_US-1_and_EU.pdf
History for this incident
Resolved
Service has been restored. A root cause analysis (RCA) will be posted here within 14 days. We sincerely apologize for the impact this had on you and your users.
Monitoring
System performance has returned to normal and we are continuing to monitor. Barring any changes in performance, our next update will be in 30 minutes.
Identified
We identified the root cause and applied the fix to both EU and US-1 regions. We will continue to monitor both environments. Next update will be within 30 minutes.
Investigating
EU is recovering and error rates are less than 1%. US-1 is still experiencing partial outage. Our team is working on mitigation actions and estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.
Investigating
Our US-1 region began experiencing similar issues at 15:21 and has been added to this incident. Our team is working on mitigation actions but we do not have an estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.
Identified
We have isolated the problem to our extensibility service. We are seeing ~8% of requests resulting in errors or timeouts, with very high latency for many other requests. Our team is working on mitigation actions but we do not have an estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.
Identified
We have isolated the problem to our extensibility service. Our team is working on mitigation actions. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.
Identified
We have isolated the problem to our extensibility service. Our team is working on mitigation actions. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.
Identified
Our Engineers have identified the issue and are continuing to work towards resolution. We will provide our next update in 15 minutes.
Identified
We have isolated the problem to our extensibility service. Our team is working on mitigation actions. We apologize for the impact this is having on you and your users. Our next update will be in 15 minutes.
Identified
Ongoing issues have been identified and investigation is continuing
Monitoring
Engineering have applied a fix and are monitoring the environment.
Investigating
We are continuing to investigate this issue.
Investigating
We are experiencing an elevated error rate in logins for our EU region. Our Engineers are investigating currently.