History

Elevated error rate in logins

Elevated error rate in logins

Current Status: Resolved | Last updated at October 8th 2021, 22:34 UTC

Affected environments: US-1 Preview, EU Production, US-1 Production, EU Preview

A root-cause analysis for this issue has been performed and is now available at https://cdn.auth0.com/blog/2021-09-24&2021-09-28_Extensibility_Service_Disruption_in_US-1_and_EU.pdf


History for this incident

September 28, 202116:43 UTC

Resolved

Service has been restored. A root cause analysis (RCA) will be posted here within 14 days. We sincerely apologize for the impact this had on you and your users.

September 28, 202116:24 UTC

Monitoring

System performance has returned to normal and we are continuing to monitor. Barring any changes in performance, our next update will be in 30 minutes.

September 28, 202116:08 UTC

Identified

We identified the root cause and applied the fix to both EU and US-1 regions. We will continue to monitor both environments. Next update will be within 30 minutes.

September 28, 202115:52 UTC

Investigating

EU is recovering and error rates are less than 1%. US-1 is still experiencing partial outage. Our team is working on mitigation actions and estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.

September 28, 202115:36 UTC

Investigating

Our US-1 region began experiencing similar issues at 15:21 and has been added to this incident. Our team is working on mitigation actions but we do not have an estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.

September 28, 202115:20 UTC

Identified

We have isolated the problem to our extensibility service. We are seeing ~8% of requests resulting in errors or timeouts, with very high latency for many other requests. Our team is working on mitigation actions but we do not have an estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.

September 28, 202115:05 UTC

Identified

We have isolated the problem to our extensibility service. Our team is working on mitigation actions. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.

September 28, 202114:49 UTC

Identified

We have isolated the problem to our extensibility service. Our team is working on mitigation actions. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes.

September 28, 202114:30 UTC

Identified

Our Engineers have identified the issue and are continuing to work towards resolution. We will provide our next update in 15 minutes.

September 28, 202114:15 UTC

Identified

We have isolated the problem to our extensibility service. Our team is working on mitigation actions. We apologize for the impact this is having on you and your users. Our next update will be in 15 minutes.

September 28, 202113:46 UTC

Identified

Ongoing issues have been identified and investigation is continuing

September 28, 202113:36 UTC

Monitoring

Engineering have applied a fix and are monitoring the environment.

September 28, 202113:25 UTC

Investigating

We are continuing to investigate this issue.

September 28, 202113:19 UTC

Investigating

We are experiencing an elevated error rate in logins for our EU region. Our Engineers are investigating currently.