Overview

On February 26th between 19:14 UTC and 19:29 UTC, customers in our AU environment experienced elevated error rates when calling all Auth0 services (Authentication API, Management API, Dashboard).

We would like to apologize for the inconvenience and explain what caused this outage and what measures we are taking to prevent this from happening again.

What Happened

As part of our continuous effort to improve our security, we are working on improving our main database auditing capabilities. This requires an upgrade of all nodes in all MongoDB replica sets. We execute this procedure one environment at a time, one database instance at a time during low traffic hours in for the specific environment.

On February 26th at 18:58:02 UTC we upgraded secondary nodes in the AU environment. As part of the procedure, we step down the primary gracefully so that the replica set can pick another primary before we upgrade the node. After a successful upgrade on the secondary nodes, at 19:14:06 UTC we stepped down the primary node.

Normally, an election takes a few seconds. This time, the election took around a minute and a new primary was elected at 19:15:06 UTC. As we observed that a primary wasn’t being elected, we decided to stop the MongoDB service on the node in an attempt to force an election. We did this at at 19:15:08 UTC, roughly 2 seconds after a node had been elected primary (there was a race condition between us checking that no primary had been elected and when we restarted the node). The election took 60 seconds because of an outdated replicaset configuration. By default the time limit in milliseconds for detecting when a replica set’s primary is unreachable for a MongoDB replicaset is 10 seconds but in this case it was set to 60 seconds due to how our former infrastructure was set up and had not been updated since.

Once the primary was elected, majority of our services came back healthy at 19:16 UTC However, some of our services did not. When we noticed this we started investigating, and we realized that these services couldn't pull their configuration from our secrets datastore, as the the read limit quota was reached. We believe this was due to services retrying reads during the time Mongo was electing a primary. We increased read capacity manually at 19:29:37 UTC and restarted the services that were down. At 19:30:31 UTC, all services were back online.

Timeline

• 18:58:02 UTC - Started the rolling upgrade to MongoDB replicaset in AU.
• 19:14:06 UTC - MongoDB Primary node manually stepped down.
• 19:15:06 UTC - MongoDB runs an election and chooses a primary.
• 19:15:08 UTC - MongoDB service is manually stopped on the node in an attempt to force an election.
• 19:15:59 UTC - MongoDB service is started on the node and the node joins the replicaset.
• 19:16:36 UTC - MongoDB election process starts.
• 19:16:59 UTC - A primary is selected and majority of services are back online.
• 19:29:37 UTC - Read capacity on datastore is increased and services are restarted.
• 19:30:31 UTC - All services are back online.

What Are We Doing About It?

• Making sure our replicaset configuration is updated in every environment and check it before each update.
• We are updating our procedures to review database configuration before applying any change.
• We are implementing caching and exponential backoff in our secret storage client to prevent bursts from consuming all available quota
• We are looking to implement auto-scaling for our secret datastore so that our services don’t fail due to reaching a limit.
• We are working to implement better monitoring for our services so that we are alerted when they don’t start and why.

Summary

We would like to apologize for the impact this incident had on you and your customers. We use this opportunity to continue to improve our services. Thank you for choosing and supporting Auth0.