On 6 September 2018 between 7:01am UTC and 11:16am UTC, approximately 80% of authentication requests using the Github strategy failed. The errors were caused by timeouts attempting to connect to Github’s API.
We would like to apologize for the impact this had on you and your customers and explain what caused the problem, how we reacted to the incident, and what we are doing to prevent incidents like this from happening in the future.
A customer raised a support ticket via Zendesk stating that users cannot login with Github
A customer raised a support ticket via Zendesk to inform us that their users were unable to login with Github. Our Technical Support team confirmed with the customer that the issue affected approximately 95% of their users attempting to login with Github. Technical Support further investigated the issue by examining logs, and found an increasing number of errors for multiple customers in the US region for requests to Auth0’s login/callback endpoint (called during the authentication process). Technical Support escalated the issue to our Engineering team.
Our Engineering team investigated and contacted Github
Initial investigation indicated that the failure was with the Github API but with their status page reporting all systems normal, our Engineering team looked into other possible causes to rule out issues with Auth0’s services.
With no evidence of issues with Auth0’s services, we contacted Github support with details of the failed requests to Github’s API, including the IP addresses that Auth0 attempted to connect to.
Our investigation highlighted the need to improve our monitoring systems so that in future we can improve our response time to authentication strategy errors.
Github’s response and confirmation of the cause of the incident
Github support responded to state that their data center had investigated an issue during the time window that Auth0 customers could not login with Github. This investigation identified some packet loss on the way out of AWS. Github stated that they have setup additional monitors to alert them to similar issues in the future.
This incident has been resolved.
We are no longer seeing errors from authentication transactions using the Github strategy. We believe the issue was caused by a change made by Github. We are confirming this with Github.
A number of authentication transactions using the Github strategy are failing to process correctly. The team is currently investigating the root cause. We will keep you updated.