Performance Degradation for the Dashboard in US-4 environment
Current Status: Resolved | Last updated at September 27, 2023, 19:30 UTC
Affected environments: US-4 Production
History for this incident
Resolved
This incident has now been resolved, and service has been restored. We have identified minor misconfigurations related to the platform migration for US-4 which impacted the Management Dashboard internal service operations. While this resulted in failure calls with 5xx and 429s responses when attempting to retrieve tenant information from that environment, we confirmed that Authentication was not impacted. We apologize for the impact this has had on you and your users.
Monitoring
We applied a fix for the issue and are monitoring the system.
Identified
We saw a spike in error rates that has impacted a subset of customers in our US-4 environment. We confirmed that impacted users would have experienced degraded functionality or saw some 5xx and 4xx error codes when accessing Manage Dashboard. Authentication is not affected. Our team is rolling out a fix that may take ~90 minutes to complete. We will provide an update once it's implemented.
Identified
We've identified the root cause and are rolling out a fix. Next update will be in 30 minutes.
Investigating
We are seeing degraded Dashboard performance for the impacted environments and are investigating the cause.