Android Guardian MFA Push Notification Failures in US-1
Current Status: Resolved | Last updated at November 3, 2023, 21:14 UTC
Affected environments: US-1 Production, US-1 Preview
History for this incident
Resolved
This issue has now been resolved for all impacted customers and MFA push notifications are now functioning as expected. Some customers using the Guardian app may need to update their application as we have applied a software patch for this issue and we are seeing full recovery with services functioning as expected.
Identified
Our Engineering team is now simultaneously working on a software configuration patch to correct this issue and also working with our vendor to support a more immediate remediation plan. At most, we expect this issue to be resolved within several hours once our software patch is released. Impact is limited to end users utilizing MFA for Guardian on a subset of devices and affects a limited number of use cases. We will confirm here once services are fully restored and this issue has been resolved.
Identified
We are currently performing steps to verify a fix for what we believe to have been the root cause of this issue. We will post additional updates here within 45 minutes.
Identified
We continue to work with the configurations identified which we believe are related to the root cause of this issue. We will post additional updates here 45 minutes as we work on determining any necessary mitigation actions.
Identified
We are now reviewing additional configurations related to the errors identified. Multiple Engineering resources are engaged on this triage step and we're working urgently to determine if additional mitigation actions are required.
Identified
We have identified MFA API errors of significance while reviewing our logs. We continue to review this for root cause and our Engineering team are working through our logs urgently to determine root cause and any necessary remediation steps
Investigating
We are currently reviewing any recent changes made within the environment, however we do not yet have a root cause at this time as part of our investigation. We are working urgently on remediation for this issue and we will provide additional updates here within 15 minutes. We are continuing to triage this and review with additional Engineering resources.
Investigating
We are investigating a spike in error rates which impacted some customers in our US-1 Public Cloud environment. Customers may notice an issue with MFA push notifications and our Engineering team is urgently investigating this issue.