[Feature Disruption] Tenant Logs Indexing Delays
Current Status: Resolved | Last updated at February 23, 2018, 19:15 UTC
Affected environments: EU-1 Production, EU-1 Preview
History for this incident
Resolved
The new elasticsearch cluster is handling traffic as expected, all metrics continue to be within healthy ranges. We're marking this incident as solved.
Monitoring
A new Elasticsearch cluster has been deployed and all data and traffic has been moved there. Log ingestion rate is back to normal, as the rest of our database metrics. Queries to the logs endpoints and dashboard log listings are showing the last results.
Identified
The Elastic Search cluster we use in the EU environment to index account logs to enable their search is ingesting slow at a slow rate. The database has reached a large state size, which is making all operations either slow or fail. No data has been lost. Our team is working on creating a new clean cluster, but it might take some time to resolve this. In the meantime, queries to the logs endpoints and dashboard log listings will show outdated results.
Investigating
Our tenant logs index processing pipeline is running behind. Queries to the Search API will return stale results until the system is caught up. We will post an update with an ETA as soon as possible.