Indexing delays on Search API for Free accounts
Current Status: Resolved | Last updated at September 8, 2017, 14:20 UTC
Affected environments: US-1 Preview, US-1 Production
History for this incident
Resolved
The Search API is responding well and we are going to close the incident. To summarize the status of things: - Customers who are on paid plans (until August 31st) that had LESS than 100 unique fields of metadata were migrated as-is. You will be able to continue to search by those fields in metadata as you were before. In the near term we will introduce limits to avoid introducing more arbitrary fields for these accounts. The end goal is to keep the cluster stable. We will let you know when that happens. - Customers who are on paid plans (until August 31st) that had MORE than 100 unique fields of metadata were migrated analyzing case by case. For some of them we created special indexes according to their search patterns. - Customers who are on free account who had active usage during the month of August were migrated to the new cluster with a restricted set of metadata indexes (see below**). If your application was relying on certain metadata queries, please contact us to searchapi-issues@auth0.com and we will do our best to accommodate your request. In some cases that might mean moving you back to the old cluster with the caveat that we cannot provide uptime guarantees for that. In the near term we might introduce certain fields to solve common use cases across all customers. Open issues we identified. - The Search API returns 503 when paging through more than 10,000 user records This is typically being used for user export. We are working on providing a longer term solution to user export that can scale and doesn't stress the Search cluster. In the meantime you can open a support ticket if you need your users exported. - Sorting (`sort=...`) is not working but we are working on a fix that should be deployed in the following days. - If you were doing a search like this: `q=user.app_metadata.approved:false` you will have to drop the `user`. Should look like: `q=app_metadata.approved:false` - Search that includes a space like `q=email:"foo "` might fail. We are investigating the reason and will fix. Thanks for your patience again while we dealt with this issue. We will publish a postmortem as we said, once things are settled. ** Normalized profile indexes available: `user_id`, `name`, `username`, `email`, `email_verified`, `phone_number`, `phone_number_verified`, `nickname`, `created_at`, `updated_at`, `family_name`, `give_name`, `last_ip`, `last_login`, `logins_count`, `blocked`, `identities.connection`, `identities.isSocial`, `identities.provider`, `identities.user_id`, `identities.profileData.email`, `identities.profileData.email_verified`, `identities.profileData.phone_number`, `identities.profileData.phone_verified`, `identities.profileData.phone_username`, `user_metadata.name`, `user_metadata.nickname`, `user_metadata.given_name`, `user_metadata.family_name`, `user_metadata.picture`.
Monitoring
We have finished the migration of free accounts to the new cluster and it is responding well. In order to protect the cluster, we have restricted the amount of indexes for these accounts. You can search based on the normalized profile fields: `user_id`, `name`, `username`, `email`, `email_verified`, `phone_number`, `phone_number_verified`, `nickname`, `created_at`, `updated_at`, `family_name`, `give_name`, `last_ip`, `last_login`, `logins_count`, `blocked`, `identities.connection`, `identities.isSocial`, `identities.provider`, `identities.user_id`, `identities.profileData.email`, `identities.profileData.email_verified`, `identities.profileData.phone_number`, `identities.profileData.phone_verified`, `identities.profileData.phone_username`, `user_metadata.name`, `user_metadata.nickname`, `user_metadata.given_name`, `user_metadata.family_name`, `user_metadata.picture`. If you were relying on certain query pattern that is not included here, please reach out to searchapi-issues@auth0.com explaining your use case. We have identified an issue in the new cluster when paging through more than 100 pages. The Export users extension uses that and we are looking for alternatives to solve this. If you need to export users before we come up with the final solution, please reach out via support with subject "Export Users Request". We will continue to monitor and close the incident during the day. Thanks for your patience again, we will publish a postmortem as soon as possible.
Identified
We continue to make progress. During the weekend we detected an issue in the way we indexed certain fields and we had to re-index some of the accounts, that delayed the whole process. We've also spent significant time working with a handful of customers who had thousands of attributes stored in metadata. We had to go case by case and that also took time. We continue to migrate free accounts in the meantime, we have done 35% of them. Note that logins were unaffected throughout the whole migration and the Search API was always working, but was not indexing new users/updated users. The team is working 24/7 on bringing back the fully functional Search API for all users. Thanks for your patience while we work on this. Please don't hesitate to contact support for any question.
Identified
All paid accounts were migrated at this point, except for the a handful that have an excessive amount of indexed fields and data who are being followed up individually. The new cluster is responding well. We will continue with free accounts now.
Identified
Update: we have migrated most of the production accounts. We will continue with free accounts next. If you experience any issue please contact us via support or twitter.
Identified
Update: we continue to make progress on the migration, things are running smoothly and we are 50% towards the full migration.
Identified
Update: we have migrated 25% of the user base and the new cluster is handling the load well. We will continue to migrate throughout the day and provide updates here. If you have an urgent request to prioritize, please contact support through support.auth0.com. Thanks for your patience.
Identified
Update on the Search API: - Our Search cluster is healthy but we can't ingest more data, since every update triggers indexing and that gets the cluster into an unhealthy state. That means that if you use the Search API or the Dashboard you will see information updated until 7:40PM UTC approximately. - This should not affect your logins, unless you are using the Search API as part of it. If that's the case you might want to check your implementation to avoid using Search. For e.g., if you use a rule that link accounts which uses search you can disable that rule. - We are moving customer accounts as fast as we can to a new cluster. We are prioritizing first production accounts. This cluster runs a newer version of the engine along with other improvements and also has some restrictions to avoid getting into the same situation. We have teams all hands on deck working on this. Thanks for understanding.
Identified
We are experiencing an indexing delay in users for search. Should not affect authentication for customers not using search in authentication rules.
Identified
We are seeing 2-3% of the Search API calls failing due to capacity issues. Authentication transactions are unaffected. We are working hard on this and expect to have a permanent fix soon. If you are using a rule to merge accounts, that uses the Search API, disable it temporarily so it doesn’t affect your logins.