Certain external APIs and Admin services slow/unresponsive

Incident Report for Chameleon

Resolved

This incident has been resolved and response times/rates are back within normal range. Any tracked Events that arrived during this time were queued and are currently processing (eta is 20 minutes to be back to real time). The team will add any additional information to this incident if there is anything new to report.

Posted Feb 25, 2021 - 11:12 PST

Monitoring

We are continuing to monitor this incident. The main cause has a work around in place but the team are still seeing elevated error rates and things return to normal. Will update again in approximately 20 minutes

Posted Feb 25, 2021 - 10:50 PST

Identified

A datastore responsible for rate-limiting and queueing is responding much slower than normal causing slow responses to API requests both internal and external. Will update shortly with more information.

Posted Feb 25, 2021 - 10:21 PST

This incident affected: Sidebar API and Data API.