This incident is now resolved. Our key metrics are back to normal and all queued jobs have been processed.
Posted Jun 15, 2023 - 10:51 PDT
Monitoring
We're seeing timeouts on approximately 4% of requests coming into Chameleon with the url prefix of https://fast.chameleon.io/observe/ and are still working to get back to fully operational
Posted Jun 15, 2023 - 10:28 PDT
Identified
We've identified a problem in how Heroku handles DNS resolution from Private spaces and are in contact with the support team. A large percentage of our servers are connecting to our database properly but the rest are failing to resolve the cluster address down to the addresses of the cluster instances. We're working on a possible workaround at the moment
Posted Jun 15, 2023 - 08:54 PDT
Investigating
We're seeing elevated error rates and retries related to processing Events and updates to Company properties. We're investigating the source of the problem and will report back here with more information.