tag:status.chameleon.io,2005:/historyChameleon Status - Incident History2024-03-28T20:55:02-07:00Chameleontag:status.chameleon.io,2005:Incident/201873122024-03-07T11:00:00-08:002024-03-07T10:49:03-08:00Microsurvey summaries delayed<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>11:00</var> PST</small><br><strong>Resolved</strong> - Some Microsurvey summaries from the previous 7 days were delayed and but will be re-run and delivered over the next few hours</p>tag:status.chameleon.io,2005:Incident/199691042024-02-09T12:00:00-08:002024-02-12T11:39:37-08:00Snooze settings incident<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>12:00</var> PST</small><br><strong>Resolved</strong> - On Feb 9th, Chameleon determined that a small percentage of some customers' end-user were not able to dismiss a specific type of Chameleon Experience (a Tour or a Microsurvey) due to an issue the application of the Experiences' Snooze settings from a Snooze that first happened prior to Jan 30th. We quickly fixed the issue and migrated the relevant data to bring your users back to the expected state. If your users were impacted by this, you will receive an email from our customer success team with more details.</p>tag:status.chameleon.io,2005:Incident/199239692024-02-07T10:39:31-08:002024-02-07T10:39:31-08:00Redis instance connection timeouts (2)<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>10:39</var> PST</small><br><strong>Resolved</strong> - We're going to close this incident as Chameleon has now been operating at full capacity for the last 12 hours. If you have any questions about this or anything else Chameleon-related please reach out to Chameleon support.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>17:28</var> PST</small><br><strong>Monitoring</strong> - We have increased the network capacity of our Redis cluster and we're in contact with the Heroku Redis team about the latency issues we're still intermittently experiencing. At this time, we're going to consider this incident closed but will continue to monitor until more time has passed without latency issues</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>10:21</var> PST</small><br><strong>Investigating</strong> - This is a repeat of the previous incident (https://status.chameleon.io/incidents/x12r73srmcg2). Please check back for updates as we will update when we have more information on when all of Chameleon's services are back up and running.<br /><br />Customers may experience issues opening Chameleon's Builder in their applications, exporting reports, or loading certain Chameleon Dashboard pages.</p>tag:status.chameleon.io,2005:Incident/199226752024-02-06T09:32:00-08:002024-02-06T09:34:35-08:00Redis instance connection timeouts<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>09:32</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved. The analytics data received for the time between Feb 6th, 15:31 UTC and Feb 6th, 16:28 UTC were not processed in real time. Our team in the process of re-running any data received during this time frame including experience analytics. Additionally many background jobs during this will also be re-run including data exports. If you do not receive your data export by the end of the day today please re-export the data or reach out to support</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>08:07</var> PST</small><br><strong>Monitoring</strong> - Our main redis instance had a failover to its secondary and is catching back up with traffic. This redis instance handles background processing and will take some time to catch back up with realtime.<br /><br />Customers may experience issues opening Chameleon's Builder in their applications, exporting reports, or loading certain Chameleon Dashboard pages.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>07:43</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.chameleon.io,2005:Incident/196795152024-01-11T08:30:00-08:002024-01-11T08:35:28-08:00Missing redirect from legacy domain (trychameleon.com, now chameleon.io)<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>08:30</var> PST</small><br><strong>Resolved</strong> - This issue caused in users who clicked the "Dashboard" button from the Chameleon website (https://www.chameleon.io) to be redirected to https://app.trychameleon.com (which then redirected to the Login page at https://www.chameleon.io/login) instead of directly to the Dashboard at https://app.chameleon.io and caused it to appear as through you were logged out. The incident lasted a ~6 hours and was fixed as of Jan 11 08:30 pacific time.</p>tag:status.chameleon.io,2005:Incident/193813992023-12-08T15:14:18-08:002023-12-08T15:14:18-08:00Chameleon sub-processors notification<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>15:14</var> PST</small><br><strong>Resolved</strong> - We are contacting you as per the terms of our Data Processing Agreement (DPA) to inform you of new sub-processors being implemented by Chameleon.<br /><br />At Chameleon, data privacy and security are a priority and we understand the importance of keeping your company's data secure. We have recently completed our third SOC 2 Type II audit and leverage a continuous security monitoring solution to maintain ongoing compliance.<br /><br />Name Data Location Description<br />OpenAI United States AI API platform<br />Pipedream United States API automation platform<br /><br />These sub-processors have been evaluated according to Chameleon’s third-party risk management process. If you'd like to know more about our privacy and security practices, please visit our security page (https://chameleon.io/security) or contact us at security@chameleon.io.<br /><br />For more information pertaining to GDPR compliance, check out: How Chameleon complies with GDPR (https://help.chameleon.io/en/articles/1632723-how-chameleon-complies-with-gdpr).<br /><br />Thanks,<br />Chameleon Security Team</p>tag:status.chameleon.io,2005:Incident/189338962023-10-26T05:03:35-07:002023-10-26T05:03:58-07:00HelpBar in GA<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>05:03</var> PDT</small><br><strong>Resolved</strong> - HelpBar.ai is now officially live and publicly available. We will be monitoring usage of the API as usage climbs. If you're interested in the launch and supporting it please visit https://www.producthunt.com/posts/helpbar-by-chameleon or www.helpbar.ai</p>tag:status.chameleon.io,2005:Incident/188430522024-02-13T07:38:12-08:002024-02-13T07:38:12-08:00Chameleon support for iframes using a different subdomain than parent page<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>07:38</var> PST</small><br><strong>Monitoring</strong> - There is now a beta version of Chameleon iFrame support available. This beta allows for Chameleon Tours and experiences to appear across iframes with different domains (both subdomains and primary domains).<br /><br />Please message Chameleon Support to request to be included in this beta.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>10:36</var> PDT</small><br><strong>Identified</strong> - Impact: If your use of Chameleon depends on selecting page elements that reside within iframes, those specific Chameleon Experiences may not not be running correctly.<br /><br />Working: Main page is app.acme.com and the iframe is app.acme.com<br />Not working: Main page is app.acme.com and the iframe is content.acme.com<br /><br />Further context is that prior to Chrome 115, iframes on the same origin (e.g. app.acme.com and content.acme.com) allowed their content is be accessible to the parent page. Chameleon uses this same-origin access pattern to find and attach Chameleon Experiences to the element found within an iframe. However, starting in Chrome 115, Google removed support for modifying a property called `document.domain` which permits this behavior. Now, because Google has disabled this feature, Chameleon cannot currently directly access content of iframes with subdomains that differ from the parent page.<br /><br />The Chameleon team is working quickly to restore full support for same-origin iframes. We plan to ship an early version of this to impacted customers in the coming days. If you are using iframes in the way described above, please reach out to your account manager or hello@chameleon.io let us know. Also indicate if you want to be added to the iframe feature preview when it's ready.<br /><br />Workaround: In the meantime, the workaround is to match the domain of the content inside an iframe with the content outside of the iframe (referenced as "Working:" above). Some teams we're in contact with have had success in simply proxying their iframed content through the same domain as their main page (let us know if you want more information on how to do this).</p>tag:status.chameleon.io,2005:Incident/183197532023-08-30T06:15:41-07:002023-08-30T06:26:32-07:00Chameleon Experience published status<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>06:15</var> PDT</small><br><strong>Resolved</strong> - Your live Experiences should now be correctly displayed with the Live label and should be running normally. As such we're marking this incident as resolved. The key impact, in addition to a currently live Experience displaying with the [Inactive] label, is that any Tours / Microsurveys that would have started for the first time during this time period ending 5:45am PDT were kept from starting out of an abundance of caution by our backend -- thanks for being patient while we worked on and deployed this fix.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>05:31</var> PDT</small><br><strong>Identified</strong> - We've identified an issue that is causing the display of Experiences as [Inactive] in the dashboard and are testing a fix at the moment. Will update when the fix is live.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>04:53</var> PDT</small><br><strong>Investigating</strong> - We're investigating reports of published Chameleon experiences showing as unpublished/Inactive in the Chameleon dashboard. Will update when we have more information</p>tag:status.chameleon.io,2005:Incident/177806082023-07-06T04:30:00-07:002023-07-06T07:21:57-07:00App + API slowness<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>04:30</var> PDT</small><br><strong>Resolved</strong> - For approximately 2.5 hours from 4:30am PDT to 7am PDT our App and API saw longer response times than normal. Operations that includes writes (vs read only) were impacted most.</p>tag:status.chameleon.io,2005:Incident/175835422023-06-15T10:51:00-07:002023-06-15T10:51:00-07:00Processing delays - Events and Companies<p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>10:51</var> PDT</small><br><strong>Resolved</strong> - This incident is now resolved. Our key metrics are back to normal and all queued jobs have been processed.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>10:28</var> PDT</small><br><strong>Monitoring</strong> - We're seeing timeouts on approximately 4% of requests coming into Chameleon with the url prefix of https://fast.chameleon.io/observe/ and are still working to get back to fully operational</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>08:54</var> PDT</small><br><strong>Identified</strong> - We've identified a problem in how Heroku handles DNS resolution from Private spaces and are in contact with the support team. A large percentage of our servers are connecting to our database properly but the rest are failing to resolve the cluster address down to the addresses of the cluster instances. We're working on a possible workaround at the moment</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>07:40</var> PDT</small><br><strong>Investigating</strong> - We're seeing elevated error rates and retries related to processing Events and updates to Company properties. We're investigating the source of the problem and will report back here with more information.</p>tag:status.chameleon.io,2005:Incident/175753402023-06-14T20:05:29-07:002023-06-14T20:05:29-07:00HelpBar search results error rates<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>20:05</var> PDT</small><br><strong>Resolved</strong> - We're marking this as resolved because the HelpBar search and DNS lookup are working as expected but we will plan to remove the temporary fix when Heroku gets to the bottom of the networking issue.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>14:30</var> PDT</small><br><strong>Monitoring</strong> - We've identified the issue as isolated to resolving ip DNS via IPv6. Our engineers have deployed a temporary fix while Heroku works on the resolution of this incident. As of now they have not posted on their status page (https://status.heroku.com/).</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>12:44</var> PDT</small><br><strong>Identified</strong> - HelpBar search results are seeing elevated error rates as some of the calls into our search service are not correctly resolving the DNS on the remote host. We're working with the Heroku team to identify the cause of this issue and will report back when we know more. In the meantime, we're keeping track of the hosts that are not correctly resolving the DNS and rotating them out of the Heroku cluster.</p>tag:status.chameleon.io,2005:Incident/171371172023-05-03T15:15:00-07:002023-05-03T15:54:01-07:00Elevated rates of error - Timeouts<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>15:15</var> PDT</small><br><strong>Resolved</strong> - We've resolved the second half of yesterday's incident which involved an single member of our application server cluster being reported as connectable and available to handle requests but internally the connections were being dropped by the heroku routing layer. This resulted in the a client-facing 503 error in approximately 4% of requests when contacting urls with the prefix https://fast.chameleon.io/observe/*.</p>tag:status.chameleon.io,2005:Incident/171199622023-05-02T09:53:46-07:002023-05-02T09:53:46-07:00Chameleon app / API slowdown<p><small>May <var data-var='date'> 2</var>, <var data-var='time'>09:53</var> PDT</small><br><strong>Resolved</strong> - While we're confident this incident is over for the immediate future, and while everything is catching back up (for a few types of data updates, mainly to companies) as everything returns to normal, we will be monitoring a few things more closely over the next few days.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>06:45</var> PDT</small><br><strong>Monitoring</strong> - This incident is now on it's way to resolution; from our testing and metrics the Chameleon dashboard (app.chameleon.io) is now loading and Experiences are being displayed (e.g. Launchers, Tours). We will provide a final update here when all of our metrics/signals have returned to normal.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>06:33</var> PDT</small><br><strong>Update</strong> - As an update to the ongoing issue we're experiencing, our database provider's team (MongoDB Atlas) has identified the underlying issue is actively working on the resolution. Chameleon, for the most part, will continue to be unavailable until this is resolved. The primary impact is that new Chameleon Experiences will not be displayed to your users and the Chameleon dashboard (app.chameleon.io) is not loading.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>06:03</var> PDT</small><br><strong>Identified</strong> - app.chameleon.io continues to timeout when loading -- the underlying cause appears to be a recent maintenance event at our MongoDB instance which has left it only partially available and are working with their team toward resolution</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>05:44</var> PDT</small><br><strong>Investigating</strong> - We're investigating a slowdown in serving API requests to Chameleon</p>tag:status.chameleon.io,2005:Incident/169849822023-04-27T21:30:00-07:002023-04-27T12:14:33-07:00Database failover causing slowness in Chameleon Builder<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>21:30</var> PDT</small><br><strong>Resolved</strong> - An unexpected increase in traffic to Chameleon's database servers caused a brief slowdown with some requests taking up to 30 seconds, and some requests timing-out. This caused issues when trying to access the Chameleon Builder inside customer apps or use interfaces inside the Chameleon customer dashboard.<br /><br />The issue resolved itself at approximately 9:30am PST, when Chameleon's database servers automatically detected the issue and adjusted server resources accordingly.</p>tag:status.chameleon.io,2005:Incident/166193422023-03-23T16:05:53-07:002023-03-23T16:05:53-07:00Builder upgrades rollout<p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>16:05</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>08:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>07:43</var> PDT</small><br><strong>Scheduled</strong> - Chameleon is launching a set of performance improvements, new integrations, and general updates across many of Chameleon's internal services during this window. We aim to minimize impact with an account-based staged rollout over the 24 hours of this maintenance window.<br /><br />Feel free to reach out to us a hello@chameleon.io (+ cc: brian@chameleon.io) with any quirks or issues you encounter and we'll be happy to take a look.</p>tag:status.chameleon.io,2005:Incident/158147312023-01-10T11:00:00-08:002023-01-17T13:11:28-08:00iFrame access errors<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>11:00</var> PST</small><br><strong>Resolved</strong> - At around 11am PST today the Chameleon team released an update to our iFrame support that was intended to provide better support for multiple-layers of iFrames that show Chameleon Tours. For a small number of customers that had iFrames configured with subdomain origins may have seen errors reported from the Chameleon script attempting to access iFrames improperly.<br /><br />The change was reverted at 2:53pm PST with the release of the new feature coming at a later date.</p>tag:status.chameleon.io,2005:Incident/157855332023-01-06T07:19:31-08:002023-01-06T07:19:31-08:00Schedule publishing delay<p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>07:19</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>07:08</var> PST</small><br><strong>Identified</strong> - We're restoring scheduled publishing / unpublishing to realtime and will be caught back up shortly</p>tag:status.chameleon.io,2005:Incident/153554232022-12-16T19:05:03-08:002022-12-17T21:09:24-08:00Email delivery delays<p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>19:05</var> PST</small><br><strong>Resolved</strong> - As of 7PM pacific time Friday night transactional email services was restored and the pending deliveries were delivered albeit delayed.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>14:34</var> PST</small><br><strong>Update</strong> - The Postmark team continues to work on this => update here https://ea5d5bbc.sorryapp.com/<br /><br />We have seen some transactional emails being delivered but not yet back to 100%.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>13:28</var> PST</small><br><strong>Identified</strong> - Postmark confirmed it's upstream from them at the deliverability service -- emails will be sent as the upstream provider comes back from this incident.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>12:23</var> PST</small><br><strong>Investigating</strong> - Transactional emails (Magic links, Data export completion, Invites, etc.) are not being delivered at the moment. We've identified an issue with out email provider and are in contact with them and are following along here => https://status.postmarkapp.com/ which also appears to be having issues.<br /><br />Support email and everything else Chameleon is not affected by this -- reach out to us via hello@chameleon.io with anything that needs our urgent attention.</p>tag:status.chameleon.io,2005:Incident/103720412022-06-22T12:32:40-07:002022-06-22T12:32:40-07:00TLS Certificate errors<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>12:32</var> PDT</small><br><strong>Resolved</strong> - The monitoring tool has confirmed they have resolved the issue.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>08:25</var> PDT</small><br><strong>Monitoring</strong> - We're continuing to monitor this but have no indication that browsers are rejecting our valid TLS certificates -- the issue appears to be in how our monitoring tool interprets the validity window. We will keep this open until they report back and we see the intermittent errors no longer showing up.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>10:40</var> PDT</small><br><strong>Investigating</strong> - Our CDN provider is investigating an issue with occasionally serving a TLS certificate that is either expired or only valid starting a small distance in the future. Our TLS certificates are provided by Lets Encrypt.</p>tag:status.chameleon.io,2005:Incident/101344092022-06-02T12:18:29-07:002022-06-02T12:18:29-07:00Reference error causing console logging<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>12:18</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>12:14</var> PDT</small><br><strong>Monitoring</strong> - We've deployed the update for the cause of this issue and are watching the error rates drop.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>10:45</var> PDT</small><br><strong>Identified</strong> - We've reproduced an issue that causes an error to be logged to the console when an Automatic Tour / Microsurvey is displayed -- this currently only logs the error but does not impact the Tour being displayed nor the behavior of the Tour itself.</p>tag:status.chameleon.io,2005:Incident/101267442022-06-01T13:18:20-07:002022-06-01T13:18:20-07:00Builder loading error<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>13:18</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>13:10</var> PDT</small><br><strong>Monitoring</strong> - We're seeing an error that prevents the Builder from opening, have deployed and fix and are monitoring the update</p>tag:status.chameleon.io,2005:Incident/100132772022-05-17T14:50:00-07:002022-05-17T14:50:00-07:00Manually delivered tours not displaying<p><small>May <var data-var='date'>17</var>, <var data-var='time'>14:50</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. Please note that shortlinks **do not require any changes** to continue working as they previously did.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>12:29</var> PDT</small><br><strong>Monitoring</strong> - We're testing this fix on staging and with a limited subset of accounts -- will report back when it's live for everyone</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>10:19</var> PDT</small><br><strong>Identified</strong> - We're investigating / identified an issue that caused Manually delivered Tours not to display. Typically they show immediately on page load after the short link is clicked. Quick reminder that Manually delivered tours are ones where your "Copy the short link" and paste that short link in an email or help center content.</p>tag:status.chameleon.io,2005:Incident/99842432022-05-13T20:47:52-07:002022-05-13T20:47:52-07:00Issues with opening builder from app.chameleon.io<p><small>May <var data-var='date'>13</var>, <var data-var='time'>20:47</var> PDT</small><br><strong>Resolved</strong> - With the update to the Chrome extension to version 3.1.9 you should now be able to open the Chameleon Builder from the extension directly in your app again.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>14:49</var> PDT</small><br><strong>Monitoring</strong> - We will continue to leave this open until we're confident you can open the Builder in all of the ways we expect you to</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>14:46</var> PDT</small><br><strong>Update</strong> - The most recent update here is that you can now open the Builder from the Chameleon dashboard https://app.chameleon.io/ and Clicking "Create Tour" in the upper right. As the newly submitted Chrome extension is approved and is auto-updated you will also be able to open the Builder from the Chrome extension. Thanks for bearing with us today</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>09:34</var> PDT</small><br><strong>Update</strong> - As an update to this incident, some people have reported their CSV export links were not working, the fix is to login and click the link from your inbox again.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>07:37</var> PDT</small><br><strong>Identified</strong> - We're able to reproduce an issue that prevents some users from opening the Builder directly from the Chameleon dashboard on (app.chameleon.io) -- in the meantime please navigate to your web application and open the Builder from the chrome extension.</p>tag:status.chameleon.io,2005:Incident/99804432022-05-12T21:33:33-07:002022-05-12T21:33:33-07:00Chameleon is moving to chameleon.io<p><small>May <var data-var='date'>12</var>, <var data-var='time'>21:33</var> PDT</small><br><strong>Resolved</strong> - We've completed our switchover to chameleon.io! As a parting thought, to be directly logged into https://app.chameleon.io simply visit https://www.chameleon.io and click the link in the upper right for "Dashboard" (or "Sign in" if you were already logged out of Chameleon and generate a new Magic link).<br /><br />We have taken great lengths to be your partner over the years, here's to many more!</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>20:56</var> PDT</small><br><strong>Monitoring</strong> - The last phase of the switch to chameleon.io is nearly complete, we're monitoring the suite of changes against a subset and will release more broadly throughout the next hour.</p>