tag:status.saasquatch.com,2005:/historySaaSquatch Status - Incident History2024-03-27T19:52:38-07:00SaaSquatchtag:status.saasquatch.com,2005:Incident/190025472023-11-02T07:30:00-07:002023-11-02T11:07:27-07:00Potential Network Timeouts<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>07:30</var> PDT</small><br><strong>Resolved</strong> - Between 7:25AM to 7:29AM Pacific Time (UTC 2:25PM - 2:29PM) on November 2, 2023, some customers may have experienced intermittent network timeouts when making calls to the SaaSquatch system, resulting in data potentially not being received as expected. <br /><br />While retries and other reinforcements are in place in the event of these scenarios, not all calls were successful during this time period but we can confirm that the issue has since been resolved.<br /><br />We apologize for the inconvenience caused by the brief disruption in service and recommend that customers who saw discrepancies in their logs around this time when sending data to the SaaSquatch system review information within their programs and on their participants for accuracy and send remedial data if needed. <br /><br />For users that are sending events to the SaaSquatch system and want to ensure that they were/are received with no duplication, we highly recommend incorporating idempotency keys into your setup to support these edge cases if you are not already making use of them: https://docs.saasquatch.com/developer/event-idempotency<br /><br />You can reach out to the SaaSquatch support team at: saasquatch-support@impact.com</p>tag:status.saasquatch.com,2005:Incident/136689012022-11-23T08:00:00-08:002022-11-24T13:55:30-08:00Issue with Managed Identity Platform relating to SaaSquatch Hosted Portals and Microsites<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>08:00</var> PST</small><br><strong>Resolved</strong> - Between approximately 7:58AM and 11:14AM PST on November 23, 2022, SaaSquatch's Managed Identity Service encountered a series of unhandled errors originating from Google Firebase causing disruptions in service affecting Hosted Portal and Microsite logins and registrations for customers that stood up those features with SaaSquatch later than August of 2021. <br /><br />After a preliminary investigation, our team returned the Managed Identity Service to fully operational by 11:14AM Pacific on November 23, 2022. An in-depth investigation is being conducted and after findings are concluded, there will be steps put in place to monitor any potential repeats of this issue. <br /><br />We estimate that this affected about eighteen (18) SaaSquatch customers in total, and a Customer Success Manager will be reaching out to those customers individually to update them and answer any questions that may arise.<br /><br />Edit: An update was made to this status at 1:55PM PST on November 24, 2022 to adjust the date of affected customers to customers with Hosted Portals and Microsites from August 2022 to August 2021, bringing the total count of potentially affected tenants from ten to eighteen. We apologize for any confusion and for the initial inaccurate number published earlier today.</p>tag:status.saasquatch.com,2005:Incident/97062282022-04-04T13:46:24-07:002022-04-04T13:46:24-07:00Increased error rates and latency<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>13:46</var> PDT</small><br><strong>Resolved</strong> - We have implemented a fix for the issue, and are currently not seeing any further degraded performance.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>13:45</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>09:08</var> PDT</small><br><strong>Investigating</strong> - Error rates and latency have increased again. It appears to be an issue with our cloud service provider network.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>09:06</var> PDT</small><br><strong>Monitoring</strong> - The sporadic increase in error rates and latency has ended. We are continuing to investigate the root cause, but all systems are operational.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>08:50</var> PDT</small><br><strong>Investigating</strong> - We're seeing reports of increased error rates and latency, and are currently investigating.</p>tag:status.saasquatch.com,2005:Incident/96433062022-03-27T17:04:32-07:002022-03-27T17:04:32-07:00Delay in events processing<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>17:04</var> PDT</small><br><strong>Resolved</strong> - Events are being normally processed and the backlog has been cleared.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>15:02</var> PDT</small><br><strong>Monitoring</strong> - A fix has been deployed to production. The backlog has started to be processed. We are monitoring to make sure the backlog is processed properly and that no new data issues are occurring.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>14:36</var> PDT</small><br><strong>Identified</strong> - We've identified the problem with our events processing. A small number of events are not processing due to an internal error in our database access layer, and that is causing all events in the pipeline to backlog. We are implementing a fix to handle the identified case. Once that fix is deployed, the backlog will start to get cleared.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>13:39</var> PDT</small><br><strong>Investigating</strong> - Our user event processing pipeline is backlogged. Events are being received and stored but are delayed in their processing.</p>tag:status.saasquatch.com,2005:Incident/94463472022-03-02T10:42:51-08:002022-03-02T10:42:51-08:00Login with Google<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>10:42</var> PST</small><br><strong>Resolved</strong> - Issue has been resolved and a strategy has been put in place to prevent future incidents. <br /><br />Scope of issue only affected Google login into SaaSquatch administrative portal; username and password logins were unaffected. <br /><br />First failed Google login was seen at 2022-03-02 13:31:39.051 UTC.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>10:15</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>09:54</var> PST</small><br><strong>Identified</strong> - There is an issue affecting the ability for administrators to log in using Google on app.referralsaasquatch.com / auth.saasquatch.com<br /><br />Anyone that is already logged in is unaffected, this only applies to logging in and creating new sessions.</p>tag:status.saasquatch.com,2005:Incident/94004622022-02-24T09:55:05-08:002022-02-24T09:55:05-08:00Connection issues with our service provider (Heroku)<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>09:55</var> PST</small><br><strong>Resolved</strong> - The issue has now been resolved and all systems are operating as intended.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>09:17</var> PST</small><br><strong>Monitoring</strong> - The issue has been identified with our service provider (Heroku) and is being resolved upstream. We are monitoring for when this will resolve for SaaSquatch. https://status.heroku.com/incidents/2402</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>09:02</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue that is causing SaaSquatch programs to fail that appears to be related to an outage with our service provider (Heroku).</p>tag:status.saasquatch.com,2005:Incident/85524242021-11-16T15:16:40-08:002021-11-16T15:16:40-08:00SaaSquatch Service Issue<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>15:16</var> PST</small><br><strong>Resolved</strong> - Service issues with the SaaSquatch platform are resolved.<br /><br />For more details see the following GCP status update: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh<br /><br />We apologize for any inconvenience this may have caused. Please don’t hesitate to contact support@referralsaasquatch.com if this</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>15:15</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:14</var> PST</small><br><strong>Monitoring</strong> - Service issues with the SaaSquatch platform are now tentatively resolved.<br /><br />Likely evidence points to a GCP outage and our Development team is continuing to investigate to confirm: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5REh<br /><br />The latest information about the status of this issue can also be found on our status page.<br /><br />We apologize for any inconvenience this may have caused. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:05</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing service issues with the SaaSquatch platform.
<br />Our development team is currently investigating the issue and is working on a resolution in order to restore service ASAP.
<br />
<br />The latest information about the status of this issue can also be found on our status page.
<br />
<br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/79581752021-09-20T18:55:52-07:002021-09-20T18:55:52-07:00SaaSquatch Scheduled Maintenance - September 20, 2021<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>18:55</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>18:03</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>13:52</var> PDT</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting impactful System Upgrade Maintenance on September 20th, 2021. The expected maintenance window is from 6:00PM PST to 10:00PM PST, with no expected downtime.<br /><br />During this maintenance window API requests and interactions with the SaaSquatch platform may be temporarily delayed for up to 30 seconds.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact success@saasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/74995232021-07-13T21:59:36-07:002021-07-13T21:59:36-07:00Delay in event persistence propagation<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>21:59</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>21:30</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>20:32</var> PDT</small><br><strong>Identified</strong> - An internal issue with event persistence has been identified which is delaying the persistence of logged user events. <br /><br />Events are being processed for the purpose of triggering rewards programs and rewards are being generated. There is no data loss or other impact on the functioning of referrals and rewards.<br /><br />Our engineers are working to resolve the issue.</p>tag:status.saasquatch.com,2005:Incident/72333782021-06-14T01:00:33-07:002021-06-21T15:07:48-07:00Delay with SaaSquatch for Salesforce<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>01:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>18:02</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results. Syncs from SaaSquatch to Salesforce should be occurring on a regular basis.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>15:17</var> PDT</small><br><strong>Identified</strong> - We have identified an issue with the job manager used in the Salesforce integration that is preventing enough jobs from running in parallel at the same time. We are developing a hotfix to replace the affected code so that syncs are run more consistently and the backlog is cleared.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>13:10</var> PDT</small><br><strong>Investigating</strong> - The SaaSquatch for Salesforce integration is processing data syncs slower than normal. This is affecting getting data from SaaSquatch into Salesforce.<br /><br />Slowdowns appeared to have started around 7:30AM PST.<br /><br />Note that SaaSquatch forms that write to Salesforce, and data received from Salesforce via Process Builder and Invocable Methods are not affected. It is only affects the background synchronisation of data from SaaSquatch to Salesforce.</p>tag:status.saasquatch.com,2005:Incident/69043962021-05-03T17:47:08-07:002021-05-03T17:47:08-07:00SaaSquatch Platform is down<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>17:47</var> PDT</small><br><strong>Resolved</strong> - On initial review SaaSquatch has determined that a large number of database queries received in rapid succession around 6am Pacific time caused the primary SaaSquatch database to become unresponsive.<br /><br />This influx of queries has been reviewed and it was determined that it was not the result of a malicious action. <br /><br />SaaSquatch is actively implementing improvements to reduce the chances of a similar issue globally disrupting SaaSquatch systems. <br /><br />A detailed analysis of the situation is underway and relevant findings will be shared with SaaSquatch clients upon completion of the analysis.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>13:13</var> PDT</small><br><strong>Update</strong> - We are no longer experiencing any issues with our services and continue to actively monitor the situation.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>08:21</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the situations. User widgets should now be loading correctly again.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>07:23</var> PDT</small><br><strong>Investigating</strong> - At this moment, various components of the SaaSquatch platform including the widget, portal and API are not functioning. We are investigating and will be sure to post updates as they are available.</p>tag:status.saasquatch.com,2005:Incident/67981822021-04-20T17:50:41-07:002021-04-20T17:50:41-07:00Issue with Setting up Salesforce Integration<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>17:50</var> PDT</small><br><strong>Resolved</strong> - Auth0 is reporting the issue has been resolved and we are no longer seeing any issues on our end. Everything is online and functioning.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>16:02</var> PDT</small><br><strong>Monitoring</strong> - The service we rely on has restored services from their outage and we can confirm that our services have also returned to full functionality. We're continuing to monitor the situation.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:51</var> PDT</small><br><strong>Identified</strong> - Due to the ongoing outage with the Auth0 service, clients may be unable to set up and/or configure their Salesforce integration in SaaSquatch. We are actively monitoring the issue and will provide updates as they come in.</p>tag:status.saasquatch.com,2005:Incident/60003452021-01-12T11:21:24-08:002021-01-12T11:21:24-08:00Delay in Analytics Processing<p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>11:21</var> PST</small><br><strong>Resolved</strong> - This issue has now been resolved and all systems are functioning optimally.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>09:24</var> PST</small><br><strong>Monitoring</strong> - A fix to the underlying causes of the analytics delay was implemented last night and the processing backlog has since been cleared. We are monitoring the update and processing to ensure everything is functioning optimally.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>15:31</var> PST</small><br><strong>Identified</strong> - Due to an internal issue, SaaSquatch is currently seeing a delay in providing updated analytics in the SaaSquatch Portal. We are currently working on a fix to this issue.</p>tag:status.saasquatch.com,2005:Incident/39843062020-04-28T18:31:37-07:002020-04-28T18:31:37-07:00SaaSquatch Scheduled Maintenance - April 28th, 2020<p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>18:31</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>18:30</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>17:30</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>09:17</var> PDT</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting an impactful System Upgrade Maintenance starting on April 28th, 2020. The expected maintenance window is from 5:30PM PST to 7:30PM PST, with no expected downtime.<br /><br />During this maintenance window updates to referrals and programs may be delayed by up to 5 minutes.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact success@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/38528342020-04-08T17:23:19-07:002020-09-30T09:47:57-07:00Issue with Analytics not displaying correctly.<p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>17:23</var> PDT</small><br><strong>Resolved</strong> - The issue with analytics writes was identified as a partial outage with our analytics database service provider (Google BigQuery). The issue was resolved by 8:19am PST.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>11:18</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>08:45</var> PDT</small><br><strong>Investigating</strong> - Issue identified with analytics write pipeline causing analytics in the admin portal to not display accurate information. Currently investigating.</p>tag:status.saasquatch.com,2005:Incident/34817362020-01-28T18:11:33-08:002020-01-28T18:11:33-08:00SaaSquatch Scheduled Maintenance - January 28th, 2020<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>18:11</var> PST</small><br><strong>Completed</strong> - Maintenance is now complete and all systems are operational.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>17:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>09:24</var> PST</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting an impactful System Upgrade Maintenance starting on January 28th, 2020. The expected maintenance window is from 5:30PM PST to 6:30PM PST, with no expected downtime.<br /><br />During this maintenance window updates to referrals and programs may be delayed by up to 5 minutes.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact success@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/34382392020-01-17T09:41:38-08:002020-09-30T09:47:57-07:00Rewards Not Expiring<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>09:41</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>18:53</var> PST</small><br><strong>Monitoring</strong> - A 3rd party vendor updated their database version today and it caused a period of degraded service on the SaaSquatch platform. We have contacted the vendor and they have reverted the update. System should be functioning normally and we are continuing to monitor the situation.</p>tag:status.saasquatch.com,2005:Incident/33071102019-12-11T11:27:42-08:002020-09-30T09:47:57-07:00SaaSquatch Service Issue<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>11:27</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>11:24</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>11:05</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>11:04</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>11:04</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing a partial outage affecting our analytics writes.<br />Our development team is currently investigating the issue and is working on a resolution in order to restore service ASAP.<br /><br />The latest information about the status of this issue can also be found on our status page.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/26809902019-07-10T18:00:54-07:002019-07-10T18:00:54-07:00SaaSquatch Scheduled Maintenance<p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>18:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>14:28</var> PDT</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting an impactful System Upgrade Maintenance starting on Wednesday, July 10th, 2019. The expected maintenance window is from 5PM PDT to 6PM PDT, with an expected downtime of 5 minutes.<br /><br />This scheduled maintenance is intended to improve overall system and database performance and stability.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/24844222019-06-02T17:55:21-07:002020-09-30T09:47:58-07:00Major Service Outage<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>17:55</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>14:11</var> PDT</small><br><strong>Investigating</strong> - We're investigating an issue in one of our data centers that has resulted in request timeouts. Our data center provider is actively working to mitigate and restore service. We will continue to update you as we receive more information.<br /><br />For more information: https://status.cloud.google.com/incident/cloud-networking/19009</p>tag:status.saasquatch.com,2005:Incident/21861152019-02-04T17:30:15-08:002019-02-04T17:30:15-08:00SaaSquatch Scheduled Maintenance - February 4<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>17:30</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>16:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>15:22</var> PST</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting an impactful System Upgrade Maintenance starting on February 4th (originally scheduled for February 1st). The expected maintenance window is from 4:30pm PST to 5:30pm PST, with an expected downtime of 5 minutes.<br /><br />This maintenance window is due to our database provider needing to perform an emergency database security patch.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/21813892019-02-01T15:23:53-08:002019-02-01T15:23:53-08:00SaaSquatch Emergency Maintenance - February 1<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>15:23</var> PST</small><br><strong>Completed</strong> - Today's scheduled maintenance has been moved to 4:30-5:30pm PST on Monday February 4th.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>16:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>17:25</var> PST</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting an impactful System Upgrade Maintenance starting on February 1st. The expected maintenance window is from 3pm PST to 4pm PST, with an expected downtime of 5 minutes.<br /><br />This maintenance window is due to our database provider needing to perform an emergency database security patch.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/18397832018-08-09T20:00:29-07:002018-08-09T20:00:29-07:00SaaSquatch Scheduled Maintenance - August 9<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>20:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>17:09</var> PDT</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting an impactful System Upgrade Maintenance starting on Thursday, August 9th, 2018. The expected maintenance window is from 7PM PDT to 8PM PDT, with an expected downtime of 5 minutes.<br /><br />This scheduled maintenance is intended to improve overall system and database performance and stability.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/17926552018-07-05T20:00:04-07:002018-07-05T20:00:04-07:00SaaSquatch Scheduled Maintenance - July 5<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>20:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>19:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>16:49</var> PDT</small><br><strong>Scheduled</strong> - SaaSquatch will be conducting an impactful System Upgrade Maintenance on Thursday, July 5th, 2018. The expected maintenance window is from 7pm PDT to 8pm PDT, with an expected downtime of 5 minutes.<br /><br />This scheduled maintenance is intended to improve overall system and database performance and stability.<br /><br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>tag:status.saasquatch.com,2005:Incident/17773762018-06-21T17:15:59-07:002020-09-30T09:47:58-07:00SaaSquatch Service Issue<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>17:15</var> PDT</small><br><strong>Resolved</strong> - An investigation into the outage concluded that a specific legacy search functionality could cause an unintentionally large amount of data to be returned by a search query. This large response slowed down the system as it processed the data, causing a backlog of requests, and resulted in the experienced outage.<br /><br />This specific legacy search functionality has since been updated so that it can no longer cause this issue. <br />We also took this opportunity to review the other search functionality in our system to ensure that similar, or related, problems don't happen in other parts of our system.<br /><br />We apologize for any inconvenience this may have caused. If you have any additional questions or concerns about this please contact our support team at support@referralsaasquatch.com.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:15</var> PDT</small><br><strong>Monitoring</strong> - Our product team has identified the cause of the service issue, and has implemented steps to resolve it.
<br />
<br />All of our components are now operating as expected, and we will continue to monitor our services closely until we are confident that all outstanding issues have been resolved.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:03</var> PDT</small><br><strong>Investigating</strong> - We are currently experiencing service issues with the SaaSquatch platform.
<br />Our development team is currently investigating the issue and is working on a resolution in order to restore service ASAP.
<br />
<br />The latest information about the status of this issue can also be found on our status page.
<br />
<br />We apologize for any inconvenience this may cause. Please don’t hesitate to contact support@referralsaasquatch.com if this raises any questions or concerns.</p>