tag:status.spiff.com,2005:/historySpiff Status - Incident History2024-03-29T05:23:41ZSpifftag:status.spiff.com,2005:Incident/190766892023-11-09T20:42:14Z2023-11-09T20:42:14ZLong load times<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>20:42</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>20:42</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>19:14</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/187437612023-10-11T15:18:07Z2023-10-11T15:18:07ZTeam Functionality Degradation<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>15:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>20:05</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>14:11</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>14:09</var> UTC</small><br><strong>Investigating</strong> - We’ve experienced unexpected behavior in Team functionality. We have identified the issue and are applying a fix. All unfrozen September statements are being recalculated. We’ll post an update as soon as statements have been recalculated.</p>tag:status.spiff.com,2005:Incident/179599932023-07-25T23:14:01Z2023-07-25T23:14:01ZSpiff App is down.<p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>23:14</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>23:01</var> UTC</small><br><strong>Update</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>22:56</var> UTC</small><br><strong>Monitoring</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>22:56</var> UTC</small><br><strong>Identified</strong> - We have isolated the root cause, and things are starting to come back online now.</p><p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>22:56</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>22:52</var> UTC</small><br><strong>Investigating</strong> - The Spiff application is currently down. We are working to bring things back online as quickly as possible.</p>tag:status.spiff.com,2005:Incident/179558692023-07-25T12:30:00Z2023-07-25T14:16:45ZDifficulties loading designer. Potentially delayed syncs.<p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>12:30</var> UTC</small><br><strong>Resolved</strong> - This incident was detected and resolved.</p>tag:status.spiff.com,2005:Incident/179404922023-07-24T00:22:56Z2023-07-24T00:28:29ZDegraded Performance<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>00:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>23:55</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>23:18</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/177535432023-07-04T02:15:34Z2023-07-05T14:42:06ZPotential issue delaying Salesforce synchronization<p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>02:15</var> UTC</small><br><strong>Resolved</strong> - All backlogged work has been processed and the application is running normally.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>21:34</var> UTC</small><br><strong>Update</strong> - The root cause of the issue was determined and resolved. It is expected that residual delay in performance may continue while the application processes backlogged work.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>19:29</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>17:22</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>17:22</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>16:11</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>15:29</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/174247892023-05-30T21:00:20Z2023-05-31T17:01:25ZPartial Outage Impacting exports<p><small>May <var data-var='date'>30</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>20:39</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>20:33</var> UTC</small><br><strong>Identified</strong> - We have isolated the problem and are currently implementing a fix.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>19:51</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an issue with rendering exports. More details will be provided when available.</p>tag:status.spiff.com,2005:Incident/173126482023-05-22T19:02:47Z2023-05-23T21:36:24ZSpiff app failing to load pages / Returning a blank white page<p><small>May <var data-var='date'>22</var>, <var data-var='time'>19:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>18:55</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>14:48</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/169342472023-05-20T16:14:43Z2023-05-20T16:14:43ZDegraded Performance<p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:14</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>23:23</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>16:46</var> UTC</small><br><strong>Update</strong> - Several performance fixes have been implemented. Continued work is being done to ensure the performance of "Designer" is improved further within specific features.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>23:16</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and an fix is being implemented.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>22:52</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing intermittent degraded performance in certain areas of the application.</p>tag:status.spiff.com,2005:Incident/171218142023-05-01T23:00:51Z2023-05-02T16:01:10ZData out of sync with Salesforce<p><small>May <var data-var='date'> 1</var>, <var data-var='time'>23:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Identified</strong> - We have isolated the problem, and have implemented a fix, data is now flowing through and we expect everything to be current by this afternoon. If you require current data sooner than this please reach out to your CSM to work through your options.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>15:55</var> UTC</small><br><strong>Investigating</strong> - Information coming from Salesforce has not successfully sync'd since April 30th for all customers using that data source.</p>tag:status.spiff.com,2005:Incident/163601522023-03-07T15:48:16Z2023-03-07T15:48:16ZCustomers receiving "Are you lost?" on specific pages.<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>22:10</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>20:37</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/162268152023-02-21T22:50:35Z2023-02-21T22:50:35ZAll Rule Metric Cards have currently disappeared.<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>22:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>21:33</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>21:28</var> UTC</small><br><strong>Identified</strong> - The issue has been currently identified and a fix is currently being issues</p>tag:status.spiff.com,2005:Incident/158634902023-01-17T17:00:59Z2023-01-20T17:55:18ZAn issue has been detected with timely commission calculations.<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>16:54</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>16:14</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>03:15</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/158066262023-01-10T02:03:38Z2023-01-10T02:03:38ZScheduled Database Restart<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>02:03</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>02:02</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>01:05</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing a scheduled database restart that will cause an interruption for a few minutes.</p>tag:status.spiff.com,2005:Incident/158058842023-01-10T01:08:25Z2023-01-10T01:08:25ZScheduled Database Restart<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>01:08</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>01:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>22:33</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing a scheduled database restart that will cause an interruption for a few minutes.</p>tag:status.spiff.com,2005:Incident/109505532022-08-27T11:00:31Z2022-08-28T02:23:35ZDelays in timely commission calculation<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Resolved</strong> - The root cause of this issue has now been identified and a remediation has been put in place to allow the majority of calculations to proceed at normal speed. Some calculation errors will be temporarily ignored until a more stable fix is available.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>10:03</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>14:43</var> UTC</small><br><strong>Investigating</strong> - Some delays in timely commission calculation is are being observed. We are currently investigating the issue.</p>tag:status.spiff.com,2005:Incident/108699192022-08-20T22:48:27Z2022-08-20T22:49:02ZScheduled Database Improvements<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>22:49</var> UTC</small><br><strong>Update</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>22:48</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>21:58</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>21:58</var> UTC</small><br><strong>Scheduled</strong> - We are currently applying improvements to our databases. You may experience degraded performance throughout portions of the app during this time. We appreciate your patience.</p>tag:status.spiff.com,2005:Incident/107049412022-07-28T23:18:20Z2022-07-28T23:18:20ZPartial Outage on Spiff Analytics<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>23:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>22:22</var> UTC</small><br><strong>Update</strong> - We are continuing to troubleshoot the issue. Updates are being applied.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>00:29</var> UTC</small><br><strong>Update</strong> - The issue seems related to the Chrome browser. Firefox may be used as a workaround.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>18:30</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/106941552022-07-26T17:03:06Z2022-07-26T17:03:07ZDesigner Outage<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>17:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>15:06</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>14:57</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>14:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/105391462022-07-11T15:59:29Z2022-07-11T15:59:29ZIncident #34 - Nginx Instance - 500 - site availability issues<p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>15:59</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p>tag:status.spiff.com,2005:Incident/103234682022-06-16T20:25:18Z2022-06-16T20:25:18ZDegraded Performance<p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>20:25</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>19:52</var> UTC</small><br><strong>Monitoring</strong> - We believe that we've identified the issue. We are working on testing and confirming. Performance has already returned and continuing to monitor.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>19:52</var> UTC</small><br><strong>Identified</strong> - We believe identified the issue, working on testing and confirming.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>17:48</var> UTC</small><br><strong>Monitoring</strong> - Performance has been restored, we are continuing monitoring.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>17:15</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/97233782022-04-08T16:34:54Z2022-04-29T16:35:57ZApp outage, Sporadic<p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>16:38</var> UTC</small><br><strong>Monitoring</strong> - Issue seems to be resolved, we are monitoring for further performance problems.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Identified</strong> - Reports on designer slowness are still being received.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>18:05</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Update</strong> - Spiff application has recovered, however, the designer feature is still slow and being triaged.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>17:45</var> UTC</small><br><strong>Update</strong> - Currently looking into sporadic slowdowns and issues with the application.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>17:24</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>17:06</var> UTC</small><br><strong>Investigating</strong> - We are currently looking into slowdowns with the designer feature of Spiff app.</p>tag:status.spiff.com,2005:Incident/95193812022-03-11T21:28:29Z2022-03-11T21:28:29ZPartial Outage<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>21:28</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>18:15</var> UTC</small><br><strong>Monitoring</strong> - The issue with the upstream provider has apparently resolved.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>18:11</var> UTC</small><br><strong>Identified</strong> - A problem was identified with an upstream provider that may be impacting a small number of customers.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>18:07</var> UTC</small><br><strong>Update</strong> - A problem was identified with an upstream provider that may be impacting a small number of customers.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>18:01</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating reports of a possible outage.</p>tag:status.spiff.com,2005:Incident/93174842022-02-15T00:01:11Z2022-02-18T18:38:51Zwww.spiff.com is down<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>00:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>23:21</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.spiff.com,2005:Incident/92681032022-02-14T23:23:35Z2022-02-18T18:38:51ZAPI Outage<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>23:23</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>23:22</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>23:22</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>19:11</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>19:04</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>