tag:status.autopilothq.com,2005:/historyAutopilot Journeys Status - Incident History2024-03-29T00:06:49ZAutopilot Journeystag:status.autopilothq.com,2005:Incident/182429262023-08-24T17:20:55Z2023-08-24T17:20:55ZDegraded Database Performance<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>17:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>07:44</var> UTC</small><br><strong>Investigating</strong> - We have notified our cloud database provider of performance issues we are seeing. They have acknowledged the issue and are investigating. Autopilot application performance will be degraded.</p>tag:status.autopilothq.com,2005:Incident/178958152023-07-19T17:21:15Z2023-07-19T17:21:15ZDegraded Database Performance<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>17:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>06:55</var> UTC</small><br><strong>Monitoring</strong> - We are continuing to monitor requests made by our application to our database provider. We see significant improvement relative to performance degradation seen at start of this incident.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>03:15</var> UTC</small><br><strong>Investigating</strong> - We have notified our cloud database provider of performance issues we are seeing. They have acknowledged the issue and are investigating. Autopilot application performance will be degraded.</p>tag:status.autopilothq.com,2005:Incident/178463892023-07-16T06:25:19Z2023-07-16T06:25:19ZDegraded Database Performance<p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>06:25</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>18:34</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor requests made by our application to our database provider. We see significant improvement relative to performance degradation seen at start of this incident.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>04:11</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>17:37</var> UTC</small><br><strong>Investigating</strong> - We have notified our cloud database provider of performance issues we are seeing. They have acknowledged the issue and are investigating. Autopilot application performance will be degraded.</p>tag:status.autopilothq.com,2005:Incident/176362622023-06-21T17:49:43Z2023-06-21T17:49:43ZDegraded Database Performance<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>06:48</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>00:33</var> UTC</small><br><strong>Investigating</strong> - We have notified our cloud database provider of performance issues we are seeing. They have acknowledged the issue and are investigating. Autopilot application performance will be degraded.</p>tag:status.autopilothq.com,2005:Incident/172759902023-05-17T13:50:30Z2023-05-17T13:50:30ZIssue with journey processing and other slowness<p><small>May <var data-var='date'>17</var>, <var data-var='time'>13:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>12:52</var> UTC</small><br><strong>Update</strong> - Systems are back to performing as normal. We have found the cause of the issue and are working to prevent this from happening again.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>12:21</var> UTC</small><br><strong>Monitoring</strong> - The system is returning to normal and we are monitoring closely to ensure performance continues and things catch up.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>11:23</var> UTC</small><br><strong>Update</strong> - We are still working on resolving the performance issues.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>10:05</var> UTC</small><br><strong>Identified</strong> - We have found that there is an issue with our database and we are working to restore full functionality.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>09:04</var> UTC</small><br><strong>Investigating</strong> - We are investigating an issue with slow journey processing and publishing. We believe our database provider is experiencing issue and will update as soon as we have more information.</p>tag:status.autopilothq.com,2005:Incident/171145672023-05-02T03:51:07Z2023-05-02T03:51:07ZDegraded Database Performance<p><small>May <var data-var='date'> 2</var>, <var data-var='time'>03:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>00:31</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>00:10</var> UTC</small><br><strong>Investigating</strong> - We have notified our cloud database provider of performance issues we are seeing. They have acknowledged the issue and are investigating. Autopilot application performance will be degraded.</p>tag:status.autopilothq.com,2005:Incident/169652962023-04-26T00:40:25Z2023-04-26T00:40:25ZCampaigns App API Outage<p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>00:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>23:13</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>19:16</var> UTC</small><br><strong>Investigating</strong> - We are investigating an API outage impacting the loading and saving of emails within the Campaign App.</p>tag:status.autopilothq.com,2005:Incident/169203082023-04-20T17:50:16Z2023-04-20T17:50:16ZApplication outage<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>17:50</var> UTC</small><br><strong>Resolved</strong> - Thank you for your patience, we are now considering this resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>13:40</var> UTC</small><br><strong>Update</strong> - The system is back to normal and task processing is catching up.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>13:11</var> UTC</small><br><strong>Monitoring</strong> - The application servers are now responding as normal again and journeys are beginning to work through the backlog which built up during the outage. We are continuing to monitor and will do so to make sure everything continues to stay online, and that all of the journey backlog is cleared.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>12:41</var> UTC</small><br><strong>Update</strong> - Our database provider is working on resolving the issue. We will update as soon as we have more information.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>12:01</var> UTC</small><br><strong>Identified</strong> - We have identified a database issue causing a full outage to accessing the application and slow processing of journeys. We are working with our database provider to resolve this issue.</p>tag:status.autopilothq.com,2005:Incident/169068552023-04-19T04:54:24Z2023-04-19T04:54:24ZDegraded Database Performance<p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>04:54</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>01:47</var> UTC</small><br><strong>Investigating</strong> - We have notified our cloud database provider of performance issues we are seeing. They have acknowledged the issue and are investigating. Autopilot application performance will be degraded.</p>tag:status.autopilothq.com,2005:Incident/168674832023-04-14T14:00:00Z2023-04-14T18:19:15ZDatabase Related Outage<p><small>Apr <var data-var='date'>14</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Resolved</strong> - We have reached out to our cloud database provider regarding an outage on their end that lasted one hour. This outage degraded the application's functionality during this one hour window.</p>tag:status.autopilothq.com,2005:Incident/165337062023-03-18T03:40:46Z2023-03-18T03:40:46ZInfrastructure Maintenance<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>03:40</var> UTC</small><br><strong>Completed</strong> - Maintenance is complete. Our application and api was intermittently down during the first 2 hours of maintenance. Apologies for any inconvenience and thank you for your patience.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>02:52</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>01:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>21:07</var> UTC</small><br><strong>Scheduled</strong> - We will be performing necessary infrastructure maintenance mandated by our cloud server provider which will impact the speed and availability of the application and API.</p>tag:status.autopilothq.com,2005:Incident/165141752023-03-15T23:28:40Z2023-03-15T23:28:40ZApplication outage<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>23:28</var> UTC</small><br><strong>Resolved</strong> - We are now considering this resolved. Thank you for your patience.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>19:13</var> UTC</small><br><strong>Monitoring</strong> - We experienced a multi host problem that that resulted in downtime on email links, apis and the application.<br /><br />We were able to resolve the root problem shortly after out status page went up, but some of the resulting secondary problems took longer to resolve. We are continuing to monitor and investigate if any further action is needed. <br /><br />As a result, customers may experience a minor delay as Autopilot processes the backlog of tasks. We will continue to keep customers updated.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>17:09</var> UTC</small><br><strong>Investigating</strong> - We are investigating issues with our web application that are preventing access.</p>tag:status.autopilothq.com,2005:Incident/158146072023-01-11T04:09:49Z2023-01-11T04:09:49ZDegraded database performance<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>04:09</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>01:04</var> UTC</small><br><strong>Monitoring</strong> - Our database provider have identified the issue and implemented a fix. We are monitoring their solution.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>23:58</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing degraded performance across the application due to a problems with a database provider. We are working with them to identify and resolve the issue.</p>tag:status.autopilothq.com,2005:Incident/144386012022-12-05T22:40:30Z2022-12-05T22:40:30ZNetwork issue affecting some users<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:40</var> UTC</small><br><strong>Resolved</strong> - We believe this issue was related to a greater issue with connectivity between ISPs and AWS and is now resolved.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:39</var> UTC</small><br><strong>Monitoring</strong> - We believe this issue was related to a greater issue with connectivity between ISPs and AWS. We have not seen issues for some time now so this is likely resolved, but we will continue to monitor.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>21:01</var> UTC</small><br><strong>Investigating</strong> - We are investigating a network issue affecting some users access to the application. It does not impact the product beyond access to the user interface.</p>tag:status.autopilothq.com,2005:Incident/123353512022-10-22T18:52:11Z2022-10-22T18:52:12ZInfrastructure Maintenance<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>18:52</var> UTC</small><br><strong>Completed</strong> - Maintenance is now completed. Thanks for your patience.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>22:54</var> UTC</small><br><strong>Scheduled</strong> - We will be performing necessary infrastructure maintenance which will impact the speed and availability of the application and api. We estimate that the maintenance should be done within an hour.</p>tag:status.autopilothq.com,2005:Incident/122585232022-10-19T20:29:47Z2022-10-20T17:59:59ZDegraded database performance<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>20:29</var> UTC</small><br><strong>Resolved</strong> - The degraded database performance has improved and we are now considering this resolved. We will continue working to try to mitigate this type of issue in future. Thanks for your patience.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>02:53</var> UTC</small><br><strong>Update</strong> - We are working with the provider and are continuing to investigate the issue. Most users should not be impacted, but we expect that some users will experience some degraded performance.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>17:24</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing degraded performance across the application due to a problems with a database provider. We are working with them to identify and resolve the issue.</p>tag:status.autopilothq.com,2005:Incident/120403142022-10-14T20:16:36Z2022-10-14T20:16:36ZIntermittent failure loading early access campaign and email manager<p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>20:16</var> UTC</small><br><strong>Resolved</strong> - We are now considering this incident resolved. Thanks for your patience.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>20:04</var> UTC</small><br><strong>Monitoring</strong> - We deployed a change which seems to have resolved the issue. We are now monitoring to ensure the issue is resolved. Thank you for your patience.</p><p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>18:59</var> UTC</small><br><strong>Investigating</strong> - We are investigating some timeouts that are causing intermittent failure loading early access campaign and email manager.</p>tag:status.autopilothq.com,2005:Incident/99710452022-05-13T04:05:14Z2022-05-13T04:05:14ZApplication Maintenance<p><small>May <var data-var='date'>13</var>, <var data-var='time'>04:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>02:55</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>01:13</var> UTC</small><br><strong>Scheduled</strong> - We will be impacted by a provider's planned maintenance and expect delayed updates to contacts and journeys and some temporary loss of functionality on our application and api.</p>tag:status.autopilothq.com,2005:Incident/99605232022-05-10T20:05:45Z2022-05-10T20:05:45ZPipedrive Token Refresh and Salesforce Activity Sync<p><small>May <var data-var='date'>10</var>, <var data-var='time'>20:05</var> UTC</small><br><strong>Resolved</strong> - We have fixed the underlying server issues and are processing the backlog</p><p><small>May <var data-var='date'>10</var>, <var data-var='time'>18:44</var> UTC</small><br><strong>Identified</strong> - We are have identified the issue and working on the fix</p>tag:status.autopilothq.com,2005:Incident/98992032022-05-04T04:00:24Z2022-05-04T04:00:25ZApplication Maintenence<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>03:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>22:48</var> UTC</small><br><strong>Scheduled</strong> - We will be impacted by a provider's planned maintenance and expect some downtime on our application and api.</p>tag:status.autopilothq.com,2005:Incident/98752742022-04-29T05:00:23Z2022-04-29T05:00:25ZWebsite Maintenence<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>05:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>22:06</var> UTC</small><br><strong>Scheduled</strong> - We will be impacted by a providers planned maintenance and expect some downtime on our website.</p>tag:status.autopilothq.com,2005:Incident/97065832022-04-04T14:30:00Z2022-04-04T16:38:03ZDatabase provider failure<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Resolved</strong> - Our database provider experienced an outage which temporarily impacted access. We are working with them closely and will provide updates as more information becomes available.</p>tag:status.autopilothq.com,2005:Incident/91717352022-01-27T01:04:05Z2022-01-27T01:04:05ZDatabase Related Outage<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>01:04</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>23:08</var> UTC</small><br><strong>Monitoring</strong> - We see database operations stabilizing and are monitoring performance</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>21:55</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue with one of our databases. This is currently breaking the user interface on multiple pages and some features may be degraded or unavailable at this time.</p>tag:status.autopilothq.com,2005:Incident/91380852022-01-24T18:06:12Z2022-01-24T18:06:12ZEmail Delivery Issue<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>18:06</var> UTC</small><br><strong>Resolved</strong> - This issue has now been resolved. Thank you for your patience.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>19:23</var> UTC</small><br><strong>Identified</strong> - We are currently experiencing issues reaching the inboxes of some email receivers and we are working hard to resolve this issue.</p>tag:status.autopilothq.com,2005:Incident/89008592021-12-21T07:51:57Z2021-12-21T07:51:57ZDatabase delay causing journey delays<p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>07:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>07:20</var> UTC</small><br><strong>Monitoring</strong> - The system has returned to normal. We will continue to monitor to check that all journeys continue as expected.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>07:10</var> UTC</small><br><strong>Identified</strong> - Our database provider is bringing the affected nodes back online.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>06:16</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>06:16</var> UTC</small><br><strong>Investigating</strong> - Our database provider has identified an issue causing some database requests to fail. This is causing journeys to be delayed and in some cases not publish. We are working on this now and will update as soon as this issue is resolved.</p>