Autopilot Journeys Status

Incident Report for Autopilot Journeys

Postmortem

A notification server which our login service uses to operate ran out of memory. The issue wasn’t immediately obvious as this server plays a very low-importance role and the login service should not be dependent on it to operate.

We fixed the issue by reducing the memory usage of the service and this immediately fixed the error.

To follow up, we will remove the direct dependence on login on this service so that this cannot happen again.

Posted Aug 18, 2021 - 10:00 UTC

Resolved

All systems fully operational.

Posted Aug 18, 2021 - 09:57 UTC

Monitoring

We have found the cause of the issue and resolved it. Everything is operating as normal now. Note that is only affected logging into the system: journeys, API, campaigns etc. all continued to function normally throughout.

Posted Aug 18, 2021 - 09:53 UTC

Update

We are continuing to investigate this issue.

Posted Aug 18, 2021 - 09:27 UTC

Investigating

People are reporting 502 errors when trying to login. We are looking into this now.

Posted Aug 18, 2021 - 09:27 UTC

This incident affected: Application.