Post-Incident Review - Friday 29th May 2021

On the morning of Friday 29th May, after processing all direct debits and direct credits as normal, one of our jobs that handles outbound notifications failed to complete.

Impact

This resulted in:

• Some debit states becoming out of sync; therefore
• the webhook notifications were not fired

Issue

Once debits are cleared, other asynchronous workers handle the sync and notifications. Unfortunately, due to a previously unnoticed memory leak, the pods running the workers terminated ungracefully and resulted in some jobs not completing.

Fix

The memory leak has been identified and a permanent fix is in the works. In the meantime, our pods’ memory allocation has been significantly increased to prevent the issue.