On the morning of Friday 29th May, after processing all direct debits and direct credits as normal, one of our jobs that handles outbound notifications failed to complete.
This resulted in:
Once debits are cleared, other asynchronous workers handle the sync and notifications. Unfortunately, due to a previously unnoticed memory leak, the pods running the workers terminated ungracefully and resulted in some jobs not completing.
The memory leak has been identified and a permanent fix is in the works. In the meantime, our pods’ memory allocation has been significantly increased to prevent the issue.