DE Notification - Payment Requests
Incident Report for Zepto
Postmortem

Post-Incident Review - Friday 29th May 2021

On the morning of Friday 29th May, after processing all direct debits and direct credits as normal, one of our jobs that handles outbound notifications failed to complete.

Impact

This resulted in:

  • Some debit states becoming out of sync; therefore
  • the webhook notifications were not fired

Issue

Once debits are cleared, other asynchronous workers handle the sync and notifications. Unfortunately, due to a previously unnoticed memory leak, the pods running the workers terminated ungracefully and resulted in some jobs not completing.

Fix

The memory leak has been identified and a permanent fix is in the works. In the meantime, our pods’ memory allocation has been significantly increased to prevent the issue.

Posted Jun 02, 2021 - 14:51 AEST

Resolved
This incident has now been resolved.
Posted May 28, 2021 - 17:00 AEST
Monitoring
All PaymentRequest.Debit events and webhooks have now been updated.

We will continue to monitor the situation and begin a post-incident review.
Posted May 28, 2021 - 12:59 AEST
Identified
We are continuing to investigate this issue.

- All PaymentRequest.Debits were processed and cleared as normal although some status update and webhook events failed and will be resent within the next 1 hour

- All PaymentRequest.Credits were processed and cleared as normal
Posted May 28, 2021 - 12:31 AEST
Update
We are continuing to investigate this issue.
Posted May 28, 2021 - 09:29 AEST
Investigating
We’ve noticed that some direct debits have not cleared in a timely fashion. Our team is currently investigating this as a matter of urgency.
Posted May 28, 2021 - 09:27 AEST
This incident affected: API and User interface.