The production failures nobody flagged
Backups that had stopped, order confirmations that never sent, logins that silently failed. None of them raised an alert. How they were found and fixed on three products.
, filed under production, operations, reliability, delivery, aws
The worst failures are the quiet ones. Nothing crashes, no alert goes off, and the dashboard stays green. Over a few months across three products, I found and fixed these, and nobody had flagged any of them.
Off-site backups had stopped
On a UK rental deposit-replacement insurance platform, the off-site backups had stopped. The storage bucket they were meant to land in had nothing in it. Nothing had failed loudly. The job just wasn’t doing its job.
I restored the backup path with two small Lambda functions and, the part that matters, verified a real dump rather than trusting the job. A backup you haven’t checked is a hope, not a backup.
Order confirmations weren’t sending
On a mobile SIM sign-up platform for a telecoms reseller, real orders were coming in after launch, but customers weren’t getting confirmation emails. The orders were fine. The customers just had no idea.
I built branded customer emails and separate operations emails, made sending retry safely without duplicates, and backfilled the confirmations for the orders that had missed them.
Every magic-link login was failing
On an insurance quoting platform, I was setting up production for user acceptance testing and found the production email configuration didn’t match what the app expected. Every magic-link login was failing, and nothing was reporting it. Users would just never get their link.
PDF generation broke in production
On the deposit-replacement platform, PDF generation worked locally and broke in production, because the production containers run with a read-only filesystem and the headless browser generating the PDFs needed to write to disk. I replaced it with a library that builds the PDFs directly, so production no longer needs a browser at all.
The pattern
Every one of these looked like success from the outside. A job that ran. An order that saved. A login form that submitted. The failure was in the part nobody was watching.
The lessons I take from them:
- check the backups actually contain something, don’t just check the job ran
- follow a real user journey end to end in production, including the email
- treat “it reported success” as a claim to verify, not a fact
If you’ve got a live product and a nagging feeling about what nobody’s checking, let’s talk.