Your backup is a rumour until you restore it

Ask a founder whether they have backups and you'll get a confident yes. Ask when anyone last restored one and you'll get a pause you could park a van in. That pause is the finding. A backup that has never been restored isn't a backup. It's a rumour about a backup.

Write-only systems

In most businesses, backups are a write-only system. Data goes in on a schedule someone set years ago, and nothing ever comes back out. Nobody checks that the thing being backed up is still the thing that matters. I've seen backups faithfully protecting a database that was retired two systems ago, while the one carrying the trade had never been backed up at all.

The fix costs an afternoon: pick a backup, restore it somewhere safe, and time it. Now you know three things you didn't know this morning: that it works, how long it takes, and what's missing.

The bad day, on paper

Disaster recovery sounds like an enterprise phrase, but it's three questions on one page. What breaks? Who does what? How long are we down and what does an hour cost us? You don't need a rehearsal with hard hats. You need the page written before the bad day, because during the bad day everyone is too busy having it.

When I do a technical review, this page is one of the first things I ask for. Whether it exists tells me most of what I need to know about how the infrastructure is run.

The server nobody remembers building

Somewhere in most stacks is a machine someone set up in a hurry, years ago, that has worked ever since. Nobody knows quite how it's configured, so nobody touches it, so nobody knows quite how it's configured. It works until the day it doesn't, and that day it takes the mystery down with it.

You don't have to rebuild it. You have to be able to. Those are different things, and the second one is mostly documentation.

Cheap insurance

Restore one backup. Write one page of disaster recovery. Name one owner for each. That's the whole prescription, and it's the cheapest insurance in the building. I've run my own production infrastructure for twenty years, serving hundreds of thousands of daily users, and the boring discipline above is most of the reason it's still standing.

If you'd rather have a second pair of eyes on the whole picture, codebase included, that's what the Technical Review is. It starts with a free 30 minute call via the contact page.