Backup and recovery

Restores are performed on a schedule, into an environment that is not production, and the result is checked rather than glanced at: row counts against expectations, the most recent records present, relationships intact, and the application actually starting against the restored copy.

That last check catches the failure people are surprised by. A dump can restore cleanly and still be useless, because it captured the tables and not the extensions, the sequences, the stored procedures, or the storage bucket the rows point at. A database restore is not the same thing as a system restore.

Recovery time is measured by doing it, then written down. "We have backups" is not a recovery plan; "the database restores in about forty minutes and the uploaded files take another hour, so a total loss is most of a morning" is one, and it is the sentence that lets a business decide whether that is acceptable.

Recovery point is stated as plainly as recovery time. Backups taken nightly mean that a failure at four in the afternoon loses the day's work, and whether that is tolerable is a business decision that has to be made by the business — not inherited silently from whatever the default schedule happened to be.

Backups live somewhere the production system cannot reach with its own credentials. A backup in the same account, deleteable by the same key, protects against hardware failure and against nothing else — not against a mistaken command, and not against anybody who obtains that key.

Backups are versioned and retained in a form that a compromised credential cannot quietly rewrite. The scenario that defeats an otherwise sound arrangement is not hardware failure — it is an attacker or a mistaken script with production access deleting the backups first, or encrypting them along with everything else. Immutable retention for a defined window is what separates a backup from a second copy of the problem.

Backups are encrypted, and where the key lives is part of the design, not an afterthought. A backup encrypted with a key held only inside the system being backed up is not a backup, and the discovery that it cannot be opened happens at exactly the moment it is needed. Somebody outside that system must be able to obtain the key, and who that is gets written down.

More than one copy, and more than one location. The cost of a second destination is small and the scenario it covers is the total loss of a provider account, which is rare, unrecoverable, and has happened to real businesses.

Partial recovery is designed for, because it is the case that actually occurs. Total loss is the scenario everybody plans against and almost nobody experiences; what happens weekly is one table corrupted by a bad migration, or one customer's records deleted in error. Being able to restore a subset into a scratch environment and copy the affected rows back is a different capability from restoring everything, and it is the one used.

Where several systems reference each other, the order of restoration is written down. Bringing a database back before the storage it points at, or an application before the schema it expects, produces a sequence of confusing errors that look like corruption and are only ordering — and working that out from first principles during an incident is expensive.

What counts as recovered is defined in advance. Without an agreed criterion the incident ends when somebody feels it has, which is reliably earlier than the point at which everything is actually working, and the remainder surfaces over the following days as a series of apparently unrelated faults.

Retention is deliberate, because the most common data-loss event is not dramatic. It is a corruption or a bad migration that goes unnoticed for a week, by which time a three-day retention window contains nothing but copies of the damage.

Uploaded files are backed up too, and this is the gap found most often. Databases get attention because they are obviously important; the storage bucket holding every document a customer ever submitted frequently has no backup at all, on the assumption that the platform is handling it.

The written procedure names who does what, in order, with the commands. During an incident the person recovering is stressed, possibly not the person who built it, and quite possibly reading this at two in the morning — which is the audience the document is written for.

Access to run a recovery is held by more than one person. A recovery plan gated on an individual being reachable is a plan with a single point of failure written into its first step.

Restores are rehearsed against realistic volume. A procedure verified against a small development copy tells you nothing about how long the real one takes, and duration is the number the business actually needs.

Where data cannot be reconstructed from anywhere else, that is identified explicitly and treated differently from data that could be re-derived. Not everything deserves the same protection, and deciding which is which is cheaper before an incident than during one.

Backup failures alarm loudly, because a backup job that quietly stops is the purest form of the failure this whole area exists to prevent: everything looks normal, nothing reports an error, and the loss is only discovered at the exact moment the backup was supposed to be the answer.

All of this is verified rather than asserted, which is the single distinction worth keeping. Every business believes it has backups. The ones that recover are the ones that found out in advance.