Your backups are not backups until you have restored one
Almost every business that loses data had a backup system. The failure is rarely “we had no backups”. It is “the backups had been failing quietly for four months”, or “the backup was on the same server”, or “nobody had ever tried to restore one and it turned out to take three days”.
A backup is a claim. A restore is evidence. Until you have done the second, you have the first.
The five ways backups fail
1. They stopped and nobody noticed
Backup jobs fail. Disks fill, credentials expire, a path changes. If the failure notification goes to a mailbox nobody reads, or to someone who left, the job has been failing since whenever that happened. Alert on failure, and alert on silence — a job that stops running produces no failure notice at all.
2. They are on the same machine
A copy on the same server protects against deleting a file. It protects against nothing else: not a failed disk array, not fire, not theft, and certainly not ransomware, which will happily encrypt the backup folder along with everything else.
3. They can be deleted by whatever deleted the original
This is the ransomware lesson of the last several years. Attackers look for the backups first. If your backup destination is writable by the same account that runs your servers, it is not a safety net. Immutable storage — where a backup cannot be altered or deleted for a set retention period, even by an administrator — is the answer.
4. They back up the wrong things
The database is backed up. The uploaded files are not. The configuration lives on a machine nobody thought of as a server. You discover the gaps during the restore, which is the worst possible moment.
5. The restore takes longer than the business can survive
Restoring several terabytes over a slow link is arithmetic, and the arithmetic does not care about your deadline. If the answer is four days and the business can tolerate one, the backup strategy has already failed — you just have not found out yet.
Two numbers to agree before anything else
- How much data can you afford to lose? Nightly backups mean up to a day of work gone. If that is unacceptable for your order system, nightly is the wrong frequency for that system.
- How long can you be down? Not the aspiration — the honest answer. This number decides your architecture and your budget more than any other.
Different systems get different answers, and that is correct. Your accounting archive and your live storefront do not need the same protection.
The 3-2-1 rule, still
Three copies of the data, on two different kinds of media, with one off-site. It predates the cloud and still holds. The modern amendment is a fourth clause: one of them immutable.
The test that actually proves it
Once a quarter, restore something for real. Not “verify the backup completed” — actually bring a file, a database or a whole machine back somewhere it can be examined, and confirm the contents are right. Time it. Write down how long it took and what went wrong.
The first test always finds something. That is not a failure of the test; that is the entire point of it, discovered on a Tuesday afternoon instead of during an outage.
Backup verification and restore testing are part of every security review we run, and part of how we think about security at every layer rather than as a product you buy once.