The dashboard is green. Notifications arrive every morning. Storage usage grows normally. Everything says the backups are working.
Then comes the day you have to restore for real. And the archive is incomplete — it has been for eight months, without a single indicator flagging it, because no indicator was measuring that.
What "success" actually means
A successful backup job asserts one thing: the program finished without error. That is all. It does not assert that the right data is in there, nor that it will come back out intact.
Four silent failures cover most real cases:
The overly broad exclusion. A pattern written to skip caches also catches a data directory. The backup finishes perfectly, faster than before — which looks like an improvement.
The database copied hot. Backing up a database's files while it writes produces a perfectly valid archive containing mutually inconsistent files. Nothing signals it before you attempt a restore.
The unfindable encryption key. Archives are encrypted, which is good. The key is stored on the backed-up server — that is, precisely the machine you just lost.
Retention that ate the good version. A corruption that happened six weeks ago, a thirty-day retention: every version you keep already contains the problem.
None of these four produce an error. All of them surface on the first restore — often too late.
What a real test looks like
A valid restore test follows three rules.
Restore to a fresh location. A temporary directory, a throwaway container, a test machine. Restoring over existing data is both risky and less conclusive: you may declare success when the files you verified were already there.
Verify content, not presence. A truncated file exists. An empty file exists. Verification must compare hashes against the source, or perform a genuine application check.
Verify completeness. Compare the expected item count against what was restored. That check, and only that check, reveals accidental exclusions — the most common and most discreet failure mode.
Automating without spending your weeks on it
A full daily test is unrealistic for most setups. A partial but automated test is entirely achievable, and it captures most of the risk.
The principle: every week, restore a sample into a temporary directory, compare hashes against the source, verify the count, clean up, and report the result into monitoring like any other metric.
Three precautions make that test genuinely useful:
- Vary the sample week to week rather than always testing the same files: otherwise you validate the same path forever.
- Include at least one application item — a restored database must start and answer a query, not merely exist.
- Treat absence of result as failure. A test that has not run for three weeks must alert, exactly like a test that fails. That is the classic monitoring trap: you watch for errors, not for silence.
The annual full restore
The automated test validates the data. It does not validate the procedure, and that is a different matter.
Once a year, restore for real onto a clean system, without consulting your usual notes, with a stopwatch. What that exercise reveals is consistently the same kind of thing: an undocumented dependency, a password that only existed on the machine you lost, a manual step nobody wrote down, a restore order that turns out to matter.
Above all, it gives you the real duration. That is frequently the bad news: an impeccable backup whose full restore takes three days is not a recovery plan if your tolerance is four hours. That duration cannot be guessed, only measured.
In short
A backup never restored is not a backup: it is a hypothesis. Turning it into a certainty takes little — a partial automated restore every week, a timed full restore every year — but it is the only spend that separates real protection from a reassuring dashboard.
Comments