Engineering · Part 14 · restore is the boot path

Backups you've actually restored

platform

Reference documentation for keeping a database durable on a disposable box: continuous backups to object storage, a recovery-point of about a minute, and — the point — making restore the boot path itself, so the recovery you depend on is exercised on every single replacement rather than living untested in a runbook.

Every team has backups. Far fewer have backups they have restored, and the gap between the two is where disasters live: a backup format that turns out to be subtly broken, a restore procedure nobody has run in a year, a step that assumed a human would be watching. This series runs Postgres on a single, deliberately disposable instance (Parts 5–6), which forces the issue in a productive way: the box will be thrown away and rebuilt, so restore cannot be a special occasion — it has to be routine, automatic, and proven.

Continuous backup: full snapshots plus a stream of changes

The durability tool is a Postgres-aware continuous-backup system (this design uses pgBackRest; the shape generalises). Two ingredients combine to bound data loss tightly:

• Periodic base backups — a full copy on a schedule (say daily), with cheaper incrementals in between (say every few hours), each written to an object-storage bucket.
• Continuous write-ahead-log archiving — every committed change is streamed to the same bucket as it happens, via Postgres's archive hook:

# postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'pgbackrest --stanza=app archive-push %p'   # ship each WAL segment to the repo
archive_timeout = 60s          # force a segment at least once a minute → RPO ≤ ~60s
synchronous_commit = on

The recovery-point objective — how much data a total loss can cost — is set by archive_timeout: with a segment forced at least every 60 seconds, a catastrophe loses at most about a minute of writes, without the cost or complexity of a synchronous replica. A base backup plus the archived change-stream after it can reconstruct the database to any point in time, which is also what makes point-in-time recovery (rewind to just before a bad migration) possible from the same machinery.

The key idea: restore is the boot path

Here is the move that turns "we have backups" into "we have recovery". The database's boot unit (Part 5) does not assume a database exists. It asks one question and branches:

flowchart TD
    Boot["box boots — db unit runs"]:::b --> Q{"is the data
directory empty?"}:::q
    Q -- "empty (fresh / replacement)" --> R["RESTORE from the latest
backup + replay WAL"]:::r
    Q -- "present (a reboot)" --> K["keep the existing cluster"]:::k
    R --> Up["Postgres up"]:::u
    K --> Up
    Up --> App["…only now is the app allowed to start"]:::a
    classDef b fill:#eff6ff,stroke:#1d4ed8;
    classDef q fill:#fffbeb,stroke:#b45309;
    classDef r fill:#ecfdf5,stroke:#047857;
    classDef k fill:#f3f4f6,stroke:#374151;
    classDef u fill:#eff6ff,stroke:#1d4ed8;
    classDef a fill:#fef2f2,stroke:#b91c1c;

Empty data directory ⇒ restore from the latest backup and replay the change-stream; otherwise keep the existing cluster. A brand-new box (first launch, or an ASG replacement after a failure) has an empty directory, so it restores itself — automatically, before the application is allowed to start, because the app's boot unit is ordered after the database unit (Part 5). A plain reboot finds its data intact and skips the restore. There is no separate "disaster recovery procedure"; there is the boot path, and the boot path restores.

The consequence is the whole reason to do it this way: your restore runs on every replacement. Every time the ASG swaps a box — which, in a recover-by-replacement design, is a routine event — a real restore-from-backup executes and then serves live traffic. A broken backup, an incompatible format, a bug in the restore step: all of these surface as a box that fails to come up healthy, caught by the health check and the boot-failure alert (Part 5), not as an unwelcome discovery during an actual outage. The recovery path cannot rot, because it is the same path you use to boot.

The single-node catch: timelines, and backing up the moment you promote

Restoring on every boot hides a subtlety that only bites once the same box is both the source of the backups and the target of the restore — which is exactly this disposable-single-node case. Finishing a restore promotes the recovered cluster, and every promotion starts a new Postgres timeline (the timeline id increments). Left at the backup tool's default, recovery follows the latest timeline it can find in the repo — so a fresh box tries to pick up the tip of whatever timeline some previous, now-terminated box promoted to. If the repo doesn't fully contain that timeline's starting checkpoint — a box died mid-write, a segment never shipped — the restore fails with could not locate required checkpoint record. And because the half-finished boot can itself write to the repo, one bad boot poisons the next: a self-feeding loop that gets worse every cycle, with timelines piling up as a growing chain of generations each new box must chase.

The fix is two small, coupled decisions:

• Pin recovery to the backup's own timeline, not latest. Restore with an explicit --target-timeline=current: recover exactly the timeline the restored backup sits on, and stop chasing the tip of an unbounded chain. A box now depends only on the one timeline its own backup came from — never on a foreign generation that a since-dead box left half-shipped — so terminating boxes can't leave a trail of timelines the next one must reason about.
• Back up immediately on promote. Pinning to current alone would freeze the data: every box would re-restore the same old backup and throw away everything written since it. So the instant a box finishes restoring and promotes onto its new timeline, take a fresh full backup. That backup captures the live state and becomes the anchor the next box restores from (again pinned to current). The database advances across box cycles, while the history stays bounded and self-consistent — a re-anchored, known-good backup each time, not an ever-growing pile of timeline generations to keep and replay.

Together they close the loop: each replacement restores the newest backup on its timeline, promotes, and immediately re-backs-up — so "chase latest" never happens, a terminated box leaves nothing the next one must follow, and a corrupt boot can't compound into a spiral. (A graceful shutdown also forces the final WAL segment to the repo before Postgres stops, so an orderly terminate loses nothing at all; an ungraceful one still loses only up to the archive timeout.)

Bootstrapping the app after a restore, without a human

A restored database is the real production data, so the app must come up ready — no manual "create the admin user" or "seed initial rows" step, because there is no human in a 3 a.m. replacement. The technique (detailed in Part 8's neighbours and Part 5) is that any bootstrap state the app needs — an admin account, a synthetic monitoring user — is created by idempotent data migrations that run on every boot's migrate step. On a restored box they find their rows already present and do nothing; on a genuinely fresh one they create them. Either way the box is ready to serve the instant the boot graph completes, with no interactive step in the critical path.

Drills: prove it on purpose, too

Exercising restore on every replacement is strong, but complement it with two deliberate proofs, because some failure modes only appear when you go looking:

• An automated boot/restore test. In your test matrix (Part 3), a boot-layer test stands up the real init graph against real Postgres and a real backup store in a container, and asserts the restore-or-init branch actually restores. This runs the recovery code in a place cheap enough to run often, so a regression in the restore path fails a test, not a production replacement.
• Manual disaster drills. Keep a small set of scripts that force the scenarios reality eventually will — a graceful failover, an ungraceful kill mid-write, a restore-and-verify — and run them before launch and periodically after. The graceful drill proves the happy path; the ungraceful one proves you survive a crash between backups (you lose ≤ the archive timeout, no more); the restore-check proves the restored data is actually correct, not merely present.

Between "restore on every replacement" and "drill the nasty cases on purpose", the question "have we actually restored this backup?" has a continuous, current answer of yes — which is the only answer that means anything when the box is gone.

The shape to copy

1. Continuous backup to a repo: periodic base backups + write-ahead-log archiving; set the archive timeout to your acceptable data-loss window. The repo is a config detail, not an architecture: object storage on a cloud box, a plain directory (a mounted disk or remote filesystem) on a self-hosted one — restore-or-init reads either identically, so the zero-cloud host self-rebuilds exactly like the cloud one.
2. Make the database boot unit branch on an empty data directory: restore-and-replay if empty, keep-existing if not — so restore is the boot path and runs on every replacement.
3. On a disposable single node, pin recovery to the backup's own timeline (--target-timeline=current) and take a fresh full backup the moment a restore promotes — so a replacement never chases a terminated box's timeline, and generations stay bounded.
4. Order the app after the database, so it never serves before the restore completes.
5. Bootstrap app state with idempotent migrations, so a restored box comes up ready with no human step.
6. Gate the first backup on observable state — "this live database faces a repo with zero backups" — never on a fresh-install marker. A marker expires with the boot that set it; if that boot fails after consuming it, the database silently never gets a base backup and the entire recovery story above is fiction until someone notices.
7. Prove it twice more: an automated boot/restore test, and periodic manual drills for the graceful, ungraceful, and verify-the-data cases.

Part 15 closes the operational arc with something smaller and human-facing: how the app reaches people who are not currently looking at it.

← All posts