Engineering · Part 5 · the systemd boot graph

Booting without cloud-init

platform

Reference documentation for booting a stateful single-VM service — one that must restore its database before the application starts — as an ordered systemd unit graph rather than a cloud-init script — restoring the database before it serves a request, and failing loudly if it cannot.

This is the operational substrate for a single-box deployment: one VM in an auto-scaling group of size one, an elastic IP, a reverse proxy terminating TLS, Postgres on the box, and backups to object storage. The design question that shapes it: when the machine boots — first launch, a replacement after failure, a plain reboot — how does it bring itself up in the right order, restoring data before serving traffic, and fail loudly if it cannot?

Why not cloud-init

The obvious answer is a startup script (cloud-init / user-data). Resist it, for a concrete reason. A boot script runs top-to-bottom in one context; the moment you need real ordering — "render secrets, then restore the database, but build the code venv in parallel, then start the app only once both branches are done" — you are hand-coding a dependency graph in bash, and racing the init system that is already a dependency-graph engine. Worse, mixing the two invites ordering cycles: pin your units after the cloud-init finish target while that target orders after the multi-user target your units belong to, and the init system will break the cycle by silently dropping a job — sometimes the job that starts your app, so the proxy comes up, the app does not, and you serve errors. The fix is to stop fighting the init system and model boot as units. Cloud-init drops out of the critical path entirely.

The unit graph

Express the boot as a set of systemd units with explicit After= (ordering) and Requires= (propagate failure) edges. A single target pulls the whole graph; checking one thing after boot — systemctl status app.target — tells you if everything came up.

flowchart TD
    prep["identity + secrets
attach EIP, swapfile, render env"]:::os
    prep --> db["db · DATA branch
restore-from-backup OR init Postgres"]:::data
    prep --> fetch["fetch · CODE branch
pull the release artifact"]:::code
    fetch --> venv["venv
build an offline virtualenv"]:::code
    db --> release["release · the JOIN
migrate + collectstatic + symlink flip"]:::join
    venv --> release
    release --> gunicorn["gunicorn
the app"]:::app
    gunicorn --> target(("app.target")):::app
    caddy["caddy
TLS reverse proxy"]:::app --> target
    classDef os fill:#f3f4f6,stroke:#374151;
    classDef data fill:#eff6ff,stroke:#1d4ed8;
    classDef code fill:#ecfdf5,stroke:#047857;
    classDef join fill:#fffbeb,stroke:#b45309;
    classDef app fill:#fef2f2,stroke:#b91c1c;

Read the shape: two independent branches run in parallel — a data branch (identity + secrets → database) and a code branch (secrets → fetch → venv) — and they join at release, which needs both (the code to run migrations, the database to run them against). Only then does the app start. Each preparatory unit is a Type=oneshot with RemainAfterExit=yes, so it runs once and then counts as "active" for the units that depend on it; the app itself is a long-running Type=simple. A representative unit:

# app-release.service — the join point
[Unit]
After=app-venv.service app-db.service
Requires=app-venv.service app-db.service     # a failure in either propagates here
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/app/bin/code-deploy.sh release   # migrate + collectstatic + atomic symlink flip
[Install]
WantedBy=app.target

The distinction between After= and Requires= is the whole art: After= only orders, Requires= also fails-forward. Order against a unit you merely want to follow; require a unit whose failure must abort you. Because every unit is idempotent (oneshot + RemainAfterExit), the identical graph handles first boot, replacement and reboot without special cases.

Restore-or-init: the database branch

The data branch is where a single-box design earns its keep. On boot the database unit does not assume a database exists — it restores from the latest backup if the data directory is empty, and initialises a fresh cluster otherwise. That one behaviour is what makes an auto-scaling group of size one durable: kill the box, the group launches a replacement, and the replacement restores itself from object storage before the app is allowed to start (release is After=db). Backups run continuously to that same object store; recovery is not a runbook, it is the boot path, exercised on every replacement.

Configuration: fail closed, no `.env`

The secrets unit renders one environment file from a parameter store, and the application reads every required setting from the environment with no default — not even in development. A missing value is a hard startup failure, so a missing secret can never silently become an insecure default:

def require(key):
    val = os.environ.get(key)
    if not val:
        raise ImproperlyConfigured(f'{key} must be set — config is fail-closed, no defaults')
    return val

SECRET_KEY = require('DJANGO_SECRET_KEY')

There is no .env file and no dev relaxation: local development states the same variables in the task runner (a Makefile block), CI states them in its environment, and the box gets them from the parameter store — three sources, one fail-closed reader. One wrinkle worth copying: a parameter store often cannot hold an empty string, yet "this optional feature is off" needs to be expressible. Reserve a sentinel — the literal string blank — that the settings layer maps back to an empty value in exactly one place. Fail-closed and "explicitly off" then coexist without a special case. (Part 8 is devoted to this configuration discipline.)

Where the code that runs on the box comes from — the artifact the fetch unit pulls, who builds it, and how the machine image itself is baked — is the subject of Parts 6 and 17. The boot graph only cares that the artifact exists; the rest is a deploy concern.

From one service to N

The graph above draws one service, but nothing in it is singular. Today the per-service units are systemd templates — fetch@, venv@, release@, gunicorn@ — instantiated from a small manifest at boot, so "add a service to this host" is a data edit and starting gunicorn@<svc> pulls that service's whole chain by Requires=. Two idempotence rules matured with it, and both generalise beyond Postgres:

Anything an external actor can pre-create, ensure every boot. The database boot unit renders its configuration drop-in before every start — not only on the fresh-cluster path — because on some platforms the OS package manager creates the cluster before your code ever runs, and a fresh-path-only install silently never happens.
Gate one-shot work on observable state, not on a marker left by the boot that intended to do it. The first backup is armed whenever a live database faces a zero-backup repo, not by a "fresh install" flag — a flag expires with the boot that set it, and if that boot failed after the flag was consumed, the work is silently never done.

Self-healing without a load balancer

A single box does not warrant a load balancer, but you still want the auto-scaling group to replace a wedged instance. Have the box report on itself: a timer curls the local health endpoint and, after a few consecutive failures, marks the instance unhealthy so the group replaces it. One parameter is load-bearing — the health-check grace period must exceed worst-case boot including a database restore, or a replacement gets killed mid-restore and the group loops. Size it against the slow path, not the happy path. Pair the self-report with an OnFailure= alerting unit so a failed boot pages you rather than silently retrying.

The shape to copy

Model boot as a systemd graph, not a script; one target pulls it; cloud-init is not in the critical path.
Two parallel branches — data (restore-or-init) and code (fetch → venv) — joining at a release unit that needs both; oneshot + RemainAfterExit everywhere, idempotent across reboot and replacement.
After= orders, Requires= fails-forward — choose per edge.
Fail-closed config from a parameter store; no .env; a sentinel for "off".
Self-report health to the scaling group; grace period > worst-case boot incl. restore; page on boot failure.

Part 6 is the other half of operations: the CloudFormation, the admin and CI commands that create and update all of this, and the fast-recovery-over-high-availability bet that motivates the restore-on-boot you just met.

← All posts