Engineering · Part 3 · the five-layer hexagon

Testing a system you can't afford to run for real

testingstaff

Reference documentation for testing a system with cloud dependencies, an LLM, a browser UI and a remote OAuth server — hermetically, in seconds, on a laptop, with the real integrations covered somewhere cheaper than "everywhere".

The application talks to object storage, a message queue and a serverless renderer; it streams from an LLM; it serves HTML through htmx and a remote MCP server over OAuth. If every test needed all of that live, the suite would be slow, flaky and expensive, and nobody would run it. The strategy below keeps the fast inner loop fully in-process while guaranteeing every faked boundary is exercised against the real thing somewhere.

Five layers, arranged as a hexagon

Think of the system as ports and adapters. The driving adapter (a browser, an API call) pushes requests in; the driven adapters (storage, queue, LLM, payments) reach out. Each test layer fakes the adapters another layer owns — so together they cover the whole system without every test paying for real cloud.

flowchart TB
    U[Unit — TestCase]:::fast
    A[Acceptance — real browser + real WSGI]:::fast
    I[Integration — real S3 / queue / LLM]:::slow
    B[Boot — real systemd in a container]:::slow
    S[Smoke — the actually-deployed system]:::deploy
    U --> A --> I --> B --> S
    U -.covers.-> DC[domain core + registry]
    A -.covers.-> DP[driving port: browser to DB]
    I -.covers.-> DR[driven ports: wire formats]
    B -.covers.-> OP[boot graph, restore-on-boot]
    S -.covers.-> EE[the whole thing, live]
    classDef fast fill:#ecfdf5,stroke:#047857;
    classDef slow fill:#eff6ff,stroke:#1d4ed8;
    classDef deploy fill:#fffbeb,stroke:#b45309;

The contract that makes fakes honest is a single matrix: every faked boundary names where it is covered for real. Storage is faked in acceptance (local filesystem) and covered by the integration test (real S3) plus smoke (real presigned URLs). The LLM is a deterministic fake in acceptance and covered by an optional integration test against a real endpoint. Payments are an in-memory fake gateway and covered by a deploy-time validator plus a staging run in the provider's test mode. Keep the matrix in the docs and in review: a new faked boundary without a "covered for real by …" entry is an incomplete test.

Acceptance: a real browser on the real request path

The acceptance layer is behave (Gherkin) driving Playwright (a real Chromium) against the live application. Two rules make it trustworthy rather than theatre:

• Navigate like a user. A scenario's actions reach a page the way a person would — click the nav, a card, a button — never a deep link. This catches wiring bugs a test-client call cannot: a scenario that deep-links to a working view passes while the real link is broken. Deep links are allowed only for access-control probes (open a forged URL, assert 404) and for @given data setup.
• No writes in an action step. A @when must persist through the UI — fill the form, click submit — never call the ORM or a service directly, or the real route/view/form is never exercised and the test passes even if it is broken. ORM writes belong only in @given preconditions.

Run it under the same server your production uses. If production is gunicorn with a gevent worker, run the acceptance live server under gevent's WSGI too, so tests exercise the same cooperative-greenlet path — which is what makes server-sent-event streaming testable in-process rather than manual-only. The login form is real but slow, so most scenarios take a fast-path: force_login a fixture user via the test client, then inject the session cookie into Playwright. The real login and signup forms are still covered — by the handful of scenarios whose subject is authentication.

The database story — reset to the migrated baseline, fast

This is where most suites are quietly slow. Two mechanisms, one per layer, and they are not the same:

flowchart LR
    subgraph Unit ["Unit — TestCase"]
      U1[migrate ONCE, keep the DB] --> U2[each test in a transaction] --> U3[ROLLBACK]
      U3 --> U2
    end
    subgraph Behave ["Acceptance — behave"]
      B1[FRESH DB, migrate once] --> B2[each scenario commits for real] --> B3[flush + restore snapshot]
      B3 --> B2
    end

• Unit tests reuse a kept database. Migrate once (--keepdb; new migrations apply incrementally), and isolate each test in a transaction that rolls back. Rollback is a structural reset to the migrated state and cannot dirty the kept DB — so reuse is provably safe and the per-run migration cost is paid once, ever.
• Acceptance uses a fresh database each run. Its scenarios hit a live server and commit for real, so a transaction wrapper would be invisible to the server. Instead, each scenario flushes and restores a serialized snapshot of the post-migration state (serialized_rollback). Because a completed run ends on a flush, a kept DB would be left empty for the next run — so acceptance takes a fresh DB, on a separate database name from the unit suite, and the whole "poisoned kept DB" problem disappears.

Define "the migrated baseline" precisely, because everything resets to it: it is exactly the rows your data migrations and post-migrate hooks seed — a bootstrap admin, a synthetic smoke user, its one smoke tenant — and nothing else. Every layer starts there: unit rollbacks return to it, each acceptance scenario restores a snapshot of it, a fresh deploy boots into it.

Two more speed levers, both free: set a weak password hasher under test (PASSWORD_HASHERS = ['django.contrib.auth.hashers.MD5PasswordHasher']) — the production KDF is otherwise the suite's dominant cost, since almost every test creates a user — and iterate with a scoped, fail-fast command (test <module> --keepdb --failfast), running the full suite only before commit. A test that takes seconds is a smell; a suite you must wait minutes for is one you stop running.

Contracts locked by tests

Some invariants are cheaper to enforce than to remember. Three patterns, each a few lines, each failing with a precise message:

# 1. Registry-derived completeness: every write operation must assert its effects.
def test_every_write_operation_has_an_effects_test(self):
    covered = {m[len('test_'):] for m in dir(EffectsTests) if m.startswith('test_')}
    writes = {n for n, op in OPERATIONS.items() if op.kind == 'write'} - EXEMPT
    self.assertEqual(writes - covered, set(),
                     'write operations with no effects test — add one, or add to EXEMPT with a reason')

# 2. AST call-layer: transports never call services directly (Part 1).
# 3. AST no-signals: no @receiver / .connect() anywhere in the apps —
#    object-lifecycle behaviour must live at explicit seams.
def test_no_signals(self):
    for path in app_python_files():
        for node in ast.walk(ast.parse(path.read_text())):
            if is_receiver_decorator(node) or is_connect_call(node):
                self.fail(f'{path}: signal usage — use an explicit seam instead')

The pattern generalises: when a design rule ("every mutation emits effects", "the agent package imports nothing from the domain package", "no private APIs") could silently rot, derive its check from the code itself. These run in seconds, in the unit suite, and turn architecture from a document into a build error.

Testing the hard integrations without the cloud

The LLM: a deterministic fake drives everything

Acceptance selects a fake backend that returns scripted responses — including scripted tool calls — with no network. Because the whole agent loop (Part 4) runs against it, streaming, tool-dispatch and provenance are all exercised deterministically. The real backends are covered by an optional integration test that points the same adapter at a live endpoint and skips cleanly (exit 0) when none is reachable, so the suite runs on a box with no LLM.

The MCP server: the real SDK client, a real browser for consent

You do not hand-roll a client to test the MCP server from Part 2 — you drive it with the official MCP SDK, so discovery, DCR, PKCE and JSON-RPC are the SDK's code, not yours. The SDK is async and its OAuth flow needs a browser; bridge both into the acceptance harness. The SDK performs the token dance and, at the authorize step, calls a redirect handler you supply — which hands the URL to Playwright to drive the real login and the real consent screen; the one-time code comes back off the redirect page's URL:

# the connector runs the REAL SDK on its own event-loop thread; its OAuth
# redirect handler is bridged to the behave main thread, where Playwright
# drives the actual login + consent UI, then reads the code off the URL bar.
oauth = OAuthClientProvider(
    server_url=f'{base_url}/mcp',
    client_metadata=OAuthClientMetadata(redirect_uris=[f'{base_url}/legal/'],
                                        token_endpoint_auth_method='none',  # public + PKCE
                                        scope='api:read api:write'),  # requested-but-vestigial: v3 wire tokens are scope-less (§24.4)
    storage=InMemoryTokens(),
    redirect_handler=lambda url: authorize_url_queue.put(url),   # → Playwright
    callback_handler=lambda: code_queue.get())                   # ← from the URL bar

@when('the connector sends me to the consent page')
def follow_authorize(context):
    context.page.goto(context.connector.authorize_url_queue.get())   # the one legitimate raw goto:
                                                                     # OAuth entry IS a redirect
@when('I approve access to "{blog}"')
def approve(context, blog):
    context.page.locator(f'label:has-text("{blog}")').locator('input').check()
    context.page.click('button[value="grant"]')
    code = parse_qs(urlparse(context.page.url).query)['code'][0]
    context.connector.code_queue.put(code)

The result is the entire Part 2 dance — 401 → discovery → registration → consent → token → authenticated tools/call — exercised hermetically on every run, with a real client and a real browser. A second, independent client guards conformance separately: the reference Inspector CLI with strict schema validation, pointed at a dev server with a pre-minted token.

The evidence: a screenshot gallery that cannot lie

Because the box is headless, capture each scenario's final state as a full-page screenshot and render them as a gallery from a small manifest the run writes. One trap to avoid: API-level scenarios (raw HTTP probes, in-process checks) never navigate the page, so a full-page shot is a meaningless blank — indistinguishable from a genuine blank-render bug. Detect them (the page never left about:blank), flag them in the manifest, and render a labelled "no UI under test" card instead of a screenshot. Now a blank image in the gallery always means a real rendering bug, never a by-design blank.

In this codebase: see it live at /staff/test.

Smoke: the real user path, on every deploy

The final layer is not a fake anywhere. On every deploy, in every environment including production, a pure-stdlib script signs in as a synthetic non-staff user and drives the real path end to end: liveness over TLS → login → create → read → delete → a real AI turn (assert at least one streamed token) → an email send. It is self-cleaning, so it is safe against production, and it is the ultimate "covered for real" entry — the row in the matrix that catches whatever the fakes could not.

The shape to copy

1. Unit: --keepdb + transaction rollback, MD5 hasher, scoped + fail-fast.
2. Acceptance: real browser, real WSGI path, fresh DB + serialized snapshot on a separate name, navigate-like-a-user.
3. Lock invariants with registry-derived and AST contract tests.
4. Drive the LLM with a deterministic fake; the MCP server with the real SDK + a real browser for consent.
5. Integration/boot against the real systems, run locally; smoke against the deployed system, run every deploy.
6. Keep the real-vs-faked matrix current — it is the definition of "the tests are enough".

Part 4 is the agent that the fake backend stands in for: a backend-neutral transcript, two LLM providers behind one adapter, and a sync engine that streams tools and effects.

← All posts