The state a check binds

A verdict about mutable state has a lifetime. Checks over things that do not change — whether a program type-checks against a pinned compiler, whether a JSON document satisfies a schema, whether line items sum to the stated total — return answers that are true indefinitely, and they can be run anywhere, cached, and replayed. Checks over live data return answers that are true for as long as the state they read holds, and the useful question about any such check is how long that is and what can invalidate it.

The distinction is invisible while a check stays where it was built, which is why it usually surfaces as a performance change. Somebody notices that the checker is expensive, moves it somewhere quieter, watches the latency fall, and ships. The code is identical, the deployment diagram has one arrow redrawn, and the set of outputs the check will accept has grown — silently, by the amount of staleness the new location carries.

Where the authority in a trial transaction came from#

Take the provisioning agent's check from the previous chapter: a candidate set of INSERT statements applied inside a transaction against production Postgres and then rolled back, with the foreign keys and unique indexes rendering the verdict. That trial was never valuable because it executed SQL. It was valuable because the thing rendering the verdict was the same arbiter that would render the verdict at commit time: the same unique indexes with the same rows in them, the same foreign keys pointed at the same parent table, the same triggers, and — this is the part that gets lost — the same transactions currently in flight. When another session has inserted finance@acme.example and not yet committed, a unique index holds that key speculatively, so a second insert of the same value blocks until the first session resolves and then fails if it committed. The trial transaction inherits that behaviour for free, which means it is correct about a class of conflicts that has not been written down anywhere yet.

The proposal that undoes all of this is a good one, and it is worth stating in its strongest form before it is taken apart. The trial always rolls back. It leaves nothing behind, changes nothing durable, and its net effect on the database is identical to having read some rows — so it is, in effect, read-only, and read-only traffic is exactly what a replica exists to absorb. Every clause of that is true. The conclusion does not follow, because "read-only" describes what the check leaves behind and the thing that matters is what the check depends on. This trial is not reading data in order to reason about it. It is putting a question to an arbiter and recording the arbiter's answer, and the arbiter's answer is a function of every row committed up to this instant plus every row currently being written by somebody else. A replica is a different arbiter holding a different set of rows. The code is unchanged; the authority is not, because authority was never in the code.

A read replica cannot reproduce either half of that. It is behind the primary by however much replication lag there is, and it has no visibility at all into uncommitted state on the primary, because uncommitted rows are not yet in the write-ahead log, the WAL, which is the ordered record of committed changes that the replica replays to catch up. Both gaps widen exactly when the system is busiest. This cluster's replica lag was 80 ms at p50 and 2.4 seconds at p95, and the p95 lag occurred under the same write-heavy load that produced the 900 ms checks in the first place, so the substitution trades a check that is slow under load for a check that is wrong under load.

Two requests a few hundred milliseconds apart is not an exotic scenario in a provisioning service; it is what happens when a customer double-submits a form, when an upstream retry fires, or when a bulk import runs two rows for the same organisation. What the replica check produces in these situations is a false accept, and a false accept is worse here than in an invoice extractor, because the loop is structurally incapable of catching it: the candidate passed, so no reason string was generated, no regeneration was triggered, and the failure surfaces later as an unhandled exception in the write path with no context about which of the eleven inserts in the candidate caused it.

Backend engineers already know this shape from access() followed by open(), where the permission you checked can be revoked between the two calls and the classic mitigation is to stop checking and just attempt the operation with the privileges you have. The generate-then-verify loop reintroduces exactly that structure, because a trial-then-commit is a check-then-act. Even against the primary the window is real: the rollback releases the locks, and the actual write happens in a new transaction some milliseconds later. In this service that gap measured about 4 ms, and over 90 days it produced two commit-time conflicts out of 1.6 million writes, which is a rate the team was happy to handle as an exception. The replica moves that window from 4 ms to somewhere between 80 ms and 2.4 seconds and removes uncommitted-state visibility entirely, which is a four-hundred-fold widening of the race plus a new class of collision.

Both ends of that comparison repay a moment's arithmetic. Two conflicts in 1.6 million writes is a bit over one in a million, which is the reason a four millisecond window is tolerable: for a collision to occur, two requests have to target the same key inside the same four milliseconds, and at this service's traffic that essentially never happens. Now scale the window. The replica's lag is twenty times the primary's window at the median and six hundred times it at the tail, so the headline four-hundred-fold figure is the tail-weighted version — and the tail is the right place to weight it, because replication lag peaks under exactly the write-heavy load that also makes two requests for the same key more likely. The two conditions arrive together. A race window that scales with load, in a system whose collision rate also scales with load, is not a rate multiplied by a constant; it is a rate multiplied by something that grows.

The second half is worse than the first and is not a matter of degree at all. Widening a window makes an existing race more frequent. Losing sight of uncommitted state removes a guarantee the primary was providing: two concurrent requests for the same key could not both be told yes, because the index held the key speculatively for whichever arrived first. On a replica, both can be told yes, and there is no lag figure at which that stops being true. One is a probability that got worse. The other is a property that is gone.

Ranking checks by how fast the state they bind moves is a more useful exercise than arguing about deployment topology, because it tells you directly which relocations are free:

The bottom row is the one that decides the provisioning case. Uniqueness over an email column in a service that provisions accounts is decided at a granularity finer than any replication topology can track, so there is no copy of the database against which the answer is knowable in advance.

There is one comparison generating the whole of that last column, and having it explicitly means you can fill the column in for a check the table does not mention. Put the copy's staleness next to the interval between changes in the state being bound. Where the staleness is far smaller than that interval, the copy is almost always identical to the original at the moment of the check, and running against it changes nothing. Where the staleness is comparable to or larger than the interval, the copy is routinely different in exactly the way the check is asking about. A pinned compiler has an interval measured in release cycles and a staleness of zero, so it is not really a copy at all. A structure-only dump has a staleness of hours against a schema that changes over days, which is close enough to be safe and close enough to need watching, hence the alarm on it. A unique index over a contended column changes on the same millisecond scale as the replica's lag, and often faster, which is the row that has no safe answer. Notice that the compiler and the unique index differ by more than a dozen orders of magnitude on this one axis, and not at all on any property of the checking code.

Make the commit the check where you can#

When the candidate output is a set of writes and nothing else, the cheapest correct design removes the trial entirely and lets the real transaction produce the verdict, catching the constraint error and feeding it back into regeneration the same way the trial's error text was fed back.

CONSTRAINT_SQLSTATES = {
    "23505": "unique violation",
    "23503": "foreign key violation",
    "23514": "check constraint violation",
}

def apply_or_reject(conn, candidate: CandidateWrites) -> Verdict:
    try:
        with conn.transaction():          # real write, not a trial
            for stmt in candidate.statements:
                conn.execute(stmt.sql, stmt.params)
    except psycopg.errors.IntegrityError as exc:
        kind = CONSTRAINT_SQLSTATES.get(exc.sqlstate)
        if kind is None:
            raise                          # not a verdict; a genuine fault
        detail = exc.diag.constraint_name or exc.diag.message_detail
        return Verdict(False, f"{kind} on {detail}")
    return Verdict(True, None)

This halves the database work per attempt, removes the race window to zero, and produces a reason string that is strictly more accurate than the trial's, because it is the error that actually occurred rather than an error that occurred under a slightly different state a few milliseconds earlier. Rejection is still cheap, since a failed transaction rolls back on its own and the candidate is regenerated exactly as before.

Each of those three claims is visible in the code, and it is worth pointing at the lines. The halving is the disappearance of a transaction: the trial version opened one transaction to ask and a second to act, and this version opens one. The race window closes because there is no longer an interval between asking and acting for anything to change in — the question and the action are the same event, which is the only construction that removes a race rather than shrinking it. And the reason string comes from exc.diag, so it names the constraint the database actually violated on the statements actually submitted, where the trial's string named a constraint violated by a different transaction operating on a slightly older database.

The line that carries the most weight is the raise in the middle, and it is easy to read as boilerplate. CONSTRAINT_SQLSTATES lists three error codes, and those three are verdicts: the candidate is well-formed and the data says no. Every other error — a connection dropped, a deadlock, a permission problem, a disk full — is a fault, meaning something went wrong that says nothing about the candidate. Mapping the whole IntegrityError family to Verdict(False, ...) would be the natural shortcut, and it would convert a database outage into a stream of rejections, which the loop would respond to by regenerating three times per request and then escalating everything to a human queue. The service would report a quality problem while having an availability problem, and every instrument pointed at the loop would agree with it. The discipline that avoids that is to enumerate the codes that constitute an answer and re-raise everything else, because a checker's contract is to return a verdict or fail, never to turn its own inability to decide into a "no".

The reason the trial version existed is that this only works when the whole candidate is transactional. The provisioning agent's output was not, at the point the design was written: accepting a candidate also enqueued a welcome email and called a billing provider's API, and neither of those unwinds. Getting a verdict before any of that fires is worth a round trip. The clean resolution is to make the transactional part authoritative and everything else consume its outcome — write the rows and the outbox row in the same transaction, and let the email and the billing call be driven by the outbox after commit. That is ordinary transactional outbox work, the outbox being a table of pending side effects written inside the same transaction as the data and drained by a separate worker once the commit is durable, and it converts a check-then-act into an act-then-propagate, which is the same trick as attempting open() instead of calling access().

Following why that dissolves the problem is worth the two extra sentences, because the outbox looks at first like a way of organising background jobs rather than a correctness argument. The reason the trial had to exist was ordering: the email and the billing call fire on acceptance, neither can be undone, and so the system needed to know the answer before it acted. Writing an outbox row inside the same transaction as the data changes what triggers those side effects. They are no longer triggered by a decision to accept; they are triggered by a row that only exists if the commit succeeded. The commit is therefore free to be the decision, because nothing irreversible can now happen ahead of it. The requirement for a prior verdict was never about verification at all — it was about the irreversible steps being wired to the wrong event.

Copies that are the same artefact#

None of this is a claim that verification belongs in production. Plenty of good checks run nowhere near it: a migration harness that runs against a scratch database restored from a structure-only dump is correct precisely because a migration binds the schema, which changes on a timescale of days and is under the team's own control. The scratch copy is not an approximation of the real arbiter, it is the same artefact, and the failure mode it lets through is schema drift between the dump and production, which is why that harness refuses to run if the dump is more than 24 hours old.

Nor is it an argument against caching, replicas or staging environments, which exist for good reasons and stay useful. What changes is the accounting. A replica-backed pre-filter followed by an authoritative commit-time check is a perfectly sound design, and its soundness comes entirely from the second half. What is not sound is deleting the authoritative check and keeping its name.

Where the candidate genuinely cannot be attempted — an agent that files a wire transfer, sends a message to a customer, or deletes an S3 prefix — the trial against live state remains the right pattern, and its verdict's lifetime becomes a design constraint you carry explicitly. That usually means the accepted candidate has to be committed quickly, that the irreversible step re-asserts whatever it can at execution time, and that the loop's error handling assumes the verdict can be stale rather than treating a post-acceptance failure as impossible.

A checker is defined by what it reads, so its location is part of its specification rather than a deployment detail. The version of the question worth asking during any optimisation is not "does this still run the same code" but "does this still bind the state that will decide" — and a checker that gets faster while simultaneously accepting more is reporting something about itself, which is rarely good news.

Relocating a check is one way to change its accept set while believing you have changed only its cost. Replacing it is the other, and the version that comes up in every design review is swapping a narrow deterministic check for a larger model asked to review the output, on the grounds that the model can see semantics the deterministic check cannot. That trade is real and it is sometimes worth taking, but the properties being given up are the ones that made the check worth building — reproducibility, independence from the generator, and a cost small enough to run on every candidate including the rejected ones. The next chapter prices that swap.