Server-owned conversation history

Conversation state is everything a chat has accumulated so far — the customer's opening complaint, the order number they gave in the second message, the replacement the assistant offered in the fourth — kept in some form that lets the next request carry it. Somebody has to keep it because of a property of the chat endpoint that this stage of the book has already established: the endpoint is stateless, meaning the provider retains nothing about a call once it has answered and does not associate your next call with your last one in any way. A turn here means one user message and the assistant reply it draws, and the array of turns that goes over the wire is rebuilt from scratch on every call by a step this topic will keep calling history assembly.

The word "stateless" is carrying more weight there than it looks, and the mechanism underneath it is worth having in front of you, because every design decision in this chapter follows from it. A language model scores the next piece of text given the text in front of it, and the text in front of it is whatever arrived in the request body. There is no store the model consults alongside that body, and no identifier in the request that would let it find one. A fact stated in the first message is therefore available when the ninth is answered under one condition only: the bytes carrying that fact are physically present in the ninth request. The assistant that still knows the order number is not remembering it. It is reading it, in the same pass and from the same input as the sentence the customer has just typed.

That converts a memory problem into a transport problem, which is the easier of the two to get right, because nothing has to be recalled and something merely has to be reassembled and re-sent. It also settles what the next problem will be. The outbound array grows with every exchange, since turn nine carries turns one through eight in full as well as turn nine, and what to do when a conversation outgrows what the model will accept is a question this topic returns to later.

The problem that state solves is narrow and unavoidable, because without it there is no multi-turn conversation at all, only a series of unrelated single questions. Every product with a chat surface meets it: a support assistant that has to know which order is under discussion, a coding assistant that must remember which file you were editing, an internal reporting tool where the second question is "and for last quarter?". What differs between such products is ownership — where the turns are stored, who is permitted to change them, and which code assembles them into the outbound array. The version worth learning first is the one where a table on your own database is the record, because it is the arrangement in which every one of those three questions has an answer you can point at, and it takes three pieces and about forty lines to build.

The three pieces#

Storage first. One row per turn, with an explicit position, scoped to a conversation that is scoped to a user.

CREATE TABLE conversations (
  id         uuid PRIMARY KEY,
  user_id    uuid NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE messages (
  id              bigserial PRIMARY KEY,
  conversation_id uuid NOT NULL REFERENCES conversations (id),
  seq             integer NOT NULL,
  role            text NOT NULL CHECK (role IN ('system', 'user', 'assistant')),
  content         text NOT NULL,
  created_at      timestamptz NOT NULL DEFAULT now(),
  UNIQUE (conversation_id, seq)
);

CREATE INDEX messages_conversation_seq_idx ON messages (conversation_id, seq);

The seq column is doing work that created_at cannot. Postgres timestamps have microsecond resolution, so two inserts landing in the same microsecond is unlikely but not impossible, and more to the point ORDER BY created_at has no defined behaviour for ties — the planner is free to return them in either order, and it will happily return them in different orders on different days depending on which plan it picks. An integer position with a uniqueness constraint on (conversation_id, seq) gives the ordering a name and makes a colliding write fail loudly instead of quietly producing an ambiguous conversation. That constraint will matter a great deal once two requests arrive on one conversation at the same moment, which is chapter 4; for now it is cheap insurance.

The insurance is worth pricing, because "fail loudly" is a mechanism here rather than a hope. The uniqueness constraint is enforced by an index that physically cannot hold two entries for the same (conversation_id, seq) pair, so a second insert claiming a position already taken cannot be committed, and the transaction around it aborts with an error the handler is forced to deal with. Take the constraint away and that same insert succeeds. The table then holds two rows claiming one position, and ORDER BY seq has a tie to break with no rule for breaking it — which is the created_at problem again, except now it is written to disk permanently rather than resolved differently by one query plan and the next.

The read is the second piece, and its shape is the whole of the correctness argument.

SELECT role, content
FROM messages
WHERE conversation_id = $1
ORDER BY seq ASC;

Two clauses, both load-bearing. The WHERE scopes to one conversation, which is the difference between assembling this chat and assembling a blend of every chat the user has ever had. The ORDER BY is what makes the array an array rather than a bag; the model reads position as sequence, so a permuted history is a different conversation, not the same conversation shuffled.

That last claim reads like a convention and is a consequence of what the request body is. The array is serialised into a single piece of text before the model scores anything, with each entry's role marker and content laid down in array order. Two arrays holding the same messages in different orders therefore serialise to two different pieces of text. Different text produces different scores, so the reply that comes back is not the same reply reached by another route — it is the reply to a conversation that did not happen. The concrete form this takes is an assistant answering the question before last, because a swap has left the model reading the customer's second message as though it followed the reply to their third.

The third piece is the loop, and it fits on a screen.

SYSTEM_PROMPT = {
    "role": "system",
    "content": "You are a support assistant for Northwind Returns.",
}

def handle_turn(conn, conversation_id: str, user_text: str) -> str:
    with conn.transaction():
        next_seq = conn.fetchval(
            "SELECT coalesce(max(seq), 0) + 1 FROM messages WHERE conversation_id = $1",
            conversation_id,
        )
        conn.execute(
            "INSERT INTO messages (conversation_id, seq, role, content)"
            " VALUES ($1, $2, 'user', $3)",
            conversation_id, next_seq, user_text,
        )

    rows = conn.fetch(
        "SELECT role, content FROM messages WHERE conversation_id = $1 ORDER BY seq ASC",
        conversation_id,
    )
    messages = [SYSTEM_PROMPT] + [{"role": r["role"], "content": r["content"]} for r in rows]

    response = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
    reply = response.choices[0].message.content

    with conn.transaction():
        conn.execute(
            "INSERT INTO messages (conversation_id, seq, role, content)"
            " VALUES ($1, $2, 'assistant', $3)",
            conversation_id, next_seq + 1, reply,
        )
    return reply

Three details of that function are load-bearing and easy to read past. coalesce(max(seq), 0) + 1 is there because max() over an empty set of rows returns NULL rather than zero, and NULL plus one is NULL, so a conversation on its first turn would otherwise try to insert a row with no position at all; coalesce supplies the zero that a first turn needs to count up from. The user's message is written before the model is called rather than after, which means a request that dies in flight leaves the question recorded and the answer missing — a gap that is visible in the rows and repairable, where the opposite order loses the question and leaves nothing to repair from. And the assistant's row goes in at next_seq + 1 rather than at a freshly recomputed maximum, because rows may have been appended by another request since the read, and recomputing would place this reply after messages the model was never shown. Using next_seq + 1 instead makes that situation collide with the uniqueness constraint, which is the loud failure the schema above bought.

The system prompt is prepended at assembly time rather than stored as row zero, which is a deliberate choice with a trade-off: prepending means a prompt edit takes effect on the next turn of every live conversation, and storing means each conversation keeps the prompt it started with. Support tools usually want the second, because a conversation whose instructions changed halfway through is hard to reason about after the fact. Either is defensible; what is not defensible is doing it inconsistently, so that some conversations carry a stored prompt and some get a prepended one and nobody can tell from the row which is which.

Behind the preference for storing it is a mechanism worth stating, because "hard to reason about" is a complaint and not yet a reason. The rows are the only durable record of what happened, and a prepended prompt is not a row. Nothing in the table says which text occupied the first slot at turn three, so an attempt to reconstruct that turn later assembles today's prompt in front of yesterday's messages, and the reconstruction looks entirely plausible while being an input the model was never given. Storing the prompt as row zero costs one row per conversation and makes the record complete, which is why it is the usual answer wherever anybody expects to have to account for a past reply.

Every arrow that leaves the pod carries everything the next hop needs. Nothing survives in the pod between the last arrow and the first arrow of the next turn, which means the conversation state lives in exactly one place and the process is as stateless as the provider is.

The test is byte-identical, not merely coherent#

A conversation that reads correctly is weak evidence. If the reply to turn two references the order number from turn one, that is consistent with an array rebuilt from rows and equally consistent with a warm in-process dictionary that happened to survive. The observation that actually discriminates is stronger and easy to run: kill the process between turn one and turn two, then compare the serialised request body your handler sends for turn two against the body it sends without the kill. If assembly reads durable rows and nothing else, those bodies are identical byte for byte, because the same rows in the same order produce the same JSON.

The comparison is between bytes rather than between meanings, because the model is a function of the exact text it is handed and not of the conversation a person would summarise from it. Two arrays differing by a stray space, a transposed pair of turns or one missing exchange are two different inputs, and there is no level at which they are the same conversation assembled slightly differently. Holding the design to equality of the serialised body is therefore not fussiness about formatting. It is the only comparison with the same resolution as the thing being compared.

Making that checkable takes a few lines. Serialise the outbound array canonically and hash it before the call:

import hashlib, json

def payload_fingerprint(messages: list[dict]) -> str:
    canonical = json.dumps(messages, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()[:16]

log.info(
    "assembled",
    conversation_id=conversation_id,
    turns=len(messages),
    fingerprint=payload_fingerprint(messages),
)

Both arguments to json.dumps are there to remove a difference that is not a difference. Two dictionaries holding the same keys and the same values can serialise to different strings if the keys come out in a different order, or if the separators carry different whitespace, and either would make two fingerprints disagree while the conversation behind them was identical. sort_keys=True fixes the order and the explicit separators fix the spacing, so what is left is a string determined by the data alone. Hashing that string then compresses it to something short enough to log on every request, and truncating the digest is acceptable here because the comparison is between two values you produced minutes apart while looking for an accidental difference, not a defence against anybody trying to manufacture a match.

Then the check is a shell loop rather than an argument in review:

# turn 1
curl -s -XPOST localhost:8080/chat -d '{"conversation_id":"c-7f3a","text":"Order 88231 arrived cracked"}'
kill -9 "$(pgrep -f 'uvicorn app:api')"; sleep 2; ./run_api.sh &
# turn 2
curl -s -XPOST localhost:8080/chat -d '{"conversation_id":"c-7f3a","text":"What did I say the order number was?"}'
# compare the logged fingerprint against a run with no restart

Two fingerprints, one number. If they differ, something in the process contributed to the array, and the interesting question is what. In practice the answer is a memoised loader, a request-scoped object that turned out not to be request-scoped, a lazily-initialised summariser that caches its output, or a "warm start" optimisation added during a latency push. The fingerprint finds all of them without anybody having to reason about the code.

Running two instances is the same test with the kill replaced by a load balancer. Send turn one to pod A and turn two to pod B; the fingerprint for turn two should be what pod A would have produced. A design that passes the restart test but fails the two-pod test usually has state in a place that survives requests but not machines, which is to say in the process after all.

What neither version of the check can see is state that survives machines. A shared cache holding assembled arrays returns the same bytes to every pod and through every restart, so both fingerprints match while the array is still not a function of the rows — it is a function of what was last written to that cache and of whether anything has invalidated it since. The restart check rules out the heap and the two-pod check rules out anything pinned to one host, and a reviewer who wants the property itself rather than two of its symptoms should ask for a third demonstration: change a row directly in the database between two turns, and confirm that the next fingerprint moves. If it does not, something between the table and the model is answering from a copy.

The cache that puts the state back#

The change that undoes all of this arrives with a good motive. History assembly runs on every turn, and on a busy conversation that is a database round trip per request; someone profiling the endpoint sees 4 ms of Postgres time and 40,000 assembly queries an hour, and adds a sixty-second TTL cache in front of it, keyed by conversation id, holding the assembled array. Read throughput drops, p50 improves by a few milliseconds, the change reviews cleanly because it touches one function.

Traffic is spread across four pods with no session affinity, so consider what a conversation's next few turns do. Turn 3 lands on pod A, misses, reads six rows, sends them, caches the six-message array. Turn 4 lands on pod C, misses, reads eight rows, sends them, caches its own eight-message array. Turn 5 lands back on pod A, twenty seconds later, and hits: pod A serves the six-message array it cached, which does not contain the user's turn-4 question or the assistant's turn-4 answer. The model receives a conversation that stops two turns ago and answers accordingly, so the user watches the assistant ask something it was already told, or repeat a suggestion it already made, or forget a decision that was reached in the exchange that just happened.

The failure rate in the table below falls out of the pod count. With no session affinity, each turn is routed without reference to where the last one went, so it lands on any given one of the four pods one time in four. Early in a conversation only one pod is holding a copy, and the chance of coming back to that pod is exactly that figure, which is the step in the walkthrough above where turn 5 comes back to pod A. The word "roughly" in front of the figure in the table is carrying two conditions. A hit only serves stale content while the cached array is still inside its sixty-second window, so a conversation whose turns are further apart than that misses every time and never sees the bug at all. And the rate drifts upward as a conversation runs on, because every pod that serves a turn leaves a copy behind, and a long conversation has more pods holding something out of date than a short one does. Both conditions point the same way: the bug is worst on a conversation being typed quickly, which is the same sentence as saying it is worst while a human is waiting.

The last two rows are why this bug is expensive. A single-pod local environment never reproduces it, because the one cache is always the freshest. Load tests do not catch it, because synthetic conversations rarely check that turn 5 knows what turn 4 said. It surfaces as user reports of the assistant "being forgetful", which routes the ticket to whoever owns prompts and models rather than to whoever owns the cache, and chapter 3 is about how much time that misrouting can burn.

The distinction to hold is between caching the input to assembly and caching its output. Caching rows under a key that includes the conversation's current message count, or invalidating on write, keeps assembly a function of durable state. Caching the assembled array under a bare conversation id asserts that the conversation has not changed since you last looked, and on a system where any of four pods can append a turn, that assertion is false whenever the conversation is active — which is exactly when anyone is reading it.

The belief a cache of assembled arrays rests on is that a conversation is a thing you can hold a copy of, and it is tempting because the copy and the rows agree for nearly all of a conversation's existence. Between turns nothing is appended and the two are identical. They diverge at one moment only, the moment a turn is written, and that is the moment every subsequent read has to survive. Measured over wall-clock time the cache is almost always right, which is what makes it profile so well. Measured over the reads that matter it is wrong whenever the conversation has moved since that pod last looked, and on an active conversation that is most of them.

None of this forbids in-process state generally. Connection pools, compiled regexes, tokeniser instances and prompt templates all live in the process quite happily, because none of them is per-conversation data whose staleness changes what the model sees. The line is narrower than "no caching": no piece of the outbound array may depend on which process handled the previous turn.

Owning the history yourself is also not the only option on the market. Several providers now offer server-side conversation state, where you send one message plus a handle to the previous response and their infrastructure carries the rest. That is a real product decision with real benefits, mostly around latency and payload size, and it is not the same claim as "the API is stateless". It relocates ownership rather than removing it, and the questions it raises are ownership questions — what retention applies to the stored turns, whether you can enumerate and export them, what happens to an evaluation pipeline when the input you want to replay lives in someone else's store, and whether an expired handle leaves you any way to reconstruct the conversation. OpenAI's own Conversation state guide is direct about the boundary between what the API remembers and what you must carry, and is worth reading with those questions in hand; topic 2.4 takes them one at a time.

The word this chapter opened with travels further than it should, so it is worth fixing its scope before it leaves. The definition used above — the provider retains nothing about a call once it has answered — is the operative sense for every argument here, and it is narrower than the words sound. What it claims is that nothing from an earlier call influences a later one, so no fact you sent yesterday can reappear in an answer today by a route you did not build yourself. It does not claim that requests go unlogged, unretained or unread, and those are separate questions with different answers per provider and per contract. Every argument above depends on the first claim and none of them touches the second.

Multi-turn conversation is a table, an ordered read scoped to one conversation, and an array rebuilt inside the request handler, and the reason to write it that way rather than to hold a session object is that the outbound array then depends on nothing but durable rows. That dependency is checkable rather than arguable: fingerprint the serialised array before each call, restart the process between two turns, and see whether the numbers match. Every optimisation that puts state back — a session dictionary, a warm cache of assembled arrays, a memoised loader — breaks that equality first and produces confused customers second, which is why the fingerprint is worth having before anybody profiles the endpoint.

What the design assumes, silently and throughout, is that the server owns the table. Every guarantee here — the fingerprint check, the ordering, the ability to hand the same array to an evaluation job tomorrow — rests on the history being your record rather than something handed to you at request time. The obvious way to make the backend simpler is to drop that assumption and let the browser keep the conversation, posting the full array with each request. It is a tempting design and it is not obviously wrong, so the next chapter takes it seriously enough to work out precisely what changes when the array arrives from outside.