Tracing a hosted conversation

A claims assistant for a home insurance product is small enough to build in an afternoon and large enough to exercise everything: a customer uploads a photograph of a damaged appliance and a PDF of their policy schedule, describes what happened over several turns, and the assistant answers coverage questions using the policy document plus a hosted search over the insurer's published terms. Four turns, one uploaded file, one provider-executed tool — a tool the provider both defines and runs inside your turn, with no handler on your side.

The rule for this build is that nothing counts unless it is captured. Every request body and every response body goes to a file, verbatim, before anything else happens with it, because the whole point of building it is to answer questions about what persisted rather than to reason about what should have. The four findings below are all readable off that capture, and three of them are invisible in the SDK's return value.

The smallest build that can be interrogated#

Three objects exist on the provider's side before the first user turn: an uploaded file, a store that makes it searchable, and a conversation. The middle one is a vector store, the provider's own index over documents you have uploaded, which their hosted file search queries on the model's behalf so that you never run a retrieval system yourself. Each one is created by an API call, each returns an identifier, and each of those identifiers is now a thing your database has to know about — which is a foreshadowing of chapter 3 rather than an aside.

The three objects are created in that order because each is a precondition for the next. An uploaded file is bytes the provider holds and nothing more; nothing can search it yet, because searching text means comparing a query against passages that have already been cut out of the document and indexed. A vector store is where the result of that indexing lives, which is why the store is created with the file's identifier passed into it rather than the file being uploaded into the store. The conversation is independent of both and is created empty, before any turn exists, because its only job is to be the thing later requests point at. The dependency between the first two is not an idiom of one SDK; it follows from a document being unsearchable until something has indexed it, and any provider offering hosted retrieval has an object playing the store's part whatever it is called there.

import json, pathlib, time
from openai import OpenAI

client = OpenAI()
MODEL = settings.MODEL_ID          # pinned in config, not inlined; see chapter 7
TRACE = pathlib.Path("trace/CS-88213.jsonl")

policy = client.files.create(
    file=open("policy_schedule.pdf", "rb"), purpose="assistants"
)
store = client.vector_stores.create(
    name="case-CS-88213", file_ids=[policy.id]
)
conversation = client.conversations.create(
    metadata={"case_id": "CS-88213", "tenant": "acme-insure"}
)

case.provider_conversation_id = conversation.id   # our Postgres keeps the handles,
case.provider_file_ids = [policy.id]              # even though it no longer keeps
case.provider_store_id = store.id                 # the transcript
case.save()

The turn function is deliberately dull. It sends exactly one new message plus the conversation handle, declares the hosted tool, and writes both halves of the exchange to the trace file.

def turn(text: str) -> str:
    request = {
        "model": MODEL,
        "conversation": conversation.id,
        "input": [{"role": "user", "content": text}],
        "tools": [{"type": "file_search",
                   "vector_store_ids": [store.id]}],
        "store": True,
    }
    t0 = time.monotonic()
    resp = client.responses.create(**request)
    elapsed_ms = round((time.monotonic() - t0) * 1000)

    with TRACE.open("a") as fh:
        fh.write(json.dumps({"dir": "req", "body": request}) + "\n")
        fh.write(json.dumps({"dir": "res", "elapsed_ms": elapsed_ms,
                             "body": resp.model_dump()}) + "\n")
    return resp.output_text

That is the entire integration. There is no message table, no serialisation of prior turns, no retrieval code, and no HTTP client for the insurer's terms site — which is the appeal, and also why the trace file is the only local record that any of this happened.

Two lines of the turn function decide how much the trace is worth. It returns resp.output_text, which is one field selected out of a response body carrying many, so a service that logs its own return value has logged the assistant's prose and nothing else. The trace instead writes resp.model_dump(), the whole body as a dictionary, and writes it before the return value is computed and before any caller has had a chance to act on it. Three of this chapter's four findings live in fields output_text never touches — the usage counts, the tool-call item, and the absence of that item's contents — which is why the capture is worth the handful of lines it costs.

The two arrows that never touch your service are the ones this chapter is about. The read from the conversation store decides what the model was shown, and the file search decides what got added to it, and both happen between your request and your response with no interaction available to you.

Those two arrows have to exist, and the reason is the property that makes hosted state a design problem at all: the model is stateless, so the whole conversation must be present in the input of every call that depends on it. The turn-2 request in the diagram carries a handle and one new message. Everything else the model needs therefore has to be fetched and assembled by something between the request arriving and the model running, and the conversation store is the only object in the picture that survives from one turn to the next. The same argument settles where the retrieved text goes: a tool result that is to be useful on a later turn has to be written somewhere a later turn reads, and the diagram shows the conversation being appended to as the search returns.

Reading the trace instead of the docs#

Four turns went through: a description of the damage, a question about whether accidental damage is covered, a follow-up asking what the excess is, and a request to summarise the claim. Pulling the interesting fields out of the captured responses gives this.

Two of those columns come from opposite sides of the boundary, and reading them as though they came from the same side is where this table stops teaching anything. The user-text column is a fact about the request your service built, and you could have written it down without calling anybody. The token columns are the provider's own count of the input it assembled and the output it produced, reported back in the usage field of the response, and they are the only quantitative statement available anywhere about material you never held. Every finding below is an inference from the gap between those two.

The handle persists and is sufficient. Turn 3 sent nine words and the model answered a question that depended on the policy excerpt retrieved during turn 2, so both the earlier conversation and the earlier tool result were in context. Nothing in the turn 3 request mentioned the file, the store, or the previous answer.

A coherent answer on its own would not have established that, because a model that answered well might have been guessing from the question. The token count is what closes it. Turn 3's request carried nine words, and nine words cannot be six thousand one hundred and ten tokens under any tokenisation, since a token is a short word or a fragment of one and a nine-word message therefore runs to tokens in the tens rather than the thousands. The input the model was charged for was assembled from something other than the text in the request, and the conversation handle is the only field in that request capable of pointing at prior turns.

The retrieved chunks became part of the conversation. A hosted file search never hands the model a whole document; it returns chunks, the passages of a few hundred tokens each that the store cut the PDF into when it indexed it, and those passages are the material the answer was written from. Input tokens jump from 1,690 on turn 2 to 6,110 on turn 3 without the user sending anything of substance, and the difference — around 4,400 tokens — is the policy text the hosted search pulled in during the previous turn. Hosted retrieval is not a transient enrichment that evaporates when the turn ends. It is written into the stored conversation and re-billed as input on every subsequent turn, exactly as if you had pasted the chunks into your own message history. This is the single most expensive behaviour in the whole arrangement and it is invisible unless you are reading usage.

Two other explanations for that jump in input tokens are available, and the same table rules both out. It is not turn 3's own retrieval, because the table records that no hosted tool ran on turn 3. It is not the ordinary growth of a conversation either: the only new material between the two counts is turn 2's reply, which the table prices at 402 output tokens, and turn 3's nine-word question. Those cover a few hundred tokens of the jump between them and leave the great bulk of it unexplained by anything the customer or the model said. What remains is material that entered the conversation during turn 2 without appearing in turn 2's input count, which is what a tool result appended after the model ran would look like.

The mechanism underneath is worth stating apart from the observation, because it explains why the behaviour could not have been designed away without giving something up. A stateless model has the chunks in context on turn 3 only if the chunks are in the input on turn 3. The input on turn 3 is assembled by the provider from the conversation, since that is the only thing the request pointed at. Retrieved text that is to stay useful after the turn that fetched it must therefore become an item in the conversation, and an item in the conversation is input on every later turn by construction. The alternative, discarding the chunks when the turn ends, would mean the model losing the policy excerpt the moment it stopped talking about it.

The natural reading of a bill that charges for the same passage on every turn after the one that fetched it is that something is being counted twice, and it is worth blocking that reading before it settles. Nothing is charged twice for one piece of work. The model reads those tokens again on every turn, because reading the whole input is the only way a stateless function has of knowing anything, and the provider is charging each time for work it performed each time. What is surprising is the durability rather than the pricing: a retrieval step a reader would expect to be scoped to the turn that ran it has instead written into a store that outlives it, and that is a fact about where the result was put rather than about how it was billed.

The tool result came back without its contents. The response contains an item of type file_search_call with status: "completed" and the queries the model issued, and results: null. The retrieved chunks — the text that actually drove the answer and that now occupies 4,400 tokens of every future turn — are not in the response body by default. You get them only by asking for them explicitly at request time:

resp = client.responses.create(
    model=MODEL,
    conversation=conversation.id,
    input=[{"role": "user", "content": text}],
    tools=[{"type": "file_search", "vector_store_ids": [store.id]}],
    include=["file_search_call.results"],   # off by default
)

Every trace captured without that flag records that a search happened and not what it found. Chapter 7 is largely about what that costs three days after an incident; for now it is enough to note that the default for provider-executed tools is to return the conclusion and drop the evidence, and that turning the evidence back on is a per-request decision you have to have made in advance.

That default is defensible once you see what a response body is for, and seeing it is what makes the trap predictable rather than merely annoying. The body's job is to be read by your code and, under hosted state, to be fed back into the next request; the chunks are already in the provider's conversation, so returning them would be sending you material you will never need to resend. Nothing in that reasoning involves an audit trail, and nothing about it is wrong. The belief it defeats is the one carried over from application-owned state, where the request body your service serialised was the model's input and could be logged as such. Under hosted state the request is a subset of the input and the response is a projection of the turn, and neither one is a record of what the model was shown.

The fourth number is not in the table, because it is in the request rather than the response. On turn 4 the body your service serialised and put on the wire was 312 bytes: a model name, a conversation id, a store id, and seven words of user text. The application-owned version of the same turn sends the entire transcript plus the retrieved chunks, which for this conversation is about 34 kilobytes of JSON. The payload shrank by two orders of magnitude and input_tokens did not move at all, because the provider reassembled on their side exactly what you would have sent from yours. Hold onto that pair of facts; chapter 4 is built on the gap between them.

The two orders of magnitude are worth working out rather than accepting, because the arithmetic is what shows the two quantities to be unrelated rather than merely different. About thirty-four thousand bytes of JSON, divided by the three hundred and twelve bytes actually sent, comes to a factor of a little over a hundred. Division is the operation the comparison allows because both quantities measure the same thing — bytes serialised for one turn — so the units cancel and the ratio reads directly as how many times smaller the payload became. The token count did not shrink by that factor, or by any factor at all. What moved is the number of bytes your service put on the wire, and what did not move is the amount of text the model processed, which is the clearest available demonstration that network payload and billed input are independent quantities.

Latency separates cleanly along the same line. The turns where no hosted tool ran came back in a little over two seconds; the two that triggered a search took 5.8 and 6.9 seconds. That gap is not the model thinking harder. It is a retrieval round trip that happens inside your request, on a system whose queue depth you cannot see and whose timeouts you do not set, and it lands entirely inside the single latency number your service reports.

The reason it cannot be separated out is that the search happens between your request and your response. Your service made one call and is blocked on it, so the provider's retrieval, the model call that follows it, and any queueing in front of either are all inside that one wait. With a retrieval system you run yourself the same work is still serialised, but each stage is a span you can time, alert on and put a deadline on. Here the two-second turns and the six-second turns differ by a step you have no instrument for, which is why the trace records elapsed_ms per turn next to whether a tool ran.

What a captured run does not establish#

The mirror-image failure is quieter and no local code causes it. Providers apply their own compaction or truncation to long hosted conversations, and when it fires, input_tokens stops growing and may fall. A conversation whose usage went 6,110 → 6,530 → 6,980 → 4,120 did not get shorter because the user said less; something in the middle of it was summarised or dropped, on a policy you did not configure and are not told about at the moment it happens. The behaviour is reasonable — it is what keeps a long thread inside a context window — and it means the input to turn 12 is not a function of anything your system stores. Chapter 3 turns that observation into the replay problem.

Reasonable understates it: compaction of that kind is forced, which is why some version of it will be present with any provider offering hosted state. A context window is a fixed ceiling on the input of a single call. A conversation that keeps appending — user messages, assistant replies, and every chunk a hosted search deposits — has no ceiling. Those two facts cannot both hold indefinitely, so at some length the provider either refuses the call or removes material, and refusing would break a product whose entire selling point is that you no longer manage length. Removal is what is left, and a reader who expects to be told each time it happens is expecting a notification the arrangement gives them no reason to receive.

What the usage numbers cannot settle is which kind of removal occurred. A count that falls is consistent with old turns being dropped outright and equally consistent with them being replaced by a shorter summary, and the two differ in what survives: dropped turns are gone from the model's view entirely, while summarised ones persist as a paraphrase written by a model, with whatever it omitted unrecoverable. Telling them apart needs the items endpoint rather than the usage field, and even that shows only the current contents rather than the history of how they came to be that way.

Fetching the stored conversation back is worth doing once, deliberately, as part of this build, because it is the only way to see the provider's copy at all:

items = client.conversations.items.list(conversation.id, limit=100)
for item in items.data:
    print(item.type, getattr(item, "role", ""), str(item)[:120])

What comes back is a list of items — user messages, assistant messages, and the tool calls — and comparing it against your trace file is the check that matters. Anything present in the provider's copy but absent from your capture is state you are carrying without a local record of it. Anything present in yours but absent from theirs has been compacted away and will not be part of the next turn's input.

Two conditions on that comparison are worth stating, because it is easy to read more into it than it supports. The call above asks for a bounded page of items, and a conversation longer than the page you requested returns the page rather than the conversation, so a read-back that looks complete on a four-turn case can be silently partial on a long one. The endpoint also reports what the provider is currently willing to expose, at the moment you ask. It is not a log of what the model received on any particular past turn, and no sequence of calls to it reconstructs one, because the assembly step that turned those items into an input left no artefact on your side. A reviewer told that the conversation can be enumerated should ask to see the read-back for the longest conversation in production rather than for the demo.

One more probe is worth running while the conversation is still alive, because it settles a question people argue about. Drop the tools array entirely on turn 5 and ask a question that depends on the policy excerpt. It answers correctly. The hosted tool's results live in the conversation, so removing the tool removes the model's ability to run a new search and removes nothing that a previous search already deposited. The same holds if you point turn 5 at a different vector store: the old chunks are still there, from a store you are no longer referencing and may have deleted. Tool configuration governs future calls; the conversation governs what is in context, and the two are not the same lever.

What the trace does not do is worth stating, because a captured run has a way of feeling like a specification. It tells you what this provider did on this day at this conversation length, and nothing about the policy behind it. Compaction thresholds, whether a tool result is stored verbatim or summarised, how long an item stays retrievable through the items endpoint — none of those are things the trace establishes, and none are things you can hold the provider to on the strength of having observed them once. The trace is also not an archive. It is your record of the bytes that crossed the boundary, which is a different and much thinner artefact than the conversation itself; the provider's copy contains items your capture never saw, and your capture contains request bodies the provider has no reason to keep. Treating either one as the authoritative history of the conversation is a mistake in a different direction each time.

Building the smallest hosted integration that can be interrogated takes an afternoon, and the reason to do it before the design argument rather than after is that four of the load-bearing facts in this topic are observations rather than deductions. The handle carries the whole conversation. Hosted retrieval writes durable, re-billed text into it. The tool's own output is dropped from the response unless you asked for it in advance. And the input to any turn is assembled by the provider from material you no longer hold. Each of those changes what a later chapter is allowed to claim, and each is a line in a JSONL file rather than a position.

Those four facts are what makes the next question sharp rather than theoretical: given that the transcript, the retrieved chunks, and the uploaded file all live somewhere else, what exactly happens when a user asks you to delete them, or when a regulator asks what the model was shown on 14 March?