Hosted conversation state and hosted tools

The previous topic established the mechanics of conversation state: the model is stateless, so somebody has to hold the transcript and resend it on every turn, and the interesting question is who. This topic is about what changes when the answer is "the provider", and about a second primitive that usually rides along in the same commit because it is switched on from the same request body.

Two terms carry the chapter. Provider-hosted conversation state is the arrangement where the first call creates a conversation object inside the provider's system, the provider returns an identifier for it — a conversation handle, an opaque string of the form conv_68d1f4… — and every later turn sends that handle plus the single new user message. The provider looks up the stored thread on their side, assembles the full input from it, and runs the model against the result. A provider-executed tool, called a server tool in most vendor documentation, is a capability the provider both defines and runs: you switch it on by naming it in the request, you never write a handler for it, and the execution happens inside the provider's infrastructure while your turn is in flight. One relocates data. The other relocates an action, and the review that catches the first does not catch the second.

What each line replaced#

Both changes fit in a diff of a few lines, which is part of why they travel together.

- messages = db.load_messages(thread_id)          # our Postgres, our backups
- resp = client.responses.create(model=M, input=messages + [turn])
+ resp = client.responses.create(model=M, conversation=thread.provider_id,
+                                input=[turn])

  tools = [
      issue_refund_tool,
+     {"type": "web_search"},
  ]

The first change means the transcript of every conversation now lives in the provider's storage rather than yours: their retention schedule, their region, their deletion semantics, their idea of how long an object sticks around after you ask for it to go. The second means that partway through a turn, on the provider's network, under the provider's egress identity, an HTTP request goes out to a page nobody at your company chose, and the text of that page lands in the same context window as your system prompt.

Nothing about the model's statelessness changes when the transcript moves. The model still has to be shown the whole conversation in one input, because a stateless function has no other way of knowing what was said three turns ago, and the previous topic established that this is a property of the model rather than of the API in front of it. What the conversation field relocates is the step that concatenates stored turns into that input. Your code used to read rows and build an array; the provider now reads their own objects and builds the same array. The bytes the model is shown are the same bytes either way.

That is worth settling early, because the natural reading of a handle replacing an array is that the provider has given the model a memory. It has not. The handle is a pointer to storage, the storage is read on every call, and the assembled input is as long as it ever was. Chapter 4, which prices the migration on a twelve-turn workflow, only makes sense once that reading has been discarded — a change that removes an array from your request body and removes no tokens from the provider's input is a change that can make the bill go up.

Hosted state exists because the plumbing it removes is real, dull, and easy to get wrong. Owning the resend means a messages table, an ordering, a serialisation step, a policy for what to drop when the thread outgrows the context window, and a migration every time the message format changes. Hosted state is the provider offering to run that code instead of you, and the offer is attractive for the ordinary reason that nobody's product is differentiated by its message-serialisation layer. You meet it as a field on a request rather than as a product you install: OpenAI's Responses API accepts a conversation id, or a previous_response_id that chains each call to the one before it; Anthropic's Messages API takes the whole message array and leaves the transcript with you, while their managed-agent products hold it on their side. Whatever the field is called in a given SDK, the tell is that the request body no longer contains the conversation and something outside your system does.

Hosted tools exist because the alternative is operating the capability yourself, which for search means an integration, an API key and a crawl budget, and for code means a sandbox with resource limits, a filesystem and a network policy — infrastructure few teams want to run in order to answer a question about a delivery date. Claude ships web_search, web_fetch and code_execution this way, and OpenAI ships hosted file search and a code interpreter. Where you meet them is the same tools array that already holds the functions your own code implements. An entry carrying a type the provider recognises, and no implementation on your side, is a hosted tool. An entry carrying a name, a description and a JSON schema you wrote is a tool that will come back to you as a request to execute. Both are a few lines of JSON in the same array, and everything else about them differs.

What a data retention review actually asks#

Hand the conversation to the provider and you have made a storage decision, in the ordinary sense that a compliance team already knows how to reason about. There is a body of user-authored text — questions, order numbers, symptoms, whatever your product collects — and it now rests in a system you do not operate. The questions that follow are the same ones you would ask about any third-party datastore, and the awkward part is that the answers are contractual rather than technical.

Retention is the first: how long does the provider keep a conversation object after your last call, and can you shorten that? Most providers publish a default window measured in days or weeks and offer a zero-retention or short-retention tier under specific agreements, sometimes with features excluded from it. Residency is second: which region the object rests in, and whether "region" describes storage only or also the inference that reads it. Deletion is third, and it is the one that surprises people, because deletion of a hosted conversation is an API call you have to make, log, and be able to evidence, rather than a row disappearing when your ON DELETE CASCADE fires. Replay is fourth: given a run from six weeks ago, can you reconstruct the exact bytes the model was shown? Chapter 3 takes that one apart properly, because the honest answer is worse than it first looks.

Residency is the hardest of the four to close out, and it is worth saying so rather than leaving a reviewer feeling slow. A region name answers where an object is written. Whether the inference that reads that object also runs there is a separate fact, it is not always stated in the same document, and the two can differ without anything being wrong from the provider's point of view. A reviewer who has a region in writing and stops there has closed out half of what their obligation probably asks.

Deletion is worth slowing down on next, because the reason it behaves differently is structural rather than a matter of vendor generosity. ON DELETE CASCADE works because the row being deleted and the rows referencing it sit inside one transaction domain that your database controls. The engine can see the foreign key, it can take the locks, and it can guarantee that either every dependent row goes or none of them does. A conversation object in a provider's system participates in none of that. It holds no foreign key into your schema, your transaction never reaches it, and your commit cannot fail on its account. Deleting your local record therefore propagates nowhere, and erasure becomes an outbound request instead — one that can time out, can be retried, and can succeed for the conversation while failing for an uploaded file.

That is also why the evidence has to be manufactured rather than observed. The proof that a row is gone is the row's absence, which anybody can check at any later date by looking. A remote object leaves no such artefact behind, so the only thing that ever existed on your side is the fact that you asked and were told yes — and if nobody recorded that at the time, it cannot be recovered afterwards. Chapter 3 turns this into an operational problem of several object types and several calls.

The numbers attached to those four questions are the sort you have to look up per provider and per contract rather than reason about, and they move. As of the current documentation the shapes are: a default retention window in the region of thirty days for stored conversation objects and uploaded files, a zero-retention arrangement available to enterprise agreements that switches off some features entirely — a model may refuse every request from an organisation configured below its required retention floor — and deletion endpoints that return success synchronously while the provider's own documentation describes purge from backups on a longer schedule. A questionnaire answer of "we support deletion" is compatible with all of that. What your compliance obligation usually needs is narrower: the date the object stops existing for retrieval, the date it stops existing at all, and a record on your side that you asked.

Two answers a questionnaire commonly returns are easy to accept as answers to the retention question and are not. The first is that your data will not be used for training. That is a statement about what the provider may do with the bytes, and the obligation driving most of these reviews is about how long the bytes exist and where; an object excluded from training rests for exactly as long as one included in it. The second is that traffic is encrypted in transit and at rest, which describes who can read the object and says nothing about when it stops being there. Both answers are worth having. Neither one shortens a retention window by a day.

None of the four questions above have anything to do with what a hosted tool does. Enabling the provider's web search adds no user-authored data to their storage that was not already in the request — the user typed what they typed, and the tool did not make them type more. That claim needs its condition attached now, because the two primitives interact once both are switched on: the tool's own retrieved output is written into the stored conversation and re-read on every later turn, which chapter 2 establishes from a capture rather than from an argument. What the tool adds to the review is not a second datastore but an actor.

The second line adds an actor, not a datastore#

Because the tool is declared and never implemented, the call and its result both happen inside the provider's turn. You write down that the model may search the web, or run code, or search a file store you uploaded to, and from then on you do not see a request to execute anything, you do not decide whether to allow it, and the first evidence you have that it happened is a result block in the response.

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=4096,
    tools=[
        {"type": "web_search_20260209", "name": "web_search"},
        {
            "name": "issue_refund",
            "description": "Issue a refund against an order.",
            "input_schema": {
                "type": "object",
                "properties": {"order_id": {"type": "string"},
                               "amount_cents": {"type": "integer"}},
                "required": ["order_id", "amount_cents"],
            },
        },
    ],
    messages=[{"role": "user", "content": "Where is my order 4471?"}],
)

Both entries in that tools array are called tools and they are opposites. issue_refund comes back to you as a request: the response stops with stop_reason: "tool_use", your code decides what to do, and you send a result. web_search does not come back at all — the provider runs it, and resp.content contains a web_search_tool_result block whose content is text your process never fetched, from a domain your process never resolved, retrieved through an egress path your network team has never seen. Chapter 5 walks the first loop in detail; the point here is that they sit on adjacent lines of the same request body and belong to different threat models.

The reason one of them comes back and the other does not is a single branch inside the provider's serving loop, and every consequence in this section follows from it. Both entries reach the model the same way, as descriptions of available capabilities rendered into the input. The model, generating one piece of text at a time, can emit a sequence that the serving loop recognises as an invocation of one of them. What happens next depends on whether that loop has an implementation for the name it has just read. For issue_refund it does not: you supplied a schema and no code. The only thing the loop can do with a name it cannot execute is stop generating and hand the turn back to you, and that halt is what stop_reason: "tool_use" reports. For web_search the loop does have an implementation, sitting on the provider's side of the API boundary. It therefore never needs to stop. It calls that implementation, appends the result to the input it is already holding, and carries on generating from the longer input.

That inner loop is the whole of the difference. Your turn is one HTTP request and one HTTP response, and everything between them — the decision to search, the fetch, the retrieved text entering the input, the continued generation — happens without the connection ever yielding control back to you. There was no moment at which your process was asked. There was therefore no moment at which it could have refused, and the absence of an approval point is not an oversight in the design. A provider-executed tool is defined by the property that its execution does not require you.

Concretely: a user asks about a shipping delay, the model decides a carrier status page would help, and eleven seconds later your response contains 12 kilobytes of text from a domain you have never heard of. Your application emitted one HTTPS request, to the provider, which is the only line in your egress log. The carrier page — and the three unrelated pages the search returned alongside it — do not appear anywhere in your infrastructure's record of what happened, because as far as your infrastructure is concerned nothing happened except one API call that took eleven seconds instead of three.

Those eleven seconds are the only trace, and that is the part which defeats the instinct to go and look. A tracing system working exactly as designed shows one span for the turn, correctly attributed, with a duration on it. Any span it could have shown for the inner loop would have had to come from an instrumented call in your process, and there was no call in your process to instrument. The signal that something else happened is therefore a latency distribution with a second hump in it, which nobody reads as "a third party was contacted", and which is indistinguishable from the provider having a slow afternoon.

The consequences are concrete enough to enumerate. Your egress logs do not contain the fetch, so if the question is "did anything in this system reach paste.example", your own network data cannot answer it. Your allowlists do not apply, because the allowlist your infrastructure enforces governs traffic leaving your VPC. Your rate limits do not apply. And the retrieved bytes — which is the part chapter 6 is built around — arrive inside the same context as your instructions, with nothing in the message format distinguishing text you wrote from text a stranger wrote.

The first three of those are one property stated three times. A log shipper, an allowlist and a rate limiter are all enforced at a boundary — the edge of the network segment your own services run inside — and a boundary can only govern what crosses it. The fetch to the carrier's status page originates inside the provider's network and terminates at the carrier. No part of it comes near a boundary of yours. The failure is therefore not that the allowlist was evaded. It is that the allowlist was never consulted, because nothing arrived at that boundary for it to have an opinion about. The distinction matters as soon as somebody proposes a fix, since any fix placed at your own perimeter is answering a question about traffic that was never there.

The fourth item is not a variation on the other three, and reading it as one is the mistake chapter 6 is written to correct. The first three describe a control that was absent. The fourth describes something that arrived: text chosen by a stranger, now sitting in the model's input alongside the instructions you wrote, with the message format offering nothing that marks which is which. Moving the perimeter would not touch it, because the text has already crossed every boundary there was by the time you see the response.

What can be said about the provider's side is narrower than it is comfortable to assume. Whether a given provider exposes any control over what its own tool may reach, and whether anything it reached is recoverable after the turn, is a per-provider question with a per-provider answer. The step to insist on is that somebody produce the documentation for the specific tool you enabled, on the tier you are actually on, rather than reasoning from the category — chapter 7 is about what happens when the answer turns out to be that the apparatus was discarded when the turn ended.

The bottom two rows are where the asymmetry lies. Turning off a hosted tool is a one-line revert that takes effect on the next request. Undoing hosted state means the six months of conversations already in the provider's store either come back to you in whatever export format exists, or stay there.

Sorting a tool before reviewing it#

The instinct that puts the weight on the state is half right, and it is right about the smaller half. Hosted state is a genuine data risk and deserves the review it gets. But it is a static risk: the data sits, and the questions about it are answerable in advance from documents. The hosted tool is a dynamic one, because it introduces an action that happens at request time, chosen by a model, on inputs neither you nor the provider curated. Static risks are the ones organisations are already good at. Dynamic ones are where the incidents come from, and this one is under-reviewed precisely because it does not look like a data flow.

The data processing agreement already signed with the provider is the most tempting reason to skip the second review, and it is tempting for a sound reason: it is the document governing data sent to this provider, and this provider is the one running the tool. What such a document addresses is the processing of the data you send — where it rests, who may read it, what may be done with it. The fetch is a different transaction. It is originated by the provider, directed at a party the agreement does not name, and it carries that party's text back into your turn. Whether any clause reaches that is a question for whoever owns the document rather than something to settle in a code review, and the point worth holding is only that having answered it about your data does not answer it about the fetch. The two have been treated as one question because the same vendor's name sits at the top of both.

The useful sorting question is not "does this tool touch user data" but "what does the tool's output touch, and what does the tool itself reach". Provider-hosted code execution in a sandbox with no network egress, operating on a CSV your own code uploaded a moment earlier, in an agent whose only other capability is writing text back to the user, is a tool you can enable without a review. The blast radius is a container that gets destroyed; the inputs are yours; the output re-enters a context that cannot act on anything. Provider-hosted web search in an agent that also holds issue_refund is not that. The input is arbitrary internet text, the output lands next to your instructions, and there is a privileged action one model decision away. Same product, same vendor, same tools array, and one of them is a config change while the other is a design question.

Every clause of the permission granted to the first of those two — the sandboxed code execution, enabled without a review — is load-bearing, and each is a claim somebody has to demonstrate rather than assume. That the sandbox has no network egress is a property of a particular provider's implementation on the tier you are on, not a property of the word "sandbox", and the way to establish it is to make somebody produce the documentation for the tool you enabled. That the inputs are yours holds only for as long as nothing else can put a file into that store. That the output cannot act on anything is the clause which decays quickest, because it stops being true on the day a privileged tool joins the same agent — and the agent that acquires one will not be re-reviewed, since the review that mattered happened when the code interpreter went in. The permission is genuine and it is conditional, and the conditions are the part worth writing down beside it.

Two neighbouring ideas get confused with hosted state and are worth separating now, because the rest of the topic assumes the distinction. Hosted conversation state is not memory: a conversation object holds this thread's turns, not durable facts extracted across threads, and providers ship those as different products with different retention. It is also not retrieval. A hosted file store that the model searches is a corpus you uploaded on purpose, and while it raises the same retention and deletion questions, it is not the transcript and is usually deleted through a different endpoint — a detail chapter 3 turns into an operational problem. Hosted tools have their own confusion: "the provider runs it" is not a statement about sandboxing quality. Provider-executed code runs in an isolated container with resource limits and typically no internet access, and that isolation is real and good. It says nothing about whether the result of the tool should be trusted by whatever reads it next.

Keeping the two primitives apart is the whole of this chapter, and the practical form it takes is two reviews with different participants. The state question goes to whoever already owns third-party data processing, and it is answered from contracts and configuration: retention window, residency for storage and for inference, a deletion path you can evidence, and whether a past input can be reproduced. The tool question goes to whoever owns the agent's capabilities, and it is answered from the agent's own surface: what the tool reaches, where its output lands, and what privileged action sits downstream of that output. A vendor questionnaire that answers the first tells you nothing about the second.

What all of this leaves open is that so far it is argument. The claims — that handles persist, that file references survive across turns, that tool results appear in responses without appearing in your logs, that you cannot see the fetch — are checkable, and until you have checked them on the provider you actually use, they are somebody's blog post. The next chapter builds the thing: several turns driven by a provider-managed conversation object, one hosted tool enabled, one uploaded file referenced, and a capture of everything sent and everything returned on every turn. Every later chapter in this topic argues from that trace rather than from first principles, which is the only way the arguments about deletion, cost, and incident reconstruction stay honest.