Function calling

Tool calling, which the OpenAI documentation and most job adverts call function calling, is the protocol by which a language model asks your program to run one of your functions and then reads what it returned. You attach a list of function descriptions to the request; the model, on a turn where one of them would help, replies with a specially marked block naming a function and filling in its arguments; your code executes the real function and sends the return value back so the model can continue. Every difficulty in this topic sits in one half or the other of that exchange, and the first of them is that the second half does not happen by itself.

The protocol exists because a language model can do exactly one thing, which is produce text, and text on its own does not cover the work assistants are asked to do. A model writing to a customer about a missing refund has no route to an orders table, no view of today's exchange rate, and no way to move €89.90 from a merchant account back to a card; everything it knows comes from training data months old and from the conversation in front of it. Tool calling joins the two halves, so that your systems supply the current facts and perform the effects while the model supplies the judgement about which function suits this turn and what to pass it. You meet it wherever a model appears to know something it could not have been trained on, or does something outside the chat window: a support assistant quoting a delivery date is calling lookup_order between the customer's question and its own reply, and a coding assistant that edits a file, a research assistant that searches the web, and every product currently sold as an agent run this same exchange underneath, usually several times per answer. Every major provider ships it as a tools field on the ordinary chat request, which is why it looks like a small addition and turns out to be a protocol with a round trip you have to implement.

What the protocol gives you is a request#

The model runs in the provider's process. It has no route into your VPC, no database credentials, no access to a refunds service, and no ability to execute a line of your Python. The only thing it can physically do is emit tokens. Tool calling is layered on that single capability: you attach a list of function descriptions to the request, and the model is trained to emit a specially marked block that names one of them and fills in arguments, at which point decoding stops and the response comes back to you carrying a stop reason that says a tool was requested rather than that the answer is finished.

Each link in that arrangement is worth making explicit, because everything else in this topic rests on it. The model produces one token at a time, and a token is an entry in a vocabulary and nothing more. During training it is shown transcripts in which a request for a function appears as a particular sequence: a marker, a name, a serialised argument object, a closing marker. Emitting that sequence is therefore something it has learned to do in the same way it has learned to emit a sentence. The provider's decoder, for its part, watches for the closing marker. When it sees one it holds a complete block and has no reason to keep generating, so it halts and returns the response with a field recording why it halted. That is the whole mechanism — a learned output shape, plus a decoder that recognises where the shape ends. No step in it reaches outside the provider's process, because no step in it can.

Attaching a tool list is not a switch that puts the model into a different mode. The definitions are serialised and placed in the request next to the messages, and the model reads them the way it reads any other text in the prompt. That is why they cost input tokens, and why the per-definition token figure later in this chapter is a real recurring charge rather than an accounting curiosity.

If you want an analogue from work you have already done, the closest one is a remote procedure call, or RPC, with the roles inverted from the direction you are used to. Your service is normally the caller: it sends a request to a payments API and waits. Here the model is the caller, your process is the server implementing the interface, and the transport between them is not a socket but the loop you wrote — you receive the call in an HTTP response, dispatch it however you like, and deliver the return value by putting it into the next HTTP request. Nothing about that is automatic, and no part of the provider's infrastructure is watching to see whether you did it.

That is a picture rather than a description, and it is worth saying where the picture stops being true. In an ordinary RPC the caller blocks: a stub holds a connection open, and the runtime guarantees that either a reply or an error comes back to it. Here the caller does not block, because by the time the block reaches you the generation that produced it has finished and the model's working state no longer exists. There is no connection waiting to be answered, no timeout that fires, and nobody notified if you never reply at all. The only thing that resumes the call is your process building a fresh request that happens to contain the earlier one. That single difference is why the failure in this chapter's scenario is silent rather than loud.

A response asking for a refund looks like this, with the identifiers shortened. It is an ordinary message; the only unusual thing about it is the block type.

{
  "id": "msg_01Q4rXbW8yN2kLpZ",
  "role": "assistant",
  "model": "claude-sonnet-4-5",
  "stop_reason": "tool_use",
  "content": [
    {
      "type": "text",
      "text": "Let me look that up and issue the refund."
    },
    {
      "type": "tool_use",
      "id": "toolu_01A9fRk3",
      "name": "refund_order",
      "input": { "order_id": "AB-40912", "amount": 89.90 }
    }
  ]
}

The stop_reason field is the part a handler has to branch on. A value of end_turn means the model has finished and the text is the answer; tool_use means the model has suspended mid-thought and is waiting for something from you. Treating those two identically sends a second request whose conversation ends in an unanswered request, and there is no training data anywhere in which that situation resolves into a completed refund.

The belief that causes trouble here is that a well-formed block is evidence that something happened. It is a reasonable thing to believe, because nearly everywhere else in a working system a well-formed structured message is produced by an operation: a webhook body exists because a charge settled, an audit row exists because a write committed, and reading one is a way of learning that the event occurred. The tool-use block inverts that order. It is produced before anything happens, by a process that has no way of making anything happen, and it is well formed precisely because the decoder was held to a shape while producing it. Validity reports on the request and says nothing at all about the world. A transcript full of immaculate tool_use blocks and a refunds ledger with no rows in it are perfectly consistent with one another.

The conversation is the only state there is#

The chat endpoint holds nothing between calls. Every request carries the full message list, and the provider reconstructs the situation entirely from what you sent, so continuing a turn after a tool call means appending two things and resending everything: the assistant message exactly as it came back, including the tool-use block, and then a new user message carrying the result, tagged with the identifier of the call it answers.

import anthropic, json

client = anthropic.Anthropic()

TOOLS = [{
    "name": "refund_order",
    "description": "Issue a refund against a completed order.",
    "input_schema": {
        "type": "object",
        "properties": {
            "order_id": {"type": "string"},
            "amount": {"type": "number"},
        },
        "required": ["order_id", "amount"],
    },
}]

messages = [{"role": "user", "content": "Please refund order AB-40912."}]

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=TOOLS,
    messages=messages,
)

if resp.stop_reason == "tool_use":
    # 1. Keep the assistant turn verbatim, tool_use block included.
    messages.append({"role": "assistant", "content": resp.content})

    results = []
    for block in resp.content:
        if block.type != "tool_use":
            continue
        try:
            payload = refunds.issue(**block.input)          # your code, your credentials
            content, is_error = json.dumps(payload), False
        except RefundRejected as exc:
            content, is_error = json.dumps({"error": str(exc)}), True
        results.append({
            "type": "tool_result",
            "tool_use_id": block.id,                        # 2. tie the result to the call
            "content": content,
            "is_error": is_error,
        })

    # 3. Results ride back as a user turn, and the whole history goes out again.
    messages.append({"role": "user", "content": results})
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        tools=TOOLS,
        messages=messages,
    )

Three details in that snippet cause most of the errors people hit in the first week. The assistant message has to go back unmodified — dropping the tool-use block while keeping the text produces a result that answers a call the provider cannot see, and the request is rejected. The tool_use_id is not decorative; it is how a result is matched to a call, and it matters enormously once a turn contains several calls at once. And the results travel in a message with the user role, which reads strangely the first time, but the role field here is a transport convention marking whose side of the exchange the content came from, and the tool result did come from your side.

It is fair to ask why an identifier is needed when the results could simply be listed in the same order as the calls. Position would work if the two lists were guaranteed to line up, and they are not. Your dispatcher may run the calls concurrently and collect them as they finish, one call may fail and come back before a slower one that succeeded, and a policy check may refuse a call while still owing the model a result for it. Any of those reorders the results relative to the calls. Matching on an identifier the provider generated makes the pairing independent of order, which is what lets you dispatch however suits your system rather than however suits the wire format.

The shape differs between providers without the mechanics differing. On OpenAI's Chat Completions API the assistant message carries a tool_calls array in which arguments arrive as a JSON string rather than a parsed object, and each result goes back as a separate message with role: "tool" and a tool_call_id. Anthropic nests everything in typed content blocks and returns arguments already parsed. The round trip is identical: the model names a function, your process executes it, the return value goes into the next request tagged with the call's identifier, and the entire conversation is resent.

Resending everything has a price that shows up on the bill and on the clock. A single tool call costs two model invocations rather than one, and the second invocation carries the first request's tokens plus the assistant turn plus the result. On a support conversation with a 900-token system prompt, four tool definitions at roughly 110 tokens each, and 600 tokens of history, the first request is around 1,940 input tokens and the second is around 2,150, so a turn that used one tool bills about 4,090 input tokens instead of 1,940. Latency stacks the same way: two round trips of 1.5 to 3 seconds each, plus however long your function takes, so an assistant that felt instant in a no-tools prototype answers in four to seven seconds once a lookup is in the path. Prompt caching, where the provider keeps the encoded form of a request's leading tokens and bills a later request that starts with the identical bytes at a fraction of the input rate, takes most of the money back on the repeated prefix, since the system prompt and tool definitions are byte-identical across both requests. It does nothing for the wall clock, and users notice the wall clock.

Those input figures are worth reproducing rather than taking on trust, because the shape of the sum is what carries over to your own numbers. The first request contains three things and nothing else: the system prompt at 900 tokens, the four tool definitions at roughly 110 tokens each — four times a hundred and ten, so a little over four hundred — and 600 tokens of prior conversation. Adding those three gives the 1,940 figure. The second request contains all three again, byte for byte, plus two new items: the assistant turn that just came back, its tool-use block included, and the tool result you appended. Those two together account for a little over two hundred tokens, which is the entire distance between 1,940 and 2,150. Addition is the right operation for the turn's total rather than taking the larger of the two, because the two requests were separate invocations and each was billed in full, so the turn's input charge is the two sizes added and that is where about 4,090 comes from.

The tempting conclusion from the caching sentence above is that a cache makes the second request free. It does not, for two reasons that are visible in the same arithmetic. A cache bills the repeated prefix at a lower rate; it does not remove the prefix from the request, and the discount applies only to the stretch that is byte-identical. The part that is not identical is precisely the part that grows — the assistant turn and every tool result accumulated so far sit after the cached prefix, and they are charged at the ordinary rate on every subsequent request in the conversation. The latency arithmetic behaves the same way and has no cache at all: two round trips at 1.5 to 3 seconds each is three to six seconds of model time before your own function has run, which is how a prototype that answered instantly becomes an assistant that answers in four to seven.

Serialising the result is your decision, and JSON is a convention rather than a requirement — the field is text, and a plain string like Refund rf_8812 issued, €89.90, expected in 3–5 business days is read perfectly well. What matters is that the content is unambiguous about what happened, because it is the only thing the model will ever know about the outcome. The model cannot inspect the return value your function produced, cannot see the HTTP status the refunds service returned, and cannot tell a partial success from a complete one except by reading the string you wrote; a result that names the outcome in the vocabulary the reply will need is the difference between an assistant that quotes a settlement window and one that constructs a plausible sentence in place of it. The same reasoning covers failure, which is why the snippet above sends exceptions back as ordinary results with the error flag set rather than raising: a rejected refund the model can read about is a turn that ends with an explanation to the customer, and an exception that escapes the handler is a turn that ends with a stack trace.

Size is the other thing to decide deliberately, since the result sits in the conversation for the rest of the session and is resent on every subsequent request. A lookup_order implementation that returns the full ORM row serialised — forty-odd fields including internal warehouse codes, a nested address, and two timestamps per state transition — puts about 700 tokens into the history to answer a question about a delivery date, and does it again on every turn that follows. Selecting the eight fields a support conversation can actually use brings that to roughly 90 tokens and measurably improves the answers, because the model is no longer choosing which of forty plausible fields the customer meant.

Divide the larger figure by the field count to see where it comes from. Seven hundred tokens spread over forty-odd fields is something in the region of fifteen to twenty tokens for each one, and that is about right for a serialised JSON field: a quoted name, a colon, a quoted or nested value, a comma, and the tokeniser splitting an internal warehouse code or an ISO timestamp into several pieces because it has never seen that exact string before. Eight fields pay the same per-field cost and simply have fewer of them, which accounts for the whole of the drop to roughly 90. Nothing was compressed. Thirty-odd fields were removed that no support conversation was ever going to read. The saving also compounds, because that result stays in the message list: on a turn that carries it for the fifth time, the difference has been paid five times.

Where the boundary gets misplaced#

A tool call is not the model executing your function. That distinction sounds pedantic until you notice how much design follows from it: your process decides whether to run the call at all, which user's credentials to run it under, whether a human has to approve it first, and what to say if the answer is no. A refusal is a perfectly ordinary tool result — {"error": "agent lacks refund permission above €50"} sent back with the error flag set — and the model will read it and explain the situation to the customer. The name in the block carries no authority whatsoever, and the moment you treat it as authorisation you have handed your permission model to a text generator.

It is worth naming why that mistake is tempting rather than assuming nobody makes it. A tool definition looks like a contract, and contracts normally bind both parties. The schema really does bind the model: order_id arrives as a string and amount as a number, because the decoder was constrained while producing them. What a schema has no vocabulary for is whether this order belongs to this customer, whether this session may move this much money, or whether the model had any grounds for choosing that order rather than a neighbouring one. A validator that passes has told you the request is well typed. Well typed and authorised are different properties, and the distance between them is exactly the space a policy check occupies.

The practical shape of that is a dispatcher with a switch on the tool name and a policy check in front of it, rather than a lookup table mapping names to callables. Between the block arriving and the function running there is room for whatever your product needs there: a check that this session's customer owns the order named in the arguments, a ceiling above which a human approves, a queue that defers the write until a review lands. None of that is visible to the model and none of it needs to be, because every one of those decisions has a result representation the model can read.

There is one real exception worth knowing about, because it muddies the vocabulary. Providers also ship server-side tools — code execution in a sandbox, web search, file retrieval — where the provider runs the thing and the result appears in the response without you doing anything. Those look superficially like the tools you define, they appear in the same tools array, and they are the reason a colleague may insist that the model does run tools. The tools you define are the other kind, and everything in this topic concerns those.

Topic 5.2, on schema-constrained generation, covers how a schema narrows what the decoder may emit; tool arguments ride on that same machinery, which is why input arrives as an object with your field names already in it rather than as prose you have to parse. The new part is not the constraint on the arguments — it is that a name and a protocol have been wrapped around them, so the output is addressed to a specific function of yours and there is a defined way to hand something back. A tool call also is not a promise of being called: attaching a tool tells the model the option exists, and on any given turn it may answer directly instead, which is the behaviour you usually want and occasionally do not.

Read as a whole, the exchange has exactly one asymmetry worth remembering. The model produces a request and stops; everything after that — running the function, choosing the credentials, deciding whether the operation is permitted, phrasing what came back — is yours, and the conversation you resend is the only place any of it is recorded. A handler that honours the request keeps the model in a situation it can reason about. A handler that ignores it leaves a hole in the transcript, and a text predictor asked to continue past a hole will fill it.

What is far less obvious is why the model reached for refund_order at all, and why it filled amount with 89.90 rather than asking. It could not see the function, the service, or the ledger. Everything it had to work from was the small envelope of name, description and schema that the request carried alongside the messages, which is what the next chapter takes apart.