Chat API shape

Read these first

• Base vs instruct vs reasoning vs multimodal variants

What you will know

• Construct a multi-turn request by hand, without an SDK, and get a response

  Construct the JSON by hand with curl once. Knowing what the SDK sends removes a whole layer of mystery.

• Run equivalent requests with an instruction placed in each role and compare the results

  Role precedence and provider-specific handling can change compliance, especially when assistant messages are used to steer generation.

• Trace a multi-turn client and verify that each request contains the prior turns rather than relying on server-side memory

  A retry on another worker or a process restart loses context unless the application persists and resends the conversation.

• Calculate cumulative input tokens and estimated cost for full-history resend, then compare it with a truncation policy

  With similarly sized turns, full-history input grows O(n²) across the conversation, while a bounded window keeps it roughly linear.

• Identify a malformed message array from the error alone

  Wrong role, missing content, or an unpaired tool result — the error text usually names the index.

Chapters

1. 1

   The messages array

   A chat API request is one JSON document: a model, a ceiling, and an ordered array of role-tagged messages. Build one by hand with curl, read the response envelope to find where the generated text sits and where the token counts and stop reason sit, then extend it into a multi-turn conversation. Once you have seen that body, every SDK convenience becomes a transformation you can predict.

   Read chapter
2. 2

   The system role

   The instruction channel is either a top-level field beside the messages array or an entry at the head of it, and on providers of the first kind a mid-array system message is a schema violation that fails with an error naming the index. Underneath the schema question is a behavioural one: post-training weights the instruction channel above user content, which is measurable on your own workload in about twenty minutes.

   Read chapter
3. 3

   Assistant prefill

   A partial assistant message at the end of the messages array makes some providers continue from it rather than reply to it: send an opening brace and JSON comes back, minus the brace you sent. Other providers reject a trailing assistant turn outright. This chapter covers how to probe which behaviour a model has, how to re-prepend the prefill before parsing, and what constrained decoding replaces it with.

   Read chapter
4. 4

   Tool calls and tool results

   Beyond system, user and assistant there is a fourth shape in the messages array: an assistant entry carrying a structured tool call, and a separate entry answering it, joined by an id the provider issued. Send one by hand to see both entries and the id linking them, then see the provider reject an array where the pairing is broken — a constraint that makes some cut positions in a conversation unsafe later.

   Read chapter
5. 5

   Stateless requests and conversation history

   Each chat API request is independent: the model sees only the array you send, and nothing on the provider's side accumulates a transcript. This chapter gives the two-request experiment that demonstrates it, the storage schema that survives three workers and a deploy, and the distinction between owning conversation history and the prompt caching people confuse it with.

   Read chapter
6. 6

   Quadratic growth of input tokens

   Turn n of a conversation carries every prior turn with it, so with roughly equal turns the cumulative billed input follows n(n+1)/2 rather than n. Work the arithmetic on a realistic support-bot shape — forty turns, six hundred tokens per pair, fifty thousand conversations a month — set it against a bounded window and against prompt caching, and see which ceiling each one moves.

   Read chapter
7. 7

   Trimming conversation history

   A window of the last eight messages can cut between the assistant entry that issued a tool call and the entry answering it, orphaning a pair the provider validates by id. This chapter covers reading a malformed messages array from the error text alone, trimming on a token budget cut at legal boundaries, validating the array before it goes out, and replaying real conversation shapes offline to find the cut positions production will hit.

   Read chapter
8. 8

   Retries and duplicate generations

   When a client times out after the provider has produced a full response, the queued retry sends a byte-identical messages array and bills a second full generation of a conversation that already grows quadratically. An unguarded write then leaves two assistant messages answering one user turn, which the next request feeds back as real conversation. Compare optimistic and post-call persistence, then build the claim record and idempotency key that make a redelivered job harmless.

   Read chapter