The image detail parameter

Some providers expose a per-image detail field, usually taking high or low, alongside the image itself. It looks like a quality setting and it is priced like a bargain: the low path bills a small fixed number of tokens per image regardless of what you upload, commonly 85, against several hundred to a couple of thousand for the same image on the high path. On a pipeline with volume, that ratio is large enough that somebody will eventually propose flipping it globally.

One parameter, two entirely different requests#

The two values do not sit at either end of a scale. They route the image through different processing.

{"type": "image_url",
 "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "high"}}   # tiled path
{"type": "image_url",
 "image_url": {"url": f"data:image/png;base64,{b64}", "detail": "low"}}    # single small view

The high path is the staircase from chapter 4: normalise, cut into 512-pixel tiles, pay a base charge plus a fixed amount per tile. The low path skips all of it. The image is reduced to a single small view — on the order of 512 pixels on a side — and billed at a fixed rate with no dependence on what you uploaded. Send a 4000×3000 photograph or a 600×450 thumbnail and both cost 85 tokens, because both become the same small view before the model sees anything.

Those are two paths rather than two settings on one dial because of how a picture becomes model input at all. The model consumes a sequence of positions, and an image enters that sequence as fixed-size crops, each crop occupying a fixed run of positions. Token cost is a count of positions, so the only quantity available to change is the number of crops: the high path cuts the picture into as many crops as its dimensions require and pays for each, and the low path reduces it to one small view, which is a single crop's worth of positions whatever was uploaded.

Nothing sits between them, because there is no fractional crop. Half the detail of the high path would mean a grid two and a bit crops wide, and a crop is either sent or it is not. The field is called detail and takes two values that read as more and less, which is what makes the dial reading tempting; what they select is whether a grid gets computed at all.

For a 1600×1200 product screenshot the difference comes out like this:

The token row is arithmetic rather than two quoted prices. This screenshot normalises to 1024 × 768, which is a grid of four 512-pixel tiles, and the charge is a base of 85 plus 170 for each of those four tiles, so the four tile charges added to the base give 765. The low path pays the base and stops. The same 85 stands in both columns for the same reason: it is what the scheme charges before any tile is counted, and the low path is the tiled scheme with the grid step skipped.

That is what fixes the ratio. Dividing 765 by 85 gives nine, and the nine is not a discount rate a provider chose — it follows from how many tiles this image bought, since each tile adds twice the base charge to a total that started at one base. A picture normalising to two tiles therefore rides at five to one, and one needing eight tiles at seventeen to one. Nine is this image's ratio rather than the parameter's, and it moves as soon as the dimensions do.

That 9× ratio is real for this image, and it is why the setting is tempting. At 200,000 images a month the same table says $52 against $460, and a saving of $400 a month is not usually what gets a change like this approved — the saving that gets it approved is the one on a system running two million pages, where it is $4,000 a month and somebody has been asked to cut infrastructure spend by a quarter.

What a single small view can carry#

Take a US Letter page rendered into one 512-pixel view. A Letter sheet is eleven inches on its long edge, so that view is 512 pixels spread across eleven inches, and dividing the pixels by the inches gives roughly 46 dots per inch. That figure is the exchange rate between the units a page is specified in and the units the model receives it in.

Type is specified in points, and a point is one seventy-second of an inch, so a ten-point font sets capitals about ten seventy-seconds of an inch tall — a length on paper, not yet a count of pixels. Multiplying a length in inches by a rate in dots per inch cancels the inches and leaves dots, and that cancellation is what makes the next two figures arithmetic rather than assertion: ten-point line-item text has a cap height of 10 × 46/72 ≈ 6.4 pixels and an x-height near 3. The same conversion on a 20-point bold total, twice the point size against the same dots per inch, gives a cap height near 12.8 pixels.

Those two results fall on opposite sides of one line. Chapter 3 puts the floor for character identity — the cap height below which the strokes that tell one glyph from another stop being separable — at around 10 pixels. The line items land well under it, so their digits are not merely hard to read; what distinguishes one digit from another is absent from what the model receives. The total lands above it, in the marginal band where reading mostly works — and a total has three advantages that body text does not: it is bold, it is isolated in whitespace, and it sits next to the word TOTAL and a currency symbol, so a partially-resolved digit string has strong context pulling it toward the right answer.

That gap between the largest field on a page and the ordinary ones is what makes the low path deceptive on documents, and the trap it sets is not a fact about invoices.

Somebody verifying an extraction checks the field they can find fastest, which is the field set in the largest type; largest type carries the most pixels per stroke, and the most pixels per stroke is the largest margin above the legibility floor. The field chosen to validate the change is therefore, systematically, the field least able to detect it, and the reviewer's confidence grows with exactly the property that makes their check uninformative.

What happens to the line items is more interesting than simple failure. The model can still see that there is a table, roughly how many rows it has, that the rightmost column is right-aligned numerals, and what the total is. It cannot resolve the numerals. Asked to return the line items, it returns line items — the right number of them, formatted correctly, in a currency it inferred, with amounts that are individually plausible. There is no marker in the response distinguishing a digit that was read from a digit that was constructed. The JSON is well-formed, the schema validates, the confidence you never asked for is not reported, and the only thing that would catch it is a comparison against the original.

Why nothing declines is the part worth working through. An answer comes out of the model one piece at a time, and each piece is chosen by scoring the candidate continuations and taking the strongest. The positions carrying the unreadable region are still positions in that sequence: they still produce vectors, and those vectors still contribute to the scores. What they lack is any content separating one digit from another, which the downscale destroyed before the model saw it.

Nothing anywhere in that loop is a step at which legibility is consulted, and there is no branch in it that can report an absence. Reporting an absence needs a place in the machinery where the answer is allowed to be nothing, and every position here produces a full score over the candidates with a strongest one always available. A search over a table finds no rows and returns no rows; this has no equivalent step to return them from. So the strongest continuation is whatever most often follows text of that shape — a plausible number of rows, numerals right-aligned in the last column, a currency consistent with the page, amounts individually of a size that invoice amounts are.

Put plainly, the downscale did not take away the model's ability to answer. It took away the part of the answer that was coming from the page and left the part that was coming from what pages of that kind usually say, and the two are indistinguishable in the output because they were never separate quantities inside it.

That property is what makes low detail dangerous rather than merely limited. A parser that cannot read a region emits nothing or emits garbage that fails validation. A model that cannot read a region emits something reasonable, and reasonable output flows straight through every downstream check you built to catch unreasonable output. Unreasonable output is the one thing every validator, tolerance check and alert in a pipeline was constructed to notice, which is why this failure is worse than a parser's rather than merely different from it.

The setting belongs to the task, not to the client#

Now take the case where the cheap path is the right answer. A triage bot receives a 1600×1200 screenshot and answers one question: is there a red error dialog on this screen, yes or no. A red modal on a greyed-out background occupies a substantial fraction of the frame and is one of the most robust signals that survives downscaling — large, high-contrast, positionally distinct. At 512 pixels it is unmistakable. High detail on that request pays nine times the image tokens to resolve text the question never reads. Ship low.

The class of question that behaves this way can be named more precisely than "large high-contrast things". It is the questions whose answer is carried by a property that survives a reduction in resolution: whether a region is present at all, roughly where it sits, which colour dominates it, how the page divides into blocks. Questions answered by a specific string of characters are the complement of that set, and contrast does not rescue them, because the strokes were discarded rather than dimmed.

There is a ten-second test for whether a task belongs on the low path, and it does not involve the API. Open the image, scale it so its long edge is 512 pixels, and look at it yourself on a screen at 100%. If you can answer the question from what you see, the model very likely can too; if you find yourself zooming, it cannot, and no prompt engineering will change that. The test works because the low path is not doing anything mysterious — it is a downscale and nothing else — no sharpening step, no separate channel carrying the text — so a person looking at that downscale holds the same information the model holds, and a thumbnail is a thing you already have good intuitions about. Teams that adopt this as a review step stop having the argument in the abstract.

The correspondence has a limit rather than being an identity, and the limit sits where the total sat. A person reading a blurred amount brings the same context the model brings — bold type, whitespace around it, a currency symbol beside it — and often reads it correctly, which makes the test optimistic on precisely the fields a reviewer checks. It errs the other way too: a person will call a region illegible and stop, where the model answers from the same pixels anyway. A fail is therefore decisive and a pass on small text is provisional.

Cost is also not the only thing that moves. Nine times fewer input tokens on the image means less prefill work — the pass a model makes over the whole of its input before it can produce a single output token. The ordering in that clause is the causal link: no output piece can be scored until every input position has been processed, the image's crops are input positions like any other, and so removing most of the crops removes most of the work standing between the request arriving and its first token leaving. The saving therefore lands as latency rather than only on the invoice. That shows up as a lower time to first token: on interactive triage where a human is waiting, the low path typically responds a few hundred milliseconds sooner on a single image and noticeably sooner on a request carrying several. For a batch job that difference is irrelevant. For a support console where an agent is watching a spinner it can be the reason the feature gets used at all, and it is worth putting on the same scale as the token saving rather than discovering it afterwards.

The reason a correct choice still needs care is that the setting will outlive the question it was chosen for. Three months later the ticket template changes and the bot is asked to include the error code from the dialog, or a product manager adds "and summarise the message," or the screenshots start arriving from a desktop client at a different aspect ratio where the dialog is smaller relative to the frame. The detail field lives in a shared client wrapper, it was set once, and the person adding the new field has no reason to look at it. The pipeline then answers the new question fluently, from an image that cannot support it, with no error anywhere. The belief being punished is that a flag chosen for a task stays correct for that task, and it is tempting because it is nearly true: the task keeps its name, its endpoint and most of its prompt. What moved is the question the prompt asks, and the detail field sits in the transport layer rather than beside the prompt.

Two things prevent that. The first is binding the setting to the question rather than to the transport, so the choice is made in the same place the prompt is written and is visible to whoever changes the prompt.

DETAIL_BY_TASK = {
    "dialog_present":   "low",    # large high-contrast element, yes/no
    "screen_summary":   "low",
    "read_error_code":  "high",   # small text, exact characters
    "extract_line_items": "high",
}

def vision_request(task: str, image_b64: str, question: str):
    return {"role": "user", "content": [
        {"type": "image_url", "image_url": {
            "url": f"data:image/png;base64,{image_b64}",
            "detail": DETAIL_BY_TASK[task]}},
        {"type": "text", "text": question},
    ]}

A dictionary keyed by task is not sophisticated, and its lack of sophistication is what makes it work: a KeyError when someone adds read_error_code without deciding its detail level is a better outcome than a silent default. The second protection is a small permanent evaluation set for each task — twenty images with known answers, run on every deploy — because the low-detail failure mode produces no runtime signal at all and only a held-out comparison surfaces it.

Where the question is not known in advance, escalation is the honest structure: run at low detail, and re-run at high when the answer needs detail the first pass could not have had. The trigger has to be something you can compute, not the model's own self-report, since a model that invented an amount will not tell you it did. Practical triggers include the task type, a schema field requiring exact characters, a checksum failing (line items not summing to the total), or a downstream validator rejecting the format. Escalating on 12% of traffic keeps 88% of the saving and removes most of the exposure.

Two defaults, one mistake#

"Always high, because accuracy matters more than cost" is the responsible-sounding version and it is where most teams start, and the instinct under it is sound nearly everywhere else: on how much context to retrieve, on which model to call, on how many tokens of reasoning a request is allowed, paying more does buy accuracy. What breaks the pattern here is that the extra payment buys resolution, and resolution is worth only what the question spends of it — a presence check about a dialog spends almost none of it. Always-high therefore pays several times the tokens on every yes/no check, every "which screen is this," every layout classification — and at real volume that is the difference between a feature that ships and one cancelled at the budget review. "Always low, because it is nine times cheaper" is the version that produces invented invoice lines. The two errors have the same shape: a global default standing in for a per-task judgement about whether the answer depends on detail the small view cannot carry.

It is also worth knowing that not every provider offers this control. Area-billed providers have no detail parameter, and the equivalent decision is made by you, when you choose what dimensions to send. That is strictly more flexible — you can pick any point on the curve rather than one of two — and strictly less forgiving, because there is no flat floor to land on and nothing labelled low to make the trade-off visible in a code review.

Treated as a per-task decision, the detail parameter is one of the better cost controls available on an image pipeline: it is a single field, its cheap path is genuinely nine times cheaper on an image that tiles into four, and a large fraction of production vision traffic — the presence checks, the screen classifications, the coarse layout questions — asks questions that a thumbnail answers perfectly well, while none of the traffic that reads characters does. Treated as a global setting, it is a way of converting a document extractor into a plausible-fiction generator without changing a line of prompt or producing a single error. What separates the two uses is where the value is chosen and whether anything checks the output afterwards.

Escalation, though, carries an assumption that usually goes unexamined: that a cheap first pass and an expensive second pass are the only options, and that both are the same model looking at the same picture. For document extraction specifically there is an older, entirely different instrument available, and it fails in a way that makes it a much better first pass than a downscaled image.