Describing an image turns cross-modal search into text search

Neither corpus in this topic has usable captions. The four million product images carry filenames like SKU-88231-alt3.jpg and an alt attribute that is the SKU again. The 200,000 engineering diagrams carry a Confluence page title and nothing else. So the caption baseline that keeps outscoring cross-modal retrieval on question-shaped queries is not actually available — unless someone writes the captions, and at four million images nobody is writing them by hand.

Generating them is the obvious move and the estimate is the first thing to get right, because the number that comes back is not in the same class as the embedding bill. A product photograph at full resolution costs roughly 1,100 input tokens once the encoder has tiled it, and a description detailed enough to be worth indexing runs about 180 output tokens. The provider prices in the table below are quoted per million tokens, abbreviated MTok, with the input price first and the output price second.

Two orders of magnitude separate the top row from the third, and the comparison teams usually make — "captions beat cross-modal on recall, so use captions" — silently assumes the captions are free because a human wrote them years ago. When you have to produce them, the caption index is the expensive system and the embedding index is the cheap one. The $8,000 also repeats on every prompt change, exactly as re-embedding repeats on every model change; the difference is that a prompt change is a thirty-second edit that feels like nothing, which makes the trap easier to fall into. Version the prompt, store its hash on every row, and treat "improve the captioning prompt" as a scheduled backfill rather than a tweak.

The call, and the schema that makes it indexable#

The description has to be built for retrieval rather than for a human reader, which mostly means it must be exhaustive about things a query might name and must never editorialise. Constraining the output to a schema does more work here than any amount of prompt wording, because it forces the model to fill the fields that queries actually hit.

import base64, json
from anthropic import Anthropic

client = Anthropic()

SCHEMA = {
    "type": "object",
    "properties": {
        "caption":   {"type": "string", "description": "One sentence naming subject, setting, framing."},
        "objects":   {"type": "array", "items": {"type": "string"}},
        "relations": {"type": "array", "items": {"type": "string"},
                      "description": "Spatial or functional relations, e.g. 'kettle on butcher block'."},
        "text_in_image": {"type": "string", "description": "Verbatim transcription of rendered text, '' if none."},
        "attributes": {"type": "array", "items": {"type": "string"},
                       "description": "Colour, material, condition, wear, defects actually visible."},
        "capture":   {"type": "string", "description": "Lighting, angle, focus, e.g. 'overhead, daylight, shallow depth'."},
    },
    "required": ["caption", "objects", "relations", "text_in_image", "attributes", "capture"],
    "additionalProperties": False,
}

PROMPT = (
    "Describe this image for a search index. Name only what is visible. "
    "Transcribe any rendered text exactly, including labels inside diagrams. "
    "Do not infer brand, price, material, or purpose that you cannot see. "
    "If a property is not visible, omit it rather than guessing."
)

def describe(path: str) -> dict:
    data = base64.standard_b64encode(open(path, "rb").read()).decode()
    resp = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}, "effort": "low"},
        messages=[{"role": "user", "content": [
            {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": data}},
            {"type": "text", "text": PROMPT},
        ]}],
    )
    return json.loads(resp.content[0].text)

For the full corpus pass, the same request body goes through the Batches API at half price, keyed by custom_id so results can be reassembled in any order:

batch = client.messages.batches.create(requests=[
    {"custom_id": sku, "params": {"model": "claude-opus-5", "max_tokens": 1024,
                                  "output_config": {"format": {"type": "json_schema", "schema": SCHEMA},
                                                    "effort": "low"},
                                  "messages": build_messages(path)}}
    for sku, path in shard
])

The difference between a description that helps and one that does not is visible in the output. Left to a generic "describe this image" prompt, the appliance corpus produced strings like "A stylish red kettle, perfect for any modern kitchen." — one usable content word, one aesthetic judgement, and a marketing claim. Under the schema above, the same photograph produced caption "Red enamel stovetop kettle on a butcher-block counter, shot from slightly above", objects ["kettle", "counter", "tiled backsplash"], attributes ["red enamel", "chrome handle", "small chip on spout rim"], text_in_image "", capture "overhead-ish, daylight from left, whole object in frame". The chip on the spout — the detail from the blind evaluation that no human caption ever recorded — is now in the index and findable by typing "chipped kettle".

How those fields land in the index matters nearly as much as their contents. Flattening the whole object into one blob and indexing it as a single string throws away the distinction between a term that appeared in text_in_image and one that appeared in caption, which is exactly the distinction a query for a model number depends on: the string "RK-2200" printed on the appliance is strong evidence, the same string in a speculative caption is not. A field-weighted lexical index — text_in_image and objects weighted above caption, capture weighted below — moved nDCG@10 by 0.06 on the appliance evaluation over the flattened version, for the cost of declaring the fields. The description corpus also wants a dense text index over the caption field alongside the lexical one, since a query phrased differently from the description will miss on BM25 and hit on embeddings, and that immediately raises the same question this topic keeps circling: two ranked lists, two score scales, one result set.

Ingestion is not a single event either, and the ongoing rate is the number that belongs in the budget. The appliance catalogue took on about 9,000 new images a day, which at the small tier through the Batches API is $9 a day, or $3,300 a year — an order of magnitude less than the backfill and a recurring line rather than a project cost. It also has a latency requirement the backfill does not: a product listed at 09:00 that is unfindable until the nightly batch returns is a real complaint, so the steady-state pipeline usually runs synchronously at full price for new arrivals and reserves batch pricing for backfills and re-runs. Sizing the two paths separately, rather than assuming the backfill's economics apply forever, is what keeps the number honest.

Omission and invention are not symmetric failures#

The index inherits the description model's coverage exactly. Anything the description does not mention cannot be retrieved by any query, no matter how well the text retriever works, because the retriever is searching text that does not contain the answer. Anything the description invents is retrievable with full confidence, because the retriever cannot tell an invented attribute from an observed one.

Both happen, at rates worth measuring rather than assuming. Sampling 200 generated descriptions from the appliance corpus and checking each field against the photograph by hand gave 6.0% containing at least one attribute that is not true of the image — most commonly a material claim ("stainless steel" for a chrome-plated plastic body, twice) and a state claim ("packaged in box" for an item photographed against white). Those 12 rows are now permanently retrievable by a query naming the wrong attribute, and the user sees a confident, well-ranked, wrong result.

Omission was both more common and harder to see: comparing the generated attributes list against a human's exhaustive list for 50 images, the model recorded a median of 4 of 7 visible attributes, and the ones it dropped clustered in exactly the region that motivated cross-modal search in the first place — small defects, unusual crops, things at the edge of frame. Query "chipped rim" against the description index returned recall@10 of 0.08; the same query against the image-embedding index returned 0.41. The description pipeline had been told to record defects and still missed most of them, because a fixed-length description is a lossy summary and the model chooses what to keep.

The asymmetry matters for how you spend effort. Invention is bounded and auditable: sample, count, tighten the prompt, re-measure, and you can drive it down. Omission is unbounded and invisible — there is no way to audit for the absence of something you did not think to ask about, and the only signal is a query class quietly returning nothing useful. Which is a strong argument for keeping a pixel-based retriever alongside the description index rather than treating one as the replacement for the other.

The audit that produces those percentages is small enough to run on a schedule and specific enough to act on. Sample 200 descriptions stratified by category, show each alongside its image, and have a reviewer mark every field-level claim as observed, wrong, or unverifiable — perhaps four hours of work. Wrong claims cluster, and the cluster names the fix: material errors respond to an explicit instruction not to infer material from appearance, packaging errors respond to adding a context enum with a "studio background" value, and errors that persist across three prompt revisions are usually a signal to move that field out of the description and into structured product data you already hold. Omissions cannot be audited this way, since a reviewer comparing description against image will also miss what the model missed; the only handle on them is the per-class recall table below, where a class quietly sitting at 0.19 is the signature of a systematic omission rather than a retrieval failure.

Which system wins which query#

With both built over the same appliance corpus and evaluated on the 900 blind queries from chapter 2, split by the classes established in chapter 4:

The last two rows are where the description index is not merely better but categorically so. Text drawn inside the picture is transcribed at full resolution by a VLM that tiles the image, so the 1.5-pixel labels that vanished under a 224-pixel encoder are read cleanly, and an exact identifier lands on a lexical retriever that can match a string. No embedding, of either modality, does exact match — that is what BM25 is for, and a hybrid text index gives it to you for free.

This is what settles the engineering-diagram corpus. Running OCR and a schema-constrained description over the 200,000 diagrams, then indexing text_in_image plus relations with BM25 and a text embedding, moved "which service talks to the payments queue" from 0.11 recall to 0.73, at a one-time cost of about $400 on the small tier through the Batches API. The remaining 0.27 is mostly diagrams where co-occurrence of two service names in the transcription does not imply an arrow between them, and closing that gap means asking a VLM the question against the top few candidates rather than improving retrieval.

What the description index is not#

It is not a replacement for the embedding index, as the middle three rows show, and a team that builds it and deletes the vectors will discover the loss on defect and style queries months later.

It is not stable across model versions any more than vectors are. A description corpus has a model identity and a prompt identity, both of which change its contents, and both belong on the row next to the text. The failure mode is subtler than the mixed-space bug from chapter 3 — nothing breaks, the corpus just becomes inconsistent, with descriptions written by two models under two prompts competing in one ranking.

And it is not evaluation-neutral. Generating descriptions with a model and then evaluating with queries generated by the same model rebuilds the leak from chapter 2 in a new form: the query and the document are now both samples from one model's distribution, and they will match each other more readily than either matches a human's phrasing. The blind human-written query set stays the ground truth precisely because it comes from outside both systems.

What you have at this point is two retrievers over the same corpus with complementary strengths, which is a good position and an unfinished one. Each returns a ranked list with scores attached, and those scores come from different machinery on different scales — cosine similarities squeezed into a narrow band, and BM25 values that depend on corpus statistics and query length. Producing one result set from two lists is the step where most of the remaining quality is won or lost.