Every image on a fetched page is attacker-controlled too

Two teams sat in the same review. The first shipped a support agent with an upload button and had spent a month on the image-scaling defence from the previous chapter. The second shipped a research assistant that took a URL, fetched the page, and summarised it for the user — no upload button anywhere in the product — and had concluded that image injection was a problem for teams with upload features, which they were not. Their agent was vision-capable because summarising a page well means describing its images, and it fetched arbitrary URLs a user pasted in. The reviewer asked the second team a single question: when your agent fetches https://competitor-review.example/comparison, who chose the images on that page? The room went quiet, because the answer is: whoever runs that domain, and it is not the user and it is not you.

This is the whole of the chapter. The trust boundary you learned to reason about in Stage 7 does not sit at your upload form. It sits between content that originated inside your control and content that did not, and a pixel that arrives from a page you fetched is on the far side of it just as surely as a file a stranger uploaded. The upload button is one opening in the boundary. It is not the boundary.

The upload form was never where the line was#

It is easy to see why the line gets drawn at the upload form. The upload is deliberate, visible, and yours — a user picks a file, your handler receives it, and the whole apparatus feels like an obvious place to inspect incoming content. So the mental shortcut forms: untrusted images enter through uploads, we have no uploads, we have no untrusted images. Each step sounds fine and the conclusion is wrong, because the first step is not true. Untrusted images enter through any path that carries pixels from outside your trust domain, and a browsing agent is a firehose of exactly those.

Consider what the research assistant actually does when it handles a URL.

def summarize_url(url: str) -> str:
    html = http_get(url)                       # attacker controls this response
    images = extract_image_urls(html)          # attacker chose every one of these
    blocks = [{"type": "text", "text": "Summarize this page for the user."}]
    for img_url in images:
        img_bytes = http_get(img_url)          # fetched from wherever the page pointed
        blocks.append({
            "type": "image",
            "source": {"type": "base64",
                       "media_type": "image/png",
                       "data": b64(img_bytes)},
        })
    return vision_agent.run(blocks)             # every image lands in the context window

Nothing in that function came through an upload. Every image URL was written into the HTML by whoever controls the page, and the agent dutifully fetches each one and places it in the model's context. If one of those images carries a rendered instruction — a plain caption, a low-contrast overlay, a scaling payload tuned to the agent's known resize — it reaches the model exactly as if it had been uploaded, and it does so on a page the user merely asked the agent to read. The user's intent was "summarise this," which is not consent to execute instructions the page's owner embedded in a banner image.

This is the direct answer to the question the second team should have been asked earlier: a team argues image injection does not apply to them because their product accepts no uploads, and their product is an agent that browses pages and summarises them — are they right? They are not. Every image on a fetched page is content an attacker controls and can plant. The absence of an upload feature removes one channel and leaves wide open the channel the product is built around.

Where pixels cross into your trust domain#

Once you stop looking for the upload form and start looking for the boundary, the list of crossings gets long, and it is worth enumerating because each one feels, individually, like it doesn't count.

Read down the right-hand column and the pattern is the same each time: the pixels arrive through a mechanism that feels internal — an authenticated inbox, a retrieval index, a partner API — and that feeling is doing the work of a trust decision it has not earned. An email attachment reaches your agent through your own mail infrastructure, which is why it feels like part of the system; the sender of that email is a stranger, which is why the pixels are untrusted. A screenshot pasted into a document that later got indexed into your knowledge base arrives through retrieval, which reads as "our data"; the document was written by someone outside your organisation, so the screenshot is theirs, not yours. The channel's trustworthiness and the content's trustworthiness are different properties, and image injection lives in the gap between them.

The unifying rule is worth stating plainly so it can be applied to a channel not on this list: treat as untrusted any pixel that did not originate inside your trust domain, regardless of how it reached the model. Origin, not arrival mechanism, decides. A photo your own service generated and stored is inside the boundary. A photo anyone outside your organisation could have influenced — by uploading it, by hosting it, by emailing it, by writing it into a document you later ingest — is outside, and the model cannot tell the two apart on its own.

The retrieval row deserves a second look, because it is where the confusion is most seductive. A knowledge base feels like the most trusted content in the system — it is "our documentation," curated, indexed, the source of truth the agent is supposed to defer to. But a knowledge base is only as trustworthy as its least trustworthy contributor, and most real ones ingest content from outside: partner documents, customer-submitted tickets, scraped vendor pages, PDFs a support rep saved from an email. A screenshot embedded in any of those arrives at the agent through the retrieval layer wearing the knowledge base's authority, and the agent, correctly trained to trust retrieved context, has every reason to believe it. The attacker did not have to breach your retrieval system; they only had to get one document with one embedded instruction into the pipeline that feeds it, which for many teams is an open submission form or a shared drive. Origin still decides, and the origin of that screenshot was a stranger, however trusted the shelf it now sits on.

The interview turns on "when," not "whether"#

The second team's real mistake, and the one the interview question is built to surface, was temporal. Asked whether image injection applies to a browse-and-summarise product, the tempting answer is: not yet — we accept no uploads, so the risk arrives the day we add an upload feature, and we should revisit it then. That answer treats the exposure as a feature you have not shipped, when in fact you shipped it on day one. The agent already fetches attacker-chosen images and places them in the model's context. There is no future upload button that opens the door; the door is the core loop of the product, and it has been open since launch.

So the honest answer has two parts. First, they are wrong about whether: the risk is present now, not contingent on a feature they might add later, because a browsing agent's inputs are attacker-controlled by construction. Second, once they accept that, the thing that changes is what they have to reason about. It is no longer "protect the upload handler," a bounded component they don't have. It is "the model's context routinely contains pixels chosen by whoever owns the pages we fetch," which reframes the exposure as a property of the agent's normal operation rather than an edge case. Every page fetch is an ingestion of untrusted images, every summary is a turn in which those images could carry instructions, and the mitigations have to live in the agent's architecture — what tools it can reach, how it treats image-derived content — rather than in a form-level scanner that has nothing to scan.

It is worth marking what this chapter does not claim, so the boundary of the argument is as clear as the boundary it describes. It does not claim every fetched image is malicious; the overwhelming majority are ordinary content, and treating them as untrusted is a statement about possibility, not prevalence. It does not claim the browsing agent is more exposed than the upload product — they are exposed through different openings in the same boundary, and a product with both openings has both. And it does not, on its own, tell you what to do about untrusted pixels once you have correctly identified them as untrusted. Naming the boundary is a prerequisite for defending it, not a defence.

When the model reads the instruction anyway#

The previous three chapters and this one have been mapping the attack surface: the channel text filters miss, the three paths one file takes, the transformation that develops a hidden payload, and now the full set of openings through which untrusted pixels reach the model. Running underneath all of it has been an assumption worth making explicit — that the model, having received an instruction it should not obey, might obey it. The natural reflex is to fix that at the source: tell the model, in its system prompt, to ignore instructions found inside images. The next chapter examines why that instruction is a request rather than a control, and what kind of boundary actually holds when the model's compliance cannot be relied on. The move is from keeping the payload out of the model — which the surface map shows is never fully achievable — to making it not matter that the payload got in.