Delimiter collision rates

Choosing a delimiter looks like a matter of taste and is usually argued as one. Triple backticks are the idiomatic markdown way to say "this is a literal chunk". Angle-bracket tags are what most published prompts and provider guides use. A banner line such as ### TRANSCRIPT ### reads clearly to a human and costs nothing. All three plausibly mark a region, and a review thread can run for two days on which is best.

The question that settles it is not aesthetic. A marker fails when the text it encloses contains the marker, because then the boundary the template drew is no longer the only place that boundary appears in the rendered prompt. How often that happens is a measurable property of the text your service actually receives, it differs by two orders of magnitude between candidate markers on the same traffic, and nobody in the argument usually has the number.

That first sentence deserves its steps, because "the boundary appears twice" sounds untidy rather than dangerous. Work forward from what the model has to do. It receives one sequence of tokens and no separate record of which stretch of it your template supplied, so where a region stops is something it works out from the sequence itself — and the only evidence in the sequence is the marker. Now put a second copy of that marker inside the payload. There are now two readings of the same text, both entirely consistent with what is written: the region runs to the payload's copy, or it runs to your template's copy. No fact in the prompt distinguishes them, because the two copies are the same tokens in a sequence that records nothing about origin. The model resolves the ambiguity the only way it resolves anything, by predicting what usually follows text shaped like that, and a payload author who understands the first reading gets to write everything after their copy in the voice of your template.

That is why a collision is not a rendering blemish. It is the payload being handed the one thing a delimiter was supposed to keep for you, which is the ability to say where your own text resumes. Ordinary customers do it by accident constantly, and how often they do it is the number the argument is missing.

Count the collisions before you argue about syntax#

The measurement is a grep over a corpus you already have. Take a summarisation service that turns a support chat transcript of up to about four thousand words into a three-sentence summary plus an escalation flag; a day's worth of transcripts sits in the analytics warehouse, so the question is answerable in a few minutes.

import re

MARKERS = {
    "triple backticks":  re.compile(r"`{3}"),
    "hash banner":       re.compile(r"^###\s", re.MULTILINE),
    "<transcript> tag":  re.compile(r"</?transcript\b", re.IGNORECASE),
}

total = 0
hits = {name: 0 for name in MARKERS}

for transcript in load_transcripts(days=1):     # 41,882 transcripts
    total += 1
    for name, pattern in MARKERS.items():
        if pattern.search(transcript):
            hits[name] += 1

for name, n in hits.items():
    print(f"{name:20s} {n:6d}  {n / total:6.2%}")

triple backticks       4738  11.31%
hash banner             879   2.10%
<transcript> tag          6   0.01%

Three details of that script are deliberate. It counts transcripts rather than occurrences, because what matters for the design is how many requests are affected and not how many backticks a single chatty customer managed to type. A single day is enough to separate eleven percent from one percent, though it is not enough to characterise anything rarer. And the window was chosen to include a Tuesday rather than a weekend, because the traffic mix differs.

The percentages in that output are one division each, and doing them by hand is worth the minute because it fixes what the columns mean. Forty-seven hundred and thirty-eight transcripts out of forty-one thousand eight hundred and eighty-two is a shade over one in ten, which prints as 11.31%. Eight hundred and seventy-nine out of the same total is about one in fifty, which prints as 2.10%. Six out of that total is one in seven thousand or so, which prints as 0.01% and is the point at which the second decimal place stops carrying information — six hits and five hits round to the same figure. So the ranking is not three points on a scale. It is one marker that collides routinely, one that collides occasionally, and one that collides so rarely that the count matters more than the rate.

Eleven percent is not an edge case. On a service handling forty thousand transcripts a day, choosing triple backticks means roughly forty-seven hundred prompts a day in which the marker that is supposed to delimit the transcript also appears inside it. The hash banner is better by a factor of five and still puts nearly nine hundred prompts a day into a state where a line in the customer's own text is typographically identical to the structure of your template. The six hits on <transcript> were all from customers pasting examples of prompts they were building, which is a real category and one that will grow, but it is three orders of magnitude away from the backtick number.

Those percentages are properties of that corpus. A service summarising restaurant reviews would see almost no backticks and might see ### never at all; a service processing scraped HTML would see <transcript> about as often as it sees any other tag name, which is to say occasionally. The measurement is cheap and it is the only thing that turns three plausible arguments into one decision, which is why it is worth running before the design discussion rather than after something breaks.

The three options against the criteria that matter#

Two rows on that table are decisive and one is a distraction. The distraction is token cost: across five sections, the difference between markdown headings and XML-style tags is on the order of fifteen tokens per request, which at Claude Sonnet 5's list price of $3 per million input tokens is about four thousandths of a cent per request. Nobody should be choosing a delimiter on that basis, and if someone in the review argues token efficiency, the number is the fastest way to move the conversation on.

That figure is one step of arithmetic and it is worth doing out loud, because a price quoted per million is hard to feel and a price per request is not. A million tokens costing three dollars means one token costs three millionths of a dollar. Fifteen tokens therefore cost forty-five millionths of a dollar. A cent is a hundredth of a dollar, so dividing by a hundredth — multiplying by a hundred — turns forty-five millionths of a dollar into forty-five hundred-thousandths of a cent, which is a bit over four thousandths of one. To spend a single cent on the choice, a service has to send a couple of hundred requests. The argument is not that the difference is small enough to ignore for now. It is that the difference cannot become large enough to matter before the collision rate has already decided the question.

The decisive rows are the collision rate and the direction of the marker. A fence and a banner are symmetric: the same string opens and closes the region, so a payload containing one produces a rendered prompt in which the region boundaries are genuinely ambiguous — not merely wrong, but ambiguous, in the sense that two different readings are equally consistent with the text. Paired tags are asymmetric, so a stray </transcript> inside the payload produces a prompt that is wrong in a specific, inspectable way. That difference matters much less for the model's behaviour than people assume and much more for your ability to write a test that catches it.

Symmetry is the reason, and it is a counting argument rather than a matter of taste. With a fence, every occurrence of the marker is both a possible open and a possible close, so a rendered prompt containing three of them supports several readings and no rule over the string can prefer one — a checker looking at it can report how many fences there are and nothing else, because there is no such thing as a fence that is out of place. With paired tags the two ends are different strings, so a checker can require exactly one open, then exactly one close, in that order, and a payload that contributes an extra </transcript> breaks that requirement at a known offset. The model may well be confused by either prompt to a similar degree. Only one of the two lets your render path find out.

The property you are actually buying#

Rank the three options by collision rate and you get an ordering. Notice what the ordering is made of and you get the design rule, which is that none of these markers is safe and one of them is merely less likely to collide on this particular corpus this particular week. What you want is a marker that cannot occur in the data, and probability is not the same as cannot.

You get "cannot" from one place: treating the payload before it is rendered. A marker becomes impossible in the data when the render path guarantees it, and the render path guarantees it by scanning and doing something about a hit.

The reason that particular place is the only one available follows from where the components sit. Downstream of the render call there is nothing left that can look at the payload separately, because after the render call the payload is not a separate object — it is a substring of a longer string, indistinguishable from its neighbours by anything except the marker whose reliability is the question. Upstream, in the render path, the payload is still its own value, held in its own variable, with a known beginning and end that came from your storage layer rather than from anything in the text. So there is exactly one moment in the request's life when "does this contain my closing tag" is a question with a definite answer, and that moment ends the instant you concatenate. Every technique in the rest of this topic is a variation on doing something at that moment.

CLOSING = "</transcript>"

class DelimiterCollision(ValueError):
    pass

def render(template: str, transcript: str) -> str:
    if CLOSING.lower() in transcript.lower():
        raise DelimiterCollision(
            f"payload contains {CLOSING!r} at offset "
            f"{transcript.lower().index(CLOSING.lower())}"
        )
    return template.format(transcript=transcript)

Raising is one policy among several and it is the wrong one for a summarisation service, where a hard failure on six transcripts a day is worse than a slightly mangled summary. The point of writing it this way is that the collision is now a named, observable event with a counter attached to it, instead of a silent property of the rendered string. A team that has never scanned does not know its collision rate; a team that raises knows it exactly, and can then argue about what to do with the six.

There are three things you can do with a hit, and the collision rate decides which is affordable. Reject is the strictest and only survives at a rate near zero, because every rejection is a request the customer does not get an answer to. Escape — rewriting the offending substring into something that cannot close the region — keeps every request working and pays for it by sending the model a payload that differs from the one the customer supplied, which matters enormously for some tasks and not at all for others. Route sends colliding payloads to a variant template with a different marker, so nothing is rewritten and nothing is refused, at the cost of a second template to keep in step with the first. What makes this a design decision rather than a detail is that the right answer changes with the rate: at 0.01% any of the three is fine and rejection is defensible, at eleven percent only escaping and routing are on the table, and escaping is exactly the option that a code-heavy corpus cannot tolerate.

Follow that last clause through, because it is where the three policies stop being interchangeable. Rejecting six transcripts a day means six customers a day who get an error instead of a summary, which is a support cost somebody can price. Rejecting forty-seven hundred a day is not a policy, it is an outage with a nicer error message. Escaping scales fine at either rate and pays in a different currency: the model is now summarising a transcript that differs from the one the customer sent. For a marker occurring once in seven thousand messages that difference is invisible. For a marker occurring in one message in ten, on a corpus where the marker is a code fence and the fences are delimiting the code the summary is about, escaping mangles the substance of the very thing being summarised. Routing avoids both by keeping a second template alive, and the cost lands on the team rather than on the request: two templates that must be changed together, and a bug class where they drift. Which cost you can carry is a question about your service, and the collision rate tells you which of the three conversations you are actually having — whether the decision is affecting six transcripts a day or several thousand.

This reframes the original question. It is not "which of these three syntaxes is safest", because the answer to that is "none of them, they are all text". It is "which syntax has an acceptable collision rate on our payload, given the treatment we are going to apply anyway".

Two markers not to reach for#

Two ideas come up in this discussion often enough to be worth heading off, and both are attempts to drive the collision rate to zero by choosing a cleverer string rather than by treating the payload. That is the shape to watch for, because a marker chosen purely to minimise collisions optimises one column of the table at the expense of columns nobody thought to check.

The first is an exotic sentinel — a rare Unicode character, a private-use codepoint, a string of control characters — chosen on the grounds that no customer will ever type it. The collision argument is sound and three other things go wrong. Rare characters tokenise badly, so a sentinel that looks like one character can cost eight or ten tokens on each side of every region. They are invisible in logs and in terminal output, so the first time you are debugging a rendered prompt at three in the morning you cannot see your own boundaries. And anything upstream that normalises text — a Unicode NFKC pass in an ingestion pipeline, which rewrites characters into a canonical equivalent form and folds many rare ones into ordinary ASCII, a database column with a restrictive collation, a JSON round-trip through a service that strips control characters — can silently remove or alter the sentinel, at which point your region has no boundary at all and nothing errors.

The second is a natural-language marker: The transcript begins now: before and The transcript ends here. after. It reads well, costs almost nothing, and gives you no boundary you can test on, because a regex for it will match the same sentence appearing in a customer's own message and because there is no mechanical way to assert that exactly one of each appeared in the right order. It also inverts the property you want — a phrase that reads naturally is exactly a phrase a payload might contain naturally. The marker should be conspicuous to a machine and unremarkable to a reader, which is roughly what an angle-bracket tag is.

The belief underneath both ideas is worth naming, because it is what makes them feel clever: a marker rare enough in the data is a marker that cannot be forged. Rarity and impossibility are different properties, and the distance between them is the subject of this chapter. Rarity is a statement about traffic already seen. It says nothing about traffic not yet seen, and nothing at all about traffic chosen by somebody who has read your prompt, since the moment a marker is discoverable its collision rate among adversarial payloads is whatever the adversary decides it should be. A check in the render path is different in kind: it makes a claim about the message actually in front of you, and it works exactly as well on a common marker as on a rare one. Rarity buys fewer treatments, which is a cost question. It never buys the guarantee.

Whole-template layout is the same question#

The same reasoning settles the other version of this argument, which is whether a whole prompt should be structured with markdown headings (## Task, ## Rules, ## Document) or with XML-style tags around the same five sections. This is the form the question usually arrives in, because most teams have a working template before anyone thinks about payload collisions, and the review that raises the delimiter question is a review of the whole layout. Four of the five sections are written by you and cannot collide with anything, so the argument is really about the fifth, which is the only one holding text you did not write.

Markdown headings read better in a pull request diff, they are what a human writing documentation would produce, and a long prose section under ## Rules looks like a document rather than like configuration. XML-style tags give you an explicit end to every region, which matters when a section is long enough that the model has to decide where ## Rules stopped and ## Examples started, and they let a test parse the template strictly and assert that all five sections are present exactly once. Provider documentation tends to recommend the tags, and that recommendation is real.

Then someone says the documents are scraped web pages that frequently contain markdown and occasionally contain HTML, and asks whether that changes the answer. It does not change it in the direction people expect. Scraped pages contain ## at the start of a line often enough that a markdown-delimited document section is unreliable, which is the argument for tags. Scraped pages also contain HTML, and an untreated page containing </document> breaks the tagged template exactly as thoroughly as a fenced block breaks the markdown one. Both templates fail on the same class of input; they just fail on different subsets of it.

Choosing tags is also not choosing XML. Nothing parses the prompt — not your code, not the model, which sees tokens — so the tags need not be well-formed, need not be namespaced, and gain nothing from being valid against a schema. <transcript> and [[TRANSCRIPT]] and <<<transcript>>> are the same kind of object, and the only reasons to prefer the angle-bracket form are that it is asymmetric, that it is what most published prompts look like so the model has seen a great deal of it, and that other engineers will recognise it. Do not let a reviewer send you down the path of escaping payloads to be XML-safe as though a parser were waiting downstream.

What transfers between services, then, is not a conclusion but a procedure, and the eleven-percent figure belongs to one corpus rather than to markdown. Grep your own traffic for each candidate marker, rank by collision rate, pick a marker whose collisions you can afford to treat, decide in advance and in writing whether a hit is escaped, rejected or routed to a variant template, and instrument the treatment so the rate stays visible. The syntax argument is then a ten-minute decision that a measurement makes for you.

That leaves the case the scan finds. A transcript arrives containing the literal string </transcript>, the render path has to do something, and every available something has a consequence — for the model's behaviour on that request, for the token prefix you were caching, and for what your test suite is able to assert. That is the next chapter.