A fixed window cuts the body off the signature

A billing monorepo — one repository holding every service and library the billing organisation owns — roughly 4,300 Python files and 1.1 million lines, indexed the obvious way: read each file as text, split it into 1,200-character windows with no overlap, embed every window, store the vectors. The ticket that arrives says refunds on mid-cycle downgrades are prorating by whole days again and should follow the same seconds-based rule as upgrades. Retrieval returns ten chunks and the top one, at cosine 0.83, comes from billing/proration.py. It is the right file, it is the right function, and the agent's patch is wrong before it writes a line, because of where character 1,200 landed.

That pipeline has a name. Repo-scale code retrieval is the step that decides which few thousand lines of a repository a coding agent is allowed to look at before it writes anything: the repository is read once and cut into pieces, the pieces are stored in a searchable index, and when a task arrives the index returns a small set of them to be pasted into the model's prompt. The pieces are called chunks and the build step is indexing, in the same sense the chunking-strategies topic used both words for documents. What is different here is only the material — source code instead of prose — and that difference turns out to run deeper than it sounds.

It exists because handing over the whole repository does not work arithmetically. At roughly twelve tokens per line, 1.1 million lines of Python is on the order of thirteen million tokens, which is past any context window on the market and, at list prices around $3 per million input tokens, would cost about $39 in prompt alone for every ticket even if it fit. Latency would be measured in minutes and the model's accuracy would fall rather than rise, since the useful thirty lines are buried in noise. Something has to reduce 4,300 files to something like the twenty or thirty the agent actually reads, and it has to do that before the model sees a character. Retrieval is that something, and it sets the agent's ceiling, because an agent cannot correctly change a function it was never shown.

You meet it whenever a coding assistant works on a codebase nobody trained it on: an editor agent answering "where is refund proration handled?", a bot that opens a pull request from a ticket, a review tool deciding which files a change ought to have touched. Each of them runs some version of the pipeline in the first paragraph before anything visible happens. The rest of this chapter is the smallest thing that can go wrong in it, which is a boundary landing in the wrong place.

Where the boundary actually fell#

The file opens with a module docstring, four imports, a comment, and a constant, and then declares the function the ticket is about.

"""Proration helpers for subscription changes.

All amounts are in minor units (cents). Nothing in this module
touches the database; callers pass in a materialised Subscription
and a change timestamp, and get back a signed delta.
"""

from datetime import datetime
from decimal import Decimal, ROUND_HALF_EVEN

from billing.calendar import cycle_bounds
from billing.models import Subscription

# Charges below this threshold are dropped rather than billed,
# because the processor's per-transaction fee exceeds them.
MIN_BILLABLE_CENTS = 50


def calculate_proration(sub: Subscription, at: datetime, new_plan_cents: int) -> int:
    """Return the signed proration delta in cents for a plan change.

    The unused portion of the current plan is credited and the
    remaining portion of the new plan is charged, both prorated by
    the number of whole days left in the billing cycle. A positive
    return value is a charge; a negative value is a credit.
    """
    start, end = cycle_bounds(sub, at)
    cycle_seconds = Decimal((end - start).total_seconds())
    remaining = Decimal((end - at).total_seconds())
    fraction = remaining / cycle_seconds

    credit = (Decimal(sub.plan_cents) * fraction).quantize(
        Decimal("1"), rounding=ROUND_HALF_EVEN
    )
    charge = (Decimal(new_plan_cents) * fraction).quantize(
        Decimal("1"), rounding=ROUND_HALF_EVEN
    )
    delta = int(charge - credit)

    if 0 < abs(delta) < MIN_BILLABLE_CENTS:
        return 0
    return delta

The preamble — docstring, imports, comment, constant, blank lines — runs to character 806. The def line adds 92 more, and the function docstring carries the count to 1,198. The window closes at 1,200, which is two characters into the first line of the body, so the first chunk ends with st and the second chunk opens with art, end = cycle_bounds(sub, at). The split does not merely separate the interface from the implementation; it severs an identifier down the middle.

What the index now holds for this function is two objects that no longer describe each other:

chunk 4,118  (chars 0–1199 of billing/proration.py)
  module docstring, imports, MIN_BILLABLE_CENTS, the def line,
  the four-line function docstring, and the string "    st"

chunk 4,119  (chars 1200–2062 of billing/proration.py)
  "art, end = cycle_bounds(sub, at)" followed by the arithmetic,
  the quantize calls, the MIN_BILLABLE_CENTS guard, and the returns.
  Contains no "def", no function name, and no docstring.

Chunk 4,119 has every line the ticket cares about. It contains ROUND_HALF_EVEN, the seconds-based fraction, and the floor that drops sub-threshold amounts. It also contains nothing that identifies it. A lexical search for calculate_proration misses it, because the name appears only in the chunk above. An embedding of it lands in the region of vector space occupied by generic decimal arithmetic, which is nowhere near the region occupied by an English sentence about refunds.

The half made of prose is the half that wins#

Ranking these candidates against the task text makes the asymmetry explicit.

The ordering is not a bug in the embedding model. A task description is a paragraph of English about money and time, and the chunk that scores highest is the chunk that is also a paragraph of English about money and time. Docstrings, comments and module headers are written in the same register as tickets, so they occupy the same neighbourhood in embedding space. Executable arithmetic is written in a different register entirely, and its similarity to a sentence is low no matter how central it is to the change. Every fixed-window index over a code repository has this bias, and the cases where it bites hardest are exactly the cases where a boundary happens to isolate the prose.

It is worth being clear about where the 1,200 came from, because nobody chose it for code. Character windows in the range of 800 to 1,500, often with 10% to 15% overlap, are the settled default of document retrieval pipelines, and they are settled there for a good reason: a window of that size holds two or three paragraphs of English, which is roughly the span over which prose stays on one topic. The number encodes an assumption about how far a unit of meaning extends in writing. A Python function has its own answer to that question, the answer varies from four lines to four hundred, and the character count that was tuned against paragraph length has no relationship to it. Pointing a document pipeline at a repository imports that assumption silently, along with the rest of the defaults.

The failure also does not register as a failure by the retriever's own accounting. The correct file was retrieved. If the labelled evaluation records that billing/proration.py is the file the task touches — and it is — then recall at ten counts this as a clean hit. There is a whole class of retrieval defects that live inside a nominally successful retrieval, and this is the first of several the topic will keep running into.

Rewriting a function from its documentation#

The agent receives chunk 4,118, the policy document, and the two upgrade tests. From that context it can see the signature, the return contract, and a four-line statement of intent. It cannot see how the intent is currently implemented, so it writes an implementation that satisfies the docstring.

def calculate_proration(sub: Subscription, at: datetime, new_plan_cents: int) -> int:
    """Return the signed proration delta in cents for a plan change.

    Prorated by the number of whole days remaining in the cycle.
    """
    start, end = cycle_bounds(sub, at)
    total_days = Decimal((end - start).days)
    remaining_days = Decimal((end - at).days)
    fraction = remaining_days / total_days

    credit = (Decimal(sub.plan_cents) * fraction).quantize(
        Decimal("1"), rounding=ROUND_HALF_UP
    )
    charge = (Decimal(new_plan_cents) * fraction).quantize(
        Decimal("1"), rounding=ROUND_HALF_UP
    )
    return int(charge - credit)

Three things changed that nobody asked to change. The granularity moved from seconds to whole days, which is the opposite of what the ticket requested and is what the stale docstring said. The rounding mode moved from half-even to half-up, because ROUND_HALF_UP is the more common idiom in training data and there was no reason on screen to prefer the other. The MIN_BILLABLE_CENTS guard disappeared, since it existed only in the severed half.

The test suite catches one of the three. A downgrade case that expects a credit of 1,183 cents now produces 1,200, and the run fails with assert -1200 == -1183. The rounding change is invisible at this scale because half-even and half-up differ only on exact .5 boundaries, which the fixtures do not contain. The dropped floor is invisible because no test ever exercised a delta smaller than 50 cents.

The floor is the one that costs money. Over the following three weeks the service issues 4,106 charges below the threshold, averaging 11 cents each, for about $451 of revenue against a processor fee of $0.30 per transaction — roughly $1,232 in fees, a net loss of $781 and a support queue full of customers asking about seven-cent card charges. The line that prevented this was two lines of Python that were sitting in the repository, unretrieved, three characters after a window boundary.

There is a second-order effect worth naming. Because the agent could not see the body, it did not produce a small diff. It produced a full-function replacement, and a full-function replacement passing review looks like a refactor rather than a mistake, so the reviewer reads it for internal consistency rather than for what it removed. Chunking defects arrive at code review disguised as style.

Running the boundary-crossing case on your own index#

None of this has to be taken on trust, because the property that causes it is directly measurable against an index that already exists. Take every function definition the parser can find, look up which chunks its byte range overlaps, and count.

def boundary_report(files: Iterable[str], chunks: ChunkIndex) -> Counter:
    report = Counter()
    for path in files:
        for fn in functions_in(path):                 # (start_byte, end_byte, def_end)
            spans = chunks.overlapping(path, fn.start, fn.end)
            report["functions"] += 1
            if len(spans) > 1:
                report["split"] += 1
            if chunks.chunk_at(path, fn.def_end) is not chunks.chunk_at(path, fn.end):
                report["signature_separated_from_tail"] += 1
            if len(spans) == 1:
                report["intact"] += 1
    return report

On the billing monorepo, 1,200-character windows with no overlap produced 38,914 function definitions, of which 27,600 crossed at least one boundary — about 71%. Of those, roughly a fifth had the signature and the closing lines of the body in different chunks, which is the population where the retrievable half can be documentation with no implementation attached. Running the same report after switching the index to syntax boundaries takes the split count to zero for every file that parses, and the difference between the two reports is a number that can be put in front of a team that is not sure the problem is real.

The report is also useful as a targeting tool, because the functions that split are not uniformly distributed. Long functions split by definition, and long functions are disproportionately the ones carrying business logic that nobody has refactored, which is disproportionately what tickets are about. Sorting the split list by function length and by how often the file appears in recent commits produces a short list of the places where this failure is most likely to be actively costing something.

What overlap and bigger windows do not fix#

The reflex repair is overlap, and it helps in a specific and limited way. Two hundred characters of overlap means chunk 4,119 now begins 200 characters earlier, so it starts inside the function docstring instead of inside an identifier, and it does carry a fragment of the contract. It does not carry the def line, so the chunk still has no function name and still will not be found by anyone searching for one. Overlap moves every boundary; it does not remove boundaries, and on a 400-line file with fifteen functions there are still fourteen places where something is cut. What overlap buys is that the cut is less likely to be catastrophic on any given function, at the cost of storing and searching roughly 17% more chunks.

There is a variant of the overlap repair that does slightly better and is worth knowing about, which is to make the split points aware of blank lines and indentation without doing a real parse. Splitting at the last blank line before the character limit, and preferring a line at indentation level zero, moves most boundaries to the gaps between top-level definitions in Python and to the gaps between blocks in C-like languages. It costs nothing, it is language-agnostic, and on the same repository it took the proportion of split functions from 71% down to about 34%. That is a large improvement for an afternoon of work and it is still one function in three, because a function longer than the window has to split somewhere and indentation heuristics have nothing to say about where.

Bigger windows trade one failure for another. Raise the window to 8,000 characters and calculate_proration survives intact, along with six unrelated functions, a class, and a block of imports. The embedding of that chunk is an average over seven unrelated topics, which is a worse match for any single one of them than a tight chunk would be, and the retrieved unit now costs about 2,000 tokens of context to deliver one 30-line function. Chunking strategies covered this trade in general; code sharpens it, because the unit whose integrity matters is a syntactic unit whose length nobody controls.

The honest scope of the problem is narrower than "fixed windows are broken". On a 40-line utility module the window contains the whole file and nothing is severed. On a README, a migration log, or a changelog, character windows are entirely appropriate, since those files have no syntactic structure worth preserving and their content is prose that the embedding model handles well. The failure is specific to files where a semantic unit — a function, a method, a class, a type declaration — is longer than the window or straddles its boundary, and where the interface half and the implementation half are separated by that boundary. In a monorepo, that describes most of the source.

What follows from this chapter is a boundary rule rather than a boundary size: cut where the language says one thing ends and another begins, so that a chunk is a function or a method rather than a span of characters. That fixes the severed body and immediately creates a different gap, which the next chapter opens with — a method retrieved whole, patched correctly on its own terms, breaking a rule stated fifteen lines above it in a place the chunk no longer reaches.