Regulations and 120-token tickets fail in opposite directions

Every strategy in this topic so far has been chosen against one body of documents. A parent relevance ratio is measured on one collection, a window radius is swept on one collection, a diversity setting is tuned against one query mix. Production systems put one search box in front of material that does not share a shape, and the strategy that was correct for one half is then being applied to the other half by default rather than by decision.

The two shapes that pull hardest in opposite directions are a long document whose meaning depends on its own structure, and a short record that is complete on its own. A regulation is the first: clause (b) means nothing without the subsection that scopes it, and a definition stated once is referenced by number for sixty pages afterwards. A support ticket is the second: a symptom, a diagnosis and a resolution in 120 tokens, independent of whatever was filed before it. Chunking both at 512 tokens is the configuration everyone starts from, and it is wrong for each of them in a different direction — too small for one, too large for the other — which is why the resulting bug reports look unrelated and have one cause.

Two shapes, two prescriptions#

A regulation is a document whose meaning is carried by its hierarchy. Clause (b) means nothing without the subsection that scopes it, definitions are stated once and referenced by number for sixty pages, and a numbered heading path is a reliable, machine-readable structure that most corpora do not have. The strategy that follows is to index leaves at clause level with the ancestor heading path prepended to the embedded text, keep the numbered subsection as the parent, and merge upward when several clauses of one subsection match.

A ticket is the opposite kind of object. It is short, complete, independent of its neighbours, and one of thirty near-identical records describing the same recurring problem. Splitting is not just unnecessary but actively harmful, since any chunk boundary drawn at 512 tokens falls in the middle of the fourth ticket. The strategy is one ticket to one chunk, no expansion of any kind, and a diversity constraint over a wide pool with a per-source cap.

The two columns are the same seven decisions answered oppositely, and the returned-tokens row is where the opposition becomes a number you can act on. Six ticket hits at 720 tokens is exactly six times 120, because a ticket is sent whole and a ticket is 120 tokens; that row is a multiplication with nothing measured in it. Six regulatory hits at 2,600 tokens divides out to a little over four hundred tokens a hit, which sits at the top of the clause range in the first row and above it — the excess being the merges that fire and replace clause leaves with the subsections above them. So one column's figure is derived and the other's is measured, which is the same asymmetry chapter 3 found between a window and a parent, arriving here as a difference between two corpora rather than between two techniques.

The prepended heading path is the cheapest high-value move on the regulatory side and it is easy to miss. Embedding the string Part 4 > 4.2 Notification requirements > 4.2.1(b) — The issuer shall provide such notice within thirty days... gives the vector something to match on for queries phrased in the regulation's own vocabulary, and it gives the generator the citation it needs to attribute the sentence. It does not solve the dangling reference — "such notice" still needs §4.2 — which is what the parent link and the merge rule are for.

What a shared index and a shared top-k break#

Putting both corpora behind one index and one k produces three failures that are independent of chunking, and they compound.

The first is that the numbers work against the regulations. There are 40,000 tickets and roughly 96,000 regulatory clauses, but tickets are written in the same register as user queries — "checkout fails with a 502" — while regulations are written in statutory language. Dense retrieval rewards that lexical and stylistic proximity, and on the observed traffic the top-10 for a mixed query was 7.2 tickets and 2.8 clauses on average. Those two figures account for the whole list with nothing unclassified, so the composition is complete rather than a sample of it: a user asking what the regulation requires got seven parts in ten of what other users had said about it.

The second is that one k denotes two different amounts of context. Six regulatory hits after merging is 2,600 tokens; six ticket hits is 720. Follow that to a shared k=10 and the range is wide in a way no budget can absorb. An all-ticket result set is ten units of 120 tokens, which is 1,200 exactly. An all-regulation set at the same k is ten units whose size depends on how often the merge rule fired, and where it fires on every hit the units are subsections rather than clauses — the same kind of object as the §4.2 preamble in the bug report below, which sits 900 tokens above the clause that referenced it. Ten of those is 9,000. So the prompt lands anywhere from 1,200 to 9,000 tokens depending on a mix the query does not control, and any budget or truncation logic downstream is operating on a distribution it cannot predict — the same tail problem from chapter 3, now driven by which corpus won rather than by which section was long.

The third is authority. A ticket in which an agent told a customer the retention period was 90 days sits in the same prompt, with the same apparent standing, as the clause specifying 180. Mixing sources of unequal authority without labelling them leaves the arbitration to the generator, which has no basis for it: nothing in the passage text says one of them is a requirement and the other a report of practice.

Fanning out to a retriever per corpus and fusing by rank fixes all three at once, and each fix is a different property of the diagram. Each corpus gets its own strategy, its own k and its own diversity setting, so the first failure disappears by construction — the regulation branch returns its six whatever the ticket branch does. Reciprocal rank fusion combines the two lists using positions rather than scores, which matters because cosine scores from two corpora with different length distributions are not on a common scale and comparing them is the arithmetic equivalent of adding metres to seconds. And the source label survives into the prompt, so the generator can be instructed that a regulation outranks a ticket.

def rrf_fuse(rank_lists, k_const=60):
    """Reciprocal rank fusion. Uses positions only — no score comparison."""
    scores = {}
    for hits in rank_lists:
        for rank, h in enumerate(hits, start=1):
            scores[h.id] = scores.get(h.id, 0.0) + 1.0 / (k_const + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)

regs   = regs_retriever.search(q, top_k=6)     # clause leaves, auto-merged
tickets = ticket_retriever.search(q, top_k=8)  # flat, MMR-selected
fused  = rrf_fuse([regs, tickets])
final  = reranker.rerank(q, fetch(fused[:20]), top_n=6)

The k_const in that function is what keeps the fusion from being dominated by first places. Without it, a rank-1 passage would score one and a rank-2 passage a half, so the top of either list would outweigh everything else; adding a constant of 60 to the rank flattens the curve, so the difference between rank 1 and rank 2 becomes small and the difference between rank 1 and rank 20 stays meaningful. That is a design decision rather than a requirement. The requirement is only that the two lists be combined without comparing their scores, and any monotone function of rank satisfies it.

The alternative to fanning out is routing: classify the query and send it to one corpus. It is cheaper — one retrieval instead of two — and it is the right call when latency budgets are tight and the two corpora genuinely never co-occur in an answer. On this system they co-occur constantly, because the useful answer to "do we have to notify customers of a breach within 72 hours" is the clause plus the three tickets where it came up in practice, and a hard router serves one of those. Where a router does earn its place here is as a modifier rather than a switch. Adjust the per-corpus k by predicted intent, so a clearly statutory query gets k_regs=10, k_tickets=2, and an operational one gets the reverse, so that no query gets zero from either side.

Splitting without a migration project#

Nothing above requires two vector databases. Adding a corpus field to every record's metadata and issuing two filtered searches against the same physical store gets the per-corpus k and the per-corpus selection policy immediately, and the two searches run concurrently so the added latency is one round trip rather than two. What separate namespaces or collections buy on top of that is independent reindexing — re-embedding the clause leaves without touching 40,000 tickets — and independent ANN parameters — the settings of the approximate nearest-neighbour structure the vector store uses to avoid comparing a query against every vector it holds — which matter once the corpora differ enough in size that one wants a different graph configuration than the other. Both are worth having eventually and neither has to block the split.

The metadata schema is where the two corpora stop pretending to be alike. Regulations carry a jurisdiction, a part number, an effective date and a superseded-by pointer; tickets carry a product area, a resolution state and a customer tier. In a shared schema, every record has half its fields null, which is harmless in storage and misleading in code, because a filter written for one corpus silently excludes the entire other one — effective_date <= today drops every ticket, since tickets have no effective date. Filters have to be applied per branch of the fan-out, which is another reason the branches want separate retriever objects rather than one function with an if.

The effective-date field is load-bearing in a way that has no counterpart on the ticket side. A corpus of 8,000 regulatory PDFs contains superseded amendments, and a clause that was correct in 2022 is indistinguishable from a current one by embedding similarity — they are near-identical text differing in a number. Filtering to in-force versions at query time, or at minimum labelling superseded material in the prompt, is the difference between an assistant that is occasionally wrong and one that is wrong in the specific way a regulator would notice. This is the same near-duplicate problem the previous chapter treated as redundancy, arriving here as a correctness problem, and geometry cannot solve it: the two clauses differ in a date held in metadata, and the embedding of a clause is computed from its text.

The prompt assembly deserves as much design as the retrieval. Passages from both corpora go into the context labelled by type and, for regulations, by citation, with an instruction that a regulation states the requirement and a ticket describes practice, so where they conflict the regulation governs and the ticket may be cited as context. On the judged set, adding those labels and that instruction moved accuracy on queries where the two sources disagreed from 58% to 81%, without any retrieval change at all. Twenty-three points from a prompt edit is the cheapest fix in this chapter, and it works only once the sources are distinguishable, which is what the fan-out provides.

One caution about the shared reranker at the end of the fan-out. Cross-encoders see the query and the passage as a single sequence, so a 120-token ticket and a 2,600-token merged clause are not being scored under comparable conditions, and long passages get truncated to the model's input limit in ways that can cut off the relevant sentence. The reranker is being trusted to arbitrate between the two corpora while its own scoring quality varies with a property — passage length — that differs by a factor of twenty between them. Reranking before expansion, with clauses at leaf size and tickets whole, keeps the comparison closer to like-for-like, and merging afterwards preserves the ordering the cross-encoder produced.

The measurement that tells you the split was right#

Aggregate accuracy over mixed traffic will not confirm any of this, for the reason chapter 4 established about pooled numbers over a heterogeneous set. The evaluation has to be stratified by where the gold evidence lives, which means the judged set needs that annotation on each query.

Three strata and three numbers each. For regulation-answered queries, measure clause-level evidence recall — did the specific clause containing the requirement reach the prompt — and separately whether the cited section number in the answer is correct, since a plausible answer citing the wrong clause is worse in this domain than a refusal. For ticket-answered queries, measure facet coverage on the redundant slice from the previous chapter. For queries whose gold evidence spans both, measure whether the prompt contained at least one passage from each corpus, which is the number that distinguishes fan-out from routing and is exactly what a hard router destroys.

One additional diagnostic is worth wiring into production rather than the eval harness: the fraction of served prompts containing zero passages from the corpus that holds the gold evidence. Before the split, on regulation-answered queries, that was 21% — one query in five never saw a regulation at all. After fan-out with fixed per-corpus k, it is zero by construction, and the phrase is exact rather than rhetorical: the regulation branch is asked for its own k and returns it, so a prompt with no regulation in it cannot be produced unless the branch itself fails. Watching that number stay at zero is also the cheapest alarm for a misconfigured index, because a corpus that stops being retrievable shows up here immediately instead of as a slow accuracy drift.

Two corpus shapes, then, and two of everything: two units, two expansions, two selection policies, two values of k, and a fusion step that never compares their scores. What makes this more than a preference is that every failure the shared configuration produced was a fixed setting meeting a quantity that varied — 512 tokens against two different sizes of complete thought, one k against two different token costs, one similarity scale against two different score distributions, one reranker against two different passage lengths. None of those is repaired by tuning the constant, because there is no value of a constant that fits two things at once.

Splitting the strategies fixed the failures both bug reports described and left one class of question untouched. A query like "which of our incident types are most affected by the notification rules that changed this year" needs material from forty documents at once, and nothing in the fan-out above can construct that, because every retriever on both sides returns passages and no passage contains the answer. Both corpora are indexed at the granularity of things that were written down, and the question asks about a pattern across them.