Writing a relevance rubric

A language model's training data stops at a fixed date, and it never contained your company's internal documents at any date, so the only way an assistant can say anything about them is to be handed the relevant text at the moment the question arrives. That handing-over is retrieval: before the model writes anything, a search runs over the collection of documents the system is allowed to read — its corpus — and the few passages that score highest are pasted into the prompt above the user's question. The arrangement of a retrieval step feeding a generation step is called retrieval-augmented generation, written RAG almost everywhere, and it is what sits behind any product that answers from documents the model was never trained on: a support assistant, a help-centre search box that replies in sentences instead of links, an internal tool that answers questions out of a company wiki.

Which passages score highest is decided by a ranking function, and the usual one works geometrically. Each passage, and the query, is converted into a list of several hundred numbers called an embedding, positioned so that texts about the same subject land close to each other, and a pair is scored by cosine similarity, a number that runs from around 0 for unrelated text to 1 for two texts pointing in exactly the same direction. Relevance is a different quantity: the judgement of whether a passage that came back was the right material for what the user was actually trying to do. The retrieval problem, which this topic is about, is that the score and the judgement agree often enough to be mistaken for one another, and they diverge on the cases that reach a customer.

What a query carries besides its words#

A query is a short string — a handful of tokens, the subword pieces text is chopped into before any model reads it — and also a set of conditions that the requester never types, because in human conversation they go without saying. Nobody asking about a refund window means the 2021 one. Nobody asking "does Northwind have an active NDA with us?" means the NDA with a differently named company. Nobody asking their support agent for an escalation runbook means the internal-only version they are not cleared to see. These conditions are part of what makes an answer correct, and they are invisible to any scoring function that only sees the query string and the passage string.

Take the concrete case. A superseded policy page, ranked first for what is our refund window? at a cosine similarity of 0.91 against 0.88 for the page currently in force, is an excellent topical match and a wrong retrieval. Nothing in the embedding model is broken: it was asked which passage in the corpus is most about refund windows and it answered that question accurately. The question it was asked is the problem, because the task carries a constraint the score has no way to encode — the answer must reflect the policy in force — and a scoring function that saw the constraint would still have no field to put it in.

It is worth pressing on why the obvious defence does not work, because almost every team reaches for it first. The defence is a similarity floor: pick a cutoff, discard anything scoring below it, and the stale page goes away. Put this case's own numbers into that plan and the cutoff has nothing to bite on. The superseded page scores 0.91 and the page in force scores 0.88, so the wrong document is the higher of the two. Any floor low enough to admit the correct page at 0.88 is therefore also low enough to admit the wrong one at 0.91, and any floor high enough to exclude 0.91 excludes 0.88 with it and returns nothing at all. The property underneath that arithmetic is the one to carry forward. Two revisions of the same policy page are written in the same vocabulary about the same subject, so their embeddings point in nearly the same direction, while the fact that separates them — which one is in force — appears in neither text. A defence that stays roughly constant while the thing it has to discriminate changes completely is not a weak defence. It is not a defence.

Naming those conditions is the whole of rubric writing, and the person who applies them, the annotator, needs each one written as a test rather than as a value. There are five that come up often enough to be worth stating explicitly in almost every rubric, and each of them catches a distinct failure that topical similarity waves through.

Two of these are worth dwelling on because they are the ones teams leave out. Answerability is what separates a rubric from a topic classifier: a superseded policy page and the current one are equally about refunds, and only one of them answers the question as asked. Authority is what stops your labels from being determined by whoever wrote the most fluent prose in your corpus, which in most companies is a marketing page and in some is a Slack export.

It helps to see answerability fail on a passage that a person skimming would wave through. One policy corpus contains policies/refunds-overview.md, three paragraphs that explain the company's philosophy of refunds, mention that the window is short and generous relative to competitors, link to the detailed policy, and never state a number. Its similarity to what is our refund window? is 0.87, a hundredth of a point under the 0.88 the current policy page scores, because it uses the phrase "refund window" four times in three hundred words while the policy page uses it twice in nine hundred. Those two counts are the whole explanation if you turn them into densities. Four mentions across three hundred words is one every seventy-five words; two across nine hundred is one every four hundred and fifty. The overview writes about refund windows six times as densely as the document that defines them, which is enough to put a page that states no number level with the page the number lives on. An annotator reading quickly sees a document about exactly the right thing and reaches for relevant. Applying the test as written — a reader holding only this passage can state the answer without guessing — makes it a clean failure, and it is a failure worth catching, because a model handed that passage and nothing else will produce a fluent paragraph about the company's generous refund philosophy and no number at all.

The list is not fixed. A rubric for a medical or legal assistant will add jurisdiction. A rubric for an incident-response assistant will add environment, because a runbook for staging retrieved during a production outage is worse than nothing. What matters is that every dimension your task depends on is written down before anyone labels a single pair, because a criterion that lives in an annotator's head produces labels nobody can reproduce or audit.

Writing one for a named task#

A rubric that starts "rate how relevant this passage is on a scale of 1 to 5" produces numbers with no meaning. The version that works starts by naming the task in one sentence specific enough that it excludes things, then gives each dimension a test that an annotator can apply without judgement calls, then defines a scale so short that there is nowhere to hide.

task: >
  A support-agent-facing assistant that answers questions about the
  policies currently in force, for a customer in any region, using only
  the published policy corpus. The agent reads the answer to the customer.
as_of: 2026-08-01

dimensions:
  answerability:
    test: >
      A reader who sees ONLY this passage can state the answer without
      guessing. Mentioning the topic is not answering it.
  authority:
    test: >
      Source is under policies/ or legal/. Marketing pages, help-desk
      macros and Slack exports fail this test even when correct.
  recency:
    test: >
      Passage is from the version in force on `as_of`. A document with a
      superseded_by field set fails, regardless of content.
  coverage:
    test: >
      Every clause of the query is addressed. If the query names a region
      or a plan, the passage addresses that region or plan.
  entity:
    test: >
      Any named account, product or plan in the query matches the one in
      the passage exactly, not by family resemblance.

labels:
  relevant: passes every dimension
  partial: passes answerability, authority, recency, entity; fails coverage only
  irrelevant: fails any of answerability, authority, recency, entity

required_fields:
  - label
  - decided_by     # which dimension decided it; required even for `relevant`
  - doc_version    # the document revision the annotator actually read

The three label rules are worth checking for holes before trusting them, because a rubric with a hole in it grows the middle value it was written to avoid. Take any pair and write down pass or fail against each of the five dimensions. If all five pass, the first rule fires and the label is relevant. If coverage is the only failure, the second rule fires and the label is partial. If any of the other four fails, the third rule fires and the label is irrelevant. Every combination of passes and failures across the five dimensions lands in exactly one of those three cases. No combination matches two rules, because the partial rule requires the other four to pass while the irrelevant rule requires one of them to fail. None escapes all three, because a pair either passes everything, or fails coverage alone, or fails something outside coverage. That closure is what lets the scale be short without being lossy, and it converts a stuck annotator into a bug report: a pair that fits nowhere is evidence of a missing dimension rather than a case for the middle of a scale.

Two design decisions in there do more work than they look like they do. Failing recency, authority or entity sends a pair straight to irrelevant with no partial credit, because those three failure modes produce answers that are wrong rather than incomplete, and the cost of a wrong answer to a customer is not on the same scale as the cost of a thin one. And decided_by is mandatory on every label, including the positive ones, which turns each label into a claim you can check rather than a number you have to trust.

Requiring decided_by on the positive labels as well looks like paperwork until somebody tries to audit one. A negative label points at a single test. A row reading irrelevant with decided_by: recency can be re-checked by one person opening one document and looking at one field, and the version is either in force or it is not. A positive label makes no such offer, because it asserts that all five tests passed and re-deriving it from scratch means applying all five. Recording the dimension that came closest to deciding against the pair collapses that work back down to one test. It also aggregates into something the rubric author needs: if the positive labels pile up on one dimension, that dimension is the one doing the discriminating for this task, and a rubric whose positives are decided almost entirely by a dimension the task does not really care about has been aimed at the wrong thing.

The three-value scale is also deliberate. Five-point relevance scales are standard in academic evaluation because ranking metrics can use graded judgments, and they are a poor fit for the first labelled set a product team builds, because the middle values absorb every case the rubric failed to decide. A pair marked 3-out-of-5 tells you nothing about which dimension was in doubt, and once a fifth of your set sits at 3 the aggregate scores move for reasons you cannot trace. Three values with an explicit rule for each, plus a decided_by field, gives up some resolution and gains the ability to answer "why is this pair labelled this way" in one lookup.

Which pairs to put in front of the annotator is a separate decision and a consequential one. Labelling random query-document pairs from a corpus of twelve thousand documents wastes almost all the effort, because the overwhelming majority of pairs are trivially irrelevant and teach you nothing. The standard approach is pooling: run every candidate ranker you are considering, take the top five from each, deduplicate, and label the union. For a hundred queries and two rankers that comes to somewhere between six and seven hundred distinct pairs rather than 1.2 million, and it concentrates the labelling on exactly the region where the rankers disagree with each other.

Both of those figures repay being derived, because the distance between them is the entire argument for pooling. The exhaustive alternative pairs every query with every document, so a hundred queries over twelve thousand documents multiplies out to 1.2 million pairs. Which two things are being multiplied is the point: the corpus size is one of them, so every document ingested next quarter makes the labelled set more expensive to build. Pooling multiplies a different pair of things. A hundred queries, taken five deep, from each of two rankers is one thousand pairs before deduplication, and the corpus size appears nowhere in that product. Deduplication then takes the thousand down to between six and seven hundred, and that shrinkage records a fact about the rankers rather than about the labelling: ten slots per query collapsing to six or seven means the two rankers are returning three or four of the same documents inside their top five. The overlap is the discount, and what survives it is exactly the region where the two rankers disagree — which is the only region in which a labelled set can tell them apart.

The cost is a known bias — documents no ranker surfaces never get labelled, so your measured recall, the share of queries whose evidence the ranker actually found and made precise in chapter 4 on ranking families and metrics, is an optimistic estimate against the pool rather than against the corpus. That bias is worth walking through rather than asserting, because the direction of the error is the part you can use. A document that no candidate ranker placed in its top five is never shown to an annotator, so it finishes the exercise with no label. Scoring has nothing available to do with an unlabelled document except treat it as not relevant. A genuinely relevant document that every ranker missed is therefore counted, at scoring time, as a document correctly left out — the single error the measurement cannot see is the error the measurement exists to detect. The consequence is one-sided rather than merely uncertain. A pooled recall that already looks bad is unambiguously bad news, while a pooled recall that looks good has been tested only against documents that at least one ranker already liked.

Labelling then produces rows like these, and they are worth reading closely because the second and third pairs are the interesting ones.

{"query": "what is our refund window?", "doc": "policies/refunds-current.md#window", "label": "relevant", "decided_by": "answerability", "doc_version": "2023-03-14"}
{"query": "what is our refund window?", "doc": "policies/refunds-2021-q3.md#window", "label": "irrelevant", "decided_by": "recency", "doc_version": "2021-08-02"}
{"query": "what is our refund window?", "doc": "help/macros/refund-reply.md", "label": "irrelevant", "decided_by": "authority", "doc_version": "2024-11-30"}
{"query": "what is our refund window in Brazil?", "doc": "policies/refunds-current.md#window", "label": "partial", "decided_by": "coverage", "doc_version": "2023-03-14"}

The third row is the one that surprises people. help/macros/refund-reply.md states fourteen days, correctly, in a sentence an agent could paste into a chat. It is labelled irrelevant because it is a copy, and a copy has no mechanism that keeps it in step with the policy it copies. Retrieving it produces the right answer today and an unpredictable answer after the next policy change, which makes it a source of future incidents that happens to be correct at labelling time.

The same document, two tasks, two labels#

Suppose a legal team wants an assistant over the same corpus for contract disputes: given a customer and a signature date, what terms applied to them when they signed. A dispute about a purchase made in October 2021 turns on the thirty-day window, and policies/refunds-current.md is actively misleading for it.

Under a rubric for that task, as_of is not a fixed date but a value supplied by the query, the recency test inverts, and authority narrows to documents with an archived effective-date range. Run the two rubrics over the same two documents and the labels invert cleanly.

It is worth being precise about what moved, because the natural reading is that the second rubric disagrees with the first, and it does not. The recency test is worded identically in both — is this the version in force at the time the question concerns — and neither the query string nor either document changed. What changed is where as_of comes from. The support rubric fixes it in the rubric file, so the version in force is the current one and the 2021 page fails. The dispute rubric takes it from the query, so for a purchase made in October 2021 the version in force is the 2021 one and the current page fails. One field changed owner, and every label that field decided turned over with it.

The third row is the control, and it is why the table has three rows rather than two. help/macros/refund-reply.md fails authority in both columns and stays irrelevant in both. Were the second rubric simply the first one negated, that row would have flipped as well. It does not flip, which pins the inversion on the dimensions that read a date and clears the ones that do not — and that is the difference between a rubric that encodes a task and a rubric that encodes an opinion about a corpus.

Neither column is a mislabelling of the other. A relevance label is not a property of a document, and it is not a property of a query-document pair either; it is a property of the triple (task, query, document), and dropping the first element is what makes labelled sets stop transferring between products and start disagreeing with each other for reasons nobody can articulate. This is also why a labelled set carries an expiry: the support rubric's as_of field means the labels are true as of a date, and a policy change in September invalidates every recency-decided label taken in August. Storing doc_version on each row is what lets you find and re-check exactly those rows instead of relabelling the whole set.

Ranking, measurement and objectivity#

It does not rank. The rubric says whether a passage belongs in the answer, and says nothing about whether the current policy page should sit above or below the regional exceptions appendix when both are relevant. Ordering within the relevant set is a separate question with separate metrics, and conflating the two produces five-point scales whose middle values mean "I could not decide".

It does not measure the retriever either. A rubric plus labels is a definition of the target; the score comes later, from running a ranker against those labels. Teams that write a careful rubric and then evaluate by reading a handful of outputs have built the expensive half of a measurement and thrown away the cheap half.

And it does not make labelling objective. Writing down that recency decides a label does not guarantee two people apply it the same way, or that either of them applies it at all rather than reaching for it after the fact to justify a gut call. What decided_by buys against that is narrow and worth naming exactly: it does not prevent the rationalisation, it records it. A reviewer sampling the set can put one row in front of the annotator who wrote it and ask them to show the superseded document — and the answer to that request separates a test that was applied from a field that was filled in.

A rubric is a small artefact and it is the one every later number in this topic depends on. It names the task in a sentence, gives each dimension the task carries a test an annotator can apply, defines a scale short enough to force a decision, and records which dimension decided each label. What it buys is not agreement — that comes next, and it comes with an argument — but the ability to say, for any row in the set, exactly why that row says what it says.

Which is where the rubric above meets its first test. It went to two annotators, and on a set of two hundred pairs from an API documentation corpus they disagreed on thirty-one of them, concentrated almost entirely in one class of question.