Retrieve a document with its own text to test the encoder alone

Self-retrieval is a run in which each query is a document that is already in the index, and the correct answer is that same document. It exists because the diagnostics in the previous chapters all measure a relationship between two different things — a user's phrasing against a document's phrasing, one formatting arm against another — and when those come back ambiguous you need a measurement with only one thing in it. Taking a chunk's own text as the query removes the query-document relationship from the experiment entirely, because the two sides now contain identical content.

What is left in the frame is worth listing, since it is what the run can speak to: the encoder's ability to give distinct documents distinct vectors, the index's ability to return a row when that row is the best match, the approximate-search parameters, any filter that silently excludes documents, and whether the stored text is what the source store says it is. Everything else — formatting, vocabulary mismatch, query phrasing, the quality of the labelled set — is held outside. A control is defined by what it excludes, and the value of this one is that the exclusion list is short and nameable.

A query that is guaranteed to have an answer#

Take a sample of chunks, use each chunk's own text as the query, and check whether that chunk comes back at rank 1. The relevance judgement is free and exact — the correct answer is the chunk you started from — so you can run this over thousands of chunks without labelling anything.

The reason the judgement is free is worth naming rather than assuming, because it is the property that makes the whole instrument cheap. Every other evaluation in this topic needs a human to decide whether a document answers a question, and that decision is expensive, contestable and slow. Here the decision is definitional: the chunk whose text you submitted is the chunk that best matches that text, by construction, so no judgement is being made at all. The cost of the run is the cost of the searches.

import random

def self_retrieval(index, chunks, n: int = 2000, k: int = 10) -> dict[str, float]:
    sample = random.sample(chunks, n)
    at1 = at10 = 0
    for c in sample:
        ranked = index.search(embed_query(c.text), k=k, ef_search=128)
        if ranked and ranked[0] == c.id:
            at1 += 1
        if c.id in ranked:
            at10 += 1
    return {"self@1": at1 / n, "self@10": at10 / n}

print(self_retrieval(INDEX_API_REF, CHUNKS_API_REF))
# {'self@1': 0.991, 'self@10': 0.998}

Read 0.991 against the size of the haystack rather than as a percentage on its own. It says that 99.1% of the time, when a chunk's own text is submitted against a corpus of 380,000 chunks, that chunk is its own nearest neighbour. Winning rank 1 out of 380,000 candidates requires the encoder to have placed that chunk's vector strictly closer to the query vector than every other chunk in the corpus without exception, which is a much stronger statement than a percentage suggests. The vectors are distinct, the index returns the row when the row is a match, the approximate nearest neighbour (ANN) parameters — which trade search accuracy for speed by visiting only part of the index — are not throwing away neighbours, no filter is silently excluding documents, and the stored text is what the source store says it is.

The blind spot that comes with the control#

The reason this run is a clean control is the same reason it is limited, and this is the part most likely to be misread. Identical content on both sides makes it insensitive to the formatting defect entirely.

Follow the strings. If your pipeline puts query: on both sides, then the query built from a chunk's own text is query: To rotate a credential…, and the string that was indexed for that chunk was also query: To rotate a credential…. They are character-for-character identical, so their vectors are identical, so the cosine between them is 1.0. Self-retrieval scores near-perfectly. Now do the same for a correctly formatted pipeline: the query is query: To rotate a credential… and the indexed string was passage: To rotate a credential…, which are different strings producing slightly different vectors — and self-retrieval still scores near-perfectly, because a small perturbation is not enough to lose rank 1 against a field of unrelated documents sitting much further away.

Both formattings therefore produce the same reading. A measurement whose value does not move when the thing it is supposed to detect changes cannot detect that thing, and the belief that a healthy self-retrieval run means the prefixes are fine is the exact shape of error this topic keeps returning to. It measures the encoder and the index while holding the cross-side relationship out of the frame, which is what a control is for and exactly why it can never be the only thing you run.

The table earns its keep by being a two-dimensional reading rather than a threshold on either number alone. Neither column identifies anything by itself: a low real-query recall is consistent with four different causes, and a high self-retrieval score is consistent with three. The pair narrows it, and the second row is the one that sends you back to chapter 3's trace.

How you draw the 2,000 chunks affects what the run can tell you. A uniform sample over the whole index gives a corpus-level health figure and will average away a problem confined to one part of it, in exactly the way the aggregate metrics discussed later in this topic do. Stratifying by source, by ingestion date and by chunk length costs one extra argument and turns a single number into a small table where an anomaly has somewhere to show up: chunks under 40 tokens self-retrieve worse than long ones in most corpora, because there is less content to distinguish them, and a source whose self@1 sits fifteen points below the others is worth a look regardless of what the total says.

A harder version, when the easy one is too easy#

The exact-text run has a ceiling built into it, because retrieving a document with its own text is close to a lookup. Nothing about matching a string to itself requires the model to have understood anything; two identical inputs give identical vectors under any encoder whatsoever, and the only work being done is telling that vector apart from other documents' vectors. To get a ceiling that requires actual semantic work, hold out part of each chunk and query with the rest. Take the chunk's first sentence, or its heading, or a randomly chosen sentence from the middle, index the chunk without that sentence, and search for it.

def held_out_self_retrieval(chunks, k: int = 10) -> float:
    probes, bodies = [], []
    for c in chunks:
        sentences = split_sentences(c.text)
        if len(sentences) < 3:
            continue
        held = sentences[len(sentences) // 2]
        probes.append((c.id, held))
        bodies.append(Chunk(c.id, " ".join(s for s in sentences if s is not held)))
    index = build_index([embed_passage(b.text) for b in bodies], [b.id for b in bodies])
    hits = sum(cid in index.search(embed_query(q), k=k) for cid, q in probes)
    return hits / len(probes)

# API reference corpus: 0.88
# product documentation corpus: 0.91

An 0.88 here says the encoder can find the right document from one sentence of its own content, which sets a defensible ceiling for the content itself. It is a genuinely different measurement from the exact-text run, because the probe sentence and the indexed body now share a subject and not a string, so the model has to place them near each other on the strength of meaning. Note also that the two sides here carry different prefixes, so unlike the exact-text run this variant is not immune to a formatting defect — it is a harder control and a slightly less clean one.

Reusing the model to find near-duplicates#

Document-to-document comparison, which is what the self-retrieval run is close to, is also the setup for deduplication — and the two uses have different obligations. A diagnostic run only has to be internally consistent. A deduplication feature makes claims about pairs of documents, and an asymmetric retrieval model was never optimised to make them.

Start with the role. Your stored vectors already carry the passage role, and there is no way to re-role a vector after the fact, so comparing documents against the index you already have means the passage role on both sides. That is also the stance that matches what you are asking: both objects are documents, and asking whether two documents are near-duplicates is a question about them as documents. Some model cards nominate a specific role for symmetric use; adopting one means embedding the whole corpus a second time into a separate index, and that cost is worth paying only if a validation set says the second index is better.

Which brings in the obligation that is easy to skip. The training objective these models optimised was question-to-answer, and passage-to-passage geometry is a by-product of it. Nothing in the batch objective from chapter 1 ever compared two passages to each other for similarity — passages appeared in it only as targets and as negatives for questions — so whatever arrangement they ended up in relative to one another was never rewarded or penalised. Sometimes the by-product is excellent — E5 and BGE both score respectably on semantic-similarity benchmarks — and sometimes it is distorted in ways that only show up on your data.

The public evidence is available and it is worth reading before you assume transfer. MTEB reports retrieval and semantic textual similarity (STS), the task of scoring how alike two sentences are with no query-and-answer relationship between them, as separate task groups precisely because a model can be strong on one and unremarkable on the other, and the ordering of models differs between the two boards. A checkpoint that sits near the top for retrieval may be mid-table for STS, which is a direct statement that its passage-to-passage similarities are less well calibrated than its query-to-passage ones. That is a signal about the family rather than a verdict on your corpus, and the reason it cannot be a verdict is that near-duplicate detection on your data is a narrower and stranger task than STS: STS pairs are sentences with graded human similarity scores, and your duplicates are two versions of the same runbook where one has an extra paragraph about a deprecated flag.

Earning the transfer, concretely, means a small labelled set: 300 pairs you consider duplicates and 300 you do not, drawn from your actual corpus. Then sweep a threshold across them.

def sweep(pairs: list[tuple[str, str, bool]], thresholds: list[float]) -> None:
    scored = [(cosine(embed_passage(a), embed_passage(b)), dup) for a, b, dup in pairs]
    for t in thresholds:
        tp = sum(1 for s, d in scored if s >= t and d)
        fp = sum(1 for s, d in scored if s >= t and not d)
        fn = sum(1 for s, d in scored if s < t and d)
        print(f"t={t:.2f}  precision={tp/(tp+fp):.2f}  recall={tp/(tp+fn):.2f}")

Read that table as the gaps between consecutive rows rather than as five candidate settings. Moving the threshold from 0.90 to 0.94 buys thirty-six points of precision and costs three points of recall, which is the steepest exchange in the table by a wide margin. The next step, to 0.96, buys twenty-two points of precision for eight of recall. The step after that buys seven for seventeen, and the last buys three for twenty-nine. The exchange rate inverts somewhere between 0.94 and 0.97, and that inversion — rather than any single row — is what a threshold decision is actually made against. The flagged column moves with it: four hundredths of threshold take the review queue from 1,840 pairs per ten thousand down to 430, which is the difference between a project and an afternoon.

The distribution behind that table is the part that transfers badly between models. On these normalized passage embeddings, two entirely unrelated documents score around 0.72 rather than anywhere near zero, because the passage role occupies a compact region of the sphere and everything in it is somewhat close to everything else. That compresses the whole usable scale into the span between 0.72 and 1.0, and the sweep above shows the decisions concentrated in the top tenth of it. A threshold of 0.8, which sounds conservative and is a number people carry over from other models out of habit, sits barely above the unrelated-document floor and flags roughly a third of all pairs. The usable range is narrow and sits high, and its location is a property of this model on this corpus that you can only learn by measuring.

Where the instrument stops being a control#

The fourth row of the interpretation table is not a failure. When self@1 drops but self@10 stays near 1.0, what usually happened is that the corpus contains many chunks with nearly the same text, so the "correct" one loses rank 1 to a twin. In the API reference corpus, error-code listings differing by a single numeric code produced exactly this, and it is worth measuring rather than dismissing, since it puts a number on how much redundancy the index carries.

The labelled pairs behind a threshold sweep have a trap of their own. Hand-constructed pairs are the wrong material, because a person asked to write a duplicate will paraphrase evenly across the whole text, while real duplicates in a documentation corpus are usually identical for 80% of their length and differ in one section. Those two kinds of pair sit in completely different places in the score distribution, and a threshold calibrated on the first will be far too low for the second. Draw the pairs from the corpus you actually have.

There is also a scale asymmetry between the diagnostic use and the product use that none of the tables shows. Self-retrieval touches 2,000 chunks and does 2,000 searches. Deduplication is a question about pairs, and the number of pairs among a set is the count of ways to choose two of its members, which is the size times one less than the size, divided by two — for 380,000 chunks that comes to 7.2 × 10¹⁰ of them. Nobody computes that exhaustively; the real implementation retrieves the top 20 neighbours of each chunk from the index and scores only those, which is 380,000 searches and a bounded amount of arithmetic. That shortcut has its own consequence for the threshold sweep, because a pair that never appears in either chunk's top 20 can never be flagged regardless of threshold, and your labelled set should therefore contain a few duplicate pairs that the neighbour search misses so the reported recall reflects the pipeline rather than the scorer.

For a corpus where the duplicates are mostly copied text with small edits, MinHash over 5-grams — hashing every overlapping five-word window of a document and measuring how many of those hashes two documents share — settles it in minutes on a CPU and no embedding model is warranted. For paraphrase-level duplicates the embedding approach earns its place, and reusing the passage-role index you already maintain is the cheap version of it. What you should not do is ship the embedding version without the labelled sweep, because the threshold is the entire product and it cannot be guessed from experience with another model.

A self-retrieval run costs nothing and answers one question completely: whether the encoder and the index can tell your documents apart. When it comes back high and real recall is low, the fault is in the relationship between the two sides, and this topic has already named the two things that live there — formatting, and a genuine gap between the words users type and the words the documents use. When it comes back low, no formatting work will help, because the vectors themselves are not doing their job.

Everything measured so far has assumed one corpus, built by one pipeline, wrong or right in one uniform way. Corpora do not stay that way. They grow a second ingestion path, and a defect that touches only part of the index behaves nothing like the ones in this chapter.