Corpus quality filtering for AI use

Read these first

• Embedding-based deduplication and near-duplicate detection

What you will know

• Strip boilerplate that pollutes retrieval

  Navigation, repeated legal text, and template fragments create false matches and can crowd out content-specific passages.

• Score documents for quality and exclude the bottom

  A scoring rule should expose its thresholds and failure cases rather than silently discarding rare but valuable material.

• Show excluded content improving results by its absence

  An offline retrieval or answer benchmark should demonstrate fewer irrelevant hits without masking regressions caused by lost coverage.

• Compare retrieval quality against ordinary data-cleaning metrics on the same corpus

  A document may be valid, complete, and schema-conformant while still being useless or actively harmful to retrieval.

• Decide what never belongs in the index

  Secrets, duplicate derivatives, navigation-only pages, and unsupported content can create security, cost, or trust failures if indexed.

Chapters

1. 1

   A footer on every page becomes the largest cluster in the vector space

   Two hundred thousand wiki pages each ending in the same 300-word confidentiality notice, chunked at 512 tokens, yield a chunk per page that is mostly notice plus a fragment of real content. A few percent of corpus tokens becomes the largest tight cluster in the space, competing for top-k slots on almost any formally phrased query, so the retrieval damage far exceeds the storage cost. Two stripping approaches are compared, and near-duplicate detection is tested as the substitute it is assumed to be.

   Read chapter
2. 2

   A document can be schema-conformant, complete and worthless to retrieval

   The data platform certifies 99.4% schema conformance, 98% field completeness, zero parse failures and duplicates removed by exact hash; retrieval is still poor, and a colleague concludes the embedding model is the remaining variable. Every one of those checks passes on a navigation stub, on a PDF re-export differing from its source by a header, and on a template with all fields filled and nothing said. This chapter names the single measurement that settles corpus against model.

   Read chapter
3. 3

   The same content in three formats takes three of your ten slots

   Exact-hash deduplication happily certifies a corpus where one page also exists as a PDF export and as a re-published knowledge-base copy, none of them byte-identical. Find that family with shingling or MinHash over normalised text, or by clustering embedding neighbourhoods, then choose a canonical copy — prefer the source format, prefer the version that kept its structure — and exclude the derivatives outright rather than down-ranking them. Leave all three in and one document's worth of information occupies three of the top ten slots on every query it matches.

   Read chapter
4. 4

   A quality score punishes structure, and reference pages pay for it

   Mark documents down for short length, low unique-token ratio and absent headings and the hardest-hit class is error-code reference entries: terse, repetitive, headingless, and exactly what support queries are looking for. A score measures form, not value. Before any threshold ships, read a sample of what it dropped grouped by document class, hunting for types penalised structurally rather than for anything a reader would actually miss, and publish the thresholds and their known casualties.

   Read chapter
5. 5

   A benchmark with no queries for a class cannot see it disappear

   Dropping template-heavy pages lifts nDCG@10 six points and halves the irrelevant-hit rate; six weeks later escalations on one product line have doubled, because that line's documentation is template-heavy and no benchmark query touches it. Precision gains get measured and coverage losses do not. You will learn how to confirm the mechanism inside a day, what the benchmark was actually missing, and why moving the threshold from 15% to 5% treats a coverage failure as a dial setting.

   Read chapter
6. 6

   Ranking cannot un-embed a secret, so some content never enters the index

   A product manager wants the entire wiki indexed — engineering runbooks, draft pages, an archive of exported Slack channels — arguing that ranking keeps irrelevant content out of answers. Once a live credential sits in a chunk it is recoverable by anything that can query the store, and a retrieval-time filter guards one path while the eval harness, a debug endpoint and any second consumer reach the index directly. This chapter draws the refuse-outright line and orders the remediation when it is crossed: rotate the credential first, since that is the only genuinely reversible step, then purge the vectors and the stored chunk text, then replicas, caches and every backup taken since, then derived artefacts — and record what has already reached customers or a provider and cannot be recalled.

   Read chapter