An index only forgets what ingest tells it to forget

A search index over a company's documents is a store of passages — chunks, each a few hundred words cut out of a document — held under identifiers, with a vector and a payload beside each one. Keeping such a store in step with its source is almost always written as an upsert: hand the store a record carrying an identifier, and it writes the record, creating it if that identifier is new and replacing it if the identifier is already there. Upsert covers the two things a crawler mostly does, which is add a page and update a page, and it is the reason a working sync job can be twenty lines long. It offers no third verb.

That missing third verb is not an inconvenience to be worked around. It is a structural difference between updating and deleting, and it surfaces as a class of answer that is fluent, well formatted, confidently cited and describing something that stopped being true months ago. No property of the stored record gives it away, because a chunk left behind by a deletion at the source is indistinguishable from a current one — same identifier shape, same vector arithmetic, same payload, same score.

A sync job written the obvious way looks like this, and the first line deserves more attention than the rest of the function.

def sync(space: str) -> None:
    for page in confluence.list_pages(space=space):      # what still exists
        if page.version == last_seen_version.get(page.id):
            continue
        chunks = chunk(parse(page))
        index.upsert([
            {
                "id": f"{page.id}:{i}",
                "vector": embed(c.text),
                "payload": {"doc_id": page.id, "version": page.version, **c.meta},
            }
            for i, c in enumerate(chunks)
        ])
        last_seen_version[page.id] = page.version

The loop iterates over what the source currently has. A deleted page is not an item in that iteration with a deleted flag on it; it is the absence of an item, and absence produces no loop body, no upsert, and no signal of any kind. Upsert offers two verbs, add and overwrite, and both of them require an identifier that the current run produced. There is no third verb, and no amount of care inside the loop invents one, because the information that a deleted page is gone does not exist anywhere in this program's control flow.

It is worth being exact about why that is a difference in kind rather than a harder version of the same problem, because the exactness is what tells you which repairs are even candidates. An update is triggered by evidence, and the evidence arrives as an object: the page appears in the listing, it carries a version, and the version differs from the one recorded. Both facts are properties of something the loop is holding in its hand. A deletion has no such object. Its evidence is a non-event — something that would have arrived and did not — and a data structure that represents things cannot represent the absence of a thing it never received. Any program that wants to act on a non-event has to construct one, and there is exactly one way to construct it: hold two sets and subtract. That is why the fix is not a more careful loop but a second set, and why a run that never enumerates what the index already holds has nothing to subtract from. The deletion it needs to perform is not undiscovered; it is unrepresentable.

Enumeration says what exists; only a diff says what stopped existing#

The fix that comes to mind first is to subscribe to deletion events. Confluence emits a page_removed webhook, S3 emits ObjectRemoved:*, most sources emit something, and wiring those into a delete-by-doc_id handler does work for the cases it covers. It is worth being clear about how many cases it does not cover, because the appeal of event-driven deletion is that it looks complete and is not.

Events are lost when the consumer is down and the source does not replay, which for most SaaS webhooks means a fifteen-minute deploy costs you every deletion in that window, permanently. Events also only describe deletion, and deletion is one of several ways a document leaves your corpus. A page moved to a space the crawler does not index emits a move event, not a removal. A page whose restrictions changed so that the crawler's service account can no longer read it emits a permissions event that a deletion handler will not be listening for, and from the crawler's point of view the page has vanished with no notification whatsoever. A space archived in bulk may emit one event for the space and none for its four hundred pages. A source-side hard delete performed by a database migration emits nothing at all. In each of these the index keeps serving content whose right to be there has expired.

Reconciliation handles all of them with one mechanism, because it does not ask the source what changed. Each run publishes the complete set of identifiers that are live right now; the job reads back the set of identifiers the index currently holds for that source; the difference in one direction is what to write and the difference in the other is what to delete.

def sync(space: str) -> None:
    live_chunk_ids: set[str] = set()

    for page in confluence.list_pages(space=space):
        chunks = chunk(parse(page))
        ids = [f"{page.id}:{i}" for i in range(len(chunks))]
        live_chunk_ids.update(ids)

        if page.version != last_seen_version.get(page.id):
            index.upsert([...])                       # unchanged from before
            last_seen_version[page.id] = page.version

    indexed_ids = set(index.scroll_ids(filter={"space": space}))
    stale = indexed_ids - live_chunk_ids

    if len(live_chunk_ids) < 0.8 * len(indexed_ids):  # crawl looks truncated
        raise CrawlIncomplete(live=len(live_chunk_ids), indexed=len(indexed_ids))

    index.delete(ids=sorted(stale))
    metrics.gauge("ingest.stale_deleted", len(stale))

Two things in that function are doing more work than their line count suggests. The live_chunk_ids set is populated for every page the crawler sees, including the ones it skips re-embedding because the version has not changed, which means the live set is a statement about existence rather than about work performed. Keeping those two ideas separate is the whole trick: the expensive path is gated on version, and the correctness path is not gated on anything.

The guard against a truncated crawl is the second. A reconciliation job holds a delete-everything button, and it will press it the first night the source API returns an empty page of results because a token expired or a rate limit returned a 429 that the client swallowed. On a corpus of 4,200 Confluence pages producing roughly 51,000 chunks, a normal night's diff is between 20 and 200 stale identifiers; anything in the thousands is a crawl failure wearing a deletion's clothes. The threshold above is crude and it is enough — the property it enforces is that a partial crawl fails loudly instead of quietly emptying the index, and the cost of getting it wrong in the conservative direction is that stale chunks survive one extra day.

That guard is there because the whole method rests on three separate things being true, and they multiply rather than add. The first is that the live set is complete: a crawl that silently returns one page of results short reports documents as absent when they are merely unlisted. The second is that the indexed set is enumerable: a store with no way to scroll identifiers under a filter cannot say what it is holding, so there is nothing to subtract from. The third is that the identifier scheme is derivable, so that one document yields the same string on both sides of the subtraction. Any one of the three at zero takes the whole thing to zero, and the failure is not a degraded answer but a confidently wrong one — subtracting a complete set from an incomplete one does not return fewer deletions, it returns deletions of live content. That is the specific direction in which this method is dangerous, and it is why the guard checks the shape of the live set before the delete rather than after it.

The same bug at chunk scale, where nobody looks for it#

Reconciliation at document level still leaves a smaller version of the same hole, and it is the one that produced the stale trial-length answer in the previous chapter. Version 9 of a page splits into 12 chunks and version 10 splits into 9, because someone deleted two paragraphs. The upsert writes 4417:0 through 4417:8 and the index still holds 4417:9, 4417:10 and 4417:11 from the previous revision. Those three orphans contain real prose that a human wrote, they embed to sensible vectors, and they will be retrieved for exactly the questions the deleted paragraphs were about. This is worse than a deleted page in one respect: the parent document still exists, so the citation resolves, the URL opens, and the page it opens does not contain the sentence the assistant quoted.

That example closes exactly, and the closure is the part worth carrying away. Version 9 produced twelve identifiers, numbered from zero. Version 10 produced nine, and every one of those nine had also been produced by version 9, so all nine writes were overwrites and not one was an insert. Nine overwritten plus three left behind accounts for all twelve, with no residual. What the closure licenses is arithmetic in place of inspection: the number of orphans a shrinking document leaves is exactly its old chunk count minus its new one, and that is computable before a single identifier is scrolled out of the store. It also gives a check with teeth in the other direction. If a per-document diff comes back with a number that is not the difference of the two counts, the discrepancy is not orphans at all — it is a write that failed, or a concurrent edit, or an identifier scheme that is producing different strings on the two sides — and each of those three wants a different repair.

The identity runs in one direction only, and that asymmetry is why the bug feels intermittent. When the new chunk count exceeds the old one the difference is negative and there are no orphans whatever: every old identifier is overwritten and the surplus identifiers are inserts. Documents that grow are safe under upsert alone. Documents that shrink are not. So the failure appears only on edits that remove text, which is precisely the kind of edit made when a policy is withdrawn, an exception is folded into a general rule, or a caveat stops applying — the edits whose stale remains are most worth quoting and most damaging to quote.

The two figures given for that Confluence space agree with the worked example in a way worth checking rather than assuming. Four thousand two hundred pages producing roughly fifty-one thousand chunks works out at about a dozen chunks a page, and the page in the example splits into twelve. Those are independent measurements — one an aggregate over a whole space, the other a single document — and their landing in the same place is weak evidence that neither figure is a typo, and a reason to be suspicious of a per-document chunk count that comes back an order of magnitude away from it. Every orphan counted so far arrived because somebody edited a document, and edits are not the only way they arrive.

The same thing happens without any edit at all when the chunker changes. Switching from a 512-token fixed window to a 900-token window with 15% overlap re-splits every document in the corpus, so every document's chunk count changes, and a run that only upserts leaves orphans behind on tens of thousands of documents at once. So does swapping the embedding model, which does not change the identifiers but does change what the vectors mean — and a collection holding two models' vectors ranks on a similarity that has no interpretation, as the previous chapter's contract table noted.

There are two workable shapes for chunk-level correctness. The first is delete-then-insert per document: remove everything matching doc_id == X, then write the new chunk set. It is simple and it costs write amplification on documents that barely changed, plus a window of a few hundred milliseconds in which the document is absent from the index entirely, which matters if the job runs during business hours. The second is a per-document reconcile: upsert the new chunks, then issue one filtered delete for doc_id == X AND version != 10, which requires that the version travels into the payload and that the store supports deletion by filter. The second is what most production jobs converge on, and it degrades gracefully — if the delete fails, the fresh chunks are already in place and the orphans get swept by the next nightly full reconcile.

That table is more useful read as gaps than as a list of situations. The second column is what a run leaves in the index that should not be there; the fourth is what those same records say when a user asks them a question, so the two columns are one quantity seen at ingest time and again at query time. Reading only the second column, two rows say "Nothing", and those two rows are exactly the cases where the set of identifiers does not shrink — a new page, and an edit that produces the same number of chunks. Every other row is a row where the identifier set contracts, and the gap between the old set and the new one is what the fourth column is describing. The rule generalises past the eight rows: upsert is a complete implementation for any change that grows or preserves the identifier set, and no implementation at all for any change that shrinks it.

Between hard deletion and doing nothing there is a middle option that is worth knowing about and worth defaulting away from. Instead of removing the stale identifiers, mark them with a deleted_at timestamp and add deleted_at IS NULL to every retrieval filter. The appeal is recoverability: a reconciliation bug that deletes 12,000 chunks by mistake is a one-line rollback rather than a full re-crawl, and the deleted set is available for forensics. The cost is that correctness now depends on every query path remembering the predicate, and query paths multiply — the main search, the evaluation harness, an admin debugging tool, whatever the next team builds against the same collection. Each one that forgets the filter is a route by which deleted content answers questions again, which is the original bug reintroduced with more machinery. Soft deletion is defensible when the store makes filtered search cheap and there is exactly one retrieval function in the codebase, and it becomes a liability at the point where there are three.

The shape of that trade is a product rather than a sum, which is why it goes bad faster than it looks like it should. Soft deletion is correct only if every retrieval path applies the predicate, so its correctness is one factor per path, each either one or zero, all multiplied together. A single path at zero takes the whole product to zero, and the result is not a partially correct system: one admin tool scrolling the collection without deleted_at IS NULL restores the original bug in full, for every soft-deleted chunk simultaneously. Each new query path multiplies in another factor that can be zero. That is why the arrangement degrades with the number of teams touching the collection rather than with the number of documents in it, and why "there is only one retrieval function" is the condition to check rather than the size of the index.

The last row is the one case reconciliation does not solve, because the identifiers are unchanged and the diff is therefore empty. Model swaps are a rebuild: write into a new collection, verify it, then repoint the reader. Anything that changes the meaning of the vector rather than the set of identifiers has to be handled by versioning the collection, not by diffing its contents.

Freshness, source truth, and the cost of scrolling#

Reconciliation is not a freshness mechanism. A nightly full reconcile means a document deleted at 09:15 keeps answering until 02:00 the following morning, which is roughly seventeen hours of a policy that no longer applies being quoted as current. If the corpus needs minute-level freshness, event handling still has to exist; reconciliation then becomes the floor underneath it that catches whatever the events missed, and the two are complementary rather than alternatives. Sources that support cheap change feeds — an S3 inventory report, a database updated_at index, a Confluence CQL query on lastmodified — let the fast path run every few minutes and the full diff run nightly, and the nightly diff is the only thing that ever needs to enumerate everything.

It is also not a correctness guarantee about content. A page that still exists and is wrong reconciles perfectly, every night, forever. Reconciliation only enforces that the set of things in the index equals the set of things at the source; whether the source is right is somebody else's problem, and pretending otherwise is how teams end up believing an index is authoritative because a job succeeded. The property being enforced is an equality between two sets, and an equality between two sets says nothing whatever about the contents of either.

The cost is real but small at this scale. Scrolling 51,000 identifiers out of a vector store with a payload filter takes about 9 seconds and a few megabytes of memory; the set difference is instant; the deletes are a single batched call. On corpora in the tens of millions of chunks the scroll becomes the expensive part, and the usual answer is to reconcile per partition — one space, one bucket prefix, one tenant per run — so that no single job ever has to hold the whole corpus in memory, and a failure in one partition cannot delete another. Partitioning also narrows the blast radius of the guard above, since a truncated crawl of one space can only ever propose deletions inside that space.

An index is not a mirror of its source. It is a record of statements some run once made, and it holds every one of them until a later run explicitly withdraws it. Upsert can only make statements; there is no sentence in it that means "this identifier should no longer exist", and the reason is arithmetic rather than negligence — the evidence for a deletion is a thing that failed to appear, and a loop over what did appear has no place to put it. Reconciliation supplies the missing sentence by building the non-event out of two sets that do exist, everything the source lists now and everything the index holds now, and subtracting one from the other. The rest of this chapter is detail about performing that subtraction safely: gate the expensive path on version while leaving the correctness path ungated, refuse to act on a crawl that looks truncated, and run the same diff one level further down at chunk granularity, where the sets are larger by whatever the average chunk count per document happens to be, and nobody is watching.

There is one more absence that produces no event and no diff. A document can stay in the index, unchanged, correctly versioned, fully reconciled, and still be something a particular user is not allowed to read, because the permission changed rather than the document. Reconciliation would catch that eventually, on whatever schedule it runs, and eventually is not a defensible answer when the failure mode is disclosure rather than staleness. Access control needs a contract that holds at query time, per user, which is the subject of the next chapter.