Ingestion pipelines

Read these first

• Document parsing
• Index maintenance

What you will know

• Sync a source incrementally rather than reprocessing everything

  A cursor or watermark advanced before durable processing can silently skip changes after a crash or connector restart.

• Make ingestion idempotent under retries

  Stable document identities and deterministic writes prevent duplicate chunks and conflicting versions when delivery is repeated.

• Handle a document deleted at the source

  Without tombstones or deletion reconciliation, stale content remains retrievable even though the source of truth no longer contains it.

• Build a dead-letter path for documents that fail to parse

  Quarantining malformed inputs preserves throughput while retaining enough payload and error context for repair and replay.

• Backfill without taking the index offline

  Separate generations, controlled cutovers, or equivalent versioning prevent mixed corpus state and inconsistent retrieval during large rewrites.

Chapters

1. 1

   Committing the cursor before the upsert opens an invisible gap

   A connector that writes the new cursor to Redis as soon as a page arrives, then parses, embeds and upserts, loses the whole in-flight page to an OOM kill, and the next poll asks only for changes after a timestamp that has already moved past them. Nothing records that those documents were ever in flight and the source will never offer them again. Covers where the commit belongs and the delivery semantics each placement gives you.

   Read chapter
2. 2

   Deterministic chunk keys turn a redelivery into an overwrite

   At-least-once delivery plus a transient 500 from the embedding provider leaves the same paragraph in the index three times. The minimal fix is not deduplication after the fact but a primary key derived from document identity and chunk index, so a repeated delivery writes exactly the rows it wrote before. This is also the answer to the claim that committing late is the worse design: duplicates are countable and removable, a gap is neither.

   Read chapter
3. 3

   An idempotent write can still overwrite fresh content with stale

   Deterministic keys plus upsert make a redelivery harmless and a late one dangerous: revision N-1 arriving after revision N is indexed overwrites current content perfectly correctly, which is why the write needs a version guard — compare the source etag or an ingest sequence number and refuse to write backwards. The second case is quieter. Re-ingest a document that now yields five chunks where it used to yield nine, upsert on (doc_id, chunk_index), and four retrievable ghosts stay behind; a document write has to be delete-then-write within the document, or a generation tag followed by a sweep. These two guards are also what make any later replay of an old payload safe to run at all.

   Read chapter
4. 4

   An updated_after query can never return a deleted record

   Legal asks why a wiki page deleted six weeks ago is still being retrieved and cited. No polling frequency closes this, because the incremental path fetches only documents the source reports as changed and a deleted page is never in that set, so a 404-on-fetch check never runs on the documents it was written for. Covers deletion feeds where the source offers them and full-ID reconciliation over 4M pages where it does not.

   Read chapter
5. 5

   A dead-letter record is only useful if it can be replayed exactly

   Three percent of PDFs fail to parse and today the consumer retries three times, logs the exception and drops the message. A quarantine path earns its place only if what it stores reconstructs the attempt: source ID, source version or etag, the cursor position the document arrived at, the connector and parser versions in effect, plus the error class and stack trace. Without the versions, a later replay cannot distinguish a fixed parser from an edited file.

   Read chapter
6. 6

   412,000 dead letters beside a green dashboard describe an ownership gap

   A queue whose oldest message is eleven months old, next to a parse-failure rate that has read 0.4% and green all year with no alert ever firing, is telling you the failure path has no owner and no threshold. This chapter reads that state properly, weighs draining against replaying messages captured under parser and connector versions that no longer exist, and specifies the drain that is actually safe — re-fetch each source document by ID instead of replaying the stored payload, and land it behind the version guard — plus the alerting and retention that stop the queue becoming a landfill again.

   Read chapter
7. 7

   Backfill into a new generation and flip the alias, never in place

   Re-chunking 8M documents in place, one atomic transaction per document, prevents torn reads inside a document and does nothing about the corpus running two chunking regimes at once for days, which makes every score taken during the window incomparable. The alternative is a separate generation the backfill writes into while the current one serves, an evaluation run against it, an atomic alias flip and a retained rollback target. Covers what the incremental connector does meanwhile.

   Read chapter