Committing the cursor before the upsert opens an invisible gap

An ingestion pipeline is the set of jobs that move documents out of the systems where people actually write them — a wiki, a shared drive, a ticket tracker, a bucket of PDFs — and into a search index an AI assistant queries when somebody asks it a question. That arrangement, where an application looks up relevant passages and puts them into a model's prompt so it can answer from them, is retrieval-augmented generation, usually shortened to RAG, and what such an assistant can answer is bounded by what its pipeline managed to put in that index. A pipeline of this kind exists because source systems cannot serve that lookup themselves: text has to be pulled out of whatever format it arrived in, split into passages, turned into vectors and written somewhere a query can reach in milliseconds. You meet one from the outside whenever a colleague reports that an assistant does not know about a page published yesterday, or quotes one deleted last month.

Running that work once over a fixed corpus is a script; running it continuously while several thousand people edit those documents every day is a pipeline, and most of the difficulty lives in what happens when a run fails partway through. The per-source half of it is a connector, the component that knows one source's API, its authentication, its pagination and its idea of what counts as a change. It keeps a cursor, a single stored value marking how far through a source's history it has read, which is what lets it ask for recent changes instead of re-reading every document every ninety seconds. Each document it processes ends with an upsert into that index, meaning a write that creates the rows if they are absent and overwrites them if they are already there. One line of code decides where the cursor write sits relative to that upsert, and that line is the difference between a crash costing a repeat and a crash costing a hole nobody can find.

A cursor is a claim about durable state#

The updated_after cursor — a watermark, a checkpoint, a sync token, depending on whose API you are reading — is a single value that stands in for a set. When the connector holds 2026-03-09T09:14:02Z, it is asserting that every change the source made before that instant has been fully processed and is durably visible in the index. Everything downstream depends on that assertion being true, because the source will never again volunteer a document whose updated_at falls below it. There is no reconciliation loop running behind the connector to catch what it dropped, and no error is generated by the absence of a document nobody asked for.

A single timestamp is not a description of progress; it is a compressed statement about an unbounded set of documents, and it is exactly as true as the weakest thing that happened to any member of that set. One document in a page of a hundred that never reached the index makes the whole claim false, and the value in Redis looks identical either way.

What makes the assertion easy to break is that the cursor is cheap to write and the work it stands for is expensive to do. Reading the last item's timestamp out of a page of JSON and putting it in Redis takes under a millisecond; parsing, chunking, embedding and upserting that page takes somewhere between thirty seconds and several minutes depending on document size. Code tends to be written in the order the author thinks about it, and the author thinks about the page first, so the cursor write lands next to the fetch that produced the value. It reads naturally, it passes every test that does not involve killing the process mid-page, and it is wrong in a way that only manifests under conditions — OOM kills, rolling deploys, spot instance reclamation — that are absent from development and routine in production.

Why a skipped document is never offered again#

The claim above has a sharp consequence that is easy to state and easy to under-rate: a document skipped this way is not delayed, it is gone. That is worth deriving one step at a time, because each step is checkable on its own.

An updated_after endpoint is a filter over the source's current state. It takes a timestamp and returns the records whose updated_at is greater than the timestamp supplied. That is the whole contract, and two things follow from it immediately.

The first is that the answer is a function of two inputs only: the source's state at the moment of the call, and the timestamp the caller passed in. Nothing about the caller's history is an input. The source does not record that an earlier call was made, which records that call returned, or whether the process that received them is still alive. It has no field in which such a thing could be stored, because nothing in the contract requires one.

The second is that the only lever the caller holds over that answer is the timestamp. A record comes back exactly when its updated_at exceeds the value supplied, and is withheld exactly when it does not. There is no other parameter to reach for.

Now add the one fact about failure that matters. A document's updated_at lives on the source's side and records when a person edited it. It does not move when a client fails to finish processing it, because nothing the client does touches the source's row. So once a stored cursor has passed a document's updated_at, every future poll supplies a timestamp at or above that value, the filter withholds that document on every one of those polls, and it stays withheld until somebody edits the page again for reasons of their own.

That is why the loss is permanent rather than delayed, and it is a stronger guarantee of loss than most storage systems can offer of anything. A queue that drops a message can often be drained again from a retained log. A log reader that loses its position can be rewound to an earlier offset. A filter that has been asked the wrong question does not fail at all — it answers correctly, and the correct answer is silence. Every component behaved: the source told the truth, the connector processed everything it was handed, and the index accepted every write it was given. The document is gone anyway.

The two writes cannot happen at the same instant#

Name the two moments and reason about their order rather than about the code. Write C for the instant the cursor value becomes durable in Redis, and W for the instant the last chunk of a document becomes durable in the index. Both are real instants, and they are never the same instant, because the cursor store and the index are separate systems reached over separate network calls with separate flushes to disk. Between C and W lies an interval in which exactly one of the two writes has happened.

A crash inside that interval produces one of two failures, and which one depends only on the sign of the difference.

If C comes first, a crash between C and W leaves the cursor advanced and the document unwritten. By the derivation above, that document is now unreachable. That is at-most-once delivery: every document is processed zero times or once, and which of the two you got is not recorded anywhere.

If W comes first, a crash between W and C leaves the document written and the cursor still pointing behind it. The next poll returns that document again and it is processed a second time. That is at-least-once delivery: every document is processed once or more, and the excess is visible in the index.

The interval cannot be closed. Moving the two writes closer together in the source file narrows the window and lowers the probability that a crash lands inside it, and changes nothing whatsoever about what happens when one does. Two systems cannot be made to acknowledge a write simultaneously without a protocol that neither Redis nor a vector index is offering here. So the decision in front of a connector author is not between a safe design and a risky one. It is a choice of sign.

That makes cursor placement a question about ordering side effects against checkpoints, which is the same question a Kafka consumer answers when it decides whether to call commitSync() before or after it has written the record to the database. Committing the offset first turns the consumer into at-most-once delivery: a crash between commit and write loses the record permanently. Committing after the write gives the other guarantee, at-least-once delivery, where the same crash means the record comes back and is processed a second time rather than being lost. The connector's Redis write is exactly that commit, and the upsert into the vector index is exactly that write.

The comparison is a comparison of one decision, not of two equivalent systems, and the place it breaks is the place the connector is worse off. What makes the connector version worse in practice is that a Kafka topic retains its records for days, so a lost offset can be rewound by hand, whereas an updated_after API keeps no history of what it returned to whom. Rewinding the cursor to 09:11:47 would re-fetch the page, but nothing in the system knows that 09:11:47 is the number to rewind to. A Kafka offset is a coordinate in a sequence that still exists; a cursor is a coordinate in a filter over state that has moved on. Rewinding the first is addressing something; rewinding the second is guessing.

Tracing one page through the gap#

The sequence that produces such a gap is worth tracing precisely, in a connector called acme-docs that reads a documentation source a page of a hundred documents at a time, because the failure lives in a five-second window that no single component can see.

The fourth line is the whole fault. The cursor is set to the last item's timestamp while the loop below it has not run, so from that line onward the stored claim covers a hundred documents of which none has been written.

Docs 37 through 100 exist in exactly one place during the gap, which is the crashed process's heap. When the heap goes, the only record that they were ever selected for processing goes with it. Note what the final two lines of the trace are: a correct request and a correct answer. The source is not lying and the connector is not confused. The information that would make the request wrong was in memory that no longer exists.

Where the commit belongs, and what each placement buys#

Separating what the source imposes from what the pipeline chose matters because only one of the two is negotiable. The source imposes that changes are offered by timestamp, that they are offered once, that no delivery is recorded, and that a caller must supply a timestamp to be given anything at all. Everything else is a decision made on your side of the wire. Where the cursor is written, whether it advances once per page or once per document, whether a safety margin is subtracted from it before it is stored, and whether a second durable record of in-flight work exists are four choices, and a source behaving in exactly the same way will accept all four. A fault that follows from the first list is a constraint to design around. A fault that follows from the second list is the design.

Four placements show up in real connectors, and they are not variations on a theme; they give genuinely different delivery guarantees and leave genuinely different residue behind after a crash.

Read the table as differences between rows rather than as four independent options. The gap between the first row and the third is the sign of the interval and nothing else — the same code, the same page, one statement moved. The gap between the third row and the fourth is not a change of sign at all, since both are at-least-once; it is a change in where the evidence of an interrupted page survives. And the gap between the first two rows is nothing, which is the most useful cell in the table.

The first two rows collapse into one failure because an embedding held in memory is not durable state. Teams sometimes reason that the expensive part is done by the time embeddings exist, so the cursor may as well move; the pod that dies holding 100 embedding vectors has the same effect on the index as the pod that dies holding 100 raw JSON bodies. The reasoning is tempting because effort feels like progress, and durability is indifferent to effort. Nothing that has not been acknowledged by a system that survives the process counts, however much it cost to compute.

The bound in the third row is the one in the table that can be checked exactly. With the commit after the last upsert of a page of a hundred, a crash while processing the k-th document leaves the documents before it already written and the cursor unmoved, so the replay rewrites those earlier documents and nothing else. When the crash lands on the first document, none has been written yet and the replay duplicates none; when it lands on the last, every earlier document is duplicated. Counting the earlier documents at each end of that range gives none at the bottom and one short of the page at the top, which is the pair of numbers in the cell. The same counting applied to the first two rows runs the other way — the documents lost are the one being processed plus every one after it — and the worst case there is the whole page, since a crash on the very first document loses all of them. That upper end is stated one short in the table, and it is left as printed here.

The third row is the correct default for most connectors, and it is one line of code away from the broken version:

def sync_page(cursor: str) -> str:
    page = source.changes(updated_after=cursor, limit=100)
    if not page.items:
        return cursor

    for doc in page.items:
        blob = source.fetch(doc.id)
        chunks = chunk(parse(blob))
        index.upsert(
            [
                {"id": f"{doc.id}:{i}", "values": embed(c), "metadata": {"doc_id": doc.id}}
                for i, c in enumerate(chunks)
            ]
        )
        # cursor advances one document at a time, after that document is durable
        cursor_store.set(CURSOR_KEY, doc.updated_at)

    return page.items[-1].updated_at

Advancing per document rather than per page shrinks the replay window from a whole page to a single document, which matters when a page is 500 items and each item costs 300ms of embedding time. It requires the source to return changes in non-decreasing updated_at order, which most APIs that offer updated_after do guarantee, and which is worth verifying rather than assuming — a source that returns a page sorted by relevance or by ID will produce a cursor that jumps forward and then backward, and the forward jump strands everything between.

The stranding is worth spelling out, since it is the same permanence argument arriving through a different door. Suppose the page is ordered by ID, and the connector reaches a document edited this afternoon early in the page. It commits that afternoon timestamp. Every document later in the page that was edited this morning now has an updated_at below the stored cursor, so the filter will withhold it forever, and it was withheld while sitting in a page the connector already had in hand. A later commit with an earlier timestamp moves the cursor back and re-covers those documents, which is why the fault only shows up when the highest timestamp in the page is not the last one processed.

Two details around the cursor value itself cause more incidents than the placement does. Sources that expose updated_at at one-second granularity will have several documents sharing the boundary timestamp, and a strict updated_after skips every tied document that happened to be in the next page. The fix is a composite cursor of (updated_at, id) where the API supports it, and where it does not, deliberately rewinding the committed cursor by a safety margin — 30 seconds is a common choice — so that each poll re-requests a small overlap. Overlap means documents get reprocessed on every single poll, which is only acceptable if reprocessing is harmless, and making it harmless is the subject of chapter 2, where a chunk's row key is derived from its document's identity and its position so that a repeated write lands on the row it wrote before.

Per-document commits also change what a restart costs. With page-level commits and a 500-item page, a pod that dies on item 480 redoes 480 documents, and at 300ms of embedding time each that is two and a half minutes of recomputation plus the embedding spend for 480 documents you had already paid for once. With per-document commits the same crash redoes one. The price is one Redis round trip per document rather than per page, which at 500 documents is 500 extra network calls of roughly 0.3ms each — 150ms spread across a batch that already takes two and a half minutes. When the cursor store is Postgres rather than Redis and the write is a synchronous commit, the arithmetic shifts and a middle ground of committing every 10 or 25 documents is reasonable, because the replay window is still small enough that nobody notices.

Those two figures are the same three digits a thousandfold apart, and the reason is that one Redis round trip costs a thousandth of what one document's embedding costs. That ratio, rather than either number on its own, is what decides the question, and the general statement is short. Let k be the number of documents between commits. The number of cursor writes per page is the page size divided by k, so the overhead falls as k rises. The replay window after a crash is the work done since the last commit, which is at most k documents, so it rises as k rises. Multiply the two and the k cancels exactly: the product is the page's total embedding time multiplied by the cost of one cursor write, whatever value of k you pick.

That cancellation says something a table of recommended batch sizes cannot. The batch size is not a dial that reduces cost. It is a dial that moves cost between two axes whose product is fixed, so every value of k sits on the same curve and the only way off the curve is to change the ratio itself. A Postgres synchronous commit has to reach stable storage before it returns, which makes a cursor write far more expensive relative to an embedding call than a Redis round trip is, and that moves the whole curve rather than your position on it. Batching then buys back a position on the new curve.

The cancellation holds under one assumption worth naming, which is that documents cost about the same as each other. When one item in a page is enormous and the rest are small, the replay window is dominated by that single item and no choice of k helps: committing every document still means the crash on the big one replays the big one. The residual, in other words, is the tail of the size distribution, and a page containing a scanned PDF has a very long tail.

The fourth row exists for pipelines where the fetch itself is expensive or rate-limited and you would rather not re-fetch a whole page. The connector writes the page's document IDs into a durable work table in the same transaction that advances the cursor, then marks each row done as its upsert lands. A crash leaves rows visibly stuck in in_flight, and a sweeper picks them up after a timeout. The cost is a second piece of state that can itself drift, and the benefit is that the gap becomes a row in a table you can query rather than an absence you cannot.

That row is not an exception to the interval argument. It does not make C and W simultaneous. It adds a third durable record whose write is ordered with the cursor's, so that the set of documents a crash interrupted survives the crash even though the documents themselves do not. The interval is still open; what has changed is that the evidence of falling into it now exists somewhere you can run a query against.

Exactly-once delivery, and change data capture#

Committing after the upsert does not give exactly-once delivery, and no arrangement of these two operations will. The index write and the cursor write live in different systems, so there is always an instant where one has happened and the other has not; all you are choosing is which side of that instant you would rather be caught on. Exactly-once in this setting means at-least-once delivery plus a write that produces the same result when repeated, which is a property of the write rather than of the transport.

Incremental sync over updated_after also is not change data capture, even though the two are often used interchangeably in design documents. CDC reads the source's own change log — a Postgres WAL, a MySQL binlog, an audit stream — and therefore sees every mutation in commit order, including deletions, and including two edits to the same row that a polling connector would collapse into one. A polling connector sees a snapshot of current state filtered by a timestamp, which means it can never observe an intermediate revision and can never observe a deletion at all. That second limitation is structural rather than a matter of polling frequency, and chapter 4 takes it apart, since a deleted row has no updated_at left for a timestamp filter to compare against and so cannot appear in any answer the filter gives.

A change log is a record of deliveries that outlives the client; a timestamp filter is a query over current state that remembers nothing. Polling more often narrows the window in which an intermediate revision can be missed and never closes it, exactly as moving two lines of code closer together narrows the crash window and never closes it.

Why the two failure modes are not mirror images#

The argument that shows up whenever a team debates this is that the choice is a trade-off between losing documents and duplicating them, so it should be decided by whether the workload minds gaps more than it minds noise. That framing treats the two outcomes as symmetric, and they are not symmetric in the only dimension that matters operationally, which is whether you can find out that it happened.

Duplicates are a state of the index. They can be counted with a group-by over doc_id, they show up as repeated passages in retrieval results, and they can be removed by rewriting the affected documents. Every one of those operations is available to you at any time, without cooperating from the source.

A gap is the absence of a state. Nothing in the index records that 64 documents were once in flight, nothing in the source records which documents it handed to a client that then died, and the query that would find the gap — "which source documents are missing from my index" — requires enumerating every document in the source, which is the full resync you were trying to avoid by syncing incrementally in the first place. For the 4M-page wiki that later chapters use as a running example, a full resync at a sustained 120 documents per second is a little over nine hours of wall clock and roughly 1.6 billion tokens through an embedding model, which at $0.02 per million tokens is about $32 in inference and considerably more in engineering attention. That is the recovery procedure for a fault you cannot detect, which is why the placement that produces duplicates is the one to build on.

Those two figures come from instruments that share no step, and the gap between them is the argument. Dividing a document count by a sustained rate gives the wall clock; multiplying a token count by a price per token gives the money. Neither calculation touches the other's inputs, and they disagree completely about how bad a full resync is — trivial in dollars, most of a working day in elapsed time, with a source API being hammered throughout. The dollar figure is the one that gets quoted in a design review and the one that measures the least, because it prices embedding inference alone: it does not price the parser, the source's rate limits, the index write amplification, or the person who has to decide whether the resync is safe to start. Reaching for the cheaper instrument is how a recovery procedure gets described as affordable.

Building on it means the duplicates have to stop being duplicates. A connector that commits late will replay documents on every crash, every deploy, and every poll if you are using an overlap window, so the write path has to be able to absorb the same document arriving twice without leaving two copies behind.

The line to move is one line, and moving it does not make the connector safe. Committing after the upsert guarantees repeats; it is a choice of which failure to have, made on the grounds that one of the two failures leaves evidence and the other leaves nothing. A duplicate is a row you can select, count, and delete. A gap is a document sitting in a wiki, indistinguishable from every document that was correctly indexed, waiting for somebody to search for it and be told there is no such page.