A dead-letter record is only useful if it can be replayed exactly

Some fraction of any real document set will not parse. An e-signature provider emits PDFs carrying an incremental update and an encrypted signature dictionary; a fax arrives as a scanned image with no text layer in it at all. A pipeline that meets one of these has three options and only three. It can retry the document forever, it can throw the document away, or it can set the document aside somewhere a person can find it later. The first choice stops everything behind it. The second is what most systems ship, and its defining property is that it leaves nothing behind — a dropped document appears in no count, in no queue, and in no answer to the question of which documents are missing from an index.

A dead-letter queue is the third choice. It is where a message goes once a pipeline has decided it cannot be processed: instead of being retried forever or thrown away, it is set aside in a store somebody can inspect and act on later, which is why the same thing is often called a quarantine. What decides whether it is worth anything is what goes into each record. A record that names a failure is a log line with a longer retention. A record that reconstructs the attempt — which bytes, which code, where in a connector's timeline — is the only kind a person can act on once the code and the bytes have both moved on, which they will have.

Two jobs, and the one that dropping the message gives up#

A dead-letter path exists so that one document cannot stop a pipeline and cannot vanish from it. Those are two separate jobs, and they get conflated because a single mechanism serves both.

Start with why an unfinishable message is dangerous at all, because the answer is a chain of checkable links. A broker hands a message to a consumer and waits for an acknowledgement, which is the consumer's statement that the message is finished with. A consumer that crashes, or raises, sends no acknowledgement. The broker has no view inside the process, so it cannot tell a crashed consumer from a slow one. Its only safe response to a missing acknowledgement is therefore to deliver the message again. That is at-least-once delivery, and it is a deliberate choice rather than an accident: the alternative loses a message on every crash.

What happens next depends on where the cause of the failure lives. If the cause was the moment — a timeout, a restart, a full disk — the next delivery meets different conditions and succeeds. If the cause is a property of the message itself, every later delivery reproduces the failure exactly, and the sequence never converges, because nothing in it changes between iterations.

On a partitioned log that is worse than an endless loop over one document. A Kafka consumer reads a partition in offset order, and the ordering guarantee it offers downstream is worth something only if it never skips ahead, so it cannot advance past a message it has not finished. The partition stops advancing, and every document queued behind the failing one waits, including the documents that would have parsed in milliseconds. One malformed PDF becomes a stalled source.

Taking the message out of the delivery path is therefore a requirement rather than a preference, and dropping it and quarantining it both satisfy that requirement equally. Only quarantine also does the second job, retaining enough of the failure that somebody can act on it later. Choosing between them is a design decision about evidence, it changes throughput by nothing at all, and that is why it so often gets made by default, inside an except block, by whoever was trying to clear the stall.

Sorting failures by whether waiting can help#

The neighbouring mechanism a dead-letter queue gets confused with is the retry queue, and the distinction is which side of the failure taxonomy the message falls on. A 429 from the embedding provider, a connection reset to the index, a 503 from the source — these are conditions of the environment, they resolve on their own, and a message that hits one should go back through with backoff. A password-protected PDF is not going to become readable on the fourth attempt. Sending both classes down the same path produces a queue that is mostly things that would have succeeded, which is why so many dead-letter queues are simultaneously enormous and uninformative.

The belief worth naming before the taxonomy is that a retry count is itself a defence. It is tempting for good reasons: a retry costs one configuration line, it is uniform, it requires no knowledge of what failed, and by count it does repair most of the failures a pipeline meets. What makes it a poor defence is that the number stays fixed while the thing it depends on does not. Three attempts is three attempts whether the cause sits in the network, where it will be gone within a second, or in the bytes, where it will still be there next year. The strength of the defence is constant, the property deciding whether it can work varies message by message, and nothing in the mechanism measures that property. On the failures where waiting cannot help, a retry budget does not fail neutrally — it multiplies the cost of the failure by the size of the budget.

Read the last column as gaps rather than as levels. It runs from five-to-eight down to one, one and zero, and it is not a severity ranking — a downstream contract violation is a more serious thing to discover than a bad encoding, and it sits at zero while the encoding sits at one. The column answers one question and only one: can the environment change between one attempt and the next? Only the first row can answer yes. That is why the entire retry budget lives in that row and every other row gets at most a single further attempt. The gap between the first row and all the rest is the taxonomy.

The two zero rows arrive at zero by different routes, and the difference decides where the check goes in the code. A permanent input can be recognised from the input. A downstream contract violation — a chunk longer than the model's context window, a vector of the wrong width — can only be recognised after the document has been parsed and chunked, so its zero means do not try a second time rather than do not try. It also alerts, because a chunk that exceeds a context window is usually a configuration change rather than a bad document, and the next thousand documents will do the same thing.

That column is also off by a word, which is worth correcting before implementing from it. The heading says attempts, and the values count retries after the first delivery. Read literally, the encrypted-PDF row asks for a document to be classified as encrypted without ever having been fetched, and nothing can do that: encryption is a property of a file's contents. Zero there means no second attempt, and the ambiguous-parse row's one means one redelivery, for two deliveries in total.

Quarantining a permanent input immediately rather than after three attempts is worth more than it sounds. Three retries against an encrypted PDF costs three full downloads of the file, three parser initialisations and three stack traces, and at 1,600 failures a day with an average PDF of 4 MB that is 19 GB of pointless egress per day. Classifying up front costs one if.

That arithmetic is worth doing rather than accepting. Three downloads at four megabytes each is twelve megabytes thrown away per document, and sixteen hundred documents a day at twelve megabytes each comes to a little over nineteen thousand megabytes. The word carrying the figure is pointless: the first download is not pointless, since finding out that a file is encrypted means having the file, so three is the count of downloads that produced nothing the first had not already produced. The saving is the whole of the retry budget, not the whole of the traffic.

The resource-fatal row is the one that needs machinery rather than an exception handler, because a process that is OOM-killed does not get to run its except block. The message never reaches the dead-letter path under its own power; it comes back on the next poll, kills the next consumer, and repeats. Brokers expose a counter for exactly this — SQS carries ApproximateReceiveCount on every receive, Kafka consumers have to keep their own in a state store — and the consumer's first action on receiving a message should be to check it:

FATAL_AFTER = 2

def handle(message):
    if int(message.attributes["ApproximateReceiveCount"]) > FATAL_AFTER:
        quarantine(message, error_class="resource_fatal",
                   detail="exceeded receive count without acknowledgement")
        return

    doc = json.loads(message.body)
    head = source.head(doc["id"])
    if head.content_length > MAX_PARSE_BYTES:          # 64 MB
        quarantine(message, error_class="permanent_input",
                   detail=f"content_length={head.content_length}")
        return

    try:
        process(doc)
    except (EncryptedDocument, UnsupportedMimeType) as e:
        quarantine(message, error_class="permanent_input", exc=e)
    except ParseError as e:
        if int(message.attributes["ApproximateReceiveCount"]) >= 2:
            quarantine(message, error_class="ambiguous_parse", exc=e)
        else:
            raise                                       # let it be redelivered once

The two branches that read the counter disagree with each other, and reading them together is what makes the disagreement visible. ApproximateReceiveCount starts at one on the first delivery, so the ParseError branch's >= 2 fires on the second delivery, which is one redelivery, which is the table's one. Apply that same reading to the guard at the top: FATAL_AFTER is two and the comparison is strict, so the guard fires on the third delivery, after two consumers have already been killed, where its table row asks for one. Either branch on its own looks reasonable. The pair of them fixes the meaning of the counter and then breaks it.

The size pre-flight is what would have saved the pod in chapter 1. A 180 MB scanned PDF is a resource-fatal input whether or not the parser can theoretically handle it, and finding that out from a HEAD request costs 40ms.

That check is the only decision in the function made before a byte of the document has been transferred, because content length is metadata a source will state on request. It is also why the pre-flight records permanent_input while the sentence above calls the same file resource-fatal. The sentence describes what the file would do to a parser; the field records how the decision was actually reached, which was from a property of the input with no process yet at risk. resource_fatal is reserved for the case where nothing survived to record anything, which is precisely why that row has to be enforced from outside the process.

The record has to reconstruct the attempt#

A dead-letter record is an evidence bag. The reason to be pedantic about what goes in it is that somebody will open it months later and have to decide what a replay result means, and that decision depends entirely on knowing the conditions under which the failure happened.

{
  "dlq_id": "01JQ7X0F5C3N4K8Y2M6P9RTVAZ",
  "source": "wiki",
  "doc_id": "1147829310",
  "source_version": "W/\"5c9f-1b3d2a7\"",
  "source_updated_at": "2026-02-11T09:41:07Z",
  "cursor_at_arrival": "2026-02-11T09:41:07Z",
  "ingest_seq": 88213774,
  "connector_version": "wiki-connector@3.14.2",
  "parser_version": "pdfminer.six@20221105",
  "chunker_version": "recursive-800-100@2",
  "embedder_version": "text-embedding-3-large@1536",
  "payload_uri": "s3://ingest-quarantine/wiki/1147829310/9f2a4c1e.pdf",
  "payload_sha256": "9f2a4c1e6b8d0f37a25c194e7b0d6f8213ac54e9d7b1c0a3f6e2d8b4a1c7e50f",
  "payload_bytes": 4718592,
  "error_class": "ambiguous_parse",
  "error_type": "PDFSyntaxError",
  "error_message": "No /Root object! - Is this really a PDF?",
  "stack_trace_uri": "s3://ingest-quarantine/traces/01JQ7X0F5C3N4K8Y2M6P9RTVAZ.txt",
  "attempts": 2,
  "first_seen_at": "2026-02-11T09:42:13Z",
  "last_seen_at": "2026-02-11T09:44:51Z"
}

Four of those fields do work that is not obvious.

source_version is what makes a replay interpretable. Suppose a replay of this record succeeds in June. Either the parser now handles files that pdfminer.six@20221105 choked on, or somebody re-exported the contract from a different tool and the bytes at the source are no longer the bytes that failed. Those two outcomes call for opposite follow-up actions: the first means every other record with the same error_type and parser version is now repairable and the whole cohort should be replayed, and the second means nothing has been learned and the other 400 records in that cohort are still broken. Comparing the stored etag against the source's current etag at replay time separates them in one request.

The shape of that problem is worth stating generally, because it recurs wherever a replay is used as evidence. A replay outcome depends on two things that can both have moved since the failure: the code that reads the bytes, and the bytes. One replay yields one observation, success or failure, and one observation cannot separate two unknowns. Holding either one fixed makes the other readable. The etag is how you hold the bytes fixed — it is the source's own statement that its content has not changed — and once the bytes are known to be unchanged, the outcome of the replay is a statement about the code alone.

That is also the limit of what an etag buys. An etag is a source's opinion of its own content rather than a measurement of it. A source that re-saves a document without editing it can issue a fresh etag for identical bytes, which reads as a change that never happened, and a source deriving its etag from a modification timestamp with second resolution can miss two edits inside the same second. The digest stored beside the payload is the stronger instrument, because it is computed from the bytes themselves — and it can only be checked against bytes you still hold, which is exactly what a replay against the current source revision does not have. Neither instrument covers both blind spots.

parser_version and its siblings turn the queue from a bag of failures into cohorts. A well-kept dead-letter queue answers "how many records failed under a parser version we no longer run, with an error type that the current version has a test for", which is the query that tells you what a drain is likely to accomplish before you run it. Without the versions, every record is an independent mystery and the only way to learn anything is to run all of them.

What a cohort licenses is a jump from one replay result to many, and the jump is legal in only one direction. If the bytes are unchanged and the replay succeeds, what changed is the code, the code is common to every record in the cohort, and the result generalises to all of them. If the bytes changed, it generalises to nothing, because the only thing that moved was specific to that one document. The residual is that a cohort is defined by a fingerprint, a fingerprint is a hash of an exception type and a few stack frames, and two documents can reach the same frames from different underlying defects.

cursor_at_arrival and ingest_seq place the failure in the connector's timeline. They let you answer whether a burst of failures corresponds to a particular sync window — a source migration, an outage, a bad deploy — and they give the replay a defensible position in the ordering that chapter 3's version guard arbitrates.

payload_uri with payload_sha256 exists because the message body cannot hold the document. SQS caps a message at 256 KB and Kafka's default max.message.bytes is 1 MB, so anything larger has to live in object storage with the record holding a pointer. Recording the digest alongside it means a replay can verify that the bytes it is about to feed the parser are the bytes that failed, rather than a re-fetch that silently substituted a different revision. Storing the payload at all is a decision with consequences, which is that the quarantine bucket is now a datastore holding full document bodies — the same fan-out obligation chapter 4 raised for deletion applies to it, and an erasure request has to reach into it.

For the record above, the size question is settled rather than open. Its payload_bytes is four and a half mebibytes, which is exactly eighteen times the SQS cap and several times the Kafka default. At that size the pointer is forced rather than chosen, and the only decisions left are where the bytes live and for how long.

A queue you cannot query is a queue you will not use#

Brokers offer a dead-letter queue as a built-in — SQS redrive policies move a message after maxReceiveCount, and Kafka setups usually publish to a .dlq topic — and it is tempting to let that be the whole implementation. The problem is the access pattern. Everything useful you will ever ask of quarantined documents is a query: how many records for source=wiki with error_type=PDFSyntaxError under a parser version older than the current one, grouped by month; is document 1147829310 in the queue; what is the oldest record for each connector. Answering any of those against SQS means draining the queue into memory, and draining is destructive.

Destructive is a consequence of what a queue is, and the built-in looks so nearly sufficient that it is worth spelling out. A queue's read operation is a receive, and a receive is a lease: the message becomes invisible to other consumers and is then either deleted or released back. No operation reads a message and leaves the queue as it was, and none reads the tenth message without first reading the nine in front of it. A queue is a transport with a cursor, and every question above wants an index.

The arrangement that survives contact with an incident is a quarantine table in Postgres holding the record above, one row per (source, doc_id) with the attempt history appended rather than duplicated, plus the broker's own dead-letter queue as the transport that gets the message out of the consumer's way. The table is small — 584,000 rows a year at a few hundred bytes each is under a gigabyte — and it supports the one property a queue cannot: upsert on (source, doc_id), so a document that fails on every sync for eight months occupies one row with attempts: 214 instead of 214 rows that make the queue look like 214 distinct problems.

Both halves of that sizing are worth checking rather than accepting. The row count is a year of a daily failure count of sixteen hundred documents: sixteen hundred a day, times a year, lands on 584,000 with nothing left over — so the figure holds only for as long as that daily count does. The byte count follows from the record above, which is a few dozen short strings and no document bytes at all — at a few hundred bytes a row that is a few hundred megabytes, inside a gigabyte even if the per-row estimate is out by a factor of two.

584,000 is an upper bound rather than a forecast, and the upsert is why. It counts messages, and the table counts documents: a document that fails on every sync contributes one row and many attempts, so the table is smaller than the message count by whatever the average attempt count turns out to be. What the upsert changes is the instrument rather than the loss — eight months of failures are eight months of failures at one row or at 214 — and what it fixes is a count that would otherwise rank one stuck document above a cohort of hundreds of genuinely distinct ones.

Grouping into cohorts only works if error identity is stable, and raw exception messages are not. "No /Root object! - Is this really a PDF?" groups cleanly; "invalid literal for int() with base 10: '\\x00\\x12' at offset 41827" produces a distinct string for every document. Storing a fingerprint alongside the message — the exception type plus the top three frames of the traceback, hashed — gives a grouping key that holds across documents while the full message and trace stay in object storage for whoever has to read one. Without it, a GROUP BY error_message over 412,000 rows returns 380,000 groups and tells you nothing.

The gap between those two figures is the whole argument. Grouping 412,000 rows into 380,000 groups means the average group holds barely more than one row: thirty-two thousand rows' worth of duplication collapsed, and the rest did not. A key that leaves almost as many groups as there were rows has not grouped anything, and the query still costs what a full scan costs.

Retention is the last field decision. A record whose payload is stored indefinitely is a document body held outside every access control the source applies to it, so quarantine buckets need the source's own retention and the same erasure fan-out as the index. Expiring the payload while keeping the metadata row is a reasonable middle position, and it changes what a replay can do — the subject of the next chapter — because a record without bytes can only be replayed by going back to the source.

The rate metric is the part that lies#

Emitting a counter on every quarantine is obvious and correct: ingest.quarantined tagged by source, error_class, error_type and parser_version. What teams then do is put the ratio of that counter to total documents on a dashboard, call it the parse failure rate, and treat it as the health signal for the whole failure path.

That is the second belief in this chapter that a defence works, and it is the more expensive one. The strongest form of why it fails is worth stating before the numbers: the quantity the dashboard holds steady is a ratio, and the quantity that hurts is a total. Hold the ratio perfectly constant and the total grows without bound, at a fixed number of documents per day, indefinitely. A reading that does not move while the danger grows is not a weak defence. It is an instrument pointed at a different variable.

A rate is a statement about a window. It says nothing about accumulation, and a failure path is a place where things accumulate. A steady 0.4% against 400,000 daily documents is 1,600 records a day and 584,000 a year, and the dashboard reads identically on day one and day four hundred. The rate is also insensitive to the thing that actually matters for a corpus that people trust, which is what fraction of the source is currently absent from the index — a number that only goes up while the rate holds flat.

The same figures taken from the other end expose a second way the rate misleads. Two statements in this chapter share no step with each other: three percent of the fifty-three thousand PDFs re-ingested on a normal day, and 0.4% of the 400,000 documents processed on a normal day. Both land on the same sixteen hundred, and the residual between them is a handful of documents, well inside the precision of the word about. That agreement is not a coincidence and it is not a check on either figure. It is one quantity written against two denominators, and the ratio between the two rates is exactly the PDF share of a day's traffic.

Which is why the headline rate moves when nothing has changed. Divide the same daily failure count by a larger denominator and the rate falls. A week of heavy wiki editing, a one-off import of a marketing site, any shift in the mix of content types — each moves the published figure while the number of documents lost that day is identical. A rate reported per content type does not have that defect, because its denominator is the population its numerator was drawn from.

There is a further limitation worth stating plainly: a quarantine path does not improve retrieval by a single point. It converts an invisible loss into a visible one. The documents in the queue are exactly as absent from the index as the documents that were being dropped, and everything downstream — coverage, recall on the affected sources, the procurement lead's three contracts — is unchanged on the day you ship it. What you have bought is the ability to find out, and to act later without going back to the source for a full resync.

So the test of a quarantine record is not whether it captured the failure. It is whether somebody opening it long after both the code and the document have moved on can say which of the two moved. Every field that earns its place above answers that question from a different side: the etag and the digest pin the bytes, the version strings pin the code, the cursor and the sequence number pin the moment, and the fingerprint says which other records share the answer. A record missing them still records that something failed, which is a log line — and a log line is what a fourteen-day retention had already thrown away.

Acting later is a process rather than a data structure, and the record schema says nothing about who owns the queue, what threshold makes it an incident, how long records live, or what a safe drain looks like. A pipeline with an excellent dead-letter record and none of those things produces a queue that grows for a year while every dashboard stays green.