How to reprocess a Kafka dead letter queue: fix and replay failed records

How to reprocess a Kafka dead letter queue: fix and replay failed records

Inspect, correct and replay Kafka dead letter queue records to the original topic

To reprocess a Kafka dead letter queue (DLQ), read the failed records, correct whatever made the consumer reject them, and produce the corrected records back to the topic they belong to. Replaying them unchanged only helps if the cause was temporary, such as an unavailable downstream service; if the data itself is wrong, the records fail again and return to the DLQ.

The topic names orders and orders.dlq in this article are examples.

Why records end up in a dead letter queue

A dead letter queue is a separate topic that collects the records a consumer or connector could not process, so the partition is not blocked and nothing is lost. Typical causes are:

  • The record cannot be deserialized, or it does not match the expected schema.
  • A field is missing or malformed and validation rejects the record.
  • A dependency, such as a database or an API, was unavailable and the retries ran out.

If the cause was temporary, the records are fine and can be replayed as they are. If the cause is in the record, replaying it blindly gives the same result: the same record meets the same code and lands in the DLQ again. So the first step is always to look at the failed records.

The usual ways to reprocess a DLQ

  • A one-off script. A small consumer reads the DLQ, applies the fix, and a producer writes the result to the original topic. It is flexible and quick to write, but it needs credentials, a place to run, and your own handling of offsets, serialization and errors.
  • A replay service or retry topics. Some teams build replay into the application: retry topics with a delay, or a service that republishes DLQ records on request. This works well when failures are frequent and mostly temporary. It is software to build and operate, and it usually replays records as they are.
  • Kafka Connect. Sink connectors can route failed records to a DLQ topic, and a connector can copy a topic to another one, with single message transforms for simple changes. This fits if you already run Connect; more involved corrections are awkward to express in transforms.

All three work. In each, the correction has to be deployed before you see what it does to the records.

Fix and replay failed records step by step

The steps below use Kadeck Pipes. A pipe writes the records of a topic to another topic, processed the way a topic view shows them.

1. Inspect the failed records

Open the dead letter topic, here orders.dlq, in the Explorer and select a record to see its key, value and headers. To compare failed records with healthy ones, click Add topic or view and add orders. The two views form a topic set, and All combined shows the records of both topics in one list.

2. Correct the records with the JavaScript Processor

In the orders.dlq view, select the JavaScript Processor tab above the record list and click Start. Your code runs once for every record, which is available as rec (see the processor reference). Return rec to keep the record with your changes, or false to leave it out.

Say the consumer rejected orders because total arrived as text such as "49,90" instead of a number. This processor repairs those records and leaves out the rest:

if (typeof rec.value.total !== "string") return false;
rec.value.total = Number(rec.value.total.replace(",", "."));
return rec;

A failed record with total stored as the text 49,90 is corrected by the JavaScript Processor and written to orders with total as the number 49.9.
The processor turns the text value into a number before the record is written back.

Click Apply. The records are read again and shown as the code returns them. The records stored in the topic are not changed.

3. Check the record list

The record list is the preview of the pipe: it shows the records the way they will be written. Add filters if only a part of the records should go back, for example the records of one error type.

4. Convert the view into a pipe and test it

Click Pipe in the toolbar of the view, next to Produce. The Convert to Kadeck Pipe dialog opens. The settings that matter here:

  • Target: the cluster and the topic to write to, here orders. The topic must exist.
  • Apply: choose Selected parts and switch End condition off. Otherwise the pipe writes at most the view's maximum number of results per partition, 50 by default.
  • Run: choose Run once. It writes what the view shows now, then completes.
  • Write as: check that the codecs for key and value are the ones the consumers of orders expect.
  • On error: Drop leaves a record that fails in the pipe out and counts it. Dead letter topic writes it to another topic, with the reason in its headers.

By default, a pipe also applies the data protection policies of its author, so attributes that are masked for you are written masked.

Then click Test run. It takes one record through decoding, processing and encoding for the target without writing anything, and shows the record the pipe would write, or the step that failed and why.

5. Write back to the original topic

Click Start pipe. The pipe completes when it reaches the end of the dead letter topic as it was when the pipe started. Its details show how many records were processed, written, filtered, dropped and dead-lettered, and Target records opens orders.

What to watch out for

  • Duplicates. A pipe delivers at least once and does not use transactions. It commits its position in the source topic only after the write succeeded, so no record is left out, but a record can be written twice. The records also stay in the DLQ, so a second pipe over them writes them again. Make the consumers of orders idempotent, or make the records identifiable by a key or an id in the value.
  • Ordering. The source partition is not kept: it is assigned on write, by the key that is written, and the order of replayed records relative to the other records in orders is not guaranteed. They arrive after everything produced in the meantime, so check that a replayed record does not overwrite newer state for the same key. The record timestamp is kept.
  • Loops. A record the application rejects again returns to orders.dlq. With Run once, the pipe has a fixed end and does not pick it up again. With Keep running, it would. For the pipe's own On error setting, pick a dead letter topic other than the one the pipe reads from.
  • Traceability. Switch Lineage header on: every written record then carries the header kadeck.pipe with the pipe name, which helps tell replayed records apart.
In the orders topic, the replayed record arrives after the records produced since the failure and keeps its original timestamp.
A replayed record lands at the end of the topic but keeps its original timestamp.

Questions

Can I replay a Kafka dead letter queue without writing a consumer?

Yes. Any tool that reads one topic and writes to another can do it: a command-line tool, a connector, or a pipe. If the records need a correction first, the tool must be able to change them on the way.

How do I replay only some of the records in a DLQ?

In a Kadeck view, add filters, or set the offset and the maximum number of results in the display settings and apply the Entire view, so the pipe writes only that range.

Are records removed from the DLQ after they are replayed?

No. Reading from a Kafka topic does not remove records. They stay until retention deletes them or you delete them, so note which records you have already replayed.

Next steps

Kadeck works with Apache Kafka, Redpanda, Amazon MSK and Aiven. The Free plan includes one active pipe, and running, paused and failed pipes count as active; Desktop Professional and Platform allow multiple active pipes. You can get started here and compare the plans on the pricing page.