Kafka dead letter queue monitoring: get alerted on the first failed record
Monitor a Kafka dead letter queue and get an alert on the first failed record

Monitor a Kafka dead letter queue and get an alert on the first failed record
To get alerted on the first failed record, monitor the dead letter queue (DLQ) topic for arriving records: fire an alert when more than zero records arrive within a short time window, and send it to the channel your on-call engineer actually reads. Consumer lag will not tell you this, and neither will a dashboard nobody has open.
The usual failure is not technical. The application does the right thing and moves a record it cannot process to a topic such as orders.dlq (an example name). Then nobody watches that topic, and the records sit there until a customer asks where their order went.
Two signals are useful. Which one to alert on depends on how the topic normally behaves.
Consumer lag is the distance between the end of a partition and the offset a consumer group has committed. It describes a consumer, not the topic. Many DLQ topics have no consumer group that runs continuously, so there is no lag to measure. If a group only runs when someone reprocesses the DLQ by hand, lag grows with every failure, but it also contains every record nobody has dealt with yet, so it says little about the last minute. And if a service drains the DLQ automatically, lag stays near zero while failures pour in. What you want to know is whether something was just written to the topic, and that is a question about the producing side.
A metrics exporter plus an alert rule. A Kafka metrics exporter publishes the end offset of every partition to a monitoring system such as Prometheus. An alert rule sums the end offsets of the DLQ topic and fires when the sum has increased over the last few minutes. This fits teams that already run a metrics stack. The cost is upkeep: the exporter has to cover every cluster, and the rules need a naming convention or a hand-maintained list of DLQ topics.
A small consumer that pages someone. A service subscribes to the DLQ topics and posts to a chat channel or an incident tool for each record. The message can include the record key and the error details. The drawbacks: it is one more service to run, it needs rate limiting so that ten thousand failures do not become ten thousand messages, and when it crashes the result is silence, which looks like a healthy DLQ.
Both work. A topic alert in a Kafka UI is a third option that needs no extra component.
Kadeck's Topic Data Arrival Alerts count the records arriving in a topic over a rolling time window and notify you when the count crosses a threshold. The steps below use the values of the DLQ example in the documentation.
orders.dlq.0.1.15.Creating or editing a topic alert requires the TopicAccessRead permission. Alert creation and configuration changes are recorded in the audit log. In the Data Catalog, the alerts-configured tag filters for topics that have alerts, a quick way to check that no DLQ was left out.

A DLQ alert only fires when failures are written somewhere. If the producer stops, or a consumer crashes before it can dead-letter anything, the DLQ stays empty. The second pattern covers that case: Less than 1 record on the main topic, for example orders, fires when no data arrived in the window. The documented "pipeline silent" example: less than 1 record, a 10 minute time window, a 30 minute cooldown, PagerDuty.

Pick a window longer than the longest normal gap between records. A topic that only receives a nightly batch would trip a 10 minute window all day.
Not every problem record ends up in a DLQ. A Kadeck pipe can filter records that match a condition into their own topic, so that a topic alert fires when one arrives.
Read the failed records first. Many frameworks write the error reason into the record headers; a Kadeck pipe that sends failed records to a dead letter topic, for example, adds kadeck.pipe.error.stage and kadeck.pipe.error.message along with the source topic, partition and offset. Fix the cause before touching the records, otherwise they fail again. After that, the failed records can be inspected and repaired: a pipe can correct them and write them back to the original topic.
Alert on records arriving in the DLQ topic. Either compare end offsets over time in your metrics system, or use a topic alert with the condition "more than 0 records in 1 minute".
Usually not. Lag needs a consumer group, and many DLQs either have none or have one that runs only during reprocessing. Record arrival is the direct signal.
Set a cooldown so the same alert cannot fire again right away, and raise the record count or widen the time window until normal background failures stay below the threshold.
Yes. A topic alert notifies Slack, Microsoft Teams, PagerDuty or email. Kadeck works with Apache Kafka, Redpanda, Amazon MSK and Aiven.
Topic data arrival alerts and their notifications are part of Kadeck Platform; they are not included in the Free or Desktop plans. Platform is priced per user, from 10 users, billed annually. You can get started here and find the details on the pricing page.