Kafka dead letter queue monitoring: get alerted on the first failed record

Kafka dead letter queue monitoring: get alerted on the first failed record

Monitor a Kafka dead letter queue and get an alert on the first failed record

To get alerted on the first failed record, monitor the dead letter queue (DLQ) topic for arriving records: fire an alert when more than zero records arrive within a short time window, and send it to the channel your on-call engineer actually reads. Consumer lag will not tell you this, and neither will a dashboard nobody has open.

The usual failure is not technical. The application does the right thing and moves a record it cannot process to a topic such as orders.dlq (an example name). Then nobody watches that topic, and the records sit there until a customer asks where their order went.

What to monitor on a dead letter queue

Two signals are useful. Which one to alert on depends on how the topic normally behaves.

  • Arrival of any record. For most DLQs the healthy state is that nothing arrives. A single record means a consumer hit something it could not handle, such as a schema change or a malformed payload. That is worth a message to the owning team.
  • Arrival rate. Some DLQs see a steady trickle the team has accepted, for example a few invalid records a day from an external partner. There, the first record is not news. A jump from a handful per hour to hundreds per minute is.

Why consumer lag on the DLQ is a poor signal

Consumer lag is the distance between the end of a partition and the offset a consumer group has committed. It describes a consumer, not the topic. Many DLQ topics have no consumer group that runs continuously, so there is no lag to measure. If a group only runs when someone reprocesses the DLQ by hand, lag grows with every failure, but it also contains every record nobody has dealt with yet, so it says little about the last minute. And if a service drains the DLQ automatically, lag stays near zero while failures pour in. What you want to know is whether something was just written to the topic, and that is a question about the producing side.

Do-it-yourself approaches

A metrics exporter plus an alert rule. A Kafka metrics exporter publishes the end offset of every partition to a monitoring system such as Prometheus. An alert rule sums the end offsets of the DLQ topic and fires when the sum has increased over the last few minutes. This fits teams that already run a metrics stack. The cost is upkeep: the exporter has to cover every cluster, and the rules need a naming convention or a hand-maintained list of DLQ topics.

A small consumer that pages someone. A service subscribes to the DLQ topics and posts to a chat channel or an incident tool for each record. The message can include the record key and the error details. The drawbacks: it is one more service to run, it needs rate limiting so that ten thousand failures do not become ten thousand messages, and when it crashes the result is silence, which looks like a healthy DLQ.

Both work. A topic alert in a Kafka UI is a third option that needs no extra component.

Setting up the alert step by step

Kadeck's Topic Data Arrival Alerts count the records arriving in a topic over a rolling time window and notify you when the count crosses a threshold. The steps below use the values of the DLQ example in the documentation.

  1. Make sure an integration exists. A Kadeck administrator creates alert integrations under Administration → Alerts → Integrations: Slack, Microsoft Teams, PagerDuty or email (SMTP). Slack, Teams and PagerDuty each need a name and a webhook URL.
  2. Find the topic. Open the Data Catalog and look up orders.dlq.
  3. Open the alert settings. Click ⋯ next to the topic and select Configure Alerts. The alert is set up in the Topic Data Arrival Alert dialog.
  4. Notify to: select one or more integrations, for example the operations team's Slack.
  5. Threshold (Records received): choose More than and enter 0.
  6. Time Window (Minutes): enter 1.
  7. Cooldown Period (Minutes): enter 15.

Creating or editing a topic alert requires the TopicAccessRead permission. Alert creation and configuration changes are recorded in the audit log. In the Data Catalog, the alerts-configured tag filters for topics that have alerts, a quick way to check that no DLQ was left out.

What each parameter does and how to tune it

  • Threshold. More than or less than, plus a record count. The count is the topic total across all partitions. More than 0 means "the first record". For a DLQ with an accepted trickle, raise the count above the normal level.
  • Time window. The rolling window over which records are counted. For a rate alert, widen the window so that one short burst does not decide the outcome: "more than 100 in 10 minutes" is steadier than "more than 10 in 1 minute" (illustrative values).
  • Cooldown period. The minimum time before the same alert can fire again, and the main control against noise. With 15 minutes, an ongoing incident produces at most one notification per 15 minutes. Lengthen it for a busy chat channel.
  • Notify to. Match the destination to the urgency: a chat channel for DLQs that can wait until morning, PagerDuty for the ones that cannot.
Timeline for orders.dlq: the first record triggers an alert within the 1 minute time window, a record arriving during the 15 minute cooldown sends no new alert, and a record after the cooldown triggers the next alert.
Time window and cooldown: one alert per incident instead of one per record.

The opposite problem: a topic that goes silent

A DLQ alert only fires when failures are written somewhere. If the producer stops, or a consumer crashes before it can dead-letter anything, the DLQ stays empty. The second pattern covers that case: Less than 1 record on the main topic, for example orders, fires when no data arrived in the window. The documented "pipeline silent" example: less than 1 record, a 10 minute time window, a 30 minute cooldown, PagerDuty.

Two alert patterns: More than 0 records in 1 minute on orders.dlq catches a failed record, Less than 1 record in 10 minutes on orders catches a silent topic.
Two patterns: alert when something arrives, and alert when nothing does.

Pick a window longer than the longest normal gap between records. A topic that only receives a nightly batch would trip a 10 minute window all day.

Not every problem record ends up in a DLQ. A Kadeck pipe can filter records that match a condition into their own topic, so that a topic alert fires when one arrives.

After the alert fires

Read the failed records first. Many frameworks write the error reason into the record headers; a Kadeck pipe that sends failed records to a dead letter topic, for example, adds kadeck.pipe.error.stage and kadeck.pipe.error.message along with the source topic, partition and offset. Fix the cause before touching the records, otherwise they fail again. After that, the failed records can be inspected and repaired: a pipe can correct them and write them back to the original topic.

Questions

How do I monitor a Kafka dead letter queue?

Alert on records arriving in the DLQ topic. Either compare end offsets over time in your metrics system, or use a topic alert with the condition "more than 0 records in 1 minute".

Should I alert on consumer lag for a DLQ topic?

Usually not. Lag needs a consumer group, and many DLQs either have none or have one that runs only during reprocessing. Record arrival is the direct signal.

How do I stop a busy DLQ from flooding the channel?

Set a cooldown so the same alert cannot fire again right away, and raise the record count or widen the time window until normal background failures stay below the threshold.

Can I get DLQ alerts in Slack, Microsoft Teams or PagerDuty without writing a consumer?

Yes. A topic alert notifies Slack, Microsoft Teams, PagerDuty or email. Kadeck works with Apache Kafka, Redpanda, Amazon MSK and Aiven.

Topic data arrival alerts and their notifications are part of Kadeck Platform; they are not included in the Free or Desktop plans. Platform is priced per user, from 10 users, billed annually. You can get started here and find the details on the pricing page.