Kafka Consumer Lag Catch-Up Calculator
How long a backlog takes to clear, and whether it clears at all. A backlog drains from the surplus between the two rates, not from the consume rate, which is why the usual estimate is optimistic.
kafka-consumer-lag.txt
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Reading lag as a duration
Lag is a count of records. Whether it is a second or an hour depends on the consume rate, which the number does not carry.
Instead:Divide by the consumption rate. That gives time to drain, which is the useful figure.
Alerting on a lag threshold
A constant lag of 10,000 on a fast topic is healthy. A lag of 100 that never clears is not. The absolute number says little.
Instead:Alert on the trend and on time-to-drain, not the raw count.
Ignoring whether the lag ever clears
Lag that grows during peak and drains overnight is capacity working as designed. Lag that never returns to zero means the consumer is permanently slower than the producer.
Instead:Check the shape over a full cycle before adding consumers.
Why consumer lag clears more slowly than expected
The arithmetic is one subtraction, and the subtraction is the part that gets skipped.
Only the surplus drains the backlog
A consumer group spends its capacity on two things: keeping up with what is arriving now, and working through what is already there. The first has first call on it. What remains, the difference between the consume rate and the produce rate, is the only rate the backlog shrinks at. When that difference is small the catch-up time is enormous even though the group looks fast, and when it is zero or negative there is no catch-up at all.
A thin surplus makes the estimate fragile
If consumers manage 5% more than the arrival rate, the backlog clears at 5% of the consume rate and a modest traffic increase during the catch-up turns a finite answer into never. This is why the useful question is not how long it will take but how much margin there is, and it is why catch-up usually happens overnight rather than during the day.
Consumers past the partition count do nothing
Partition assignment gives each partition to exactly one consumer in a group, so a group larger than the partition count has idle members. Scaling out a lagging consumer group is the first instinct and it stops working precisely at the partition count, which is often lower than the number of pods someone has just added.
Lag can fall because data was deleted
Lag measures the gap between the committed offset and the end of the log, and retention moves the start of the log forward. A consumer far enough behind has its unread records deleted, the gap closes, and the metric recovers with nothing having been processed. The current delay figure on this page is what to compare against your retention: if it is longer, this is already happening.
What this cannot see
It assumes both rates hold steady, and real traffic has a daily shape, so if the produce rate has an overnight trough the true answer is better than this one. It also cannot know whether your consumer is slow because of processing, because of a rebalance loop, or because it is being throttled by a quota, and those have different fixes. For whether the consumer configuration itself is the problem, the consumer config linter on this site reads the rebalance budget and the poll timings.