Auto-commit on
The default that acknowledges messages the application has not finished processing
group.id=my-group enable.auto.commit=true max.poll.interval.ms=300000 auto.offset.reset=latest
Kafka Consumer Config Linter
Paste a consumer .properties file and get the rebalance, offset and delivery problems named with the exact line: the batch that cannot finish inside max.poll.interval.ms, the auto-commit that quietly gives you at-most-once, the default that skips your whole topic, and the assignor change that cannot be rolled out in one deploy. Validity first, then the analysis. It runs in this tab and nothing is uploaded, which matters for a file that usually carries a SASL password.
Paste below, or drop a file anywhere on this panel
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Lint. Nothing leaves this tab.
Formatted
Results
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Examples
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
No group.id
A consumer with no group, which cannot commit offsets or participate in a rebalance
enable.auto.commit=true max.poll.records=5000 session.timeout.ms=45000
Common mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Leaving enable.auto.commit at its default
It commits on a timer, so offsets advance for records the application has not finished processing. A crash then skips them, silently.
Instead:Commit manually after processing, or accept at-most-once and say so in the code.
Raising max.poll.records without raising max.poll.interval.ms
More records per poll means longer between polls. Exceed the interval and the broker considers the consumer dead and rebalances, mid-batch, repeatedly.
Instead:Size the two together: worst-case per-record time times max.poll.records must fit inside the interval.
Relying on auto.offset.reset as a policy
It only applies when there is no committed offset, so it fires on a brand new group or after retention removed the committed position. latest silently skips a backlog; earliest silently reprocesses everything.
Instead:Set it deliberately and monitor for it firing, because when it does something else has already gone wrong.
What it checks
Validity first: a unit suffix that Kafka rejects, a key one character off the real one and silently ignored, a duplicate that means the line you are reading is not the line in force. Then the six behaviours that decide whether this consumer keeps its partitions and keeps your records.
The rebalance that is really your for-loop
max.poll.interval.ms is a deadline on your code, not on the network, and it defaults to 300000. poll() hands you up to max.poll.records, which defaults to 500. That is 600 ms per record before the coordinator decides this member is gone, revokes its partitions and gives them away. Your code hears about it at the next commit, as CommitFailedException. The tool shows the division, and if you add a comment line reading # ms-per-record=25 it does the multiplication against your own number and tells you the largest safe max.poll.records.
Two timeouts, two entirely different failures
Heartbeats have run on a background thread since 0.10.1, which split liveness in two. session.timeout.ms catches a process that has died: the container was killed, the network went. max.poll.interval.ms catches a process that is alive and heartbeating happily while wedged inside your own code. Conflating them is the most common misreading in this whole area, and it is why raising the session timeout never fixes a slow-processing rebalance. The tool names which is which, checks the heartbeat is under a third of the session, and checks the session sits inside the broker's own group.min and group.max limits, because outside them the join fails with an error that quotes neither number.
At-most-once delivery, arrived at by accident
enable.auto.commit defaults to true. Auto-commit happens inside poll(), committing offsets for records earlier polls already returned, so a crash part way through a batch skips records that were fetched and never processed. No error, no lag, no retry: they are simply gone. Nearly everyone running this configuration believes they have at-least-once. This is the highest-value line in the file and it is usually the line that is not there.
The default that skips your entire topic
auto.offset.reset defaults to latest, so a brand-new group attaches to the end of the log and receives nothing that is already in it. The bug report reads my consumer receives no messages. earliest is the other direction and replays everything inside the retention window, which reads as a flood, and is what a typo in group.id produces because a misspelled group is a new group. The setting also applies when a committed offset has fallen off the log through retention, which is how a consumer that was down over a weekend loses its backlog.
cooperative-sticky takes two rolling restarts
Incremental rebalancing avoids the stop-the-world revoke, and you cannot switch to it in one deploy. Eager and cooperative members cannot share a group: if no assignor is common to every member the join fails with InconsistentGroupProtocolException and the group does not form, which takes the application down rather than slowing it. Phase one adds cooperative-sticky alongside the old assignor and rolls. Phase two removes the old one and rolls again. The tool tells you which phase the pasted file is, including the case where somebody stopped after phase one and has been paying for it since.
A credential in the file, named but never printed
An inline password in sasl.jaas.config, or an ssl.keystore.password, is reported as a fact about the file rather than about Kafka: that file gets committed, baked into images, mounted from ConfigMaps and pasted into tickets. The value is never echoed back, not in a finding, not in an excerpt, not in the exported report. The fix named is Kafka's own indirection, ${file:/path:key}, which lets the properties file reference the secret without containing it.
What it cannot see
This reads one file. It does not know your broker's group.min.session.timeout.ms, your topic's max.message.bytes, how many partitions this group is assigned, or how long your code takes per record, so where those matter it says which number to go and look up rather than guessing one. It also cannot see your commit calls: a consumer with enable.auto.commit=false that never commits looks identical here to one that commits correctly. For the running group, kafka-consumer-groups.sh --describe is the tool that answers what this page cannot.
How a consumer group actually behaves
Almost every consumer setting is about one of three questions: does the group still believe you are alive, where does reading start, and when is an offset written down. The failures are hard to debug because none of them produce an error where the cause is. A rebalance loop looks like a broker problem, a skipped topic looks like a broken producer, and lost records look like nothing at all.
Membership is a lease, and the lease has two clocks
One member of the group holds an assignment of partitions granted by the group coordinator, a broker. The assignment survives only as long as the coordinator believes the member is healthy, and since 0.10.1 it judges that in two independent ways. A background thread sends a heartbeat every heartbeat.interval.ms and the coordinator gives up after session.timeout.ms without one. Separately, the coordinator expects the application to call poll() again within max.poll.interval.ms, and gives up if it does not. The first clock catches a dead process. The second catches a live process stuck in your code, which is the case where heartbeats keep arriving and everything looks fine.
defaults, modern client
session.timeout.ms 45000 no heartbeat for this long -> member is dead
heartbeat.interval.ms 3000 a third of it, so two can be lost safely
max.poll.interval.ms 300000 no poll() for this long -> member is stuck
max.poll.records 500 records handed to you per poll()
the number nobody computes
300000 / 500 = 600 ms per record, for everything you do
with it, including the retryWhere the offset lives, and what a crash costs
A committed offset is a record in the internal __consumer_offsets topic, keyed by group, topic and partition. It is not where the consumer is reading: that is the position, which lives in memory and moves on every poll. The gap between the two is what a crash costs, and the setting that decides the gap is enable.auto.commit. On, the gap is one auto.commit.interval.ms of records committed possibly before you processed them, which loses records. Off and committing after processing, the gap is one batch reprocessed, which duplicates them. There is no configuration that gives exactly-once on its own: that needs read_committed plus a transactional producer writing the offsets in the same transaction as the output.
enable.auto.commit=true commit happens inside poll(), on a timer
poll -> 500 records
poll -> commits those 500 <- your code may still be on record 3
crash records 3..500 never processed, offsets gone
= at most once, silently
enable.auto.commit=false you commit after processing
poll -> 500 records
process all 500
commitSync()
crash before the commit the 500 arrive again on restart
= at least once, so make it idempotentauto.offset.reset only fires when there is no offset to use
It is not where the consumer starts. It is what happens when there is no committed offset for a partition, or when the committed offset no longer exists in the log. Three situations reach it: the first run of a new group, a partition added to a topic after the group started, and a group whose committed offset was deleted by retention because it was away too long. In all three the default, latest, discards whatever is sitting in the log, and none of them logs anything that looks like a problem.
group has a committed offset? -> resume there, this setting is ignored
no committed offset, and:
latest start at the end the backlog is skipped, silently
earliest start at the beginning the whole retained topic replays
none throw NoOffsetForPartitionException
new group + latest is the reason a working consumer receives nothing
a typo in group.id + earliest is the reason one suddenly floodsEager and cooperative rebalancing, and why the upgrade is two deploys
Under the eager protocol every member revokes every partition at the start of a rebalance and waits for a new assignment, so the whole group stops consuming for the duration. Cooperative rebalancing revokes only the partitions that are actually moving, so most members never stop. The catch is in how the assignor is chosen: the coordinator picks the most preferred assignor that every member supports, so a group holding both eager-only and cooperative-only members has no common choice and fails to form. That is why the change is two rolling restarts and never one. Since 3.0 the client default is [RangeAssignor, CooperativeStickyAssignor], which still resolves to Range and therefore to eager, so having the cooperative entry in the list is not the same as using it.
phase one, roll every member with both:
partition.assignment.strategy=\
org.apache.kafka.clients.consumer.CooperativeStickyAssignor,\
org.apache.kafka.clients.consumer.RangeAssignor
the group keeps using Range, because the members not yet
restarted support nothing else. nothing improves yet.
phase two, roll again with only:
partition.assignment.strategy=\
org.apache.kafka.clients.consumer.CooperativeStickyAssignor
now every member supports it, so it is chosen.
skipping phase one: InconsistentGroupProtocolException,
and the group does not form at allisolation.level, the half of exactly-once that is easy to miss
A transactional producer marks its records committed or aborted after the fact. Filtering the aborted ones out is the consumer's job, and the default does not do it: read_uncommitted returns everything in the log, including work that was explicitly rolled back. Setting read_committed also changes what the consumer can see at all, because it reads only up to the last stable offset. An open transaction on a partition therefore blocks progress on it until the producer commits or transaction.timeout.ms expires, and reported lag includes records the consumer is not yet allowed to read. Both of those are normal here and both get debugged as consumer problems.
read_uncommitted (default) every record, aborted ones included
read_committed up to the last stable offset only
symptom under read_committed: lag stops moving on one partition
cause: a producer died mid-transaction
fix: nothing on the consumer, wait for transaction.timeout.msMore kafka tools
Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka