Idempotence without acks=all
A combination Kafka rejects at startup, with the reason
enable.idempotence=true acks=1
Kafka Producer Config Linter
Paste a producer .properties file
and get the durability, ordering and throughput problems named with the exact
line: the settings that stop the client starting, the reordering that happens
with no error at all, and the defaults that changed in Kafka 3.0 so that the
same file now means two different things.
Paste below, or drop a file anywhere on this panel
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Lint. Nothing leaves this tab.
Formatted
Results
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Examples
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
acks=1
Acknowledged by the leader alone, so a leader failure loses the write
acks=1 retries=2147483647 enable.idempotence=false
Common mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Enabling idempotence with acks=1
Kafka rejects the combination at startup. Idempotence requires acks=all, because the guarantee depends on the full ISR acknowledging.
Instead:Set acks=all, or turn idempotence off deliberately.
Using acks=1 for data that matters
The leader acknowledges before replicating. A leader failure immediately after the ack loses the write with no error anywhere.
Instead:acks=all with min.insync.replicas=2 on a replication factor of 3.
Setting retries high without max.in.flight bounded
With retries and more than one in-flight request, a retried batch can be written after a later one, silently reordering records within a partition.
Instead:Enable idempotence, which handles this, or set max.in.flight.requests.per.connection=1.
What it checks
Validity first: whether the client starts at all. Then the pitfalls, which are almost all relationships between two settings rather than one wrong value.
The defaults changed under you in 3.0
enable.idempotence became true by default, and it pulls acks to all and retries to Integer.MAX_VALUE with it. So an unspecified acks means 1 on a 2.x client and all on a 3.x one, and the same file describes two different durability profiles. Where the answer depends on the client version, the finding says so instead of picking one.
Startup failures, separated from opinions
enable.idempotence=true with acks=1 is not a warning: the KafkaProducer constructor throws ConfigException and the process does not start. Same for retries=0 with idempotence, for more than five requests in flight with it, and for a delivery.timeout.ms below linger.ms plus request.timeout.ms. Those come first, before anything about durability.
The config that starts on 2.8 and refuses to start on 3.0
max.in.flight.requests.per.connection above 5 is rejected whenever idempotence is on, and from 3.0 it is on unless you turned it off. Nothing else has to change for a working producer to stop starting after a client upgrade, and the exception names the in-flight setting rather than the default that turned on.
Silent reordering, which nobody expects
Without idempotence, more than one request in flight plus a retry reorders records within a partition: batch one fails, batch two lands, batch one is retried behind it. Partitioning by key does not save you. It guarantees the records go to the same partition, not the order they arrive in it, and there is no error and no log line when it happens.
retries is not the bound, delivery.timeout.ms is
Since 2.1 the producer gives up when delivery.timeout.ms expires, whichever comes first, and that clock covers waiting for metadata and sitting in the accumulator as well as retries. So retries=0 does not make a send fail fast, a large retries value does not mean the attempts happen, and lowering retries no longer shortens the time to a failed callback.
Throughput settings that cancel each other
batch.size is a ceiling, not a target, so raising it while linger.ms stays at 0 changes nothing: the producer sends as soon as a sender thread is free. Compression works per batch for the same reason, so linger.ms=0 also weakens it. Both are reported as the pair they are rather than as two independent settings.
send() is not as asynchronous as it reads
It returns a Future, so it looks non-blocking. When buffer.memory is full or the topic metadata has not arrived, the calling thread blocks for up to max.block.ms, 60 seconds by default, and then throws TimeoutException. On a request-handling thread pool that turns producer backpressure into a stalled service.
The limits that live on the broker
max.request.size is checked against the broker's message.max.bytes and the topic's max.message.bytes, and only the client side is in this file. Raising one of the three turns a local rejection into a RecordTooLargeException from the broker that looks like a client bug. Those settings are named where they matter rather than checked, because a config file cannot show them.
Your producer config has a password in it. It stays in this tab.
A real producer file usually carries
sasl.jaas.config with an
inline password, or a keystore password, or both. That is exactly the
kind of file nobody should paste into a stranger's backend, so this
page does not have one. There is no network request of any kind: the
parser and every rule are JavaScript in your browser, the file is
never uploaded, and nothing is written to storage. Close the tab and
there is nothing left to delete.
The credential is reported, because a password sitting in a file that is probably in git is worth knowing about. It is reported without its value: not in the finding, not in the copied report, and not in the source excerpt, which is why the excerpts here show a single line rather than the usual few lines of context. A tool that prints the password back to prove it found one has just copied it into the page, the clipboard and any screenshot of the result.
If a credential is in the file and the file has been committed anywhere, deleting the line is not the fix. Rotate the credential, and inject it at startup from a secret store or the environment instead.
What a producer config actually decides
Fifteen lines of properties decide whether a record survives a broker failure, whether two records for the same key stay in order, and how many requests per second your cluster has to serve for the same bytes. None of that is visible in the file, and most of it is a relationship between two settings rather than a value in one.
acks is a count of copies, not a boolean
acks=0 means the producer treats a successful socket write as success: it does not wait for the leader, and the send callback has nothing to report because no answer was expected. acks=1 waits for the leader's own log and no further, so a leader failure in the replication window loses a record that was already reported as sent. acks=all waits for every replica currently in sync, which is where min.insync.replicas comes in: it is a broker or topic setting, and at its default of 1 a lone in-sync leader satisfies acks=all on its own.
acks=0 leader never asked loss is invisible
acks=1 leader's log only loss on leader failure
acks=all every in-sync replica loss only if the ISR shrank to one
acks=all + min.insync.replicas=1 -> one copy can be enough
acks=all + min.insync.replicas=2 -> writes fail rather than under-replicate
replication.factor=3 keeps them possibleThe 3.0 idempotence default, and how it fails quietly
From Kafka 3.0 enable.idempotence defaults to true, which implies acks=all and retries=Integer.MAX_VALUE. The validation that follows behaves differently depending on whether you wrote enable.idempotence yourself. If you did, a contradiction throws. If you did not, the client backs down: it turns idempotence off and logs one INFO line. That second path is why a config can be upgraded to 3.x, keep working, and quietly not have the guarantee the upgrade advertised.
enable.idempotence=true
acks=1 ConfigException, the producer does not start
acks=1
(enable.idempotence unset) 3.0+: idempotence turned off, INFO log, starts
2.x : already off, nothing to log
max.in.flight.requests.per.connection=10
(enable.idempotence unset) 3.0+: ConfigException, does not start
2.x : starts, and can reorder on retryOrdering is a property of the retry path
Kafka guarantees the order of a partition's log, not the order your sends reach it. With several requests in flight to one partition and a retry, the failed batch is re-sent after the batches behind it were already accepted, and the log order is now different from the send order. The idempotent producer fixes this properly: each batch carries a sequence number and the broker refuses one that arrives out of sequence, which is why five in flight is safe with idempotence and unsafe without it.
enable.idempotence=false
max.in.flight.requests.per.connection=5
send A -> in flight A fails, will be retried
send B -> in flight B is accepted, appended
retry A A is appended after B
log order: B, A same key, wrong order, no error anywhere
enable.idempotence=true
max.in.flight.requests.per.connection=5 order preserved by sequence numberdelivery.timeout.ms is the clock that matters
KIP-91 replaced the retry count with a deadline. delivery.timeout.ms, 120000 by default, bounds the whole time from send() returning to the callback firing: waiting for metadata, sitting in the accumulator, every attempt and every backoff. request.timeout.ms bounds one attempt inside that. The client also validates the relationship, and how it validates depends on whether you set the deadline yourself.
delivery.timeout.ms >= linger.ms + request.timeout.ms
set explicitly and smaller ConfigException at construction
left unset and smaller raised for you to the sum, WARN logged
retries=0 with the default deadline
a callback can still take up to 120 s, for example while the
partition leader is unknown. "fail fast" is delivery.timeout.ms,
not retries.Batching, compression and the request rate
The producer accumulates records per partition and sends a batch when it reaches batch.size or when linger.ms elapses, whichever comes first. At linger.ms=0 there is no waiting, so under anything short of saturation the batches are tiny: the byte count is the same and the request count is many times higher, which is broker CPU rather than bandwidth. Compression is applied per batch and the brokers store the batch compressed, so a batch of one record compresses to roughly nothing saved.
linger.ms=0 batch.size=1048576 batches stay small anyway
linger.ms=20 batch.size=65536 up to 20 ms of records per request
compression.type=none the default, and rarely the right answer for text
compression.type=lz4 cheap CPU, works everywhere
compression.type=zstd best ratio, needs brokers and consumers on 2.1+Where a producer blocks, and where it throws
buffer.memory, 32 MiB by default, is the pool records wait in before they are sent. When it is full, send() blocks the calling thread for up to max.block.ms and then throws TimeoutException, and the same happens while the metadata for a topic is still being fetched. Everything after that point is asynchronous and reaches you through the callback, which is why a failed send and a failed delivery are two different pieces of error handling.
send() -> metadata known? no: block, up to max.block.ms
-> buffer space? no: block, up to max.block.ms
-> appended to a batch returns a Future here
-> sent, retried, timed out callback, up to delivery.timeout.ms
max.block.ms default 60000 a request thread can stall for a minute
buffer.memory default 33554432 must exceed batch.size and max.request.sizeMore kafka tools
Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka