Kafka Producer Batching Calculator
How large your batches actually get, which of batch.size, linger.ms or the in-flight limit ends them, and whether batch.size times your partition count still fits in buffer.memory.
kafka-batching.txt
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Believing linger.ms=0 disables batching
It does not. The producer still batches whatever has accumulated while the previous request was in flight, so throughput under load is far better than the setting suggests.
Instead:Raise linger.ms only if latency budget allows. At high throughput it changes little.
Raising batch.size without raising buffer.memory
batch.size is per partition. With many partitions the producer blocks on buffer.memory long before batches fill.
Instead:Size buffer.memory against batch.size times the partitions one producer writes to.
Assuming a larger batch always compresses better
It usually does, but the batch is capped by max.request.size and by the broker's max.message.bytes, and exceeding either fails the send.
Instead:Check all three limits together.
What batch.size and linger.ms actually do
Both are per partition, both are ceilings rather than targets, and the one that ends your batches is usually neither of them.
linger.ms=0 does not mean unbatched
This is the most common misreading of the producer. With linger.ms at 0 the sender thread sends a batch as soon as one is ready, but it can only keep max.in.flight.requests.per.connection requests outstanding at a time. Once it is saturated, the next batch accumulates for as long as it takes a slot to free, which is roughly a round trip divided by the in-flight limit. So batching happens anyway, and its size is decided by network latency rather than by anything you configured. The same code batches differently in a different network, and a producer that batches well in staging can batch poorly in production for no reason visible in the config.
batch.size is a cap, and a record bigger than it defeats it
A batch never exceeds batch.size, but a single record larger than batch.size is still sent, alone, in a batch of its own. At that point batching is off, linger.ms cannot help, and compression has one record to work with instead of a hundred. If your average record is anywhere near batch.size, that is the first number to change, and it is worth checking against the largest record rather than the average.
A request is per broker, not per partition
One ProduceRequest carries a batch for every partition on that broker that has data ready. That means request rate falls as partitions concentrate and rises as they spread, and it means counting a request per partition overstates the rate by the partitions-per-broker factor. The overstated figure is what makes people conclude they need more brokers than they do.
What this cannot see
It assumes records spread evenly across partitions, which a keyed topic with skewed keys does not do: a hot partition batches well and the rest batch badly, and this page shows you the average of that. It does not know your compression codec, your GC behaviour, or whether your broker is the bottleneck rather than your producer. The compression ratio you enter is applied to the batch, but a fuller batch really does compress better than a sparse one, so the figure moves with the batching you achieve.