Kafka Delivery Semantics

The consumer commits offsets on a timer, every auto.commit.interval.ms, whether or not the records in that range have been processed. If the process dies after a commit and before the work finishes, those records are never reprocessed. They are simply gone, and lag reads zero throughout.

What to do: If losing records is not acceptable, set enable.auto.commit=false and commit after the work has actually succeeded. This is the single most common correctness bug in Kafka consumers, and it is invisible in every metric.

enable.auto.commit=false
# then, only after the work succeeded:
consumer.commitSync();

The producer does not wait for any acknowledgement. A record lost in the network, or arriving at a broker that is not the leader, is never retried because the producer never learns. Throughput is the highest available and the loss is silent.

What to do: Only correct where loss genuinely does not matter, which in practice means metrics and sampled telemetry. Note that acks=0 is incompatible with idempotence and the producer fails to start if both are set.

The producer waits for min.insync.replicas to acknowledge, and retries on a retriable error. A retry after an acknowledgement that was sent but not received produces a duplicate, which is the whole reason this is not exactly-once.

What to do: acks=all is the default since 3.0. Set min.insync.replicas=2 on a replication-factor-3 topic: with the default of 1, acks=all waits for the leader alone and a leader failure still loses acknowledged data.

# topic side, and this is the half people forget
kafka-configs.sh --bootstrap-server broker:9092 \
  --alter --entity-type topics --entity-name orders \
  --add-config min.insync.replicas=2

Process the batch, then commit. A crash between the work and the commit means the batch is processed again on restart, so downstream has to tolerate a repeat. That tolerance is where the real design work is.

What to do: Make the downstream effect idempotent: an upsert keyed on something in the record, or a de-duplication table keyed on topic, partition and offset. This is more robust than transactions and works with systems that have no transaction support at all.

Since 2.1 this is the total time a send may take, retries included, and it supersedes reasoning about retries and retry.backoff.ms separately. When it expires the send fails permanently whatever retries is set to.

What to do: Set delivery.timeout.ms to how long the application can tolerate a send taking, and leave retries at its default of Integer.MAX_VALUE. Tuning retries directly is the old way and gives a bound nobody can compute.

delivery.timeout.ms=120000
request.timeout.ms=30000
retries=2147483647

Deduplicates retries within one producer session, per partition. The producer gets a producer id and a sequence number per partition, and the broker rejects a duplicate sequence. This removes duplicates caused by retries and nothing else. It does NOT survive a producer restart: a new session gets a new producer id.

What to do: It is the default since 3.0 and costs almost nothing, so leave it on. Note that it requires acks=all, max.in.flight.requests.per.connection at most 5, and retries above 0, and the producer refuses to start if you contradict any of them.

enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5

Makes a set of writes across partitions atomic, and ties the consumer's offset commit into the same transaction. This is what makes consume-transform-produce exactly-once: either the output records and the offset commit both land, or neither does. It survives a restart, because the transactional.id lets a new instance fence the old one.

What to do: The transactional.id must be STABLE per logical producer instance across restarts. Generating a fresh one at startup, which is a very common mistake, gives none of the guarantee and leaks a producer id per restart into the transaction state topic.

transactional.id=payments-processor-${POD_ORDINAL}

producer.initTransactions();
producer.beginTransaction();
producer.send(...);
producer.sendOffsetsToTransaction(offsets, groupMetadata);
producer.commitTransaction();

The default is read_uncommitted, which means a consumer reads records from transactions that were later aborted. A producer using transactions while its consumer uses the default gets no benefit at all: the whole apparatus is running and the reader still sees aborted data.

What to do: Set isolation.level=read_committed on every consumer downstream of a transactional producer. This is the single most commonly missed setting in an exactly-once pipeline, and the pipeline appears to work without it.

# on EVERY consumer downstream of a transactional producer
isolation.level=read_committed

Exactly-once is a property of the Kafka-to-Kafka path. The moment a side effect leaves Kafka, an email, an HTTP call, a write to a database without a shared transaction, the guarantee stops at the boundary. Kafka cannot roll back an email.

What to do: For an external side effect, make it idempotent rather than transactional. A de-duplication key derived from the record is more robust than any distributed transaction, and it works with systems that offer none.

False before 3.0, true from 3.0. A producer that was plain at-least-once becomes idempotent on a client upgrade with no config change, which is a strict improvement and worth knowing about.

What to do: The trap is the interaction below: setting acks=1 explicitly, which was harmless before, now conflicts with the new default and silently turns idempotence off, or fails startup depending on the version.

The default was 1 and is now all. A producer relying on the old default is now slower and safer, which is usually welcome and occasionally a surprise in a latency-sensitive path.

What to do: If you set acks=1 explicitly to get the old behaviour back, you have also disabled idempotence, because idempotence requires acks=all. Set enable.idempotence=false deliberately in that case rather than leaving the combination to be resolved for you.

# these two are the same statement on 3.0+
acks=1
enable.idempotence=false

Idempotence supports at most 5 in-flight requests per connection. Above that the producer cannot guarantee ordering on a retry, so it refuses to start rather than silently reordering.

What to do: Leave it at 5. Raising it for throughput trades away both ordering and idempotence, which is almost never the trade somebody intends to make.

Kafka guarantees order within a partition and offers nothing across partitions. Records for one key are ordered only because the default partitioner sends one key to one partition, so ordering per key depends on the key being set and the partition count not changing.

What to do: Increasing a topic's partition count changes the key-to-partition mapping, so records for one key exist either side of the change and their relative order across the boundary is not preserved. Plan the partition count before ordering matters, not after.

With idempotence off and more than one request in flight, a failed first request that succeeds on retry lands after a second request that succeeded first. The records are written out of order and nothing reports it.

What to do: Idempotence prevents this, which is another reason to leave the 3.0 default alone. Before 3.0 the equivalent was max.in.flight.requests.per.connection=1, which costs a great deal of throughput.