Kafka Exactly-Once Config Generator
Every piece exactly-once needs, including the one that is code rather than configuration. Any single piece missing gives at-least-once while everything still appears to work.
exactly-once.properties
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Configuring only the producer half
Transactions without read_committed consumers means aborted records are still read. Half of exactly-once is worse than none, because it costs the throughput and delivers nothing.
Instead:Configure both sides, or do not claim the guarantee.
Sharing one transactional.id across instances
It fences the previous holder, so two instances with the same id continuously fence each other and neither makes progress.
Instead:One stable transactional.id per instance, stable across restarts.
Setting transaction.timeout.ms above the broker's maximum
transaction.max.timeout.ms on the broker caps it, and a producer asking for more is rejected at initialisation.
Instead:Keep the producer value at or below the broker maximum.
Exactly-once is four things, and configuration is only three of them
The broker, the producer, the consumer, and the code that ties the read to the write.
The transaction state topic is where the guarantee lives
transaction.state.log.replication.factor and its min ISR decide whether a transaction survives a broker failure. At replication factor 1 the transaction log has one copy, so losing that broker loses the record of which transactions committed. Exactly-once on a cluster smaller than three brokers is a development configuration.
read_committed is not optional and is easy to forget
Without it the consumer reads records from aborted transactions, which is the single most common way a transactional pipeline turns out not to be exactly-once. Everything else can be configured perfectly and this one default undoes it, because read_uncommitted is the default.
The offsets have to be committed inside the transaction
This is the part that is not a setting. sendOffsetsToTransaction makes the read and the write atomic; committing through the consumer, with auto-commit or commitSync, makes them two operations and a crash between them duplicates or loses work. Every property in the generated files can be correct while the code makes the guarantee impossible.
Streams handles it, at a cost in transaction volume
processing.guarantee=exactly_once_v2 is genuinely one line, because Streams manages the transaction and the offset commit itself. The cost is the commit interval: 100 ms under exactly-once against 30 s otherwise, which is a three hundredfold increase in transactions on the brokers and the bound on the topology's end-to-end latency.
It covers Kafka and nothing else
Exactly-once means reading from Kafka and writing to Kafka atomically. A transform that calls an external service still calls it twice on a retry, because that call is not in the transaction. A pipeline described as exactly-once that writes to a database outside the transaction is at-least-once with extra steps.