acks=all does not wait for all replicas, and the ISR shrinks under load

acks=all is what you set when a message must not be lost. The name promises that every replica has the record before the producer is told the write succeeded.

It does not say that. It says every in-sync replica, and the size of that set is not a constant. Under exactly the conditions where durability matters, it shrinks.

The ISR is a moving target

A replica stays in the in-sync replica set while it keeps up with the leader, within replica.lag.time.max.ms (30 seconds by default). When one falls behind because of a GC pause, a slow disk, a network blip, or a rolling restart, the leader removes it from the ISR and carries on serving.

The write path does not stop. It now waits for the smaller set.

With replication.factor=3:

ISR size	What acks=all waits for	Effective durability
3	All three	Survives two broker losses
2	Two	Survives one broker loss
1	The leader alone	Same as acks=1

At ISR=1, acks=all has the durability of acks=1 with extra latency, and the producer is receiving successful acknowledgements the entire time. Lose that broker and the data is gone. Nothing in the producer’s view distinguishes this from the fully healthy case.

That is the whole problem: the guarantee degrades silently, and it degrades precisely when brokers are unhealthy, which is when you were relying on it.

The setting that means what acks=all sounds like

# broker or topic config, NOT producer
min.insync.replicas=2

This is where it gets missed. acks is a producer setting; min.insync.replicas is a broker or topic setting. The person configuring the producer often does not own the topic, and the two settings live in different repositories, different review queues, and sometimes different teams.

With it set to 2, a produce request using acks=all fails with NotEnoughReplicasException when the ISR drops below two. That is the behaviour people believe they configured: a write that cannot be made durable is rejected rather than silently accepted at lower durability.

The relationship is worth stating baldly:

• acks=all without min.insync.replicas is a guarantee that quietly degrades to nothing.
• min.insync.replicas without acks=all does nothing at all, because the producer is not waiting on the ISR in the first place.

They are one setting split across two files. Either alone is a false sense of security, and the second case is worse because the topic config looks correct in isolation.

Picking the number

With replication.factor=3, min.insync.replicas=2 is the standard choice. It tolerates one broker down while still accepting writes, and it guarantees an acknowledged write is on at least two brokers.

min.insync.replicas=3 with replication.factor=3 is a common overcorrection after an incident, and it is strictly worse. Any single broker being down stops writes entirely, because the ISR can never reach three. You have built a cluster that cannot tolerate a rolling restart, in exchange for no additional durability in the failure cases that actually occur.

The rule: min.insync.replicas must be less than replication.factor. If they are equal, routine maintenance is an outage.

For a topic where availability matters more than durability, RF=3 with min.insync=1 is a legitimate choice. It is only wrong when it is accidental.

How the failure shows up

Three shapes, and only the first is obvious:

NotEnoughReplicasException in producer logs. The system working correctly. Writes are being rejected because they cannot be made durable. Retryable, and the producer will retry, so it often appears as elevated latency rather than errors.

Under-replicated partitions on a dashboard, and nothing else. UnderReplicatedPartitions above zero means at least one partition has an ISR smaller than its replica count. If min.insync.replicas is not set, this is the only signal that your durability guarantee is currently weaker than you think. It is worth alerting on for that reason alone.

Silent data loss after a broker failure. No exception, no error, no metric at the time. Records acknowledged while ISR was 1, on a broker that then died before a follower caught up. You find out when a consumer reports a gap, and by then the evidence is gone.

The wrong instinct: turn up the retries

When produce requests start failing, the reflex is more retries and a longer timeout. It relieves the symptom and can create a worse problem:

max.in.flight.requests.per.connection=5
retries=2147483647
enable.idempotence=false

With more than one request in flight and retries enabled, a failed batch can be retried after a later batch has already succeeded. The partition ends up with records permanently out of order, and nothing reports it. Ordering is one of the guarantees people assume Kafka gives unconditionally; it does not, and this is how it is lost.

enable.idempotence=true fixes it, and it has been the default since Kafka 3.0. Which is exactly why it is worth checking rather than assuming: a config carried forward from a 2.x cluster may still set it explicitly to false, or may set max.in.flight.requests.per.connection above 5, which silently disables idempotence rather than raising an error.

Idempotence also gives you exactly-once semantics per partition for free at the producer level, deduplicating retried batches by sequence number. There is very little reason to have it off.

What changed recently

Three shifts that matter if your config predates them:

Kafka 3.0 flipped the producer defaults. enable.idempotence=true, acks=all, and max.in.flight.requests.per.connection=5 became the defaults together. A producer written against 2.x defaults and carried forward is now inconsistent with its own cluster’s expectations, and the explicit values in the config are often the old ones.

KRaft replaced ZooKeeper. ZooKeeper mode is removed in 4.0. This does not change acks semantics, but it does change where min.insync.replicas is administered and it changes the failure modes around controller election. If you are planning a migration, audit topic-level overrides before you move, because they are the settings most likely to be assumed rather than known.

Tiered storage changes the retention conversation, not the durability one. Worth saying because the two get conflated: offloading old segments to object storage does not make an unacknowledged write safe. The ISR arithmetic above is unchanged.

Fixing it on a running cluster

The order matters, because doing this in the wrong sequence causes an outage:

1. Audit before changing anything. For every topic that matters:

kafka-topics.sh --bootstrap-server broker:9092 --describe --topic orders

Read the Isr: list against the Replicas: list on each partition. If they already differ, you have an unhealthy cluster and setting min.insync.replicas now will start rejecting writes immediately.

2. Fix under-replication first. Get every partition to full ISR. Setting a durability floor on a cluster that cannot currently meet it converts a silent problem into a loud outage, which is the right end state and the wrong way to arrive at it.

3. Set min.insync.replicas per topic, not cluster-wide, at least initially:

kafka-configs.sh --bootstrap-server broker:9092 --alter \
  --entity-type topics --entity-name orders \
  --add-config min.insync.replicas=2

Per topic means the blast radius of a mistake is one topic. Roll it out starting with the ones where loss is unacceptable.

4. Then set the producer. acks=all and enable.idempotence=true.

5. Alert on UnderReplicatedPartitions and on NotEnoughReplicasException. The first tells you durability is degraded; the second tells you writes are being correctly refused. You want to know about both, and they mean different things.

When acks=all is the wrong choice

Not every topic needs this, and treating them all the same is its own kind of mistake:

High-volume telemetry and metrics. If losing a small fraction of records during a broker failure is acceptable, acks=1 is a legitimate and substantially faster choice. Durability has a throughput cost and paying it for data you would drop anyway is waste.

Anything already idempotent downstream. If your consumer deduplicates by key and reprocesses cleanly, the marginal value of the strongest producer guarantee falls sharply.

Genuinely latency-critical paths where a slow write is worse than a lost one. Rare, and real when it is real. Make it an explicit decision recorded somewhere, not a default nobody revisited.

The point is not that acks=all is always right. It is that when you choose it, you should be getting what you think you are choosing.

The short version

• acks=all waits for the in-sync replicas, and the ISR shrinks under load.
• At ISR=1 it is acks=1 with extra latency and no warning.
• min.insync.replicas is a broker/topic setting, and it is the other half of acks=all. Either alone is close to useless.
• min.insync=2 with RF=3. Setting it equal to RF makes a rolling restart an outage.
• acks=-1 and acks=all are the same thing, in case you are reading an old config.
• Check enable.idempotence, which is on by default since 3.0 and is often explicitly disabled in configs carried forward.
• Fix under-replication before setting a durability floor.

The Kafka producer config linter reports the acks and min.insync.replicas pairing, the reordering combinations, and the defaults that changed at 3.0, which are the ones most likely to be wrong in a config nobody has revisited since the upgrade.