Kafka Docker Compose Generator (KRaft)

A KRaft Docker Compose file with no ZooKeeper, and with the dual listener setup that makes it reachable from both another container and your own machine. That addressing is the single most common reason a Compose Kafka does not work.

Cluster

A KRaft cluster with no ZooKeeper. Three brokers is the smallest that lets you set a replication factor of 3.

Fine for a laptop. For anything real, controllers belong on their own nodes.

Ports and storage

Without these the cluster loses everything on docker compose down, including the formatted metadata.

Convenience

Off is the production-shaped choice: an auto-created topic gets default partitions and replication factor with nobody deciding.

Adds kafka-ui on port 8080, reachable at localhost:8080 and talking to the brokers over the internal listener.

docker-compose.yml

updates as you type

    Common mistakes

    These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

    1. Advertising a single listener

      A broker reachable from inside the compose network is not reachable from the host, because the advertised host differs. Clients connect, get the advertised address and fail on the second call.

      Instead:Define two listeners with different advertised addresses, one internal and one for the host.

    2. Leaving internal topic replication at the default

      __consumer_offsets defaults to a replication factor of three and cannot be created on a one-broker cluster.

      Instead:Set the offsets and transaction log replication factors to 1.

    3. Using the container name as the advertised host for external clients

      It resolves inside the compose network and nowhere else, so anything on the host fails with an unresolvable address.

      Instead:Advertise localhost with the mapped port for the external listener.

    Why your Compose Kafka works from one place and not the other

    One broker cannot advertise one address that works both inside and outside Docker. That single fact is behind almost every broken Kafka Compose file.

    A container and your laptop need different addresses

    A client discovers brokers through metadata, and the broker tells it whatever advertised.listeners says. From another container the right answer is kafka-1:9092, resolved on the Compose network. From your machine it is localhost:9092, because that is where the published port lands. There is no single string that is both, so a broker needs two listeners with two advertised addresses. With one listener the cluster works from exactly one of those two places and fails from the other, and the failure looks like a broker problem: the client connects to the bootstrap address, gets metadata pointing somewhere it cannot reach, and times out.

    KAFKA_LISTENERS: INTERNAL://:9092,HOST://:9092
KAFKA_ADVERTISED_LISTENERS:
  INTERNAL://kafka-1:9092,HOST://localhost:9092

# from a container:  kafka-1:9092
# from your machine: localhost:9092
# both work, because the broker advertises both

    Internal topic replication has to fit the broker count

    offsets.topic.replication.factor defaults to 3, so on a one or two broker Compose cluster __consumer_offsets cannot be created. The cluster starts, looks healthy, and the first consumer group fails to commit. This generator sets those factors to match the broker count, which means a config that works on one broker will not transfer to a real cluster unchanged, and the output says so.

    No ZooKeeper, and no separate format step

    The official apache/kafka image runs in KRaft mode and formats its log directory on first start when CLUSTER_ID is set, so there is no kafka-storage command to run. Confluent's cp-kafka uses different environment variable names and its own setup, so a compose file for one is not a compose file for the other. Every node in the cluster needs the same cluster id, and it should be a fresh one from kafka-storage.sh random-uuid rather than a copied example.

    Without volumes, down means gone

    KRaft keeps the cluster metadata in the log directory alongside the data, so losing that directory loses the formatted state as well as the messages. Without named volumes, docker compose down leaves you recreating the cluster rather than restarting it, which is fine for a throwaway and annoying in a test suite that expects to stop and start.

    Two settings here are deliberately wrong for production

    group.initial.rebalance.delay.ms is set to 0, which on a real cluster wastes rebalances during a fleet start and on a laptop just removes a three second wait from every test. Combined broker and controller mode puts the quorum on the nodes serving traffic, which is fine locally and not how a production cluster should be laid out. Both are marked in the output rather than left to be discovered when the file gets copied somewhere real.

    More kafka tools

    Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka

    Elsewhere on the site

    Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer