Kafka CLI Command Builder

The exact command for kcat, kafka-topics.sh, kafka-consumer-groups.sh and kafka-configs.sh, including the librdkafka property names that are not Kafka's.

Tool

One page for four tools, because the flags differ and the questions do not.

kcat

kcat is a librdkafka client, so its property names are not Kafka's. That difference is most of what goes wrong with it.

Adds a -f format string. Without one kcat prints the value and nothing else, which is rarely what you want when debugging.

Without it kcat waits for more messages forever, which in a script is a hang rather than an empty result.

kafka-topics.sh
kafka-consumer-groups.sh

Off means --dry-run. A reset with neither flag prints what it would do and changes nothing, which is a good default and does surprise people.

kafka-configs.sh
Topic
Security

No password field. The Java tools read a properties file, and kcat takes its credential from an environment variable rather than the command line, so it does not end up in your shell history.

command.sh

updates as you type

    Common mistakes

    These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

    1. Expecting Java client property names to work

      kcat is built on librdkafka, which has its own names. max.poll.records and several others simply do not exist.

      Instead:Use the librdkafka names. The overlap is partial and the differences are silent.

    2. Omitting -e when consuming a finite set

      Without it kcat waits for more messages forever, which in a script hangs the pipeline.

      Instead:Add -e to exit at the end of the partition.

    3. Using -X for a property that needs a prefix

      Consumer and producer specific properties need the right prefix in some contexts, and a wrong one is accepted and ignored.

      Instead:Check with -X list, which prints what librdkafka actually recognises.

    kcat is a librdkafka client, and that is most of what goes wrong

    The flags are the easy half. The half that costs an afternoon is that kcat does not read Kafka's configuration language, and half of a working client.properties transfers into it while the other half is silently ignored.

    There is no sasl.jaas.config in librdkafka

    The credential is two separate properties, sasl.username and sasl.password, passed with -X. There is no truststore either: librdkafka uses OpenSSL, so it is ssl.ca.location pointing at a PEM file, and a .jks there fails with an error about the file format. What makes this genuinely confusing is that security.protocol and sasl.mechanism do have the same names, so a config that half works looks like a credential problem rather than an ignored setting.

    kcat -C -b broker:9093 -t orders \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanism=SCRAM-SHA-512 \
  -X sasl.username=app \
  -X sasl.password="$KAFKA_PASSWORD"

    kcat's default partitioner is not Java's

    librdkafka uses consistent-random CRC32 by default and the Java client uses murmur2, so the same key lands on a different partition depending on which produced it. Producing test data with kcat into a topic whose ordering matters will interleave it with the real records in a way that looks like a broker fault. Set -X partitioner=murmur2_random to match Java. The key to partition mapper on this site shows what each one gives for a key.

    The config flag is different on every Java tool

    kafka-topics.sh, kafka-configs.sh, kafka-acls.sh and kafka-consumer-groups.sh take --command-config. kafka-console-consumer.sh takes --consumer.config and kafka-console-producer.sh takes --producer.config. Passing the wrong one is accepted and ignored by some versions, and the tool then connects with no security at all and times out against a listener that will not talk to it. The error says nothing about the flag.

    A reset does nothing without --execute, and that is the good news

    kafka-consumer-groups.sh --reset-offsets requires either --dry-run or --execute, and with neither it prints its usage and exits, which reads as a broken command rather than as a safeguard. It also only works when the group has no active members: with a consumer still running it fails, and the message is about the group being active. --to-datetime takes an ISO-8601 string with no zone and reads it in the JVM's local time zone, which is the quietest way to land somewhere unintended.

    Increasing partitions is not reversible and moves your keys

    kafka-topics.sh --alter --partitions can only increase the count, and increasing it changes which partition a key hashes to. Every key produced after the change lands somewhere new, so per-key ordering is broken across the boundary and a compacted topic ends up with two live versions of the same key on different partitions. There is no way to reduce the count afterwards.

    The password is never on the command line here

    kcat commands read it from an environment variable, because a command line ends up in your shell history and in ps output for every process on the host. The Java tools read it from a properties file instead, and the client.properties generator on this site emits that file without a password in it either. Nothing on this page is sent anywhere: the command is built in your browser.

    More kafka tools

    Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes

    Elsewhere on the site

    Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer