Kafka tools
The config that decides whether a message survives, and the bytes on the wire.
Kafka Confluent Wire Format Decoder
The five junk bytes in front of your payload
Decode the five bytes in front of a Kafka payload: the magic byte, the big-endian schema id, and where the Avro actually starts.
Kafka Key to Partition Mapper
Which partition does this key land on?
Work out which Kafka partition a key lands on. murmur2 over the UTF-8 bytes, and why adding partitions breaks ordering for existing keys.
Kafka Topic Name Validator
Legal, risky, or 249 characters too long?
Check Kafka topic names for the offending character, the 249 limit, and the dot-versus-underscore collision that breaks JMX metrics.
Kafka Replication Safety Checker
How many brokers can you lose
Work out exactly how many broker failures a topic survives before writes are refused and before an acknowledged write is lost, from replication.
Kafka Producer Config Linter
Will it start, and will it lose a record?
Check a Kafka producer config for the settings that lose records: acks=1, idempotence without acks=all, and silent batch reordering.
Kafka Message Payload Decoder
The first five bytes are usually not data
Decode a Kafka record payload from base64 or hex and find out what it actually is.
Kafka Connect Source Connector Generator
tasks.max is a ceiling, not a count
Generate the JSON for a Kafka Connect source connector, with the task count you will actually get rather than the one you asked for.
Kafka Connect Sink Connector Generator
A dead letter queue with no context headers is a pile of records
Generate the JSON for a Kafka Connect sink connector with a dead letter queue that is actually diagnosable.
Kafka Connect SMT Chain Builder
The order is the transforms list
Build a single message transform chain and see what each transform does to a record that does not match its expectations.
Kafka MirrorMaker 2 Config Generator
It renames every topic by default
Generate a MirrorMaker 2 configuration, and see what your topics are called on the other side.
Kafka Partition Reassignment Generator
The throttle is not optional
Generate the reassignment JSON and the three commands that go with it, including the throttle.
Strimzi Kafka Resource Generator
Without the cluster label, nothing happens
Generate Strimzi Kafka, KafkaTopic and KafkaUser resources. Without the strimzi.io/cluster label the operator ignores them silently.
Kafka mTLS Config Generator
The certificate is the identity
Generate the keytool and openssl commands for a Kafka mTLS client, plus the config, with no password in it.
Kafka Schema Registry Config Generator
The compatibility direction is your deployment order
Generate a Confluent Schema Registry configuration, with the compatibility level explained as what it actually decides: which side you can deploy first.
Kafka Exactly-Once Config Generator
Half of it is worse than none
Generate the complete set of configuration exactly-once needs: the broker settings, the producer, the consumer, and the part that cannot be configured at all.
Kafka Broker and KRaft Config Generator
The internal topics that break a one-broker cluster
Generate a Kafka server.properties, including the internal topic replication factors that stop a single-broker cluster working.
Kafka Quota Generator
Byte rates are per broker, not per cluster
Generate Kafka client quotas. Byte rates are per broker, not per cluster, so a 10 MB/s quota on six brokers permits 60 MB/s.
Kafka Streams Config Generator
application.id is four things at once
Generate a Kafka Streams config. application.id is the group id, the internal topic prefix and the state directory all at once.
Kafka Connect Worker Config Generator
Security three times, or the tasks fail
Generate a Kafka Connect worker config. Security has to be set three times, or the source and sink tasks fail to connect.
Kafka Retention and Unit Converter
log.retention.hours does not take milliseconds
Paste Kafka settings and see each duration and size converted, with the unit that setting actually takes.
Kafka Timestamp Converter
Two sentinels and two meanings
Convert Kafka record timestamps, which are milliseconds since the epoch, from whichever unit you have.
Kafka .properties to YAML Converter
Dotted keys stay flat
Turn a Kafka .properties file into YAML for a ConfigMap or Strimzi, with dotted keys kept flat because Kafka reads them literally.
Kafka Streams Internal Topic Predictor
Create them before Streams does
Predict the changelog and repartition topics a Kafka Streams application will create, from its application.
Kafka ACL Generator
The grant you forgot is on another resource type
Generate Kafka ACLs, including the grant people forget: a consumer needs READ on the group as well as on the topic, or it cannot join.
Kafka Topic Config Generator
min.insync.replicas is the one that matters
Generate Kafka topic config. min.insync.replicas is the setting that provides durability; the replication factor alone does not.
Kafka client.properties Generator
The file every CLI tool asks for
Generate the Kafka client.properties every CLI tool asks for, with SASL and TLS set correctly and no password written into the file.
Kafka Producer Config Generator
No password field, on purpose
Generate a Kafka producer config with no password field, on purpose. Covers acks, idempotence and the retry settings that reorder records.
Kafka Consumer Config Generator
The commit mode decides the semantics
Generate a Kafka consumer config where the commit mode decides your delivery semantics, and auto-commit quietly means at-most-once.
Kafka Disk and Retention Calculator
retention.bytes is per partition
Work out the disk a Kafka topic needs from its throughput, retention and replication factor, and the retention.
Kafka Partition Count Calculator
The number you can never reduce
Choose a partition count from target throughput, per-partition throughput and consumer group size, and see which of the three actually binds.
Kafka Cluster Sizing Calculator
The traffic no client metric shows
Size a Kafka cluster on network, disk and the replication floor at once, and see which binds.
Kafka Consumer Lag Catch-Up Calculator
Whether it ever clears, not just when
Work out how long a Kafka consumer backlog takes to clear, and whether it clears at all.
Kafka Producer Batching Calculator
linger.ms=0 still batches
Work out how large your producer batches actually get, and which of batch.
Kafka Segment and Index Sizing
Why retention.ms is a lower bound
Work out when a Kafka segment actually rolls, and why retention.ms is a lower bound rather than a promise about deletion.
Kafka Rebalance Duration Estimator
What a rolling restart really costs
How long a consumer group rebalance takes, split into detecting the departure, waiting for the slowest member to poll, and assigning.
Kafka Cost Estimator
Your rates, so nothing goes stale
What a Kafka cluster costs a month at rates you supply, split into storage, cross-zone replication and consumer egress.
Kafka Config Explorer by Version
The answer depends on the release
Look up a Kafka setting in a specific release: type, default, valid values and update mode, because defaults move between versions.
Kafka Default Config Reference
What moved under a config you never edited
Every Kafka default for a release you choose, filterable by name and importance, or narrowed to just the defaults that changed from the previous release.
Kafka OAuth Bearer Token Decoder
Will Kafka accept it, and can it refresh
Decode a Kafka SASL/OAUTHBEARER token and check the claims Kafka reads, including which one supplies the principal for ACLs.
Kafka Record Header Viewer
Headers are a list, not a map
Decode Kafka record headers and see what each costs on the wire, including the varint lengths the broker writes.
Kafka Topic Regex Subscription Tester
Kafka matches the whole name
Test a Kafka pattern subscription against real topic names. Kafka calls Matcher.matches(), so the pattern must match the whole name.
Kafka ACL Permission Matrix Viewer
DENY beats every ALLOW
Turn kafka-acls output into a principal-by-resource matrix, with denials kept separate because a DENY beats every ALLOW.
Kafka Connect Config Validator
The mistakes that raise no error
Check a Kafka Connect config for what Connect accepts and you did not mean, like a Filter with no predicate dropping every record.
Kafka Consumer Group Id Validator
Which broker coordinates the group
Work out which __consumer_offsets partition a group id hashes to, and therefore which broker coordinates it, using Kafka's own String.
Kafka Partition Assignment Visualizer
Leadership is the load, not replicas
See how partition replicas and leadership sit across your brokers, from a reassignment JSON or kafka-topics --describe output.
Kafka Consumer Assignment Visualizer
The three assignors disagree
Which partitions each consumer would get, under RangeAssignor, RoundRobinAssignor and CooperativeStickyAssignor, computed with Kafka's real algorithms.
Kafka ZooKeeper to KRaft Config Converter
The authorizer class nobody changes
Convert a Kafka ZooKeeper server.properties to KRaft, sorted into what carries across, what is renamed and what has no equivalent.
Kafka Config to Strimzi
Half of it belongs elsewhere
Turn a Kafka server.properties into Strimzi resources, splitting the settings the operator owns from the ones you still set.
Kafka Docker Compose Generator (KRaft)
Reachable from inside and outside
A Kafka KRaft Docker Compose file with the dual-listener setup that makes brokers reachable from inside the network and from your host.
Kafka JAAS Config Decoder
The line that stops SASL working
Decode a Kafka sasl.jaas.config value without printing the password back, and find the missing semicolon that stops SASL working.
Kafka CRC32C Calculator
Which CRC, over which bytes
Compute CRC32C and CRC32 over any bytes, or paste a whole record batch and have its stored checksum verified.
Kafka Config Upgrade Checker
What breaks when you upgrade Kafka
Pick the Kafka version you run and the one you are moving to, and see which settings are removed, renamed or change their default.
Kafka Config Diff
Which change actually changed something
Compare two Kafka configs by what they mean rather than by their text.
Kafka Consumer Config Linter
Why the group rebalances, and where the records went
Check a Kafka consumer config for the rebalance and offset problems: auto-commit losing records, and max.poll.interval.ms timing out.
Kafka Avro Schema Validator
The defaults Avro accepts and rejects
Check an Avro schema and get the exact path to what is wrong, not a bare message.
Kafka Schema Compatibility Checker
What the registry will say, before you ask it
Will the Schema Registry accept this Avro version? Reader and writer resolution worked through, before you try to register it.
Kafka Avro Schema Diff
Which direction each change breaks
Two Avro schemas compared as schemas rather than as text, with the direction each change breaks worked out from Avro's resolution rules.
Kafka Compression Comparison
Measured on your bytes
Measure real gzip, snappy and lz4 ratios on your own Kafka records, because compression ratio depends almost entirely on the data.
Kafka Delivery Semantics
Exactly-once has a consumer half
At-most-once, at-least-once and exactly-once, with the settings that actually produce each.
Kafka ksqlDB Query Builder
It looks like SQL and the rules are not
Build a ksqlDB statement with the rules attached to the choices that trigger them.
Kafka Connect SMT Predicate Tester
negate reads backwards
Describe a record, paste your predicate config, and get told which transforms actually run.
Kafka Streams Topology Viewer
Count the repartitions
Paste Kafka Streams topology.describe() output and see its shape. The number that matters is the sub-topology count, not the nodes.
Kafka Connect Pipeline Visualizer
The order things really run in
See a Kafka Connect pipeline in execution order, which is the transforms list rather than the order the properties appear in the file.
Kafka Protobuf Binary Decoder
Works without the .proto
Decode a protobuf payload with or without the .proto. The wire format carries a field number and a type, so every reading is shown.
Kafka Protobuf JSON Converter
Why your JSON does not round-trip
Check a JSON document against a .proto and get canonical proto3 back, including why int64 is encoded as a string rather than a number.
Kafka Protobuf to Avro Schema
What does not survive the conversion
Convert a .proto to an Avro schema and see what was lost. uint64 and fixed64 have no Avro type, and proto3 has no null.
Kafka Avro Binary Decoder
Wrong schema, no error
Decode an Avro payload against its schema, byte by byte with an offset per field.
Kafka Avro JSON Converter
Why the console producer rejects your line
Convert between the Avro JSON encoding and plain JSON, in either direction, against a schema.
Kafka Avro Sample Data Generator
Records that actually serialize
Generate sample records from an Avro schema, one JSON document per line, in either the plain form or the Avro JSON encoding a producer takes.
Kafka JSON to Avro Schema
What JSON cannot tell you
Infer an Avro schema from sample JSON, with what JSON cannot tell you named: int versus long, and which fields are really optional.
Kafka JSON Schema to Avro
What does not survive the conversion
Convert a JSON Schema to an Avro schema, with every keyword that has no Avro equivalent listed rather than dropped quietly.
Kafka SASL JAAS Generator
One login module, four syntaxes
The same Kafka SASL login module written four ways: client properties, a JAAS file, a broker KafkaServer stanza and Kubernetes.
Kafka CLI Command Builder
kcat is librdkafka, not Kafka
Build the exact kcat, kafka-topics.sh or kafka-consumer-groups.sh command. kcat is librdkafka, so the property names are not Kafka's.