Kafka Topic Name Validator

Paste topic names, one per line, and find out which the broker will reject, which it will accept and make you regret, and exactly which character is at fault. The limit is 249 rather than 255 for a reason worth knowing, and two legal names that differ only by a dot against an underscore report one set of metrics between them.

Paste below, or drop a file anywhere on this panel Upload a file

Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.

The answer appears here

Paste on the left and press Validate. Nothing leaves this tab.

Formatted

Results

Examples

Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.

Dot and underscore collision

Kafka's metric names treat . and _ as the same character, so these two topics collide

my.topic_name
my_topic.name

Over the length limit

The 249 character ceiling, which exists because the topic name becomes a directory

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Common mistakes

These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

Mixing dots and underscores in topic names
Kafka's metric names replace both with the same character, so my.topic_name and my_topic.name collide in JMX and one silently overwrites the other's metrics.
Instead:Pick one separator for the whole cluster and stay with it.
Naming a topic after a path
The topic name becomes a directory name on every broker, so length and characters are constrained by the filesystem as well as by Kafka.
Instead:Keep names short and flat. 249 characters is the hard ceiling and far shorter is better.
Using a name that differs only by case
Topic names are case sensitive to Kafka but the log directories may not be on a case-insensitive filesystem.
Instead:Use lowercase throughout.

What it checks

The four rules the broker applies, and the five things it accepts and then makes you live with. Both halves are decidable from the name, which is why neither needs a cluster.

The alphabet, and the two things it allows that you expect it to reject

A topic name is ASCII letters, digits, ".", "_" and "-" and nothing else, so a slash, a colon, a space or an accented letter is refused with InvalidTopicException. Each one is reported with a caret under the exact character, and a character outside ASCII is reported with its code point, because an emoji or a non-breaking space looks like nothing in a terminal. Two things are legal and surprise people: uppercase, so Orders and orders are two separate topics that no tool will ever tell you about, and a leading hyphen, which is legal to the broker and unusable from a shell, since kafka-topics.sh --topic -orders reads -orders as an option.

249 characters, which is a filesystem limit wearing a protocol's clothes

Not 255, not 256, and nothing in the protocol says 249. The name becomes a directory: every partition is a directory called <topic>-<partition> under log.dirs, and one path component stops at 255 bytes on ext4, xfs and NTFS alike. 249 is what Kafka keeps under that for the hyphen and the partition number, and it needs the room, since a deleted log directory is renamed <topic>-<partition>.<32 hex characters>-delete. Both sides of the boundary are checked here, and the caret points at character 250, which is where to cut.

A "." and a "_" are the same character to Kafka's metrics

This is the finding worth the page. A metric name is a dotted hierarchy, so Kafka replaces "." with "_" when it puts a topic name into one, which means my.topic and my_topic report a single set of metrics between them: bytes in, bytes out, messages in and every lag figure derived from them, added together, with nothing downstream able to separate them again. kafka-topics.sh prints a warning about it on every create and it scrolls past. Paste both names and the collision is reported outright, with the metric name they share. Paste one and the risk is reported at low severity, because the name is legal and the second name is the accident.

The two names Kafka refuses outright, and the one it should

An empty name is rejected before anything else is checked, and "." and ".." are rejected by name rather than by character, because they are directory references and the name is about to become a directory. Kafka checks nothing else about dots, so "..." is a perfectly valid topic name: the log directory is "...-0", which ls does not show and a glob does not match, and it reports its metrics as "___". That one is reported here as legal and a bad idea, which is what it is.

The topics that already exist, and the prefix that says so

__consumer_offsets holds every consumer group's committed offsets, __transaction_state is the transaction coordinator's own log, and on KRaft __cluster_metadata is the metadata log the cluster is described in. All three are legal names and none of them are available. A leading double underscore on a topic of your own is legal too, and costs you something quieter: much of the tooling around Kafka filters on the prefix rather than asking the broker which topics are internal, so the topic works perfectly and goes missing from listings, dashboards and retention audits.

Legal here, and too long by the time Kafka Streams has finished with it

Streams names its internal topics <application.id>-<name>-changelog and <application.id>-<name>-repartition, where <name> is a store or processor node name, which is very often the topic name it came from. The same 249 limit applies to the result, so a name that is legal on its own fails at Streams startup with an InvalidTopicException naming a topic you never created. The suffix is 12 characters and the application.id costs its own length plus a hyphen, so 236 is the last length that leaves room for an application.id at all. Add a line reading application.id: your-app to the box and the arithmetic is exact instead of a budget, and the group id gets checked while it is there.

What makes a Kafka topic name legal, and what makes a legal one a mistake

Kafka's own validation is 15 lines long and easy to satisfy: an alphabet, a length, and two names refused outright. The interesting half is everything the broker accepts and then makes you regret, which is a shorter list than you would think and contains one entry that quietly merges two topics' monitoring together. All of it is decidable from the name itself, which is why this is a page and not a cluster query.

The rule, as Topic.validate applies it

This is the whole of the broker-side check, in the order it runs. Note what is absent: no lowercase requirement, no rule about where a hyphen or a dot may sit, no minimum length beyond one character, and no reserved-word list other than the two directory references.

1  not empty
2  not "." and not ".."
3  at most 249 characters
4  every character in [a-zA-Z0-9._-]

failing any of them:
  InvalidTopicException, at create time, and
  the same error to a producer relying on
  auto.create.topics.enable

Why the limit is 249 and not 255

Because the name is not only a name. Kafka stores each partition as a directory under log.dirs, named after the topic and the partition, and a single path component stops at 255 bytes on every filesystem you would run a broker on. The headroom is for the hyphen and the partition number, and for the suffixes Kafka renames a log directory with when a topic is deleted or a replica is moved between log directories.

log.dirs/
  orders-0/                    <topic>-<partition>
  orders-1/
  orders-2.9f3c...a1-delete/   a deleted log dir
  orders-3.4b71...c8-future/   moving log dirs

255  the filesystem limit on one component
249  the limit on the name
  6  the hyphen and the partition number

The "." and "_" collision, which is the one that costs money

A metric name is a dotted hierarchy, so a dot inside a topic name would split it into two levels. Kafka's answer is to replace every "." with "_" when it builds the metric name, which makes two legal, distinct topics indistinguishable everywhere the metrics are read: Prometheus, Datadog, Grafana, an alert on bytes in, a lag dashboard. kafka-topics.sh prints the warning below on every create whose name contains either character, once, into a terminal nobody is reading. Kafka has a function for the pair test, Topic.hasCollision, and the ZooKeeper-based create path used it to refuse the second topic outright. Do not depend on that: what is true on every cluster is that the metrics merge.

orders.v1   ->  metric name  orders_v1
orders_v1   ->  metric name  orders_v1
                             ^ one name

WARNING: Due to limitations in metric names,
topics with a period ('.') or underscore ('_')
could collide. To avoid issues it is best to use
either, but not both.

what merges: BytesInPerSec, BytesOutPerSec,
MessagesInPerSec, and every lag figure built
on top of them

Uppercase is legal, and that is a trap of its own

Every other naming rule in this part of the stack is lowercase only, so people assume this one is too. It is not: a topic name is case-sensitive and case-preserving, and there is no folding anywhere. Orders and orders are two topics with two sets of partitions and two sets of offsets, and the only symptom is a consumer that reads nothing while the producer reports success.

Orders     valid
orders     valid, and a different topic
ORDERS     valid, and a third one

a Kubernetes Service named Orders is rejected
a Kafka topic named Orders is created

The names refused by name, and "..." which is not one of them

Two strings are refused because of what they mean to a filesystem rather than because of their characters. Kafka stops there and checks nothing else about dots, so a name of three or more dots is created without complaint. It is legal, and its log directories are hidden files, and its metric name is a row of underscores.

""      rejected, before anything else
"."     rejected by name
".."    rejected by name
"..."   created. log dir "...-0", metric "___"
".x"    created. log dir ".x-0", hidden from ls
"-x"    created, and unusable from a shell:
        --topic -x parses -x as an option

The internal topics, and the prefix that claims to be one

Three names are the cluster's own. The first two are what Topic.isInternal returns true for, which is what makes a consumer skip them on a pattern subscription unless exclude.internal.topics is set to false, and what makes kafka-topics.sh refuse to delete them. Both are readable if you actually want them, and both hold Kafka's own binary format rather than anything you wrote. A topic of your own under the same prefix is legal and gets filtered out by tooling that pattern-matches instead of asking.

__consumer_offsets    committed offsets, 50
                      partitions by default
__transaction_state   the transaction
                      coordinator's log
__cluster_metadata    the KRaft metadata log

__my_audit_log        yours, legal, and skipped
                      by anything that filters
                      on the prefix

Names that pass here and are too long somewhere else

The same shape as a Kubernetes CronJob name capped at 52: the name is accepted, and something builds a second name out of it and runs out of room. Kafka Streams is the common one, and MirrorMaker 2 is the one that catches people twice, because its default replication policy prefixes the source alias and a dot, which both lengthens the name and introduces the collision character into a name that did not have one.

Kafka Streams
  <application.id>-<name>-changelog
  <application.id>-<name>-repartition
  14 + 1 + 223 + 12 = 250, and the limit is 249
  fails at Streams startup, naming the derived
  topic rather than yours

  236  the longest name that leaves room for an
       application.id of any length at all

MirrorMaker 2, default replication policy
  orders_v1 on us-east becomes
  us-east.orders_v1 on the target
  which shares a metric name with a local
  topic called us-east_orders_v1

A group id is not a topic name, and has no rules at all

Worth knowing because the two sit next to each other in every client config and only one of them is validated. The broker enforces no alphabet and no length on a group id: dots, slashes, spaces and unicode are all accepted, and the only hard ceiling is the 32,767 byte limit on a protocol string. So there is no such thing as an invalid group id, only a group id that is not the one holding the offsets you expected, and a typo in one is a silent reset rather than an error. Java's Properties.load keeps trailing whitespace on a value, which is how a stray space at the end of a client.properties line becomes a group with no committed offsets.

group.id=orders-consumer     joins the group
group.id=orders-consumer     a second group,
                             trailing space

no committed offsets, so it starts wherever
auto.offset.reset points:
  earliest   reprocess the retention window
  latest     skip everything backed up

application.id in Kafka Streams is the group id
and the prefix on every internal topic, so it
is measured against 249 as well

What this does not check

Only the text. It does not know which topics already exist, so the collision check covers the names in the box and nothing else: a name that collides with a topic already on the cluster looks perfectly clean here, and kafka-topics.sh --list is what settles that half. It does not know your partition count, your replication factor, whether the broker has auto.create.topics.enable set, or whether you have Create ACLs on the cluster. It also does not know your own naming convention, which is the thing most likely to reject the name after this page accepts it.

checked      the characters, the length, the
             metric name, and the collisions
             between the names you pasted

not checked  whether the topic exists
             whether it collides with one that
             does
             partitions, replication, ACLs
             your own naming convention

More kafka tools

Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka

Elsewhere on the site

Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer