Kafka Protobuf to Avro Schema

Convert a .proto to an Avro schema, and get told what did not survive. Unsigned 64-bit types have no Avro equivalent, enums renumber when they are sparse, and a singular proto3 scalar has no presence to map onto a nullable union.

The root message becomes the top-level Avro record. Everything it reaches is emitted inline at its first use, and a second reference becomes a name, which is what makes a recursive message representable.

Paste below, or drop a file anywhere on this panel

Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.

The answer appears here

Paste on the left and press Convert. Nothing leaves this tab.

Examples

Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.

A proto3 message

The Avro record it becomes, and where the two type systems do not line up

syntax = "proto3";
message User {
  int32 id = 1;
  string email_address = 2;
}

An enum and a repeated field

How proto enums and repeated fields map, including the zero-value rule

syntax = "proto3";
enum Status {
  UNKNOWN = 0;
  ACTIVE = 1;
}
message Order {
  Status status = 1;
  repeated string tags = 2;
}

Common mistakes

These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

  1. Expecting proto3 optionality to survive

    Proto3 scalars have no null. A field left unset is the zero value, which is not the same as an Avro null union.

    Instead:Decide per field whether absence means null or zero, and encode that deliberately.

  2. Mapping enums directly

    Proto enums require a zero value and allow unknown values through. Avro enums are closed and reject anything unlisted.

    Instead:Map to a string, or include a default symbol.

  3. Ignoring field numbers

    Protobuf identifies fields by number and Avro by name and position. A rename is safe in protobuf and breaking in Avro.

    Instead:Keep names stable when both formats are in play.

The easy eighty per cent maps straight across, and the rest is why this page exists

Both formats describe records of typed fields, so most of a schema converts without a decision to make. A converter is worth a page for the parts that do not, because a schema the registry accepts and the data does not fit is worse than a failed conversion.

Two protobuf types have no Avro equivalent at all

Avro's long is signed 64-bit, so uint64 and fixed64 cannot be represented: every value above 9223372036854775807 has nowhere to go. Avro has no unsigned types and nothing wider, so there is no lossless choice to offer. uint32 and fixed32 have a milder version of the same problem, since Avro's int is signed 32-bit, and they widen to long, which is lossless but changes the generated Java accessor from int to long.

Avro enums have no numbers, so a sparse protobuf enum renumbers

A protobuf enum is a name-to-number mapping and the number is what goes on the wire. An Avro enum is an ordered list of symbols and the ordinal is what goes on the wire. The two agree only when the protobuf numbers happen to be 0, 1, 2 in declaration order. An enum numbered 0, 5, 9 becomes 0, 1, 2 in Avro, so every symbol after the first gap silently changes value. This is reported with the actual numbers rather than as a general warning.

Presence decides whether a field is nullable, and proto3 mostly has none

A singular proto3 scalar cannot tell unset from zero, so there is nothing to map a nullable union onto: it becomes a plain Avro field with the type's default, which is the closest true statement. A field declared optional does have presence and becomes ["null", T] with a default of null. So does a message field, and so does a wrapper type, which is the one well-known type that maps onto Avro exactly. Adding optional in the .proto and converting again is the way to get a nullable Avro field.

Timestamps lose three digits and durations have no counterpart

google.protobuf.Timestamp carries seconds plus nanoseconds. Avro's finest timestamp logical type is microseconds, so the last three digits are dropped. google.protobuf.Duration is worse: Avro's duration is a twelve-byte fixed of months, days and milliseconds, which cannot express a signed span of seconds and nanos at all, so it is converted to a record of seconds and nanos instead, which is lossless but is not Avro's own type.

The output is checked with this site's own Avro parser

Every schema this produces is parsed back before it is shown, using the same parser the Avro validator and compatibility checker on this site read through. That is a rule every generator here follows, and it is not decoration: it caught a union nested inside another union in this converter, which Avro forbids and which the wrapper types produce if the nullable wrapping does not flatten.

What this cannot see

Anything behind an import, since it reads one file. Whether the registry will accept the result under your compatibility setting, which depends on the version already registered rather than on the schema alone: the compatibility checker on this site answers that once you have both. It also does not convert data, only the schema.

More kafka tools

Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka

Elsewhere on the site

Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer