This lesson explains how we can use Apache Avro as a serialization choice when Kafka producers send messages.

We'll cover the following

Using Avro with Kafka

Kafka comes with serialization classes for simple types such as string, integers, and byte arrays. However, one has to use a serialization library for complex types. We can use JSON, Apache Avro, Thrift or Protobuf for serializing and deserializing Kafka messages.

Using Avro with Kafka

For this course, we’ll use Avro for serialization and deserialization and discuss it in the context of Kafka. Apache Avro is a language neutral serialization framework and thus a good choice for Kafka. The Avro project was developed by Doug Cutting of Hadoop fame and later incubated at Apache.

Avro provides robust support for schema evolution. A schema can be thought of as a blue-print of the structure of each record in an .avro file. For instance, we can define a schema representing a car as follows: