Posted in Software Engineering / Data-platform

Apache Avro is a data serialization system. You describe your data with a schema, Avro writes the data in a compact binary form, and a reader turns the bytes back into objects because it knows the schema they were written with. Avro also lets data written with one version of a schema be read with another, which is why it’s everywhere data moves between systems that change at different speeds: Kafka topics, data lakes, Hadoop and Spark jobs, long-term archives.

The usual way to learn Avro is to read the documentation and copy a main method. This series takes a different route: every concept is a test. You read a short explanation, look at a real example, run the tests that prove it, then break something on purpose and watch what fails. It’s the same approach as my JUnit 6 and REST Assured courses, applied to Avro.

This first post covers Module 1: Getting Started. It turns the official Getting Started (Java) guide into eight sections and 41 tests. By the end you’ll have defined a schema, generated a Java class from it, created users three ways, written them to a data file, read them back with and without generated code, and switched on the guide’s beta feature for faster code. Along the way you’ll also hit two things the guide doesn’t tell you about Avro 1.12.2.

The course at a glance

Module 1 is the first of twelve. The full learning outline lists every section; here is the route from beginner to advanced:

ModuleLevelYou will learn
1. Getting StartedBeginnerThe Getting Started guide as tests: schema, code generation, writing and reading a data file, with and without generated code
2. SchemasBeginnerEvery primitive and complex type, names, namespaces, aliases, defaults, building and parsing schemas
3. Code GenerationBeginnerThe Maven plugin’s goals and options, generated classes, schemas in several files
4. Data ModelsIntermediateDatum readers and writers, encoders and decoders, generic, specific and reflect data
5. Binary and JSON EncodingIntermediateThe encodings byte by byte, and serializers you write yourself
6. Logical TypesIntermediateDecimals for money, UUIDs, dates, timestamps, durations, a logical type of your own
7. Schema Resolution and EvolutionIntermediateReading old data with a new schema, the resolution rules, compatibility checks
8. Object Container FilesAdvancedThe file format, codecs, metadata, sync markers, splitting files, avro-tools
9. Single-Object Encoding, Sort Order and FingerprintsAdvancedSelf-describing messages, comparing encoded data, canonical form
10. Avro IDL, Protocols and RPCAdvancedSchemas in IDL, and a working RPC client and server
11. Untrusted Data and RobustnessAdvancedTrusted classes, size limits, malformed input, compression bombs
12. Avro with Kafka and a Schema RegistryAdvancedSchema ids on the wire, serializers, subjects, compatibility in a registry

The Apache Avro documentation is the source for everything: the Getting Started (Java) guide, the Specification and the Java API.

Before you start

You need:

The project uses Apache Avro 1.12.2, JUnit 6.1.3 and AssertJ 3.27.7.

Get the code and run it:

git clone https://github.com/andre-mare/java-apache-avro.git
cd java-apache-avro
mvn test -pl module-01

You should see two test runs: 38 tests in the default run and 3 more in a run called custom-coders (section 1.8 explains why), then BUILD SUCCESS.

How the module is organised

module-01/
├── pom.xml
├── src/main/resources/avro/
│   └── user.avsc                         the guide's schema
├── src/main/java/com/example/gettingstarted/
│   ├── Users.java                        Alyssa, Ben and Charlie, created three ways
│   ├── SpecificUsersFile.java            write and read users.avro with the generated class
│   └── GenericUsersFile.java             the same without code generation
└── src/test/java/com/example/module_01/
    ├── section_01/ProjectSetupTest.java
    ├── section_02/DefiningASchemaTest.java
    ├── section_03/CompilingTheSchemaTest.java
    ├── section_04/CreatingUsersTest.java
    ├── section_05/SerializingTest.java
    ├── section_06/DeserializingTest.java
    ├── section_07/GenericUsersTest.java
    └── section_08/CustomCodersTest.java

The class example.avro.User doesn’t appear in that tree. The Avro Maven plugin generates it from user.avsc during every build, into target/generated-sources/avro.

How to study each section:

  1. Read the section of the Avro guide it links to.
  2. Read the example and run its tests: mvn test -pl module-01 -Dtest=ClassName.
  3. Do the “Try this” exercise: break something, predict the result, run the tests again.

Section 1.1: Setting up the project

Goal: a Maven module that generates a Java class from an Avro schema on every build, without any generated code in your source tree.

What you will learn:

  • The one dependency Avro needs, and how the avro-maven-plugin hooks into the build
  • Why this course keeps schemas in src/main/resources/avro and generated code in target/, where the guide uses src/main/avro and src/main/java
  • How a test can check the build itself

Reference: Getting Started: Download

Example: the dependency and plugin from the guide, as they appear in module-01/pom.xml (the versions come from the parent POM):

<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro</artifactId>
</dependency>

<plugin>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-maven-plugin</artifactId>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>schema</goal>
            </goals>
            <configuration>
                <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

The course makes two changes to the guide’s configuration:

  1. Schemas live in src/main/resources/avro instead of the plugin’s default, src/main/avro. Maven copies everything under resources to target/classes, so the schema is also on the classpath at runtime and packaged in the jar. Section 1.7 uses that.
  2. Generated code stays in target/generated-sources/avro. The guide sets <outputDirectory> to src/main/java, which puts User.java next to the code you write, where someone can edit and commit it. Every generated file starts with Autogenerated by Avro ... DO NOT EDIT DIRECTLY, and the plugin only regenerates a file when the schema is newer than it. A hand edit can survive for weeks and then vanish silently on the next schema change. Leaving the output in target/ means mvn clean removes it, it is never committed, and editing it is pointless.

The plugin adds target/generated-sources/avro to the compile sources by itself, so your code and your IDE see User as usual.

Tests: ProjectSetupTest

@Test
void theGeneratedSourceIsBuildOutput() {
    assertThat(Path.of("target/generated-sources/avro/example/avro/User.java")).isRegularFile();

    // The guide's outputDirectory, src/main/java, would mix generated code with code you write and commit.
    assertThat(Path.of("src/main/java/example/avro/User.java")).doesNotExist();
}
TestWhat it proves
theUserClassIsGeneratedDuringTheBuildUser.getClassSchema().getFullName() is example.avro.User; if the test compiles, the plugin ran first
theGeneratedSourceIsBuildOutputUser.java is in target/generated-sources/avro, not in src/main/java
theSchemaFileIsWhereThePluginLooksForItThe schema is in src/main/resources/avro, and src/main/avro doesn’t exist
theSchemaIsAlsoOnTheClasspathtarget/classes/avro/user.avsc exists, and getResource("avro/user.avsc") finds it

Try this: run mvn generate-sources -pl module-01 and open module-01/target/generated-sources/avro/example/avro/User.java. Read the header comment, then find SCHEMA$, the builder and the custom encode method at the bottom.

Section 1.2: Defining a schema

Goal: read the guide’s schema as Avro sees it: a named record with typed fields, two of which may be missing.

What you will learn:

  • How a record schema is written in JSON: namespace, type, name and fields
  • How a union with null makes a field optional
  • How to parse a schema with SchemaParser, the parser Avro 1.12 introduced
  • Why “nullable” and “has a default” are two different things

Reference: Getting Started: Defining a schema, and in the specification Records, Unions and Names. API: SchemaParser, Schema.

Example: src/main/resources/avro/user.avsc, exactly as in the guide:

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

The namespace and the name together give the full name, example.avro.User, which becomes the Java package and class. name is a required string. favorite_number and favorite_color are unions: each value is either an int (or a string) or null.

@BeforeAll
static void parseTheSchemaFile() throws IOException {
    schema = new SchemaParser().parse(new File("src/main/resources/avro/user.avsc")).mainSchema();
}

@Test
void favoriteNumberIsAUnionOfIntAndNull() {
    Schema favoriteNumber = schema.getField("favorite_number").schema();

    assertThat(favoriteNumber.getType()).isEqualTo(Type.UNION);
    assertThat(favoriteNumber.getTypes()).extracting(Schema::getType).containsExactly(Type.INT, Type.NULL);
    assertThat(favoriteNumber.isNullable()).isTrue();
}

SchemaParser.parse(..) returns a result that can hold several schemas; mainSchema() is the one the file defines.

Tests: DefiningASchemaTest

TestWhat it proves
theSchemaIsARecordWithAFullNameType RECORD, name User, namespace example.avro, full name example.avro.User
theSchemaHasThreeFieldsInOrderThe fields, in order, and that favorite_color is at position 2
nameIsARequiredStringname is a STRING and not nullable
favoriteNumberIsAUnionOfIntAndNullThe union’s branches are INT then NULL
favoriteColorIsAUnionOfStringAndNullThe same for the colour
noFieldHasADefaultNone of the three fields has a default. This matters in section 1.4
theSchemaPrintsBackAsJsontoString() turns the parsed schema back into JSON

Try this: change "string" to "strin" in a copy of the schema inside a test, parse it, and read the SchemaParseException.

Section 1.3: Compiling the schema

Goal: understand what the compiler makes of a schema, so the generated class holds no surprises.

What you will learn:

  • How the guide compiles with avro-tools, and how the Maven plugin does the same on every build
  • How schema names become Java names: favorite_number becomes getFavoriteNumber()
  • Why a nullable int becomes an Integer, and why strings are CharSequence by default
  • That every generated class carries its own schema

Reference: Getting Started: Compiling the schema. API: SpecificRecordBase.

Example: the guide’s command-line compiler:

java -jar avro-tools-1.12.2.jar compile schema src/main/resources/avro/user.avsc /tmp/generated

The tests use reflection to look at the class the plugin generated:

@Test
void aNullableIntIsABoxedInteger() throws NoSuchMethodException {
    // ["int", "null"] can hold null, so it can't be a primitive int.
    assertThat(User.class.getMethod("getFavoriteNumber").getReturnType()).isEqualTo(Integer.class);
}

@Test
void stringsAreCharSequencesByDefault() throws NoSuchMethodException {
    assertThat(User.class.getMethod("getName").getReturnType()).isEqualTo(CharSequence.class);
}

The guide’s plugin configuration doesn’t set stringType, so the default applies, and getters for strings return CharSequence. Section 1.6 shows why that matters after reading data back.

Tests: CompilingTheSchemaTest

TestWhat it proves
fieldNamesBecomeCamelCaseAccessorsgetFavoriteNumber(), setFavoriteNumber(Integer) and getFavoriteColor() exist
aNullableIntIsABoxedIntegerThe getter returns Integer, so it can return null
stringsAreCharSequencesByDefaultgetName() returns CharSequence
theClassEmbedsItsSchemaUser.SCHEMA$, User.getClassSchema() and user.getSchema() are the same object
theEmbeddedSchemaIsTheSchemaInTheFileThe embedded schema equals the parsed user.avsc

Try this: add <stringType>String</stringType> to the plugin’s configuration, rebuild, and watch stringsAreCharSequencesByDefault fail. Module 3 covers the generator options in detail.

Section 1.4: Creating users

Goal: create objects of the generated class in the guide’s three ways, and know which one checks your data.

What you will learn:

  • The no-argument constructor with setters, the all-arguments constructor, and the builder
  • That only the builder validates, and what it checks
  • Why a field that may be null still has to be set in a builder when it has no default
  • That generated records are values, with equals, hashCode and a JSON toString()

Reference: Getting Started: Creating Users. API: AvroMissingFieldException.

Example: Users.java holds the guide’s three users:

public static User alyssa() {
    User user1 = new User();
    user1.setName("Alyssa");
    user1.setFavoriteNumber(256);
    // Leave favorite color null
    return user1;
}

public static User ben() {
    return new User("Ben", 7, "red");
}

public static User charlie() {
    return User.newBuilder()
        .setName("Charlie")
        .setFavoriteColor("blue")
        .setFavoriteNumber(null)
        .build();
}

The guide’s .setFavoriteNumber(null) looks redundant, but it isn’t. The builder fills an unset field from its default, and favorite_number has no default (section 1.2), so leaving it out fails:

@Test
void theBuilderRejectsAFieldThatIsNeitherSetNorDefaulted() {
    assertThatThrownBy(() -> User.newBuilder().setName("Charlie").setFavoriteColor("blue").build())
        .isInstanceOf(AvroMissingFieldException.class)
        .hasMessageContaining("favorite_number");
}

The constructors check nothing at all: new User() gives a user without a name, and that mistake only shows up when the user is written (section 1.5).

Tests: CreatingUsersTest

TestWhat it proves
settersLeaveUnsetFieldsNullAlyssa’s colour is null
theConstructorTakesFieldsInSchemaOrderBen’s fields match the constructor’s arguments
theBuilderValidatesAndBuildsCharlie’s fields, including the null number
theBuilderRejectsAFieldThatIsNeitherSetNorDefaultedAvroMissingFieldException naming favorite_number
theNoArgumentConstructorChecksNothingA User without a name can be created
usersAreValuesWithEqualsTwo Bens are equal and have the same hash code
toStringShowsTheUserAsJson{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}

Try this: add "default": null to favorite_number in the schema. Remember that a union’s default must match one of its branches, then rebuild. Which test now fails, and why?

Section 1.5: Serializing to a data file

Goal: write the three users to users.avro, and see what an Avro data file contains.

What you will learn:

  • The two layers of writing: a DatumWriter turns a record into Avro binary, and a DataFileWriter wraps it in the file format
  • That a data file stores its schema in its header
  • Why closing the writer matters
  • What happens when a record breaks its schema

Reference: Getting Started: Serializing, and Object Container Files in the specification. API: SpecificDatumWriter, DataFileWriter.

Example: SpecificUsersFile.write is the guide’s code, with try-with-resources in place of an explicit close():

public static void write(File file, List<User> users) throws IOException {
    DatumWriter<User> userDatumWriter = new SpecificDatumWriter<>(User.class);
    try (DataFileWriter<User> dataFileWriter = new DataFileWriter<>(userDatumWriter)) {
        dataFileWriter.create(users.get(0).getSchema(), file);
        for (User user : users) {
            dataFileWriter.append(user);
        }
    }
}

The tests write to a JUnit @TempDir (see the JUnit 6 user guide), a fresh folder per test that is deleted afterwards:

@TempDir
Path dir;

@Test
void threeUsersAreWrittenToUsersAvro() throws IOException {
    File file = dir.resolve("users.avro").toFile();

    SpecificUsersFile.write(file, List.of(Users.alyssa(), Users.ben(), Users.charlie()));

    // Every Avro data file starts with the magic bytes "Obj" and the format version, 1.
    assertThat(Files.readAllBytes(file.toPath())).startsWith(new byte[] {'O', 'b', 'j', 1});
}

Remember the user without a name from section 1.4? Writing it fails, and the cause says exactly where:

assertThatThrownBy(() -> SpecificUsersFile.write(file, List.of(new User())))
    .isInstanceOf(DataFileWriter.AppendWriteException.class)
    .cause()
    .isInstanceOf(NullPointerException.class)
    .hasMessage("null value for (non-nullable) string at User.name");

Tests: SerializingTest

TestWhat it proves
threeUsersAreWrittenToUsersAvroThe file starts with Obj and version 1
theFileStoresTheSchemaInItsHeaderA reader given no schema finds User’s schema in the header. Without compression the header has only avro.schema: Avro writes no avro.codec entry at all
theFileIsCompleteWhenTheWriterIsClosedBefore close() the file holds only the header; Ben is still in the writer’s buffer
aUserMissingARequiredFieldCannotBeWrittenAn AppendWriteException whose cause names User.name

Try this: after the tests run, look at a file with avro-tools: getschema, getmeta and tojson --pretty. Module 8 takes the file format apart byte by byte.

Section 1.6: Deserializing from a data file

Goal: read users.avro back into User objects safely, and meet the setting Avro 1.12.2 requires for it.

What you will learn:

  • How the guide’s reading loop reuses one object, and the bug you get if you keep it
  • That strings come back as Avro’s Utf8, not String
  • That Avro 1.12.2 only instantiates trusted classes, including your own generated ones

Reference: Getting Started: Deserializing, and the Avro 1.12.2 release notes. API: SpecificDatumReader, DataFileReader, Utf8, ClassSecurityValidator.

Example: the guide reads with next(user), handing the previous object back so Avro can fill it again instead of allocating a new one:

try (var dataFileReader = new DataFileReader<>(file, new SpecificDatumReader<>(User.class))) {
    User user = null;
    while (dataFileReader.hasNext()) {
        user = dataFileReader.next(user);
        names.add(user.getName().toString());        // copy the value before the next read
    }
}

That saves garbage, but it is the same object every time. Add user itself to a list and you get Charlie three times:

@Test
void keepingTheReusedObjectKeepsOnlyTheLastUser() throws IOException {
    List<User> kept = new ArrayList<>();
    try (var dataFileReader = new DataFileReader<>(file, new SpecificDatumReader<>(User.class))) {
        User user = null;
        while (dataFileReader.hasNext()) {
            user = dataFileReader.next(user);
            kept.add(user);                              // the bug: the same object, three times
        }
    }

    assertThat(kept).extracting(user -> user.getName().toString()).containsOnly("Charlie");
}

The trust setting. Since 1.12.2, a SpecificDatumReader only creates instances of trusted classes, and by default Avro trusts only a few java.lang and java.math types. Without a setting, reading users.avro into User objects throws a SecurityException. The project’s parent POM passes this to every test run:

<systemPropertyVariables>
    <org.apache.avro.SERIALIZABLE_PACKAGES>example.avro</org.apache.avro.SERIALIZABLE_PACKAGES>
</systemPropertyVariables>

One test switches the trust off to show the exception:

@Test
void readingGeneratedClassesRequiresTrustingTheirPackage() {
    ClassSecurityValidator.setGlobal(ClassSecurityValidator.DEFAULT_TRUSTED_CLASSES);
    var untrustingReader = new SpecificDatumReader<User>(User.getClassSchema(), User.getClassSchema(),
        new SpecificData());

    assertThatThrownBy(() -> {
        try (var dataFileReader = new DataFileReader<>(file, untrustingReader)) {
            dataFileReader.next();
        }
    })
        .isInstanceOf(SecurityException.class)
        .hasMessageContaining("example.avro.User")
        .hasMessageContaining("org.apache.avro.SERIALIZABLE_PACKAGES");
}

An @AfterEach method restores the default with ClassSecurityValidator.setGlobal(ClassSecurityValidator.DEFAULT), so the other tests aren’t affected. Module 11 explains why Avro added this: a schema can name any Java class to instantiate, which is a classic deserialization attack.

Tests: DeserializingTest

TestWhat it proves
theThreeUsersAreReadBackInOrderAlyssa, Ben and Charlie come back equal to what was written
theGuidesLoopReusesOneUserObjectCopying each name before the next read gives all three names
keepingTheReusedObjectKeepsOnlyTheLastUserKeeping the object itself gives Charlie three times
readStringsAreUtf8NotStringgetName() returns a Utf8 that isn’t equal to "Ben", though its text is
readingGeneratedClassesRequiresTrustingTheirPackageWithout trust, a SecurityException naming the class and the property

Try this: run DeserializingTest with the property removed from the parent POM, and see how many tests fail.

Section 1.7: Without code generation

Goal: do the same work with no generated class at all, using generic records and a schema loaded at runtime.

What you will learn:

  • How GenericRecord holds data described only by a schema
  • When generic records are the right choice: tools, routers, and schemas you only learn about at runtime
  • How to load a schema from the classpath
  • That the bytes don’t depend on how they were written

Reference: Getting Started: Serializing and deserializing without code generation. API: GenericRecord, GenericData.Record, GenericDatumWriter, GenericDatumReader.

Example: GenericUsersFile. The schema is under src/main/resources, so it is on the classpath and can be loaded from anywhere the code runs:

public static Schema schema() throws IOException {
    try (InputStream in = GenericUsersFile.class.getClassLoader().getResourceAsStream("avro/user.avsc")) {
        if (in == null) {
            throw new FileNotFoundException("No schema on the classpath at avro/user.avsc");
        }
        return new SchemaParser().parse(in).mainSchema();
    }
}

public static GenericRecord ben(Schema schema) {
    GenericRecord user2 = new GenericData.Record(schema);
    user2.put("name", "Ben");
    user2.put("favorite_number", 7);
    user2.put("favorite_color", "red");
    return user2;
}

Field names are now strings, so mistakes surface at runtime. The guide points out one at once:

@Test
void anUnknownFieldIsRejected() {
    GenericRecord user = new GenericData.Record(schema);

    assertThatThrownBy(() -> user.put("favorite_animal", "cat"))
        .isInstanceOf(AvroRuntimeException.class)
        .hasMessageContaining("favorite_animal");
}

And the most useful lesson of the section: a file written with generic records reads back as generated User objects, because the file holds only bytes and a schema.

GenericUsersFile.write(file, schema, List.of(GenericUsersFile.alyssa(schema), GenericUsersFile.ben(schema)));

assertThat(SpecificUsersFile.read(file)).containsExactly(Users.alyssa(), Users.ben());

Tests: GenericUsersTest

TestWhat it proves
genericUsersSurviveAWriteAndReadNumbers, nulls and colours come back
aRecordIsEqualToTheRecordItWasReadFromGenericData compares strings by their text, so a String and a Utf8 count as equal
stringsComeBackAsUtf8get("name") is a Utf8: isEqualTo("Alyssa") fails, hasToString("Alyssa") passes
anUnknownFieldIsRejectedAvroRuntimeException naming the field
fieldsCanBeReachedByPositionTooget(0) is get("name")
theGenericFileReadsAsGeneratedUsersThe generic file reads back as User objects

Try this: put the number 7 as a String instead of an Integer. Does put complain? Does writing?

Section 1.8: Beta feature: generating faster code

Goal: switch on the guide’s beta feature and prove it changes the speed, not the result.

What you will learn:

  • That generated classes contain their own encode and decode methods, used only when a flag is on
  • How to run one test class in a separate Maven Surefire execution with its own system property
  • How to prove two code paths write identical bytes

Reference: Getting Started: Beta feature: Generating faster code. API: SpecificData. Maven: Surefire plugin.

Example: the feature is switched on with the system property org.apache.avro.specific.use_custom_coders=true, which Avro reads once. So the module’s POM runs every test except this class in the default execution, and this class alone in a second execution with the property set:

<execution>
    <id>custom-coders</id>
    <goals>
        <goal>test</goal>
    </goals>
    <configuration>
        <test>CustomCodersTest</test>
        <systemPropertyVariables>
            <org.apache.avro.specific.use_custom_coders>true</org.apache.avro.specific.use_custom_coders>
        </systemPropertyVariables>
    </configuration>
</execution>

The custom coders are a faster route to the same format. A GenericDatumWriter never uses them, so it makes a perfect reference:

@Test
void theBytesAreTheSameEitherWay() throws IOException {
    User ben = Users.ben();

    byte[] custom = write(new SpecificDatumWriter<>(User.class), ben);
    byte[] generic = write(new GenericDatumWriter<>(User.getClassSchema()), ben);

    assertThat(custom).isEqualTo(generic);
}

Tests: CustomCodersTest

TestWhat it proves
theFlagIsOnSpecificData.getForClass(User.class).useCustomCoders() is true in this execution
usersRoundTripWithCustomCodersAlyssa, Ben and Charlie survive a write and read
theBytesAreTheSameEitherWayThe custom coders write exactly the bytes the generic writer writes

To run it on its own:

mvn test -pl module-01 -Dtest=CustomCodersTest -Dorg.apache.avro.specific.use_custom_coders=true

Two things the guide doesn’t tell you

Running the guide’s code against Avro 1.12.2 as tests turned up two things worth knowing before you use Avro anywhere real.

1. You must trust your own generated classes. Since 1.12.2, reading generated classes with a SpecificDatumReader throws a SecurityException unless their package is listed in the system property org.apache.avro.SERIALIZABLE_PACKAGES (or set in code with ClassSecurityValidator.setGlobal(..)). The guide doesn’t mention it yet. Set it at application startup, and list your own packages rather than *, which trusts everything.

2. Strings come back as Utf8. With the plugin’s default settings, string getters return CharSequence, and after reading that is org.apache.avro.util.Utf8. A Utf8 is never equals to a String, even with the same text. Compare with toString() (AssertJ’s hasToString does that), or generate String getters with the plugin’s stringType option (Module 3).

Common pitfalls from Module 1

  • Leaving a nullable field out of a builder. ["int", "null"] without a default throws AvroMissingFieldException. Set it explicitly, or give it "default": null with null first in the union.
  • Trusting the constructors. new User() and setters check nothing; the error surfaces when the record is written.
  • Keeping a reused record. next(user) fills the same object again. Copy what you need, or call next() without an argument.
  • Comparing Utf8 with String. Use toString().
  • Forgetting to close the writer. The last block stays in the buffer. Use try-with-resources.
  • Editing generated code. Change the schema or the plugin options instead, and keep generated code in target/.

Cheat sheet

I want to…Code
Parse a schema filenew SchemaParser().parse(file).mainSchema()
Parse a schema from the classpathnew SchemaParser().parse(getClass().getClassLoader().getResourceAsStream("avro/user.avsc")).mainSchema()
Create a generated record safelyUser.newBuilder().setName("Ben").setFavoriteNumber(7).setFavoriteColor("red").build()
Create a generic recordGenericRecord r = new GenericData.Record(schema); r.put("name", "Ben");
Write a data filenew DataFileWriter<>(new SpecificDatumWriter<>(User.class)).create(schema, file), then append(..)
Read a data filenew DataFileReader<>(file, new SpecificDatumReader<>(User.class)), then hasNext() and next(..)
Trust generated classes-Dorg.apache.avro.SERIALIZABLE_PACKAGES=example.avro
Look inside a filejava -jar avro-tools-1.12.2.jar tojson --pretty users.avro

What’s next

Module 1 used one small schema. Module 2: Schemas covers the whole type system: all eight primitive types, records, enums, arrays, maps, unions and fixed, how names and namespaces work across files, aliases and defaults, building schemas in code with SchemaBuilder, and repairing old schemas that newer versions of Avro reject.

Resources

Apache Avro documentation

Java API classes used in this module

The course code

Testing tools