Apache Avro: Module 1 Section One - Environment Setup

Posted in Software Engineering / Data-platform on Apache Avro
Apache Avro: Module 1 Section One - Environment Setup

Apache Avro in Java - Setting Up a Maven Project That Generates Code from Schemas

Most Java libraries arrive as a single dependency and you start calling them. Apache Avro is different: the centre of an Avro project is not Java code but a schema, a JSON file that describes your data. From that schema a build plugin generates Java classes, and your code, your tests and every system that reads your data depend on the schema being where the build expects it. Get the project layout right and Avro feels effortless. Get it wrong and you end up with generated code committed next to hand-written code, edits that vanish on the next build, and readers that refuse to load your own classes.

This guide sets up the Maven project behind the Getting Started (Java) guide of Apache Avro 1.12.2, the first section of an example-driven Avro course in which every concept is a JUnit 6 test. You’ll add the dependency and the avro-maven-plugin, decide where schemas and generated code live and why, run the build, look at what it generated, and write tests that check the build itself.

What is Apache Avro?

Apache Avro is a data serialization system. You describe your data once, in a schema, and Avro uses it to write that data in a compact binary form and read it back. Three properties make it popular in data platforms:

  1. The schema travels with the data. A data file stores its schema in its header, so any reader can decode it years later.
  2. The binary form is small. There are no field names or type tags in the data, only values in schema order.
  3. Schemas can evolve. Data written with one version of a schema can be read with another, so producers and consumers can change at different speeds.

In Java, Avro offers two ways to work with data: generated classes (a User class with getters, setters and a builder, produced from the schema at build time) and generic records (a map-like object driven by a schema loaded at runtime). This first section sets up the build for the generated-class approach; section 1.7 covers the generic one.

How the Build Fits Together

An Avro project adds one step in front of the normal Maven build:

  1. generate-sources: the avro-maven-plugin reads every .avsc file in its source directory and writes one Java class per named type.
  2. process-resources: Maven copies src/main/resources to target/classes. If the schemas live there, they end up on the classpath too.
  3. compile: the compiler compiles your code and the generated classes, because the plugin registers its output directory as a source root.
  4. test: your tests use the generated classes like any other.

Because generation happens on every build, the schema is the single source of truth. Change user.avsc, rebuild, and the User class changes with it.

Where Schemas and Generated Code Live

The Getting Started guide and this course make two different choices here, and the difference matters in real projects.

The Guide’s Layout

The guide keeps schemas in the plugin’s default directory, src/main/avro, and sets the plugin’s outputDirectory to src/main/java. The generated User.java then sits in your source tree next to the code you write. That makes the first example simple, because your IDE sees the class without a build, but it has a cost.

This Course’s Layout

The course makes two changes:

  • Schemas live in src/main/resources/avro. The plugin reads them from there, and because they’re resources, Maven also copies them to target/classes/avro. Your code can load a schema from the classpath at runtime, and it ships inside the jar.
  • Generated code stays in target/generated-sources/avro, the plugin’s default output directory. It is build output: mvn clean deletes it, .gitignore keeps it out of Git, and nobody edits it by accident.

Why Generated Code Should Never Be Edited

Every generated file starts with this header:

/*
 * Autogenerated by Avro
 *
 * DO NOT EDIT DIRECTLY
 */

The warning is there for a reason. The plugin only regenerates a class when the schema file is newer than the generated file. A hand edit to User.java in src/main/java therefore survives build after build, until someone changes user.avsc, and then it disappears without an error. It can also leave the class disagreeing with the schema it embeds. Keeping generated code in target/ removes the temptation: there is nothing to edit that would last.

If you need a different generated class, change the schema, change a plugin option (such as stringType), or write code around the class rather than in it.

Project Configuration

The course is a multi-module Maven project, java-apache-avro, with one module per topic. Module 1 is the Getting Started guide.

Step 1: Create the Parent POM

The parent manages versions for every module and passes one Avro 1.12.2 setting to every test run (Step 4 explains it):

<groupId>com.example</groupId>
<artifactId>java-apache-avro</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>

<modules>
    <module>module-01</module>
</modules>

<properties>
    <java.version>21</java.version>
    <maven.compiler.release>${java.version}</maven.compiler.release>
    <avro.version>1.12.2</avro.version>
    <junit.version>6.1.3</junit.version>
    <assertj.version>3.27.7</assertj.version>
    <avro.trusted.packages>example.avro</avro.trusted.packages>
</properties>

Its dependencyManagement imports the JUnit BOM and pins avro and assertj-core; its pluginManagement pins the compiler, Surefire and avro-maven-plugin versions.

Step 2: Add the Module POM

The module declares the one dependency the guide asks for, plus JUnit and AssertJ for the tests, and runs the plugin’s schema goal:

<dependencies>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
    </dependency>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.assertj</groupId>
        <artifactId>assertj-core</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <executions>
                <execution>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>schema</goal>
                    </goals>
                    <configuration>
                        <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

There is no outputDirectory: the default, target/generated-sources/avro, is the point. There is no build-helper-maven-plugin either; the Avro plugin adds its output to the compile sources itself.

Step 3: Add the Schema

Save the guide’s schema as module-01/src/main/resources/avro/user.avsc:

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

Section 1.2 takes it apart field by field.

Step 4: Trust Your Generated Classes

Avro 1.12.2 only instantiates Java classes it trusts when it reads data, and that includes your own generated classes. Without a setting, reading User objects back from a file throws a SecurityException. The parent POM passes the package to every test run through Surefire:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <systemPropertyVariables>
            <org.apache.avro.SERIALIZABLE_PACKAGES>${avro.trusted.packages}</org.apache.avro.SERIALIZABLE_PACKAGES>
        </systemPropertyVariables>
    </configuration>
</plugin>

Section 1.6 shows the exception and the reason behind it.

Build Walkthrough

Running the Build

mvn test -pl module-01

The first lines of the log show the extra step and the schema being copied as a resource:

[INFO] --- avro:1.12.2:schema (default) @ module-01 ---
[INFO] --- resources:3.3.1:resources (default-resources) @ module-01 ---
[INFO] Copying 1 resource from src/main/resources to target/classes
[INFO] --- compiler:3.15.0:compile (default-compile) @ module-01 ---
[INFO] Compiling 4 source files with javac [debug parameters release 21] to target/classes

“4 source files” is the three hand-written classes of Module 1 plus the generated User.java.

What the Plugin Generated

ls module-01/target/generated-sources/avro/example/avro/

User.java

The namespace example.avro became the package and the record name User became the class. The file is 541 lines long and starts like this:

/*
 * Autogenerated by Avro
 *
 * DO NOT EDIT DIRECTLY
 */
package example.avro;

@org.apache.avro.specific.AvroGenerated
public class User extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
  public static final org.apache.avro.Schema SCHEMA$ = JsonSchemaParser.parseInternal("{\"type\":\"record\",\"name\":\"User\",...}");
  public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }

Section 1.3 walks through the rest of the class.

Run every example yourself All the code in this post is in Module 1 of the Java Apache Avro course: a Maven project where every Avro concept is a JUnit 6 test you can run, break and fix. Get the code on GitHub → · Read the course outline →

Testing the Build Itself

A build configuration is code too, so Module 1 tests it. Maven runs tests with the module directory as the working directory, so paths are relative to module-01:

class ProjectSetupTest {

    @Test
    void theUserClassIsGeneratedDuringTheBuild() {
        // If this compiles, the plugin generated the class before the test sources were compiled.
        assertThat(User.getClassSchema().getFullName()).isEqualTo("example.avro.User");
        assertThat(SpecificRecord.class).isAssignableFrom(User.class);
    }

    @Test
    void theGeneratedSourceIsBuildOutput() {
        assertThat(Path.of("target/generated-sources/avro/example/avro/User.java")).isRegularFile();

        // The guide's outputDirectory, src/main/java, would mix generated code with code you write and commit.
        assertThat(Path.of("src/main/java/example/avro/User.java")).doesNotExist();
    }

    @Test
    void theSchemaFileIsWhereThePluginLooksForIt() {
        assertThat(Path.of("src/main/resources/avro/user.avsc")).isRegularFile();
        assertThat(Path.of("src/main/avro")).doesNotExist();
    }

    @Test
    void theSchemaIsAlsoOnTheClasspath() {
        assertThat(Path.of("target/classes/avro/user.avsc")).isRegularFile();
        assertThat(getClass().getClassLoader().getResource("avro/user.avsc")).isNotNull();
    }
}

The second test is the important one. If anyone switches the plugin’s output to src/main/java “to make the IDE happy”, the build fails and says why.

Common Pitfalls

  1. Generating into src/main/java. It invites edits that the next schema change silently discards. Keep the default output directory.
  2. Adding build-helper-maven-plugin. Older tutorials add it to register the generated sources. The Avro plugin already does that.
  3. Putting schemas somewhere the plugin doesn’t look. If sourceDirectory and the files disagree, the build succeeds but generates nothing, and the compile fails on a missing User class. The location test above catches it first.
  4. Forgetting the trust setting. Writing works without it; reading generated classes fails with a SecurityException. Set org.apache.avro.SERIALIZABLE_PACKAGES for tests and for the application.

Wrapping Up

An Avro project is a schema-first project. The schema lives in src/main/resources/avro, where both the plugin and the classpath can see it; the generated class lives in target/generated-sources/avro, where nobody can edit it; and one system property tells Avro 1.12.2 that your generated package is safe to instantiate. With four small tests guarding that layout, the build can’t quietly drift away from it.

Next, section 1.2 opens the schema itself: records, fields, and the unions with null that make a field optional.

Run every example yourself Clone the repository, run mvn test -pl module-01, and watch 41 tests pass. Then change the plugin’s output directory to src/main/java and watch one of them fail. Get the code on GitHub → · Module 1 README →