Getting Started with the Annotation Processing Tool, apt

The following topics are covered

The apt Tool Has Been Deprecated

The apt tool and its associated API contained in the package com.sun.mirror have been deprecated since JDK 7 and are planned to be removed in the next major JDK release. Use the options available in the javac tool and the APIs contained in the packages javax.annotation.processing and javax.lang.model to process annotations.

Summary of apt API Replacements

The following tables summarize apt API replacements:

Package `com.sun.mirror.apt`
`apt` type	Standard Replacement
`AnnotationProcessor`	`javax.annotation.processing.Processor`
`AnnotationProcessorEnvironment`	`javax.annotation.processing.ProcessingEnvironment`
`AnnotationProcessorFactory`	`javax.annotation.processing.Processor`
`AnnotationProcessorListener`	No analog.
`AnnotationProcessors`	No analog.
`Filer`	`javax.annotation.processing.Filer`
`Filer.Location`	`javax.tools.StandardLocation`
`Messager`	`javax.annotation.processing.Messager`
`RoundCompleteEvent`	No analog.
`RoundCompleteListener`	No analog.
`RoundState`	`javax.annotation.processing.RoundEnvironment`

Package `com.sun.mirror.declaration`
`apt` type	Standard Replacement
`AnnotationMirror`	`javax.lang.model.element.AnnotationMirror`
`AnnotationTypeDeclaration`	`javax.lang.model.element.TypeElement`
`AnnotationTypeElementDeclaration`	`javax.lang.model.element.ExecutableElement`
`AnnotationValue`	`javax.lang.model.element.AnnotationValue`
`ClassDeclaration`	`javax.lang.model.element.TypeElement`
`ConstructorDeclaration`	`javax.lang.model.element.ExecutableElement`
`Declaration`	`javax.lang.model.element.Element`
`EnumConstantDeclaration`	`javax.lang.model.element.VariableElement`
`EnumDeclaration`	`javax.lang.model.element.TypeElement`
`ExecutableDeclaration`	`javax.lang.model.element.ExecutableElement`
`FieldDeclaration`	`javax.lang.model.element.VariableElement`
`InterfaceDeclaration`	`javax.lang.model.element.TypeElement`
`MemberDeclaration`	`javax.lang.model.element.Element`
`MethodDeclaration`	`javax.lang.model.element.ExecutableElement`
`Modifier`	`javax.lang.model.element.Modifier`
`PackageDeclaration`	`javax.lang.model.element.PackageElement`
`ParameterDeclaration`	`javax.lang.model.element.VariableElement`
`TypeDeclaration`	`javax.lang.model.element.TypeElement`
`TypeParameterDeclaration`	`javax.lang.model.element.TypeParameterElement`

Package `com.sun.mirror.type`
`apt` type	Standard Replacement
`AnnotationType`	`javax.lang.model.type.DeclaredType`
`ArrayType`	`javax.lang.model.type.ArrayType`
`ClassType`	`javax.lang.model.type.DeclaredType`
`DeclaredType`	`javax.lang.model.type.DeclaredType`
`EnumType`	`javax.lang.model.type.DeclaredType`
`InterfaceType`	`javax.lang.model.type.DeclaredType`
`MirroredTypeException`	`javax.lang.model.type.MirroredTypeException`
`MirroredTypesException`	`javax.lang.model.type.MirroredTypesException`
`PrimitiveType`	`javax.lang.model.type.PrimitiveType`
`PrimitiveType.Kind`	`javax.lang.model.type.TypeKind`
`ReferenceType`	`javax.lang.model.type.ReferenceType`
`TypeMirror`	`javax.lang.model.type.TypeMirror`
`TypeVariable`	`javax.lang.model.type.TypeVariable`
`VoidType`	`javax.lang.model.type.NoType`
`WildcardType`	`javax.lang.model.type.WildcardType`

Package `com.sun.mirror.util`
`apt` type	Standard Replacement
`DeclarationFilter`	`javax.lang.model.util.ElementFilter`
`DeclarationScanner`	`javax.lang.model.util.ElementScanner6`
`DeclarationVisitor`	`javax.lang.model.element.ElementVisitor`
`DeclarationVisitors`	No replacement.
`Declarations`	`javax.lang.model.util.Elements`
`SimpleDeclarationVisitor`	`javax.lang.model.util.SimpleElementVisitor6`
`SimpleTypeVisitor`	`javax.lang.model.util.SimpleTypeVisitor6`
`SourceOrderDeclScanner`	`javax.lang.model.util.SimpleElementVisitor6`
`SourcePosition`	No replacement.
`TypeVisitor`	`javax.lang.model.element.TypeVisitor`
`Types`	`javax.lang.model.util.Types`

What is apt?

The command-line utility apt, annotation processing tool, finds and executes annotation processors based on the annotations present in the set of specified source files being examined. The annotation processors use a set of reflective APIs and supporting infrastructure to perform their processing of program annotations (JSR 175). The apt reflective APIs provide a build-time, source-based, read-only view of program structure. These reflective APIs are designed to cleanly model the Java^TM programming language's type system after the addition of generics (JSR 14). First, apt runs annotation processors that can produce new source code and other files. Next, apt can cause compilation of both original and generated source files, thus easing the development cycle.

Why Should I Use apt?

Many of the intended use cases for annotations involve having annotations in a base file hold information that is used to generate new derived files (source files, class files, deployment descriptors, etc.) that are logically consistent with the base file and its annotations. In other words, instead of manually maintaining consistency among the entire set of files, only the base file would need to be maintained since the derived files are generated. The apt tool is designed for creating the derived files.

Compared to using a doclet to generate the derived files based on annotations, apt

has a cleaner model of the declarations and current type structure of programs
uses a more contemporary API design, such as returning generic collections instead of arrays and providing visitors to operate on declarations and types
supports recursive processing of newly generated files and can automatically cause compilation of original and generated source files

While intended for annotation processing, apt can be used for other reflective programming tasks too.

How to use apt

Overview

First, apt determines what annotations are present on the source code being operated on. Next, apt looks for annotation processor factories you've written. The tool asks the factories what annotations they process. Then apt asks a factory to provide an annotation processor if the factory processes an annotation present in source files being operated on. Next, the annotation processors are run. If the processors have generated new source files, apt will repeat this process until no new source files are generated.

Developing an Annotation Processor

Writing an annotation processor relies on four packages:

com.sun.mirror.apt: interfaces to interact with the tool
com.sun.mirror.declaration: interfaces to model the source code declarations of fields, methods, classes, etc.
com.sun.mirror.type: interfaces to model types found in the source code
com.sun.mirror.util: various utilities for processing types and declarations, including visitors

Each processor implements the AnnotationProcessor interface in the package com.sun.mirror.apt. This interface has one method -- process -- used by the apt tool to invoke the processor. A processor will "process" one or more annotation types.

A processor instance is returned by its corresponding factory -- an AnnotationProcessorFactory. The apt tool calls the factory's getProcessorFor method to get hold of the processor. During this call, the tool provides an AnnotationProcessorEnvironment. In the environment the processor will find everything it needs to get started, including references to the program structure on which it is operating, and the means to communicate and cooperate with the apt tool by creating new files and passing on warning and error messages.

There are two ways a factory can be found; the factory to use can be specified via the "-factory" command line option or the factory can be located during the apt discovery procedure. Using the "-factory" option is the simplest way to run a single known factory; this option may also be used when a factory needs more control over how it is run. To locate the factories on a particular path, the discovery procedure retrieves from jar files META-INF/services information in the format described below.

To create and use an annotation processor using the "-factory" option:

Write an AnnotationProcessorFactory that in turn can create an AnnotationProcessor for the annotation type(s) of interest.
Compile the processors and factories using javac with tools.jar on the classpath; tools.jar contains the com.sun.mirror.* interfaces.
Put the compiled class files, or jar files containing the class files, on the appropriate path when invoking apt

To create and use an annotation processor with the default discovery procedure use the same first two steps then,

Create a UTF-8 encoded text file in META-INF/services named com.sun.mirror.apt.AnnotationProcessorFactory whose contents are a list of fully qualified names of the concrete factory classes, one per line. (This is the same format used by sun.misc.Service.)
Package the factory, processor, and META-INF/services information into a jar file
Place the jar file on the appropriate path when invoking apt. The appropriate path is discussed in the Discovery section.

A Simple Sample Annotation Processor

import com.sun.mirror.apt.*;
import com.sun.mirror.declaration.*;
import com.sun.mirror.type.*;
import com.sun.mirror.util.*;

import java.util.Collection;
import java.util.Set;
import java.util.Arrays;

import static java.util.Collections.*;
import static com.sun.mirror.util.DeclarationVisitors.*;

/*
 * This class is used to run an annotation processor that lists class
 * names.  The functionality of the processor is analogous to the
 * ListClass doclet in the Doclet Overview.
 */
public class ListClassApf implements AnnotationProcessorFactory {
    // Process any set of annotations
    private static final Collection<String> supportedAnnotations
        = unmodifiableCollection(Arrays.asList("*"));

    // No supported options
    private static final Collection<String> supportedOptions = emptySet();

    public Collection<String> supportedAnnotationTypes() {
        return supportedAnnotations;
    }

    public Collection<String> supportedOptions() {
        return supportedOptions;
    }

    public AnnotationProcessor getProcessorFor(
            Set<AnnotationTypeDeclaration> atds,
            AnnotationProcessorEnvironment env) {
        return new ListClassAp(env);
    }

    private static class ListClassAp implements AnnotationProcessor {
        private final AnnotationProcessorEnvironment env;
        ListClassAp(AnnotationProcessorEnvironment env) {
            this.env = env;
        }

        public void process() {
            for (TypeDeclaration typeDecl : env.getSpecifiedTypeDeclarations())
                typeDecl.accept(getDeclarationScanner(new ListClassVisitor(),
                                                      NO_OP));
        }

        private static class ListClassVisitor extends SimpleDeclarationVisitor {
            public void visitClassDeclaration(ClassDeclaration d) {
                System.out.println(d.getQualifiedName());
            }
        }
    }
}

A number of new language and library features are used in the sample processor. First, static imports are used so that the simple name of various utility methods can be used; for example
"unmodifiableCollection"
instead of
"Collections.unmodifiableCollection".
Second, generic collections are used throughout. The Arrays.asList method is now a var-args method so it can accept a comma separated list of strings and create a list with the desired elements. The Collections.emptySet method is a generic method and can be used to create a type-safe empty set. The for loop in the process method is an enhanced for loop that can iterate over collections.

Specifying the Annotations to Process

To tell the tool what annotations it processes, a factory returns a collection of import-style strings, as shown in the example. A particular string entry may have one of the following three forms:

*: Process all annotations. This also processes an empty list of annotations; in other words, a factory that processes * could be asked to provide a non-trivial processor even if no annotations are present. This capability allows the com.sun.mirror APIs to be used to write general source code processing tools.
foo.bar.Baz: Process an annotation whose canonical name is "foo.bar.Baz".
foo.bar.*: Process annotations whose canonical names start with "foo.bar.".

The apt tool presents the factory with a set of annotations for the factory to process. Based on the set of annotations and the annotation processor environment, the factory returns a single annotation processor. What if the factory wants to return multiple annotation processors? The factory can use com.sun.mirror.apt.AnnotationProcessors.getCompositeAnnotationProcessor to combine and sequence the operation of multiple annotation processors.

Specifying the Command Line Options to Recognize

The supportedOptions method allows a factory to communicate to apt which command line options it recognizes. Command line options starting with "-A" are reserved for communicating with annotation processors. For example, if this factory recognizes options such as -Adebug and -Aloglevel=3, it will return the strings "-Adebug" and "-Aloglevel". In the future, apt may give an indication if -A options are given that no factory recognizes.

The apt Command Line

In addition to its own options, the apt tool accepts all of the command-line options accepted by javac. The javac options are passed to the final javac call, if any.

The apt specific options are:

-s dir: Specify the directory root under which processor-generated source files will be placed; files are placed in subdirectories based on package namespace
-nocompile: Do not compile source files to class files.
-print: Print out textual representation of specified types; perform no annotation processing or compilation.
-A[key[=val]]: Options to pass to annotation processors -- these are not interpreted by apt directly, but are made available for use by individual processors
-factorypath path: Specify where to find annotation processor factories; if this option is used, the classpath is not searched for factories.
-factory classname: Name of AnnotationProcessorFactory to use; bypasses default discovery process

How apt shares some of javac's options:

-d dir: Specify where to place processor and javac generated class files
-cp path or -classpath path: Specify where to find user class files and annotation processor factories. If -factorypath is given, the classpath is not searched for factories.

There are a few apt hidden options that may be useful for debugging:

-XListAnnotationTypes: List found annotation types
-XListDeclarations: List specified and included declarations
-XPrintAptRounds: Print information about initial and recursive apt rounds
-XPrintFactoryInfo: Print information about which annotations a factory is asked to process

How the apt Tool Operates

Discovery

After scanning the source files on the command line to determine what annotations are present, by default the apt tool looks for annotation processor factories on the appropriate path. If the -factorypath option is used, that path is the appropriate path to search for factories; otherwise, the classpath is the appropriate path. The factories are queried to determine what annotations they process. If a factory processes one of the annotations present, that annotation is considered claimed. Once all annotations are claimed, the tool does not look for additional factories. After the annotations are all claimed or no more factories can be found, apt will call the factories' getProcessorFor methods, passing in the set of annotations that factory has claimed. Each factory returns a single processor to perform the appropriate processing for the set of annotations in question. After all processors are returned, apt calls each processor in turn. If any processor generated a new source file, a recursive round of apt will occur. In recursive apt rounds, discovery calls getProcessorFor on any factory that provided a processor in a previous round, even if that factory processes none of the current annotations. This allows the factory to register a listener in subsequent apt rounds; though most factories will simply return AnnotationProcessors.NO_OP in this case. After a round where no new source files are generated, apt will invoke javac on the original and generated source files. If no processors are found or the processors found don't process the annotations present, calling apt is essentially equivalent to calling javac directly on the source files.

If a factory class is used by more than one round of annotation processing, the factory class is loaded once and the factory's getProcessorFor method will be called once per round. This allows a factory to store static state across rounds.

If the -factory option is used, the named factory is the only one queried.

Rounds of apt Processing

The first round of apt analyzes the input source files, runs the discovery procedure, and calls the resulting annotation processors. The second round of apt analyzes the new source files produced by the first round (if any), runs the discovery procedure on those new files, and calls the resulting annotation processors. Likewise, if the second round has produced new source files, the third round analyzes the new source, runs discovery, etc. The apt rounds continue until no new source files are generated. Finally, after the last round, by default the apt tool will run javac on the original and generated source files.

Listeners

Annotation processors or factories can register listeners for the end of a round using the addListener method in the environment. The tool calls the registered listeners after all annotation processors for that round have run to completion. The listener is passed information about the status of the round, such as if any new source files were written, if an error was raised, and if the just completed round was the last round. Listeners can be used to write out trailing ends of files when all annotation processing has completed. The same class can implement both the AnnotationProcessor and RoundCompleteListener interfaces so the same object can serve in both contexts.

Return Code

If javac is invoked after the last apt round, the return code of apt will be the return code of javac compiling those files. If javac is not invoked, apt will have a 0 exit status if no errors were reported, either by the tool itself or by processors. Operating on malformed or incomplete source files in and of itself is not sufficient to cause apt to have a nonzero exit status.

Declarations and Types

The mirror API represents source code constructs principally through the Declaration interface and its hierarchy of subinterfaces, in the package com.sun.mirror.declaration. A Declaration represents a program element such as a package, class, or method, and typically corresponds one-to-one with a particular fragment of source code. Declarations are the structures that may be annotated.

Types are represented by the TypeMirror interface and its hierarchy of subinterfaces in the package com.sun.mirror.type. Types include primitive types, class and interface types, array types, type variables, and wildcard types.

The API is careful to distinguish between declarations and types. This is most significant for generic types, where a single declaration can define a whole family of types. For example, the declaration of the class java.util.Set corresponds to

the parameterized type java.util.Set<String>
the parameterized type java.util.Set<Number>
the parameterized type java.util.Set<T> for some type T other than String or Number
the raw type java.util.Set.

A declaration has doc comments, a source position, modifiers, and annotations. A declaration may have different kinds of names (simple, qualified). More specific declaration subclasses provide additional information appropriate for that construct. For example, class declarations provide access to constructors and the superclass. A declaration for an enum has a method to provide the enum constants.

TypeMirrors are used to model the return types, parameter types, etc., in source code. A TypeMirror for a reference type provides a mapping from a type to corresponding declaration; for example from the type mirror for java.util.Set<String> to the declaration for java.util.Set.

FAQs

Do you know about the Debian Advanced Packaging Tool?
Yes.
How does an annotation processor compare to a doclet?
The two entities certainly have some similarities; both use build-time reflective APIs to process source code. However, the mirror APIs used in annotation processors better model the current type system of the Java programming language. Also, by default the annotation processor(s) to run are determined by the annotations present in the source code rather than only on a mechanism analogous to the -doclet option. In other words, instead of running a single fixed doclet, apt dynamically chooses potentially multiple processors to run.
How can I process annotations on local variables?
Annotations on local variables are invisible since the declaration hierarchy does not model program structure inside a method or constructor.
Why are there separate factory and processor interfaces in com.sun.mirror.apt?
It would be possible to combine determining how to process and controlling when to process; however, we choose to distinguish and separate the two steps with factory and processor interfaces.
What is a mirror?
Somewhere you find a reflection. In reflective programming, a mirror design maintains a consistent separation between the objects being represented and the objects doing the representing.
Why were the annotations a factory supports and the options a factory supports returned by methods instead of being encoded as annotations on the factory class?
Implementing an interface is the usual mechanism to indicate a class has a desired capability, such as providing an indication of what annotations it processes. There is no language mechanism to require a class be annotated with a particular annotation type. Therefore, while it would have been technically possible to encode such information in annotations, doing so would have lacked the type safety of just having methods in an interface.
I'm not familiar with visitors, do I have to use them?
The visitor pattern is one of the standard patterns in the "Gang of Four" Design Patterns book. The pattern provides a powerful mechanism to invoke a type-dependent operation without explicitly testing for the type. However, using visitors is not mandatory.