Build Tools - An incomplete paradigm - part 2

I am in the midst of making my build system for the saltations project usable by everyday developers. i.e. I am working to make it a turnkey system and in the process came across this entry in Kent Spillner's blog and nearly split my side laughing.

I don't know is he meant it to be funny but it was an accurate summation of most of my complaints about Maven and I liked his expression of it:

http://kent.spillner.org/blog/work/2009/11/14/java-build-tools.html

Ah, well, back to work.

The build lifecycle

One of the things that is critical to any build system ( and this is something that the Maven guys nailed) is that it have a clearly defined lifecycle.

For myself over the last 20 years I have developed a build lifecycle that appears to answer my needs for every build.

• Clean - Remove all build artifacts from the file system.
• Init - Initialize the build system. All properties should be set here.
• Prep - Prepare the file system for the build. Create folders as necessary.
  
• GetDependencies - Resolve all dependencies. Locate and pull down the necessary dependencies and make them available to the build
• Gen - Generate or Preprocess source code and resources of any type.
• Build - Build anything that can be compiled or assembled.
• UnitTest - Execute any tests that can be performed without deployment
• Pkg - Package anything that can be packaged
• Verify - Establish the internal integrity of the package
• Deploy - Deploy any packages that need to be deployed.
• SmokeTest - Execute any smoke tests against the deployed application(s)
  
• Stage - Publish the build artifacts for local (i.e. machine local) usage.
  
• Share - Publish the build artifacts for team wide usage.
• Release - Publish the build artifacts for general (i.e. network) usage.
• IntegrationTest - Execute any smoke tests

Most of that is not new. You have probably seen build systems that use subsets or supersets of these. The key is in making it very easy to plug in to a lifecycle event so that you can perform additional actions as necessary. Doing that in Ant takes some very careful thought but produces some very clean results.

It does require some conventions on the project and source code structure side of things. Another thing that the Maven guys nailed. :-)

The directory structure conventions I am using with my build system are the following:

I have a top level [project] Folder with several folders underneath.

• [project]/root - This is the top level location from which you do builds that execute against everything. Commands like "CleanAll", "BuildAll", etc...
  
• [project]/common-build - This contains the common build system I use across all projects. It is a separate project in my revision control system and is typically pulled down into a location under the project using Subversion's externals command or using Git's submodules.
  
• [project]/components - This folder contains all of the sub projects that make up the components of this project.
  
• [project]/apps - This folder contains all of the subprojects that make up the applications of this project.
  
• [project]/installers - This folder contains all of the subprojects used to generate the installers of this project.
  

Under each components or apps or installers folder is a named subproject with a standard structure.

For example: The project "calendar-server" with the "calendar-utils" component would look like this:

calendar-server/components/calendar-utils

The component has a series of files. The build.xml file is of course the ant file for the component. The ivy.xml is a very simple file that describes what artifacts the component is dependent on. And the .project and .classpath files are those files needed by Eclipse.

calendar-utils/build.xml
calendar-utils/ivy.xml
calendar-utils/.project
calendar-utils/.classpath

And the calendar-utils component would have a checked-in folder structure that looks like this:

calendar-utils/src/main/java
calendar-utils/src/main/conf
calendar-utils/src/test/java
calendar-utils/src/test/conf

The source trees are separated for several reasons. Often the Java test code is compiled , managed, preprocessed, executed or referenced separately from the main code. The conf folders are also separate for some of the same reasons as well as allowing separate configurations to be set up for test and actual deployment.

There are also a series of common folders that are typically generated during the build. the first set of those is for generated code and those follow the same conventions as the checked source for much the same reasons.

calendar-utils/gen-src/main/java
calendar-utils/gen-src/main/conf
calendar-utils/gen-src/test/java
calendar-utils/gen-src/test/conf

The other set of folders is for the targets or results of the build. They all lie under the target folder. The classes and test classes are kept separate for the same reasons the sources are kept separate. The distrib folder contains all of the final output artifacts of the build ( jars, documents, zip files and executables etc.). The reports folder contains reports on the tests, profiling, etc.

calendar-utils/target/classes
calendar-utils/target/test-classes
calendar-utils/target/distrib
calendar-utils/target/reports

So to clean up this directory structure for a new pristine build only requires us to delete the gen-src and target folders.

In the next post I will talk about the design of the Ant build files found in the components and the common-build folder.