<JWare/AntXtras Simple Artifact Management (SAM)>

[home] [overview] [download] [learning] [problems] [ourplans] [contacts] [legalese]

[overview]

home » overview

This site is no longer active; it remains as a historical reference.
For the latest version of JWare Software visit jwaresoftware.org

Introduction

SAM’s Four Cornerstones

The JWare/AntXtras Simple Artifact Management package (SAM or SAMS) is a set of Ant components that help you manage the significant outputs and useful by-products of your build or test processes. SAM is not a formal project management tool like Maven, nor is it a dependency manager like Ivy. Instead, SAM is an enhancement of the way you define and organize plain old Ant scripts (or POAS ;-)). SAM is useful to anyone who creates or manages multiple build or test environments built using Ant scripts that have lots in common (in intent if not necessarily in implementation).

The rest of this overview will introduce you to the most important SAM components and demonstrate some of their uses in a build program. See the SAM user [guides] for a detailed catalog of all SAM components and their options. Peruse the SaladBar demo for examples of ways you can use SAM in your own build system. Subscribe to the JWare/AntXtras announcements mailing list to be notified of new SAM releases.

Where’s My Stuff?

The SAMS world revolves around the custom “stuff” we produce directly and indirectly when we run our Ant scripts; stuff like deployable war files and product documentation and test results. In SAMS, we call this stuff artifacts and it is the lowest level of detail you will see in most SAM Ant scripts. The details regarding how artifacts are created, saved, and retrieved are moved into external antlibs and presetdefs that, like standard programming libraries, remain largely black-box entities to the main Ant script. The two requirements for something to be considered a SAM artifact are that it be a tangible, externally locatable entity and that it be somehow produced or located by an Ant script. Examples of valid SAM artifacts include java archives, log files, CVS sandboxes, and database records. Examples of things that cannot be called SAM artifacts include Ant properties and active JVM processes (like a thread).

Below is part of a SAM based Ant file for a Java library project. The three targets libraries, testables, and testreport create the build process artifacts that make up the client artifact reflected in the target’s name (the “stuff”). In a succinct manner, each target tells you the important outputs it produces; this is one of the main goals of SAM.

    <target name="libraries" depends="newfixture"
            description="Make main libraries">
        <required-thirdparty-jars/>
        <templatesources-java/>
        <javaclasses-main/>
        <libraryarchives-main/>
        <statusfile-libraries/>
    </target>

    <target name="testables" depends="libraries"
            description="Make main libraries and tests">
        <required-thirdparty-testjars/>
        <javaclasses-programmertests/>
        <required-thirdparty-runconfig/>
        <statusfile-testables/>
    </target>

    <target name="testreport" depends="testables"
            description="Make complete testset report">
        <programmertests-baselines/>
        <statusfile-baselinetests/>
        <programmertests-all/>
        <statusfile-programmertests/>
        <reports-programmertests allif="tests.dirty"/>
    </target>

Example 1: Part of a SAM based Ant script that builds a Java library.

SAMS and AntX

SAM is based on our foundation JWare/AntXtras package AntX. SAM enhances AntX in the following important ways:

SAM greatly increases your ability to categorize, share, and reuse artifacts of your organization's build and test system(s) without requiring a particular view of product creation or project artifact organization. To this end, SAM relies heavily on three core Ant capabilities: antlibs, presetdefs, and macrodefs.
SAM supports fully the concept of a single build-iteration that crosses target, project, and process boundaries. While AntX introduces the build-iteration, SAM’s usefulness is linked directly to its support for iterations instead of insular projects and sub-projects.
SAM adds extensible event monitoring to build-iterations and client-controlled decoration to the primary artifact generating activities executed during an iteration. These features extend the script-based feedback components found in AntX by giving you a way to hook into a SAM component’s internal processing.
SAM provides a complete solution to artifact management. And like AntX, SAM works equally well as a collection of standalone components (in a build or test management system) and as an artifact management service for other Ant-based development tools.

User License Agreement

JWare/AntXtras SAM, binary and source form, is released under the Free Software Foundation’s GNU LGPL; a copy of this license can be found on the Free Software Foundation’s website at http://www.fsf.org/licenses/lgpl.html. Please read the LGPL carefully before using any of the SAM source in your own application.

SAM and its documentation are written for experienced Ant users and developers. SAM is not intended for someone new to Ant or for someone evaluating Ant unless you are concerned about the issues that SAM specifically addresses.

[↑ Back to Top]

SAM’s Four Cornerstones

The SAM system is based on four cornerstones: [1] Artifact Retrieval Mechanisms (ARMs), [2] Ant Library Management Services (ALMS), [3] Artifact Definitions (artifactdefs), and [4] Artifact Announcement Callbacks (reactiondefs). How these four collaborate with each other and a build-iteration is shown generally in the diagram below.

Figure 1: SAMS Component Collaboration Overview.

[1] Artifact Retrieval Mechanisms (ARMs)

The first cornerstone of the SAM system is its Artifact Retrieval Mechanisms or ARMs. ARMs combine Ant’s macro and antlib features to let you define independent, pluggable functions for your scripts. The purpose of an ARM is to produce an artifact that is in a finished, publishable form where you or your organization has decided what is meant by “publishable”. In general, you create an ARM to combine the generic functions we all get with Ant in a way that produces an outcome that is precisely what you need for your specific purpose.

Unlike the general Ant antlib and macrodef functions, ARMs have a clearly defined purpose and must conform to a prescribed template in order to work within the SAM system. And although they are based on the Ant macro facility, you would almost never call an ARM directly from your scripts. Below is an example “mktimestamps” ARM script. In this example, the “timestamps” artifact is the file that contains the final, formatted timestamp strings. The artifact’s location is returned by the ARM through the required artifactlocation argument. This argument is always provided by the SAM system and it contains the name of the AntX variable your ARM must fill in.

    <!--
      | ARM: mktimestamps
      | PURPOSE: Creates a loadable properties file of common iteration timestamps.
      |          Works for single and multi project (driver->subbuild) uses.
      -->
    <arm>
        <macrodef name="mktimestamps">
          <attribute name="artifactlocation"/>
          <attribute name="in" default="${user.home}/.builds"/>
          <attribute name="file" default="build.timestamps"/>
          <attribute name="apply" default="no"/>
          <attribute name="overwrite" default="yes"/>
          <sequential>
            <overlay-msgs file="${mktimestamps.alms.basedir}/feedback.strings">
              <assert isdirectory="@{in}"  msg="Directory '@{in}' exists"/>
              <assign var="@{artifactlocation}" value="${$ospath:@{in}/@{file}}"/>

              <tally logic="or" truevariable="_continue_mktstamps">
                <istrue value="@{overwrite}"/>
                <isfalse value="${$filenotempty:@{in}/@{file}}"/>
                <tally>
                  <isnotset property="ITID_STAMP"/>
                  <istrue value="@{apply}"/>
                </tally>
              </tally>

              <do true="${$var:_continue_mktstamps}">
                <emit msgid="msg.gen.timestamps" level="info"/>
                <macrolocals>
                  <property name="BASE_STAMP"/>
                  <property name="LONG_STAMP"/>
                  <property name="ITID_DATE"/>
                  <property name="ITID_STAMP"/>
                  <property name="CHANGELOG_DATE"/>
                  <property name="DSTAMP"/>
                  <property name="TSTAMP"/>
                  <property name="TODAY"/>
                </macrolocals>
                <macroisolate>
                  <tstamp>
                    <format property="BASE_STAMP"     pattern="MMMddyyyy'_'kkmmss'GMT'" timezone="GMT"/>
                    <format property="LONG_STAMP"     pattern="MMMddyyyy'_'kkmmss"/>
                    <format property="ITID_DATE"      pattern="MMM-dd-yyyy"/>
                    <format property="ITID_STAMP"     pattern="MMM-dd-yyyy'_'hhmm"/>
                    <format property="CHANGELOG_DATE" pattern="dd MMM yyyy"/>
                  </tstamp>
                  <newfile path="@{in}/@{file}" overwrite="yes">
                    <line value="BASE_STAMP=${BASE_STAMP}"/>
                    <line value="LONG_STAMP=${LONG_STAMP}"/>
                    <line value="ITID_DATE=${ITID_DATE}"/>
                    <line value="ITID_STAMP=${ITID_STAMP}"/>
                    <line value="CHANGELOG_DATE=${CHANGELOG_DATE}"/>
                  </newfile>
                </macroisolate>
              </do>
              <do true="@{apply}">
                <property file="@{in}/@{file}"/>
              </do>
            </overlay-msgs>
          </sequential>
        </macrodef>
    </arm>

Example 2: An example ARM Ant script for a timestamps file.

[↑ Back to Top]

[2] Ant Library Management Service (ALMS)

Template format aside, so far ARMs sound an awful lot like what the standard Ant macrodef and antlib provide out-of-the-box. But remember, one of SAM’s goals is to enhance your ability to categorize, share, and reuse elements of your plain Ant scripts. In isolation, ARMs provide only one part of the solution. The second SAM cornerstone, the Ant Library Management Service or ALMS, is SAM’s implementation of an antlib fetch and apply on-demand service. ALMS lets you dynamically locate and load ARMs (or any antlib) from any source that is reachable with a URL. That means, from a local Ant script, you can load and use ARMs stored in local directories or ARMs stored in remote datastores (including those that are not http based like CVS repositories or relational databases).

Although ALMS separates the notions of a remote (pristine) antlib repository and a local (sandbox) antlib workspace, ALMS is not a dependency management tool. There are several open source projects working on such tools but SAM is not one of them. In fact, we expect many users will combine SAM with a repository synchronization tool so we will work hard to ensure we function well in such a scenario. Having said that, if you do not use a dependency tool but still want to store your ARMs and other antlibs in a private database or CVS repository (for instance), each ALMS instance lets you supply an Ant-based criteria that it will use to determine if a local ARM package is up-to-date. If the criteria returns false, the ALMS instance will try to refetch the antlib using your URL. Below is an example criteria downloaded-today that flags as out-of-date any ARM package that was not downloaded (or extracted) on the same day as the active build-iteration.

    <criteria id="downloaded-today" resultvar="out:downloaded-today">
        <property name="_datefile"
                  value="${alms.module.dir}/.crdate"/>

        <do true="${$filenotempty:@(_datefile)}">
            <loadfile srcfile="${_datefile}"
                      property="_alms.crdate"/>
            <do true="${$equaldate:@(CHANGELOG_DATE),,@(_alms.crdate)}">
                <assign var="out:downloaded-today" value="yes"/>
            </do>
        </do>
    </criteria>

Example 3: An example criteria that will trigger a refetch of any antlib that was not fetched today.

Often you will write an ARM that creates artifacts that are unique and useful only to you; however, with SAM you can also create ARMs that work like high level packaging solutions that others might find useful. And because SAM lets you retrieve ARMs from other sources, you might find the ARMs someone else has created useful in your own work. If you or your team is responsible for some of the product deliveray tools in an organization, SAM gives you a simple mechanism for organizing and distributing your Ant scripts using standard antlibs.

[↑ Back to Top]

[3] Artifact Definitions (artifactdef)

The third cornerstone of the SAM system, Artifact Definitions or artifactdefs connect an ALMS instance and one or more repositories of ARMs to your application through dynamically generated Ant script elements. An artifact definition lets you create a set of related elements, Ant presetdefs, that each embody the unique combination of parameters and fixture setup that is needed to invoke a single ARM in a slightly different way. And like ARMs, SAM lets you use artifactdefs from your private build environment or from trusted third-parties.

If we think of our Ant targets as reflecting the final baked results of our build or test process, “the stuff”, SAM artifact definitions are like the course ingredients with which those final results are produced. Most of the XML you use in an SAM Ant script are presetdefs that some artifactdef generated. Other specific benefits of artifact definitions include: they eliminate the redundancy from ARM calls that differ by one or two parameters; they hide non-trivial implementation details associated with seemingly trivial artifact variations; they lock down ARM invocation contexts and associate those contexts with semantically meaningful artifact names; they can link one ARM’s output and byproducts to another ARM’s inputs via location properties; they can ensure a particular artifact is generated at most once during a build-iteration; they can be linked automatically to an artifact cleanup method (called a cleanupdef); and they are based on a hard architectural model that can be enforced (unlike standard presetdefs which can be altered always from the client Ant script).

Based on the mktimestamps ARM described in an earlier section, the next example shows how we can create a custom artifactdef buildtimestamps that embodies how we intend to use the general ARM definition in our build. The artifactdef antlib shields most of our build scripts from knowing about ARMs and SAM. Instead, from our Ant scripts we can use the SAM-generated presetdef <buildtimestamps/> to create our timestamp file.

    [In our artifactdef antlib]
    <add-artifactdefs>
        <artifactdef name="buildtimestamps">
            <defaults>
                <method use="mktimestamps"/>
                <version use="standard"/>
                <parameter name="in" value="${buildmeta.dir}"/>
            </defaults>
            <version name="standard">
                <parameter name="apply" value="yes"/>
                <parameter name="overwrite" value="no"/>
            </version>
            <version name="snapshot">
                <parameter name="file" value="tstamps"/>
            </version>
        </artifactdef>
        …[more]
    </add-artifactdefs>

    [In our actual build script]
    <target name="newfixture" depends="typedefs">
        <buildtimestamps/>
        …[more]
    </target>

Example 4: An example artifact definition for our mktimestamps ARM.

While you can use SAM to create simple artifacts like our sample timestamps file, its real utility surfaces when you have nontrivial artifact packaging logic that must accomodate many independent control options that each affect the final outcome. Below is part of a build script that generates a pair of documentation packages latest-docs and stable-docs. The bulk of the artifact(s) packaging logic has been off-loaded to independent ARMs and a set of artifactdefs thereby making the driver script more readable, maintainable, and extensible. In the example, variants of the following artifacts are used: readmes, javadocs, examples, guides, umldocs, and alldocs. Note that each “.loc” property is generated by the SAM system when the matching artifactdef is called. The definition for the readmes artifact is also shown to emphasize that each artifactdef variant can produce very different outcomes.

    <--
     | Generate and package documents of all development trunks.
     -->
    <target name="latest-docs" depends="sources,…">
        <parallel timeout="${10mins}" failonany="yes">
            <readmes/>
            <javadocs/>
            <examples-latest/>
            <guides-tip/>
        </parallel>

        <datadef name="latest-docs.list">
            <properties>
                <property name="readmes"  value="${readmes.loc}"/>
                <property name="javadocs" value="${javadocs.loc}"/>
                <property name="examples" value="${examples-latest.loc}"/>
                <property name="guides"   value="${guides-tip.loc}"/>
            </properties>
        </datadef>

        <alldocs-packaged listref="latest-docs.list"
                          locationproperty="latest-docs.loc"/>
    </target>

    <--
     | Generate and package documents of last public milestone.
     -->
    <target name="stable-docs" depends="stable-sources,…">
        <parallel timeout="${10mins}" failonany="yes">
            <readmes-stable/>
            <javadocs/>
            <examples/>
            <guides/>
            <umldocs-pdf/>
        </parallel>

        <datadef name="stable-docs.list">
            <properties>
                <property name="readmes"  value="${readmes-stable.loc}"/>
                <property name="javadocs" value="${javadocs-stable.loc}"/>
                <property name="examples" value="${examples.loc}"/>
                <property name="guides"   value="${guides.loc}"/>
                <property name="archdocs" value="${umldocs-pdf.loc}"/>
            </properties>
        </datadef>

        <alldocs-packaged listref="stable-docs.list"
                          locationproperty="stable-docs.loc"/>
    </target>

    [ - Extract of the artifactdefs antlib script ----------------------]
    <artifactdef name="readmes">
        <defaults>
            <method use="mkreadmes"/>
            <version use="latest">
            <parameter name="for" value="@{package}-readmes-pkg"/>
            <parameter name="uploads" value="@{docs.dir}/readmes"/>
            <parameter name="clean" value="yes"/>
            <parameter name="license" value="MIT"/>
            <parameter name="saladbar" value="@{saladbar.dir}"/>
            <announcement>
               <include parameter="for"/>
            </announcement>
        </defaults>
        <version name="latest">
            <parameter name="downloads" value="${cvs.modules}"/>
        </version>
        <version name="stable">
            <parameter name="tag" value="r1-1build123"/>
            <parameter name="downloads" value="${cvs.modules}/r1-1build123"/>
        </version>
        <version name="xmlprototype" announce="no">
            <method use="mkreadmes-xml"/>
            <setup>
                <directory-stylesheets name="readmes"/>
            </setup>
            <parameter name="downloads" value="${cvs.modules}/branch-xml"/>
            <parameter name="xsl" value="@{stylesheets-readmes.loc}"/>
        </version>
    </artifactdef>

[↑ Back to Top]

[4] Artifact Announcements and Reactions

The fourth and final cornerstone of the SAM system, Artifact Announcements lets you customize somewhat a set of pre-existing readonly ARM and artifact definitions. Basically SAM lets you listen for artifact creation so you can take some action in response, always. You can also associate your action (called a reaction) with an artifact’s pre-creation announcement. Announcements are useful when you want an action performed everytime an artifactdef is called regardless of the variant that’s being used.

To demonstrate how you might use artifact announcements, let’s say we have a pre-existing thirdparty SAM-based build system that generates javadocs as one of the outputs for our daily integration builds. Let’s also assume we have a business requirement that our javadocs be of high quality as they are released as part of our company’s products. As a first step to ensure quality, our chief programmers want to know when the standard javadoc complains about our sources and they want these “complaints” to permeate our internal build documents so no one can claim to have missed them. Using Extreme Programming as a guide, one quick solution is to swap our javadocs stylesheets based on the kinds of complaints our source generates. If any warnings or errors are generated, the stylesheets reflect at least a “yellow traffic light” mode. If more than some threshold of errors or warnings is generated we reflect a “red traffic light” mode. The following screenshots demonstrate the effect:

Failure(Red)

Problem(Yellow)

Success

With artifact announcements we can tweak any javadocs artifact without access to either the mkjavadocs ARM or the javadocs artifactdef. We just have to install a reaction method that gets called whenever the build system generates javadoc html (identified by the javadocs-html artifact name) but before that html is further processed (like being archived or deployed to a public website). Below is the listing for our reaction script trafficlight-javadocs and the snippet of build script that installs it:

    <!--
      | NAME: trafficlight-javadocs
      | PURPOSE: Looks at the javadoc logs and tweaks stylesheet to reflect any issues.
      -->
    <antlib xmlns:i="antlib:com.idaremedia.sams.install" xmlns:c="ant:current">

        <taskdef name="interpretlog"
                classname="com.idaremedia.antx.capture.InterpretLoggedTask"/>

        <macrodef name="trafficlight-javadocs">
            <attribute name="artifactlocation"/>
            <attribute name="cssfile" default="stylesheet.css"/>
            <sequential>
                <i:assert isset="javadoc.log" msg="'javadoc.log' defined"/>
                <i:macrolocals>
                    <property name="_javadoc.result"/>
                </i:macrolocals>
                <i:macroisolate>
                    <c:interpretlog format="javadoc" logfile="${javadoc.log}"/>
                    <i:domatch property="_javadoc.result">
                        <i:equals value="problem">
                            <copy filtering="no"
                                  file="${saladbar.dir}/css/javadoc-problem-stylesheet.css"
                                  tofile="@{artifactlocation}/@{cssfile}"
                                  overwrite="yes"/>
                        </i:equals>
                        <i:equals value="failure">
                            <copy filtering="no"
                                  file="${saladbar.dir}/css/javadoc-failure-stylesheet.css"
                                  tofile="@{artifactlocation}/@{cssfile}"
                                  overwrite="yes"/>
                        </i:equals>
                    </i:domatch>
                </i:macroisolate>
            </sequential>
        </macrodef>

    </antlib>

    [ - Extract of the controlling build script ----------------------]
    <add-reactiondefs...>
        <reactionfor artifact="javadocs-html" call="trafficlight-javadocs"/>
        …
    </add-reactiondefs>

[↑ Back to Top]