Automating JavaScript builds

Automating JavaScript builds

– UPDATE (20/04/2012) –

This is an update to my original post on how to automate javascript builds using ant and a number of other tools, however, for most of my projects I’ve switched over to gruntjs. It’s far easier to use than Ant since it uses javascript and integrates most of my build tools pretty easily. Granted, it’s a little more limited than Ant, for instance: you can’t invoke shell commands from gruntjs (except the ones that have grunt tasks).
I also wrote a gruntjs plugin for running jasmine specs: grunt-jasmine-task.

Forget Ant, use gruntjs instead. :)

Below is the original post, just for sake of reference, but I consider it outdated, since I no longer use Ant, but Grunt instead.

– ORIGINAL POST (14/03/2012)–

What?

Sometimes even for smaller javascript projects it’s interesting to create an automated build process. From the moment there’s a few steps you need to take every time you want to release your application/library it’s beneficial to automate this process not only because it will save you time, but also because it will save you the headache of remembering to do every step every time.

We’re not talking about enterprise-level CI here, just automating a few necessary actions that will validate your JS files and make sure they’re ready for public/in-house consumption.

As a minimum I make sure my JS apps/libs are …

So that’s what we’re going to do: we’re going to automate the above steps. To be able to do that we need to somehow create a list of actions to take and tell our computer to execute them, conditionally and sequentially. There’s a huge number of possible scripting languages and build tools we could use, but I want to keep it simple and easy-to-use, since we’re working on a small project here, I’ll be using Apache Ant.

Example source

Download example source on github

All tools that don’t need to be explicitly installed are included with the code.
This was done for example purposes only. I have a central location where I drop all jars/js files/whatever and link to those locations from the ant build file, but I wanted to save you the trouble of having to download all the various bits and pieces to get you up and running ASAP.

In the below tools list, all applications that require installation are prefixed with an asterisk (*), so even if you just downloaded the example source, you’ll still need to download and install those packages.

Directory structure

  • The bin directory will be the destination directory into which our final .js files will be placed
  • the docs directory is the destination point for our API documentation
  • lib contains all 3-rd party tools and files needed for building
  • specs contains our unit tests
  • src contains the source .js files

Tools

We’ll be using a number of tools/applications/libs to achieve all of this, for each of them there are already a ton of resources on how to install them, so I wont go into detail on that.

(*) Apache Ant

Apache Ant is a Java library and command-line tool whose mission is to drive processes described in build files as targets and extension points dependent upon each other.

We’ll be using Apache Ant to automate the process and list out what needs to be done, how and when. Details will follow

Download Apache Ant

JSHint

JSHint is a code quality tool for JavaScript based on JSLint. I prefer JSHint for one simple reason: JSLint enforces you to move all your var declarations to the top of your functions. That’s just silly.

JSHint is a community-driven tool to detect errors and potential problems in JavaScript code and to enforce your team’s coding conventions.

It’s got a webinterface for validation, but we’ll be using the JSHint lib directly to validate our code locally and during the build process.

Download JSHint

The only thing we’ll be needing is the jshint.js file, so you can download it directly as well if you want.

No installation required, we’ll just drop it into the lib directory in its entirety.

JSDoc Toolkit

JsDoc Toolkit is an application, written in JavaScript, for automatically generating template-formatted, multi-page HTML (or XML, JSON, or any other text-based) documentation from commented JavaScript source code.

Download JSDoc Toolkit

No installation required, we’ll just drop it into the lib directory in its entirety.

YUI compressor

The goal of JavaScript and CSS minification is always to preserve the operational qualities of the code while reducing its overall byte footprint. [...] The YUI Compressor is JavaScript minifier designed to be 100% safe and yield a higher compression ratio than most other tools.

Download YUI compressor

No installation required, we’ll just drop it into the lib directory in its entirety.

Jasmine

Jasmine is a behavior-driven development framework for testing your JavaScript code.

Unit testing is essential in every language, but in JavaScript even more so, IMO. If you’re not into testing, no problem, it suffices to know for this example that with BDD you define how your code is expected to behave and run a number of tests to verify those expectations. A test passes if the expectation is fulfilled and fails if not. In Jasmine these tests are called specs.

Download Jasmine

No installation required, we’ll just drop it into the lib directory in its entirety.

(*) PhantomJS

PhantomJS is a headless WebKit with JavaScript API

Simply put: it’s a browser without a GUI client. Why’s this necessary? Since we’re automating this whole process we don’t want to be running our Jasmine tests in a browser, but from Ant. Unfortunately since the Jasmine specs runner (the file that lists out and calls all the tests) is HTML-only (there’s a number of alternatives, see below for explanation) we need to be able to parse and run that HTML file. That’s where PhantomJS comes in. It parses and renders the HTML file, waits for it to finish execution and then reads the results and passes them on to Ant.

Also, we’ll be using PhantomJS for running JSHint and parse it results.

Sidenote: there are alternatives like envjs and zombie, but since I didn’t like the envjs API and didn’t want to use Node.js for this, I chose PhantomJS. Another alternative was to run a console spec runner and catch the results in Rhino, but since I use the HTML spec runner anyway (for manual testing) this seemed the easiest option. Maybe it’s not.

Download PhantomJS

Putting it all together

Let’s take a look at our example:

The src files

Inside our src directory we find 3 .js files: build.example.js, Baz.js, Foo.js. buildexample.js defines the main containing object (ie. “namespace”) in which 2 class definitions will be put: Foo and Baz. Both have a single property (bar and qux respectively) and a single method (getBar and getQux respectively).

What we want to achieve is the following:

  • run a QA tool (JSHint) on all of the code in the .js files
  • run all unit tests
  • consolidate the 3 seperate .js files into one single .js file
  • minify that single file into a separate minified .js file
  • generate API documentation from the inline JSDoc comments in the .js files
  • use a version number in both the API documentation and in the file names of the consolidated and minified files

The build process should be aborted if any of these steps fail.

build.xml

The heart of it all is build.xml. It’s the Ant build file, that tells Ant what to do and when. I won’t be delving too deep into Ant build file syntax since, again, there are tons of resources out there. The official documentation for instance is pretty extensive and clear.

In Ant there are 3 concepts we need to know to proceed:

  1. Properties: we can define properties, they’re like constants, once a property is set it’s immutable. We’ll be using properties to make our build process configurable.
  2. Tasks: tasks are preconfigured actions that can be executed by Ant. They can be used to copy files, run an executable etc.
  3. Targets: a target is a named container that groups actions or calls other targets.

As you can guess the outlined above steps will translate into targets, that will be called in a certain sequence during our build process.

Version number

The first thing we’ll be doing is making sure a version number is defined for our build:

    <target name="version">
        <loadfile property="version.old" srcFile="version.txt" />
        <input message="Current version number is ${version.old}. Please enter the new version number:"
            defaultValue="${version.old}" addproperty="version"/>
        <echo file="version.txt" message="${version}" />
    </target>

It loads the value from version.txt file in the project root directory, then prompts us to either use that version number again or to enter a new one, which consecutively is written into the .txt file again. This version number can be any string value you want: v0.1, 1.0.0-RC2, etc

let’s test it out, in terminal:
Creynders-Laptop:dev creynder$ ant version
will present us with

Buildfile: /Users/creynder/Dropbox/Work/Projects/CREY - automate JS builds/dev/build.xml
version:
[input] Current version number is 1.0.0. Please enter the new version number: [1.0.0]

Property declaration

Only then the other properties will be declared and read from the build.properties file.
It also tries to read a build.user.properties file, which can be used to override some of the settings in build.properties. This user specific configuration file is also ignored by git, so it doesn’t get accidentally committed to our repo.

    <target name="properties">
        <tstamp>
            <format property="timestamp" pattern="yyyyMMddHHmmss"/>
        </tstamp>
        <!-- allow user-specific overrides -->
        <property file="build.user.properties"/>
        <property file="build.properties"/>
    </target>

Create build directory

Next we’ll be creating our build directory in which the temporary work files are to be placed. We won’t be placing everything directly in the bin directory just to make sure that if the build fails, the bin directory won’t be containing any “dirty”, faulty files.

    <target name="create_build" depends="properties">
        <echo>Creating build...</echo>
        <delete dir="${dir.build}" />
        <mkdir dir="${dir.build.current}" />
        <echo>Finished.</echo>
    </target>

Consolidation

On to the consolidation of our .js source files:

    <target name="consolidate" depends="properties, create_build">
        <echo>Consolidating...</echo>
        <copy file="${dir.src}/${name.base.js}" tofile="${file.consolidated.js}"/>
        <replace file="${file.consolidated.js}" token="%VERSION%" value="${version}"/>
        <replace file="${file.consolidated.js}" token="%TIMESTAMP%" value="${timestamp}"/>
        <concat id="srcfiles" destfile="${file.consolidated.js}" append="true">
            <fileset dir="${dir.src}" includes="**/*.js" excludes="${name.base.js}"/>
        </concat>
        <pathconvert pathsep=";" property="files" refid="srcfiles"/>
        <echo>${files}</echo>
        <echo>Finished.</echo>
    </target>

First it copies the base src file, in this case buildexample.js, to the build directory and renames it to buildexample-.js
Then it appends all of the other src .js files into our consolidated file.
Lastly it substitutes a few strings in our consolidated file: %VERSION% is replaced with the value from our version property and %TIMESTAMP% with the value of timestamp.

JSHint

Now we want to run JSHint on our consolidated code and make sure it adheres to proper syntax and formatting.

This is done through PhantomJS. It accepts as a first parameter a .js file, which tells phantom what to do. PhantomJS has a clean and easy-to-use API.

I wrote this phantom-jshint-runner.js file which includes the contents of the lib/jshint/jshint.js file and subsequently passes the contents of our consolidated js file to the JSHINT function. In the example source it can be found in the lib directory.
Please take a look at the file itself for an explanation of what happens exactly.

Then in our build.xml we need to call PhantomJS and pass it the required arguments:

    <target name="jshint" depends="properties, consolidate">
        <echo>Checking syntax...</echo>
          <exec executable="phantomjs" dir="${basedir}" failonerror="true" resultproperty="specs.results">
              <arg line="'${file.jshint-runner.js}'" />
              <arg line="'${file.jshint.js}'" />
              <arg line="'${file.consolidated.js}'" />
              <arg line="${timeout.phantom}" />
          </exec>
        <echo>Finished</echo>
    </target>

Jasmine

Jasmine tests are run from a .html file, in the example this is specs/index.html. Again we’ll be using PhantomJS to read and parse the html file and return the results to ant. I wrote phantom-jasmine-runner.js which can be found in the lib directory and was based on the default spec runner from the PhantomJS site.

We’ll be calling PhantomJS in the same fashion as for JSHint but with different arguments obviously.

    <target name="specs" depends="properties">
        <echo>Running specs...</echo>
        <exec executable="phantomjs" dir="${basedir}" failonerror="true" resultproperty="specs.results">
            <arg line="'${file.jasmine-runner.js}'" />
            <arg line="'${file.specs-runner.html}'" />
            <arg line="${timeout.phantom}" />
        </exec>
        <echo>Finished.</echo>
    </target>

Minifying

We’ll be using the YUI compressor to minify the contents of the consolidated file. Read the official documentation for a full explanation of YUI compressor usage.

It’s shipped as a .jar, so we’ll be running java and pass it the path to the yui compressor .jar file.
The command line execution looks like this:

java -jar yuicompressor-x.y.z.jar [options] [input file]

Which translates to the following ant build xml

    <target name="minify" depends="properties,create_build, consolidate">
        <echo>Minifying...</echo>
        <exec executable="java" dir="${basedir}" failonerror="true">
            <arg line="-jar '${file.yui_compressor.jar}'" />
            <arg line="--type js" />
            <arg line="-o '${file.minified.js}'" />
            <arg line="'${file.consolidated.js}'" />
        </exec>
        <echo>Finished</echo>
    </target>

JSDoc toolkit

First we’ll recreate the docs directory. Then we’ll be generating the docs files. JSDoc toolkit is run through the Mozilla Javascript Engine, Rhino, but fortunately JSDoc toolkit distributions include it out-of-the-box.

Commandline:

java -jar jsrun.jar app/run.js myscript.js -t=templates/jsdoc

Translated to:

    <target name="jsdocs" depends="properties, consolidate">
        <echo>recreating docs folder...</echo>
        <delete dir="${dir.docs}"/>
        <mkdir dir="${dir.docs}" />
        <echo>Generating...</echo>
        <exec executable="java" dir="${basedir}">
            <arg line="-jar '${file.jsdoc_toolkit.jar}' '${dir.jsdoc_toolkit}/app/run.js'" />
            <!-- -d tells JSDoc toolkit where to output the documentation -->
            <arg line="-d='${dir.docs}'" />
            <!-- use the default template -->
            <arg line="-t='${dir.jsdoc_toolkit}/templates/jsdoc'" />
            <!-- Create an arg element for each file you want to include in the documentation -->
            <arg line="'${file.consolidated.js}'" />
        </exec>
        <echo>Finished</echo>
    </target>

Clean up

If the build process has succeeded we need to move the temporary build files from the build directory to the bin.

    <target name="finish" depends="properties">
        <echo>Finishing...</echo>
        <delete dir="${dir.bin}" />
        <mkdir dir="${dir.bin}" />
        <move file="${dir.build.current}" tofile="${dir.bin}"/>
        <echo>Finished.</echo>
    </target>

Group it together

Finally we’ll create a grouping target which calls all of the other targets in sequence:

    <target name="build" depends="version, properties, create_build, consolidate, jshint, specs, minify, jsdocs, finish">
    </target>

Now run it!

In terminal cd to the root of the example directory and run ant with the name of target you want to see executed:

ant build

    • Will
    • May 27th, 2012

    Your build process (built on top of ANT) is cleaner and more flexible than gruntjs, which seems a bit sloppy. Go back!

    • Grunt is definitely more limited, but it takes far less time to get it all up and running. Granted, for some build processes – larger, more complex – I’d still use Ant though.

  1. I get this error when I run your build script out of the box.
    build.xml:39: refid ‘srcfiles’ does not refer to a resource collection.

  2. Sorted. The reason was my Ant being outdated. Works with 1.7.1 (1.7.0 didn’t)

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 548 other followers

%d bloggers like this: