(Thanks to Alex Sallin, this post is also available in Czech here.)

This blog post is about how I build using scripts instead of calling make directly.

Last known good

When you do “random” pulls from Mozilla-central you are rarely sure what the state of the tree is before you pull, it takes some time before tinderbox get updated and it might be burning red on a platform that you don’t care about or it might not have a build for the special configuration you are working on….

To solve this problem I have two cron scripts running on my machine, one that pulls the latest and greatest from mozilla-central every half hour, and one that tries to build what ever was pulled, if the build succeeds with all the configurations that are important to me, it will tag the revision so I can update to it when I need to – that means that at all times I can update to a known good revision, I do incremental builds, and while this is not always giving the same result as a clean build, it is close enough for day to day use.

(The rest of this post is mostly relevant in a Linux environment)

Doing scratchbox builds without being in scratchbox

The most tricky part of creating the scripts were to make the compile in scratchbox (when building for Maemo) – I was already doing:

$ sudo mount –bind ~/MozillaCode /scratchbox/users/mike/home/mike/MozillaCode

This lets me share a single copy of the source code between scratchbox and my normal development directory (The above command will make ~/MozillaCode contain the same stuff when in scratchbox and when you are outside scratchbox for a user called “mike”, remember to create the empty target directory before executing the command, and you can’t do it the other way around mounting the scratchbox directory in your home directory).

My first attempt at building in scratchbox, without first logging into scratchbox, involved calling a script running in scratchbox that did the building – but then I found that it is actually possible to use the same environment in scratchbox as you were in when launching scratchbox, this is done by using the -k flag, and you can even shift into a specific directory with the -d flag.

So to build, my script sets up the correct MOZCONFIG and then executes something like:

$ scratchbox -d “$sourceDir” -k make -f client.mk

where sourceDir points to the mozilla-central that I want to build.

Doing scratchbox builds without being in scratchbox from a cron script

Now, my Linux knowledge is too limited to know the reason why – but in order to run the scratchbox command from a script run from cron, you need to first do:

export USER=mike

and then the scratchbox command, if you happen to share my user name, otherwise substitute your own user name smiley.

Only getting the relevant output

When we build we get a lot of build output that is not very relevant in most cases, what is most important is the errors that can be hidden inside the output, they are especially hiding if you are running multi threaded builds.

The first thing I did was to add a:

mk_add_options MOZ_MAKE_FLAGS=”–no-print-directory”

to my mozconfig files, as that takes a lot of unneeded information away – I’m sure it’s useful for some people to see where the build script is, but not for me.

The other thing I did was to pipe all the standard output to a file, leaving only the error output, by executing make like:

$ make -f client.mk > stdoutLogFile.txt

But as I wanted to be able to get the error output to see failures or successes from my cron script I also needed to pipe the error output, that is done by using the 2> pipe:

$ make -f client.mk > stdoutLogFile.txt 2> stderrLogFile.txt

In my actual script the files a renamed so I can see which build generated them, and a _OK is postfixed to the filename if the build was successful, and _Fail if there were build errors – this enables me at a glance to see which platforms are currently building and which are failing, and to get the build error I just need to open the file(s) postfixed with _Fail. Very easy and very convenient (I can even see which target is being build at the moment as that gets a _Building postfixed).

At this very moment the Result* content of my LatestBuild directory is like:


These correspond to the “stderrLogFile.txt” in the above make example, but for each platform I’m building for.

It can be seen that right now everything seems fine for the platforms I care about in my daily work.

What I can do now to get the latest code, in my working directory is the execute

$ hg pull
$ hg update – u mikek-lng

As I’m always cloning my local repository rather than directly from mozilla-central when I start a new directory. This way I can be almost sure that any failures after the update are due to errors I made in the patch I’m working on, and not something that is coming from mozilla-central – which was my main goal.

Building with a script instead of the command line

Previously I used to have several command line windows open, one configured for each target that I wanted to build for (like for Firefox Mobile PC, Firefox Mobile PC Qt version, Firefox Mobile Maemo Release version, …) It was confusing and I was never sure on which platforms I had kicked of builds on, or which window belonged to what platform.

As creating the LNG scripts had given me a basal and dangerous knowledge of writing shell scripts on my Ubuntu box I got the idea of creating a single macro that could do all the building – iBuild was created, what I have now is a simple tool I can run that reports back to me in a very simple way if the build is a success or not.

I can do the quick and dirty, that takes forever:

$ ./iBuild all

This will simply build all the targets that I found relevant as seen above, or I can specify what I’m currently interested in like:


(FF_* = Firefox, FFM_* = Firefox mobile, *_P* = PC, *_N* = Maemo (Nokia), *_?R = Release, *_?D = Debug, _QT = Qt version, otherwise it defaults to Gtk) It’s all done with a big table and a number of Mozconfigs, but I hope one day it will auto generate the Mozconfigs.

A build cycle now looks something like (Note, scratchbox Maemo builds are handled inline):

Building from /home/mike/MozillaCode/100614
parameter is FFM_NR FFM_NR_QT FF_PD

Building FFM_NR
FireFox Mobile Nokia Release
Using Scratchbox for building
Build success

Building FFM_NR_QT
FireFox Mobile QT Nokia Release
Using Scratchbox for building
Build success

Building FF_PD
Firefox PC Debug
Build success

All builds were ok

The bottom line is the important one – it tells me with a single glance that everything went well, where before I need to go to each individual command line window – if it had failed it is easy to see which build failed, and I have the error log in a file saved on the drive to see the exact error message.

My iBuild script is by no means finished, it contains very little error handling, it is constantly evolving, while it works for me it might not work for anyone else, and while I didn’t intend for it to launch an ICBM attack when executed it might do, use at your own risk.

However, feel free to be inspired by it and create your own modified version – it can be found here and must be executed in the directory that contains your mozilla-central, e.g. the directory below where you would usually execute make from.

The currentSyncTarget() function contains the translation between command line arguments and mozconfig files, and the buildAll() function defines what the “all” command will build.

The script that is called from cron does a

hg pull
hg update -C

and then calls iBuild all to do the actual building, its protected against multiple executions in the same simplified way as used in the iBuild script with a lock file.

If iBuild is successful it then uses hg to tag the version

hg tag -f mikek-lng

Any feedback will as usually be appreciated.