wiki:BuildFarm

This page documents the build and test farm.

Currently two different automated build and test setups are in use:

  • Multiple periodic builds for Solaris/SPARC, FreeBSD/amd64, NetBSD/i386, CentOS Linux/x86_64, Debian Linux/i686, MacOSX/x86_64, OpenBSD/amd64.

Build Farm Setup

The following briefly describes the build farm.

Git Hooks

The main git repo is at /var/bind10/git/bind10.git

It has three hooks in place:

  • post-receive -- this runs two scripts:
    • /home/tester/bin/save-revision.sh
      • saves the revision/hash to /home/tester/LATEST_COMMIT
      • and an entry into the build farm queue (explained below)
    • /root/post-receive-email which generates the email about the commit to the list; it uses:
      • git config hooks.mailinglist
      • git config hooks.announcelist
      • git config hooks.envelopesender
      • The email prefix is hardcoded (git config hooks.emailprefix is not used)
  • pre-receive -- stops pushes that reference name containing secur, vulnerab, privat.
  • update -- to prevent force-pushing to master. It should prevent pushing rebases and other non-fast-forward changes to master by mistake.

The Build Queue

Each build farm periodically fetches a queue file of revisions to build. The queue file is automatically appended to when a commit to master is pushed. If the build servers haven't attempted that build before, they will do each build in order.

Some build servers do this every hour, one server only does it once per day, and one system does it on demand. Most are setup to not do multiple simultaneous builds, so there may be a long delay between initial commit and the final report. (For example, with using ccache for gcc, a single build on FreeBSD i386 system may take 30 minutes.)

Branches may be added to this queue too. The file is located on the git server at

/home/tester/public_html/builder/BUILD_QUEUE

It is currently owned by tester:tester (user/group).

If you have permission you can edit it. The format is simply a tab-delimited file with one entry per line. (Hashes at start of line are for comments). The fields are build name (like "BIND10"), repo URI (like "git://git.bind10.isc.org/bind10"), branch (such as "master" or "origin/trac1194", revision (a git hash), date added (in Unix epoch time), and an optional comment. (The git hook mentioned above runs a script that adds these.)

If a dash (-) is used for the revision then the latest on the branch will be checked out. If a revision is set, then the branch-name field is ignored. (TODO: fix that.) The date added field is used to periodically prune old entries (via another script).

A branch may be appended to the build queue file with /home/tester/builder/add-to-builder-queue on the git server, for example:

$ ~tester/builder/add-to-builder-queue BIND10 git://git.bind10.isc.org/bind10 trac2946

TODO: document the old build script that still is in use

Build

Each builder runs via cron a tool called build-and-upload. Each builder identifies itself with some name (like Debian5Linux-i686). The script fetches the build queue file and if has not done then it fetches a build specification tarball. This build specification includes a series of scripts for different phases to run, such as checkout, configure, build, running tests, installation, etc. Each phase is written in a simple portable shell script. The output of each phase is saved. When complete the output is sent to the build master. The build master frequently checks its queue of incoming builds and copies the data into place for viewing via http and also regenerates webpage reports indicating the success or failure of the phases.

The cronjobs are ran by either user jreed or b10builder. The build files are located in their home at ~/builder/. They may have custom environment (for various build and test behavior) located in sourced shell scripts named *.env (such as BIND10.env). These may be reviewed to see how to reproduce builds. The build jobs also periodically cleanup files (to restore diskspace) in advance; if a disk is full, likely it will remove old build files soon later.)

Emails are sent about failures. It doesn't send emails for same failure types (like generic "build") for same builder unless it has been over 24 hours. Possibly the suppressed notification is entirely unrelated. These emails are sent by the builder master (currently on the git server).

Last modified 4 years ago Last modified on Jul 31, 2013, 2:00:56 PM