The aim of this article is to achieve the following for multiple different releases of FreeBSD.
I will be covering creating builds for the following releases: 8.1, 7.1, 7.0, 6.2. These releases have been chosen because they are the releases that I have to maintain at work, you can substitute any releases you wish. In order to achieve these goals we will be employing jails, and Tinderbox, along with some glue scripts to tie it all together.
If you are planning on building packages for multiple architectures you will need to duplicate this process for each architecture. At $work we maintain builds for i386 and amd64 but this process is equally applicable to PPC and sparc64 for example.
The VCS that I will be using thought this article is Subversion (svn), you may substitute this for your prefered VCS if you wish. If you are unfamiliar with svn you may wish to take a look at the Subversion Book. At this point you may be asking why we need a VCS. The simple answer to this is that there is a high probability that you will need to make changes to certain things, both in the FreeBSD base system and in the ports tree that you will be building packages from. Some examples of the changes that we make at $WORK are:
If you are the sole SA in an organisation you may think that you can keep track of all of this information without a VCS, however as you start to introduce more builds into your network you will find it more difficult. I recommend investing the time to setup the VCS before you start even if you are only building for one release to start with.
I will not be covering how to setup a svn server in this article, and will assume that you have managed to setup an empty repository accessible via WebDav at http://svn.local/bsd.
The stuff that we will be storing in VCS will be:
We will be including vendor trees in our VCS system because this makes tracking upstream changes easier as well as tracking the points at which we integrate them into the builds.
Based on these things this is what our svn repository will look like before we start loading anything into it:
/system /system/vendor /system/vedor/releng/[release branches] /system/vendor/tags/[patch level tags] /system/releng/[internal release branches] /system/tags/[internal release tags] /ports /ports/vendor /ports/vendor/current /ports/vendor/[import-tags] /ports/[internal release branches]
Warning: The you will need a fairly powerful system to run these processes on with a lot of ram and very fast disks or they will take a very long time. At $work the system that we use has 6Gb of ram and runs amd64 as the processes get very large when importing large file systems such as the ports tree.
In order to complete this we are going to need a working area, with all of the vendor code in it. We are going to be fetching the vendor code from 2 separate VCS systems operated by FreeBSD: OS source from SVN, and ports trees from CVSUP, you may choose to operate your own internal mirrors to make operations from these systems faster, however I will not be covering these here there are plenty of tutorials on how to do this out there.
Our working area is going to look something like this (all paths are relative to the working area base directory, which you will need to decide on based on your file system layout, for the rest of this document I will be using /data/workdir as my base) :
/vendor /vendor/system /vendor/system/svn /vendor/system/exports /vendor/ports /internal <-- Svn checkout of http://svn.local/bsd
Now that we have a working directory structure setup we can proceed to fetch all of the code that we need. (The following walk though assumes you are using bash as your shell) You will need the svn_load_dirs script in your path to complete this process, you can grab this from the contrib folder of the svn distribution tarball.
RELEASES="6.4 7.0 7.1" WORKDIR=/data/workdir SVN=http://svn.local/bsd for I in $RELEASES; do cd ${WORKDIR}/vendor/system/svn ## Checkout the release engineering branch from the FreeBSD SVN Repository svn co http://svn.freebsd.org/base/releng/${I} ${I} ## Grab the current patch level information out of the checkout. TMPFILE=`mktemp` egrep 'REVISION="|RELEASE="|BRANCH="' ${I}/sys/conf/newvers.sh > ${TMPFILE} . ${TMPFILE} ## Export the working copy to make it suitable for ## loading into the our interal svn repository. svn export ${I} ${WORKDIR}/vendor/exports/${RELEASE} ## Load the release into the svn repository. cd ${WORKDIR}/vendor/exports/${RELEASE} svn mkdir ${SVN}/system/vendor/${I} -M "Create container for FreeBSD ${I} Import" svn_load_dirs.pl -t tags/${RELEASE} ${SVN}/system/vendor ${I} . done cd ${WORKDIR}/internal svn up
Now that we have the base source code in the repository we need to grab a ports tree snapshot an load it into the repository. To get the snapshot we will use csup to fetch it from one of the cvsup mirrors. Here is the sup file that you will need, it should be placed in ${WORKDIR}/vendor/ports/supfile. You will need to set the host to a suitable cvsup mirror close to you.
*default host=CHANGE_THIS.FreeBSD.org *default base=/data/workdir/vendor/ports *default prefix=/data/workdir/vendor/ports *default release=cvs tag=. *default delete use-rel-suffix *default compress ports-all
Now that the supfile is ready we can proceed with loading a snapshot of the tree into SVN. (This assumes that you are running a FreeBSD release that includes csup, if not you will need to install cvsup from ports and replace the csup command with cvsup.)
SVN=http://svn.local/bsd cd /data/workdir/vendor/ports csup supfile cd ports svn mkdir ${SVN}/ports/vendor/current -M "Create container for FreeBSD ports tree import" svn_load_dirs.pl -t `date +%Y-%m-%d` ${SVN}/ports/vendor current .