This howto should help you solve some of the problems that I have come across while trying to setup distcc on FreeBSD amd64 hosts to cross compile code to i386/ia32.

This assumes that you will be building binaries to run on i386 boxes, and running make from i386 boxes. If this is not the case this howto may be of little use to you. I will also be assuming that FreeBSD 6 is being used on all the hosts.

Required Pacakges

You will need to install the distcc package (/usr/ports/devel/distcc) on all of the hosts that you wish to use in your build cluster. On the hosts that you wish to compile on (the ones that make will be run from) you will need to install the ccache package (/usr/ports/devel/ccache) as well as the distcc package.

Setting up distccd

The distccd process needs to be setup on all servers that you wish to use to compile on, however there are a few extra things that need setting up on the amd64 hosts to make things play nicely.

Once you have installed the distcc package you need to add an entry similar to the one below to /etc/syslog.conf:

!distccd
*.*                                     /var/log/distccd.log

You should now create the log file (touch /var/log/distccd.log) and restart syslogd (/etc/rc.d/syslogd restart).

You now need to update /etc/rc.conf to enable the daemon at startup. The following entires should suffice (assuming all you clients are on the 10.0.0.0/8 subnet):

distccd_enable="YES"
distccd_flags="-a 10.0.0.0/8 -a 127.0.0.0/8 --user distcc --daemon -P /var/run/distccd.pid"

On i386 build servers you have now finished configuring distccd and can start it using the rc script (/usr/local/etc/rc.d/distccd start). On the amd64 boxes you need to make a few changes to make distcc compile 32bit code. First you need to create a wrapper for distccd that changes the path where it looks for compilers. Place the following script in /usr/local/sbin/distccd-i386:

#!/bin/sh
export PATH=/usr/local/crossdev/
 
/usr/local/sbin/distccd "$@"

Now you need to update the rc script to use the wrapper script rather than directly executing the distccd binary. The line in /usr/local/etc/rc.d/distccd that reads:

command=/usr/local/sbin/${name}

Should now read:

command=/usr/local/sbin/${name}-i386

Next you need to create the wrapper script for the compilers to add the -m32 argument to the compiler options so that 32bit code is generated. The following script should be created in /usr/local/corssdev/crossgcc:

#!/usr/local/bin/bash
 
# Note that PATH is unset; specify full path of everything
/usr/bin/`/usr/bin/basename $0` -m32 "$@"

You now need to create symlinks to the crossgcc script with the names of all compilers that you may use. The following bash script will make the common ones for you:

cd /usr/local/crossdev
for I in "gcc g++ g77 cc c++ cpp"; do
	ln -s crossgcc $I
done

Distccd is now fully configured for amd64 boxes to build 32bit code, you can start the daemon using the rc script (/usr/local/etc/rc.d/distccd start).

Setting up distcc

On the hosts that you are going to run builds from (the client) you need to have the distcc package installed, but do not need to have distccd running although there is no harm on running on the client as well. Configuring the distcc client is quite easy as there is one central configuration file to edit. In /usr/local/etc/distcc/hosts you should place a list of hosts running distccd e.g:

vader
anakin
darth

That is all that needs configuring for distcc initally the rest of the configuration incuding how to get make to use distcc will be covered in the ccache section.

Setting up ccache

Due to an anoying feature in ccache which causes it to pass an absolute path to the compiler to distcc another wrapper is required to make ccache and distcc play nicely together. The script strips the path off the compiler name and then passes the new argument set to distcc. It should be placed in /usr/local/bin/distcc_wrapper:

#!/usr/local/bin/bash
cmd=`basename $1`
shift
/usr/local/bin/distcc $cmd "$@"

If you do not have a group that all of the users that will use the build cluster to belong to you need to create on now. Once you have the group in place you are ready to create the cache directory.

mkdir /data/ccache
chgrp developers /data/ccache

All that is left now is to setup your environment so that you can use the build system. If you use bash you can add the following to your ~/.bash_profile:

export CCACHE_DIR=/data/ccache
export CCACHE_NOLINK=yes
export CCACHE_UMASK=002
export CCACHE_PREFIX=distcc_wrapper
 
dmake() {
    make CC="ccache gcc" CXX="ccache g++" "$@"
}

If you have nfs mounted home directories that do not support locking you should use the following extra entries:

export CCACHE_DIR=/data/ccache
export CCACHE_NOLINK=yes
export CCACHE_UMASK=002
export CCACHE_PREFIX=distcc_wrapper
export DISTCC_DIR=/tmp/`whoami`/distcc
 
mkdir -p /tmp/`whoami`/distcc
 
dmake() {
	make CC="ccache gcc" CXX="ccache g++" "$@"
}

If you would rather use gmake then you can change the function definition to call gmake instead of make. In order to use the build cluster rather than the local compiler you can use the dmake command with jobs argument set to a reasonable number e.g.:

dmake -j5 all