FreeBSD/High Avaliability NFS Server

From TomJudge.com
Jump to: navigation, search


Incomplete!

This page is incomplete, hopefully I will get around to finishing this some day.


Contents

Getting Started

This page aims to show you how to configure a pair of FreeBSD boxes as a high availability NFS server.

For this we will need:

  • 2 FreeBSD boxes
    • 1GB or more RAM
    • RAID 1 or 10 HDD config
    • 2 * GbE NICs
  • Ports installed:
    • sysutils/heartbeat
    • sysutils/gnu-watch (optional)

The file system layout should have sufficient room on /data (or another file system) for all your nfs data to live. Here is a typical disk layout that we use at work:

Filesystem Size
/ 500M
/tmp 1G
/usr 10G
/var 15G
/data Rest of disk

Network configuration

The first NIC on each server should be connected to your server LAN, the second nic's should be patched together as a back to back link for the servers. The DNS entries for the servers should also be configured correctly (forward and reverse).

Server 1 rc.conf 

hostname="server1.local"
defaultrouter="192.168.0.254"
ifconfig_em0="inet 192.168.0.2/24"
ifconfig_em1="inet 192.168.1.1/30"

Server 2 rc.conf 

hostname="server2.local"
defaultrouter="192.168.0.254"
ifconfig_em0="inet 192.168.0.3/24"
ifconfig_em1="inet 192.168.1.2/30"

Preparing the file systems

This assumes that you have a basic FreeBSD install on both systems.

On both servers you will need to do the following (Assuming you want 10G of NFS space): 

mkdir /data/nfs
cd /data/nfs
dd if=/dev/zero of=fileSystem bs=1M count=10240

Now we have the base file to use to put the file system in (/data/nfs/fileSystem).

Preparing the systems

Now you will need to install the following to scripts into /etc/rc.d/

File contents from SVN: ggatec
#!/bin/sh
 
# PROVIDE: ggatec
# REQUIRE: ggated
# BEFORE:  mountcritlocal
 
#Copyright (C) 2010, Tom Judge
# ----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# <tom@tomjudge.com> wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Tom Judge.
# ----------------------------------------------------------------------------
 
#ggatec_enable="YES"
#ggatec_remote_host="127.0.0.1"
#ggatec_remote_device=""
 
. "/etc/rc.subr"
 
name="ggatec"
rcvar=`set_rcvar`
 
# read configuration and set defaults
load_rc_config "$name"
 
: ${ggatec_enable="NO"}
: ${ggatec_buf_size="1048567"}
: ${ggatec_timeout="5"}
: ${ggatec_queue_size="1024"}
start_cmd=${name}_start
stop_cmd=${name}_stop
 
ggatec_start() {
	/sbin/ggatec create -u 0 \
            -q ${ggatec_queue_size} \
            -t ${ggatec_timeout} \
            -S ${ggatec_buf_size} \
            -R ${ggatec_buf_size} \
            ${ggatec_remote_host} \
            ${ggatec_remote_device}
}
 
ggatec_stop() {
	/sbin/ggatec destroy -u 0
 
}
 
run_rc_command "$1"
File contents from SVN: ggated
#!/bin/sh
 
# PROVIDE: ggated
# REQUIRE: root
# BEFORE:  mountcritlocal
 
#Copyright (C) 2010, Tom Judge
# ----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# <tom@tomjudge.com> wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Tom Judge.
# ----------------------------------------------------------------------------
 
#ggated_enable="YES"
#
 
. "/etc/rc.subr"
name="ggated"
rcvar=`set_rcvar`
 
required_files="/etc/gg.exports"
 
 
# read configuration and set defaults
load_rc_config "$name"
 
: ${ggated_enable="NO"}
: ${ggated_ip="127.0.0.1"}
: ${ggated_buf_size="1048576"}
start_cmd=${name}_start
stop_cmd=${name}_stop
 
ggated_start() {
        /sbin/ggated -S ${ggated_buf_size} \
                        -R ${ggated_buf_size} \
                        -a ${ggated_ip} \
                        /etc/gg.exports
}
 
ggated_stop() {
        /usr/bin/killall ggated
}
 
run_rc_command "$1"


You will need to add the following to rc.conf on both servers: 

nfs_server=192.168.0.1
nfs_server_enable="NO"
nfs_server_flags="-u -t -n 4 -h ${nfs_server}"
rpcbind_enable="NO"
rpcbind_flags="-h ${nfs_server}"
mountd_enable="NO"
mountd_flags="-r -p 832 -h ${nfs_server}"
 
if [ $hostname = "server1.local" ]; then
   ggate_me="192.168.1.1"
   ggate_peer="192.168.1.2"
else
   ggate_me="192.168.1.2"
   ggate_peer="192.168.1.1"
fi
 
ggated_enable="NO"
ggated_ip="${ggate_me}"
ggated_buf_size="1310720"
ggatec_enable="NO"
ggatec_buf_size="1310720"
ggatec_timeout="5"
ggatec_queue_size="2048"
ggatec_remote_host="${ggate_peer}"
ggatec_remote_device="/dev/md0"

Add the following to /etc/sysctl.conf 

net.inet.tcp.sendspace=1048576
net.inet.tcp.recvspace=1048576
kern.ipc.maxsockbuf=2049152

Add the following to /boot/loader.conf 

kern.ipc.nmbclusters="32768"

On server1 you need the following /etc/gg.exports 

192.168.1.2 RW /data/nfs/fileSystem

On server2 you need the following /etc/gg.exports 

192.168.1.1 RW /data/nfs/fileSystem

Now you need to reboot the 2 servers so that the loader settings take affect.

Configuring Heartbeat

First we need to share the key that will be used to identify the cluster members.

/usr/local/etc/ha.d/authkeys 

auth 1 
1 sha1 key-for-sha1-any-text-you-want

This file must be set to perms 600 and owned by root:wheel.

Next we need configure the global heartbeat configuration. /usr/local/etc/ha.d/ha.cf 

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility     local0
keepalive 2
deadtime 170
warntime 10
initdead 180
udpport 695
bcast   em1
auto_failback off
node    server1.local
node    server2.local
Retrieved from "http://www.tomjudge.com/index.php/FreeBSD/High_Avaliability_NFS_Server"
Personal tools