CFEngine/Managing FreeBSD Jails

Views

From TomJudge.com

The module should allow you to manage jails deployed on a large number of jail hosts using a MySQL DB server and cfengine.

1 DB Setup
2 Jail Hosts
3 Configuring Flavours
4 Activating the jail manager module in cfengine.
5 Create some jails

DB Setup

In order to make this system work we are going to need some tables in the MySQL database, here are the definitions for the tables:

CREATE TABLE `Jail` (
  `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `host_object_id` int(10) UNSIGNED NOT NULL DEFAULT '1',
  `ip` int(10) UNSIGNED NOT NULL DEFAULT '0',
  `description` text,
  `flavour_id` int(10) UNSIGNED NOT NULL DEFAULT '0',
  `deployable` enum('yes','no') NOT NULL DEFAULT 'no',
  `enabled` enum('yes','no') NOT NULL DEFAULT 'no',
  `type` enum('production','development') NOT NULL DEFAULT 'production',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `ip` (`ip`),
  UNIQUE KEY `name` (`name`)
);
 
CREATE TABLE `Flavour` (
  `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `description` text,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `name` (`name`)
);
 
CREATE TABLE `RackObject` (
  `id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  PRIMARY KEY  (`id`),
  UNIQUE KEY `name` (`name`)
);

NOTE: This is a subset of the racktables database that we use. If you want more info on the system please contact me.

Jail Hosts

The jail hosts need the following packages installed:

Perl
Perl DBD::MySQL
ezjail

They also need to have the ROLE_JailHost class active in cfengine.

Each jail host should also have a record in the RackObject table:

INSERT INTO RackObject (name) VALUES ('jailhost1');
INSERT INTO RackObject (name) VALUES ('jailhost2');
INSERT INTO RackObject (name) VALUES ('jailhost3');

The following CFEngine policy file will configure ezjail on the jail hosts for you.

File contents from SVN: ezjail.cf

#Copyright (C) 2009, Tom Judge
# ----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# <tom@tomjudge.com> wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Tom Judge.
# ----------------------------------------------------------------------------
 
## THESE VARIABLES MUST MATCH /usr/local/etc/ezjail.conf on the jail host.
control:
    ROLE_JailHost::
        AddInstallable = (
            ezjail_failed_primary
            ezjail_failed_backup
            )
 
        ezjail_jailbase = ( /data/jails/basejail )
        ezjail_jailtemplate = ( /data/jails/newjail )
 
groups:
    ROLE_JailHost::
        ezjail_has_jailbase = ( IsDir(${ezjail_jailbase}) )
        ezjail_has_jailtemplate = ( IsDir(${ezjail_jailtemplate}) )
 
copy:
    freebsd.ROLE_JailHost::
        ${configroot}/config/ezjail/ezjail.conf
            dest=/usr/local/etc/ezjail.conf
            failover=ezjail_failed_primary
            ignore=.svn
            mode=0644
            recurse=inf
            server=${primary_server}
            type=checksum
        ${configroot}/config/ezjail/flavours
            dest=/data/jails/flavours
            failover=ezjail_failed_primary
            ignore=.svn
            mode=0644
            recurse=inf
            server=${primary_server}
            type=checksum
 
    freebsd.ROLE_JailHost.ezjail_jailed_primary::
        ${configroot}/config/ezjail/ezjail.conf
            dest=/usr/local/etc/ezjail.conf
            failover=ezjail_failed_backup
            ignore=.svn
            mode=0644
            recurse=inf
            server=${backup_server}
            type=checksum
        ${configroot}/config/ezjail/flavours
            dest=/data/jails/flavours
            failover=ezjail_failed_backup
            ignore=.svn
            mode=0644
            recurse=inf
            server=${backup_server}
            type=checksum
 
directories:
    freebsd.ROLE_JailHost::
        ${ezjail_jailtemplate}/usr/home
            mode=775
            owner=root
            group=${root_group}
 
    freebsd.ROLE_JailHost.ezjail_has_jailbase::
        ${ezjail_jailbase}
            mode=755
            owner=root
            group=${root_group}
 
        ${ezjail_jailbase}/usr
            mode=755
            owner=root
            group=${root_group}
 
        ${ezjail_jailbase}/usr/lib32
            mode=755
            owner=root
            group=${root_group}
 
files:
    freebsd.ROLE_JailHost::
        /data/jails
            mode=700
            owner=root
            group=${root_group}
            action=fixall
 
 
editfiles:
    freebsd.ROLE_JailHost::
        { /etc/rc.conf
            AppendIfNoSuchLine "ezjail_enable=\"YES\""
            AppendIfNoSuchLine "jail_sysvipc_allow=\"YES\""
        }
 
shellcommands:
    freebsd.ROLE_JailHost.!ezjail_has_jailbase::
        "/usr/local/bin/ezjail-admin update -i"
            useshell=true
            inform=true
            timeout=10
            expireafter=10
 
alerts:
    ezjail_failed_backup::
        "Failed to copy the ezjail configuration file."
 
# vim:set syntax=cfengine:
# vim:set tabstop=4:
# vim:set shiftwidth=4:
# vim:set expandtab:

You can find some the example ezjail configuration files here.

For each of the flavours you wish to use with ezjail you need to insert a record like so:

Configuring Flavours

For each flavour that you create in the config/ezjail/flavours directory you need to insert a record into the Flavours table.

INSERT INTO Flavour (name,description) VALUES ('basic','basic example flavour');

Activating the jail manager module in cfengine.

You will need install the following module in your cfengine modules directory:

File contents from SVN: module:jailmanager

#!/usr/bin/perl
#Copyright (C) 2009, Tom Judge
# ----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# <tom@tomjudge.com> wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Tom Judge.
# ----------------------------------------------------------------------------
 
use strict;
use warnings;
use English;
use Fcntl ':flock';
 
open (MUTEX, "> /var/run/cfengine-jailmanager-mutex") or die ("Cant open mutex");
if (!flock(MUTEX,LOCK_EX|LOCK_NB)) {
    print "Jail Manager already Running\n";
    exit;
}
 
###Workaround to avoid annoying messages when cfengine runs on a server with no DBI.pm
eval "require DBI";
if ($@)
{ exit; }
require DBI;
###Workaround -end
 
use Data::Dumper;
my $EZJAIL_ADMIN = "/usr/local/bin/ezjail-admin";
my $EZJAIL_RC = "/usr/local/etc/rc.d/ezjail.sh";
my $hostname = `/bin/hostname -s`;
chomp $hostname;
 
my $RACKTABLES_HOST = $ARGV[0];
my $RACKTABLES_DB =   $ARGV[1];
my $RACKTABLES_USER = $ARGV[2];
my $RACKTABLES_PASS = $ARGV[3];
 
my @allclasses = split (":","$ENV{CFALLCLASSES}");
my %classes;
foreach my $class (@allclasses) {
    $classes{$class} = 1;
}
 
## We should only run on Jail Hosts
if (!defined($classes{ROLE_JailHost})) {
    exit;
}
 
## We should probably only run after we have a base jail.
if (! -e "/data/jails/basejail") {
    print "** DRAGON ALERT: No jail base yet, its not my time **\n";
    exit;
}
 
## We need the list of current jails to be able to work out what to do.
#STA JID   IP              Hostname                     Root Directory
#--- ----- --------------- ---------------------------- -------------------------
#DS  N/A   172.17.0.106    testjail.usdmm.com           /data/jails/testjail.usdmm.com
 
my %current_jails;
 
# Fetch the current jails
my @ezjail_list = `$EZJAIL_ADMIN list`;
 
## trim off the first 2 lines of headers
shift @ezjail_list; shift @ezjail_list;
 
for my $jail (@ezjail_list) {
    chomp $jail;
    my ($jail_status, $jail_id, $jail_ip, $jail_name, $jail_root) = split /\s+/, $jail;
    $current_jails{$jail_name}->{'ip'} = $jail_ip;
    $current_jails{$jail_name}->{'id'} = $jail_id;
    $current_jails{$jail_name}->{'root'} = $jail_root;
    ## Parse flags (see ezjail-admin(1))
    if ($jail_status =~ /D/) {
        $current_jails{$jail_name}->{'base'} = "dir";
    } elsif ($jail_status =~ /I/) {
        $current_jails{$jail_name}->{'base'} = "image";
        if ($jail_status =~ /B/) {
            $current_jails{$jail_name}->{'image_type'} = 'bde';
        } elsif ($jail_status =~ /E/) {
            $current_jails{$jail_name}->{'image_type'} = 'eli';
        } else {
            $current_jails{$jail_name}->{'image_type'} = 'raw';
        }
    }
    if ($jail_status =~ /R/) {
        $current_jails{$jail_name}->{'status'} = "running";
    } elsif ($jail_status =~ /A/) {
        $current_jails{$jail_name}->{'status'} = "attached";
    } elsif ($jail_status =~ /S/) {
        $current_jails{$jail_name}->{'status'} = "stopped";
    }
}
 
##  Connect to the management DB
my $dsn = "DBI:mysql:$RACKTABLES_DB:$RACKTABLES_HOST";
my $dbh = DBI->connect ($dsn, $RACKTABLES_USER, $RACKTABLES_PASS, {RaiseError=>1, PrintError=>0});
 
## Get the list of jails for this host
my $get_jails = $dbh->prepare(
        "SELECT Jail.name, INET_NTOA(Jail.ip) as ip, Flavour.name as flavour_name, Jail.deployable, Jail.enabled FROM Jail ".
            "LEFT JOIN Flavour on Flavour.id=Jail.flavour_id ".
            "LEFT JOIN RackObject on Jail.host_object_id=RackObject.id ".
                "WHERE LOWER(RackObject.name)=?"
        );
$get_jails->execute($hostname);
 
my %required_jails;
my $jail;
while ($jail = $get_jails->fetchrow_hashref()) {
    print "MY JAIL: ".$jail->{"name"}." - ".$jail->{"ip"}."\n";
    $required_jails{$jail->{"name"}}->{"ip"}=$jail->{"ip"};
 
    ####
    ### PRE FLIGHT CHECKS
    ####
    ## Check that the jail is allowed to be deployed
    if ($jail->{"deployable"} eq 'no') {
        print "** DRAGON ALERT: Jail not deployable, nothing to see here (".$jail->{"name"}."), move along please **\n";
        next;
    }
 
    ## Check that the IP is configured on an interface somewhere
    my $jailip=$jail->{"ip"};
    if (scalar(grep(/\s$jailip\s/,`/sbin/ifconfig`)) != 1) {
        print "** DRAGON ALERT: Jail IP Not Configured, nothing to see here (".$jail->{"name"}."), move along please **\n";
        next;
    }
 
    ## Check that the flavour exists
    if (! -e "/data/jails/flavours/".$jail->{"flavour_name"}) {
        print "** DRAGON ALERT: Jail Flavour Does Not Exist, nothing to see here (".$jail->{"name"}."), move along please **\n";
        next;
    }
 
    ####
    ### PRE FLIGHT CHECKS PASSED
    ###
    ### We are good to start working with the jail now
    ####
 
    if (defined($current_jails{$jail->{"name"}})) {
        print "Jail exists: ".$jail->{"name"}."\n";
        if ($current_jails{$jail->{"name"}}->{"status"} eq "stopped") {
            if ($jail->{"enabled"} eq 'yes') {
                print "Starting Jail: ".$jail->{"name"}."\n";
                my @start_command = ($EZJAIL_RC, "start", $jail->{"name"});
                system (@start_command);
            }
        } elsif ($current_jails{$jail->{"name"}}->{"status"} eq "running") {
            if ($jail->{"enabled"} eq 'no') {
                print "Stopping Jail: ".$jail->{"name"}."\n";
                my @start_command = ($EZJAIL_RC, "stop", $jail->{"name"});
                system (@start_command);
            }
        }
 
        my $ports = "/data/jails/".$jail->{"name"}."/usr/ports";
        if ( -d $ports ) {
            `rm -Rf $ports`;
            symlink ( "../../basejail/usr/ports", $ports);
        }
    } else {
        print "Creating Jail: ".$jail->{"name"}."\n";
        ## Here we go time to fly  create me a jail please
        my @create_command = ($EZJAIL_ADMIN, "create", "-f", $jail->{"flavour_name"}, $jail->{"name"}, $jail->{"ip"});
        system(@create_command);
 
        # Mount /usr/home in the jail.
        my $fstab = $jail->{"name"};
        $fstab =~ s/[\.-]/_/g;
        $fstab = "/etc/fstab.$fstab";
        my $home_fstab = "/usr/home /data/jails/".$jail->{"name"}."/usr/home nullfs rw 0 0\n";
        open FSTAB, '>>', $fstab;
        print FSTAB $home_fstab;
        close FSTAB;
 
        #Fix permissions
        #/data/jails/basejail/usr and usr/lib32 and /data/jails/newjail/basejail
        my $dirtofix;
        foreach $dirtofix ("/data/jails/basejail/usr" , "/data/jails/basejail/usr/lib32" , "/data/jails/newjail/basejail" , "/data/jails/basejail") {
            warn "Could not chmod \"$dirtofix\": $!" unless chmod (0755,$dirtofix);
        }
 
        if ($jail->{"enabled"} eq 'yes') {
            my @start_command = ($EZJAIL_RC, "start", $jail->{"name"});
            system (@start_command);
        }
 
    }
}
 
##
## Delete jails that are not supposed to be here
##
for my $jail_name (keys(%current_jails)) {
    if (!defined($required_jails{$jail_name})) {
        print "Starting Removal Of Jail: $jail_name\n";
        if ($current_jails{$jail_name}->{"status"} eq "running") {
            print "Stopping Jail: $jail_name\n";
            my @stop_command = ($EZJAIL_RC, "stop", $jail_name);
            system (@stop_command);
        }
        print "Deleting Jail Filesystem: $jail_name\n";
        my @delete_command = ($EZJAIL_ADMIN, "delete", "-w", $jail_name);
        system(@delete_command);
    }
}
 
 
$get_jails->finish();
$dbh->disconnect();
 
flock(MUTEX,LOCK_UN);
close(MUTEX);

control:
    any::
        management_db_server = ( management-db.test.com )
        management_db_name = ( racktables )
        management_db_user = ( cfengine )
        management_db_pass = ( readonly )
    freebsd::
        moduledirectory = ( /var/cfengine/modules )
        workdir = ( /var/cfengine )
        actionsequence =
            (
            packages
            copy
            links.Prepare
            files.Prepare
            directories
            links.Rest
            required
            tidy
            disable
            editfiles
            files.Rest
            processes
            shellcommands
            "module:jailmanager ${management_db_server} ${management_db_name} ${management_db_user} ${management_db_pass}"
            )

Create some jails

So all we are left to do is actually create some jails. To do this all we need to do is insert some records into the Jail table.

This jail is on jailhost1 and is a basic flavour.

INSERT INTO Jail (
        name,
        host_object_id,
        ip,
        descirption,
        flavour_id,
        deployable,
        enabled,
        type 
    ) VALUES (
        "jail1.test.com",
        1,
        INET_ATON("192.168.1.10"),
        "Jail 1 for web server",
        1,
        "yes",
        "yes",
        "production"
    );

Retrieved from "http://www.tomjudge.com/index.php/CFEngine/Managing_FreeBSD_Jails"

Categories: FreeBSD | CFEngine

CFEngine/Managing FreeBSD Jails

Views

Contents

DB Setup

Jail Hosts

Configuring Flavours

Activating the jail manager module in cfengine.

Create some jails

Navigation

Other

Search

Toolbox

Personal tools