FreeBSD/Jails/MemoryLimits

Comments/improvements/suggestions welcome.

WARNING THESE PATCHES ARE KNOWN TO CAUSE SYSTEM DEADLOCKS WITH HEAVY WORKLOADS - USE AT YOUR OWN RISK

Memory Usage Stats Collection

Jail memory usage is collected every X (default 5) seconds by a kernel thread, this thread collects both resident set size and the full vm size for all processes+threads in the prison (prison is the kernel side of the jail subsystem). This thread is called jpager and there is one jpager for every jail in the system.

Limits

The system has both soft and hard limits that are run time tunable using the jtune command.

Soft Limits

When a jail's resident memory foot print exceeds the soft limit, the vm system requests that ~6% of the largest process in the jail resident memory is released. This has 2 distinct effects depending on the level of memory contention on the system:

Case 1 free memory: Nothing really happens to the process as there is no memory contention, the deactivated pages will remain in ram until the memory system becomes contended.

Case 2 no free memory: Deactivated pages are swaped out to disk, the net affect is that processes that are using large amounts of ram will be slowed down, as they will have to have pages swapped back in on request. Other processes in the system will remain swapped in assuming that they all fit in ram.

Hard limits

The hard limit worked on the full vm footprint of the jail. (All resident and swapped out pages). When the jails footprint exceeds the hard limit then a jail local out of memory killer is invoked. This will cause the largest process in the jail to be killed. NOTE: The hard limit will only be triggered when the system starts to swap.

JPager Internals

The jpager thread will ignore threads/processes with the following conditions:

Processes:

• P_SYSTEM - Marked as a system thread (i.e its self)
• P_WEXIT - Processes waiting to exit
• P_PROTECTED - Protected processes.
• PID = 1 - Init (Killing this would be bad TM)

Threads:

• Is on the run queue.
• Is running.
• Is sleeping.

This should prevent swap from thrashing.

How To Use

This patch is for FreeBSD 7.1.

Download the above patch and place in /var/tmp

cd /usr/src
patch < /var/tmp/7.1-jail-limits.patch
cd sys/kern
make sysent
cd /usr/src
make buildworld
make buildkernel
make installkernel
make installworld
echo "security.jail.limit_jail_memory=1" >> /etc/sysctl.conf
shutdown -r now
jtune -j 1 -i
jtune -m 200 -M 300

Enjoy