FreeBSD/Tracking Ports In A Production Environment

From TomJudge.com
Jump to: navigation, search

The aim of this article is to discuss the pros and cons of tracking the ports tree in a production environment. We have a large number of servers at $WORK (some where in the order of 700 images including physical hosts, jails and virtual machines). We also have 3 internally supported FreeBSD releases in the production environment.

Contents

Where We Are

At the moment when we build a new FreeBSD release for the production environment we do the following:

  1. Check out a current ports tree from the our internal CVS mirror.
  2. Load this snapshot into our subversion server.
  3. Apply a number of changes to the tree to make it suite our environment such as:
    • Changing the default PostgreSQL version
    • Changing the default MySQL version
    • Down grading ports known to have regressions when used by our internal software systems.
    • Adding internal ports that are not available upstream
    • Adding patches to ports that are specific to our uses
  4. Load the lists of available ports into our role based package management system
  5. Add the packages for this build to each role.
  6. Run the build though tinderbox to create a binary package tree

The ports tree used to build the release is then not updated at all for the support lifetime of the release. The justification for this is that the software development team (consisting of around 50 developers) like to have a stable production environment where things are not changing that could introduce bugs in our products.

Due to this policy we have a number of packages deployed on systems that have CERT advisories issued against them which is not an ideal place to be.

The Upgrade Policy (Per Port)

  1. If a port does not exist in the ports tree, we will add and deploy the port.
  2. If a port does exist in the ports tree but has not been deployed to any role, we will deploy it.
  3. If a port has been deployed already, Infrastructure and Development will collaborate to determine a solution. This is likely to involve:
    1. Identifying who is using a port
    2. Discussions / testing to determine if the proposed change would be a breaking or undesirable change for those systems
    3. For changes that can be rolled out everywhere:
      1. Determining a sensible method / timing for upgrades so as to not put existing systems at risk.
    4. For more complex changes
      1. Evaluating other options (such as flavour scripts / local installs) to resolve issues

Where We Want To Be

In a position to track the ports tree either fully in semi real time or per port based on security issues.

Other like to haves:

  • Deploy multiple versions of a port with in the same release stream (I.e. MySQL).

Things to consider

  • Wide sweeping changes to the ports tree could cause major issues for automated. (Examples the change of the X11BASE, or splitting of Xorg into hundreds of tiny ports.)
  • Changes in shared libraries from upstream vendors that do not bump the shared library number when there is an ABI change.
  • Changes to ports that change the behaviour so that it differs from the behaviour of the version the internal software teams developed against.
  • Deploying dynamically linked binaries to systems that are not in ports, specifically product binaries.
  • Ports being removed, or having categories changed
  • The number of changes that we make to the tree internally can make updating the tree a very time consuming task.
  • New builds will take 1-2 weeks to complete due to the number of packages being built.
  • Backing out changes should be quick and simple.

How to achieve some middle ground

From spending some time considering our processes we have some options:

  1. One time snapshot per release with security updates
  2. Periodic full snapshots followed by security updates
  3. Regular tracking of vendor HEAD

One Time Snapshots + Security Updates

Taking a snapshot of the ports tree at the point we build the production release.

Pros Cons
  • Fewer changes in the production package set.
  • Builds only need updating when there are security updates.
  • Due to the smaller change sets, there is less testing.
  • One time integration of custom changes per release cycle.
  • Sweeping changes upstream can make back porting changes more difficult.

Periodic Snapshots

Taking snapshots of the ports tree monthly/bimonthly.

Pros Cons
  • All packages are updated regularly from upstream.
  • Less individual edits.
  • Sweeping changes upstream can cause large numbers of updates.
  • Updates are going to create large change sets, creating large amounts of work in testing.
  • Each snapshot requires a large amount of work integrating custom changes. (Dependent on update procedure).
  • Sweeping changes upstream can cause difficulty upgrading existing installations. Examples of changes like this are the Xorg split and the change of X11BASE.
  • The current ports HEAD is not tested against older releases that are no longer supported.

Tracking Ports HEAD

Daily updates to the tree from FreeBSD, this process generally has the same pros and cons as periodic snapshots.

Retrieved from "http://www.tomjudge.com/index.php/FreeBSD/Tracking_Ports_In_A_Production_Environment"
Personal tools