[Acadix, LLC]
IT Consulting, Software Development
Small Business, Schools, Residential

[Home] [Services] [Philosophy] [Experience] [Software] [Tips] [Contact]

Software

Auto-admin
Desktop-installer
Cluster-admin
Roboctl
OpenVex
APP - Assembly Pre-Processor

Why FreeBSD?

FreeBSD's unparalleled stability, near-optimal efficiency, and easy software management via ports and packages make it an ideal platform for high performance computing (HPC) clusters. There is no better platform for running huge computational jobs that may require weeks or months of uninterrupted uptime. FreeBSD is the only operating system we've have found that offers enterprise stability combined with a top-tier software management system (FreeBSD ports).

Many FreeBSD HPC clusters are in use today, serving science, engineering, and other disciplines. FreeBSD is a supported platform on Amazon's EC2 virtual machine service. It is also a little-known fact that the special effects for the movie "Matrix" were rendered on a FreeBSD cluster.

There are a couple of limitations with FreeBSD, although they are of no concern to most users:

  1. While FreeBSD can run most 32-bit Linux binaries natively (with better performance than Linux in some cases), it does not currently support 64-bit Linux binaries. Hence, it will not run some commercial applications such as ANSYS and Abaqus. 64-bit Linux compatibility is currently in development, so this will likely change in the near future.
  2. You may have difficulty finding drivers for proprietary network technologies such as InfiniBand, so FreeBSD clusters needing very high-speed networking may be limited to 10 GigE. This is also not an issue for most users. In fact, many Linux cluster designers have chosen 10 GigE over InfiniBand and other proprietary networks.

For typical users, there's no better system for HPC. A small FreeBSD cluster can be built from the ground up in a few hours (with the help of cluster-admin), and most mainstream mathematical and scientific software can be added from FreeBSD's ports/packages system in a matter of seconds.

Users who need to run 64-bit commercial Linux applications are advised to run some sort of enterprise Linux such as Red Hat Enterprise Linux (RHEL), SUSE Enterprise, or one of the free derivatives such as CentOS or Scientific Linux. Other Linux distributions are not generally supported by vendors of products such as ANSYS and Abaqus.

The main disadvantage of enterprise Linux platforms (compared to FreeBSD or community Linux distributions such as Debian and Gentoo) is the outdated base installations and the limited and outdated collection of packages available in the Yum repository. (Stability and long-term binary compatibility in enterprise Linux systems is maintained by running older, time-tested versions of system software.)

I've had great success using pkgsrc to manage more up-to-date open source software on RHEL. The pkgsrc system is well-supported on Linux, offers far more packages than Yum, and can install a complete set of packages that are almost completely independent from the base Linux installation. This allows the base system (including RPMs from Yum) to be updated without breaking software installed by pkgsrc.

Cluster Admin

Cluster-admin is a suite of tools for building and managing FreeBSD HPC clusters. It automates the process of configuring a head node and compute nodes for a high performance computing cluster, and managing configuration and software after installation.

Screen shot of a small cluster built with cluster-admin:

[Ganglia Screen Shot]

Cluster-admin automates the setup of a cluster using either the Sun Grid Engine or Torque scheduler, the Open MPI parallel programming suite, and the Ganglia web-based network monitoring suite. It also helps synchronize system files on the compute nodes, manage user accounts, and manage software on compute nodes.

The design philosophy centers on simplicity and minimizing interdependence of the cluster nodes. Each compute node contains a fully independent operating system installation and computational software installations on its own local storage. Compared with clusters that utilize shared storage more extensively, this strategy increases the initial cluster setup time slightly in exchange for simpler management, less "noise" on the local network, fewer single points of failure, and fewer bottlenecks.

Implementation of this design is facilitated by leveraging the extensive systems management tools provided by the FreeBSD base system, including the ports system which automates the installation of nearly every mainstream open source application in existence.

The cluster-admin tools are written primarily as Bourne shell scripts using standard Unix tools to configure the system, and utilizing ports/packages for all software management. The software is currently considered a prototype and undergoing heavy development and redesign.

[cluster diagram] In many clusters, the head node is multi-homed (has two network interfaces) and serves as the gateway for the cluster. Cluster-admin supports this configuration, but be aware that it complicates the setup of the head node as well as configuration of services running on the head node, including the scheduler and the Ganglia resource monitor.

The recommended hardware configuration uses a single network interface on all nodes, including the head node, and a separate router. Many network switches have built-in routing capability. If you're using a simple switch for your cluster, you can use an inexpensive hardware router, or quickly and cheaply build a sophisticated firewall router using another PC and pfSense.

To get started, do a basic FreeBSD installation on your head node including all source code and ports. Then install cluster-admin via ports or packages and run "man cluster-setup".