Linux CentOS Cluster Setup Tips

I made a linux cluster using 16 dual Opteron 248 machines, gigabit Ethernet and CentOS 5.4 DVD with kickstart.

Nodes can be remotely rebuilt upon command in about 3 minutes each in parallel, with no manual intervention, as long as you’re careful to treat nodes like appliances and don’t save data on them.

Some tips to save time and effort are:

• collect the MAC addresses of all nodes at one time using the most efficient possible way, either from a manifest, or simply power all the nodes on and type on one node:

  ping -b 10.0.0.255 or
  fping -A -q -c 1 -g 10.0.0.0/24 or
  nmap -sP 10.0.0.0/24
  and
  arp -n
  

• decide what your policy will be for dealing with machine failure. Fix, remove or just power it down?
• on your main client PXE test node, which you may do 50 reinstalls on, save boot time by disabling memory checking, boot splash screens, etc. and use small filesystems during initial testing
• install one machine by hand from DVD first to generate the anaconda-ks.cfg file, which contains your preferred package list (the CentOS installer itself uses kickstart even for local installs)
• I found that having kickstart fetch the distro files using HTTP was a lot easier to setup and troubleshoot than NFS, and easier to secure later.
• it’s common to use a BIOS boot order of “PXE, CD, HD” on each machine to bootstrap the cluster if the hard drive is not blank, then switch to “CD, HD, PXE” after linux is successfully installed and you’re able to login remotely. Subsequent reboots will try the HD first unless you force a PXE boot, which can be done with a script I wrote called unboot that both deactivates the boot partition and erases the MBR:

  #!/bin/bash -x
  
  # do this in a device-naming portable way...
  #
  # parted /dev/hda set 1 boot off
  # dd if=/dev/zero of=/dev/hda bs=512 count=1
  
  DEV=`perl -e '$x=qx[df -l /]; $x=~m[/dev/(hda|sda)\d];print $1'`
  dd if=/dev/$DEV of=/root/mbr.boot bs=512 count=1
  parted /dev/$DEV set 1 boot off
  dd if=/dev/zero of=/dev/$DEV bs=512 count=1
  sync
  

• do a web search for several good sample kickstart files. I found that merging 3 or 4 good ones provided very nice results.
• by default, kickstart configures your networking with DHCP if you are doing network installs, but you can overwrite that in your post-install section with multiple static IP addresses if desired.
• test your tftpd setup from the server (or another node) with tftp localhost -v -c get pxelinux.0
• do tail -f /var/log/messages on the DHCP server to monitor DHCP requests by client nodes.
• Make sure “/var/lib/dhcp/dhcpd.leases” exists.
• Note that doing yum -y update in the postinstall script will significantly increase installation time.
• Note that kickstart creates the MBR at the very end of the installation process. So if you unboot your machine and interrupt kickstart, then it will not be able to boot from the hard drive. Instead you will only see a blinking cursor on the console.

Likely I will move to Rocks Clusters later, which is also derived from CentOS. Rocks is seriously funded by the NSF and has the goal of being simple to install, manage and use – meaning, no cluster sysadmin needed.

The Rocks Clusters people handle PXE boot in a more sophisticated way, configuring PXE boot to read the kernel image from the local hard drive, sparing tftpd from being swamped on clusters of thousands of nodes. Their unboot utility is called cluster-kickstart-pxe.

I use dsh, part of the ClusterIt Toolkit, which also has the pcp (parallel copy) command. Currently ClusterIt works fine on most Unix environments, with the exception of dtop, which segfaults on linux.

Another parallel execution tool of the same name is dsh (Dancer’s Distribute Shell) for issuing cluster-wide commands.

The dsh Howto has some nice tips, including copying files over dsh and scp:

To copy files from master's /etc directory to /etc directory of all nodes:
dsh -a "scp master:/etc/passwd /etc/passwd"

hp.com: Setting up a Linux PXE server and integrating clients – Howto (c00257674.pdf)

RedHat Linux KickStart HOWTO
Remote Network Boot via PXE
communities.vmware.com: How to Pass Parameters to a Kickstart Script?
aboveaverageurl.com: PXE Booting
Howtoforge: Unattended Fedora 8 Installation With NFS And Kickstart
Yu Dong, NASA: Installing Linux over Network: PXE, DHCP, TFTP, NFS and Kickstart
Rocks Cluster 5.3: Forcing a Re-install at Next PXE Boot
[Rocks-Discuss]cluster-fork ‘/boot/kickstart/cluster-kickstart–start’has no effect?
IEEE OUI and Company_id Assignments (MAC Address Database)
ftp://ftp.rocksclusters.org/pub/rocks
Reading Dell service tag number – dmidecode -s system-serial-number
Debian – setting hostname from DHCP result
Golfing the Extraction of IP Addresses from ifconfig
The Tool Shed: ClusterIt
Fedora Cobbler Install Server
Func: Fedora Unified Network Controller
High Performance Computing Rocks at NYU: A Look at Rocks Cluster Distribution for HPC Researchers (2004)
Shell Scripting DSL in Ruby
Perl CPAN Grid::Machine
Creating Hadoop pe under SGE
Multi-level, Colorful and “Jazzy” PXE Boot Menus Made Easy
Make a Menu for PXE linux
PXE Magic: Flexible Network Booting with Menus