I’ll have more to say later about why the tips are useful, but the title of “Top 20 OpenSSH Server Best Security Practices” is not really accurate.

Top 20 OpenSSH Server Best Security Practices

Tasks:  85 total,   2 running,  83 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us, 14.8%sy,  0.0%ni, 17.0%id, 68.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8174024k total,  8132492k used,    41532k free,      284k buffers
Swap:  2096472k total,  2096472k used,        0k free,     5648k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  308 root      10  -5     0    0    0 D 17.5  0.0   0:05.14 kswapd0
15985 apache    18   0 19.4g 7.7g   84 D 15.1 98.3   0:12.09 httpd
15996 root      16   0 12740  624  368 R  5.4  0.0   0:00.48 top
    1 root      16   0 10348  124   32 S  0.0  0.0   0:01.69 init

The test server is a Dell 1950 with 8 GB RAM running CentOS 5.4 x64 and Apache 2.x.

The above problem illustrates one of the many reasons that almost all hosting providers adopted PHP instead of mod_perl.

PHP gives you good performance without the headaches of mod_perl, which get magnified in a shared environment.

However, if you have a dedicated machine, mod_perl is a great way to accelerate a Perl application as long as the program is reasonably well-behaved.

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

According to /etc/services, 224.0.0.251:5353 is multicast DNS, 631 is for printing, and 50 and 51 are “Remote Mail Checking Protocols”.

How does that help my webserver exactly? Unix is not supposed to volunteer your machine for things that were not requested.

And those extra ports are useless when in linux runlevel 3 (console mode) since no desktop environment can run without X, nevermind the INPUT and FORWARD ACCEPT defaults.

This free web tool makes a lot more sense to me:

 Generated by iptables-save v1.3.5 on Tue Mar  2 23:33:21 2010
*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
COMMIT
# Completed on Tue Mar  2 23:33:21 2010

Just put that in /etc/sysconfig/iptables on your Redhat or CentOS webserver then:

chkconfig iptables on
service iptables restart

iptables is started in script 08iptables, which is after several other services but before networking is started, which sounds ok.

mista.nu: Simple Iptables Script Generator
Redhat Product Pricing

I’m not expecting a lot of site traffic initially, but using a load balancer from Day One lets you get all the data center servers assigned, and allows sysadmins to do maintenance whenever convenient.

I was looking around at similar Open Source software, and what caught my attention about HAProxy is that Willy “obsessed with reliability” Tarreau is the author.

HAProxy has several nice features, including speed (fast enough for 10 GB connections at up to 132,000 connections per second), and epoll, cookie, multicore, chroot support and much more.

There are ports available for most Unix systems, including linux, FreeBSD and Solaris.

Here is the build script I wrote for a Dell 1950 (after installing libpcre):

#!/bin/bash

make clean
make TARGET=linux26 USE_PCRE=1 ARCH=x86_64
# no make test
make install

You can do a graceful restart of HAProxy by adding this to your startup script (the tr is needed to handle when nbproc > 1):

graceful() {
  /usr/local/sbin/haproxy -c -q -f /etc/haproxy.cfg
  if [ $? -ne 0 ]; then
    echo "Errors found in configuration file, check it with 'haproxy check'."
    return 1
  fi
  /usr/local/sbin/haproxy -V -f /etc/haproxy.cfg -p /var/run/haproxy.pid -sf
`tr '\n' ' ' < /var/run/haproxy.pid`
}

HAProxy Documentation
wht: HAproxy - Quick and Dirty HTTP Load balancing Tutorial on Redhat/Centos
Session Based Load Balancing with HAproxy
tito: Zero-Downtime Restarts with HAProxy
Building an easy and scalable load-balanced high-availability web-hosting solution. Part One : The front.
How To Tell Apache To Not Log Certain Requests In Its Access Log
Pricing for Zeus software on Amazon EC2
microsoft.com: Network Load Balancing Technical Overview
loadbalancer.org: FAQ
Tenereillo.com: Why DNS Based Global Server Load Balancing (GSLB) Doesn't Work (2005)
davew: Thoughts on Global Server Load Balancing
ksalchow: Shame on GSLB? Shame on Me?
Vegan Load Balancing Mailing List

His R44 piston was so economical that Bell shutdown production of the Jetranger line, and the R66 is another amazing machine.

And manufactured in the USA.

Heli-Expo 2010: Robinson’s new baby leads the way
avweb.com: Robinson Sets R66 Price At $770,000
Robinson Helicopter

Pluses are extreme low-power operation (under 35 watts for an entire system), dual core with HT, dual gigabit NICs and IPMI remote mgmt. (Reportedly there are still some minor bugs in the IPMI code, for example with mounting ISO images remotely.)

Minuses are that memory is limited to 4 GB non-ECC single-channel SO-DIMM (notebook) DDR2 RAM, but hey, an Atom is a slow processor suitable for file-serving and not heavy calculations. I have heard of serving ftp traffic at 300 Mbps with this mobo.

I’m thinking of using it in the data center where power is at a premium, but having a dedicated box is nice – uses like an HAProxy appliance or a PXE/kickstart build server.

One use of Atoms is “rack filler”: The low-power Atom servers can be distributed into racks where there’s not enough power for another 2-amp Dell server, but where there’s still a half amp or so left on a circuit.

The power budget looks like:

The total cost of this server is about $450 to assemble from the above mobo and Supermicro case/80% PSU.

Thanks to Colin from HE for telling me about his experiences with the D510.

wht: 0.2amp server with Intel Atom D510
anandtech.com: Intel Atom D510: Pine Trail Boosts Performance, Cuts Power
atacom.com: MB13_SUPE_X7_HF pricing
APAQ Digital: Atom Servers
GCC 4.5 Release Series Changes, New Features, and Fixes: Support for the Intel Atom processor is now available through the -march=atom and -mtune=atom options.
tomshardware.com: UPDATE: Apple Mac Mini Based on Nvidia Ion (Rumors)
engadget: Atom N470 at CES

Must be a fun time to be an IT security guy these days. *shudder*

wikipedia.org: Hamachi
An open source alternative to Hamachi: tinc
List of Hamachi Alternatives|Virtual Private Network Adapters
Hamachi: Roll Your Own VPNs the Fast and Free Way, Create and Manage Your Own Virtual Private Networks ]]> http://www.jebriggs.com/blog/2010/02/windows-user-vpn-software/feed/ 0 http://www.jebriggs.com/blog/2010/02/linux-centos-cluster-setup-tips/ http://www.jebriggs.com/blog/2010/02/linux-centos-cluster-setup-tips/#comments Sun, 14 Feb 2010 22:45:51 +0000 Administrator http://www.jebriggs.com/blog/?p=1630 I made a linux cluster using 16 dual Opteron 248 machines, gigabit Ethernet and CentOS 5.4 DVD with kickstart.

Nodes can be remotely rebuilt upon command in about 3 minutes each in parallel, with no manual intervention, as long as you’re careful to treat nodes like appliances and don’t save data on them.

Some tips to save time and effort are:

• collect the MAC addresses of all nodes at one time using the most efficient possible way, either from a manifest, or simply power all the nodes on and type on one node:

  ping -b 10.0.0.255 or
  fping -A -q -c 1 -g 10.0.0.0/24 or
  nmap -sP 10.0.0.0/24
  and
  arp -n
  

• on your main client test node, which you may do 50 reinstalls on, save boot time by disabling memory checking, boot splash screens, etc. and use small filesystems during initial testing
• install one machine by hand from DVD first to generate the anaconda-ks.cfg file, which contains your preferred package list (the CentOS installer itself uses kickstart even for local installs)
• I found that having kickstart fetch the distro files using HTTP was a lot easier to setup and troubleshoot than NFS, and easier to secure later.
• it’s common to use a BIOS boot order of “PXE, CD, HD” on each machine to bootstrap the cluster if the hard drive is not blank, then switch to “CD, HD, PXE” after linux is successfully installed and you’re able to login remotely. Subsequent reboots will try the HD first unless you force a PXE boot, which can be done with a script I wrote called unboot that both deactivates the boot partition and erases the MBR:

  #!/bin/bash
  
  parted /dev/hda set 1 boot off
  dd if=/dev/zero of=/dev/hda bs=512 count=1
  

• do a web search for several good sample kickstart files. I found that merging 3 or 4 good ones provided very nice results.
• by default, kickstart configures your networking with DHCP if you are doing network installs, but you can overwrite that in your post-install section with multiple static IP addresses if desired.
• test your tftpd setup from the server (or another node) with tftp localhost -v -c get pxelinux.0
• do tail -f /var/log/messages on the DHCP server to monitor DHCP requests by client nodes.
• Make sure “/var/lib/dhcp/dhcpd.leases” exists.

Likely I will move to Rocks Clusters later, which is also derived from CentOS.

The Rocks Clusters people handle PXE boot in a more sophisticated way, configuring PXE boot to read the kernel image from the local hard drive, sparing tftpd from being swamped on clusters of thousands of nodes. Their unboot utility is called cluster-kickstart-pxe.

hp.com: Setting up a Linux PXE server and integrating clients – Howto (c00257674.pdf)

RedHat Linux KickStart HOWTO
Remote Network Boot via PXE
communities.vmware.com: How to Pass Parameters to a Kickstart Script?
aboveaverageurl.com: PXE Booting
Howtoforge: Unattended Fedora 8 Installation With NFS And Kickstart
Yu Dong, NASA: Installing Linux over Network: PXE, DHCP, TFTP, NFS and Kickstart
Rocks Cluster 5.3: Forcing a Re-install at Next PXE Boot
[Rocks-Discuss]cluster-fork ‘/boot/kickstart/cluster-kickstart–start’has no effect?
IEEE OUI and Company_id Assignments (MAC Address Database)
ftp://ftp.rocksclusters.org/pub/rocks
Reading Dell service tag number – dmidecode -s system-serial-number
Debian – setting hostname from DHCP result

I did not ask the CentOS installer to touch the non-system partitions in any way, but it happened.

Fortunately, mke2fs writes superblock backups to each filesystem in case something bad happens.

e2fsck -b could be used to recover a superblock from a copy, but I found a friendlier tool …

I used an Open Source tool by Christophe GRENIER called TestDisk to scan for a backup superblock, and overwrote the bad superblock in about 30 seconds. Then I added the original mount label and mounted the filesystem:

# testdisk_static (or testdisk_static /log /dev/sdb)
# parted /dev/sdb name 1 /data (works on gpt partition types)
# mount -a
# ls -l /data
# tune2fs -l /dev/sdb1

TestDisk worked perfectly, even on a complex system with Perc 6i and Perc 5e RAID controllers with 4 TB partitions, but you must carefully read and navigate TestDisk’s menus, and actually write the new superblock to disk for each filesystem that was lost. TestDisk can also be used to recover files and preventively to save superblocks before an issue occurs.

There are versions of TestDisk for several operating systems, including Windows, Linux 2.4, Linux 2.6 and FreeBSD.

Note that parted also has a rescue mode for partitions:

(parted) help rescue
  rescue START END      # rescue a lost partition near START and END

Other tools to look at when fixing linux filesystems include tune2fs and partprobe.

For deeper insight into ext2 and ext3 recovery, search for the excellent articles by Ted Ts’o.

# omreport storage controller
No controllers found

That sure got my attention …

There were at least 2 issues caused by this update:

1. Although the individual packages were fine, the installer script had bugs that resulted in the combination of packages to not work correctly, regardless of whether you were updating an old system, or a fresh CentOs installation. Even though disk volumes were still mountable, most omreport options did not. Somebody posted a script that usually fixes that on the Dell forum, and I have added some modprobe commands that some people also recommended:

   #/bin/bash
   
   # this script based on Dell Forums samples
   
   /sbin/modprobe ipmi_si
   /sbin/modprobe ipmi_devintf
   
   yum remove srvadmin*
   yum install srvadmin-all
   yum install dell_ft_install
   cd /opt/dell/srvadmin/etc
   ./autoconf_cim_component.sh
   yum remove srvadmin-iws srvadmin-webserver srvadmin-jre
   srvadmin-services.sh start
   omreport storage controller # now works properly, or reboot first
   
   #Somebody really pooched the dependancies list in the OMSA 6.2 install !!!!
   

2. omreport was installed in a new location, so the commonly-used check_openmanage monitoring perl script failed to find it. A simple edit fixes that:

/usr/lib64/nagios/plugins/contrib/check_openmanage:

#
# Locate the omreport binary
#
sub find_omreport {
    # Possible full paths for omreport

    my @omreport_paths
      = (
         '/usr/bin/omreport',                            # default on Linux
         '/opt/dell/srvadmin/oma/bin/omreport.sh',       # alternate on Linux
         '/opt/dell/srvadmin/oma/bin/omreport',          # alternate on Linux
+         '/opt/dell/srvadmin/bin/omreport',               # alternate on Linux
         'c:\progra~1\dell\sysmgt\oma\bin\omreport.exe', # default on Windows
         'c:\progra~2\dell\sysmgt\oma\bin\omreport.exe', # default on Windows x64
        );

Dell power-edge list: OpenManage 6.2 Storage Controller not found fix
Dell Forums: OMSA daemons appear to crash a minute after startup