March 6th, 2010
Nice screenshot of 100% swap space being used on a popular but ill Perl app running under ModPerl::PerlRun.
Tasks: 85 total, 2 running, 83 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 14.8%sy, 0.0%ni, 17.0%id, 68.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8174024k total, 8132492k used, 41532k free, 284k buffers
Swap: 2096472k total, 2096472k used, 0k free, 5648k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
308 root 10 -5 0 0 0 D 17.5 0.0 0:05.14 kswapd0
15985 apache 18 0 19.4g 7.7g 84 D 15.1 98.3 0:12.09 httpd
15996 root 16 0 12740 624 368 R 5.4 0.0 0:00.48 top
1 root 16 0 10348 124 32 S 0.0 0.0 0:01.69 init
The test server is a Dell 1950 with 8 GB RAM running CentOS 5.4 x64 and Apache 2.x.
The above problem illustrates one of the many reasons that almost all hosting providers adopted PHP instead of mod_perl.
PHP gives you good performance without the headaches of mod_perl, which get magnified in a shared environment.
However, if you have a dedicated machine, mod_perl is a great way to accelerate a Perl application as long as the program is reasonably well-behaved.
Posted in Linux, Open Source, Perl | No Comments »
March 2nd, 2010
When I use a firewall generator, like Redhat’s system-config-securitylevel, this is not what I expect when I tell it to allow only ports 22, 80, and 443:
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
According to /etc/services, 224.0.0.251:5353 is multicast DNS, 631 is for printing, and 50 and 51 are “Remote Mail Checking Protocols”.
How does that help my webserver exactly? Unix is not supposed to volunteer your machine for things that were not requested.
And those extra ports are useless when in linux runlevel 3 (console mode) since no desktop environment can run without X, nevermind the INPUT and FORWARD ACCEPT defaults.
This free web tool makes a lot more sense to me:
Generated by iptables-save v1.3.5 on Tue Mar 2 23:33:21 2010
*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
COMMIT
# Completed on Tue Mar 2 23:33:21 2010
Just put that in /etc/sysconfig/iptables on your Redhat or CentOS webserver then:
chkconfig iptables on
service iptables restart
iptables is started in script 08iptables, which is after several other services but before networking is started, which sounds ok.
mista.nu: Simple Iptables Script Generator
Redhat Product Pricing
Posted in Linux, Open Source, Tech, Toys | No Comments »
February 27th, 2010
Just trying out HAProxy in a new data center for http load balancing.
I’m not expecting a lot of site traffic initially, but using a load balancer from Day One lets you get all the data center servers assigned, and allows sysadmins to do maintenance whenever convenient.
I was looking around at similar Open Source software, and what caught my attention about HAProxy is that Willy “obsessed with reliability” Tarreau is the author.
HAProxy has several nice features, including speed (fast enough for 10 GB connections at up to 132,000 connections per second), and epoll, cookie, multicore, chroot support and much more.
There are ports available for most Unix systems, including linux, FreeBSD and Solaris.
Here is the build script I wrote for a Dell 1950 (after installing libpcre):
#!/bin/bash
make clean
make TARGET=linux26 USE_PCRE=1 ARCH=x86_64
# no make test
make install
You can do a graceful restart of HAProxy by adding this to your startup script (the tr is needed to handle when nbproc > 1):
graceful() {
/usr/local/sbin/haproxy -c -q -f /etc/haproxy.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with 'haproxy check'."
return 1
fi
/usr/local/sbin/haproxy -V -f /etc/haproxy.cfg -p /var/run/haproxy.pid -sf
`tr '\n' ' ' < /var/run/haproxy.pid`
}
HAProxy Documentation
wht: HAproxy - Quick and Dirty HTTP Load balancing Tutorial on Redhat/Centos
Session Based Load Balancing with HAproxy
tito: Zero-Downtime Restarts with HAProxy
Building an easy and scalable load-balanced high-availability web-hosting solution. Part One : The front.
How To Tell Apache To Not Log Certain Requests In Its Access Log
Pricing for Zeus software on Amazon EC2
microsoft.com: Network Load Balancing Technical Overview
loadbalancer.org: FAQ
Tenereillo.com: Why DNS Based Global Server Load Balancing (GSLB) Doesn't Work (2005)
davew: Thoughts on Global Server Load Balancing
ksalchow: Shame on GSLB? Shame on Me?
Vegan Load Balancing Mailing List
Posted in Linux, Tech, Toys | No Comments »
February 26th, 2010
Congrats to Frank Robinson on his new helicopter product, the Turbine R66, for $770,000 with standard equipment.
His R44 piston was so economical that Bell shutdown production of the Jetranger line, and the R66 is another amazing machine.
And manufactured in the USA.
Heli-Expo 2010: Robinson’s new baby leads the way
avweb.com: Robinson Sets R66 Price At $770,000
Robinson Helicopter
Posted in Business, Flying | No Comments »
February 19th, 2010
The Supermicro X7SPA-HF is a near-server-class $200 Atom-based motherboard.
Pluses are extreme low-power operation (under 35 watts for an entire system), dual core with HT, dual gigabit NICs and IPMI remote mgmt. (Reportedly there are still some minor bugs in the IPMI code, for example with mounting ISO images remotely.)
Minuses are that memory is limited to 4 GB non-ECC single-channel SO-DIMM (notebook) DDR2 RAM, but hey, an Atom is a slow processor suitable for file-serving and not heavy calculations. I have heard of serving ftp traffic at 300 Mbps with this mobo.
I’m thinking of using it in the data center where power is at a premium, but having a dedicated box is nice – uses like an HAProxy appliance or a PXE/kickstart build server.
One use of Atoms is “rack filler”: The low-power Atom servers can be distributed into racks where there’s not enough power for another 2-amp Dell server, but where there’s still a half amp or so left on a circuit.
The power budget looks like:
| mobo/cpu |
21 watts |
| 8800 RPM fan |
2 watts |
| notebook hard drive |
5 watts |
| 4 GB DDR2 SO-DIMM RAM |
1 watt |
| 80%-efficient power supply losses |
6 watts |
| Total |
35 watts |
The total cost of this server is about $450 to assemble from the above mobo and Supermicro case/80% PSU.
Thanks to Colin from HE for telling me about his experiences with the D510.
wht: 0.2amp server with Intel Atom D510
anandtech.com: Intel Atom D510: Pine Trail Boosts Performance, Cuts Power
atacom.com: MB13_SUPE_X7_HF pricing
APAQ Digital: Atom Servers
GCC 4.5 Release Series Changes, New Features, and Fixes: Support for the Intel Atom processor is now available through the -march=atom and -mtune=atom options.
tomshardware.com: UPDATE: Apple Mac Mini Based on Nvidia Ion (Rumors)
engadget: Atom N470 at CES
Posted in Business, Linux, Tech, Toys | No Comments »
February 14th, 2010
Posted in Linux, Open Source, Tech, Toys | No Comments »
February 14th, 2010
I made a linux cluster using 16 dual Opteron 248 machines, gigabit Ethernet and CentOS 5.4 DVD with kickstart.
Nodes can be remotely rebuilt upon command in about 3 minutes each in parallel, with no manual intervention, as long as you’re careful to treat nodes like appliances and don’t save data on them.
Some tips to save time and effort are:
Likely I will move to Rocks Clusters later, which is also derived from CentOS.
The Rocks Clusters people handle PXE boot in a more sophisticated way, configuring PXE boot to read the kernel image from the local hard drive, sparing tftpd from being swamped on clusters of thousands of nodes. Their unboot utility is called cluster-kickstart-pxe.
hp.com: Setting up a Linux PXE server and integrating clients – Howto (c00257674.pdf)
RedHat Linux KickStart HOWTO
Remote Network Boot via PXE
communities.vmware.com: How to Pass Parameters to a Kickstart Script?
aboveaverageurl.com: PXE Booting
Howtoforge: Unattended Fedora 8 Installation With NFS And Kickstart
Yu Dong, NASA: Installing Linux over Network: PXE, DHCP, TFTP, NFS and Kickstart
Rocks Cluster 5.3: Forcing a Re-install at Next PXE Boot
[Rocks-Discuss]cluster-fork ‘/boot/kickstart/cluster-kickstart–start’has no effect?
IEEE OUI and Company_id Assignments (MAC Address Database)
ftp://ftp.rocksclusters.org/pub/rocks
Reading Dell service tag number – dmidecode -s system-serial-number
Debian – setting hostname from DHCP result
Posted in Linux, Open Source, Perl, Tech, Toys | No Comments »
February 9th, 2010
While testing the Dell OpenManage 6.2 update recently, the main ext3 filesystem superblock on a gpt partition was damaged by the CentOS 5.4 installer.
I did not ask the CentOS installer to touch the non-system partitions in any way, but it happened.
Fortunately, mke2fs writes superblock backups to each filesystem in case something bad happens.
e2fsck -b could be used to recover a superblock from a copy, but I found a friendlier tool …
I used an Open Source tool by Christophe GRENIER called TestDisk to scan for a backup superblock, and overwrote the bad superblock in about 30 seconds. Then I added the original mount label and mounted the filesystem:
# testdisk_static (or testdisk_static /log /dev/sdb)
# parted /dev/sdb name 1 /data (works on gpt partition types)
# mount -a
# ls -l /data
# tune2fs -l /dev/sdb1
TestDisk worked perfectly, even on a complex system with Perc 6i and Perc 5e RAID controllers with 4 TB partitions, but you must carefully read and navigate TestDisk’s menus, and actually write the new superblock to disk for each filesystem that was lost. TestDisk can also be used to recover files and preventively to save superblocks before an issue occurs.
There are versions of TestDisk for several operating systems, including Windows, Linux 2.4, Linux 2.6 and FreeBSD.
Note that parted also has a rescue mode for partitions:
(parted) help rescue
rescue START END # rescue a lost partition near START and END
Other tools to look at when fixing linux filesystems include tune2fs and partprobe.
For deeper insight into ext2 and ext3 recovery, search for the excellent articles by Ted Ts’o.
Posted in Linux, Open Source, Tech | No Comments »