Archive for the ‘Open Source’ Category

SSH Configuration Tips

Wednesday, March 10th, 2010

I came across a useful blog post with 20 SSH configuration tips.

I’ll have more to say later about why the tips are useful, but the title of “Top 20 OpenSSH Server Best Security Practices” is not really accurate.

Top 20 OpenSSH Server Best Security Practices

linux 100% swap Screenshot

Saturday, March 6th, 2010

Nice screenshot of 100% swap space being used on a popular but ill Perl app running under ModPerl::PerlRun. :)

Tasks:  85 total,   2 running,  83 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us, 14.8%sy,  0.0%ni, 17.0%id, 68.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8174024k total,  8132492k used,    41532k free,      284k buffers
Swap:  2096472k total,  2096472k used,        0k free,     5648k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  308 root      10  -5     0    0    0 D 17.5  0.0   0:05.14 kswapd0
15985 apache    18   0 19.4g 7.7g   84 D 15.1 98.3   0:12.09 httpd
15996 root      16   0 12740  624  368 R  5.4  0.0   0:00.48 top
    1 root      16   0 10348  124   32 S  0.0  0.0   0:01.69 init

The test server is a Dell 1950 with 8 GB RAM running CentOS 5.4 x64 and Apache 2.x.

The above problem illustrates one of the many reasons that almost all hosting providers adopted PHP instead of mod_perl.

PHP gives you good performance without the headaches of mod_perl, which get magnified in a shared environment.

However, if you have a dedicated machine, mod_perl is a great way to accelerate a Perl application as long as the program is reasonably well-behaved.

Redhat system-config-securitylevel Surprises

Tuesday, March 2nd, 2010

When I use a firewall generator, like Redhat’s system-config-securitylevel, this is not what I expect when I tell it to allow only ports 22, 80, and 443:

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

According to /etc/services, 224.0.0.251:5353 is multicast DNS, 631 is for printing, and 50 and 51 are “Remote Mail Checking Protocols”.

How does that help my webserver exactly? Unix is not supposed to volunteer your machine for things that were not requested.

And those extra ports are useless when in linux runlevel 3 (console mode) since no desktop environment can run without X, nevermind the INPUT and FORWARD ACCEPT defaults.

This free web tool makes a lot more sense to me:

 Generated by iptables-save v1.3.5 on Tue Mar  2 23:33:21 2010
*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
COMMIT
# Completed on Tue Mar  2 23:33:21 2010

Just put that in /etc/sysconfig/iptables on your Redhat or CentOS webserver then:

chkconfig iptables on
service iptables restart

iptables is started in script 08iptables, which is after several other services but before networking is started, which sounds ok.

mista.nu: Simple Iptables Script Generator
Redhat Product Pricing

Windows User VPN Software

Sunday, February 14th, 2010

So … Windows end-users are setting up their own VPNs with Hamachi and other networking programs.

Must be a fun time to be an IT security guy these days. *shudder*

wikipedia.org: Hamachi
An open source alternative to Hamachi: tinc
List of Hamachi Alternatives|Virtual Private Network Adapters
Hamachi: Roll Your Own VPNs the Fast and Free Way, Create and Manage Your Own Virtual Private Networks

Linux CentOS Cluster Setup Tips

Sunday, February 14th, 2010

I made a linux cluster using 16 dual Opteron 248 machines, gigabit Ethernet and CentOS 5.4 DVD with kickstart.

Nodes can be remotely rebuilt upon command in about 3 minutes each in parallel, with no manual intervention, as long as you’re careful to treat nodes like appliances and don’t save data on them.

Some tips to save time and effort are:

  • collect the MAC addresses of all nodes at one time using the most efficient possible way, either from a manifest, or simply power all the nodes on and type on one node:
    ping -b 10.0.0.255 or
    fping -A -q -c 1 -g 10.0.0.0/24 or
    nmap -sP 10.0.0.0/24
    and
    arp -n
    
  • on your main client test node, which you may do 50 reinstalls on, save boot time by disabling memory checking, boot splash screens, etc. and use small filesystems during initial testing
  • install one machine by hand from DVD first to generate the anaconda-ks.cfg file, which contains your preferred package list (the CentOS installer itself uses kickstart even for local installs)
  • I found that having kickstart fetch the distro files using HTTP was a lot easier to setup and troubleshoot than NFS, and easier to secure later.
  • it’s common to use a BIOS boot order of “PXE, CD, HD” on each machine to bootstrap the cluster if the hard drive is not blank, then switch to “CD, HD, PXE” after linux is successfully installed and you’re able to login remotely. Subsequent reboots will try the HD first unless you force a PXE boot, which can be done with a script I wrote called unboot that both deactivates the boot partition and erases the MBR:
    #!/bin/bash
    
    parted /dev/hda set 1 boot off
    dd if=/dev/zero of=/dev/hda bs=512 count=1
    
  • do a web search for several good sample kickstart files. I found that merging 3 or 4 good ones provided very nice results.
  • by default, kickstart configures your networking with DHCP if you are doing network installs, but you can overwrite that in your post-install section with multiple static IP addresses if desired.
  • test your tftpd setup from the server (or another node) with tftp localhost -v -c get pxelinux.0
  • do tail -f /var/log/messages on the DHCP server to monitor DHCP requests by client nodes.
  • Make sure “/var/lib/dhcp/dhcpd.leases” exists.

Likely I will move to Rocks Clusters later, which is also derived from CentOS.

The Rocks Clusters people handle PXE boot in a more sophisticated way, configuring PXE boot to read the kernel image from the local hard drive, sparing tftpd from being swamped on clusters of thousands of nodes. Their unboot utility is called cluster-kickstart-pxe.

hp.com: Setting up a Linux PXE server and integrating clients – Howto (c00257674.pdf)

RedHat Linux KickStart HOWTO
Remote Network Boot via PXE
communities.vmware.com: How to Pass Parameters to a Kickstart Script?
aboveaverageurl.com: PXE Booting
Howtoforge: Unattended Fedora 8 Installation With NFS And Kickstart
Yu Dong, NASA: Installing Linux over Network: PXE, DHCP, TFTP, NFS and Kickstart
Rocks Cluster 5.3: Forcing a Re-install at Next PXE Boot
[Rocks-Discuss]cluster-fork ‘/boot/kickstart/cluster-kickstart–start’has no effect?
IEEE OUI and Company_id Assignments (MAC Address Database)
ftp://ftp.rocksclusters.org/pub/rocks
Reading Dell service tag number – dmidecode -s system-serial-number
Debian – setting hostname from DHCP result

Running TestDisk on linux with Dell Perc Controllers

Tuesday, February 9th, 2010

While testing the Dell OpenManage 6.2 update recently, the main ext3 filesystem superblock on a gpt partition was damaged by the CentOS 5.4 installer.

I did not ask the CentOS installer to touch the non-system partitions in any way, but it happened.

Fortunately, mke2fs writes superblock backups to each filesystem in case something bad happens.

e2fsck -b could be used to recover a superblock from a copy, but I found a friendlier tool …

I used an Open Source tool by Christophe GRENIER called TestDisk to scan for a backup superblock, and overwrote the bad superblock in about 30 seconds. Then I added the original mount label and mounted the filesystem:

# testdisk_static (or testdisk_static /log /dev/sdb)
# parted /dev/sdb name 1 /data (works on gpt partition types)
# mount -a
# ls -l /data
# tune2fs -l /dev/sdb1

TestDisk worked perfectly, even on a complex system with Perc 6i and Perc 5e RAID controllers with 4 TB partitions, but you must carefully read and navigate TestDisk’s menus, and actually write the new superblock to disk for each filesystem that was lost. TestDisk can also be used to recover files and preventively to save superblocks before an issue occurs.

There are versions of TestDisk for several operating systems, including Windows, Linux 2.4, Linux 2.6 and FreeBSD.

Note that parted also has a rescue mode for partitions:

(parted) help rescue
  rescue START END      # rescue a lost partition near START and END

Other tools to look at when fixing linux filesystems include tune2fs and partprobe.

For deeper insight into ext2 and ext3 recovery, search for the excellent articles by Ted Ts’o.

James’ NoSQL Feature Matrix Link

Tuesday, December 22nd, 2009

A wikipedia editor (SamJohnston) has reverted the useful NoSQL Feature Comparison Matrix that I added at ApacheCon Oakland.

Here is the last NoSQL article version that has the detailed feature matrix.

Thanks to several contributors for improving it before it was suppressed.

I would be in favor of moving it to a wiki outside of wikipedia for long-term maintenance.

One nice improvement would be sort-by-column.

Jimmy Wales’ goal for wikipedia “to be the sum of all human knowledge” is not possible with a rigid NPOV and without primary sources. Instead, wikipedia will remain “the entertaining cartoon of human knowledge.”

I’ll have to think twice before contributing my time, content or money to wikipedia in the future.

rackspacecloud.com: NoSQL Ecosystem

Nagios check_http.c Patch to Display Result Page Snippet

Tuesday, December 1st, 2009

Nagios LogoHere’s a minor change to the nagios plugin check_http.c that shows the first 128 bytes of the page body if the -s match option is used:

  /* check elapsed time */
  if (strlen(string_expect) && strlen(page)) {
#define MAX_BUFFER_PAGE_SAMPLE 128
     char s[MAX_BUFFER_PAGE_SAMPLE];
     strncpy(s, page, MAX_BUFFER_PAGE_SAMPLE);
     s[MAX_BUFFER_PAGE_SAMPLE-1] = 0;
     /* Need to strip JavaScript here to prevent XSS */
     strip_xss(s);
     strip(s);

     asprintf (&msg,
            _("%s - %d bytes in %.3f second response time %s%s|%s %s"),
            msg, page_len, elapsed_time,
            (display_html ? "" : ""), s,
            perfd_time (elapsed_time), perfd_size (page_len));
  }
  else {
     asprintf (&msg,
            _("%s - %d bytes in %.3f second response time %s|%s %s"),
            msg, page_len, elapsed_time,
            (display_html ? "" : ""),
            perfd_time (elapsed_time), perfd_size (page_len));
  }

void
strip_xss (char *t)
{
   char *s;

   for (s=t;*t;*t++) {
       if (*t == ' ' ||
           *t == '.' ||
           *t == '-' ||
           *t == '\'' ||
           *t == ':' ||
           *t == ',' ||
           isalnum(*t)) {
          *s++ = *t;
       }
   }
   *s = 0;
}

Then you can use the usual Nagios check_http command with the -s option:

# 'check_http_str' command definition with options:
# -u URI without leading scheme and hostname
# -s "string" to match
# -M seconds is max acceptable age of page
# -L $HOSTADDRESS$ makes it a hyperlink to source page

define command{
        command_name    check_http_str
        command_line    $USER1$/check_http -I $HOSTADDRESS$ -u "$ARG1$" -s "$ARG2$" -M $ARG3$
        }

The result looks like:

HTTP OK: HTTP/1.1 200 OK - 377 bytes in 0.163 second response time
OK - System: 'PowerEdge 2950', SN: 'N1234', hardware working fine,
11 logical drives, 51 physical drives

The improved status display allows me to more easily use HTTP for remote nagios monitoring, instead of NRPE. Almost worth dropping into C for. :)