Archive for July, 2007

OS Virtualization on FreeBSD vs. Linux

Monday, July 30th, 2007

I was talking to somebody at OSCON last week and they made an off-hand comment about using FreeBSD jail to isolate software applications in a development environment.

Being a linux guy, it took a while to sink in that jail was a form of operating system virtualization.

On linux, there’s dozens of virtualization options these days in various states of development, reliability and licensing. There’s only 3 linux “OS virtualization” options, FreeVPS, OpenVZ and Linux V-Server.

Generally linux virtualization requires a custom kernel with the relevant patches, so installation requires installing the kernel, rebooting and testing the new kernel – usually a hassle.

FreeBSD has jail, and Solaris has zones.

What’s great about jail is that:

  • limits access to filesystem, process space and network
  • it’s been built-in since FreeBSD 4.0 – no-hassles, mature feature
  • a new jail can be created with one command
  • it’s fast – native, not emulated
  • it consumes almost no resources – only 6-10MB hard disk space per jail
  • it’s simple to use and requires no OS reinstallation or configuration – great for remote boxes
  • it’s free.

The downsides of jail are that you can’t run different versions of the OS than what was originally installed, there’s no way to limit RAM or CPU used in each jail, and demons should be configured to listen on the appropriate IP addresses instead of 0.0.0.0.

Originally adding jail support only required changing about 350 lines of kernel code and adding 400 more – not that intrusive.

Linux-VServer
wikipedia: FreeBSD jail
Virtualization with FreeBSD Jails
wikipedia: Comparison of virtual machines
ibm.com: Virtual Linux – An overview of virtualization methods, architectures, and implementations
Virtualization in Xen 3.0
Installing Linux-VServer
New virtualization option for Linux: KVM (and Linux virtualization summary)
KernelTrap: Abusing chroot
Related Projects to jail and vserver

Tweaking Wordpress with Permalinks and WP-Cache

Sunday, July 29th, 2007

I’ve been blogging for a couple years now using WordPress software hosted on one of my linux servers.

I’ve done a little customization of the sidebar by editing PHP, upgraded a couple times finally getting to WordPress 2.2, and added a weather plug-in and the PHP APC op-code cache.

So I guess it’s finally time to really make WordPress behave the way I want.

Today I enabled descriptive permalinks. Instead of the “ugly” links like /blog/index.php?p=1, descriptive permalinks like /blog/category/my-latest-post.html are also available.

There’s 3 steps to enable descriptive permalinks:

  1. Allow Apache/WordPress to write to your .htaccess file while setting up permalinks
  2. Allow apache to use the .htaccess file with an AllowOverride directive in your httpd.conf file
  3. in WordPress go to Site Admin … Dashboard … Options … Permalinks … Custom and use a template like “/%category%/%postname%.html”

Then test it, and after it works you can make the .htaccess read-only to apache. If you encounter a problem, check your error_log to see exactly what problems are being logged.

The potential benefit is that in some cases SEO is improved with descriptive permalinks. More SEO is better I guess, although I’ve been doing fine in SERPs (free “organic” search engine results pages) with just writing detailed content and “ugly” links.

I also came across a link to WP-Cache 2.0.

I already use APC, but WP-Cache creates static pages and redirects to them. Very cool, I enabled that and it seems to be working. Some timing statistics are added in the HTML as a comment, and the cache directory is working as expected.

WordPress has fairly good performance with APC, but it’s worth improving in shared hosting environments, a busy database or in case of slashdotting.

There are options to not cache pages being crawled by search engines.

Tom Raftery: Boost Search Engine Optimisation (SEO) using Permalinks

Nikon 18-200mm lens

Sunday, July 29th, 2007

The usual online stores are usually sold out on the Nikon 18-200mm VR lens. I heard that local retailers often get a few each month, so I ordered one from San Jose Camera in Campbell last weekend.

To my surprise, I got a call Tuesday morning saying that it was in. That’s fast … about one business day after I ordered it.

San Jose Camera doesn’t charge a mark-up on popular back-ordered items like some stores.

So today I picked it up. It’s smaller and lighter than I thought. It comes with the HB-35 plastic hood and a bag, but takes 72mm filters. It’s nice that Nikon still includes accessories – Canon only does that with their L-series lenses.

We’ll see how I like it on my D200. But then I’ve never had a Nikon lens where I didn’t like the image quality, although some of the AF-S zooms have a weird cam-like feel when you zoom them.

I also played with the Canon Rebel XTi and 17-55mm f2.8 IS lens. (I’ve been thinking of getting a toy DSLR to play with since my 30D with grip is big for grab shots.)

The Rebel XTi body is the same size as the older XT – very small and toy-like. The lens is a big, heavy, fast, pro-like EF-S lens that would block the flash coverage. Randal has this lens and loves it on his 20D – it’s great for flashless indoor candids.

OSCON July, 2007 – Portland

Friday, July 27th, 2007

I attended the O’Reilly Open Source convention again, making it 10 years in a row. Once again it was held at the Convention Center near downtown Portland, a convenient light rail ride from the airport.

Like many experienced developers, I spent a lot of time in the “hallway track” talking to other developers and users, as well as in one of the 15 simultaneous talks.

The general consensus was that the talks were not as strong as in previous years (not even compared to the MySQL conference this year), but it’s worthwhile to me if I can get even one juicy nugget from each talk, or gain an understanding of a developing trend in programming or system administration.

Many of the presenters griped about there not being enough time to look at source code in a 40 minute talk.

For those who want a conference summary in a nutshell:

  • OpenID is popular
  • lucene and its REST interface have more mindshare than projects like Kinosearch, language-specific bindings, etc.
  • Yahoo! released the yslow browser plug-in for front-end performance evaluation
  • Perl: no ORM appears to be gaining the upper hand, though DBIx is respected. Tim Bunce would like to see a wrapper around JDBC for each scripting language. Alison Randal is updating the Perl license.
  • PHP: no good way to do vector reporting graphics, especially since IE doesn’t support SVG and Adobe is killing the Macromedia plug-in in December. PHP4 is being EOL’ed 8/8/8 so that the PHP developers can focus on 5 and 6 only.

Google was heavily recruiting at the conference. I ran into 3 recruiters, and there were even more in the Google booth.

Pretty good food for lunch, usually chicken or fish in some kind of red sauce with steamed veggies. Better than the wilted sandwich boxes from previous years that mainly got tossed out.

Tuesday nite

I arrived at the Convention Center in time for the evening Google Open Source awards. Happened to sit next to Zak and the 20 year-old OpenID guy, David Recordon, who won $5,000 and a colored, transparent, angular plastic trophy and base that we had fun stabbing each other with.

The OpenID Foundation is offering a $5,000 bounty to the first 10 OSI-approved projects that add OpenID support. Many programmers were busy adding it, including SocialText and others. (David works at Verisign.)

I walked over to the Doug Fir Lounge with a few guys, 2 of them Austrian. I had the halibut fish and chips and lemonade for $20 including tip. It was ok. They have a log cabin motif happening with a restaurant, patio and bar upstairs, and dance club downstairs, so ID is required to enter. Open from 7 am to 2:30 am every day, 1 503 231 WOOD.

Wednesday

Nagios

- general overview of features
- Event Broker most powerful, least used

Bigger and Faster
Rasmus Lerdorf

Rasmus did his usual “PHP is as secure as any other language”, and “pick on a PHP app and make it go faster” talk.

He said he’s still not a Y! Paranoid, but his work does often touch on PHP and web security.

He used to use httpload, but now prefers siege for load testing because it has support for cookies.

http://developer.yahoo.com/yslow/
Live HTTP Headers
APC

sla.ckers.org/forum/list.php?3
php.net/filter
xdebug.org/docs/profiler
talks.php.net/show/oscon07

xdebug
jeremiah san diego xss console author
scanmus.corp.yahoo.com

PHP and Ruby Envy
- NZ programmer on Silverlight CMS (BSD licensed)
- own object system in PHP5
- Ruby less available on web servers, less mindshare
- rolled his own PHP OO frameword apparently

Exhibits

- talked to Mark Finkle of mozilla.org
- said hi to Larry. He had the whole family there.

Trac
Vivek Khera

- he uses RT for public tickets, Trac internally
- doesn’t require much resources since only a few developers
- Trac is used on many Ruby/PHP projects
- gives you wiki/tickets/etc.
- modified BSD license

Afterwards mentioned:

- uses Trac in a BSD jail
- an alternative to Trac would be basecamp (or I guess Sourceforge software). See slashdot.org threads for more ideas.
- likes pfSense firewall as an appliance
- nagios alerts too much, and no good rule builder for multiple hosts
- own web framework called Rowdy (RWDE)
- software as complicated to install as RT should be treated as an appliance
- he submitted 6 related talks on software development environment, only 1 accepted.

Steve Souders
Chief Performance Yahoo!
souders@yahoo-inc.com
Exceptional Performance Group

http://developer.yahoo.com/performance

- IBM Page Detailer Pro
- yslow (crawls the DOM, not a packet sniffer)
- firebug
- jslint – The JavaScript Verifier

80-90% of end user response time is spent on the front-end. so optimize there.

14 Rules for a Better User Experience

1. make fewer HTTP requests
2. use a CDN
3. add an Expires header
4. gzip components – even JS and CSS
5. CSS at top
6. JavaScripts to bottom
7. avoid CSS expressions
8. make JS and CSS external
9. reduce DNS lookups
10. minify Javascript
11. avoid redirects
12. remove duplicate scripts
13. configure Etags – disable in most cases if load-balanced or multiple web servers
14. make AJAX cacheable

move JS to onload
remove bottom tabs
avoid redirects
images sprites
expires

Thursday

PHP Graphics
Luke Welling, OmniTI

Luke presented an overview of raster and vector graphics modules for PHP.

He prefers vector graphics, but there’s not many free options for doing that.

He feels that Yahoo! Finance and Google analytics sites are state of the art in presentation graphics with anti-aliasing, interactivity, esthetics, text and maps. I’d say that’s aiming a little low, but it’s a start I guess.

Perl Lightning Talks

  • Vani Raja did a talk on Yahoo! JS
  • Ask did his talk on qsmtpd again
  • talk on Test::More 3?
  • talk on task lists for hit and run volunteers
  • Schwern did one talk on making tea for 5 minutes, and one on “Blame Schwern” – just do it instead of waiting for permission
  • Tim Potter did a talk on a messaging standards effort for his employer, saying that the ANSI process was too slow and looking for an alternative
  • Andy Lester did a talk on ack
  • a talk on SVN::Notify
  • http://angerwhale.org/
  • Tim Bunce talked about DBD::Gofer Proxy and next-gen cross-scripting language DB API based on JDBC API
  • guitar song about #perl

YouTube: Perl Lightning Talks on Handycam by Schtonk

Perl Auction

Larry’s talk on comparative languages and Perl6. Sounds like we’ll be able to do something like foreach (1..infinity).

Full Text Search BOF
Peter Zaitsev
– based in London, England but often in SV
– uses Sphinx on several servers
– http://boardreader.com/ one TB of searchable data
– geneology is big on full-text search

- after insert, mysql full text gets slow, run optimize.
also, doing it at insert time causes index update per keyword
- gin or gyst for Postgresql 8.4?
- Michael Kimsal, SOLR
- hard disk space is free (enough for whatever indexing is required)
- mostly news search involves last 5 minutes of feed
- MessageOne stores email for lawyers to mine. They like to search, archive and expire. Mostly Exchange lusers, rarely Unix admins.
- Lucene and REST interface
- Monty says MySQL AB hired a programmer to work on search, but he’s working on another project now. They need somebody with a burning desire to make progress in an area like that, but they recognize the importance of search.
- Monty poured out free Finnish chocolate rum from a Pepsi bottle that was so powerful it scared most people. He said it was banned for 2 years in Finland because it was so addictive.

Sun BOF

- audience talked to senior Sun staff about Java and Solaris a little.
- free beer, cheese and crackers.

Friday

A bunch of Postgres people went to the Portland wine tasting on the river event in the afternoon.

Call for Software Whiteboard

OSCON07 Call for Software Whiteboard
flickr.com: Jeff Kubina’s OSCON 2007 Whiteboard set of tiles

CentOS and yum

Sunday, July 15th, 2007

Most of the linux systems I administer these days happen to run CentOS, which is a free redistribution of Redhat.

The main package manager is yum, which seems to work nicely.

I use the default yum installation, plus I’ve added rpmforge support for less common packages.

That gives me access to about 20,000 linux packages altogether.

By default, yum works fine as long as you have working nameservers to /etc/resolv.conf.

I use rpm and yum repositories mainly for installing basic tools like sysstat (iostat), mytop, etc.

yum is very easy-to-use, and either finds the package in a few seconds and installs it, or displays an error message:

yum install pkg-name

My configurations of apache/SSL/mod_proxy/PHP/APC and mod_perl are too customized to expect to find in an existing package, so I build them from source and block rpm or yum from updating the related files with an exclude directive.

I use shell scripts for building those. The apache build script is about 100 lines long now.

Building Open Source applications is easy on CentOS, provided that you have installed the development packages, including gcc, g++ and ncurses-devel packages.

Note that for complex, threaded programs like databases, use a pre-tested binary distribution if possible. MySQL is an example of a threaded program that is sensitive to which libraries you link it to.

SVLUG: Drobo Data Robot

Wednesday, July 11th, 2007

Tonite at the Silicon Valley Linux Users Group, Richard Sharpe, from Data Robotics, Inc. gave a talk on Drobo, the World’s First Data Robot.

It’s a black, easy-to-use USB storage device with 4 hot-swappable SATA bays about the size of a small toaster. The interesting part is that it intelligently does RAID as you insert and remove hard drives, which can be of varying sizes.

It was carefully designed to have a pleasing consumer appearance with informative LEDs for drive capacity and health. Now anybody can manage and upgrade a high-capacity RAID device.

It has a microcontroller running VxWorks, but no file system. It just presents disk blocks as LUNs using SCSI over USB. Currently it has Windows, Mac and Linux support. What’s interesting is that you can remove a drive from a 2-drive setup as a snapshot backup for later.

They are adding more and more functionality to Drobo to do things that are better done on the storage device. For example, Windows can only create 32 GB FAT32 partitions, but can mount 2 TB partitions. So it’s better to do the configuration on the device. Bad block checking or defragmentation may also be better done on the storage device.

It does not run Linux or function as a stand-alone NAS unit.

4x 1-terabyte drives would result in a RAID of about 3 TB. That sounds useful as a backup device for photography or digital artists. It’s not going to be terribly fast with USB or while rebalancing RAID, so is not really made for video editing.

Their belief is that a consumer, 4 bay device is the sweet spot for their market.

It’s possible the logs are “encrypted” by XOR’ing every byte with 0xC3.

It was eye-opening to hear about all the software hackery needed to support various versions of consumer operating systems like Windows.

Drobo is $499 without any hard drives.

Sourceforge drobo-utils project (GPL3)

2009 Update:
drobo.com: Linux Support FAQ
drobo.com: supported hard drives FAQ
Daniel Krook: One week with the Drobo on Linux
Andy Grover’s Blog: Drobo and Linux
Bart’s Drobo blog post
theregister.co.uk: Drobo restrings boxes to double-up product range (Drobo S with 5 slots, and Drobo Elite with 8 slots, iSCSI, SAN-like features, and faster Marvell processor)
StorageMojo: DroboPro at SNW
StorageMojo: A data robot is eating the low end

FrontBase RDBMS

Friday, July 6th, 2007

I came across the Frontbase RDBMS recently. Their 288-page users manual was so nice and clear I decided to read through it.

The FrontBase project was started in 1996 for Copenhagen’s 911 service (MySQL has similar practical origins, written to support retail data warehousing contracts). They claim it’s a “database designed for the Internet” and requires zero maintenance. Currently they’re on version 4.2.

It’s a proprietary, closed-source database written in ANSI C, but free for most users. They currently charge for support and consulting, and sell licenses for embedded users. The FBAccess client library source is available on request for programmers to add support to their oses and devices.

It seems to have traction in the Mac community, likely because of the nice Mac client program and FileMaker and RealBasic support.

What’s nice is that it supports transactions and full-text search at the same time, unlike MySQL.

FrontBase supports:

- servers: Mac OS X, Linux, Windows, Solaris, FreeBSD
- clients: FrontBaseManager Aqua client for Mac OS X allows mgmt. and monitoring, also web, common scripting languages (Perl, PHP, Tcl), JDBC, ODBC, and the sql92 console client
- SQL92
- Unicode
- statement-based replication from transaction logs
- full SQL logging options, allowing replay (SET WRITE SQL TRUE GLOBAL;)
- full-text indexing
- stored procedures
- clusters
- table spaces
- raw device driver
- live (hot) backup due to versioning (WRITE DATA;)
- data and communications encryption
- whitelisting and blacklisting of IP addresses
- row-level locking
- row-level privileges
- read-only databases (note that replication slaves are read-only)
- import utilities for Access and FileMaker
- migration tools for FileMaker (tables only) and MySQL (no enum or set column support)

They have been changing their licensing, so contact gclem at frontbase.com for the latest info.

FrontBase sure looks like a clean, modern RDBMS implementation.

I plan to learn more about its reliability and performance.

I have noticed that it is prone to crashing on Mac OS X PPC. Even a simple query like SELECT DISTINCT(OID) FROM TABLE; on 500,000 records with mostly nulls can crash the database, but hard to tell if it’s Frontbase, the OS, or what.

Kihncert 2007, San Jose

Sunday, July 1st, 2007

I went to see the Kihncert on the Green 2007 concert in downtown San Jose. Great music, friendly audience.

The lineup was:

MAIN STAGE SCHEDULE
1:00pm - Evolution
2:30pm - Aja Vu
3:45pm - Wild Child
5:15pm - Foghat
7:00pm - Greg Kihn Band

STAGE 2 SCHEDULE (on between main stage bands)
2:00pm - He's Dead Jim
3:15pm - Crimes of Passion
4:45pm - Lane Four
6:30pm - Referral

I got down there in time to see Wild Child, Foghat and the Greg Kihn Band perform, and photographed them with my Nikon D200 and 70-200mm/f2.8 VR lens.

I heard that Evolution did a great set, and met the grandmother of the guitarist. They’re from Pittsburgh.

Wild Child is a dead-on clone of The Doors. The singer even looks like Jim Morrison.

Foghat sounded unbelievably great, especially when they had 3 guitarists on stage at once.

Greg Kihn Band also sounded good. Ry Kihn and Chris Jackson did some blistering solos.

Not bad for $9.85 advance ticket price.

I’ll process and upload my photos later this week.