The name “Gluster” was derived from the words “GNU” and “Cluster.” No relation to the Luster filesystem, and actually they have opposite overall architectures.

GlusterFS is a GPL3 distributed network filesystem that runs as a service in user mode on Linux on a network of servers (conceptually like Google GFS.) Redhat bought Gluster, Inc. in 2011 and calls it “Redhat Storage.” By running in user mode and reusing existing linux features and modules, GlusterFS gained reliability in months instead of the usual 10 years for other filesystems.

gluster.org is the community website for GlusterFS.

Previously, AB was CTO at California Digital Corporation (CDC). One of his projects was the 1024-node linux “Thunder” HPC supercomputer for LLNL, the most powerful in the USA at the time.

- after that, he still wanted to do Open Source projects, preferably without bureaucratic encumbrances
- got some angel funding, but also a seismic data company paid $500,000 to adapt HPC code in 3 months to replace IBM Regatta system, then a follow-on storage contract for 1.2 PB in a 6 months project
- audience member: “In 10 years 1 PB will fit on a SD card.”
- GlusterFS is in some ways architecturally the opposite of VMware. GlusterFS is userland code.
- in 2006 large companies like Lehman, etc. started appearing on mailing list, to some surprise and awe
- originally the company was called Z Research, renamed to Gluster, Inc. to have clearer brand name
- no in-house test storage hardware, so developed on customer hardware!
- was still doing other paying work such as embedded kernel stuff, web dev, etc to reduce burn rate initially. Too distracting.

- Google mixes app and GFS, app generates 64 MB chunks, GFS manages metadata. Too complicated for general use.
- GlusterFS is a distributed storage OS in user-space
- create container without knowledge of filesystem (Posix, ACL, etc. ) because no known common user pattern for storage
- lots of general C programmers available to recruit, but no filesystem developers without baggage about kernel
- in 2008-2009 added too many features to actually test
- VCs contacted them and invested A and B series total of $15 million, despite their storage “experts” saying it was crazy, users said it was awesome
- lowest layer is native filesystem like EXT3 or XFS, thus idiot-proof
- use extended attributes for metadata
- block, replication, striping, elastic hashing algorithm
- striping support by cleverly using sparse files with different offset on each server
- read server choice based on fastest response
- every directory has its own hash space
- good default behavior when adding servers (no thundering herd)
- striping is good for hotspot files or files too big for 1 volume, like saving HPC results
- will be unified file and object protocol for object storage
- there is a pathinfo command can query extended attribute, could be used with ssh for a fake MapReduce
- GET and PUT at command line
- GlusterFS is most heavily used for lots of files containing unstructured data
- 3.3 will have faster healing operations, better granularity for 100 VMs, KVM support, etc.
- currently shared-nothing, but with a little sharing healing can be faster
- will be HDFS clone mode

Data Storage models:

• objects
• file
• block
• structured data
• NoSQL
• semi-structured data.

- Redhat bought Gluster, Inc. for about $136 million in October, 2011. It was about 60 employees at the time. Now there’s about 40 engineers working on GlusterFS at Redhat. AB chose Redhat over other suitors because of its commitment to Open Source and linux.

A dozen people attended dinner afterwards:

- it was tough hiring people for Gluster Inc. since the concept of doing file systems in userland confused a lot of developers and managers.
- AB’s philosophy is that the Open Core model doesn’t serve end-users well, as all users need “extras” like user-friendly mgmt. programs, not just licensees
- companies seem to be happy to pay for GlusterFS support, one of the reasons being lack of in-house storage engineers
- lots of discussion about Illumos (OpenSolaris fork), ZFS and containers
- take a look at Nexenta
- An efficient WAN replication method with GlusterFS is to use the marker framework / queue using extended attributes to feed rsync a list of changed files, scales better than inotify
- no storage tiering yet for incoming/outgoing hotspot files
- Redhat is pushing xfs heavily internally, and has hired the available ex-SGI xfs developers
- some checksumming is done in GlusterFS, but no end-end checksumming. Need to look at performance and demand.
- historicaly, linux has had slower adoption and community interest in India than Western countries due to lagging Internet performance (often more practical to install linux from a magazine CD-ROM than attempting large downloads) and relatively higher cost of computers compared to local salaries
- AB got started in programming on a Spectrum microcomputer, and progressed over time to fixing minor bugs in the linux network drivers, culminating in GlusterFS.

Thanks again to Symantec for hosting this event.

GlusterFS Downloads
Chip’s Rant: Why GlusterFS is Glusterfsck’d Too
wikipedia: Extended file attributes

There was a good turnout, with about 30 outside attendees and about the same of ISC staff. 3 BIND developers from the Czech and Chinese registrars also attended, as well as Cricket Liu.

Here’s my notes.

DNS

- can learn what you need to know (50%) in a day, lifetime for the next 50%

BIND9

- monolithic program
- 10 years old
- different syntax than BIND8, 30% slower

User Story – Quick Refresh at AOL

- VMs spark up in 9.5 seconds, but DNS takes 15 minutes or more to propagate in their system
- goal is 5 minute updates across globe

BIND10

- sponsored by 10 TLDs, unlike BIND9 which was sponsored by Big Iron vendors
- TLDs represent registrars, users or citizens
- started April 1, 2009
- 5 year plan
- now on year3 – features and performance
- release every 6 weeks
- will be no slower than BIND9
- postfix processes model, not threads
- scalable across cores
- modular
- logging
- reporting
- REST API
- save cache object to memory or network, could be 10s to 100s of MB
- no query or response logging, use tcpdump
- look at DLZ

DNS Provider Issues

- Dealing with large zones
- Dealing with lots of zones
- resource estimates (for sizing new servers)
- high-availability installations and technologies
– anycasting
– cooperate better with firewalls and load balancers
– auto-clustering?
– quagga support?
- multi-master
- support for fully mastered slave
– akin to powerdns “super masters”

- some debate over how much routing is appropriate inside dns
- do it all or just be fast
- does it make sense to send acl info over port 53? needed for cluster setup

DNSSEC Issues

- Deployment and Maintenance procedures
- in-line signing
– aka “bump on the wire” 9.9.0
– hidden master sends signed zones to slaves
– aka “signer in the middle”
- registry vs. registrar
- look at DomainKeys optionality
- DLV keys (don’t need a signed parent)
- zone monitoring is crucial
- 4 HSM known, $80k for qty 1
- some firefox plugins check ad bit
- dnssec slowest on negative answer, nsec3 worst

The event photos are available here.

Thanks to Facebook for hosting this event.

Small URL Generator (in F5 iRule): Part 1 Part 2

What could possibly go wrong?

wikipedia: F5 Networks says, “iRules: a TCL-based scripting language allowing arbitrary manipulation of traffic flowing through the BIG-IP, including real-time modification of said data.”

Ten Steps to iRules Optimization

Thus the need for filesystem conformance testing.

Some popular tools are:

• File System Exerciser (fsx)
• mt_write
• File System Reliability Test Suite
• Linux Test Project
• bonnie and bonnie++ stress filesystems and can also reveal issues. bonnie++’s fsync option is one knob to try.

linux-kernel: Inode Lock Scalability V4
techforce.com.br: LVM, RAID, XFS and EXT3 file systems tuning for small files massive heavy load concurrent parallel I/O on Debian
MythTV: Advanced Partition Formatting (Large files)
redhat.com: The Ext4 File System

Pthreads Primer
LLNL: POSIX Threads Programming Tutorial
POSIX.1 Symbols: _XOPEN_SOURCE and POSIX_C_SOURCE Feature Test Macros

Interesting comments about using kickstart in EC2:

    Maybe I'm missing something: why would you ever want an EC2 instance to
    kickstart at boot time?  You should create an image for every role you
    care about and then boot the appropriate one for every instance you
    need.

Roles change, updates happen frequently, and I'd rather a machine spin up with the
latest packages.  I've always found that updating a pre-built machine is slower,
sometimes substantially so, than just building a fresh image with the newest RPMs.

Interview: Anaconda and Fedora 9 Developers
Quick History Lesson of Puppet and Chef
Cradle-to-Grave System Automation with PXE, Kickstart, and cfengine
crash, bang, wallop, what a server (creating a kickstart+puppet solution from scratch)
Systemimager & puppet
wikipedia: Software configuration management
Tips and tricks for anaconda and kickstart
Capistrano

• sdiff is a text-mode side-by-side diff and merge utility. It has -B and -W options for special handling of blank lines and whitespace.
• vimdiff lets you visually compare and edit up to 3 files in color using the vim engine.

Note: diff3 works like diff, but lets you compare and merge 3 files.

ZFS is the holy grail of filesystems. Many Database Administrators have switched from Linux to Solaris because ZFS has much better snapshot support than LLVM, as well as good SSD support.

phoronix.com: Native ZFS Is Coming To Linux Next Month (Aug. 27, 2010)
phoronix.com: Btrfs, EXT4 & ZFS On A Solid-State Drive (Aug. 9, 2010)
phoronix.com: Benchmarking ZFS On FreeBSD vs. EXT4 & Btrfs On Linux (July 27, 2010)
phoronix.com: Running ZFS With CAM-based ATA On FreeBSD 8.1 (July 26, 2010)
github: Native ZFS for Linux
FreeBSD Wiki: ZFS

It was a good conference, and we had beautiful weather all week long.

Executive Summary

The themes promoted by the conference organizers were Cloud Computing, NoSQL, Emerging Languages (Scala, Erlang, Parrot, Go) and Android phone development.

The @oscon twitter channel was heavily used to coordinate amongst organizers and attendees. I used the TwiXtreme twitter client program on my BlackBerry.

Plug Computers were very popular in the Expo area. They are 5 watt ARM-based computers running Debian Linux that fit into a power brick-sized case and cost $99 to $129 depending on features. The Marvell booth had a few models on display, from GlobalScale (GuruPlug) and Ionics. High-end models have dual gigabit NICs, multiple USB ports, a WiFi access point and other expansion ports.

There was also continuing buzz regarding Facebook’s Flashcache SSD module (GPL v2) for linux, and also ZFS snapshots.

Tutorials

I went to the Gearman Cookbook tutorial, the first half of the Chef tutorial and some of the Cloud Summit talks.

The Gearman Cookbook tutorial was excellent. After a detailed overview of the Gearman architecture and implementations in Perl and C, a number of use cases were explored in detail, including before and after code samples. The talk was both easy to listen to as an overall survey, as well as providing immediately useful info for those wanting to deploy it.

The Chef tutorial was very detailed – too much so perhaps. I went to the first half only, since I am not planning to implement Chef soon (I use PXE and anaconda/kickstart with CentOS), and did not need that level of detail at this time. cfengine, puppet and chef are ops tools for configuring servers. Chef uses Ruby data structures for its configuration files, and has include files and other useful syntax. Basically, users can “code” server configuration, as if they were traditional apps.

I went to some of the Cloud Summit talks and BOFs, but found that anybody who has done a simple project using EC2 knew as much or more than the speakers, some I would call blowhards.

Marten Mickos, president of Eucalyptus, is refreshing in that he is always clear about being in it for the money, while also promoting Open Source.

Sessions

Some of the most memorable sessions to me were:

Introduction to MongoDB, Kristina Chodorow (MongoDB)

Kristina is the maintainer of the Perl and PHP drivers for MongoDB. She gave an overview of MongoDB, a NoSQL document store, and its command-line interface, which uses JavaScript.

Some day she will release a sharding tool for MongoDB.

Scaling SourceForge with MongoDB, Nosh Petigara (10gen), Rick Copeland (SourceForge.net / GeekNet)

Nosh and Rick gave an excellent review of incorporating MongoDB into the SourceForge site.

- SF query load is mostly read-only
- ops team benchmarked a few NoSQL candidates, and MongoDB won on performance
- original MySQL servers had 64 GB RAM. After migration to MongoDB, same server machines but only 8 GB RAM
- backup dumps are verified to be bitwise the same as masters
- have to be careful not to dump all documents in your database to the network or it will max out switches
- SF relies on first-class data centers and replication slaves, less worried about MongoDB mmap (not crash-safe)
- I personally looked at their performance numbers and site graphs (on an iPad), and the end result was impressive.

Perl Lightning Talks

As always, the Perl Lightning Talks are a highpoint of the conference.

The “cartoon” of Vincent Pit’s remarkable CPAN module(VPIT) contributions was both informative and hilarious. Vincent is a French Ph.D. candidate in advanced geometry.

Cloud BOF (3 Hours)

The Cloud BOF was disorganized, starting 30 minutes late and for some reason was subdivided into 4 audience groups. Startups and vendors trying to make a cloud sales push led the BOF, including cloud and DNS service providers.

The Health Regulations subgroup came up with a couple ways to make the Cloud palatable to regulators by using encryption on all data due to the multi-tenancy issues with sharing public VMs.

I was in the NoSQL group, which discussed general issues and particular successes. Memcached was the clearest winner, while some people also had success with MongoDB and Redis.

My neighbor was an engineer at Postrank.com. He said that they were happy with HAProxy, but much less happy with the unpredictable IO available when running MySQL on EC2. He also said to carefully look at storage volumes available to your instance, as one is a useful tmpfs. They use AuthSMTP to get around EC2 being generally blacklisted for outbound email.

Database BOFs

MySQL BOF

The MySQL AB engineering staff has left Oracle. Monty Program AB (21 staff) has the core developers, and Percona Inc. (32 staff) has the consultants. Oracle still has some of the InnoDB programmers.

The business plan for Monty Program AB is 60% commercially-sponsored MySQL development, and 40% community-request development. Monty would like commercial users of MySQL to sponsor patches that would benefit them.

Mark mentioned that using Nehalem instructions for CRC were much faster, and that Facebook was using partitions for truncating tables instead of doing multi-record deletes. (See his blog for more details.)

One person mentioned using a commercial backup tool, R1Soft, that inserts a linux kernel module to allow filesystem snapshots. He said to carefully test backup and restore in your environment, especially for filesystems greater than 1 TB which may exceed certain block counter limits. Peter said that some of his clients had used it with varying success.

It worked for him in his environment, and the file browser allows selective file restore (he uses it to restore by priority where a system runs multiple applications.) It starts at $299 for the Standard Edition, and also has MySQL Add-on and Enterprise Editions.

PostgreSQL BOF

The PostgreSQL BOF talked about 30 or so changes that went into version 9.

One of the most exciting new features is a native replication feature, called streaming replication (block-based.) The advantage over Slony-I replication is that Slony-I is trigger-based, so has a variety of issues included inability to replicate DDL commands.

Some of the developers mimed replication events, which was rather amusing to watch. Yes, it was taped.

PostgreSQL is released under the PostgreSQL Licence, which is BSDish.

Peter Zaitsev, co-founder of Percona, organized 3 BOFs, including XtraDB, XtraBackup, Maatkit, Percona Server, Sphinx Search and Running Databases on Flash Storage.

Sphinx Search BOF

Andrew Aksyonoff, the original programmer of Sphinx Search (GPL v2), couldn’t make it to OSCON (the good excuse was that he was busy coding), so Richard Kelm (Sphinx sales/customer support honcho) and Peter filled in (Percona is a business partner with Sphinx, and many of Percona’s clients use it.)

Some of the attendees were existing users, like myself, and some from HP and other companies were looking for a large-scale search solution or alternative to Lucene.

Monty mentioned that the latest MySQL 5.1 should be used, as there have been a number of performance and reliability improvements. Full-text search is supposed to be 10x faster than 5.0, and replication is nearly bug-free by now.

Sphinx Search now has real-time index updates in version 1.1.0 beta. Another very nice feature is SQL+FS indexing.

Here is the full Sphinx 1.1.0 changelog.

Running Databases on Flash Storage BOF

The Running Databases on Flash Storage BOF had a combination of MySQL and Postgres users who have tested or used most of the SSD products: FusionIO, violin, Intel, OCZ, etc. Everybody was happy with SSD IOPS performance, but less so with cost and metadata RAM requirements with the add-in boards (FusionIO may require 4 GB RAM for metadata.)

Peter said that 20% to 30% of his clients are already using SSD – across the spectrum of vendors and models. Some are also trying “massive RAM” solutions, like Cisco servers with 384 GB RAM.

Some users had 1+ TB Postgres databases with very thorny backup and mgmt. issues. One solution was to start a snapshot, but not do the copy operation.

Expo Notes

I had an enjoyable talk with Austin Hook, who has operated the OpenBSD Store for many years. He lives near Calgary, the center of OpenBSD/OpenSSH/PF development. He mentioned that some perennial financial contributors had stopped because of the recession, so here’s the donations link.

I also talked to some reps from a Brazilian outsourcing firm, ActMinds. They currently have 400 employees across Brazil and a sales office in Philadelphia. Brazil is only 2 hours ahead of EST. They said the minimum project size is 2 developers and developer turnover a low 5%/annum. Their pricing is $35 to $45/hour.

And I had fun handling the plug computers on display at the Marvell booth. The Ionics boards are amazingly densely populated.

Discussions

I had the opportunity to talk to a long-time Portland resident who works as a computer consultant. He said that the Portland economy is not doing great, and really hasn’t done well since old-growth logging was stopped after 90% of the forests were cleared. And although hundreds of miles of fiber optic has been laid downtown, it’s not available for residential use. However, the Beaverton area does have ubiquitous FTTH.

I also talked to somebody who attended the Emerging Languages talks. He’s working on his M.Sc. in Computer Science, so found those talks fascinating.

Twitter Humor

There were some humorous tweets:

- “my MongoDB and CouchDB mugs are fighting each other.”
- “I got one MongoDB mug, but need two to safely store coffee.”

Notes

Note to self: skip the nightly parties unless you have a date. The bars are too loud to talk to anybody.

Note to the O’Reilly conference organizers: use meetup.com for the BOFs like ApacheCon does. The average audience was about 10 people, and with meetup it would be 4x that.

OSCON 2010 Slides
Tim Bray: Desperate Perl Hacker
Youtube: OSCON 2010 videos
blip.tv: OSCON2010 videos
wikipedia: Plug Computer
Jeremy Zawodny: MongoDB Early Impressions

He talked about his career as a programmer, college lecturer at UC Riverside, computer book author and developer of nuclear reactor control software.

It was interesting to hear first-hand that CS students during the dot com boom actually did enroll “just for the money”, regardless of interest in science or ability.

Originally his book on HAL was a download-only book, but No Starch Press was looking for content and actually contacted him for permission to publish it. It proved to be a popular book and another version is planned.

He said it takes about 2 years to learn the domain-specific knowledge about nuclear reactors, plus whatever time it takes to learn the programming languages or tools used for the project.

Using a debugger on nuclear reactor control software results in a scram, so planning ahead is a good idea.

Thanks again to Symantec for hosting the meeting.

Here’s a diff I wrote to make dtop work on recent versions of Linux (tested on CentOS 5.5 x86_64):

$ diff dtop.org.c dtop.c

311a312,322
> 	char buf2[30];
>
> 	if (strstr(c, "Swap:") != NULL) {
> 		sscanf(c, "Swap: %30s total, %*s used, %30s free", buf, buf2);
> 		nd->swap = dehumanize_number(buf);
> 		nd->swapfree = dehumanize_number(buf2);
> 		nd->inactmem = nd->wiredmem = nd->execmem = 0;
> 		return;
> 	}
>
>
344a356
> #if ! defined(__linux__)
364a377
> #endif
470a484,486
> #if defined(__linux__)
> 		case 11:
> #else
471a488
> #endif
517a535,539
> #if defined(__linux__)
> 		} else if (strstr(c, "Tasks:") != NULL) {
>                         sscanf(c, "Tasks: %d ",&nodedata[nn].procs);
> #else
519a542
> #endif

The output of dtop on linux looks like this:

HOSTNAME  PROCS  LOAD1  LOAD5 LOAD15 ACTIVE  INACT   FILE   FREE SWPFRE SWUSED
  g00-int     64   0.11   0.04   0.01	   0	  0	 0  7345M  2047M  0.00%
  g01-int     64   0.00   0.01   0.00	   0	  0	 0  7033M  2047M  0.00%
  g02-int     61   0.08   0.02   0.01	   0	  0	 0  6980M  2047M  0.00%
  g03-int     64   0.16   0.06   0.01	   0	  0	 0  7011M  2047M  0.00%
  g04-int     64   0.04   0.04   0.01	   0	  0	 0  6996M  2047M  0.00%
  g05-int     61   0.02   0.01   0.00	   0	  0	 0  7424M  2047M  0.00%

Here is the final, hardened version of dtop.c that uses secure C programming techniques (strn API and double-free safe.)

My long-term preference would be to rewrite dtop in Perl since parsing text input in old-school C is brittle.

Also, dtop should be able to handle top results from heterogeneous systems, and the linux ifdefs contribute to preventing that.

And here are some of the debugging commands I used:

ulimit -S -c unlimited > /dev/null 2>&1
valgrind -v --leak-check=full --show-reachable=yes --track-origins=yes ./dtop
gdb ./dtop core

Dan Saks: Why size_t matters
Karpov: About size_t and ptrdiff_t