The name “Gluster” was derived from the words “GNU” and “Cluster.” No relation to the Luster filesystem, and actually they have opposite overall architectures.

GlusterFS is a GPL3 distributed network filesystem that runs as a service in user mode on Linux on a network of servers (conceptually like Google GFS.) Redhat bought Gluster, Inc. in 2011 and calls it “Redhat Storage.” By running in user mode and reusing existing linux features and modules, GlusterFS gained reliability in months instead of the usual 10 years for other filesystems.

gluster.org is the community website for GlusterFS.

Previously, AB was CTO at California Digital Corporation (CDC). One of his projects was the 1024-node linux “Thunder” HPC supercomputer for LLNL, the most powerful in the USA at the time.

- after that, he still wanted to do Open Source projects, preferably without bureaucratic encumbrances
- got some angel funding, but also a seismic data company paid $500,000 to adapt HPC code in 3 months to replace IBM Regatta system, then a follow-on storage contract for 1.2 PB in a 6 months project
- audience member: “In 10 years 1 PB will fit on a SD card.”
- GlusterFS is in some ways architecturally the opposite of VMware. GlusterFS is userland code.
- in 2006 large companies like Lehman, etc. started appearing on mailing list, to some surprise and awe
- originally the company was called Z Research, renamed to Gluster, Inc. to have clearer brand name
- no in-house test storage hardware, so developed on customer hardware!
- was still doing other paying work such as embedded kernel stuff, web dev, etc to reduce burn rate initially. Too distracting.

- Google mixes app and GFS, app generates 64 MB chunks, GFS manages metadata. Too complicated for general use.
- GlusterFS is a distributed storage OS in user-space
- create container without knowledge of filesystem (Posix, ACL, etc. ) because no known common user pattern for storage
- lots of general C programmers available to recruit, but no filesystem developers without baggage about kernel
- in 2008-2009 added too many features to actually test
- VCs contacted them and invested A and B series total of $15 million, despite their storage “experts” saying it was crazy, users said it was awesome
- lowest layer is native filesystem like EXT3 or XFS, thus idiot-proof
- use extended attributes for metadata
- block, replication, striping, elastic hashing algorithm
- striping support by cleverly using sparse files with different offset on each server
- read server choice based on fastest response
- every directory has its own hash space
- good default behavior when adding servers (no thundering herd)
- striping is good for hotspot files or files too big for 1 volume, like saving HPC results
- will be unified file and object protocol for object storage
- there is a pathinfo command can query extended attribute, could be used with ssh for a fake MapReduce
- GET and PUT at command line
- GlusterFS is most heavily used for lots of files containing unstructured data
- 3.3 will have faster healing operations, better granularity for 100 VMs, KVM support, etc.
- currently shared-nothing, but with a little sharing healing can be faster
- will be HDFS clone mode

Data Storage models:

• objects
• file
• block
• structured data
• NoSQL
• semi-structured data.

- Redhat bought Gluster, Inc. for about $136 million in October, 2011. It was about 60 employees at the time. Now there’s about 40 engineers working on GlusterFS at Redhat. AB chose Redhat over other suitors because of its commitment to Open Source and linux.

A dozen people attended dinner afterwards:

- it was tough hiring people for Gluster Inc. since the concept of doing file systems in userland confused a lot of developers and managers.
- AB’s philosophy is that the Open Core model doesn’t serve end-users well, as all users need “extras” like user-friendly mgmt. programs, not just licensees
- companies seem to be happy to pay for GlusterFS support, one of the reasons being lack of in-house storage engineers
- lots of discussion about Illumos (OpenSolaris fork), ZFS and containers
- take a look at Nexenta
- An efficient WAN replication method with GlusterFS is to use the marker framework / queue using extended attributes to feed rsync a list of changed files, scales better than inotify
- no storage tiering yet for incoming/outgoing hotspot files
- Redhat is pushing xfs heavily internally, and has hired the available ex-SGI xfs developers
- some checksumming is done in GlusterFS, but no end-end checksumming. Need to look at performance and demand.
- historicaly, linux has had slower adoption and community interest in India than Western countries due to lagging Internet performance (often more practical to install linux from a magazine CD-ROM than attempting large downloads) and relatively higher cost of computers compared to local salaries
- AB got started in programming on a Spectrum microcomputer, and progressed over time to fixing minor bugs in the linux network drivers, culminating in GlusterFS.

Thanks again to Symantec for hosting this event.

GlusterFS Downloads
Chip’s Rant: Why GlusterFS is Glusterfsck’d Too
wikipedia: Extended file attributes

DARPA Grand Challenge

- Tully participated in the DARPA Grand Challenge on a team that made a self-driving car called Bob that went 3 miles, CMU went 7 miles. (Bob got confused and ended up on the wrong side of a barbed-wire fence.)
– Alice went farther with rackmount servers running Gentoo
– Little Ben used Mac minis, easier to unload nightly
– 90% of effort repeated by each team because code not designed to be reusable frameworks (Message passing, etc. ), thus ROS
– the Grand Challenge result was just a race, without a technology work product despite the massive effort invested, but did make people think about how to do robotics more efficiently

Robot Types

Robots are classified into 3 application groups:

1. Industrial – currently over 1 million robots used in factories. Foxconn plans to add 1 million. (I would argue that CNC machines are also robots.)
2. Service – also over 1 million robots used in cleaning, etc., largely from iRobot
3. Education/Hobby/Other – mostly custom-built at universities.

Robot Operating System (ROS)

– ROS protocol with listeners and clients in many languages, no project preferred language
– ROS is BSD-licensed
– goal is really good documentation so that researchers don’t feel the need to keep rewriting basic robotics software
– also maintain OpenCV and Pointclouds
– ROS uses Jenkins for Continuous Integration (CI), emails author of a build failure quickly
– Japanese industrial robot manufacturer Motoman is currently the only industrial mfg. that supports ROS. Most industrial mfgs. use the excuse that Open Source developers would pose a quality problem, but in reality want to maintain proprietary code as a trade secret.

Willow Garage PR2 Robot

– intended for general purpose combined robotics/visual processing research and education
– PR2 weighs 400 pounds, 4.5′ to 5.5′ tall, uses 4 lb. motors, and arms stay where set with series elastic actuators (springs)
– 4 lb. motors are unlikely to injure anybody, and no exclusion zone is needed between robots
– PR2 is $400k, SE is $285k, 30% off for approved Open Source projects
– 1200 watts max power consumption (500 watts per i7 computer plus 200 watts for all motors and sensors)
– can fetch beer, fold laundry in 5 or 6 min, or plug itself in
– has software simulator Gazebo of all motors and sensors, runs about 1/3 of actual PR2
– about 11 have been distributed to leading universities
– PR2 is designed to be heavy-duty, and universities can use it 3 lab shifts per day
– used industrial robots start at $4,000, so the PR2 is an expensive way to get into robotics unless you’re a vision researcher.

PR2 ($400k) Playing Pool

TurtleBot

– TurtleBot is an entry-level, affordable robot
– built from iRobot Create, Kinect and Asus EeePC 1215N (Atom), ROS
– about $1,400, assembled is $1,500
– also available is an accessory arm kit for $750, and a 6-pack holder
– Create is a Roomba minus vacuum plus serial port
– working on robot swarming, need better reconnect
– YouTube: Create Fridgemate Explained, in action

Above: TurtleBot ($1,500.) Black object on top is Kinect, black object in middle is Asus eeePc.



Above: I-Heart Create Arm ($750) plus 12 VDC Power Assembly

General Robotics Comments

- main limitation on current personal robots is battery power, followed by computing power
- the wireless network connection between robot and computing cloud is a severely limiting bottleneck, better to do on-robot
- bright lighting really helps robotic vision
- Japanese robots seem to be more interesting from a video promo standpoint than a technical one

Audience Comments

- one person expected robots to have legs and be able to walk like humans for some reason
- some were hopeful that Willow Garage was hiring, but they’re not a fast-growth company

BiliBot

- BiliBot ($1,200) is another affordable educational robot, designed at MIT. A basic arm is included that can lift 3 lbs.
- YouTube: BiliBot picking up a bunch of grapes

Above: BiliBot ($1,200.)

Toys

- HexBug makes inexpensive (under $30) but fragile toy robots with legs for ages 8+
- WowWee Roboquad (CES 2007) is an awesome $400 toy for robot enthusiasts
- USB Rocket Launcher, handy to cannibalize for PC-controlled motors ($25) with 360 degree horizontal rotation and 45 degree vertical rotation

Thanks to Symantec for hosting the meeting.

With these autonomous cars, who needs you to drive?
wikipedia: ROS
iRobot SPARK
darpagrandchallenge.com
Stanford Point Clouds FAQ
Roomba game encourages you to attack dust bunnies
IEEE Spectrum DIY: Willow Garage’s TurtleBot Brings Mobile 3D Mapping and ROS to Your Budget
Bilibot Is the Affordable ROS Platform You’ve Been Looking For
everything-robotic.com: Where are the robots? Where’s that $100 billion industry we’ve seen in the forecasts?
Open source DARwIn-OP bot can be yours for just $12,000
Used-Robots.com
robotsltd.co.uk: Industrial Robot Applications
7 Jobs That Can Be Done By A Robot

I couldn’t make it there tonite, but here’s the slideshare and PDF. It’s an excellent presentation.

Olve is a software developer at Cisco Systems. He has a blog and twitter.

Even if you’re not a full-time C/C++ programmer, it pays to stay current. I had to drop into C twice this year to solve problems, in addition to reading the usual strace/oprofile dumps.

My recommendation is to use lint and build with 2 different compilers, then try to fix all warnings. (Brian Aker goes further and says that all warnings should be fixed, and enforces that on the Drizzle project.)

Thanks again to Symantec for hosting the meeting.

Tomer did a great job of answering a variety of technical questions.

In order to improve High Availability (HA), MapR has:

• rewritten Hadoop Distributed File System (HDFS) in C++ as MapR-FS. 3 compressed copies are stored. It is snapshottable and exportable as NFS.
• distributed the namenode across all storage nodes, resulting in 1000x namenode performance and avoiding Java GC
• made the jobtracker restartable by checkpointing
• granular permissions allow multi-tenancy
• added a nice mgmt. User Interface (UI) with heatmaps.

So MapR has replaced HDFS, and modified Apache Hadoop to change how namenode and jobtracker work. Hadoop-ecology layers above that are not affected, like Pig or Hive. Nobody has requested HCatalog support yet.

MapR benchmarks appear to be 2x to 4x faster than Apache Hadoop, esp. on small files, so a cluster would need half as many nodes. (See Slideshare presentations for benchmark graphs.)

There’s 2 MapR product versions:

1. MapR M3 Free Edition is non-HA and forum-supported.
2. MapR M5 includes HA and phone support and is $4,000/node/year.

EMC is reselling MapR. They are planning to build a 1,000 node cluster for demo purposes.

Currently, the deployed MapR clusters are small (in the range of 10 to 100 nodes), but MapR is designed to handle clusters of 10,000 nodes. Apache Hadoop maxes out at almost 4,000 nodes currently.

Thanks once again to Symantec for hosting the event.

MapR on YouTube
Ted Dunning’s MapR Talks on Slideshare YouTube: Localhost mounting of MapR NFS
Apache Hadoop FAQ
Tomer Shiran on Quora.com
Ryan’s Review of Hadoop Summit 2011
gigaom.com: Investors make $20M bet on MapR to win Hadoop war
gigaom.com: Why MapR Is Right to Give Back to Apache Hadoop
wikipedia: MapR

Brian Kernighan, Ken Thompson and Dennis Ritchie are the biggest influences on my career, starting when I became aware of Unix in high school.

I never had the chance to meet Dennis, but I attended a lecture by Brian in 1984 on “Little Languages” in Waterloo.

After reading K&R, I was able to write software that ran on the Space Shuttle.

Dennis Ritchie: The shoulders Steve Jobs stood on
Wikipedia: Dennis Ritchie
guardian.co.uk: Dennis Ritchie obituary
oreilly.com: On Dennis Ritchie: A conversation with Brian Kernighan
wikipedia: The C Programming Language

There were 209 sessions and 3,414 registered attendees.

I did not attend, but have seen some of the talks at other conferences in the past year. Pretty impressive list of speakers!

List price for support was $4,000 to $11,000 annually per node, where a node is a server with possibly multiple storage devices regardless of storage capacity. (I’ve heard of volume pricing of $800/node for clusters.)

Thus if Redhat bundles GlusterFS support into RHEL, it would be a bargain for many situations.

Red Hat to Acquire Gluster
networkworld.com: Redhat Buys GlusterFS
Gluster Price List (Oct 3, 2011)
Gluster Pushes Storage Software to VMware, Amazon
arstechnica.com: Anyone using Gluster?

Here’s a plugin I wrote to calculate and display the performance of various MySQL read and write caches.

You can download the plugin source here.

Note that hit rates are seldom easy to interpret or helpful on their own. Mixing transactional and reporting loads will muddy them (as above.) This graph is more meaningful if you also monitor absolute read and write counts.

Percona Blog: What MySQL buffer cache hit rate should you target (2006)

Block diagram of Oracle’s Big Data Appliance

I can certainly see why Oracle database customers would want to know Oracle’s Hadoop support story.

It remains to be seen how well Oracle can cooperate with the Apache Software Foundation and Open Source communities in general.

It’s interesting that Oracle’s NoSQL Database is based on Berkeley DB.

Whenever I mention that as the original NoSQL database I get puzzled looks from hipsters, yet Unix developers have used DBM/BDB as a key-value store for decades.

theregister.co.uk: Oracle rolls its own NoSQL and Hadoop
theregister.co.uk: Microsoft pops Hadoop into SQL Server and Azure