The name “Gluster” was derived from the words “GNU” and “Cluster.” No relation to the Luster filesystem, and actually they have opposite overall architectures.

GlusterFS is a GPL3 distributed network filesystem that runs as a service in user mode on Linux on a network of servers (conceptually like Google GFS.) Redhat bought Gluster, Inc. in 2011 and calls it “Redhat Storage.” By running in user mode and reusing existing linux features and modules, GlusterFS gained reliability in months instead of the usual 10 years for other filesystems.

gluster.org is the community website for GlusterFS.

Previously, AB was CTO at California Digital Corporation (CDC). One of his projects was the 1024-node linux “Thunder” HPC supercomputer for LLNL, the most powerful in the USA at the time.

- after that, he still wanted to do Open Source projects, preferably without bureaucratic encumbrances
- got some angel funding, but also a seismic data company paid $500,000 to adapt HPC code in 3 months to replace IBM Regatta system, then a follow-on storage contract for 1.2 PB in a 6 months project
- audience member: “In 10 years 1 PB will fit on a SD card.”
- GlusterFS is in some ways architecturally the opposite of VMware. GlusterFS is userland code.
- in 2006 large companies like Lehman, etc. started appearing on mailing list, to some surprise and awe
- originally the company was called Z Research, renamed to Gluster, Inc. to have clearer brand name
- no in-house test storage hardware, so developed on customer hardware!
- was still doing other paying work such as embedded kernel stuff, web dev, etc to reduce burn rate initially. Too distracting.

- Google mixes app and GFS, app generates 64 MB chunks, GFS manages metadata. Too complicated for general use.
- GlusterFS is a distributed storage OS in user-space
- create container without knowledge of filesystem (Posix, ACL, etc. ) because no known common user pattern for storage
- lots of general C programmers available to recruit, but no filesystem developers without baggage about kernel
- in 2008-2009 added too many features to actually test
- VCs contacted them and invested A and B series total of $15 million, despite their storage “experts” saying it was crazy, users said it was awesome
- lowest layer is native filesystem like EXT3 or XFS, thus idiot-proof
- use extended attributes for metadata
- block, replication, striping, elastic hashing algorithm
- striping support by cleverly using sparse files with different offset on each server
- read server choice based on fastest response
- every directory has its own hash space
- good default behavior when adding servers (no thundering herd)
- striping is good for hotspot files or files too big for 1 volume, like saving HPC results
- will be unified file and object protocol for object storage
- there is a pathinfo command can query extended attribute, could be used with ssh for a fake MapReduce
- GET and PUT at command line
- GlusterFS is most heavily used for lots of files containing unstructured data
- 3.3 will have faster healing operations, better granularity for 100 VMs, KVM support, etc.
- currently shared-nothing, but with a little sharing healing can be faster
- will be HDFS clone mode

Data Storage models:

• objects
• file
• block
• structured data
• NoSQL
• semi-structured data.

- Redhat bought Gluster, Inc. for about $136 million in October, 2011. It was about 60 employees at the time. Now there’s about 40 engineers working on GlusterFS at Redhat. AB chose Redhat over other suitors because of its commitment to Open Source and linux.

A dozen people attended dinner afterwards:

- it was tough hiring people for Gluster Inc. since the concept of doing file systems in userland confused a lot of developers and managers.
- AB’s philosophy is that the Open Core model doesn’t serve end-users well, as all users need “extras” like user-friendly mgmt. programs, not just licensees
- companies seem to be happy to pay for GlusterFS support, one of the reasons being lack of in-house storage engineers
- lots of discussion about Illumos (OpenSolaris fork), ZFS and containers
- take a look at Nexenta
- An efficient WAN replication method with GlusterFS is to use the marker framework / queue using extended attributes to feed rsync a list of changed files, scales better than inotify
- no storage tiering yet for incoming/outgoing hotspot files
- Redhat is pushing xfs heavily internally, and has hired the available ex-SGI xfs developers
- some checksumming is done in GlusterFS, but no end-end checksumming. Need to look at performance and demand.
- historicaly, linux has had slower adoption and community interest in India than Western countries due to lagging Internet performance (often more practical to install linux from a magazine CD-ROM than attempting large downloads) and relatively higher cost of computers compared to local salaries
- AB got started in programming on a Spectrum microcomputer, and progressed over time to fixing minor bugs in the linux network drivers, culminating in GlusterFS.

Thanks again to Symantec for hosting this event.

GlusterFS Downloads
Chip’s Rant: Why GlusterFS is Glusterfsck’d Too
wikipedia: Extended file attributes

DARPA Grand Challenge

- Tully participated in the DARPA Grand Challenge on a team that made a self-driving car called Bob that went 3 miles, CMU went 7 miles. (Bob got confused and ended up on the wrong side of a barbed-wire fence.)
– Alice went farther with rackmount servers running Gentoo
– Little Ben used Mac minis, easier to unload nightly
– 90% of effort repeated by each team because code not designed to be reusable frameworks (Message passing, etc. ), thus ROS
– the Grand Challenge result was just a race, without a technology work product despite the massive effort invested, but did make people think about how to do robotics more efficiently

Robot Types

Robots are classified into 3 application groups:

1. Industrial – currently over 1 million robots used in factories. Foxconn plans to add 1 million. (I would argue that CNC machines are also robots.)
2. Service – also over 1 million robots used in cleaning, etc., largely from iRobot
3. Education/Hobby/Other – mostly custom-built at universities.

Robot Operating System (ROS)

– ROS protocol with listeners and clients in many languages, no project preferred language
– ROS is BSD-licensed
– goal is really good documentation so that researchers don’t feel the need to keep rewriting basic robotics software
– also maintain OpenCV and Pointclouds
– ROS uses Jenkins for Continuous Integration (CI), emails author of a build failure quickly
– Japanese industrial robot manufacturer Motoman is currently the only industrial mfg. that supports ROS. Most industrial mfgs. use the excuse that Open Source developers would pose a quality problem, but in reality want to maintain proprietary code as a trade secret.

Willow Garage PR2 Robot

– intended for general purpose combined robotics/visual processing research and education
– PR2 weighs 400 pounds, 4.5′ to 5.5′ tall, uses 4 lb. motors, and arms stay where set with series elastic actuators (springs)
– 4 lb. motors are unlikely to injure anybody, and no exclusion zone is needed between robots
– PR2 is $400k, SE is $285k, 30% off for approved Open Source projects
– 1200 watts max power consumption (500 watts per i7 computer plus 200 watts for all motors and sensors)
– can fetch beer, fold laundry in 5 or 6 min, or plug itself in
– has software simulator Gazebo of all motors and sensors, runs about 1/3 of actual PR2
– about 11 have been distributed to leading universities
– PR2 is designed to be heavy-duty, and universities can use it 3 lab shifts per day
– used industrial robots start at $4,000, so the PR2 is an expensive way to get into robotics unless you’re a vision researcher.

PR2 ($400k) Playing Pool

TurtleBot

– TurtleBot is an entry-level, affordable robot
– built from iRobot Create, Kinect and Asus EeePC 1215N (Atom), ROS
– about $1,400, assembled is $1,500
– also available is an accessory arm kit for $750, and a 6-pack holder
– Create is a Roomba minus vacuum plus serial port
– working on robot swarming, need better reconnect
– YouTube: Create Fridgemate Explained, in action

Above: TurtleBot ($1,500.) Black object on top is Kinect, black object in middle is Asus eeePc.



Above: I-Heart Create Arm ($750) plus 12 VDC Power Assembly

General Robotics Comments

- main limitation on current personal robots is battery power, followed by computing power
- the wireless network connection between robot and computing cloud is a severely limiting bottleneck, better to do on-robot
- bright lighting really helps robotic vision
- Japanese robots seem to be more interesting from a video promo standpoint than a technical one

Audience Comments

- one person expected robots to have legs and be able to walk like humans for some reason
- some were hopeful that Willow Garage was hiring, but they’re not a fast-growth company

BiliBot

- BiliBot ($1,200) is another affordable educational robot, designed at MIT. A basic arm is included that can lift 3 lbs.
- YouTube: BiliBot picking up a bunch of grapes

Above: BiliBot ($1,200.)

Toys

- HexBug makes inexpensive (under $30) but fragile toy robots with legs for ages 8+
- WowWee Roboquad (CES 2007) is an awesome $400 toy for robot enthusiasts
- USB Rocket Launcher, handy to cannibalize for PC-controlled motors ($25) with 360 degree horizontal rotation and 45 degree vertical rotation

Thanks to Symantec for hosting the meeting.

With these autonomous cars, who needs you to drive?
wikipedia: ROS
iRobot SPARK
darpagrandchallenge.com
Stanford Point Clouds FAQ
Roomba game encourages you to attack dust bunnies
IEEE Spectrum DIY: Willow Garage’s TurtleBot Brings Mobile 3D Mapping and ROS to Your Budget
Bilibot Is the Affordable ROS Platform You’ve Been Looking For
everything-robotic.com: Where are the robots? Where’s that $100 billion industry we’ve seen in the forecasts?
Open source DARwIn-OP bot can be yours for just $12,000
Used-Robots.com
robotsltd.co.uk: Industrial Robot Applications
7 Jobs That Can Be Done By A Robot

I couldn’t make it there tonite, but here’s the slideshare and PDF. It’s an excellent presentation.

Olve is a software developer at Cisco Systems. He has a blog and twitter.

Even if you’re not a full-time C/C++ programmer, it pays to stay current. I had to drop into C twice this year to solve problems, in addition to reading the usual strace/oprofile dumps.

My recommendation is to use lint and build with 2 different compilers, then try to fix all warnings. (Brian Aker goes further and says that all warnings should be fixed, and enforces that on the Drizzle project.)

Thanks again to Symantec for hosting the meeting.

Tomer did a great job of answering a variety of technical questions.

In order to improve High Availability (HA), MapR has:

• rewritten Hadoop Distributed File System (HDFS) in C++ as MapR-FS. 3 compressed copies are stored. It is snapshottable and exportable as NFS.
• distributed the namenode across all storage nodes, resulting in 1000x namenode performance and avoiding Java GC
• made the jobtracker restartable by checkpointing
• granular permissions allow multi-tenancy
• added a nice mgmt. User Interface (UI) with heatmaps.

So MapR has replaced HDFS, and modified Apache Hadoop to change how namenode and jobtracker work. Hadoop-ecology layers above that are not affected, like Pig or Hive. Nobody has requested HCatalog support yet.

MapR benchmarks appear to be 2x to 4x faster than Apache Hadoop, esp. on small files, so a cluster would need half as many nodes. (See Slideshare presentations for benchmark graphs.)

There’s 2 MapR product versions:

1. MapR M3 Free Edition is non-HA and forum-supported.
2. MapR M5 includes HA and phone support and is $4,000/node/year.

EMC is reselling MapR. They are planning to build a 1,000 node cluster for demo purposes.

Currently, the deployed MapR clusters are small (in the range of 10 to 100 nodes), but MapR is designed to handle clusters of 10,000 nodes. Apache Hadoop maxes out at almost 4,000 nodes currently.

Thanks once again to Symantec for hosting the event.

MapR on YouTube
Ted Dunning’s MapR Talks on Slideshare YouTube: Localhost mounting of MapR NFS
Apache Hadoop FAQ
Tomer Shiran on Quora.com
Ryan’s Review of Hadoop Summit 2011
gigaom.com: Investors make $20M bet on MapR to win Hadoop war
gigaom.com: Why MapR Is Right to Give Back to Apache Hadoop
wikipedia: MapR

1. Python At Google (A Google IO talk)
2. YouTube for Your Business

Python At Google (A Google IO talk)

Python 2.6 is an officially-supported production language, along with C++ and Java.

The cluster and meta-cluster management software was about 70% Python/30% C++, may have changed.

For applications that are to be deployed on a million servers, saving even 5% memory is important, so C++ is used there.

Around a dozen well-known Python developers and authors work at Google around the world.

Even internal programmers are encouraged to use Google AppEngine.

YouTube for Your Business, presented by JJ Behrens

JJ Behrens is a developer advocate at Google specializing in YouTube APIs.

Some year-old YouTube numbers:

• 3 billion videos viewed per day
• every minute, 48 hours more video is uploaded
• usage seemed to be about 10:1 of PC vs mobile viewing
• new video formats and resolutions are added over time in a reasonably aggressive manner

One promotion strategy that has been found to work well for many businesses is to create instructional and demonstration videos and upload them to YouTube and serve them from there.

You can also embed the video on your site in an iframe, or for more advanced users use the API.

Read TOS, Monetization, Branding and API TOS guides. Spirit is that if the page only contains a YT video, then you can’t place your own ads.

Although it’s easy to broadcast videos to promote your business, reporting on who views them is not available.

You can do live streaming while saving your video via YouTube upload and download, though it requires some organization to do it well.

Many games allow directly uploading videos from in-game to YouTube now, including PC and PS3 and games.

Youtube Direct on AppEngine
Shortform.com has manually curated content
Gdata apis
Storify.com
Memolane.com – curation app that helps you create a lane (channel) of all your friends’ social media items.

YT/create
Squrl.com
Lvzzk

JJ mentioned that Python was the likely reason that the original YouTube team was able to add features more quickly than the Google Videos team. YouTube used Python, while Google Videos used C++. He felt that Python was 5x more productive.

Thanks once again to Symantec for hosting the event.

Google I/O 2011: Python@Google
wsj.com: YouTube Tees Up Big Talent
Your own movie database in 5 minutes with IMDb API and Perl

It was held in parallel at the Oregon Convention Center with the O’Reilly OSdata and OSjava Conferences at the beginning of the week, and then later a knitting conference.

The conferences were well-managed, as usual. Great economy: lots of job notices and recruiting appeals. There was some chatter about Tim’s anti-harassment blog post.

Executive Summary:

• HTML5/CSS3/AppCache are what should have been available 20 years ago, and are significant improvements that allow both desktop and mobile development in HTML. Although the HTML5 video tag gets a lot of press, HTML5 includes equally important forms improvements.
• DNSSEC is complex and new signatures should be generated every 30 days or less (to reduce replay attacks by limiting the signature validity period), which is a burden on companies without a full-time DNS hostmaster. Third-party DNS hosting companies are salivating over DNSSEC.
• MySQL long-term stewardship is still in question, with Oracle hemorrhaging MySQL developers and closing access to their bugs database, but MontyProgram and Percona maintaining strong forks.

Here’s my notes on some of the tutorials and talks I attended:

Monday

HTML5 & CSS3: The Good Enough Parts
Estelle Weyl, Standardista.com
Slides

- transform-origin is key to snowflake demo looking realistic, easy to use
- background resets everything, so use individual properties
- background-position – use all 4 values
- background-size auto contain cover, handy for iPhones
- text-overflow: ellipsis
- minimal HTML5 document:

<!doctype html5>
<meta charset=utf8>
<title></title>

(head and body are implied)
- or even send tags in server headers
- changed most elements
- <i lang=”"> useful to style
- small tag useful for legal smallprint, since there’s no copyright metatag yet
- html5shim
- Modernizr
- HTML5 Boilerplate – good way to learn HTML5 and CSS3
- tabindex=”-1″ allows JS to set focus and not bother user otherwise
- spellcheck=”true” | “false”
- itemtype=”http://data-vocabulary.org/Person”
- new input types
- placeholder, pattern, required, spellcheck, validate
- a@b is deliverable for internal email servers. hmm.
- meter, progress, output widgets
- HTML5Rocks
- button generator at css3button.net
- Voronoi diagram demo
- aquarium.js
- web workers

Monday Lunch

Benjamin, Ubuntu
- loves CloudFlare
- likes Linode
- nimbula

Talked to an open mapping data fellow about various projects. Google ToS is scary when it comes to that kind of data.

Monday Afternoon

Moose is Perl: A Guide to the New Revolution
Ricardo Signes, Pobox.com
Slides

- detailed talk about Moose features and syntax
- chatted with other folks at break time about topics like CSS::Inliner and Mojolicious web framework (with minimal dependencies) by Sebastian Riedel.

Tuesday Afternoon

Three For Five – Functional HTML5 & CSS3 for Designers & Developers
Jason VanLue, Envy Labs and CodeSchools.com

- good training session with fun sample – a beer menu created from 1 photo (CSS3 text scaling and rotation) and HTML5/CSS3 styled text
- training class is available online for $75 (also jQuery and 2 Rails classes)

Click to Enlarge

Tuesday Night

Photo credit: Jacob Helwig

Creating a Scalable JavaScript Application Architecture
Nicholas Zakas, NCZConsulting
Slides

An AJAX client only cares about getting the data it wants, not response codes, etc.

Use layered JavaScript client architecture:

- sandbox
- application
- library (Dojo, YUI, MooTools, etc. )

www.nczonline.net
@slicknet
Author of “High Performance JavaScript”

Wednesday Lunch

I talked to Ben Golub, CEO of Gluster.

- 80% business, 20% scientific
- users include box.net, Pandora.com
- written in C
- minimum is 2 nodes for replication
- lots of people use it in EC2
- office located in Sunnyvale.

Wednesday Afternoon

HTML5: All about Web Forms
Estelle Weyl, standardista.com

- use label tag with forms to ease navigation for end-users
- use placeholder attribute, better for screenreaders than JS coding
- multiple autofocus defaults to last one in HTML5
- type=”text” is default, so tel, email, etc. degrades on all browsers back to text
- form element can disassociate parent form, useful for AJAX multiform pages
- input types good for mobile devices to show useful soft keyboard for url or email input types
- numeric step options
- test date and numeric input types for usability. Scrolling birthdays or zip codes is painful
- still need JS
- Opera is first with new UI features but last with artistic design, so currently has hideous tooltip appearance
- list and datalist like exploded select. Include select for IE backward compatibility
- meter, progress and output UI elements
- input type=text x-webkit-speech, now on Google homepage

HTML5 in Your Pocket: Application Cache and Local Storage
Scott Davis, ThirstyHead.com

- 4 million Macs, 32 million iDevices in last quarter
- Basecamp Mobile
- “Dive into HTML5″ by Mark Pilgrim online
- cookies should be called thimbles, only 4k
- HTML5 localstorage supported in IE8, FF 3.5, so practically all
- 5 MB, QUOTA_EXCEEDED_ERR, can’t increase now
- StatCounter browser stats
- gist 350433: Storage polyfill using window.name and cookies
- no version of IE or FF support web SQL and they probably won’t, FF for philosophical reasons
- cache manifest
- genManifest bookmarklet
- FF about:cache and Firebug are handy to see caches
- appcache has no expiry date
- date stamping manifest file causes re-download
- 404 causes none to be saved
- treat appcache as only slightly more secure than cookies, which are round-tripped
- webplication
- still sandboxed from local file access, could use node.js or signed app
- See W3C HTML5 offline

Profiling and Detecting Bottlenecks in Software
Bryan Call, Yahoo!/Apache Committer
PowerPoint .pptx

- usual savings (machines, moving parts, get smart)
- top, htop
- vmstat, dstat
- time cmd
- Boost logging does small writes, allocates memory when it gets behind, causing both IO and memory pressure
- profilers like oprofile and google profile cause 1% to 8% slowdown
- valgrind’s callgrind much more resources
- oprofile has script to convert output to kcachegrind
- opcontrol –deinit
- sysctl nmi_watchdog off
- opcontrol –no-vmlinux
- opcontrol –daemon
- google profiler userland, LD_PRELOAD
- env CPUPROFILE=/tmp/mybin.prof /usr/local/bin/my_binary_compiled_with_libprofiler_so
- caching: don’t do the same work twice
- choose the correct algorithms and data structures: dqueue vs. List, hash vs. trees, locks vs. r/w locks, bloom filter
- reuse memory, stack vs. heap, tcmalloc
- make fewer system calls (larger reads and writes)
- faster hardware, bonded NICs, SSDs, RAID, CPU, more cores
- read How to Profile Apache Traffic Server
- bootchart
- http_load now uses epoll
- he made ab multi-core

From kernel.txt: “nmi_watchdog: Enables/Disables the NMI watchdog on x86 systems. When the value is non-zero the NMI watchdog is enabled and will continuously test all online cpus to determine whether or not they are still functioning properly. Currently, passing “nmi_watchdog=” parameter at boot time is required for this function to work. If LAPIC NMI watchdog method is in use (nmi_watchdog=2 kernel parameter), the NMI watchdog shares registers with oprofile. By disabling the NMI watchdog, oprofile may have more registers to utilize.”

CoffeeScript: A New Hope for JavaScript
Scott Davis, ThirstyHead.com

- trainer, author, worked on Comcast/Time Warner TVs which mostly use WebKit
- little language that compiles into JS
- JS V8 headless, like node.js
- PhantomJS is headless HTML, handy for testing
- Google GWT compiles Java to JS
- “transpiler”
- install node.js
- install npm
- npm install -g coffeescript
- –tokens, –nodes like java p
- immediately invoked function expression IIFE
- coffeescript: string interpolation #{name}, “”"
- objects with left-hand spacing like python

Thursday

DNSSEC @ Mozilla
Shyam Mani, Mozilla Corporation
Slides

- BIND 9.7 is nice for DNSSEC
- Keys are everything, protect them. Have a backup plan.
- Cisco core routers by default don’t expect large DNS transfers:

policy-map global policy class inspection_default inspect dns maximum-length 4096

- DS was live, no signed zones
- watch log levels, can be chatty and quickly fill disk with logs
- DNSSEC has no immediate benefit to end-users, since resolvers don’t honor it
- their logs show 1000:1 dns vs dnssec queries for last 6 months, but growing
- IOS Firewall DNSSEC

Ask Google Engineers Anything
Chris DiBona, Google

- 55 Google employees attending OSCON this year
- mostly end-user questions about Google+ circles and API
- or running Go on android
- or why does my telco not do firmware releases for my smartphone
- or not happy with Google search results this month
- I asked about original reason for GFS. Originally, the hardware was really that flaky, and Google even actively bought bulk refurbed computers and RAM, sometimes off the back of a truck. Got a gopher plushie in return.
- also some good feedback complaints: google groups UI inadequate for managing 350 groups in an Education scenario
- inadequate data import tools for non-profit users of groups, mentioned by a religious charity.

How Not to Design Like a Developer: Open Source Can Look Good Too!
Chrissie Brodigan, Mozilla/Firefox

- KPI vs. git (different goals)
- @sirupsen
- story about the downloads map graphic for FF 4 – a developer silently removed social button graphics, limiting participation of wider audience. Marketing needs to explain why and how other staff fit into outreach programs.
- hang out on #projectdesign
- design contests are a good way to get them to come out of the woodwork
- designers hang out on twitter, not irc
- programmers should avoid big red buttons that scare users, and improve accessibility
- Inkscape, Blender, HTML and CSS are some Open Source tools for design mockups
- do AB testing or survey users
- designers want to be martyrs, so be careful they don’t offer more than you are willing to accept (start with 1 icon rather than the whole set)
- take a look at graphics libre for icons
- Quitestrong.com 5 girls who do design

Gearman BOF
Brian Aker

- Gearman polls, beanstalk busy waits
- nice to have feature to give up to another thread
- monitor projects handle launching of workers
- Gearman has durable and non-durable queues
- is a superset of the crap you handrolled. Most of the homegrown apps peak at 50% to 60% of Gearman’s features
- Gearman is production ready, but the postgresql driver less so because of fewer test cases and Brian’s lesser familiarity
- setup ntp and use Gearman coalescence for redundant cron servers
- can inspect queue
- agnostic to backend
- 99designs.com looking at this, same use case as original developer
- I still think that if you already have a database app, adding a status column gets you a lot of Gearman functionality without one more moving part.

MariaDB BOF
Monty Widenius, MontyProgram AB

- Monty mentioned that the latest release of MariaDB 5.3-beta has faster replication from group commit and performance improvements on the master, which also help the slave. Also subqueries and joins work much better.
- Monty talked about his Aria storage engine, which is a replacement for MyISAM that has both transaction and non-transaction modes. It’s intended for users who want the space savings of MyISAM. Over time it may compete with InnoDB.
- Monty’s responsibility is to convince Percona to merge into 1 source base sometime
- it’s estimated that although Oracle still has the InnoDB team, they may only have 1 general MySQL server programmer left.
- he explained that MontyProgram developers work 50% on feature requests from end-users, and 50% Open Source-related. So paid requests for 1 week of work really need to cover 2 weeks of developer time for that model to work. Typically a medium-sized change is roughly $12,000 and includes development, testing and documentation.
- Zmanda got FLUSH TABLES WITH READ LOCK NO CHECKPOINT for a beer, though. Monty was able to find a code path that could be locked to prevent Aria and MyISAM from writing anything during the backup.
- I sponsored WL#232 for USD$100 to add a SHUTDOWN statement to MySQL
- Monty explained that MERGE tables may be a better choice than MySQL partitions for logging applications.
- attendees from MontyProgram, SkySQL, Percona, DeNA

AskMonty: MySQL “Wishlist” Session from an online travel agency
MySQL Manual 5.5: FLUSH Syntax

Perl Lightning Talks
Hosted by Geoff Avery

- a talk on why arrogant community members telling others that “they need a thick skin” is unhelpful
- a talk by a young Perl community member on getting commit access, and how others can get the spirit and contribute
- Larry did several talks, mostly encouraging backporting Perl6 features to Perl5 it seemed, perhaps as a replacement to going Moose
- nice song on the importance of public libraries, which face shutdown due to economic budgeting problems in Australia and USA
- nice comedy juggling act comparing programming languages. Perl6 was omitted as “nothing has been updated in 5 or 6 years”, prompting Larry to say that he was happy he has a thick skin. See above.
- afterward, I talked to a booking.com rep about why a European company needed to actively recruit in USA and world-wide. He said that European developers are happy working where they are now, and it’s easier to recruit in places with mobile workforces like the USA. He would like to hire a couple developers per week to meet their development schedule.

“ATA over Ethernet (AoE) is a network protocol designed for simple, high-performance access of SATA storage devices over Ethernet networks. It is used to build storage area networks (SANs) with low-cost, standard technologies.

Learn about a new storage infrastructure that eliminates bottlenecks such as costly controllers, addresses the right user requirement with the right disk solution, and can scale to Petabytes using the same platform, with linear scaling cost and performance.

More about the Speakers: Rob Przykucki (sha-koo-ski) is Director of Product Management at Coraid.

Alan Weisenburger is a Corporate Sales Representative at Coraid and formerly was an Enterprise Account Manager at EMC.

John Masci (unavailable tonite) is a Solutions Engineer with Coraid. He has worked as a Sr. Solutions Engineer at EMC and was an integral part in the development of VCE, the joint venture between VMware, Cisco, and EMC.”

Some notes:

• Coraid is marketing AoE as “EtherDrive” (trademarked in 2004) now
• most operating systems supported (Solaris, OpenSolaris, Linux, Windows, vmWare, Xen, etc.), but no BSDs – could port linux driver to *BSD. AoE still appears to be in the linux kernel as of this blog post.
• Vblade is a software-based AoE target, a virtual EtherDrive Blade. It exports local block storage to hosts on an Ethernet local area network. Hosts with an ATA over Ethernet (AoE) initiator, like the AoE driver for Linux, can then access the storage over Ethernet.
• Coraid has a max single LUN 512 TB. Multiple LUNs can be used for petabytes
• FCoE and iSCSI are mostly interchangeable.
• iSCSI has its own CRC because TCP recalculates between connections, so useless for end-to-end integrity across devices, and there are better/different algorithms than what TCP uses, checksum method 0 being 16-bit 1′s complement-based. See the iSCSI CRC RFC. Also, CRC32 is essentially free on Intel Nehalem CPUs now in SSE 4.2.
• AoE: connectionless, parallel delivery
• AoE has similar latency to FC, most direct and fastest SAN protocol.
• Plan9 Distributed Operating System (!) used for mgmt.
• Coraid HBAs for better monitoring than generic Ethernet port. 10 GbE and 1 GbE
• MACSEC can be enabled, other features available and can be enabled when needed
• 95% of support calls about switch problems (use recommended switches, not Netgear, etc. Likes Arista switches.) Also issues with non-certified disk drives.
• Marine Corps has 2 PB online now.
• NY SAG uses MS SQL on it, much better performance than a more expensive, famous competitor
• Many LAMP users, so definitely MySQL works on Coraid products
• Certified Disk Drives are required now. 3 TB SATA disks are available. Mix and match of drive sizes and types is supported, including SSD.
• Coraid has more than 1,400 customers.

Thanks once again to Symantec for hosting the meeting.

theregister.co.uk: Souped-up Coraid aims to change its spots (2011)
Coraid Blog
sourceforge: ATA over Ethernet Tools
Kernel Korner – ATA Over Ethernet: Putting Hard Drives on the LAN (2005)
storagemojo.com: Will FCoE save storage networks? (2008)
theregister.co.uk: Disk-over-Ethernet startup trousers another $50m (Nov, 2011)

“This presentation will cover the basics of “plain text” and its importance in providing meaningful content, and will explore some recent developments in Unicode that allow otherwise unencodable characters, such as variant forms of CJK Unified Ideographs, to be reliably represented in a “plain text” paradigm through the use of the Ideographic Variation Database (IVD).

Examples from other scripts will also be provided. Various “plain text” pitfalls and bad-practices that undermine Unicode’s success, such as PUA usage, CJK Compatibility Ideographs, and code point poaching, will be touched upon.

Finally, emerging environments that thrive on “plain text” data, such as mobile, will be discussed.”

Some of the interesting points:

• Adobe’s PDF creator has a 2 level scheme to separate text and presentation, but Microsoft’s PDF exporter only has 1. (I need to confirm the product names.)
• keep up with Unicode versions, now 6.0. Several code points are added each year.
• Unicode is not just code points, but also properties (Digit, etc.)
• the Basic Multilingual Plane (BMP) is full, so apps need to support additional planes
• testing HTML or SGML with numeric character references lets you control the input better as you test for rendering problems. The Unix od -h utility is helpful for examining files at the hex byte level.

Thanks to Adobe for hosting and broadcasting the meeting.

Character References Explained
wikipedia: Unicode

Dr. Hugh Williams has spent the past sixteen years researching and developing search engines and web services. He is presently the Vice President of Buyer Experience Engineering at eBay Inc., where his responsibilities include eBay’s search engine. In the past he managed a large R&D team at Microsoft’s Bing, and was a tenured professor in Australia.

He has published 99 works, mostly in the field of Information Retrieval, and including two books: “Web Database Applications with PHP and MySQL” and “Learning Mysql” for O’Reilly Media Inc. He holds 2 US patents and has 25 patents pending. He has a PhD from RMIT University in Australia

[Modern Search Engine Architecture Diagram]

Web Nodes

Cache

Aggregator Nodes

Row Nodes

Storage

Analysis (Links Graph)

Crawlers

Crawling

1 make a queue for links
2 add some seed links
3 while crawling, add new links to tail of queue.

1 fetch new resources from new domains or pages.
2 fetch new resources from existing domains or pages
3 refetch existing resources that have changed

A successful crawl is a new resource being indexed and visited by a user.

There are many challenges:

1 HTTP HEAD and GET can see different headers
2 Not found pages often return 2xx
3 pages can redirect to self
4 pages can look different to browsers and crawlers
5 JavaScript
6 cookies
7 Flash, non-HTML content
8 human vs machine challenges
9 authenticated sites (Facebook has 11% of page views, but cannot be crawled by outsiders)

If you write a video search crawler you have to start by processing Youtube and Hulu. Other sites may be emphasized less.

The web is now an infinite number of pages

Remove spam, illegal, dupe, traps, automatic generated, don’t exist, stale, not understandable.

Most search engines index 20 to 50 billion documents.

How do we choose the right pages?

Store those that meet future information needs!

There are many ways to choose the right pages:

Right page: users visit, popular in links graph, match queries, popular sites, clicked, shown by competitors, language or market, distinct, change at moderate rate.

Whatever choice is made:

1 the head is stable (4 to 6 billion)
2 the tail “wags around”, billions of candidate pages have similar or identical scores

Decay rate: maybe hour, day, weekly.

Choosing Pages in Practice

2 solutions to choosing pages for the index:

1 in real-time decide yes or no
2 store the pages and process them offline to construct an index

Former is Altavista chunk method. Latter is Google.

Compressed inverted index takes 10% to 20% of original content

Index Compression

Benefits of index compression:

1 save disk space
2 save communications bandwidth
3 more index can fit in memory.

Compression is only a benefit if uncompressing is faster than plain reading.

Rows

Aggregators cache results. They can also intelligently pick rows.

Wider rows decrease latency (ms).
More rows increase throughput (qps).

In practice, row nodes don’t exhaustively evaluate queries on the inverted index.

Nodes stop searching when;

1 time runs out
2 result sets are stable
3 enough results have been found
4 too much load.

So depending on query load, different results could be obtained depending on time of day, etc.

Ranking

Broder (2002) proposed this taxonomy of user queries:

1. Informational 80%
2. Navigational 10%
3. Transactional 10%

Median is 2 words, mean is 2.4. (2000).

48% of users submit 1 query in session, 21% two, 31% three or more (session duration is less than 30 seconds.)

Users don’t like boolean and phrase searches, and use wrong syntax when they try. (wrong number of quotes, quoting 1 word, etc.)

Good answer is relevant to user’s information need.

10 results per page

Consumer search engines are optimized for precision, but intelligence-oriented engines need total recall (all relevant documents.)

Relevance

Web search engines enforce boolean ‘and’.

More frequent is better. Term frequency.

More rare words is better. IDF

Some TF math …

More is better, log().

tf.idf ranking function.

Okapi BM25

Recall

Scoring recall.

Precision

Scoring precision.

One off-hand comment was that eBay encourages bots to crawl their site these days.

Thanks again to Symantec for hosting the meeting.

The Anatomy of a Large-Scale Hypertextual Web Search Engine – Sergey Brin and Lawrence Page

I watched the lecture via Adobe Connect webcast, which worked fairly smoothly.

Thanks again to Adobe for hosting the meeting.