James' World » Conferences http://www.jebriggs.com/blog Observations by a Programmer of Silicon Valley and Beyond Mon, 06 Feb 2012 11:03:37 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 Silicon Valley Codecamp 2011 http://www.jebriggs.com/blog/2011/10/silicon-valley-codecamp-2011/ http://www.jebriggs.com/blog/2011/10/silicon-valley-codecamp-2011/#comments Mon, 10 Oct 2011 06:42:55 +0000 Administrator http://www.jebriggs.com/blog/2011/10/ Continue reading →]]> Silicon Valley Codecamp 2011 was held this weekend at Foothill College near San Jose.

There were 209 sessions and 3,414 registered attendees.

I did not attend, but have seen some of the talks at other conferences in the past year. Pretty impressive list of speakers!

]]> http://www.jebriggs.com/blog/2011/10/silicon-valley-codecamp-2011/feed/ 0 OSCON 2011, Portland http://www.jebriggs.com/blog/2011/07/oscon-2011-portland/ http://www.jebriggs.com/blog/2011/07/oscon-2011-portland/#comments Sun, 31 Jul 2011 06:03:21 +0000 Administrator http://www.jebriggs.com/blog/?p=3735 Continue reading →]]> Once again, the O’Reilly Open Source Conference (OSCON) was held in Portland, Oregon.

It was held in parallel at the Oregon Convention Center with the O’Reilly OSdata and OSjava Conferences at the beginning of the week, and then later a knitting conference.

The conferences were well-managed, as usual. Great economy: lots of job notices and recruiting appeals. There was some chatter about Tim’s anti-harassment blog post.

Executive Summary:

  • HTML5/CSS3/AppCache are what should have been available 20 years ago, and are significant improvements that allow both desktop and mobile development in HTML. Although the HTML5 video tag gets a lot of press, HTML5 includes equally important forms improvements.
  • DNSSEC is complex and new signatures should be generated every 30 days or less (to reduce replay attacks by limiting the signature validity period), which is a burden on companies without a full-time DNS hostmaster. Third-party DNS hosting companies are salivating over DNSSEC.
  • MySQL long-term stewardship is still in question, with Oracle hemorrhaging MySQL developers and closing access to their bugs database, but MontyProgram and Percona maintaining strong forks.

Here’s my notes on some of the tutorials and talks I attended:

Monday

HTML5 & CSS3: The Good Enough Parts
Estelle Weyl, Standardista.com
Slides

- transform-origin is key to snowflake demo looking realistic, easy to use
- background resets everything, so use individual properties
- background-position – use all 4 values
- background-size auto contain cover, handy for iPhones
- text-overflow: ellipsis
- minimal HTML5 document:

<!doctype html5>
<meta charset=utf8>
<title></title>

(head and body are implied)
- or even send tags in server headers
- changed most elements
- <i lang=”"> useful to style
- small tag useful for legal smallprint, since there’s no copyright metatag yet
- html5shim
- Modernizr
- HTML5 Boilerplate – good way to learn HTML5 and CSS3
- tabindex=”-1″ allows JS to set focus and not bother user otherwise
- spellcheck=”true” | “false”
- itemtype=”http://data-vocabulary.org/Person”
- new input types
- placeholder, pattern, required, spellcheck, validate
- a@b is deliverable for internal email servers. hmm.
- meter, progress, output widgets
- HTML5Rocks
- button generator at css3button.net
- Voronoi diagram demo
- aquarium.js
- web workers

Monday Lunch

Benjamin, Ubuntu
- loves CloudFlare
- likes Linode
- nimbula

Talked to an open mapping data fellow about various projects. Google ToS is scary when it comes to that kind of data.

Monday Afternoon

Moose is Perl: A Guide to the New Revolution
Ricardo Signes, Pobox.com
Slides

- detailed talk about Moose features and syntax
- chatted with other folks at break time about topics like CSS::Inliner and Mojolicious web framework (with minimal dependencies) by Sebastian Riedel.

Tuesday Afternoon

Three For Five – Functional HTML5 & CSS3 for Designers & Developers
Jason VanLue, Envy Labs and CodeSchools.com

- good training session with fun sample – a beer menu created from 1 photo (CSS3 text scaling and rotation) and HTML5/CSS3 styled text
- training class is available online for $75 (also jQuery and 2 Rails classes)


3-for-5 Beer Menu
Click to Enlarge

Tuesday Night


Puppet Labs Logo

- went to Puppet Labs office for CloudCamp lightning talks, which started about 90 minutes late
- nice office, typical start-up look across from a small park. Comfy little meeting rooms with leather sofas a la Netflix.
- about 5 lightning talks total, 2 were sales pitches, 2 had 40 slides crammed into 5 minutes. ick.
- Puppet Labs CEO gave a good talk on optimizing Puppet for a client with 10,000+ nodes. Converted XML::RPC to REST, which doubled performance from 500 to 1,000 qps (I talked to Randy Ray about that, and he wasn’t surprised and that would be the case on simple requests), did some more work and maxed out at 2,500 qps. Enabling SSL did not slow down requests.
- got too crowded for me, and also fire department, who manned the exits and counted people as they entered and left.

Puppet Labs Office
Photo credit: Jacob Helwig

Wednesday

Creating a Scalable JavaScript Application Architecture
Nicholas Zakas, NCZConsulting
Slides

An AJAX client only cares about getting the data it wants, not response codes, etc.

Use layered JavaScript client architecture:

- sandbox
- application
- library (Dojo, YUI, MooTools, etc. )

www.nczonline.net
@slicknet
Author of “High Performance JavaScript”

Wednesday Lunch

I talked to Ben Golub, CEO of Gluster.

- 80% business, 20% scientific
- users include box.net, Pandora.com
- written in C
- minimum is 2 nodes for replication
- lots of people use it in EC2
- office located in Sunnyvale.

Wednesday Afternoon

HTML5: All about Web Forms
Estelle Weyl, standardista.com

- use label tag with forms to ease navigation for end-users
- use placeholder attribute, better for screenreaders than JS coding
- multiple autofocus defaults to last one in HTML5
- type=”text” is default, so tel, email, etc. degrades on all browsers back to text
- form element can disassociate parent form, useful for AJAX multiform pages
- input types good for mobile devices to show useful soft keyboard for url or email input types
- numeric step options
- test date and numeric input types for usability. Scrolling birthdays or zip codes is painful
- still need JS
- Opera is first with new UI features but last with artistic design, so currently has hideous tooltip appearance
- list and datalist like exploded select. Include select for IE backward compatibility
- meter, progress and output UI elements
- input type=text x-webkit-speech, now on Google homepage

HTML5 in Your Pocket: Application Cache and Local Storage
Scott Davis, ThirstyHead.com

- 4 million Macs, 32 million iDevices in last quarter
- Basecamp Mobile
- “Dive into HTML5″ by Mark Pilgrim online
- cookies should be called thimbles, only 4k
- HTML5 localstorage supported in IE8, FF 3.5, so practically all
- 5 MB, QUOTA_EXCEEDED_ERR, can’t increase now
- StatCounter browser stats
- gist 350433: Storage polyfill using window.name and cookies
- no version of IE or FF support web SQL and they probably won’t, FF for philosophical reasons
- cache manifest
- genManifest bookmarklet
- FF about:cache and Firebug are handy to see caches
- appcache has no expiry date
- date stamping manifest file causes re-download
- 404 causes none to be saved
- treat appcache as only slightly more secure than cookies, which are round-tripped
- webplication
- still sandboxed from local file access, could use node.js or signed app
- See W3C HTML5 offline

Profiling and Detecting Bottlenecks in Software
Bryan Call, Yahoo!/Apache Committer
PowerPoint .pptx

- usual savings (machines, moving parts, get smart)
- top, htop
- vmstat, dstat
- time cmd
- Boost logging does small writes, allocates memory when it gets behind, causing both IO and memory pressure
- profilers like oprofile and google profile cause 1% to 8% slowdown
- valgrind’s callgrind much more resources
- oprofile has script to convert output to kcachegrind
- opcontrol –deinit
- sysctl nmi_watchdog off
- opcontrol –no-vmlinux
- opcontrol –daemon
- google profiler userland, LD_PRELOAD
- env CPUPROFILE=/tmp/mybin.prof /usr/local/bin/my_binary_compiled_with_libprofiler_so
- caching: don’t do the same work twice
- choose the correct algorithms and data structures: dqueue vs. List, hash vs. trees, locks vs. r/w locks, bloom filter
- reuse memory, stack vs. heap, tcmalloc
- make fewer system calls (larger reads and writes)
- faster hardware, bonded NICs, SSDs, RAID, CPU, more cores
- read How to Profile Apache Traffic Server
- bootchart
- http_load now uses epoll
- he made ab multi-core

From kernel.txt: “nmi_watchdog: Enables/Disables the NMI watchdog on x86 systems. When the value is non-zero the NMI watchdog is enabled and will continuously test all online cpus to determine whether or not they are still functioning properly. Currently, passing “nmi_watchdog=” parameter at boot time is required for this function to work. If LAPIC NMI watchdog method is in use (nmi_watchdog=2 kernel parameter), the NMI watchdog shares registers with oprofile. By disabling the NMI watchdog, oprofile may have more registers to utilize.”

CoffeeScript: A New Hope for JavaScript
Scott Davis, ThirstyHead.com

- trainer, author, worked on Comcast/Time Warner TVs which mostly use WebKit
- little language that compiles into JS
- JS V8 headless, like node.js
- PhantomJS is headless HTML, handy for testing
- Google GWT compiles Java to JS
- “transpiler”
- install node.js
- install npm
- npm install -g coffeescript
- –tokens, –nodes like java p
- immediately invoked function expression IIFE
- coffeescript: string interpolation #{name}, “”"
- objects with left-hand spacing like python

Thursday

DNSSEC @ Mozilla
Shyam Mani, Mozilla Corporation
Slides

- BIND 9.7 is nice for DNSSEC
- Keys are everything, protect them. Have a backup plan.
- Cisco core routers by default don’t expect large DNS transfers:

policy-map global policy class inspection_default inspect dns maximum-length 4096

- DS was live, no signed zones
- watch log levels, can be chatty and quickly fill disk with logs
- DNSSEC has no immediate benefit to end-users, since resolvers don’t honor it
- their logs show 1000:1 dns vs dnssec queries for last 6 months, but growing
- IOS Firewall DNSSEC

Ask Google Engineers Anything
Chris DiBona, Google

- 55 Google employees attending OSCON this year
- mostly end-user questions about Google+ circles and API
- or running Go on android
- or why does my telco not do firmware releases for my smartphone
- or not happy with Google search results this month
- I asked about original reason for GFS. Originally, the hardware was really that flaky, and Google even actively bought bulk refurbed computers and RAM, sometimes off the back of a truck. Got a gopher plushie in return.
- also some good feedback complaints: google groups UI inadequate for managing 350 groups in an Education scenario
- inadequate data import tools for non-profit users of groups, mentioned by a religious charity.

How Not to Design Like a Developer: Open Source Can Look Good Too!
Chrissie Brodigan, Mozilla/Firefox

- KPI vs. git (different goals)
- @sirupsen
- story about the downloads map graphic for FF 4 – a developer silently removed social button graphics, limiting participation of wider audience. Marketing needs to explain why and how other staff fit into outreach programs.
- hang out on #projectdesign
- design contests are a good way to get them to come out of the woodwork
- designers hang out on twitter, not irc
- programmers should avoid big red buttons that scare users, and improve accessibility
- Inkscape, Blender, HTML and CSS are some Open Source tools for design mockups
- do AB testing or survey users
- designers want to be martyrs, so be careful they don’t offer more than you are willing to accept (start with 1 icon rather than the whole set)
- take a look at graphics libre for icons
- Quitestrong.com 5 girls who do design

Gearman BOF
Brian Aker

- Gearman polls, beanstalk busy waits
- nice to have feature to give up to another thread
- monitor projects handle launching of workers
- Gearman has durable and non-durable queues
- is a superset of the crap you handrolled. Most of the homegrown apps peak at 50% to 60% of Gearman’s features
- Gearman is production ready, but the postgresql driver less so because of fewer test cases and Brian’s lesser familiarity
- setup ntp and use Gearman coalescence for redundant cron servers
- can inspect queue
- agnostic to backend
- 99designs.com looking at this, same use case as original developer
- I still think that if you already have a database app, adding a status column gets you a lot of Gearman functionality without one more moving part.

MariaDB BOF
Monty Widenius, MontyProgram AB

- Monty mentioned that the latest release of MariaDB 5.3-beta has faster replication from group commit and performance improvements on the master, which also help the slave. Also subqueries and joins work much better.
- Monty talked about his Aria storage engine, which is a replacement for MyISAM that has both transaction and non-transaction modes. It’s intended for users who want the space savings of MyISAM. Over time it may compete with InnoDB.
- Monty’s responsibility is to convince Percona to merge into 1 source base sometime
- it’s estimated that although Oracle still has the InnoDB team, they may only have 1 general MySQL server programmer left.
- he explained that MontyProgram developers work 50% on feature requests from end-users, and 50% Open Source-related. So paid requests for 1 week of work really need to cover 2 weeks of developer time for that model to work. Typically a medium-sized change is roughly $12,000 and includes development, testing and documentation.
- Zmanda got FLUSH TABLES WITH READ LOCK NO CHECKPOINT for a beer, though. :) Monty was able to find a code path that could be locked to prevent Aria and MyISAM from writing anything during the backup.
- I sponsored WL#232 for USD$100 to add a SHUTDOWN statement to MySQL
- Monty explained that MERGE tables may be a better choice than MySQL partitions for logging applications.
- attendees from MontyProgram, SkySQL, Percona, DeNA

AskMonty: MySQL “Wishlist” Session from an online travel agency
MySQL Manual 5.5: FLUSH Syntax

Perl Lightning Talks
Hosted by Geoff Avery

- a talk on why arrogant community members telling others that “they need a thick skin” is unhelpful
- a talk by a young Perl community member on getting commit access, and how others can get the spirit and contribute
- Larry did several talks, mostly encouraging backporting Perl6 features to Perl5 it seemed, perhaps as a replacement to going Moose
- nice song on the importance of public libraries, which face shutdown due to economic budgeting problems in Australia and USA
- nice comedy juggling act comparing programming languages. Perl6 was omitted as “nothing has been updated in 5 or 6 years”, prompting Larry to say that he was happy he has a thick skin. See above. :)
- afterward, I talked to a booking.com rep about why a European company needed to actively recruit in USA and world-wide. He said that European developers are happy working where they are now, and it’s easier to recruit in places with mobile workforces like the USA. He would like to hire a couple developers per week to meet their development schedule.

]]> http://www.jebriggs.com/blog/2011/07/oscon-2011-portland/feed/ 0 DataStax Apache Cassandra Training in SF http://www.jebriggs.com/blog/2011/07/datastax-apache-cassandra-training-in-sf/ http://www.jebriggs.com/blog/2011/07/datastax-apache-cassandra-training-in-sf/#comments Wed, 13 Jul 2011 03:20:35 +0000 Administrator http://www.jebriggs.com/blog/?p=3576 Continue reading →]]> I went to the DataStax training for Apache Cassandra today at the Hilton San Francisco Financial District Hotel.

Ben Coverston, Director of Operations at DataStax, did a great job presenting some very technical material.

Topics included data modeling, client programs, server configuration, replication, tuning, performance monitoring and operations.

Labs were done using the provided USB drive containing a VM (system requirements: 2 GB RAM and 12 GB disk free) of Cassandra and other tools like JConsole running on Debian Linux.

(I should have used VirtualBox, which is free (GPL) and doesn’t require registration, instead of the VMware Fusion trial, which has restrictive licensing and only works for a month.

To use VirtualBox4, create a new VM and instead of creating new disk space, attach “Riptano Training.vmdk”. You may also need to rename the parent directory from “datastax_training_dist client.vmware” to “datastax_training_dist client” for the navigator to open it.)

About 45 people attended, a new class record.

If you get a chance to take Ben’s class, I highly recommend it – send your whole team.

To get the most out of the class, I recommend reading about Cassandra data modeling and trying the various tools (cassandra-cli, cql, nodetool, JConsole, JNA, JMX, Opscenter, etc.) and hanging out on the #cassandra channel on irc.freenode.net. beforehand.

The hotel was a good choice, located near the financial district and Chinatown, with reliable WiFi, conveniently organized meeting space, and a very good lunch.

I had an interesting discussion with some veteran DBAs about the popularity of NoSQL solutions on the East Coast. The most popular request was for – get this – SQL access via Hive or cql. :)

Hilton San Francisco Financial District
750 Kearny Street, San Francisco, California, United States 94108

@bcoverston
Ben Coverston – The Apache Cassandra Project
sfgate.com: DataStax Training for Apache Cassandra – SF
Apache Cassandra training for $30? from @spyced of Datastax? No way!
Let’s play with Cassandra… (Part 1/3)

Some notes about 0.8 and previous releases:

- Cassandra Clients

  • cassandra-cli
  • Hector
  • CQL
  • pycassa
  • Perl and PHP clients are not as reliable because of poor Thrift bindings, should be rewritten to use cql binary protocol
  • Java

- 3 common cassandra-cli commands are:

  1. show
  2. describe
  3. list

- Counters

  • If you use counters, then don’t delete SStables. decommission and bootstrap.
  • TimeoutException can cause overcount with counters.
  • No secondary indexes on counters.
  • Repair does not fix counters.

- nodetool

  • nodetool flush
  • nodetool drain

- Repair

  • Send schema changes 5 seconds apart to avoid wedging other nodes. To fix a wedge, delete old schema (leave data alone) and restart.
  • mlockall is enabled by default if JNA is found on the classpath (must be installed separately because of licensing). Also a good idea to turn swap off.
  • nodetool repair (or rsync all files then nodetool repair)
  • read at consistency level of all (ghetto repair)
  • adding a new node can be more efficient than removing a node then adding it.

- Compaction

  • rows over 64 MB need 2-pass compaction
  • minor compaction can purge tombstones > 0.6.6
  • avoid major compaction, but keep an eye on SStables-per-read histogram

- Monitoring

  • watch -d -n 5 nodetool info or cfstats or tpstats
  • JConsole and JMX
  • storageproxy latency, reads, writes
  • oadcdb storageproxy
]]> http://www.jebriggs.com/blog/2011/07/datastax-apache-cassandra-training-in-sf/feed/ 0 Cassandra SF 2011, San Francisco http://www.jebriggs.com/blog/2011/07/cassandra-sf-2011-san-francisco/ http://www.jebriggs.com/blog/2011/07/cassandra-sf-2011-san-francisco/#comments Tue, 12 Jul 2011 07:02:17 +0000 Administrator http://www.jebriggs.com/blog/?p=3571 Continue reading →]]> I went to the Cassandra SF 2011 conference today, at the UCSF Mission Bay Conference Center.

It was hosted by Datastax.com, where many of the Cassandra developers work.

Here’s an overview of some of the talks I attended.

Keynote
Jonathan Ellis, DataStax CTO and Apache Project Chair of Cassandra

Talked about the latest features and use cases for Cassandra and Brisk, Hadoop powered by Cassandra.

Cassandra Use Cases at Twitter
Chris Goffinet, Twitter (ex-Yahoo! Performance Engineer)

Ext4-writeback-512k best, xfs also good
Noop best for reads p90, p99, avg, max
All schedulers same for writes because of journal.
Multi-threaded compaction all the time:
CASSANDRA-2191 Multithread across compaction buckets
CASSANDRA-2156 Compaction Throttling
Compression reduces data by 7x
memcached most efficient space-wise
99th percentile went from 200-800 ms -> 2.5 ms
Cuckoo replaces Ganglia, updates every 60s
140 MB/s writes, 70 MB/s reads
500 GB new data / hour
x3 replication factor
Slab allocator fixed size allocs of 2 MB
CASSANDRA-2252 off-heap memtables
Reduce GC time by 1000x
Don’t use TTL, drop SSTables after N days
CASSANDRA-2819 Split rpc timeout for read and write ops
mlockall recommended with JNA jar to fix JVM allocation problems
git, Hudson/Jenkins, murder (Twitter Open Source), Bittorrent to deploy to 100s of nodes in 20s
Cassie is a light-weight Cassandra Client
Based on finagle
Yahoo! Cloud Serving Benchmark (YCSB)-based test framework
Generate 1,200 reports per build, takes 48 hours to run
Understand your stack
Measure everything
Invest in your storage technology (right components, understand failure modes, etc.)
Automate
Expect everything to fail
Linux page cache did not have good read performance since not LRU and fixed 4K, thus memcached esp. with narrow rows
Need to evaluate alternative JVMs like JRockit and Azul for GC
Apache Mesos cluster manager and virtualization project.
Twitter Ops improvements – 6 people for 6 months.
Bittorrent originally used to update 1000s of apache servers, create 1 seed and have it then do neighbor. Can update 300-node Cassandra cluster in 20s. :)

(An attendee said speed of deployment is important more for recovering after a bad push than anything else in his case. It’s also helpful for software developers writing deployment scripts, and also when you have network architecture issues like overcommitment or latency-sensitive production traffic.)

Eric Onnen, Urban Airship

Committer on HBase, Cassandra, Zookeeper projects
Hosting for mobile services
United API for services across platforms
Common SLA is 10,000 or 120,000 messages / s
Over 160 million app installs on 80 million devices
Half of requests are device check-ins (ie. when an app foregrounds)
Now 25 million per day
Used early MongoDB, not so good
EC2 does not offer enough RAM for caching all their data
Cassandra starting in summer 2010 for android
6 EC2 himems, 1000 reads/s, 750 writes/s
30 GB node
Rolling upgrades
Column TTLs
Nice community
Good with EC2
No SPOF
Ability to alter CLs on a per-operation basis like checkins vs schema changes
Problems
Know your data model
Creating indexes later is a PITA
Wide rows are bad – IO, thrift, count
JSON better than packed binaries
Careful with thrift in the stack
Read timeout vs connection refused, GC vs failure, library assumptions and exception handling
Verify client
Ops
Ensure dynamic switch is enabled
IO
Avoid EBS except for snapshot backups or use S3
Stripe epehemerals, not EBS volumes
Use common init scripts
Avoid smaller instances – more steal
Bare metal will rock your world (compared to VMs or EC2)
Uselargepages
Look at thread dumps
Java TDA (Thread Dump Analyzer) – love it, only good thing to come out of SAP :)
Eclipse Memory Analyzer (MAT)
Perform major compactions daily in lull periods
Monitor JMX religiously
Cassandra exposes a lot of performance counters, use them
Still too much time looking at GC
Upgrading from 0.6 to 0.7 was rocky, had to do a read repair surprise
Need CQL or Pig to offload ad hoc reporting from 12 developers to 11 administrative staff :)
Sample end-user ad hoc query: How many android devices are using this version of software title “whatever”?

Introduction to Brisk
Jake Luciani, DataStax

Great talk on Brisk.

Highly Available DNS and Request Routing Using Apache Cassandra
David Strauss, Founder/CTO, Pantheon Systems

We accept HTTP requests and forward them to app servers (mostly for hosted Drupal)
Reconfiguring Varnish, nginx, and proxies not fun, disruptive
DNS has a nice model
And client tools like dig
nginx did not do round-robin
Node.js ok using DNS, HTTP proxy and Fugue modules, 99 lines of code
Why another? DNS has no ring replication, defined master
MS AD and ApacheDS too heavy
DNS needs to withstand DoS
Github cassandra-dns
Twisted doesn’t leak memory, but clients may
TxSQL is another way, and is non-blocking
Other ideas, point client resolvers at haproxy

Replacing Datacenter Oracle with global Apache Cassandra on AWS
Adrian Cockcraft, Netflix

DC capacity is 4 Billion requests per month
exponential growth to 30 billion requests per month
10 minutes to create a cluster in SG, would take 9 months to build a DC there
Full backup – daily cron snapshot
Incremental -
Continuous – commit log to EBS
Archive – S3, encrypt, region, separate account, different cloud
Guide to NoSQL, redux. Mark Atwood
Practicalcloudcomputing.com
Chaos monkey runs 9 am to 3 pm, need to opt-out
For BI take backup after midnite, but restore and chop off extra seconds after 00:00:00

Real World Capacity Planning Cassandra on Blades and Big Iron
Ed Capriolo, Media6degrees

Compacting tombstones at nite (batch time) can help with day-time traffic (real-time)
nodetool
JBOD not commonly used, but you could, like Hadoop or Riak
Puppet
Cacti graphs to acquire JMX data
Uses all SCSI/SAS drives
Likes blades, though only 1 or 2 drives per blade
Centos 5.5 ext4
Writing “High Performance Cassandra Cookbook”

Lightning Talks

Mike Weir, Mktg VP, Datastax

Compaction performance improvements C-1608

Ankit Shah, Principal Software Engineer, Verisign Authentication Group, Symantec

DB is 1.2 GB or so (small) but want HA
Can be read-intensive, some writes
Want good read performance, failover, replication
Secure gossip over IPSec or similar
Doesn’t like existing AWS database product limitations
So Cassandra is it

The Auto-Clustering Brisk AMI
Joaquin Casares, Datastax

Also RAIDs ephemeral volumes

Twitter

Largely done programming:

CASSANDRA-47 SSTable compression
CASSANDRA-674 New SSTable Format
CASSANDRA-16 Memory efficient compactions
CASSANDRA-2319 Promote row index
CASSANDRA-293 remove_key_range operation and
CASSANDRA-494 add remove_slice to the api
CASSANDRA-808 Need a way to skip corrupted data in SSTables
CASSANDRA-2398 Type specific compression

Mike Bulman, Opscenter Team, Datastax

Uses Cassandra itself for logging
Ring view graph
Add a node to ring tool

Cassandra Anti-Patterns
Matt Dennis, Datastax

Use Sun JVM u22 or newer, not OpenJDK or Blackdown
6-8 GB for java heap, not above 16 GB
Separate spindles for. data and commit log unless EBS
No EBS, use ephemeral
Don’t read before write if possible
Use bulk loader in 0.8, Java SStable program
Supercolumns need to be rewritten.

SlideShare: Cassandra Query Language (CQL)
IBM and SAP Open Source their JVM Diagnostics Tools
Migrating Netflix from Datacenter Oracle to Global Cassandra
Urban Airship raises $15 million (Nov., 2011)

]]> http://www.jebriggs.com/blog/2011/07/cassandra-sf-2011-san-francisco/feed/ 0 O’Reilly Velocity Conference Web Performance and Operations Conference 2011, Santa Clara http://www.jebriggs.com/blog/2011/06/oreilly-velocity-conference-2011/ http://www.jebriggs.com/blog/2011/06/oreilly-velocity-conference-2011/#comments Sun, 26 Jun 2011 23:22:57 +0000 Administrator http://www.jebriggs.com/blog/?p=3502 Continue reading →]]> I was too busy to make it to the Velocity Conference this year from June 14-16, but here’s some talk links.

YouTube: O’Reilly Velocity 2011 Playlist
YouTube: Velocity 2011 Search Results

YouTube: O’Reilly Velocity 2010 Playlist
Overclocking SSL at Google (2010)

]]> http://www.jebriggs.com/blog/2011/06/oreilly-velocity-conference-2011/feed/ 0 MySQL 5.5 News from Oracle OpenWorld http://www.jebriggs.com/blog/2010/10/mysql-5-5-news-from-oracle-openworld/ http://www.jebriggs.com/blog/2010/10/mysql-5-5-news-from-oracle-openworld/#comments Sun, 03 Oct 2010 06:18:08 +0000 Administrator http://www.jebriggs.com/blog/?p=2390 Continue reading →]]> I received an interesting email about MySQL news from Oracle.

It was badly formatted in my email client, so I have reformatted it below.

Sender: “Oracle MySQL”
Subject: News from MySQL Sunday at Oracle OpenWorld
Date: Sat, 2 Oct 2010 11:17:17 -0700

Did you miss the first MySQL Sunday at Oracle OpenWorld?

Here’s your opportunity to catch up on the exciting news and developments. In his keynote, Edward Screven, Oracle’s Chief Corporate Architect, underlined Oracle’s investment in MySQL and demonstrated how Oracle is making MySQL better every day.

As a testament to that investment, Edward announced the immediate availability of the release candidate version of MySQL 5.5, delivering major enhancements in performance & scalability, availability and usability.

Watch Edward’s presentation

With MySQL 5.5 You Get:

  1. Improved performance and scalability: Benchmarks showed up to 1,561% performance gains for Read/Write operations on Windows, and up to 364% performance gain in Read/Write operations on Linux!
  2. Higher availability: New semi-synchronous replication and replication Heart Beat improve the reliability of data and the speed of failover for continuous application availability.
  3. Improved usability: Enhanced index and table partitioning, signal/re-signal support and diagnostics further improve the usability of MySQL 5.5.

Learn More and Get Going with MySQL 5.5:

Contact your MySQL sales representative for more information online or at the number for your region below:

  • USA/Canada – Toll Free: +1-866-221-0634 USA. From abroad: +1-208-338-8100
  • Additional phone numbers for EMEA, Latin America and APAC

]]> http://www.jebriggs.com/blog/2010/10/mysql-5-5-news-from-oracle-openworld/feed/ 0 Defcon 18, Las Vegas http://www.jebriggs.com/blog/2010/08/defcon-18-las-vegas/ http://www.jebriggs.com/blog/2010/08/defcon-18-las-vegas/#comments Mon, 02 Aug 2010 02:13:15 +0000 Administrator http://www.jebriggs.com/blog/?p=2200 Continue reading →]]> DEF CON 18 was held once again in Las Vegas at the Riviera Convention Center.

There were a handful of talks on the subjects of DNS and IPv6.

The hacker Jeopardy session was a lot of fun. I think the audience got more correct answers than the panel. I was impressed with the software somebody wrote to show the game categories – very convincing. Afterward, the EFF had an interesting fundraiser (your photo beside a “model”.)

The weather was hot but clear. The McDonald’s across the street is open 24 hours and has free WiFi.

I walked over to the Fashion Show Mall (about 1 mile.) It has a variety of restaurants on different levels, including a Maggiano’s, the Capital Grille, and a gourmet burger stand.

theregister.co.uk: Defcon speaker calls IPv6 a ‘security nightmare’

]]> http://www.jebriggs.com/blog/2010/08/defcon-18-las-vegas/feed/ 0 O’Reilly Open Source Conference 2010, Portland http://www.jebriggs.com/blog/2010/07/oscon-conference-2010-portland/ http://www.jebriggs.com/blog/2010/07/oscon-conference-2010-portland/#comments Fri, 23 Jul 2010 23:22:51 +0000 Administrator http://www.jebriggs.com/blog/?p=2140 Continue reading →]]> Once again, the O’Reilly Open Source Conference (OSCON) was held in Portland, Oregon.

It was a good conference, and we had beautiful weather all week long.

Executive Summary

The themes promoted by the conference organizers were Cloud Computing, NoSQL, Emerging Languages (Scala, Erlang, Parrot, Go) and Android phone development.

The @oscon twitter channel was heavily used to coordinate amongst organizers and attendees. I used the TwiXtreme twitter client program on my BlackBerry.

Plug Computers were very popular in the Expo area. They are 5 watt ARM-based computers running Debian Linux that fit into a power brick-sized case and cost $99 to $129 depending on features. The Marvell booth had a few models on display, from GlobalScale (GuruPlug) and Ionics. High-end models have dual gigabit NICs, multiple USB ports, a WiFi access point and other expansion ports.

There was also continuing buzz regarding Facebook’s Flashcache SSD module (GPL v2) for linux, and also ZFS snapshots.

Tutorials

I went to the Gearman Cookbook tutorial, the first half of the Chef tutorial and some of the Cloud Summit talks.

The Gearman Cookbook tutorial was excellent. After a detailed overview of the Gearman architecture and implementations in Perl and C, a number of use cases were explored in detail, including before and after code samples. The talk was both easy to listen to as an overall survey, as well as providing immediately useful info for those wanting to deploy it.

The Chef tutorial was very detailed – too much so perhaps. I went to the first half only, since I am not planning to implement Chef soon (I use PXE and anaconda/kickstart with CentOS), and did not need that level of detail at this time. cfengine, puppet and chef are ops tools for configuring servers. Chef uses Ruby data structures for its configuration files, and has include files and other useful syntax. Basically, users can “code” server configuration, as if they were traditional apps.

I went to some of the Cloud Summit talks and BOFs, but found that anybody who has done a simple project using EC2 knew as much or more than the speakers, some I would call blowhards.

Marten Mickos, president of Eucalyptus, is refreshing in that he is always clear about being in it for the money, while also promoting Open Source.

Sessions

Some of the most memorable sessions to me were:

Introduction to MongoDB, Kristina Chodorow (MongoDB)

Kristina is the maintainer of the Perl and PHP drivers for MongoDB. She gave an overview of MongoDB, a NoSQL document store, and its command-line interface, which uses JavaScript.

Some day she will release a sharding tool for MongoDB.

Scaling SourceForge with MongoDB, Nosh Petigara (10gen), Rick Copeland (SourceForge.net / GeekNet)

Nosh and Rick gave an excellent review of incorporating MongoDB into the SourceForge site.

- SF query load is mostly read-only
- ops team benchmarked a few NoSQL candidates, and MongoDB won on performance
- original MySQL servers had 64 GB RAM. After migration to MongoDB, same server machines but only 8 GB RAM
- backup dumps are verified to be bitwise the same as masters
- have to be careful not to dump all documents in your database to the network or it will max out switches
- SF relies on first-class data centers and replication slaves, less worried about MongoDB mmap (not crash-safe)
- I personally looked at their performance numbers and site graphs (on an iPad), and the end result was impressive.

Perl Lightning Talks

As always, the Perl Lightning Talks are a highpoint of the conference.

The “cartoon” of Vincent Pit’s remarkable CPAN module(VPIT) contributions was both informative and hilarious. Vincent is a French Ph.D. candidate in advanced geometry.

Cloud BOF (3 Hours)

The Cloud BOF was disorganized, starting 30 minutes late and for some reason was subdivided into 4 audience groups. Startups and vendors trying to make a cloud sales push led the BOF, including cloud and DNS service providers.

The Health Regulations subgroup came up with a couple ways to make the Cloud palatable to regulators by using encryption on all data due to the multi-tenancy issues with sharing public VMs.

I was in the NoSQL group, which discussed general issues and particular successes. Memcached was the clearest winner, while some people also had success with MongoDB and Redis.

My neighbor was an engineer at Postrank.com. He said that they were happy with HAProxy, but much less happy with the unpredictable IO available when running MySQL on EC2. He also said to carefully look at storage volumes available to your instance, as one is a useful tmpfs. They use AuthSMTP to get around EC2 being generally blacklisted for outbound email.

Database BOFs

MySQL BOF

The MySQL AB engineering staff has left Oracle. Monty Program AB (21 staff) has the core developers, and Percona Inc. (32 staff) has the consultants. Oracle still has some of the InnoDB programmers.

The business plan for Monty Program AB is 60% commercially-sponsored MySQL development, and 40% community-request development. Monty would like commercial users of MySQL to sponsor patches that would benefit them.

Mark mentioned that using Nehalem instructions for CRC were much faster, and that Facebook was using partitions for truncating tables instead of doing multi-record deletes. (See his blog for more details.)

One person mentioned using a commercial backup tool, R1Soft, that inserts a linux kernel module to allow filesystem snapshots. He said to carefully test backup and restore in your environment, especially for filesystems greater than 1 TB which may exceed certain block counter limits. Peter said that some of his clients had used it with varying success.

It worked for him in his environment, and the file browser allows selective file restore (he uses it to restore by priority where a system runs multiple applications.) It starts at $299 for the Standard Edition, and also has MySQL Add-on and Enterprise Editions.

PostgreSQL BOF

The PostgreSQL BOF talked about 30 or so changes that went into version 9.

One of the most exciting new features is a native replication feature, called streaming replication (block-based.) The advantage over Slony-I replication is that Slony-I is trigger-based, so has a variety of issues included inability to replicate DDL commands.

Some of the developers mimed replication events, which was rather amusing to watch. Yes, it was taped.

PostgreSQL is released under the PostgreSQL Licence, which is BSDish.

Peter Zaitsev, co-founder of Percona, organized 3 BOFs, including XtraDB, XtraBackup, Maatkit, Percona Server, Sphinx Search and Running Databases on Flash Storage.

Sphinx Search BOF

Andrew Aksyonoff, the original programmer of Sphinx Search (GPL v2), couldn’t make it to OSCON (the good excuse was that he was busy coding), so Richard Kelm (Sphinx sales/customer support honcho) and Peter filled in (Percona is a business partner with Sphinx, and many of Percona’s clients use it.)

Some of the attendees were existing users, like myself, and some from HP and other companies were looking for a large-scale search solution or alternative to Lucene.

Monty mentioned that the latest MySQL 5.1 should be used, as there have been a number of performance and reliability improvements. Full-text search is supposed to be 10x faster than 5.0, and replication is nearly bug-free by now.

Sphinx Search now has real-time index updates in version 1.1.0 beta. Another very nice feature is SQL+FS indexing.

Here is the full Sphinx 1.1.0 changelog.

Running Databases on Flash Storage BOF

The Running Databases on Flash Storage BOF had a combination of MySQL and Postgres users who have tested or used most of the SSD products: FusionIO, violin, Intel, OCZ, etc. Everybody was happy with SSD IOPS performance, but less so with cost and metadata RAM requirements with the add-in boards (FusionIO may require 4 GB RAM for metadata.)

Peter said that 20% to 30% of his clients are already using SSD – across the spectrum of vendors and models. Some are also trying “massive RAM” solutions, like Cisco servers with 384 GB RAM.

Some users had 1+ TB Postgres databases with very thorny backup and mgmt. issues. One solution was to start a snapshot, but not do the copy operation.

Expo Notes

I had an enjoyable talk with Austin Hook, who has operated the OpenBSD Store for many years. He lives near Calgary, the center of OpenBSD/OpenSSH/PF development. He mentioned that some perennial financial contributors had stopped because of the recession, so here’s the donations link.

I also talked to some reps from a Brazilian outsourcing firm, ActMinds. They currently have 400 employees across Brazil and a sales office in Philadelphia. Brazil is only 2 hours ahead of EST. They said the minimum project size is 2 developers and developer turnover a low 5%/annum. Their pricing is $35 to $45/hour.

And I had fun handling the plug computers on display at the Marvell booth. The Ionics boards are amazingly densely populated.

Discussions

I had the opportunity to talk to a long-time Portland resident who works as a computer consultant. He said that the Portland economy is not doing great, and really hasn’t done well since old-growth logging was stopped after 90% of the forests were cleared. And although hundreds of miles of fiber optic has been laid downtown, it’s not available for residential use. However, the Beaverton area does have ubiquitous FTTH.

I also talked to somebody who attended the Emerging Languages talks. He’s working on his M.Sc. in Computer Science, so found those talks fascinating.

Twitter Humor

There were some humorous tweets:

- “my MongoDB and CouchDB mugs are fighting each other.”
- “I got one MongoDB mug, but need two to safely store coffee.”

Notes

Note to self: skip the nightly parties unless you have a date. The bars are too loud to talk to anybody.

Note to the O’Reilly conference organizers: use meetup.com for the BOFs like ApacheCon does. The average audience was about 10 people, and with meetup it would be 4x that.

OSCON 2010 Slides
Tim Bray: Desperate Perl Hacker
Youtube: OSCON 2010 videos
blip.tv: OSCON2010 videos
wikipedia: Plug Computer
Jeremy Zawodny: MongoDB Early Impressions

]]> http://www.jebriggs.com/blog/2010/07/oscon-conference-2010-portland/feed/ 1 MySQL Conference 2010 http://www.jebriggs.com/blog/2010/04/mysql-conference-2010/ http://www.jebriggs.com/blog/2010/04/mysql-conference-2010/#comments Mon, 12 Apr 2010 18:58:43 +0000 Administrator http://www.jebriggs.com/blog/?p=1774 Continue reading →]]> The MySQL Conference was this week in Santa Clara. It was a well-organized and educational event with everybody involved in the MySQL community showing up once again.

Executive Summary

The highlights were:

  • after 2 years of effort, the performance schema foundation is available as a 5.5.x patch. With another year of effort, it could be useful.
  • the various community forks (Percona/XtraDB, MariaDB, OurDelta) will merge in the next 3 months into a maintenance fork by Monty Program, since MP has the most original MySQL developers.
  • the various MySQL vendors are soldiering along, all releasing new, improved versions of their hw and sw products this year.
  • The largest independent MySQL-centric consulting companies are Percona with 32 staff, and Monty Program 40 with staff, with a target of 50 employees.
  • the MySQL source code will have to be modified to make MySQL fast enough to keep up with Fusion IO SSD devices. Currently, better than SSD performance can be gained by installing enough RAM to fit the entire database in buffer pool.
  • Drizzle development is going nicely, but note that it’s not backward compatible with MySQL. Drizzle is a 64-bit only fork of MySQL with emphasis on community code development, increasing performance and maintainability through a plug-in architecture and strict code cleanliness.

Monday Morning Tutorial

Using Partitioning in MySQL 5.1 and 5.5 with Giuseppe Maxia (Oracle)

- available in MySQL 5.1 and later only
- TO_DAYS and YEAR() special and recommended as they can prune partitions from lookups.
- when using TO_DAYS() as a partitoning function, the first partition matters. Use value less than zero for first partition to create NULL partition to double performance as a bug workaround.
- consider lock before inserting for all table types
- for performance, consider non-partitioned on masters, partitions on slaves.
- or different partition types

He also gave a nice tutorial on mysql sandboxes script.

Partition Limitations:

- cannot mix table types
- cannot make read-only

I talked to some advanced users, and none have found a practical use for partitions in their environment that was better than using regular table types for logging type applications.

This is due to the fact that partitions do not increase fault-tolerance, often don’t benchmark any faster, and have little in the way of administrative mgmt. support after partition creation.

Partitions can increase performance in applications where the index serves to stripe operations, but most people are just using dates for logging, with no practical benefit, as most operations fall into the current date partition.

Slides

Monday Afternoon Tutorial

Talked to Arjen Lenz and a friend at lunch.

- OpenQuery is suitable for affordable, long-term contract databae admin, not firefighting
- former partition tester and bugfixer
- replication bug with TCP errors, nagios plugin should compare both replication lag seconds and log position
- need SSL or heartbeat to detect/fix

memcached

- set all clients to same values
- use JSON or YAML, not Storable or Pickl

Tuesday

Performance Schema with Peter Gulutzan

- coded by Alff, but not GA yet
- PERFORMANCE_SCHEMA database optionally populated with events (mutex, lock, io) timing and count info
- allows simple SQL reporting of performance

EXPLAIN Demystified with Baron Schwartz (Percona Inc.)

- perennial nice EXPLAIN overview
- nice example of using mysql command prompt as pipeline for non-trivial processing

Introduction to InnoDB Monitoring System and Resource & Performance Tuning with Jimmy Yang

An Overview of Flash Storage for Databases with Vadim Tkach, Percona Inc.

- nice talk with useful performance graphs

Linux Performance Tuning and Stabilization Tips with Yoshinori Matsunobu, Sun Microsystems

- nice talk with detailed slide examples – he’s a hard worker
- he’s a fan of xfs, so some info not always useful for ext3. ie. deadline scheduler may be better on xfs, but it feels the same to me as cfq on ext3.

Wednesday

More Mastering the Art of Indexing with Yoshinori Matsunobu, Sun Microsystems

- second-part continuation of his talk from last year (!) Were you there?
- his understanding of the space requirements of blobs in Innodb is different than Peter Zaitsev’s.

Faster Than Alter – Less Downtime with Chris Schneider (Ning.com)

- Hipster presentation on doing practical DBA tasks
- likes doing dump and restore on Innodb tables, 30% faster afterwards on his tables.

InnoDB Architecture and Performance Optimization with Peter Zaitsev, Percona Inc.

- perennial comprehensive overview of Innodb
- talked about differences between Antelope and Barracuda file formats
lwn.net: A look at the MySQL forks

BOFs

O’Reilly failed to use meetup.com to promote the BOFs once again at this conference, so turnout was light to moderate as in past years.

Sphinx BOF hosted by Andrew Aksyonoff

I’ve been familiar with SphinxSearch for years and am a production user, so the general audience discussion was not interesting to me.

However, I had a chance to talk to Andrew about my take on the October Apache (Search!) Conference last October and suggested a few things:

  • explain collections on the Sphinx homepage, since many users insist on this feature. The question, of course, is what does the term ‘collections’ mean to various people?
  • make it possible for a non-technical end-user (like a marketing asst.) to highlight 10 items for feature on the first page of results
  • Microsoft is EOLing FAST for linux users, so think about promotion to that segment, who is considering migration to Lucene mainly – because Lucene is free, and migration is the same cost to any other product.
  • look at the myriad “value-added features” of commercial search engines, mostly related to adserver integration, and decide what can be supported.

MariaDB BOF hosted by Monty

Not much talk about MariaDB, but lots of drinking! (See Monty’s keynote for more detailed info.)

Conference Wrapup

Overall, another good MySQL conference. The organizers restored balance to the presentations, with a fair number of independent consultants and end-users doing talks. (Though I miss the awesome Percona Performance Conference from last year.)

BOFs should be promoted on meetup.com to double participation.

There should be a room with exotic hardware to demonstrate high performance MySQL and MySQL Cluster configurations – SANs, Infiniband, failover, etc.

The lunch food was quite good on all days, as noted by several people. (Important because the suburban venue is not within walking distance to outside restaurants.)

]]> http://www.jebriggs.com/blog/2010/04/mysql-conference-2010/feed/ 0 ApacheCon 2009 Oakland http://www.jebriggs.com/blog/2009/11/apachecon-2009-oakland/ http://www.jebriggs.com/blog/2009/11/apachecon-2009-oakland/#comments Fri, 06 Nov 2009 21:52:43 +0000 Administrator http://www.jebriggs.com/blog/?p=1225 Continue reading →]]> I went to ApacheCon 2009 in Oakland. Why Oakland? The ASF was founded here 10 years ago.

Executive Summary

Most of the attendees that I talked to were primarily interested in search technologies, or were Apache project comitters. The search users were already using either Lucene and Solr, or were using commercial software and evaluating Lucene and Solr.

Also a lot of interest in Hadoop, Zookeeper and NoSQL projects.

I added a wikipedia NoSQL project features table after the NoSQL BoF.

The conference was very well-organized, with tutorials, BoFs, a BarCamp, and sessions. Meetup.com was used to generate the highest BoF turnout that I’ve ever seen – close to 100 at the Lucene and Hadoop BoFs. (O’Reilly Conferences can learn from that.)

The Oakland Convention Center was a good venue for this conference, though the attached Oakland Marriott hotel is $$$$ and fond of surcharges, like $33/day for parking, $5 draught beer and $3.75 for a bottle of water in-room.

The keynotes and one track per day were recorded and are available for $99 at Linux Pro Magazine Streaming.

StoneCircle Productions was the conference organizer.

Conference Notes

Monday

Although I live in San Jose, Oakland is far enough away that I’ve never been there. Oakland has a compact downtown full of historical-era buildings, and Alameda is also nice, but things get less pretty at night.

I went to the Lucene tutorial on Monday.

Lunch Conversations

- awesome views of Bay Area past Golden Gate bridge from 21st floor
- FAST pretty good indexing and search solution, but bought by Microsoft recently (going to continue linux support or not?)
- FAST has FQL (users pronounce it fecal) query language :)
- 150 FAST servers replaced by 40 lucene servers by 1 company
- FAST4 to FAST5 upgrade tough, similar to port to say lucene, forced upgrades for support
- linguistics is 60% of value of Fast according to Monster, 13 languages supported
- “bad stems” can be a nightmare
- SOLR gives you 90% of what you would need to program in java, built on top of Lucene
- Open Source search is not really about price, but about control and flexibility

Monday Afternoon – Lucene Tutorial

- user-assigned document id not mandatory, but great idea for many reasons, including after an index-rebuild
- lucene-assigned id only valid for that snapshot (life of score doc)
- parameter to keep or delete old index directory
- StringBuilder is more efficient than strcat
- populating title column is a good idea
- results boosting handy for ecommerce, specials, etc.
- LUKE – handy tool for index statistics, etc.
- Searcher class, snapshot in time, won’t see new merges
- contrib/ has more analyzers
- snowball stemmers
- use 1 tokenizer and 0 or more token filters
- precision-recall curve ??
- n-grams and shingles (“the president”, “United states”)
- pre-2.9 lucene, numbers and dates really strings
- 2.9 NumericField builds tri structure, help optimize range queries
- SOLR analysis tool apache-solr
- relevance feedback with MoreLikeThis

Monday BoFs

Couchdb

- “ground computing”
- “offline by default”
- now an ubuntu service
- mozilla raindrop to combine chat client msgs
- lockless
- append-only btree
- rsyncable since append-only, also replication
- checksums everywhere
- windows not first class yet, mozilla improving it

@mozilla

- browsercouch
- don’t like sql
- brasstacks test tool storage
- store now, index later
- replicate to handle large indexing load
- testbot ci

Marklogic

- commercial
- xml-centric
- great for articles, books
- transactional
- search-centric
- structure-aware
- schema-free
- xquery-driven
- extremely fast, largest 200 TB xml, 166 on hosts
- clustered
- database server
- 180 clients, 150 employees
- markmail.org demo contains 42 million email messages, very impressive performance with 5 views in almost realtime. Search is distributed across 160 nodes.

JCR in 15 minutes

- Bertrand Del
- JCR is JackRabbit,
a fully conforming implementation of the Content Repository for Java Technology API (JCR). A content repository is a hierarchical content store with support for structured and unstructured content, full text search, versioning, transactions, observation, and more.
- the ultimate content store
- content repo, union of database and filesystem, best of both worlds
- full-text search combined with structured search

Solr Flair

- information forage
- “resume-driven design”

Lucene Numerics

- available in 1.4
- tune by modifying precisionStep

HBASE

One bewildered attendee wished for a NoSQL product matrix, so I added that to the wikipedia NoSQL page.

Wednesday Sessions

Becoming a Pig Developer, Alan Gates

- Apache Pig is a sub-project of Apache Hadoop.
- this talk was really how to use PIG as an end-user, not to become a Pig project developer

Apache Hadoop in the Cloud, Tom White

- general comments on using EC2 with Hadoop mostly

Practical HBase, Michael Stack

- Apache HBase is the Apache Hadoop database, similar to BigTable.
- HBASE usage

mod_jk / mod_proxy and others, Jean-Frederic Clere and 2 others

- mod_jk, mod_proxy, mod_serf and mod_cluster original topics
- mostly focused on mod_jk, mod_proxy and isapi_redirect
- good talk by 3 long-term project contributors
- jk is kind of Java-centric, with support for Apache JServ Protocol (AJP) only available in Java back-end servers for now, like Tomcat
- isapi_redirect is primary way to do redirects on Windows IIS
- survey of audience showed several mod_proxy users, maybe one intentional mod_jk user

Thursday Sessions

“Apache Lucene and Apache Solr Performance Tuning with Mark Miller” was packed, so moving along to a different room …

Scalable Internet Architectures, Theo Schlossnagle

- amazing and thought-provoking talk, also one of the most popular
- think about performance from network packet level to application level
- carp, vrrp, whackamole
- alterdns, neustar
- dynact
- anycast (shared IP), geoip (but need actually accurate database)
- activemq, rabbitmq instead of Spread
- “memcached is the worst thing that ever happened to our industry – it solves a problem, just not the original problem”

- many apps today are so poorly designed that network issues never become scalability concerns – ie. RoR applications :)
- max out at 500 requests per second across 40 boxes – RoR
- firebug and yslow have been fantastic at making front-end engineers aware of networking performance
- 10 gb nics suck
- instead of one big 20 Gbps loadbalancer, use anycast from core router to 5x 4 gpbs cheaper load balancers
- spiky load or DDoS – announce a /32 to separate load balancer, use symmetric return path

- jms, aqmp, spread

durable message queues

- activemq (java)
- openamq (c) – hard to use
- rabbitmq (erlang) – nice except in durable mode because erlang disk io blows

- most common protocol Stomp is awful and slow (hard to read 100k messages per second) and not binary, but lots of clients exist.

- activemq and stomp is a good start.
- rabbitmq and native connectors are better, but no perl client.

- PCI compliance requires a stateful firewall. Hard to do 1.5 million packets per second traffic for most medium-sized data centers, need to use a CDN to distribute static requests and distribute the packets somewhere else
- leaving trailing / off causes 302, doubles traffic
- Slides
- read/write ratio is 1 … likely IM or email?
- went over some networking details with Paul L. afterwards

Recent Developments in SSL and Browsers, Rick Andrews, Thawte

- 1.6 billion OCSP requests per day, need good infrastructure to support that
- intermediate CA allows root CA to be offline – chained hierarchy – SSLCertificateChainFile,
needs intermediate certificates before cross-certificates, some clients need in proper order
- EV hierarchy more complex. wanted new EV root, but older browsers don’t know about it.
- browser ubiquity problem with any new feature, hash or crypto algorithm
- logotypes – trademark and copyright issues with using other companies’ logos in a product
- Verisign does not have apache httpd committers, but should
- 1 attendee wanted to sign JavaScript files, but what does it mean if most sites link to 10 advertising and tracking scripts? what do you tell the user if 1 JS is not signed?

Subversion Meetup

Organizers didn’t show up, so spent 10 minutes talking to a handful of end-users about subversion gripes and moved along to …

Hadoop Meetup

Zookeeper

- zk is persistent to disk
- can run on one node, but 3 is minimum non-toy
- zk is popular in academia now for some reason
- avoid split-brain partitioning between 2 data centers – bad
- very recent merge to fix -368, not ready for production yet
- people using it for a message queue, perhaps more reliable than many other Open Source ones
- need 1 zk node for testing, but 3 zk nodes for non-trivial implementation

Scribe

- github
- 4x to 5x compression with lzo. similar disk bw improvement

A local owner of a gelato store handed out 6 free samples from a portable gelato freezer. :)

Friday Sessions

Building Intelligent Search Applications with the Lucene Ecosystem, Ted Dunnin

- some matrix math
- using his matrix math optimization, a perl program on 1 server was faster than Mahout running on a $250k cluster :)
- tdunning.blogspot.com

- the original LLR in NLP paper
“Accurate Methods for the Statistics of Surprise and Coincidence” check on citeseer
- Mahout project
tdunning [at] apache.org

Realtime Search, Jason Rutherglen

- many technical issues prevent Lucene from being able to do realtime search
- lots of patches done, lots to do
- audience member thanked author for great work so far

Closing Plenary: Brian Behlendorf on Open Source and Charity

Talked to Alex Karasulu a little after the final presentation. He’s a committer on the Apache Directory project. He suggested adding dbm to the NoSQL product matrix. Wants a MacBook Air with 8 GB RAM to run his Java apps. :)

Conference Schedule Grid

]]> http://www.jebriggs.com/blog/2009/11/apachecon-2009-oakland/feed/ 1