There was a good turnout, with about 30 outside attendees and about the same of ISC staff. 3 BIND developers from the Czech and Chinese registrars also attended, as well as Cricket Liu.

Here’s my notes.

DNS

- can learn what you need to know (50%) in a day, lifetime for the next 50%

BIND9

- monolithic program
- 10 years old
- different syntax than BIND8, 30% slower

User Story – Quick Refresh at AOL

- VMs spark up in 9.5 seconds, but DNS takes 15 minutes or more to propagate in their system
- goal is 5 minute updates across globe

BIND10

- sponsored by 10 TLDs, unlike BIND9 which was sponsored by Big Iron vendors
- TLDs represent registrars, users or citizens
- started April 1, 2009
- 5 year plan
- now on year3 – features and performance
- release every 6 weeks
- will be no slower than BIND9
- postfix processes model, not threads
- scalable across cores
- modular
- logging
- reporting
- REST API
- save cache object to memory or network, could be 10s to 100s of MB
- no query or response logging, use tcpdump
- look at DLZ

DNS Provider Issues

- Dealing with large zones
- Dealing with lots of zones
- resource estimates (for sizing new servers)
- high-availability installations and technologies
– anycasting
– cooperate better with firewalls and load balancers
– auto-clustering?
– quagga support?
- multi-master
- support for fully mastered slave
– akin to powerdns “super masters”

- some debate over how much routing is appropriate inside dns
- do it all or just be fast
- does it make sense to send acl info over port 53? needed for cluster setup

DNSSEC Issues

- Deployment and Maintenance procedures
- in-line signing
– aka “bump on the wire” 9.9.0
– hidden master sends signed zones to slaves
– aka “signer in the middle”
- registry vs. registrar
- look at DomainKeys optionality
- DLV keys (don’t need a signed parent)
- zone monitoring is crucial
- 4 HSM known, $80k for qty 1
- some firefox plugins check ad bit
- dnssec slowest on negative answer, nsec3 worst

The event photos are available here.

Thanks to Facebook for hosting this event.

Here is a partial list of problems with MySQL application accounts having SUPER privilege:

• allows server configuration updates: replication, system variables, logs, views and stored procedures.
• The init_connect variable is not executed for accounts with SUPER
• SET GLOBAL READ_ONLY=ON is not respected for accounts with SUPER privilege, for example, allowing direct updates of slaves.
• the reserved connection MAX_CONNECTIONS+1 is available to all accounts with SUPER, meaning the DBA cannot login when all connections are used to investigate problems like a deadlock.

Some end-users think that SUPER is a god-like privilege like root on Unix (with all the associated emotional baggage of losing it), but it’s only about 30% as powerful.

So explain to end-users that SUPER is not a Unix-like root account, and add another MySQL account like prod_admin with SUPER for use in administration by end-users if they insist on having occasional access to SUPER.

To fix it, just do:

mysql> stop slave; start slave;


MySQL Manual: Replication Slave Options and Variables

There were 209 sessions and 3,414 registered attendees.

I did not attend, but have seen some of the talks at other conferences in the past year. Pretty impressive list of speakers!

Here’s a plugin I wrote to calculate and display the performance of various MySQL read and write caches.

You can download the plugin source here.

Note that hit rates are seldom easy to interpret or helpful on their own. Mixing transactional and reporting loads will muddy them (as above.) This graph is more meaningful if you also monitor absolute read and write counts.

Percona Blog: What MySQL buffer cache hit rate should you target (2006)

I recently did some technical consulting for Netflix.

I can say that they are really, truly committed to cloud IT, per their public pronouncements. Most large companies are still just dipping a toe in the water, so it’s significant that Netflix took the plunge head-first.

They’re Java-centric and mainly use Open Source, with the notable exception of Perforce.

The following public links explain Netflix’s engineering philosophy.

Netflix Tech Blog Github Twitter
Adrian Cockcraft’s Slideshare Presentations Twitter
Joe Sondow’s Netflix Cloud Tools Keynote HTML Twitter

Sid Anand’s Practical Cloud Computing Blog Twitter
ASF Cassandra JIRA
Netflix overtakes Bittorrent as traffic champ

AWS Forums: unusual ELB activity not from my domain
Jason Brown: Cassandra from the trenches: migrating Netflix

Note that with older versions of Perl, you can use fork() or the forks.pm module instead of threads.

#!/usr/bin/perl

use threads;
use 5.8.1;
use strict;
use diagnostics;

$|=1;

$SIG{INT} = sub {
        threads->exit();
        exit(1);
};

   my @hosts = ('master', 'slave1', 'slave2', );

   my @threads;

   for (@hosts) {
       my $t = threads->new(\&ping_mysql, $_);
       push(@threads,$t);
   }

   for (@threads) {
       my $r = $_->join;
       print "$r\n";
   }

   exit;

sub ping_mysql {
   my ($host) = @_;

   my $timeout = 5;

   my $cmd = "/usr/bin/mysqladmin -h $host --connect_timeout=$timeout ping 2>&1";
   my $ret = `$cmd`;
   if ($ret =~ /mysqld is alive|Access denied/is) {
      return "$host: ok";
   }
   else {
      if ($ret =~ /error: (.*)/) {
         return "$host: $1";
      }
      else {
         return "$host: not ok";
      }
   }
}

perlmonks.org: Things you need to know before programming Perl ithreads
perlthrtut – tutorial on threads in Perl
CPAN: forks module
PerlMonks: How to debug perl code that uses threads

I noticed some disturbing things related to Apache HTTPD:

• many apps are adding .conf files to /etc/httpd/conf.d, which was intended for virtualhosts. So guess what – by default, now nagios, awstats, mrtg, wordpress, etc. appear as aliases under every site that you host on that server. So? Now search engines look for those aliases across all your sites, and when you fix that by restricting them, then you get a ton of 404s. Not good for SERPS or SEO.
• the mod_perl 2.0.4 module appears to be broken, sending back blank pages fairly often:

  [error] :Apache2 IO flush: (103) Software caused connection abort at
  /var/www/cgi-bin/test.cgi
  

• httpd has a bug in ExecCGI/Multiviews/whatever where file.cgi.txt and file.pl.txt are treated as executables. The error log looks like this:

  (13)Permission denied: exec of '/var/www/html/test.pl.txt' failed
  Premature end of script headers: test.pl.txt
  

  This is a case of mod_mime Files with Multiple Extensions.

On a more positive note, the WordPress 3.2.1 widgets admin UI is breath-taking, with drag-and-drop configuration of sidebars. And the myriad caching plugins that now support PHP memcache and other libraries is amazing as well.

It was held in parallel at the Oregon Convention Center with the O’Reilly OSdata and OSjava Conferences at the beginning of the week, and then later a knitting conference.

The conferences were well-managed, as usual. Great economy: lots of job notices and recruiting appeals. There was some chatter about Tim’s anti-harassment blog post.

Executive Summary:

• HTML5/CSS3/AppCache are what should have been available 20 years ago, and are significant improvements that allow both desktop and mobile development in HTML. Although the HTML5 video tag gets a lot of press, HTML5 includes equally important forms improvements.
• DNSSEC is complex and new signatures should be generated every 30 days or less (to reduce replay attacks by limiting the signature validity period), which is a burden on companies without a full-time DNS hostmaster. Third-party DNS hosting companies are salivating over DNSSEC.
• MySQL long-term stewardship is still in question, with Oracle hemorrhaging MySQL developers and closing access to their bugs database, but MontyProgram and Percona maintaining strong forks.

Here’s my notes on some of the tutorials and talks I attended:

Monday

HTML5 & CSS3: The Good Enough Parts
Estelle Weyl, Standardista.com
Slides

- transform-origin is key to snowflake demo looking realistic, easy to use
- background resets everything, so use individual properties
- background-position – use all 4 values
- background-size auto contain cover, handy for iPhones
- text-overflow: ellipsis
- minimal HTML5 document:

<!doctype html5>
<meta charset=utf8>
<title></title>

(head and body are implied)
- or even send tags in server headers
- changed most elements
- <i lang=”"> useful to style
- small tag useful for legal smallprint, since there’s no copyright metatag yet
- html5shim
- Modernizr
- HTML5 Boilerplate – good way to learn HTML5 and CSS3
- tabindex=”-1″ allows JS to set focus and not bother user otherwise
- spellcheck=”true” | “false”
- itemtype=”http://data-vocabulary.org/Person”
- new input types
- placeholder, pattern, required, spellcheck, validate
- a@b is deliverable for internal email servers. hmm.
- meter, progress, output widgets
- HTML5Rocks
- button generator at css3button.net
- Voronoi diagram demo
- aquarium.js
- web workers

Monday Lunch

Benjamin, Ubuntu
- loves CloudFlare
- likes Linode
- nimbula

Talked to an open mapping data fellow about various projects. Google ToS is scary when it comes to that kind of data.

Monday Afternoon

Moose is Perl: A Guide to the New Revolution
Ricardo Signes, Pobox.com
Slides

- detailed talk about Moose features and syntax
- chatted with other folks at break time about topics like CSS::Inliner and Mojolicious web framework (with minimal dependencies) by Sebastian Riedel.

Tuesday Afternoon

Three For Five – Functional HTML5 & CSS3 for Designers & Developers
Jason VanLue, Envy Labs and CodeSchools.com

- good training session with fun sample – a beer menu created from 1 photo (CSS3 text scaling and rotation) and HTML5/CSS3 styled text
- training class is available online for $75 (also jQuery and 2 Rails classes)

Click to Enlarge

Tuesday Night

Photo credit: Jacob Helwig

Creating a Scalable JavaScript Application Architecture
Nicholas Zakas, NCZConsulting
Slides

An AJAX client only cares about getting the data it wants, not response codes, etc.

Use layered JavaScript client architecture:

- sandbox
- application
- library (Dojo, YUI, MooTools, etc. )

www.nczonline.net
@slicknet
Author of “High Performance JavaScript”

Wednesday Lunch

I talked to Ben Golub, CEO of Gluster.

- 80% business, 20% scientific
- users include box.net, Pandora.com
- written in C
- minimum is 2 nodes for replication
- lots of people use it in EC2
- office located in Sunnyvale.

Wednesday Afternoon

HTML5: All about Web Forms
Estelle Weyl, standardista.com

- use label tag with forms to ease navigation for end-users
- use placeholder attribute, better for screenreaders than JS coding
- multiple autofocus defaults to last one in HTML5
- type=”text” is default, so tel, email, etc. degrades on all browsers back to text
- form element can disassociate parent form, useful for AJAX multiform pages
- input types good for mobile devices to show useful soft keyboard for url or email input types
- numeric step options
- test date and numeric input types for usability. Scrolling birthdays or zip codes is painful
- still need JS
- Opera is first with new UI features but last with artistic design, so currently has hideous tooltip appearance
- list and datalist like exploded select. Include select for IE backward compatibility
- meter, progress and output UI elements
- input type=text x-webkit-speech, now on Google homepage

HTML5 in Your Pocket: Application Cache and Local Storage
Scott Davis, ThirstyHead.com

- 4 million Macs, 32 million iDevices in last quarter
- Basecamp Mobile
- “Dive into HTML5″ by Mark Pilgrim online
- cookies should be called thimbles, only 4k
- HTML5 localstorage supported in IE8, FF 3.5, so practically all
- 5 MB, QUOTA_EXCEEDED_ERR, can’t increase now
- StatCounter browser stats
- gist 350433: Storage polyfill using window.name and cookies
- no version of IE or FF support web SQL and they probably won’t, FF for philosophical reasons
- cache manifest
- genManifest bookmarklet
- FF about:cache and Firebug are handy to see caches
- appcache has no expiry date
- date stamping manifest file causes re-download
- 404 causes none to be saved
- treat appcache as only slightly more secure than cookies, which are round-tripped
- webplication
- still sandboxed from local file access, could use node.js or signed app
- See W3C HTML5 offline

Profiling and Detecting Bottlenecks in Software
Bryan Call, Yahoo!/Apache Committer
PowerPoint .pptx

- usual savings (machines, moving parts, get smart)
- top, htop
- vmstat, dstat
- time cmd
- Boost logging does small writes, allocates memory when it gets behind, causing both IO and memory pressure
- profilers like oprofile and google profile cause 1% to 8% slowdown
- valgrind’s callgrind much more resources
- oprofile has script to convert output to kcachegrind
- opcontrol –deinit
- sysctl nmi_watchdog off
- opcontrol –no-vmlinux
- opcontrol –daemon
- google profiler userland, LD_PRELOAD
- env CPUPROFILE=/tmp/mybin.prof /usr/local/bin/my_binary_compiled_with_libprofiler_so
- caching: don’t do the same work twice
- choose the correct algorithms and data structures: dqueue vs. List, hash vs. trees, locks vs. r/w locks, bloom filter
- reuse memory, stack vs. heap, tcmalloc
- make fewer system calls (larger reads and writes)
- faster hardware, bonded NICs, SSDs, RAID, CPU, more cores
- read How to Profile Apache Traffic Server
- bootchart
- http_load now uses epoll
- he made ab multi-core

From kernel.txt: “nmi_watchdog: Enables/Disables the NMI watchdog on x86 systems. When the value is non-zero the NMI watchdog is enabled and will continuously test all online cpus to determine whether or not they are still functioning properly. Currently, passing “nmi_watchdog=” parameter at boot time is required for this function to work. If LAPIC NMI watchdog method is in use (nmi_watchdog=2 kernel parameter), the NMI watchdog shares registers with oprofile. By disabling the NMI watchdog, oprofile may have more registers to utilize.”

CoffeeScript: A New Hope for JavaScript
Scott Davis, ThirstyHead.com

- trainer, author, worked on Comcast/Time Warner TVs which mostly use WebKit
- little language that compiles into JS
- JS V8 headless, like node.js
- PhantomJS is headless HTML, handy for testing
- Google GWT compiles Java to JS
- “transpiler”
- install node.js
- install npm
- npm install -g coffeescript
- –tokens, –nodes like java p
- immediately invoked function expression IIFE
- coffeescript: string interpolation #{name}, “”"
- objects with left-hand spacing like python

Thursday

DNSSEC @ Mozilla
Shyam Mani, Mozilla Corporation
Slides

- BIND 9.7 is nice for DNSSEC
- Keys are everything, protect them. Have a backup plan.
- Cisco core routers by default don’t expect large DNS transfers:

policy-map global policy class inspection_default inspect dns maximum-length 4096

- DS was live, no signed zones
- watch log levels, can be chatty and quickly fill disk with logs
- DNSSEC has no immediate benefit to end-users, since resolvers don’t honor it
- their logs show 1000:1 dns vs dnssec queries for last 6 months, but growing
- IOS Firewall DNSSEC

Ask Google Engineers Anything
Chris DiBona, Google

- 55 Google employees attending OSCON this year
- mostly end-user questions about Google+ circles and API
- or running Go on android
- or why does my telco not do firmware releases for my smartphone
- or not happy with Google search results this month
- I asked about original reason for GFS. Originally, the hardware was really that flaky, and Google even actively bought bulk refurbed computers and RAM, sometimes off the back of a truck. Got a gopher plushie in return.
- also some good feedback complaints: google groups UI inadequate for managing 350 groups in an Education scenario
- inadequate data import tools for non-profit users of groups, mentioned by a religious charity.

How Not to Design Like a Developer: Open Source Can Look Good Too!
Chrissie Brodigan, Mozilla/Firefox

- KPI vs. git (different goals)
- @sirupsen
- story about the downloads map graphic for FF 4 – a developer silently removed social button graphics, limiting participation of wider audience. Marketing needs to explain why and how other staff fit into outreach programs.
- hang out on #projectdesign
- design contests are a good way to get them to come out of the woodwork
- designers hang out on twitter, not irc
- programmers should avoid big red buttons that scare users, and improve accessibility
- Inkscape, Blender, HTML and CSS are some Open Source tools for design mockups
- do AB testing or survey users
- designers want to be martyrs, so be careful they don’t offer more than you are willing to accept (start with 1 icon rather than the whole set)
- take a look at graphics libre for icons
- Quitestrong.com 5 girls who do design

Gearman BOF
Brian Aker

- Gearman polls, beanstalk busy waits
- nice to have feature to give up to another thread
- monitor projects handle launching of workers
- Gearman has durable and non-durable queues
- is a superset of the crap you handrolled. Most of the homegrown apps peak at 50% to 60% of Gearman’s features
- Gearman is production ready, but the postgresql driver less so because of fewer test cases and Brian’s lesser familiarity
- setup ntp and use Gearman coalescence for redundant cron servers
- can inspect queue
- agnostic to backend
- 99designs.com looking at this, same use case as original developer
- I still think that if you already have a database app, adding a status column gets you a lot of Gearman functionality without one more moving part.

MariaDB BOF
Monty Widenius, MontyProgram AB

- Monty mentioned that the latest release of MariaDB 5.3-beta has faster replication from group commit and performance improvements on the master, which also help the slave. Also subqueries and joins work much better.
- Monty talked about his Aria storage engine, which is a replacement for MyISAM that has both transaction and non-transaction modes. It’s intended for users who want the space savings of MyISAM. Over time it may compete with InnoDB.
- Monty’s responsibility is to convince Percona to merge into 1 source base sometime
- it’s estimated that although Oracle still has the InnoDB team, they may only have 1 general MySQL server programmer left.
- he explained that MontyProgram developers work 50% on feature requests from end-users, and 50% Open Source-related. So paid requests for 1 week of work really need to cover 2 weeks of developer time for that model to work. Typically a medium-sized change is roughly $12,000 and includes development, testing and documentation.
- Zmanda got FLUSH TABLES WITH READ LOCK NO CHECKPOINT for a beer, though. Monty was able to find a code path that could be locked to prevent Aria and MyISAM from writing anything during the backup.
- I sponsored WL#232 for USD$100 to add a SHUTDOWN statement to MySQL
- Monty explained that MERGE tables may be a better choice than MySQL partitions for logging applications.
- attendees from MontyProgram, SkySQL, Percona, DeNA

AskMonty: MySQL “Wishlist” Session from an online travel agency
MySQL Manual 5.5: FLUSH Syntax

Perl Lightning Talks
Hosted by Geoff Avery

- a talk on why arrogant community members telling others that “they need a thick skin” is unhelpful
- a talk by a young Perl community member on getting commit access, and how others can get the spirit and contribute
- Larry did several talks, mostly encouraging backporting Perl6 features to Perl5 it seemed, perhaps as a replacement to going Moose
- nice song on the importance of public libraries, which face shutdown due to economic budgeting problems in Australia and USA
- nice comedy juggling act comparing programming languages. Perl6 was omitted as “nothing has been updated in 5 or 6 years”, prompting Larry to say that he was happy he has a thick skin. See above.
- afterward, I talked to a booking.com rep about why a European company needed to actively recruit in USA and world-wide. He said that European developers are happy working where they are now, and it’s easier to recruit in places with mobile workforces like the USA. He would like to hire a couple developers per week to meet their development schedule.