Fortunately it’s also easy to solve that by understanding how to design HTML and site structure in advance to support those.

One of the most helpful things that can be done is to have well-defined URLs for HTML, images, CSS and JavaScript in the following scenarios:

• homepage content rooted under /, like ., images, css and js respectively
• admin site content rooted under /app, like ., images, css and js
• CDNs where you may store content in the future, such as a network-local proxy, Amazon, Akamai or Limelight. Generally a remote URL is overlaid onto the homepage or admin site structures listed above.

If you have localized content, each of the subdirectories above may be subdivided by ISO language code also.

Generally HTML and URLs intended for web browsers and email clients needs to be considered separately:

• web browsers handle relative URLs well
• email clients do not handle relative URLs as well as browsers, if at all. For example, not using an absolute URL or not specifying the scheme (ie. ://domain.tld) will cause problems in most email clients.
• in both cases, HTTP servers can use URL rewriting if necessary to make changes after the design is done.

Relative URLs not only help with HTTP to HTTPS transitions, but also in creating developer sandboxes, and test and QA servers.

Try to make as many links as possible relative in your HTML as possible if you’re planning on using SSL or test servers in the future. Fixing the links later can be expensive as it requires testing the entire site again for broken links.

By having reserved directory paths for images, css and html, it’s possible to set far-future expiry times to improve cacheability of those assets. However, the filename may not be reused, so new version of images need to receive a new filename or else caches will continue serving the old content.

And by having a URL available for items that could be served from a CDN, it’s possible to configure your CMS to be CDN-aware from Day One and avoid site changes and testing later. One of the first things I check when evaluating CMS programs these days is how I would easily be able to change serving images from a local web server to a remote CDN.

This was a fairly non-technical general talk on Google’s efforts to realize their mission statement.

What was most notable about the talk is that no mention was made of where their i18n staff came from. Google largely gained their deep Internet i18n knowledge from hiring former Netscape and IBM ICU staff. Currently the group hires based on referrals of experienced people.

It was decided to pick the 40 most natural languages as they represented 99.7% of web traffic. (To get to 100% would require 120 more languages.) Google search itself is in 113 languages, and GMail in 54, soon to be 58. Eric Schmidt, Google’s CEO, is a strong supporter of this effort and quality of user experience is considered more important than cost of translation.

Lux-IQ: program to get feedback on international User experience and localization quality of various Google products from a network of in-market evaluators.

Example findings:

Issue type

Language/translation, interaction design, feature missing, feature bugs, visual design, data quality, other. Total.

Google translation toolkit used for ads. Machine translation. Some ad customers request translation into 50 languages for example.

Language Findits. 3 hour testing party for language-related products. Very successful.

Language console would help with finding already translated strings.

Globalization continuum

I18n prd, intl 1-stop, i18n checklist, country planningn legal, content, l10n checklist, translation, review and qa

I18n, planning, deployment

All is global, weekly pushes, 0.25 seconds for search query response

Quality is more important than cost.

High level advice alone – not effective
Deliver concrete solutions, hands-on
Adapt to product needs, constraints and priorities
Earn credibility
Success breeds success
Be persistent

Metrics: intl revenue, top10 problems
Graph of i18n api adoption

Challenges

Unicode redesign
Bidi in webapps
Broad range of environments
I18n technologies
Deep dives: android, chrome, gmail, youtube: to help critical area, new areas

40 language initiative

Take aways

I18n by design
Educate, evangelize, communicate
Design globally, implement locally
Build credibility. Success breeds success.
Retrofitting happens. C’ets la vie. Learn from it.
“Make it easy to do right, and hard to do wrong.”

3 engineers for 7 months to fix gmail

Thanks to Google for hosting.

Twitter: IMUG

# XIHA Connects Facebook and Twitter Friends with New Multilingual Translation http://tinyurl.com/2ankev2 #L10n about 3 hours ago via web

# Zynga Launches First Localized Game In China: Texas Poker. http://tinyurl.com/2fxp9tm #L10n about 3 hours ago via web

# @localization sorry didn’t see your Q (no hashtag) but their i18n team seems fairly centralized but serves all projects and offices. about 4 hours ago via web in reply to localization

# And the event is over! Thank you Manish, and thank you Andrew Swerdow & the Google i18n intergrouplet for hosting! #imug408 about 6 hours ago via web

# Question: is there a process for self-localization of smaller language? Yes, for example Search recently translated into Hawaiian. #imug408 about 6 hours ago via web

# Question: why 40 languages? Those 40 can reach 99.7% of all internet users. Actually 42 now. ~100 more needed for remaining 0.03%. #imug408 about 6 hours ago via web

# Many questions from audience. One was how well-integrated is bug-management system? Manish summarized end-to-end process for that. #imug408 about 6 hours ago via web

# @renatobeninatto approximately 60 attended tonight’s Google #i18n event. #imug408 about 6 hours ago via web in reply to renatobeninatto

# Manish now summing up: #i18n by Design and other take-aways for any organization. #imug408 about 6 hours ago via web

# Google #i18n API adoption has grown 173% since start of the 40 language initiative. #imug408 about 6 hours ago via web

# Google has a team dedicated to #Unicode “Redesign”. #imug408 about 6 hours ago via web

# First two points on successful #i18n: not high-level advice alone; deliver concrete solutions or even hands-on help. #imug408 about 6 hours ago via web

# How can an #i18n team make an impact on projects from the outside? Manish offers 6 points. #imug408 about 6 hours ago via web

# Amazing how much the Web challenges but also offers opportunities in #i18n. #imug408 about 6 hours ago via web

# Interesting timeline now of Google’s Globalization process from i18n thru Planning to L10n. But I’m not going to give it away. #imug408 about 6 hours ago via web

# Apparently this internal Language FindIt program results in far less mischief than other firms’ community translation efforts. #imug408 about 6 hours ago via web

# Manish now onto Language FindIts: Googlers identifying translation & #L10n issues to improve Google products in own languages. #imug408 about 6 hours ago via web

# Interesting discussion going on with audience about MT vs. transcreation. And rule-based vs. statistical translation. #imug408 about 6 hours ago via web

# @renatobeninatto Yes they do use Google Translator Toolkit internally for example in automated ad translation. User can then edit. #imug408 about 6 hours ago via web in reply to renatobeninatto

# @renatobeninatto I found Petra, she says Hi, but she is still making me ask your question. #imug408 about 6 hours ago via web in reply to renatobeninatto

# @ken_lunde yes similar to Wordware but the entrance fee was far less! Very good presentation, great interaction with audience. #imug408 about 6 hours ago via web in reply to ken_lunde

# Google has a program called Lux-IQ to get feedback from local market-savvy non-technical users in all 40 language markets. #imug408 about 6 hours ago via web

# #i18n quality issues include not only basic encoding and locale issues, but also missing features important locally. #imug408 about 7 hours ago via web

# The presentation is now turning to quality issues in #i18n. #imug408 about 7 hours ago via web

# Manish is also presenting case studies, such as their experience with Google Video and Unicode (pre-YouTube). #imug408 about 7 hours ago via web

# And by the way Google is hiring #i18n engineers! #imug408 #jobs about 7 hours ago via web

# Manish has given us perspective on Google’s incredible global growth, and the start of Google’s 40-language initiative in 2007 #imug408 about 7 hours ago via web

# Manish Bhargava, Google #i18n Product Manager, is presenting. #imug408 about 7 hours ago via web

# Tonight’s IMUG event at Google kicked off 1/2 hour late, big crowd not enough badges. #imug408 about 7 hours ago via web

# Going to tonight’s IMUG event @ Google? Maps, directions and more: http://www.imug.org/google/ #imug408 about 10 hours ago via web

# Shirley_Rogers Twitter Unicode Hashtags – http://bit.ly/9ciNuu about 12 hours ago via web Retweeted by i18n_mug

# 57 yes, 5 maybe RSVPs: 8 left for tonight’s 70 chairs. Will it be SRO? Google’s Manish Bhargava is an #i18n star! #imug408 about 12 hours ago via web

# Localization Project Manager, Net-Translators, Sunnyvale, CA. Just posted to IMUG Jobs: http://www.imug.org/jobs/ #L10n #jobs about 13 hours ago via web

# Maps and directions to tonight’s 7 PM Google #i18n event in Mountain View, CA: http://www.imug.org/google/ #imug408 about 14 hours ago via web

# IMUG cannot do webcasts from Google yet. Hope to see you all there tonight! http://tinyurl.com/2f9esas Hashtag will be #imug408 about 14 hours ago via web

# RT @ken_lunde Two font- and CJKV-related Tech Notes now live. http://tinyurl.com/23ffulg & http://tinyurl.com/yzd3hjj <–Kazuraki font! about 16 hours ago via web

# Kazuraki: Adobe’s Groundbreaking New Japanese Typeface http://tinyurl.com/2by46nn Next month’s IMUG event, @ Adobe #imug408 about 16 hours ago via web

# TONIGHT 7 PM: The Google i18n Story http://tinyurl.com/2f9esas Hashtag for this IMUG event @ Google will be #imug408 about 16 hours ago via web

# cathywissink RT @TalkStandards Nascent Web Open Font Format is getting boost thanks to W3C’s new initiatives http://bit.ly/biE85M #typography about 16 hours ago via web Retweeted by i18n_mug

It was a good conference, and we had beautiful weather all week long.

Executive Summary

The themes promoted by the conference organizers were Cloud Computing, NoSQL, Emerging Languages (Scala, Erlang, Parrot, Go) and Android phone development.

The @oscon twitter channel was heavily used to coordinate amongst organizers and attendees. I used the TwiXtreme twitter client program on my BlackBerry.

Plug Computers were very popular in the Expo area. They are 5 watt ARM-based computers running Debian Linux that fit into a power brick-sized case and cost $99 to $129 depending on features. The Marvell booth had a few models on display, from GlobalScale (GuruPlug) and Ionics. High-end models have dual gigabit NICs, multiple USB ports, a WiFi access point and other expansion ports.

There was also continuing buzz regarding Facebook’s Flashcache SSD module (GPL v2) for linux, and also ZFS snapshots.

Tutorials

I went to the Gearman Cookbook tutorial, the first half of the Chef tutorial and some of the Cloud Summit talks.

The Gearman Cookbook tutorial was excellent. After a detailed overview of the Gearman architecture and implementations in Perl and C, a number of use cases were explored in detail, including before and after code samples. The talk was both easy to listen to as an overall survey, as well as providing immediately useful info for those wanting to deploy it.

The Chef tutorial was very detailed – too much so perhaps. I went to the first half only, since I am not planning to implement Chef soon (I use PXE and anaconda/kickstart with CentOS), and did not need that level of detail at this time. cfengine, puppet and chef are ops tools for configuring servers. Chef uses Ruby data structures for its configuration files, and has include files and other useful syntax. Basically, users can “code” server configuration, as if they were traditional apps.

I went to some of the Cloud Summit talks and BOFs, but found that anybody who has done a simple project using EC2 knew as much or more than the speakers, some I would call blowhards.

Marten Mickos, president of Eucalyptus, is refreshing in that he is always clear about being in it for the money, while also promoting Open Source.

Sessions

Some of the most memorable sessions to me were:

Introduction to MongoDB, Kristina Chodorow (MongoDB)

Kristina is the maintainer of the Perl and PHP drivers for MongoDB. She gave an overview of MongoDB, a NoSQL document store, and its command-line interface, which uses JavaScript.

Some day she will release a sharding tool for MongoDB.

Scaling SourceForge with MongoDB, Nosh Petigara (10gen), Rick Copeland (SourceForge.net / GeekNet)

Nosh and Rick gave an excellent review of incorporating MongoDB into the SourceForge site.

- SF query load is mostly read-only
- ops team benchmarked a few NoSQL candidates, and MongoDB won on performance
- original MySQL servers had 64 GB RAM. After migration to MongoDB, same server machines but only 8 GB RAM
- backup dumps are verified to be bitwise the same as masters
- have to be careful not to dump all documents in your database to the network or it will max out switches
- SF relies on first-class data centers and replication slaves, less worried about MongoDB mmap (not crash-safe)
- I personally looked at their performance numbers and site graphs (on an iPad), and the end result was impressive.

Perl Lightning Talks

As always, the Perl Lightning Talks are a highpoint of the conference.

The “cartoon” of Vincent Pit’s remarkable CPAN module(VPIT) contributions was both informative and hilarious. Vincent is a French Ph.D. candidate in advanced geometry.

Cloud BOF (3 Hours)

The Cloud BOF was disorganized, starting 30 minutes late and for some reason was subdivided into 4 audience groups. Startups and vendors trying to make a cloud sales push led the BOF, including cloud and DNS service providers.

The Health Regulations subgroup came up with a couple ways to make the Cloud palatable to regulators by using encryption on all data due to the multi-tenancy issues with sharing public VMs.

I was in the NoSQL group, which discussed general issues and particular successes. Memcached was the clearest winner, while some people also had success with MongoDB and Redis.

My neighbor was an engineer at Postrank.com. He said that they were happy with HAProxy, but much less happy with the unpredictable IO available when running MySQL on EC2. He also said to carefully look at storage volumes available to your instance, as one is a useful tmpfs. They use AuthSMTP to get around EC2 being generally blacklisted for outbound email.

Database BOFs

MySQL BOF

The MySQL AB engineering staff has left Oracle. Monty Program AB (21 staff) has the core developers, and Percona Inc. (32 staff) has the consultants. Oracle still has some of the InnoDB programmers.

The business plan for Monty Program AB is 60% commercially-sponsored MySQL development, and 40% community-request development. Monty would like commercial users of MySQL to sponsor patches that would benefit them.

Mark mentioned that using Nehalem instructions for CRC were much faster, and that Facebook was using partitions for truncating tables instead of doing multi-record deletes. (See his blog for more details.)

One person mentioned using a commercial backup tool, R1Soft, that inserts a linux kernel module to allow filesystem snapshots. He said to carefully test backup and restore in your environment, especially for filesystems greater than 1 TB which may exceed certain block counter limits. Peter said that some of his clients had used it with varying success.

It worked for him in his environment, and the file browser allows selective file restore (he uses it to restore by priority where a system runs multiple applications.) It starts at $299 for the Standard Edition, and also has MySQL Add-on and Enterprise Editions.

PostgreSQL BOF

The PostgreSQL BOF talked about 30 or so changes that went into version 9.

One of the most exciting new features is a native replication feature, called streaming replication (block-based.) The advantage over Slony-I replication is that Slony-I is trigger-based, so has a variety of issues included inability to replicate DDL commands.

Some of the developers mimed replication events, which was rather amusing to watch. Yes, it was taped.

PostgreSQL is released under the PostgreSQL Licence, which is BSDish.

Peter Zaitsev, co-founder of Percona, organized 3 BOFs, including XtraDB, XtraBackup, Maatkit, Percona Server, Sphinx Search and Running Databases on Flash Storage.

Sphinx Search BOF

Andrew Aksyonoff, the original programmer of Sphinx Search (GPL v2), couldn’t make it to OSCON (the good excuse was that he was busy coding), so Richard Kelm (Sphinx sales/customer support honcho) and Peter filled in (Percona is a business partner with Sphinx, and many of Percona’s clients use it.)

Some of the attendees were existing users, like myself, and some from HP and other companies were looking for a large-scale search solution or alternative to Lucene.

Monty mentioned that the latest MySQL 5.1 should be used, as there have been a number of performance and reliability improvements. Full-text search is supposed to be 10x faster than 5.0, and replication is nearly bug-free by now.

Sphinx Search now has real-time index updates in version 1.1.0 beta. Another very nice feature is SQL+FS indexing.

Here is the full Sphinx 1.1.0 changelog.

Running Databases on Flash Storage BOF

The Running Databases on Flash Storage BOF had a combination of MySQL and Postgres users who have tested or used most of the SSD products: FusionIO, violin, Intel, OCZ, etc. Everybody was happy with SSD IOPS performance, but less so with cost and metadata RAM requirements with the add-in boards (FusionIO may require 4 GB RAM for metadata.)

Peter said that 20% to 30% of his clients are already using SSD – across the spectrum of vendors and models. Some are also trying “massive RAM” solutions, like Cisco servers with 384 GB RAM.

Some users had 1+ TB Postgres databases with very thorny backup and mgmt. issues. One solution was to start a snapshot, but not do the copy operation.

Expo Notes

I had an enjoyable talk with Austin Hook, who has operated the OpenBSD Store for many years. He lives near Calgary, the center of OpenBSD/OpenSSH/PF development. He mentioned that some perennial financial contributors had stopped because of the recession, so here’s the donations link.

I also talked to some reps from a Brazilian outsourcing firm, ActMinds. They currently have 400 employees across Brazil and a sales office in Philadelphia. Brazil is only 2 hours ahead of EST. They said the minimum project size is 2 developers and developer turnover a low 5%/annum. Their pricing is $35 to $45/hour.

And I had fun handling the plug computers on display at the Marvell booth. The Ionics boards are amazingly densely populated.

Discussions

I had the opportunity to talk to a long-time Portland resident who works as a computer consultant. He said that the Portland economy is not doing great, and really hasn’t done well since old-growth logging was stopped after 90% of the forests were cleared. And although hundreds of miles of fiber optic has been laid downtown, it’s not available for residential use. However, the Beaverton area does have ubiquitous FTTH.

I also talked to somebody who attended the Emerging Languages talks. He’s working on his M.Sc. in Computer Science, so found those talks fascinating.

Twitter Humor

There were some humorous tweets:

- “my MongoDB and CouchDB mugs are fighting each other.”
- “I got one MongoDB mug, but need two to safely store coffee.”

Notes

Note to self: skip the nightly parties unless you have a date. The bars are too loud to talk to anybody.

Note to the O’Reilly conference organizers: use meetup.com for the BOFs like ApacheCon does. The average audience was about 10 people, and with meetup it would be 4x that.

OSCON 2010 Slides
Tim Bray: Desperate Perl Hacker
Youtube: OSCON 2010 videos
blip.tv: OSCON2010 videos
wikipedia: Plug Computer
Jeremy Zawodny: MongoDB Early Impressions

Generally if ! or !+ characters are inserted into long email lines, the solution is use shorter lines (insert newlines) or use quoted-printable encoding.

RFC 821: “The maximum total length of a text line including the <CRLF> is 1000 characters (but not counting the leading dot duplicated for transparency).”

Re: sendmail maximum line length
Dan on Body: Line Length

Anthony has extensive experience in game localization, going back to Sierra On-Line’s Leisure Suit Larry franchise (challenging to localize because the user input was freeform text) and Valve Corporation’s Half-Life (requiring about 6 months to localize.)

Also many members of the game development community were there, including an Ubisoft localization project manager and SO-L’s fourth staff programmer.

Localization staff typically have to play the game for a week to understand how it works and what work needs to be done.

There are various localization levels for games:

1. Just localize the box art
2. Add subtitles
3. Localize everything

For networked games, some additional issues are:

• localize just the client (user) program, or also the server too?
• what if player messages to each other are in different languages?

Cheats are very helpful to expedite testing when testing levels, features, etc.
Testing on the lowest supported resolution is the fastest way to find font and string problems.

When testing PC games, particular graphics cards may be needed. Localizing console games is more involved though, requiring a developer console and software development licenses, possibly costing thousands of dollars per console.

Besides the usual challenges in localizing products, games have 2 additional steps:

1. for console games, manufacturer (Sony, Nintendo, or Microsoft) acceptance testing is required. 10 to 15 days need to be scheduled for the initial test run, followed by 5-10 days per round of additional testing for rejected games. Additional test runs are billed by the mfg.
2. government ratings boards, like ESRB in USA.

Also, holiday deadlines for games are literally that: missing a holiday means that your audience has already spent their discretionary budget, so the release is fruitless.

Thanks to Google for hosting the meeting.

He was on the Summize team that Twitter bought, so originally worked on search, then ended up being the i18n guy.

Twitter had a lot of things stacked against it i18n-wise:

- unilingual source tree
- no budget to do any real engineering initially
- Japanese tree translated by outside volunteer partner, Digital Garage
- primitive Unicode support in MySQL and Ruby
- now a legacy system full of data, hard to add metadata like lang/locale at this point.

Japanese cell phone support was challenging because:

- Shift_JIS, not Unicode
- each of 3 carriers uses a different image format and different emoji codepoints, some overlapping
- lack of l10n resources in house for Japanese-flavor design, including being cute, dense and also having an ad to demonstrate business seriousness
- Japan uses cellular emails, not SMS like other places
- mobile browsers don’t support cookies, so URL sessions needed unlike the regular Ruby web app
- hard to tokenize short messages in Japanese.
- need QR (Quick Response) code support (2D barcode)

QR Code

Crowdsourced translations using Google Groups, interns and app integrated with Twitter site. Now up to 3,700 strings and 2,600 translators. Hard to translate informal terms like tweet and follower though.

There was very good turnout, with 60 attendees live and 6 online.

Thanks to Adobe for hosting the venue. (Though I don’t understand why there is a 3-year NDA for attending a public meeting.)

techcrunch.com: Twitter Has Basically Doubled In Staff In The Past 6 Months (June, 2010)
Official Twitter Blog

Executive Summary

Most of the attendees that I talked to were primarily interested in search technologies, or were Apache project comitters. The search users were already using either Lucene and Solr, or were using commercial software and evaluating Lucene and Solr.

Also a lot of interest in Hadoop, Zookeeper and NoSQL projects.

I added a wikipedia NoSQL project features table after the NoSQL BoF.

The conference was very well-organized, with tutorials, BoFs, a BarCamp, and sessions. Meetup.com was used to generate the highest BoF turnout that I’ve ever seen – close to 100 at the Lucene and Hadoop BoFs. (O’Reilly Conferences can learn from that.)

The Oakland Convention Center was a good venue for this conference, though the attached Oakland Marriott hotel is $$$$ and fond of surcharges, like $33/day for parking, $5 draught beer and $3.75 for a bottle of water in-room.

The keynotes and one track per day were recorded and are available for $99 at Linux Pro Magazine Streaming.

StoneCircle Productions was the conference organizer.

Conference Notes

Monday

Although I live in San Jose, Oakland is far enough away that I’ve never been there. Oakland has a compact downtown full of historical-era buildings, and Alameda is also nice, but things get less pretty at night.

I went to the Lucene tutorial on Monday.

Lunch Conversations

- awesome views of Bay Area past Golden Gate bridge from 21st floor
- FAST pretty good indexing and search solution, but bought by Microsoft recently (going to continue linux support or not?)
- FAST has FQL (users pronounce it fecal) query language
- 150 FAST servers replaced by 40 lucene servers by 1 company
- FAST4 to FAST5 upgrade tough, similar to port to say lucene, forced upgrades for support
- linguistics is 60% of value of Fast according to Monster, 13 languages supported
- “bad stems” can be a nightmare
- SOLR gives you 90% of what you would need to program in java, built on top of Lucene
- Open Source search is not really about price, but about control and flexibility

Monday Afternoon – Lucene Tutorial

- user-assigned document id not mandatory, but great idea for many reasons, including after an index-rebuild
- lucene-assigned id only valid for that snapshot (life of score doc)
- parameter to keep or delete old index directory
- StringBuilder is more efficient than strcat
- populating title column is a good idea
- results boosting handy for ecommerce, specials, etc.
- LUKE – handy tool for index statistics, etc.
- Searcher class, snapshot in time, won’t see new merges
- contrib/ has more analyzers
- snowball stemmers
- use 1 tokenizer and 0 or more token filters
- precision-recall curve ??
- n-grams and shingles (”the president”, “United states”)
- pre-2.9 lucene, numbers and dates really strings
- 2.9 NumericField builds tri structure, help optimize range queries
- SOLR analysis tool apache-solr
- relevance feedback with MoreLikeThis

Monday BoFs

Couchdb

- “ground computing”
- “offline by default”
- now an ubuntu service
- mozilla raindrop to combine chat client msgs
- lockless
- append-only btree
- rsyncable since append-only, also replication
- checksums everywhere
- windows not first class yet, mozilla improving it

@mozilla

- browsercouch
- don’t like sql
- brasstacks test tool storage
- store now, index later
- replicate to handle large indexing load
- testbot ci

Marklogic

- commercial
- xml-centric
- great for articles, books
- transactional
- search-centric
- structure-aware
- schema-free
- xquery-driven
- extremely fast, largest 200 TB xml, 166 on hosts
- clustered
- database server
- 180 clients, 150 employees
- markmail.org demo contains 42 million email messages, very impressive performance with 5 views in almost realtime. Search is distributed across 160 nodes.

JCR in 15 minutes

- Bertrand Del
- JCR is JackRabbit,
a fully conforming implementation of the Content Repository for Java Technology API (JCR). A content repository is a hierarchical content store with support for structured and unstructured content, full text search, versioning, transactions, observation, and more.
- the ultimate content store
- content repo, union of database and filesystem, best of both worlds
- full-text search combined with structured search

Solr Flair

- information forage
- “resume-driven design”

Lucene Numerics

- available in 1.4
- tune by modifying precisionStep

HBASE

One bewildered attendee wished for a NoSQL product matrix, so I added that to the wikipedia NoSQL page.

Wednesday Sessions

Becoming a Pig Developer, Alan Gates

- Apache Pig is a sub-project of Apache Hadoop.
- this talk was really how to use PIG as an end-user, not to become a Pig project developer

Apache Hadoop in the Cloud, Tom White

- general comments on using EC2 with Hadoop mostly

Practical HBase, Michael Stack

- Apache HBase is the Apache Hadoop database, similar to BigTable.
- HBASE usage

mod_jk / mod_proxy and others, Jean-Frederic Clere and 2 others

- mod_jk, mod_proxy, mod_serf and mod_cluster original topics
- mostly focused on mod_jk, mod_proxy and isapi_redirect
- good talk by 3 long-term project contributors
- jk is kind of Java-centric, with support for Apache JServ Protocol (AJP) only available in Java back-end servers for now, like Tomcat
- isapi_redirect is primary way to do redirects on Windows IIS
- survey of audience showed several mod_proxy users, maybe one intentional mod_jk user

Thursday Sessions

“Apache Lucene and Apache Solr Performance Tuning with Mark Miller” was packed, so moving along to a different room …

Scalable Internet Architectures, Theo Schlossnagle

- amazing and thought-provoking talk, also one of the most popular
- think about performance from network packet level to application level
- carp, vrrp, whackamole
- alterdns, neustar
- dynact
- anycast (shared IP), geoip (but need actually accurate database)
- activemq, rabbitmq instead of Spread
- “memcached is the worst thing that ever happened to our industry – it solves a problem, just not the original problem”

- many apps today are so poorly designed that network issues never become scalability concerns – ie. RoR applications
- max out at 500 requests per second across 40 boxes – RoR
- firebug and yslow have been fantastic at making front-end engineers aware of networking performance
- 10 gb nics suck
- instead of one big 20 Gbps loadbalancer, use anycast from core router to 5x 4 gpbs cheaper load balancers
- spiky load or DDoS – announce a /32 to separate load balancer, use symmetric return path

- jms, aqmp, spread

durable message queues

- activemq (java)
- openamq (c) – hard to use
- rabbitmq (erlang) – nice except in durable mode because erlang disk io blows

- most common protocol Stomp is awful and slow (hard to read 100k messages per second) and not binary, but lots of clients exist.

- activemq and stomp is a good start.
- rabbitmq and native connectors are better, but no perl client.

- PCI compliance requires a stateful firewall. Hard to do 1.5 million packets per second traffic for most medium-sized data centers, need to use a CDN to distribute static requests and distribute the packets somewhere else
- leaving trailing / off causes 302, doubles traffic
- Slides
- read/write ratio is 1 … likely IM or email?
- went over some networking details with Paul L. afterwards

Recent Developments in SSL and Browsers, Rick Andrews, Thawte

- 1.6 billion OCSP requests per day, need good infrastructure to support that
- intermediate CA allows root CA to be offline – chained hierarchy – SSLCertificateChainFile,
needs intermediate certificates before cross-certificates, some clients need in proper order
- EV hierarchy more complex. wanted new EV root, but older browsers don’t know about it.
- browser ubiquity problem with any new feature, hash or crypto algorithm
- logotypes – trademark and copyright issues with using other companies’ logos in a product
- Verisign does not have apache httpd committers, but should
- 1 attendee wanted to sign JavaScript files, but what does it mean if most sites link to 10 advertising and tracking scripts? what do you tell the user if 1 JS is not signed?

Subversion Meetup

Organizers didn’t show up, so spent 10 minutes talking to a handful of end-users about subversion gripes and moved along to …

Hadoop Meetup

Zookeeper

- zk is persistent to disk
- can run on one node, but 3 is minimum non-toy
- zk is popular in academia now for some reason
- avoid split-brain partitioning between 2 data centers – bad
- very recent merge to fix -368, not ready for production yet
- people using it for a message queue, perhaps more reliable than many other Open Source ones
- need 1 zk node for testing, but 3 zk nodes for non-trivial implementation

Scribe

- github
- 4x to 5x compression with lzo. similar disk bw improvement

A local owner of a gelato store handed out 6 free samples from a portable gelato freezer.

Friday Sessions

Building Intelligent Search Applications with the Lucene Ecosystem, Ted Dunnin

- some matrix math
- using his matrix math optimization, a perl program on 1 server was faster than Mahout running on a $250k cluster
- tdunning.blogspot.com

- the original LLR in NLP paper
“Accurate Methods for the Statistics of Surprise and Coincidence” check on citeseer
- Mahout project
tdunning [at] apache.org

Realtime Search, Jason Rutherglen

- many technical issues prevent Lucene from being able to do realtime search
- lots of patches done, lots to do
- audience member thanked author for great work so far

Closing Plenary: Brian Behlendorf on Open Source and Charity

Talked to Alex Karasulu a little after the final presentation. He’s a committer on the Apache Directory project. He suggested adding dbm to the NoSQL product matrix. Wants a MacBook Air with 8 GB RAM to run his Java apps.

Conference Schedule Grid

It was a well-organized, fun-sized conference – just big enough to use half the convention center, which made it easy to get around without a lot of walking between session rooms.

There was also an official, parallel unconference in 2 rooms priced at $199 for non-conference attendees.

The talks were high-quality, the food was great, and wifi worked everywhere. What more could one ask for? Well, a few more power strips next time, perhaps.

I was impressed with the number of attendees from Europe, Australia – and Utah!

I went to Matthew Weier O’Phinney’s tutorials on Monday. He’s the Project Manger for the Zend Framework, including design and supervision of the framework programmers. He’s an excellent speaker and really knows his stuff. Both his Intro to Zend Framework, and Ajax with Zend Framework tutorials were excellent.

My favorite talk of the conference was Eric Farrar’s talk on Mobile Data Synchronization. His slides went through many of the pitfalls of data synchronization, then actually provided a solution: use Sybase’s mobilLink, which is free to use with MySQL and SQLAnywhere. He said a team of 24 has been working for 10 years on that, and it is deployed in millions of devices. He works on the ultraliteweb project.

A Digg sysadmin did an interesting operations talk on the evolution of the Digg data center over the past few years. They’re up to 800 servers in 2 Equinix locations now, and use pre-cabled racks of servers from Penguin Computing. Software-wise, they like Cassandra key-value pair, clusto and puppet. They tried some commercial software in 2007, and didn’t enjoy the experience.

I had some great lunch break talks with other folks. One guy from Ohio was getting interesting SEO results by serving raw XML to clients, and having client-side JavaScript provide styling for human users.

I talked to a couple folks about their experiences using MySQL NDB Cluster in production. They both said it’s flaky, with one having already abandoned it for regular MySQL database with Innodb. He was also using RightScale and Amazon for document processing, and was happy with that combo.

There were about 20 exhibitors in 2 aisles, so easy to talk to all of them.

I got personal demos of RightScale’s cloud admin app, Zend Studio IDE, and BCDSoftware’s WebSmart PHP code generator.

WebSmart PHP is a $4600 code generator for ex-RPG and COBOL programmers. It provides a basic IDE, but the interesting part is that whatever you might want to do is either documented in hundreds of online technote examples, or available by contacting their unlimited support department.

Some of the unconference talks I went to included improving cookie security by embedding the SSL session id, and part of the continuous integration session (they talked about Hudson and CruiseControl, but not BuildBot).

The unconference talk on PHP and queues was quite good, with an overview of Amazon Simple Queues (good), Gearman (no persistence), beanstalkd (rave), and custom PHP and C queues (don’t roll your own unless you want long-term job security.)

The closing keynote was what I was mainly at the conference for … the PHP Frameworks Shoot-out with the framework project leaders.

Here’s my notes from my perspective as a listener. Please email me with any corrections.

Agavi
- David
- borrow from Symfony PHPunit code
- would use Symfony as alternate
- CI ORM is a pointless reimplementation, Rails programmers are morons shaped by pragmatism of Rails model
- hates complexity of validation code, context from Majove too many interdependencies
- 5.3 nice to have universal exception handling fw
- believes 5.3 is a major new release not comparable to 5.1 or 5.2 that frameworks need to support
- bigger the team and complexity, better agavi is because more structure

CakePHP
- Nate Abele
- hates long class names
- hates ACL system needs to be redocumented or cleaned up
- PHP 4 at this point, next release on 5.3

CodeIgniter
- Ed Finkel
- Symfony generates too many files, brain hurts; input filtering in ZF overcomplicated
- hates complex routing, unlike Limonade
- CI is not recommended for complex systems

Symfony
- Fabien Potencier
- French
- full stack
- secure by default
- would use Django and ZF
- hates 1.1 form framework complexity that users painpoint
- 5.3 is nice, but no plans to port to 5.3 because of large users update cycle time

Zend Framework
- Matthew Weier O’Phinney
- ZF routing from Rails, lots of stealing
- would use CodeIgniter
- hates heavy-weight dispatch cycle in ZF, to be rewritten in 2.0
- 5.3 ZF already testing with it, backwards compatible to 5.2.

The sessions that had an audio recording will be released as podcasts, one per week, and hosted on devzone.zend.com.

Thanks to Zend for organizing a great conference.

(It was a long trip from San Jose on the Caltrain. I knew that I had arrived in San Francisco when I could smell the stench of urine upon leaving the station.)

Jeff mentioned working on ticketmaster.com before, and used S5 slides to illustrate a variety of localization issues with languages like Chinese, Japanese and Malaysian.

Some of his tips for identifying and preventing translation string corruption were:

• check for double-encoding of UTF-8 strings, perhaps with Test::utf8::is_sane_utf8()
• check complete toolchain for UTF-8 cleanliness
• can use Unicode script and block properties to identify language when possible, as documented in perldoc perlunicode
• use RCS pre-commit hook feature to inspect checkins, though can be slow with large input files.
• important to decide how much cleanup the translator is responsible for vs. internal.
• JavaScript string localization will likely require careful escaping of quotes.

Audience members also suggested:

• enable online web editing of translations as well as batch export
• consider locking columns if translators use excel worksheets.

As always, my comment is that it’s more important to focus on locale definition than charsets in i18n projects.

Several members were looking for perl jobs, so post your offers on the mailing list.

Thanks to Six Apart for hosting the meeting.

Juerd’s Perl Unicode Advice
Unicode.org
wikipedia: UTF-8
Jeff’s CPAN

(I have heard that the City of San Jose Business Development office is very, very accommodating towards conferences these days.)

There was great attendance, and plenty to see with about 15 simultaneous tracks, lots of BoFs, and an active exhibits area.

One of the changes this year was more OS talks, including some for linux and FreeBSD. This is a welcome change, though many kernel hackers won’t travel to the USA, for various legal issues.

My favorites were:

• talk – YAML by Ingy. YAML is a serialization standard for all programming languages and is a superset of JSON, in that YAML supports types and references. The Perl module is YAML.pm. Although it is a “serialization standard”, best results are obtained when both sides of the exchange are controlled by the programmer (ie. different word sizes or floating point standards will likely cause issues.)
• Perl lightning talk – Esthetic Randomness by Joseph Brenner. Joseph likes to post-process random output before display to get a more desirable appearance. He’s kind of goth-looking, so the overall subject and delivery made it an interesting 5 minutes.
• BoF – MySQL social with Monty, Percona (now 25 employees!) and Mark C.
• exhibit – Haiku OS (the Open Source BeOS clone) demo. 2 developers, now at Google, reimplemented BFS from the textbook. Haiku can run with 64 MB RAM. Posix compatible, so the gcc toolchain works. The ARM port is a GSoC project. The video support comes from from ffmpeg, xiph, etc.

Regarding the MySQL BoFs, I think they can be summarized like this: the community is not going to wait for MySQL AB/Sun/Oracle to dick around any further.

Monty Program AB, Percona and Drizzle are going to have forks regardless of what Oracle does. Either the official MySQL documentation will be freed, or rewritten by Monty Program AB and Drizzle.

Typo3 CMS also had a community booth. They are the #1 European CMS with support for 38 languages.

I didn’t see much use for the “OSCamp” attendee-organized tracks personally. Whereas at the MySQL Conference the Percona Performance Conference was necessary to fix the broken speaker selection process that was weighed towards MySQL/Sun staff instead of productive community contributors, that wasn’t an issue this time around.

The talk on Perl and Unicode was pedantic (focusing on UTF-8 bit patterns, presumably for those needing to detect and fix corruption) but comprehensive, as Tom Christiansen was in attendance to provide up-to-the-minute comments and tips. perluniintro is very helpful.

The PHP Best Practices talk was informative, as the 2 presenters have worked as PHP programming consultants and seen how projects go wrong. They tend to use whatever PHP framework the client is using, and have nothing glowing to say about any particular one.

I’d say that the world of PHP frameworks (dozens) is even more fragmented than Perl (Catalyst, Mason, embperl, CGI::Application are the major ones), which is indeed astonishing. And ironic – since PHP is itself a templating language.

Stonehenge Consulting threw another of their famous drunkfests at a local bar for those wearing their neon yellow t-shirts. If you want to get hammered for free, this is always the spot.

I talked to Randy Ray a little about what can be done with svlug.pm considering that the South Bay is suburban and thus less centralized than a dense city. Stay tuned.

Other Perl lightning talks included:

• Larry Wall’s son talking about black holes
• Scott Smith talking about Getopt::Complete, which can do svn command-line style nested args
• connie willis bellwether talked about Flocks and the hive mind as defined by 3 rules.
• Don’t Blame Perl – It’s the programmer’s problem if they don’t use modules, scoping, comments, brevity, objects.
• Cool Perl6 – hyperoperators (work on arrays) with a card game sample using extended-ASCII symbols.
• svn is not totally useless – it pointed the world towards git.

The closing talk on linux economics seemed to be an eye-opener for the audience. Most cell carriers and OEMs are at a severe disadvantage to Apple in the apps market, so they may need linux (or Haiku) to mount any kind of response that makes financial sense.

In the conference wrap-up segment, Allison Randal and an O’Reilly rep fielded questions from the audience and answered in “Twitter mode” – single sentences less than 140 characters in length.

OSCON 2009 Speaker Presentation Files
youtube.com: oscon 2009 video clips
wikipedia: Monty Program AB
mtocker: Understanding the MySQL forks