James' World » Japanese

QR Code Links

Administrator — Wed, 09 Nov 2011 13:09:05 +0000

Originally designed by Toyota Japan for factory use, QR codes are 2-dimensional barcodes commonly used in advertising.

cnn.com: 23 of the coolest QR codes
cnn.com: Does anybody actually use QR codes?
QRcodify.com
Perl Module GD::Barcode::QRcode
Successful QR Code Campaigns
msn.com: How QR codes hide privacy, security risks
theregister.co.uk: QR Codes and Spam

OSCON 2011, Portland

Administrator — Sun, 31 Jul 2011 06:03:21 +0000

Once again, the O’Reilly Open Source Conference (OSCON) was held in Portland, Oregon.

It was held in parallel at the Oregon Convention Center with the O’Reilly OSdata and OSjava Conferences at the beginning of the week, and then later a knitting conference.

The conferences were well-managed, as usual. Great economy: lots of job notices and recruiting appeals. There was some chatter about Tim’s anti-harassment blog post.

Executive Summary:

HTML5/CSS3/AppCache are what should have been available 20 years ago, and are significant improvements that allow both desktop and mobile development in HTML. Although the HTML5 video tag gets a lot of press, HTML5 includes equally important forms improvements.
DNSSEC is complex and new signatures should be generated every 30 days or less (to reduce replay attacks by limiting the signature validity period), which is a burden on companies without a full-time DNS hostmaster. Third-party DNS hosting companies are salivating over DNSSEC.
MySQL long-term stewardship is still in question, with Oracle hemorrhaging MySQL developers and closing access to their bugs database, but MontyProgram and Percona maintaining strong forks.

Here’s my notes on some of the tutorials and talks I attended:

Monday

HTML5 & CSS3: The Good Enough Parts
Estelle Weyl, Standardista.com
Slides

- transform-origin is key to snowflake demo looking realistic, easy to use
- background resets everything, so use individual properties
- background-position – use all 4 values
- background-size auto contain cover, handy for iPhones
- text-overflow: ellipsis
- minimal HTML5 document:

(head and body are implied)
- or even send tags in server headers
- changed most elements
- useful to style
- small tag useful for legal smallprint, since there’s no copyright metatag yet
- html5shim
- Modernizr
- HTML5 Boilerplate – good way to learn HTML5 and CSS3
- tabindex=”-1″ allows JS to set focus and not bother user otherwise
- spellcheck=”true” | “false”
- itemtype=”http://data-vocabulary.org/Person”
- new input types
- placeholder, pattern, required, spellcheck, validate
- a@b is deliverable for internal email servers. hmm.
- meter, progress, output widgets
- HTML5Rocks
- button generator at css3button.net
- Voronoi diagram demo
- aquarium.js
- web workers

Monday Lunch

Benjamin, Ubuntu
- loves CloudFlare
- likes Linode
- nimbula

Talked to an open mapping data fellow about various projects. Google ToS is scary when it comes to that kind of data.

Monday Afternoon

Moose is Perl: A Guide to the New Revolution
Ricardo Signes, Pobox.com
Slides

- detailed talk about Moose features and syntax
- chatted with other folks at break time about topics like CSS::Inliner and Mojolicious web framework (with minimal dependencies) by Sebastian Riedel.

Tuesday Afternoon

Three For Five – Functional HTML5 & CSS3 for Designers & Developers
Jason VanLue, Envy Labs and CodeSchools.com

- good training session with fun sample – a beer menu created from 1 photo (CSS3 text scaling and rotation) and HTML5/CSS3 styled text
- training class is available online for $75 (also jQuery and 2 Rails classes)

Click to Enlarge

Tuesday Night

- went to Puppet Labs office for CloudCamp lightning talks, which started about 90 minutes late
- nice office, typical start-up look across from a small park. Comfy little meeting rooms with leather sofas a la Netflix.
- about 5 lightning talks total, 2 were sales pitches, 2 had 40 slides crammed into 5 minutes. ick.
- Puppet Labs CEO gave a good talk on optimizing Puppet for a client with 10,000+ nodes. Converted XML::RPC to REST, which doubled performance from 500 to 1,000 qps (I talked to Randy Ray about that, and he wasn’t surprised and that would be the case on simple requests), did some more work and maxed out at 2,500 qps. Enabling SSL did not slow down requests.
- got too crowded for me, and also fire department, who manned the exits and counted people as they entered and left.

Photo credit: Jacob Helwig

Wednesday

Creating a Scalable JavaScript Application Architecture
Nicholas Zakas, NCZConsulting
Slides

An AJAX client only cares about getting the data it wants, not response codes, etc.

Use layered JavaScript client architecture:

- sandbox
- application
- library (Dojo, YUI, MooTools, etc. )

www.nczonline.net
@slicknet
Author of “High Performance JavaScript”

Wednesday Lunch

I talked to Ben Golub, CEO of Gluster.

- 80% business, 20% scientific
- users include box.net, Pandora.com
- written in C
- minimum is 2 nodes for replication
- lots of people use it in EC2
- office located in Sunnyvale.

Wednesday Afternoon

HTML5: All about Web Forms
Estelle Weyl, standardista.com

- use label tag with forms to ease navigation for end-users
- use placeholder attribute, better for screenreaders than JS coding
- multiple autofocus defaults to last one in HTML5
- type=”text” is default, so tel, email, etc. degrades on all browsers back to text
- form element can disassociate parent form, useful for AJAX multiform pages
- input types good for mobile devices to show useful soft keyboard for url or email input types
- numeric step options
- test date and numeric input types for usability. Scrolling birthdays or zip codes is painful
- still need JS
- Opera is first with new UI features but last with artistic design, so currently has hideous tooltip appearance
- list and datalist like exploded select. Include select for IE backward compatibility
- meter, progress and output UI elements
- input type=text x-webkit-speech, now on Google homepage

HTML5 in Your Pocket: Application Cache and Local Storage
Scott Davis, ThirstyHead.com

- 4 million Macs, 32 million iDevices in last quarter
- Basecamp Mobile
- “Dive into HTML5″ by Mark Pilgrim online
- cookies should be called thimbles, only 4k
- HTML5 localstorage supported in IE8, FF 3.5, so practically all
- 5 MB, QUOTA_EXCEEDED_ERR, can’t increase now
- StatCounter browser stats
- gist 350433: Storage polyfill using window.name and cookies
- no version of IE or FF support web SQL and they probably won’t, FF for philosophical reasons
- cache manifest
- genManifest bookmarklet
- FF about:cache and Firebug are handy to see caches
- appcache has no expiry date
- date stamping manifest file causes re-download
- 404 causes none to be saved
- treat appcache as only slightly more secure than cookies, which are round-tripped
- webplication
- still sandboxed from local file access, could use node.js or signed app
- See W3C HTML5 offline

Profiling and Detecting Bottlenecks in Software
Bryan Call, Yahoo!/Apache Committer
PowerPoint .pptx

- usual savings (machines, moving parts, get smart)
- top, htop
- vmstat, dstat
- time cmd
- Boost logging does small writes, allocates memory when it gets behind, causing both IO and memory pressure
- profilers like oprofile and google profile cause 1% to 8% slowdown
- valgrind’s callgrind much more resources
- oprofile has script to convert output to kcachegrind
- opcontrol –deinit
- sysctl nmi_watchdog off
- opcontrol –no-vmlinux
- opcontrol –daemon
- google profiler userland, LD_PRELOAD
- env CPUPROFILE=/tmp/mybin.prof /usr/local/bin/my_binary_compiled_with_libprofiler_so
- caching: don’t do the same work twice
- choose the correct algorithms and data structures: dqueue vs. List, hash vs. trees, locks vs. r/w locks, bloom filter
- reuse memory, stack vs. heap, tcmalloc
- make fewer system calls (larger reads and writes)
- faster hardware, bonded NICs, SSDs, RAID, CPU, more cores
- read How to Profile Apache Traffic Server
- bootchart
- http_load now uses epoll
- he made ab multi-core

From kernel.txt: “nmi_watchdog: Enables/Disables the NMI watchdog on x86 systems. When the value is non-zero the NMI watchdog is enabled and will continuously test all online cpus to determine whether or not they are still functioning properly. Currently, passing “nmi_watchdog=” parameter at boot time is required for this function to work. If LAPIC NMI watchdog method is in use (nmi_watchdog=2 kernel parameter), the NMI watchdog shares registers with oprofile. By disabling the NMI watchdog, oprofile may have more registers to utilize.”

CoffeeScript: A New Hope for JavaScript
Scott Davis, ThirstyHead.com

- trainer, author, worked on Comcast/Time Warner TVs which mostly use WebKit
- little language that compiles into JS
- JS V8 headless, like node.js
- PhantomJS is headless HTML, handy for testing
- Google GWT compiles Java to JS
- “transpiler”
- install node.js
- install npm
- npm install -g coffeescript
- –tokens, –nodes like java p
- immediately invoked function expression IIFE
- coffeescript: string interpolation #{name}, “”"
- objects with left-hand spacing like python

Thursday

DNSSEC @ Mozilla
Shyam Mani, Mozilla Corporation
Slides

- BIND 9.7 is nice for DNSSEC
- Keys are everything, protect them. Have a backup plan.
- Cisco core routers by default don’t expect large DNS transfers:
policy-map global policy class inspection_default inspect dns maximum-length 4096
- DS was live, no signed zones
- watch log levels, can be chatty and quickly fill disk with logs
- DNSSEC has no immediate benefit to end-users, since resolvers don’t honor it
- their logs show 1000:1 dns vs dnssec queries for last 6 months, but growing
- IOS Firewall DNSSEC

Ask Google Engineers Anything
Chris DiBona, Google

- 55 Google employees attending OSCON this year
- mostly end-user questions about Google+ circles and API
- or running Go on android
- or why does my telco not do firmware releases for my smartphone
- or not happy with Google search results this month
- I asked about original reason for GFS. Originally, the hardware was really that flaky, and Google even actively bought bulk refurbed computers and RAM, sometimes off the back of a truck. Got a gopher plushie in return.
- also some good feedback complaints: google groups UI inadequate for managing 350 groups in an Education scenario
- inadequate data import tools for non-profit users of groups, mentioned by a religious charity.

How Not to Design Like a Developer: Open Source Can Look Good Too!
Chrissie Brodigan, Mozilla/Firefox

- KPI vs. git (different goals)
- @sirupsen
- story about the downloads map graphic for FF 4 – a developer silently removed social button graphics, limiting participation of wider audience. Marketing needs to explain why and how other staff fit into outreach programs.
- hang out on #projectdesign
- design contests are a good way to get them to come out of the woodwork
- designers hang out on twitter, not irc
- programmers should avoid big red buttons that scare users, and improve accessibility
- Inkscape, Blender, HTML and CSS are some Open Source tools for design mockups
- do AB testing or survey users
- designers want to be martyrs, so be careful they don’t offer more than you are willing to accept (start with 1 icon rather than the whole set)
- take a look at graphics libre for icons
- Quitestrong.com 5 girls who do design

Gearman BOF
Brian Aker

- Gearman polls, beanstalk busy waits
- nice to have feature to give up to another thread
- monitor projects handle launching of workers
- Gearman has durable and non-durable queues
- is a superset of the crap you handrolled. Most of the homegrown apps peak at 50% to 60% of Gearman’s features
- Gearman is production ready, but the postgresql driver less so because of fewer test cases and Brian’s lesser familiarity
- setup ntp and use Gearman coalescence for redundant cron servers
- can inspect queue
- agnostic to backend
- 99designs.com looking at this, same use case as original developer
- I still think that if you already have a database app, adding a status column gets you a lot of Gearman functionality without one more moving part.

MariaDB BOF
Monty Widenius, MontyProgram AB

- Monty mentioned that the latest release of MariaDB 5.3-beta has faster replication from group commit and performance improvements on the master, which also help the slave. Also subqueries and joins work much better.
- Monty talked about his Aria storage engine, which is a replacement for MyISAM that has both transaction and non-transaction modes. It’s intended for users who want the space savings of MyISAM. Over time it may compete with InnoDB.
- Monty’s responsibility is to convince Percona to merge into 1 source base sometime
- it’s estimated that although Oracle still has the InnoDB team, they may only have 1 general MySQL server programmer left.
- he explained that MontyProgram developers work 50% on feature requests from end-users, and 50% Open Source-related. So paid requests for 1 week of work really need to cover 2 weeks of developer time for that model to work. Typically a medium-sized change is roughly $12,000 and includes development, testing and documentation.
- Zmanda got FLUSH TABLES WITH READ LOCK NO CHECKPOINT for a beer, though. Monty was able to find a code path that could be locked to prevent Aria and MyISAM from writing anything during the backup.
- I sponsored WL#232 for USD$100 to add a SHUTDOWN statement to MySQL
- Monty explained that MERGE tables may be a better choice than MySQL partitions for logging applications.
- attendees from MontyProgram, SkySQL, Percona, DeNA

AskMonty: MySQL “Wishlist” Session from an online travel agency
MySQL Manual 5.5: FLUSH Syntax

Perl Lightning Talks
Hosted by Geoff Avery

- a talk on why arrogant community members telling others that “they need a thick skin” is unhelpful
- a talk by a young Perl community member on getting commit access, and how others can get the spirit and contribute
- Larry did several talks, mostly encouraging backporting Perl6 features to Perl5 it seemed, perhaps as a replacement to going Moose
- nice song on the importance of public libraries, which face shutdown due to economic budgeting problems in Australia and USA
- nice comedy juggling act comparing programming languages. Perl6 was omitted as “nothing has been updated in 5 or 6 years”, prompting Larry to say that he was happy he has a thick skin. See above.
- afterward, I talked to a booking.com rep about why a European company needed to actively recruit in USA and world-wide. He said that European developers are happy working where they are now, and it’s easier to recruit in places with mobile workforces like the USA. He would like to hire a couple developers per week to meet their development schedule.

IMUG: The Power of Plain Text with Ken Lunde

Administrator — Fri, 20 May 2011 04:40:23 +0000

Dr. Ken Lunde of Adobe did a talk on “The Power of Plain Text” at IMUG tonite.

“This presentation will cover the basics of “plain text” and its importance in providing meaningful content, and will explore some recent developments in Unicode that allow otherwise unencodable characters, such as variant forms of CJK Unified Ideographs, to be reliably represented in a “plain text” paradigm through the use of the Ideographic Variation Database (IVD).

Examples from other scripts will also be provided. Various “plain text” pitfalls and bad-practices that undermine Unicode’s success, such as PUA usage, CJK Compatibility Ideographs, and code point poaching, will be touched upon.

Finally, emerging environments that thrive on “plain text” data, such as mobile, will be discussed.”

Some of the interesting points:

Adobe’s PDF creator has a 2 level scheme to separate text and presentation, but Microsoft’s PDF exporter only has 1. (I need to confirm the product names.)
keep up with Unicode versions, now 6.0. Several code points are added each year.
Unicode is not just code points, but also properties (Digit, etc.)
the Basic Multilingual Plane (BMP) is full, so apps need to support additional planes
testing HTML or SGML with numeric character references lets you control the input better as you test for rendering problems. The Unix od -h utility is helpful for examining files at the hex byte level.

Thanks to Adobe for hosting and broadcasting the meeting.

Character References Explained
wikipedia: Unicode

Sony the Multinational Cripple Gets a Diagnosis

Administrator — Fri, 06 May 2011 01:30:37 +0000

I know, I know … it’s not fair to pick on Sony being a multinational cripple and all.

But Slashdot nailed them this week.

See Poll Results and Comments

A few years ago I went to a lecture by Guy Kawasaki. I asked him, “What’s wrong with Sony? Is their string of failures part of some secret master plan, or just gross incompetence?”

I figured since Guy was both Japanese-American and a veteran of the tech industry, he would know.

His answer was that it was likely the latter, with no evidence to the contrary.

One Slashdot poster prolly got it right: the problem with Sony is that it’s not only a Japanese multinational, but it’s split into content and technology organizations, which are diametrically opposed/schizophrenic.

Content owners want to restrict how content is used by the consumer, while technology creators need to be at the forefront of liberating and enhancing consumer experience.

wikipedia: Guy Kawasaki

Google Translate API Programming Notes

Administrator — Sat, 23 Apr 2011 21:41:47 +0000

I decided to try out the Google Translate API (v1), part of the Google Language API Family this weekend.

The Google Translate API supports various housekeeping functions plus 2 main features:

language detection
language translation.

Although machine translation does not generally provide a high quality, polished result for arbitrary input, it can still be useful for more limited requirements.

For this project, I had 11 existing translations (Google Translate supports 56 languages) for a web site that needed a small incremental update of 65 short strings, and could compare the Google Translate results with a human-translated corpus for most of the languages.

Conveniently, the new strings included day and month names, which is easy for machine translation to get right.

The most important thing to do is to first read the Google Translate TOS first. There are several limitations and requirements:

you should register for an API key, provide a referer URL for the project, and provide the IP address of the requesting host
“powered by Google” must be displayed in any human-readable UI that relies on Google Translate
the maximum allowable input is 5,000 characters
Automated requests are prohibited; all requests must be made as a result of an end-user action.
All websites or apps which use Google APIs must be free of charge.

As a practical matter, you should detect a TOS error and stop submitting API requests, ie. responseDetails is ‘Suspected Terms of Service Abuse. Please see http://code.google.com/apis/errors‘.

JSON and REST are supported, so any programming language can be used. Google provides code samples in JavaScript, Flash, Java, PHP, Python and Perl. UTF-8 is the character set used.

I used the Perl sample code, fixed the string concat bug (!) in the first line, and enhanced it to comply with the TOS.

I found that no API key is needed if translation requests are throttled by 10 seconds each.

Also, you may submit input embedded in HTML, but the output translation can reorder the HTML tags, in some cases changing the final appearance. I noticed that anchor and strong elements were re-ordered in my results.

wikipedia: Machine translation
CLDR – Unicode Common Locale Data Repository
GeoNames.de – Languages of the World
Perl CPAN Module DateTime::Format::CLDR
Apertium Machine Translator API
Google to close Translation API service

MySQL Conference 2011

Administrator — Thu, 14 Apr 2011 23:50:23 +0000

I went to the MySQL Conference once again at the Santa Clara Convention Center.

Executive Summary: The talks and informal conversations centered around these main issues:

fragility of replication, and need for a global binlog ID and checksums
stalls in Innodb queue processing, resulting in query latency
general happiness with Oracle’s MySQL 5.5 release, considered by many to be the best release in several years
there were some Postgresql talks and the closing keynote was by Mike Olson, the former CEO of Sleepycat, so a segue is being made to an Open Source database conference in future years.

Some more detailed comments below …

MySQL Replication Tutorial

- read-only except for super-user and repl thread (cool for DW)
- slavereadahead, mk-prefetch

- do
– expensive queries
– non-deterministic functions

don’t
- lots of rows or network traffic

semi-sync available in 5.5
- load-plugin on master and slave

delayed replication in 5.6

stop slave;
change master to master_delay=60;
start slave;

See SQL_DELAY: !!!

mysql-sandbox

- now has options for also creating slave and cir

openark

- oak-get-slave-lag
- oak-online-alter-table
- oak-purge-master-logs
- oak-show-replication-status
code.openark.org/forge/openark-kit

MySQL Performance Tuning Tips

- sequence diagram time x flow
- show global status like ‘table_wait_%’
- on linux stats are per-process, not per-thread. mysqd is multi-threaded

- add a performance schema like Luke’s

Aspersa toolkit

- missed: iotop, dtrace, systemtap, sample “show processlist”

http://slideshare.net/MySQLGeek

Justin Swanhart

Diagnosing MySQL Replication Failiures
Justin Swanhart

- recommended to enable binlogs on slave for recovery and diagnostics
– ie. errant write to slave will have server-id=slave-id
- BBU and buggy journal file process common sources of “rolled back” master compared to slave
-use maatkit to identify and update slaves, note that checksum performance may depend on key distribution heuristics
- statement-based and row-based replication have different issues and bugs
- read the replication features 33 sections
- note there are still 37 replication bugs
- mixed innodb and myisam updates is a bad idea, overall ill-defined behavior (at least 3 changes in 5.1.x)
- CREATE TEMPORARY TABLE – bad idea, but at least use a unique name with tmp_ prefix to help the DBA

Automated, Non-stop MySQL Operations and Failover
Mats, DeNA & ngmoco

1) heartbeat + drbd

innodb-flush-log-at-trx-commit=1, sync-binlog=1 for HA, but kills write performance

2) Mysql Cluster – not innodb

3)

4) Google patch – with guid in binlog

- study relay log internals “at” and “end_log_pos”
- to catch up a slave, mysqlbinlog –start-position=101835
- actually does recovery of lost transactions
- disable purge of relay log after sql thread to preserve statements, but will fill disk, so need alternate way of deleting
- purge relay log blocks (90 seconds for 10 GB), so need hard links to do non-blocking delete outside MySQL process
- show full processlist is non-blocking way to show progress of sql thread on slave, unlike master_pos
- mysqlbinlog –base64=always, none have useful properties
- Tool: Master High Availability Toolkit
power off delays
- R610 5-10 seconds using telnet+DRAC
- DL360 4-5 econds ipmitool+iLO
- do not use 5.0.45 or lower version (end_log_pos is incorrect)
- drop user is graceful way of not accepting new app connections, but finishing existing transactions.

Drift

google

http://code.google.com/p/google-mysql-tools/wiki/OnlineDataDrift

http://goo.gl/YLr0s http://code.google.com/p/google-mysql-tools/
http://goo.gl/OZA6R https://groups.google.com/group/google-mysql-tools

we don’t monkeypatch slaves – we rebuild from a reliable source
problem with mysql permissions is cached data

SSD

XFS better with 4k blocks
mkfs.xfs -s szie=4096
mount -o nobarrier
multiple threads
Percona server or Innodb plugin or Mysql 5.5
Innodb mutex contention with 120 GB RAM, performs better with 26 GB

Linden Labs
20 TB SSD X25 MLC and SLC, one failed on POST

Interesting trick
–tmpdir=/mnt/ssd

FusionIO user for mysql db – disks on master, fusion ion on 8 slaves, very happy now, early days some driver updates

Expo

AWS RDS
- sweet features, UI, backup options
- still no way setup a remote slave or do root configuration-type commands

Schooner and Virident had booths.

Wed. Keynotes

Monty
MariaDB 5.6

Marten
Cloud

Brian Aker
Drizzle GA

MySQL State of the Art @ facebook

DB Operations
- typical DBA

DB Engineering Team
- develop code for MySQL, tools, fix bugs, port patches

DB Performance Team
- full stack solution hw to os to cache to db
- test
- improve

- Nov 2010
- 4 ms read, 5 ms write
- 38 GB traffic on database tier
- 13 million qps

- based on vanilla MySQL 5.1
- custom facebook build on launchpad
- heavily sharded, heavy replication

HW
- lots of generations
- many core, not cpu bounc
- local storage, no SAN
- some flash in production

Performance Ideas
- monitor long-term trends – why are inserts up 10% this week?
- monitor real-time problems – look at Top10, not aggregates
- monitor everything
- throughput vs. latency
– 95th, 99th percentile
- stalls (1s, 2s, 3s why)

Tools

- table statistics
- shadows
– async copy of live data on test machine for load testing for main API (not ETL stuff), starts with replication and breaks it
– percona has a tool called logjam
- slocket
– udp dump of slow queries from network
- pmysql
– parallel statement submit
– global show full processlist
– fast – thousands of servers in a second
- replication sampling
– run show full processlist in a loop, look at writes, generate % stats
- aggregated client error logs
- scribe is a distributed log processing system
- query comments
– every single MySQL statement ever
– FB has an API to insert this
– server, page, memcached info
– FB doesn’t use query cache, but could strip out comments to make it query-cache friendly
- backtraces are stored by id
- indigo (query monitor)

Stalls

- Dogpiles
- Temporary Slow Down
- Stall Tools
- dogpiled
- aspersa stalk, collect
- pmp
- gdb

stalls found
- tables extending
- drop table
- purge connection
- kernel mutex
- transaction creation
- lock creation/removal
- deadlock detection
- many more – servers don’t expect as well as you would expect

“MySQL has to do what Facebook wants. It is a submissive relationship.”

Efficiency

- increasing utiliization of hw
- memory to disk ratio
- finding bottlenecks
– disk bound normally
– sometimes network
– rarely CPU/memory bw
- poor application design
– leads to poor query design

Disk Efficiency

- normally disk found
– IOPS
- queue lengths
- Innodb page size
– default 16k (8k, 4k possible)
- smaller for flash?
- our tests range scans vs. point lookups
- need adjustable per table or index

(linux io request queue may be 128 requests, but each drive has 32 slots, so only 4 drives would be efficietly used)

Memory efficiency

- compact records
- clustered and covering index planning
- accessing historical data
– understand workloads
- full table scans
– ETL type queries
– mysqldump
– tune midpoint insertion for Innodb LRU

Flash

- many iops!
– awesome at random reads
– kind of small
- tested many vendors since a lot of mktg is misleading
– long-term performance
– reliability
– edge cases
- failure rates
- different use cases

Flashcache

- flash in front of disks
– slower disk usage possible
- write-back cache
- much more data storage
- able to utilize much more of flash card
- ver long warmup time
- open source

MySQL 2X Project

- flash is too fast, slave replication is too slow, so put multiple mysql instances per server
- use 3307, 3308, 3309 (skip 3306 to find legacy 3306 requests)

Options

- MySQL is never a solved problem
- better/new
- new hw
- new data centers

Visibility

- never assume
- add more metrics
- full stack
- more innodb info
- more application info

Replication

- lag is a big problem
- possible solutions
– better save prefetch
– maatkit version has problems
- tungsten parallel slave
- oracle parallel slave

InnoDB Compression

- originally was planned during 5.1 upgrade
- problems
– replication stream
- increased log writes
- performance in some cases
- goto db eng talk thu at 2.50 pm

env: CentOS/XFS, mysql build with debugging symbols on, xtrabackup

PCI
Percona

PCI DSS
- 5 companies created common standard
- offset losses to processors from providers, or revoked
- outsourcing credit card processing correctly makes you compliant (cc’s never go through your network or touch your disk)

- don’t do PCI unless level 1 or level 2 merchant since it’s not cost-effective
- worry more about your PCI auditor than the written docs

- design access around business processes
- only billing support and ops person needs access to chd

- securich
- MySQL 5.5 has PAM, to help with enforcing password policies

- be judicious with grants
- disable local_infile
- disable old_passwords
- set read_only=on on replication slaves
- enable secure_auth

#, name, service code, exp date can store
mag, cvv2, pin cannot store

- binlogs enabled on master and slaves
- read-only
- semi sync, use furthest ahead slave

http://usa.visa.com/download/merchants/cisp_what_to_do_if_compromised.pdf

Decider
Google

- host list
- discovers replication topology by polling hosts
- current master found based on topology, not dns
- mostly stateless
- locality to dc
- also RPC API
- python
- if decider is not online, no problem.

Sanity checks

- no more than N hosts should be down
– recommend N=1
– down slave could have most recent tx
– N < hosts/2 for quorum

Will be available in google mysql tools.

http://goo.gl/xAgzC

Thursday

- Keynotes

- Baron Schwartz, Percona
- Mike Olson, former CEO Sleepycat Software, now CEO Cloudera

- remarkable insight into overall state of business problems and databases, no specific technology tips
- consumer devices are generating so much location data that we have no data store big enough and won’t without designing for it.

- 3 Ignite talks (20 slides in 5 minutes)
Baron Schwartz: MySQL failure stats from Percona support db (Tops are replications, human processes)
Canadian dude
Josh Berkus: “Be Sexy. Go Down.” parodying the latest IT trends and unintended anti-scaling (Ruby, etc.)
– some of the younger attendees didn’t realize that the examples were real-world, like the Ruby mongrel anti-scaling, etc.

Chef Tutorial

- central chef server needed, whether yours or on opscode.com (free for 5 nodes)
- uses RSA keys

https://github/

Narada
Patrick Galbraith

- works on MySQL and Drizzle

Error Detection with MySQL Replication
Facebook

- uses stream cipher
- eventually to be Open Sourced

Closing Keynote

JC did a talk on MySQL @ Twitter with lots of statistics.

Once again, O’Reilly did a great job organizing the conference. Hope it continues for many years to come, preferably not during tax week.

Former MySQL boss: Code ‘in better shape than ever’ under Oracle

Fukushima Reactor Failures Comments

Administrator — Sat, 02 Apr 2011 07:41:05 +0000

Whenever anybody asks me about the nuclear reactor problems in Japan, my reply is this.

A few days after the tsunami, I saw a video of one of the reactors exploding, with a white high-explosive shock wave clearly visible above and outside from a hydrogen explosion, similar to bombing videos that I’ve seen from the Vietnam War.

To me that indicated major destruction inside the reactor, similar to a 500 pound bomb detonating, with all of the blast energy focused and contained inside.

So I tell people to assume the worst, and ignore any press briefings or anybody trying to minimize the scale of this nuclear disaster.

A lot of people will make excuses for this accident, and shrug off any analysis as “20-20 hindsight” as they have with the Great Housing Bubble.

But I wouldn’t use a data center at sea level next to the ocean, let alone build a nuclear reactor complex there. IT people have struggled for decades to increase data center reliability, so there is plenty of historical data on power continuity, generator failure modes, etc.

Locating the reactor backup power sources at or below sea level where they could be flooded was an entirely preventable design error.

The nuclear power industry has been tarnished by this event.

What can be done in the future?

use a standardized reactor design. Currently each reactor in the world is a unique snowflake – even within the same reactor complex.
review failure modes of each reactor (earthquake, tsunami, tornado, airplane, sabotage, power failure, cross-dependency)
have practical and comprehensive disaster plans
have adequate photographic and radiation monitoring in place
have adequate mitigation supplies in place – personnel, robots, pumps, cement, transformers, etc.
have a plan to nationalize the utility if they are unable to perform for a military-led recovery mission
transition to modern, more failure-proof, standardized reactor designs.

cnn.com: Video shows 48′ tsunami crashing into Fukushima nuclear site
msnbc.com: ‘Jumpers’ offered big money to brave nuke work (up to $5,000/day)
msnbc.com: Japanese workers braved radiation for temp job
wsj.com: Fujitsu Chief Looks at Japan’s Crisis
wsj.com: The Man Who Predicted the Tsunami
cbsnews.com: Ancient stone markers warned of tsunamis
cbsnews.com: Japan nuke workers at their limit, doctor warns
NRC chief questions blackout plans for US plants
wsj: Cores Damaged at Three Reactors (plus Reactor 4 damaged in hydrogen explosion)
Photos: When the Tsunami Hit Fukushima Daiichi
3 nuclear reactors melted down after quake, Japan confirms
AP IMPACT: US nuke regulators weaken safety rules
wsj.com: Design Flaw Fueled Nuclear Disaster
Frontline: Nuclear Aftershocks
Cabinet kept alarming nuke report secret – Fearful of scaring public, existence of document was denied for months

Honshu, Japan Earthquake and Tsunami

Administrator — Fri, 11 Mar 2011 10:46:40 +0000

Just monitoring the tsunami activity after the Honshu, Japan 8.9 earthquake:

“At 9:46 PM Pacific Standard Time on March 10, an earthquake with preliminary magnitude 8.9 occurred near the east coast of Honshu, Japan . (Refer to the United States Geological Survey for official earthquake parameters.)”

“This earthquake has generated a tsunami which could cause damage to coastal regions in a warning or advisory. The waves are expected to first reach Shemya, Alaska at 12:13 AM AKST on March 11.”

Measurements or reports of tsunami activity:

Location Lat. Lon. Time Amplitude ------------------------ ----- ------ ------- ----------- Ofunato Japan 39.0N 141.8E 0605UTC 10.8FT/03.29M Hanasaki Japan 43.3N 145.6E 0643UTC 09.3FT/02.82M

Note that there have been over 30 5.0 to 7.1 aftershocks, as well as a 6.5 near Bali the night before.

Crescent City, California, was hammered by at least three tsunami waves, including one reaching a record 8.1 feet.

“Each was a little higher than the last,” said spokesman Joey Young of Del Norte County’s emergency operations center.

The tsunami brought waves of about 6 feet to a harbor in Maui, authorities said, but other areas reported lower levels, including Honolulu at 2.2 feet and Hilo at 4.3.

wsj.com: Waves Hit Hawaii, California; One Man Swept to Sea

Documentaries on Japanese I-400 and Midget Submarines

Administrator — Sun, 09 Jan 2011 08:22:27 +0000

It seems like Japanese WW2 submarine documentary week on PBS …

“Japanese SuperSub” talks about the I-400, which was a 400′ submarine with a hangar large enough for 3 Seyron bombing aircraft, each with a 1,700-pound bomb and a 85′ runway with a steam catapult.

The internal hangar was 11′ wide, so the wings and tailplane had to fold back. The hull had a stealth treatment.

The I-400 was developed to attack the United States, and their first mission was the Panama Canal. That mission was apparently cancelled when Japan realized that it was too hard of a target for a handful of bombers to succeed.

The US captured some of the I-400′s during the surrender. This was an awkward capture as the Japanese military personnel did not know how to surrender, there was a language barrier, and the American sailors stationed on the I-400 were afraid of being trapped in the subs in case of a dive or scuttle attempt.

Two of the I-400′s were sent to Pearl Harbor after the war for analysis, and later sunk to hide the technology from Russia.

“Killer Subs in Pearl Harbor” talks about the 5 Japanese midget subs were used in the Pearl Harbor attack, with 4 lost fruitlessly, but 1 firing its two torpedos and entering West Loch. It has apparently been found 3 miles offshore.

Likely it was scuttled inside West Loch, but salvaged and dumped in the ocean during a cleanup after the war.

historynet.com: Japan’s Panama Canal Buster

SVLUG: Greg Lindahl on Blekko Search Engine

Administrator — Thu, 04 Nov 2010 05:39:36 +0000

Greg Lindahl from Blekko gave a talk tonite at Silicon Valley Linux Users Group (SVLUG.)

Blekko is positioned as an alternative search engine to the currently entrenched ones.

Some of the alternative features are:

slashtags UI feature for end users
enhanced SEO reporting, including incoming links to your competitor sites
currently no ads.

Some stats about Blekko:

positioned as “alternative search engine to Google” with public launch November 1.
project started 3 years ago
Blekko has 21 employees, 19 of them engineers
$20 million in funding. Ron Conway is an angel investor.
$5 million of that spent on hardware, including 700 servers and 5 PB storage (1/3 used currently) managed with Greg’s provisioning tools (a version Open Sourced in 1999). Nice monitoring.
hard to think of a good domain/company name. blekko.com was a domain name held by a founder, izak.com was another candidate (too boring.)
servers here and in New Jersey. Nice to have in Europe to reduce latency.
They use servers with Intel X25-M 160 GB SSD. Only 1% of available write cycles used after 3 months of operation.

They wrote their own “eventually consistent” cluster software called PM, as Hadoop and Cassandra were not awesome enough 3 years ago and they need to know every line of the source code anyway. He would like to Open Source PM, but that’s up to the board.

Perl/XS and C are used, no Java. They made a sharded architecture with 3 copies of data the goal.

Currently they crawl Top 3 billion pages, and need to expand hardware by 5x to crawl and store entire web. No rush.

To fill in more rare queries, they use Yahoo! Search BOSS (Google Custom Search Engine (CSE) is not available now except to large partners.)

They try to focus on core search technology and not get distracted by things which can be outsourced. An example might be to outsource an online calculator UI instead of re-inventing the wheel.

Each page has 100 metrics computed for it, including things like number of unique links per Class C and freshness.

Greg said that provisioning software not only needs to do initial configuration of servers, but also detect and repair changes. One concern is a rogue root process or user that can change anything.

Mahalo has banned them in robots.txt (Blekko’s crawler identifies itself as ScoutJet):
# ScoutJet has been ill-behaved in the past. # They'll fix it soon.. User-agent: ScoutJet Disallow: /

Blekko plans to crawl trending SEO-bait sites like ehow.com for statistical purposes, but not necessarily to link them in organic search results.

PowerSet’s algorithms/calculations were too slow, lucky to get bought out.

Blekko might have used the wikipedia corpus for crawler testing earlier on, but the live Internet is used for everything now.

Next stage is to work on UI and other user features.

Thanks to Symantec for hosting the meeting once again.

techcrunch.com: blekko
azreporter.com: Blekko’ ScoutJet Begins Web Crawl