James’ World

I attended the O’Reilly Open Source Conference, once again in Portland, Oregon.

Overall my impression was that the talks and vibe were oriented towards Web 2.0 primarily.

I would say that the talks were not as strong as previous years, but it’s easy to compensate for that with the “hallway track” and access to the original Open Source authors.

Several attendees used the EEE sub-notebook computer, and were happy with it as a email/browser tool.

Wednesday

PHP Taint Tool: It Ain’t a Parser

- CS’y effort at PHP parser for code analysis, reminds me of early days of Perl’s B tools
- not suitable for end-users

Write Beautiful Code (in PHP), Laura Thomson, Mozilla

- good general background on good programming practises
- not a lot of specifics about PHP, but available for questions

Hypertable, Doug Judd, Zevents

- HyperTable is a clone of Google’s BigTable, from public paper
- room was packed, some turned away
- still alpha, maybe beta in August
- preferred distributed filesystem is HDFS, works with others
- I recommend reading web site and then looking at the curt slides
- plans to do benchmarks with same hardware as Google has published.

Open Source Virtualization for People Who Feel Guilty About Using VMware So Much, andy michelle, EDA

- cute talk about VirtualBox, Xen and VMware
- Xen has weird nomenclature compared to other tools
- VMware wins on tools and polish
- showed screenshots of unreleased and alpha mgmt. tools.

Barely Legal XXX Perl, Jos Boumans, RIPE

- stunning and twisted example of overloading, short-circuiting, import-faking, whatever it takes to make a loaded module do something other than intended
- illustrates great flexibility of perl, for good or ill
- could be useful for things like testing harnesses, etc.
- motivated to win bet of $100 or 1 vertical meter of beer
- said it took 3 or 4 hours to complete.

I walked around the exhibits area.

Got a demo of Atlassian’s continuous integration (CI) tool, Bamboo. They’re also the vendors of JIRA issue tracker and Confluence wiki, which I’ve used before.

One company had a public Wii game happening.

Thursday

Scaling Databases with DBIx::Router, Perrin Harkins

Ultimate Perl Code Profiling, Tim Bunce (Shopzilla)

- talk and screenshots about NYT perl profiler

The New York Times Perl Profiler

Top 10 Scalability Mistakes, John Coggeshall (Automotive Computer Services)

- good overview of writing high-performance, maintainable Internet systems
- interesting opinion that scalability is not just about increasing performance. scalability can be about scaling up or down, performance or maintainability, etc.
- recommended php.ini settings list

Perl Lightning Talks

- popular with audience, attendees seemed to like all the talks
- Mail::ESMTP looks very interesting for testing and production

Code is Easy, People are Hard: Developing Meebo’s Interview Process, Elaine Wherry (meebo)

- struggled to find time, right approach to interview new candidates in 1996, likely at behest of VCs
- external recruiters hit-and-miss, conferences and jobs email link useless
- phase where non-founder employees doing interviews wanted a founder involved in interview process
- trying to preserve culture (finger rockets, social networking, 2 female founders, etc.)
- came up with process involving reading resumes, phone screens, and office “sim” that adds a new candidate within 3-6 weeks
- “sim” has 3 versions: office manager (plan to erect a meebo office sign), front-end engineer (write a JavaScript app), and back-end engineer (write a server) in 4 hours
- current goal is to keep interview time down to 8 hours per candidate over 10 days
- now up to about 40 employees
- my feeling was that their hiring process started off clueless due to inexperienced mgmt. and is still oriented towards junior engineers. Silicon Valley is full of expert engineers and it doesn’t take 8 hours to interview them.

BOF

mysql-sandbox

Giuseppe Maxia discussed and demoed his very useful mysql-sandbox utility for managing several versions and instances of MySQL on the same machine.

He wrote it for his testing work at MySQL AB. Very well received by attendees. This is a great example of what I call “anti-virtualization” - using ports instead of resource-intensive VMs.

MySQL Conference 2008 Presentation

State of the Onion Address, Larry Wall

- talk about Perl6, random anecdotes, etc.

Friday

Open Voices, Jim Zemlin (The Linux Foundation), Keith Bergelt (Open Invention Network), Karen Sandler (Software Freedom Law Center), Phil Robb (Hewlett Packard)

- panel discussion of various free software efforts, some little-known

An Illustrated History of Failure, Paul Fenwick (Perl Training Australia)

Paul gave an interesting talk on notable Software Failures and estimated a price tag for each. I had heard news reports of many of them, but it was interesting to hear an updated analysis of what really happened behind the scenes.

Thanks to Google for sponsoring the fairly good almost-gourmet lunches. Sure beats the O’Reilly lunchbags from the dot bomb days. (Everybody I know bailed and found a subway shop back then.)

Notes

- Burgerville popular with attendees, can upgrade combos to a shake.
- Red Lion hotel has a small cardio gym with 1 universal machine, no free weights, open til 11 pm
- WiFi password changed weekly, in middle of remodel, lobby just finished.
- There is a 24-Hour Fitness that is actually open 24 hours near downtown Portland. Has basketball court and 2-lane pool. $15 for non-member visitors.

OSCON 2008 Presentations

I’ve successfully used MySQL statement-based replication for several years across data centers and understand it’s quirks.

While at the MySQL Conference, I tried to see how DRBD could help the installations I manage, but I just can’t drink the DRBD Kool-Aid.

MySQL Replication Pluses

• Free
• Easy to setup if you already have a backup and master position
• No shared storage to manage or corrupt
• Light network load
• Can use master for r/w and slaves for r.
• can do maintenance on slave (ALTER TABLE, etc.) and failover afterwards
• works well across Internet even with high-latency
• many replication problems simple and hand-fixable

MySQL Replication Minuses

• Slaves can/will get out of sync with the master, typically noticed after a few weeks or with Maatkit
• Changing masters requires rebuilding slaves
• There is always some replication lag when there is a busy master
• no checksums or 2-phase commit

DRBD is a low-level driver to copy a disk partition in near real-time from a master to a failover node (cold standby.)

MySQL with DRBD Pluses

• Free
• No fsck or transaction log replay needed if manual failover.
• Slaves don’t need SET MASTER updated unless DRBD fails.

MySQL with DRBD Minuses

• DRBD partition corruption means failover node would be unusable (disadvantage of shared storage) and failback could destroy original master too.
• if the master panics, then after failover both fsck and transaction logs replay must be performed
• more work to setup initially than statement-based replication
• NIC and network corruption is also propagated.
• Failover node is a cold standby, cannot accept database traffic if that would change the DRBD partition
• Could generate a lot of network traffic.
• cannot do maintenance on cold standby database
• 2 heartbeats needed on a reliable, local network

I can see how MySQL/DRBD would be appealing for those who operate on a reliable network and don’t need Master-Master for load or maintenance, or who have many slaves that cannot easily be rebuilt.

Eric Bergen: DRBD in the real world.

Since IO accounting was added to the linux kernel in 2.6.20, it’s been possible to examine IO per task.

Guillaume Chazarain’s iotop.py takes advantage of that to show disk IO in a format similar to the venerable top program.

It’s unlikely yum install iotop will work on your older linux distro yet though …

Prerequisites for running iotop.py are Python 2.5+ (to preserve your existing python install, do make altinstall) and linux kernel 2.6.20+ with IO accounting enabled (TASKSTATS and TASK_IO_ACCOUNTING.)

make menuconfig

General setup --->
[*] Export task/process statistics through netlink (EXPERIMENTAL)
[*] Enable per-task delay accounting (EXPERIMENTAL)
[*] Enable extended accounting over taskstats (EXPERIMENTAL)
[*] Enable per-task storage I/O accounting (EXPERIMENTAL)

iotop.py accepts several command line options for filtering, including PID, user and process/thread-view. Once the program is loaded you can use the keyboard arrow keys to change which column gets sorted.

Guillaume Chazarain’s blog
Linux::Taskstats::Read Perl module
Tricks to diagnose processes blocked on strong I/O in linux
DTrace iotop
DTrace iotop samples
Running DTrace from Solaris Against Linux in Brandz
Fedora Daily Package: iotop - Display I/O Activity by Process

I was reading an interesting samba mailing list comment about using rsync on live MySQL databases.

The author said this:

“Assuming a short break in accessibility is tolerable, I’d

1. run rsync to the backup
2. stop the server
3. run rsync to the backup (should be much much faster now)
4. restart the server.”

Combining rsync and mysqlhotcopy we can get a little fancier:

Ghetto Innobackup-style backup with rsync

1. STOP SLAVE; FLUSH TABLES
2. run rsync to the backup
3. FLUSH TABLES WITH READ LOCK; SHOW SLAVE STATUS;SHOW MASTER STATUS
4. run rsync to the backup (should be much much faster now)
5. UNLOCK TABLES
6. START SLAVE

Note that the read lock and unlock must be done while on the same database connection, and innodb continues to update indexes even when read-locked.

Also, record the master and slave status values. They may be very useful later if you want to apply binlogs to the backup, or initialize a slave.

This technique would be very suitable for non-critical snapshots like QA copies and on quiet databases.

It may be suitable for busy databases if other methods aren’t working out, for instance you don’t have LVM snapshots setup and innobackup is locking your MyISAM tables too long.

rsync -a is also useful for backing up master binlogs every 5 minutes on a live site. Normally you’re better off setting up a slave just running the slave IO thread, though.

Many databases have features to allow “log shipping.” With MySQL, similar functionality is accomplished by doing FLUSH LOGS and rsync, or using replication (there is a command to not execute the replication stream, just save it to disk.)

FLUSH NO_WRITE_TO_BINLOG LOGS
FLUSH TABLES WITH READ LOCK

So Amazon EC2 registration is open again. Interesting … I’ve done colocation and dedicated hosting, time to do elastic cloud computing I guess.

The upsides with both dedicated hosting and EC2 are:

• no data center to provision and manage
• just whip out your credit card when you need to expand

With EC2, you can expand and contract even more easily than dedicated hosting.

The downsides with EC2 that I’ve seen mentioned include:

• lack of local persistent storage
• lack of a permanent IP address
• pricing is still a little more than dedicated hosting.

Amazon EC2 Release: Introducing New Instance Types
Forum: Amazon Elastic Compute Cloud (Beta)
I Hate Cubicle Blog with articles on EC2
Slashdot: Amazon EC2 Open To All

Cleversafe updates distributed storage idea

firstserv.com: Oracle 10g Shared Hosting
bluereef.net: Oracle 8i Shared Hosting
Amazon storage ‘cloud’ service goes dark, ruffles Web 2.0 feathers
GoGrid by ServePath

Another good talk at the Silicon Valley Linux Users Group, this time by Tapio Tolvanen of Nokia Finland. He talked about the hardware and software (Tablet OS) behind their excellent PDAs, the 770 and 800.

The Nokia 800 is a Debian distro using GTK, SDL and Pango for i18n. It can run C, C++ and Python. The 800 has host USB, WiFi and BlueTooth. It plays video well using mplayer, runs Opera 9.5 and a Skype client.

Nokia’s plan is to get the 770 and 800 into the hands of developers to prepare for a larger consumer market.

Newer versions of Internet Tablet OS don’t work on the older 770 as the hardware is different.

I have a feeling that the 800 will have stiff competition with the $399 iPhone.

Thanks to Symantec/Veritas for hosting the meeting.

wikipedia: Nokia 770 Internet Tablet
wikipedia: Nokia 800 Internet Tablet
nokia770.com

Some users prefer the 770 because the battery lasts twice as long as the 800. Some hours are building 6.5 Watt/hour external battery packs to compensate.

I gave up on Defcon after Defcon 11 because of the venue overcrowding at the Alexis Hotel. But since I had no other plans this weekend and they moved to the larger Riviera hotel last year, I decided to give them another shot this year.

What an improvement! 7,000 attendees and 5 talk tracks, yet now fairly well-organized. I registered Thursday at 9:30 am for $100, but they had already run out of the awesome white animated SMT LED badges with a “HUMAN” cut-out conference badges, and programs.

They should have done a photocopy run of the schedules, but didn’t, which is bad since the schedules are not posted outside the lecture rooms.

Security staff in red shirts called “goons” vigilantly enforced access points and fire codes. I was told that lecture room exits had to be clear in case of sudden emergencies like a smoke bomb attack.

The hotel was big enough that the attendees didn’t swamp the hotel, and the rooms for Tracks 2, 3 and 4 were usually big enough. The Track 5 room and the closing ballroom were too small. Hotel security seemed happy and stayed in the background.

The hotel coffee shop was very busy but served ok food ($10 for a hamburger and fries.) The upstairs dinner buffet was a good deal at $16. Good variety of fresh food and desserts, including prime rib, mexican and asian food.

I didn’t bring a notebook computer this year because it would just get hacked if I turned it on, and I also left my Blackberry off. I understand that some people buy a computer at Fry’s and return it after the weekend to get re-imaged, or use it as a honeypot. Some people did use their work Blackberrys with bluetooth disabled. One guy had a Nokia 770 wifi PDA that he planned to re-image after going home. Nice screen with scalable fonts.

The talks that I attended on Thursday and Friday were very strong, usually presented by the original researchers or somebody deeply involved in the topic. Defcon talks are unusual in that most audience questions are held until after the talk in a separate Q&A room. Not my preference, since expert attendees often get more out of the questions that the talk.

Thursday

Thomas Holt: The Market for Malware

Insight into mainly Russian malware industry:

- pincher programs for intercepting username and password data
- joiner programs to bind pincher program payloads with images or downloads
- like to be paid with e-gold, don’t like Western Union
- like ICQ, irc
- tools cheaper for other Russians to purchase than foreigners
- forums for promoting and rating developers and programs
- good authors provide good customer support, upgrades ($10), manuals and customization ($30)
- admin UI programs very polished and professional looking - some are even skinnable.

Pilgrim: How to be a WiFi Ninja

Pilgrim is the real deal - he knows how wifi works, owns a wifi accessories shop in Florida and is a perennial show vendor.

He gave tips on improving wifi transmission and reception:

- thinner cable is lossier, so keep under 10′
- cable is optimized for Channel 6
- wifi signals transmit better in drier air
- most omni AP transmitters can be made more directional by using a tin-foil reflector behind them, preferably parabolic shape
- used Dish satellite receiver antenna could be very useful
- made a wok dish antenna and recommends it
- recommends USB receivers over PC Cards because of external antenna
- recommends USB cable run to smart antenna instead of long runs of cable
- transmit power isn’t everything. try to balance transmitter, receiver, cable and geometry
- re-orient AP antennae to get better vertical or horizontal reception, especially in 2-storey buildings. same when war-driving.

Broward Horne: Click Fraud Detection with Practical Memetics

Broward gave a great talk.

He has the website RealMeme.com and does experiments in web site promotion and Internet mindshare. He left some blog comments on Casey Serins’ IAmFacingForeclosure.com website but received no traffic to his site initially. He posted a comment inquiring about that, got a bunch of traffic, and upon log analysis realized that it was bot traffic, implying that Casey was involved with bots for AdSense click fraud.

He showed some graphs of discussion activity before and after the Pope’s death, which expanded the bandwidth of discussion, and the SARS outbreak, which barely registered.

D.J.Capelis: Virtualization: Enough holes to work Vegas

Awesome talk on how pathetic x86 virtualization is from a security perspective.

He talked mainly about VMware Server and Xen, but problems generally applicable are:

- vulnerable to physical attacks and DoS at PCI level for shared hardware like video, network and drive controllers
- vulnerable to IP and MAC address changes
- vulnerable to practically undetectable covert channels between VMs
- vulnerable to timing attacks similar to the Intel HT ones
- all the image migration tools use plaintext, possibly across ethernet
- any rogue partition can violate all other partitions, subverting your firewall and network security
- bad default configurations, as documented.

The expensive VMware ESX product fixes a few but not all of the above problems.

He’s hoping IBM can leverage their 30 year virtualization experience on LPARs to do a good job.

He released a script to somewhat improve the default security configuration of VMware Server.

Dave Josephsen: Homeless Vikings, (short-lived BGP prefix hijacking and the spamwars)

He presented a history of spam and countermeasures timeline with commentary (he likes content filtering and thinks anything else is just a pointless technical arms race that can’t be won by the good guys).

Then he talked about how BGP can be used by spammers to spoof address blocks or commandeer unassigned IP space, likely the same techniques used by intelligence agencies now.

Gadi Evron: Webserver Botnets

Peter Gutmann: The Commercial Malware Industry

Some repetition of the talk at 10 am, but with more detail.

Daniel Peck & Ben Feinstein: CaffeineMonkey: Automated Collection, Detection and Analysis of Malicious JavaScript

They demonstrated some utilities for de-obfuscating javascript malware and presented some graphs that illustrated how malware and legitimate javascript profile very differently.

Also, they talked about spidering and analyzing some web sites and being surprised at how clean myspace is for example - no JavaScript malware found, probably a credit to their staff.

atlas: Remedial Heap Overflows: dlmalloc style

atlas did a Linux Buffer Overflow 101 class.

He used python to inject the shell code.

All the hotels surrounding the Rivera were full, so I stayed in the Hilton Vacation Getaway Hotel, a moderate walk from the Riviera. My $169 room was a very nice and new suite with a 30″ HDMI plasma TV, jacuzzi, shower, bedroom TV and laundry. The downstairs deli tuck shop is very complete and you can order custom sandwiches there. There is also an outside grill with $5 hamburgers and $4 hotdogs that’s open for lunch. The basement business center is 24 hours and has computer rentals and printing for $1/page. The hallway vending machines have $1 sodas.

Friday

Brendan O’Connor: Greater than 1: Defeating “strong” Authentication in Web Applications

Excellent talk reviewing US online banking so-called strong authentication, then attacking it.

- in-person banking is 2-factor authentication (something you have (card) and something you know (PIN)
- online banking is not 2-factor (you know a PIN but normally you don’t provide card, token or biometrics)
- browser fingerprinting is pointless because everybody buys the same configurations from Dell or HP
- browser fingerprinting is pointless because the implementations are bungled (commented source, little effort)
- banks should display all recent logins, not just the last one
- bank should not star out account numbers, then display the full check thumbnail!
- bolt-on auth systems from 3rd-party vendors weaken overall security and increase the attack surface
- SiteKey is worthless, since they have a limited image catalog indexed by alt tag
- knowledge base questions based on public databases as implemented now are worthless, but could be improved by displaying the same question until correctly answered and not randomizing choices

He finished by demonstrating a MITM attack by writing a newbie-level Perl program to relay the browser fingerprint, setting up Defcon Bank and doing a MITM attack on his personal bank which uses Sitekey.

David Byrne: Intranet Invasion With Anti-DNS Pinning

He discussed DNS pinning issues with IE and Firefox, pinning in Java and also how LiveConnect in Firefox and Opera reduce pinning. Also he showed how to use an exploited browser as a web or socks proxy and talked about using the socket capabilities in Flash 7 and above.

In his demo, he owned somebody’s browser, ran Nessus 3, and started a shell.

Billy Rios & Nathan McFeters: Biting tha Hand that Feeds You - Storing and Serving Malicious Content From Well Known Web Servers

Billy and Nathan are the reason for the recent Firefox 2.0.0.5 and 2.0.0.6 updates.

They talked about:

- XSRF
- serving warez from webmail hosts, in particular Yahoo! and gmail, because they’re free anonymous accounts, have a large storage capacity, good network bandwidth, high-reputation domain names, and plausibly deniable.
- domain substitution
- what can you trust on the Internet? only the domain name in your location bar
- Flash settings XML config file
- browser scheme and %00%00 filetype handlers
- IE 7 and Firefox URI hand-off exploits
- possibly KDE registry might also be vulnerable to filetype handler issues.

The award ceremonies went on 2 hours. It was interesting to learn about the whole Defcon community: security, logistics, press, events, speaker coordination, etc.

40 hardware kits were handed out to people wanting to hack the badge, but only 7 submissions resulted. The 2 winners built a graphical, gray-scale multimeter and a pong game. DT suggested it would be hard to top the badge next year, unless it was converted to a fibrillator or laser beams were added.

A 17 year-old won one of the lock-picking categories. The overall winner mentioned preferring home-made tools.

Some of the award winners received a Black Badge - good for free life-time show admittance. Some got a used notebook or Dish receiver.

The trivia show winning team was booed for their poor result - sometimes needed a dozen clues and still getting the wrong answer - but still got black badges. Next year there will be a pre-qualifying test.

I took one of the airport shuttle buses back to the airport. Depending on how you look at it, either I got a free ground tour of Las Vegas, or they wasted a half hour of my time trying to find a passenger who booked 24 hours in advance but didn’t show up on time. Eventually they found him … back at the Riviera.

theinquirer.ne: How to break forensics software
GData: An Online MD5 Hash Database

I was talking to somebody at OSCON last week and they made an off-hand comment about using FreeBSD jail to isolate software applications in a development environment.

Being a linux guy, it took a while to sink in that jail was a form of operating system virtualization.

On linux, there’s dozens of virtualization options these days in various states of development, reliability and licensing. There’s only 3 linux “OS virtualization” options, FreeVPS, OpenVZ and Linux V-Server.

Generally linux virtualization requires a custom kernel with the relevant patches, so installation requires installing the kernel, rebooting and testing the new kernel - usually a hassle.

FreeBSD has jail, and Solaris has zones.

What’s great about jail is that:

• limits access to filesystem, process space and network
• it’s been built-in since FreeBSD 4.0 - no-hassles, mature feature
• a new jail can be created with one command
• it’s fast - native, not emulated
• it consumes almost no resources - only 6-10MB hard disk space per jail
• it’s simple to use and requires no OS reinstallation or configuration - great for remote boxes
• it’s free.

The downsides of jail are that you can’t run different versions of the OS than what was originally installed, there’s no way to limit RAM or CPU used in each jail, and demons should be configured to listen on the appropriate IP addresses instead of 0.0.0.0.

Originally adding jail support only required changing about 350 lines of kernel code and adding 400 more - not that intrusive.

Linux-VServer
wikipedia: FreeBSD jail
Virtualization with FreeBSD Jails
wikipedia: Comparison of virtual machines
ibm.com: Virtual Linux - An overview of virtualization methods, architectures, and implementations
Virtualization in Xen 3.0
Installing Linux-VServer
New virtualization option for Linux: KVM (and Linux virtualization summary)
KernelTrap: Abusing chroot
Related Projects to jail and vserver