Archive for April, 2009

Using oprofile with the MySQL Server

Wednesday, April 29th, 2009

Monty Taylor from MySQL Consulting gave a Percona Performance Conference talk on using oprofile, the linux kernel profiler, to diagnose MySQL server performance issues.

Installing oprofile requires the oprofile package (yum install oprofile), and optionally /boot/vmlinux for kernel symbols.

I was surprised at how useful and easy it was to get useful info out of oprofile, in his examples regarding memory allocation and type conversion performance problems (typical with newbie PHP applications.)

You can do near real-time mysqld library profiling with commands like:

# opcontrol --start --no-vmlinux
# watch -n 1 "opcontrol --dump && opreport -l mysqld | head -30 ; opcontrol --reset"
# opcontrol -h

Here is an example of a table scan on a MyISAM table on a busy production box, with the terminal display updated once per second:

samples % symbol name
1182 15.4753 _mi_rec_unpack
795 10.4085 my_hash_sort_simple
671 8.7850 my_strnncollsp_simple

Note that was 1,182 calls to _mi_rec_unpack per second!

oprofile does add load to a server, but that should be ok even on a fairly busy server. opcontrol loads a kernel module, so there is a small possibility it could crash the kernel.

Below are two useful IBM Developerworks links to advanced oprofile techniques related to scheduling performance:

IBM Developerworks: Using oprofile to profile L2 data cache misses
IBM Developerworks: OProfile analysis: Branch misprediction

Another powerful profiling tool is Sun’s DTrace. Most MySQL DTrace users are using Apple Mac OS X 10.5, less so Solaris. MySQL 5.4 and MySQL 6.0 include about 60 pre-defined static DTrace probes in the source.

wikipedia: DTrace

MySQL Conference 2009

Thursday, April 23rd, 2009

The MySQL Conference was, as usual for the past few years, at the Santa Clara Convention Center (home of the $14 bistro burger.)

And once again, MySQL got bought by another company. Last year Sun, this year Oracle.

I found out about the Oracle buy while eating lunch with a DBA from Utah who was there to write the MySQL certification test and investigate MySQL as a cheaper replacement for … Oracle. :)

This year there were actually 3 simultaneous conferences – O’Reilly MySQL Conference and Expo, MySQL Camp (free) and the Percona Performance Conference (free) – 10 simultaneous tracks from 8:30 am to about 9 pm. Julian Cash was also doing creative photography next to MySQL Camp.

Storage-class Memory (NOR Flash)

Last year’s Expo theme seemed to be “3rd party commercial storage engines” (Kickfire, Tokutek, etc.) This year’s theme probably was “enterprise storage-class memory” – and likely the most exciting thing there.

Storage-class memory is NOR flash memory that is mapped to memory space using a Hypertransport controller, unlike SSD which is NAND memory mapped to IO space.

Virident and Schooner demoed Sun and Intel quad-cpu MySQL and Memcached appliances with 2 processor slots attached to 512 MB of NOR flash. When NOR flash density is improved, that will become 1 TB.

Virident, with an office in Milpitas, said that Innodb operations are 70x faster than hard drives because of Hypertransport, optimization of Innodb for byte accesses instead of blocks, and removal of usermode-kernelmode transitions. (MyISAM not optimized yet.) The Virident appliance prices range from around $15K to $45K per appliance. Demo machines and other testing arrangements are available according to the sales rep.

Schooner has a relationship with IBM support to provide on-site service.

mysqld

Regarding the MySQL software, I saw more progress and excitement from 3rd parties than MySQL AB.

The Google V3 patch and Percona patches plus XFS and tcmallloc on linux appear to finally provide good overall performance and adequate multi-core performance to 16 cores for Innodb. Mark Callaghan from Google talked about these at his Cloud talk, then did his usual shout-out to Heikki and Ken Jacobs of Oracle to pick up the ball and run with it.

MySQL Workbench Client Program

MySQL Workbench comes in two versions, no charge and SE. Both versions can print professional-looking schema diagrams from create table statements, and SE can make a remote connection.

(I asked the developers to add a legend of symbols used.)

SE also has a roles feature that a DBA can use like macros to populate the usual user, db and host tables.

Replication and Storage BOF

I attended the Replication and Backup BOF, hosted by Mats Kindahl and Charles Bell from Mysql. It was a strange experience … the MySQL employees had a rosy view of their world (belief in replication working perfectly “guaranteed” and mysqldump being fast and safe, and unaware of potential problems of “FLUSH TABLES WITH READ LOCK” and Innodb), and myself and other experienced users had a very different viewpoint (replication requiring periodic checksumming and snapshots being the slowest acceptable backup method.)

But then again, Innodb sells Innobackup, so no urgency to quiesce it for snapshots, I guess.

Open Storage BOF

Raymond Austin, Sun Workgroup Storage, organized this BOF to request wishlists from MySQL DBAs. It took a while to convey that most DBAs don’t normally run MySQL on NFS. Most of the discussion revolved around: desire for ZFS to provide backward compatibility forever, storage hardware should have checksums everywhere, double parity (RAID6), and reasonable drive controller UER error propagation and control (ie. don’t trash the array upon 1 error in the middle of a rebuild.)

Drizzle BOF

Drizzle is a project to prune MySQL to an essential core for users who want performance more than features, while building a development community that MySQL AB did not encourage hard enough.

Brian Aker did the initial setup of Drizzle.

Jay Pipes talked about anybody was welcome to contribute, whether working on the website, writing client programs, or C/C++ on the server. Just don’t add locks, and ensure the server is modular and scales connection-wise.

In the expo area I talked to Matt Gentry from iDashboards.com. They’re a Detroit company that creates BI dashboards for clients. Funnily enough, his clients most often request speedometer displays, often 4 across, which consume the most display area without providing any more information.

blip.tv: Keynote Videos
filefrag
sf.net: oprofile
performancewiki.com: oprofile
ibm developerworks: Smashing performance with OProfile
Alex’s Notes from Percona performance Conference