Recently I had the chance to use EC2 for a commercial project for 2 months, so I thought I’d share some real-world experiences.

A friend of mine with a small West Coast data center (1/2 a rack or so, slow 2000-2005 vintage Intel servers) needed to do something big: build out a flexible internal SOAS renderfarm to generate promotional morphing video clips for an entertainment-industry client.

The existing capacity was about 1/morph per second sustained according to benchmarks.

The system requirements were:

• linux OS running apache, FFmpeg and proprietary software
• static IP address to reach each server from the permanent data center
• easy and affordable to scale up or down (possibly 10x or more) with 4 hours notice or less. Start with 10 morphs/second, scalable to 100+ morphs/second.
• Core 2 Duo CPUs for rendering (number crunching)
• not much disk space, bandwidth (results hosted on Akamai) or memory needed - mostly CPU
• West Coast location for servers preferred, not critical though
• no important data to be stored off-site

The requirement to install custom software leaves you with dedicated servers or Amazon EC2. All the other grid offerings I looked at (Google Grid, Mosso, others) had limitations on what software you could install, basically PHP or python for web use - no root access, can’t install ffmpeg or proprietary binaries.

I like dedicated servers, but their problem is that generally you pay by the month, so it’s hard to quickly reduce capacity and expenses depending on advertising spikes. Sometimes new servers can be provisioned quickly, and other times they’re “sold out.” Not so “elastic.”

Some good American business dedicated server hosts are theplanet and gigeservers.

I was somewhat familiar with EC2 from various conferences, especially the Hadoop Conference last year. The talks by The New York Times (cloud OCR) and AutoCAD (vendor metadata processing) were the most influential on me that I’ve been to.

I did some benchmarking on the 20 cent instance (c1.medium: High-CPU Medium Instance - 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each). 32-bit, 1.7GB RAM, 350GB disk) and liked the performance.

After that I added 2 large 80 cent instances (c1.xlarge: High-CPU Extra Large Instance - 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each). 64-bit, 7GB RAM, 1690GB disk).

I noticed one c1.xlarge was somewhat faster than the other, so you may want to benchmark a few different instances and choose the fastest ones.

The pre-made AMI linux images are all a little different. I had no problems with the 20 cent one, but the 80 cent one was not so smooth - the Perl CPAN module was not installed, and conflicted with the Amazon tools. So I had to yum remove the Amazon tools, then yum install perl-cpan.

I ended up going with two of the monster 80 cent instances to reduce system maintenance and software release targets primarily, keeping the 20 cent instance for test/prod.

Amazon instances appear as regular linux Xen instances, so it’s just linux as usual.

In the end, we ended up using EC2 for 2 months with no problems.

Overall the project was a success, and I was happy with EC2. The invoice was about $1,300/month ($1.80/hour x 24 hours x 30 days) total for the 3 instances, including negligible bandwidth. Here’s an online AWS calculator to play with.

Some suggestions to Amazon would be to make provisioning possible from the web portal instead of just the command-line tools to help newbs get rolling, and do better testing on the images - CPAN.pm is a commonly-used Perl module indeed.

Note that FFmpeg (HEAD) is twice as fast as the older rpms floating around download sites.

Sample EC2 Commands

$ ec2-add-keypair gsg-keypair -K /root/.ec2/pk-xxx.pem -C /root/.ec2/cert-xxx.pem
$ ec2-describe-images -o self -o amazon
$ ec2-run-instances ami-2bb65342 -k gsg-keypair -t c1.medium
$ ec2-describe-instances
$ ssh -2 -i id_rsa_gsg_keypair ec2-67-202-32-93.compute-1.amazonaws.com
$ ec2-allocate-address
ADDRESS 75.101.148.165
$ ec2-describe-addresses
ADDRESS 75.101.148.165
$ ec2-associate-address -i i-aa46e5c3  75.101.148.165
$ ec2-authorize default -p 80 -K /root/.ec2/pk-xxx.pem -C /root/.ec2/cert-xxx.pem
GROUP           default
PERMISSION              default ALLOWS  tcp     80      80      FROM    CIDR    0.0.0.0/0
$ ssh -2 -i key aws01

         __|  __|_  )  Rev: 2
         _|  (     /
        ___|\___|___|

 Welcome to an EC2 Public Image
                       : - )

    Getting Started

    __ c __ /etc/ec2/release-notes.txt

[root@domU-12-31-38-00-46-01 ~]# uname -a
Linux domU-12-31-38-00-46-01 2.6.16-xenU #1 SMP Mon May 28 03:41:49 SAST 2007 i686 athlon i386 GNU/Linux

[root@domU-12-31-38-00-46-01 ~]# date
Sun Sep 28 21:17:28 EDT 2008

[root@domU-12-31-38-00-46-01 ~]# cat > /etc/yum.repos.d/dag.repo

[dag]
name=Dag RPM Repository for Red Hat Enterprise Linux
baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag
gpgcheck=0
enabled=1
^D

[root@domU-12-31-38-00-46-01 ~]# exit

$ ec2-terminate-instances i-aa46e5c3

Rent or Own: Amazon EC2 vs. Colocation Comparison for Hadoop Clusters
George Reese: On Why I Don’t Like Auto-Scaling in the Cloud

At roughly 30 minutes it was the shortest SVLUG talk I’ve been to.

Thanks to Symantec for hosting the event.

Hawaiian Airlines arrived at Honolulu Airport at noon Thursday, so I headed straight to the GA area on Lagoon Drive, ate a Hawaiian-style plate lunch at Hawaiian BBQ, and got a practice flight in right away at George’s Aviation.

Note: theBus’ Lagoon Drive route was cancelled 2 months ago, so you’ll need a rental car or catch a cab on the east side of Lagoon Dr. just south of Nimitz.

I initially flew a 5 year-old Cessna 172SP. This was not a great choice for me to practise for a flight review as the panel instruments (HSI, GPS, etc.) and handling were quite different than my usual 172N choice.

The 172 SP has a fuel-injected 0360 engine with a lot more power, and the airframe is a lot cleaner than a 30-year old N model, so handling in the landing flare is different. (Apparently George bought if from a Fedex captain who kept it as a toy.)

Then I went to my “discount-rate hotel” - the 4-star Waikiki Resort Hotel at Koa and Kauilani Streets - for $89/nite. That’s about half price, so occupancy rates must be really low. (When I was paying for my flight online, a partner popup promoted this deal. I guess I was targeted very effectively.)

After another practise flight the next day, I started my FAA Flight Review.

Every 24 months, a flight review by a CFI is required. It used to be called a Biennial Flight Review (BFR), but I understand it may become an annual requirement, hence the recent renaming to just “Flight Review.”

CFI Bob gave me a 1 hour flight review oral quiz. Mostly it involved demonstrating detailed knowledge of airspace on a sectional chart, and explaining the various FARs about pilot and aircraft currency.

After that we did a one-hour flight out of Honolulu Class B airspace to the Wheeler south practise area for private pilot maneuvers, followed by some landings at Kaleiloa Class D airport, and return for landing at HNL.

It was a challenging flight. The winds changed from quite calm for Hawaii using Tradewind procedures to Kona wind procedures mid-flight, resulting in runway changes at both airports. Also, Honolulu Approach frequency was busy for almost 5 minutes, so I had to circle outside the Class B before landing.

It appears the standard North Arrival at 1500′ has been changed to the North2 Arrival at 2000′, effective in the new Airport and Facility Guide (AFD).

I ate the buffet at Sheraton Princess Kaiulani one nite, and another nite had the Pear salad and Linguini Seafood at Sergio’s Italian (yelp reviews) in the location formerly occupied by Sam Choy’s near the Honolulu Zoo.

The city block-sized Compusa near downtown was closed due to bankruptcy, and most of the retail stores on the first level of Ala Moana were new to me since the last trip.

The hotel TV had some interesting items. The Subway $5 ad had one hilarious version with 3 hula dancers doing a hula to the ad song. There was a well-researched hour-long documentary on the history of the UFC, including interviews with the original creators and current owners and president. Plus lots of local news coverage of the Hawaiian airline industry court battles over the bankrupt Aloha airlines trademark.

The shuttle bus back to the airport was only $9, and left plenty of time to eat at the Lahaina Chicken location. The roast chicken and beef combo platter was really good.

Next time I’ll take the free airport wiki shuttle out to the mainland gate. It’s too humid to run around outside with luggage. (Any gate higher than 26 is a long walk with luggage.)

Hawaiian Airlines has the most primitive in-flight entertainment systems, but it’s only a 4.5 hour flight, so I didn’t bother to rent a headset for $5 or a digimedia player for $15. Instead I watched my seatmate cheat on beginner’s crossword puzzles, which was quite amusing.

They have a very strict space budget for code and fonts since every byte they use is one less for the end-user.

As much as possible, they try to provide the full ICU API for developers to use.

Regular Truetype fonts are used, though there’s no hint information and there’s no mechanism to add your own.

Chuck Soper (Vela Design Group) talked about porting VelaClock to the iPhone, and made some recommendations for the Apple iPhone apps store.

He would like to see longer sales history information than 7 days, ideally unlimited. Also, he would like to see feedback and ratings reviews across countries, since loading 100 country forums is tedious.

He also wanted to know best to provide mib and strings to translators.

He says half his sales come from the Apple Store.

Some of his customers use VelaClock to do things like plan night flights.

I had a chances to try out the Blackberry Bold, with it’s new UI and hi-res screen. The screen has the same number of pixels as an iPhone, but half the dimensions.

I also tried a gPhone. It has a built-in compass, so Google StreetView knows what direction you’re pointing the phone and can show real-time updates based on that direction. Very cool to see.

Thanks to Apple for hosting the event in Cupertino.

This deal is big news … it even made the front page of wsj.com.

A lot of people in the aviation industry were convinced this plane would never make it to production.

I knew that it would, but assumed the final production price would be a little higher. I always thought DayJet was a shill company of Eclipse to help write press releases.

Well, imagine my surprise when I saw TV video footage a few months back of DayJet’s operation, replete with dozens of Eclipse 500 jets. DayJet was real afterall!

DayJet has since closed, blaming equally financing problems and Eclipse roll-out problems.

What’s fascinating is that Eclipse is selling the DayJet 28-Eclipse 500 inventory with the information that each airframe has 150 to 450 cycles. So it looks like DayJet really did some flying.

Eclipse has apparently missed payroll this month, so we’ll see what happens next.

avweb.com: Eclipse Selling Off DayJet Fleet
avweb.com: Analysts Grim On Eclipse Future
avweb.com: Eclipse Shutdown Predicted
avweb.com: Eclipse Misses Payroll: TV Report
kob.com: Unpaid Eclipse employees worry about future
South Florida Business Journal: DayJet Restart Speculation

First of all, sorry to out-of-town attendees for the showery weather today. San Jose is normally sunny, but not in winter.

The Expo was 3 days (Thursday, Friday and Saturday) for $55/day including seminars and exhibits.

Exhibits Hall

The Exhibits Hall was twice as big as I expected and a lot of fun. I’m used to sparse IT conference exhibits, but aviation exhibits are a welcome relief with lots of hardware: aircraft, wrap-around simulators, parts, interior fabrics - you name it.

I only had time to spend an hour looking at the exhibits, but could easily spend a day going booth-to-booth and trying everything out.

The Cessna 162 Skycatcher in purple was on display, as well as the DJet (and a Williams engine static display) and Epic.

Frasca had a wrap-around simulator, and there was a very nice glass cockpit Cessna 172 simulator with 3 almost 180 degrees of displays. It’s available for rent in Hayward for $65/hour.

Rolls-Royce had 2 engines on display, the A300 used in the Robinson R66 helicopter and a prototype for the A400.

Noticeably absent … Eclipse did not have a booth.

I listened to 2 seminars from trainers that I haven’t seen live before, and really wanted to: John and Martha King and Mike Busch.

Pilot Risk Management, John and Martha King

John and Martha King talked about managing flying risks in a systematic fashion.

It took John a minute to warm up, then he sounded just like his pilot training tapes.

They calculate that small-time GA flying is about as risky as motorcycle operation in the US. Half the audience personally knew a pilot killed in GA.

John then went into 3 war stories: IFR letdown in a Cessna 210 with no electricity due to ignoring maintenance near St. Paul in icing, poor takeoff decision at max. weight at a high DA airport (Lone Pine, 3680′), and an unlighted flight from Big Island to Oahu over the ocean at nite (forgot to pre-flight lights for a nite flight.)

They recommend using checklist nmenonics like PAVE CARE for reducing risks.

• Pilot
• Aircraft
• enVironment
• External Pressures

• Consequences
• Alternatives
• Reality
• External Pressures

Airplane Maintenance Management, Mike Busch

Mike Busch gave an awesome talk on airplane maintenance management.

Mike is an aviation maintenance author, trainer and businessman who is famous in the GA aviation community.

His latest venture is savvymx.com, which provides professional maintenance management for owners. They represent dozens of aircraft already.

(I’ve attempted to paraphrase what Mike said below, but any errors or omissions are my fault.)

40 years ago in GA’s hayday, there was an authorized Cessna, Piper or Beech service center on every field with specialist mechanics and a building full of parts on the shelf. Now GA maintenance facilities are merely a shadow of that, unless you’re talking jets.

He recommends 5 rules/secrets for affordable maintenance:

1. Interview your mechanic/shop like you’re hiring an employee
2. Inspection, Discrepancies, Approval in writing
3. Don’t fix what’s not broken
4. Pilot needs to troubleshoot before mechanic can fix anything
5. one other …

The aircraft owner is the manager, the mechanic takes orders, and there must be a business-like relationship. Otherwise, find another shop.

However, while giving his SavvyAviation talks, he’s noticed that some people either don’t want to make the time, effort or be assertive enough to actually do the mgmt. needed.

95% of aircraft components can and should be maintained on-condition, meaning periodically inspected and replaced as needed. Examples are tires, some actuators, etc.

The remaining 5% are things maintained on a time-based schedule, like magnetos and hoses which are difficult to inspect.

When approving aircraft repairs, terminology is very important. The terms repair, overhaul and rebuild mean very different things. Normally what one wants is a repair (fix just what’s broken directly and as cheaply as possible), and not an overhaul (blindly follow an overhaul checklist from beginning to end and change and test everything, broken or not.)

He says that TBO is a psychological limit, not a maintenance one. His P210 is currently 1100 hours past TBO, FWIW.

Mike recommends using the most direct method for monitoring and troubleshooting aircraft: engine monitors, oil and filter analysis, borescope. He calls this “21st century analysis”, while old techniques like magneto RPM-drop and cylinder compression tests are “Orville and Wilbur Wright analysis.”

He uses Blackstone Laboratories for oil analysis. Unfortunately, many shops don’t have borescopes, and even when they do, nobody with training to interpret the image, since studying borescopes is not required for A&P.

In newer airplanes, there is no separation between airframe and electronics, so you need a shop that can handle integrated maintenance, like Woodgreen in SoCal.

He used a black Asus EEE PC to show his slides.

Robinson R66: a preview by Philip Greenspun, Feb. 2008
flyingmag.com: Learning to Use an IFR Rating

I have my own map.pl script for doing that that has different options for serial and parallel execution, immediate and sorted output, and timeouts.

Logging of commands and performance information might be a good idea.

I find it handy to maintain a small script that can be customized for different scenarios, especially if you maintain the server host and status information in a database.

Several Open Source projects have also sprung up.

Minimalist ssh-like command line tools:

• pssh, pscp, prsync, pnuke, pslurp
• mpssh
• clusterit (C)
• func (python)
• capistrano (ruby)

Less minimal Distributed Virtual Terminal utilities:

• clusterit (dvt)
• clusterssh
• PConsole
• MultiXTerm
• C3
• Keyboardcast

SourceForge: search for “parallel ssh”
Parallel SSH execution and a single shell to control them all
pssh: Run Command On Multiple SSH Servers
Bitmover BitCluster

5 years ago, the shuttle bus was scrapped, and the Caltrain connected to SFO at Millbrae with a $1.50 5-minute BART train that stopped at Terminal 3. The 2 ticket vending machines were often mobbed for 20 minutes. Good luck finding a live person for assistance. A hassle, but doable.

Today, the Caltrain connects to SFO at Millbrae with a $1.50 10-minute pointless BART train north to San Bruno BART station, followed by a 10-minute ride back south to SFO Terminal 3. The 3 ticket vending machines are often mobbed for 20 minutes. If you find a BART employee, they’ll say, “I didn’t design this system.” A nightmare in good weather, worse when it’s cold or windy.

I (and other passengers on the same Caltrain) missed an international flight recently because it took over 3 hours to get from San Jose Diridon station to SFO.

Although a taxi from central San Jose to SFO is $120, I have to consider that now if I don’t have half a day to waste on the trains. Cheaper than paying for trains that are misrouted and late and rebooking fees.

examiner.com: BART to halt Millbrae-SFO direct service