Meetup: Scaling Chef at Facebook

Posted on by
Reply

Chef LogoAt the Bay Area Chef User Group tonite in Mountain View, Phil Dibowitz from Facebook Operations reprised a talk he gave at ChefConf 2013, “Scaling Chef at Facebook.”

His team of 4 people write and maintain the Chef systems for hundreds of thousands of servers, divided into several clusters of 10,000 or so nodes.

Facebook used CFEngine v2 before, but migrated to Enterprise Chef (private), and Community Chef (public) on at least one cluster.

The advantage of private Chef is to get features faster, especially related to large clusters.

He does not like CFEngine because it is file-based, whereas Chef is template-based.

Phil likes idempotent behavior. Idempotent originally meant “not changed in value following multiplication by itself,” but in this case means “acting as if used only once, even if used multiple times.” Thus a file only gets downloaded or updated once per client, not over and over.

(A second principle in managing networked systems like HPC is near locality of reference. Blekko is a big proponent of this for performance reasons.)

And in any large cluster, the secret to eventual consistency of all nodes is to have clients pull changes, not have the deploy server push them.

He used sysctl.conf as an example of how Chef makes it easy for each group or engineer to customize their servers.

Also, he talked a little about how Chef can help configure a production test node as a test canary that
automatically reverts to live after an hour of non-testing.

Facebook is big on IPv6, as a significant percentage of mobile and Comcast traffic is using it.
Also, with the large number of devices they have, IPv6 helps when RFC1918 space happens to run out internally.

Phil has previously worked in Operations at TicketMaster and Google, and runs a popular Metallica
web site.

Thanks to Ooyala for the nice meeting space. 800 W. El Camino Real is a fairly historic building. Searchme.com was a Sequoia-funded visual search engine also based here.

Ooyala is a video CDN. They use Ruby, Go, Scala and Chef, and they are hiring.

ChefConf 2013: Scaling systems configuration at Facebook – Phil Dibowitz

keywords: opscode, chef

SVLUG: The Amanda Backup Utility and Zmanda

Posted on by
Reply

Zmanda LogoTonite at the Silicon Valley Linux Users Group (SVLUG), Chander Kant and Paddy Sreenivasan from Carbonite/Zmanda gave a talk on Amanda/Zmanda.

Amanda is a popular Open Source Client-Server backup program that is one of the oldest Open Source projects. Zmanda Inc. is a company that was formed in 2005 to sell support and services around Amanda.

  • Zmanda bought by Carbonite in 2012 to move into business services
  • quick to fix Coverity issues
  • stores archives in existing backup formats like tar, zip, bzip, dump
  • 32k archive header contains label and restore instructions
  • Client, Backup Server, Media Server
  • Amanda can compute an optimized schedule by itself based on times or bandwidth
  • Vaulting (multiple tiers of same backup to multiple locations)
  • full set of encryption methods, including hardware
  • compression either on client or server
  • parallelism, even to Cloud
  • Open Source, Open Formats, Open Standards
  • Microsoft Windows client is free but not Open Source, uses VSS. Can backup Exchange and other MS formats.
  • Zmanda Enterprise is Core + Extensions (Oracle Enterprise database plugin, etc.) + web-based Management Console
  • Amanda has file-based catalog, Zmanda is embedded MySQL
  • many clients migrate from Amazon S3 to Glacier to reduce storage costs
  • Zmanda Recovery Manager (ZRM) for MySQL
  • planning to use libvirt library for Xen support, etc.
  • can specify backup target by by geographic location, like Amazon region
  • backup and DR to Cloud, start instance and then terminate it when backups tasks are done
  • write your own plugin for things like object stores
  • rsync is a tool, Amanda is a lot more.

Thanks again to Symantec for hosting tonite’s meeting.

Restaurant Review: Xanh Vietnamese Restaurant, Mountain View

Posted on by
Reply

I had lunch at Xanh Vietnamese Restaurant on Castro St. in downtown Mountain View.

They only serve Vietnamese buffet for lunch, but it’s quite elaborate – I counted at least 3 different types of spring rolls (veggie, beef and fried), at least 2 soups, and a dozen vegetarian and meat entrees.

I liked the variety of food, and thought it was tasty.

The restaurant is very large, with seating for about 100, so big enough to easily handle parties of 20-30 with reservations.

Special mention goes to the lovely hostesses! :)

Xanh Restaurant
110 Castro Street (near W. Evelyn Ave.)
Mountain View, CA
650-964-1888

Restaurant Review: Amber India Restaurant, Mountain View

Posted on by
Reply

I had lunch at Amber India Restaurant in Mountain View. It is part of a chain of Indian restaurants in Silicon Valley.

Unlike most local Indian restaurants, there’s no lunch buffet, only ordering off the menu.

I had a mild curried chicken dish with side vegetables and rice.

The food was served hot and fresh.

My order was not exactly abundant, but the plate orders seemed larger, so inquire about those if you’re really hungry.

Next time I plan to try the lamb.

For special orders, like vegan, state your request clearly to avoid confusion.

Amber India Restaurant – Mountain View
2290 El Camino Real,#9
Mountain View, CA
650-968-7511