The Illusion of Certainty

Most of the people I talk to are well-educated engineers in Silicon Valley, yet even they don’t understand what is true and false in our society.

Fallacy 1: “I want to see peer-reviewed results.”

Peer review just means that some people, usually experts, have read the paper, not that they have duplicated the results. In fact, they virtually never attempt to reproduce the results in the paper.

Going deeper, there’s several problems with peer review:

  1. the only person with the grant money to write the paper is the author, so there’s usually no way to even finance reproducing the results
  2. paper authors seldom include the raw data with their paper for 2 reasons: they can get more papers out of the same data set, and they can get credit for each paper without furthering competitors. Of course without the underlying data, who knows if their are mistakes or malfeasance?
  3. When experiments are duplicated, they seldom match. Recently some drug companies have attempted to reproduce important papers to create known foundations for their own programs. They virtually always fail to see the same results. (Mendeleev himself likely published fake results, as statistics shows his ratios are too good.)
  4. peer reviewers are subject to the same politics and biases as any other human endeavor
  5. In the dismal case of “medicine science”, generally all of the New England Medical Journal published results are considered to be wrong after 10 years
  6. In the case of physics, how do you verify even a single paper on string theory? :)

Fallacy 2: “It’s good/safe if it is FDA-approved.”

FDA approval just means that a new drug is better than a placebo in trials, not that it is better than existing drugs. Also, the drug may work well in one ethnic or gender group and not in another group. The fact that drugs have unintended effects when used off-label proves that the manufacturer doesn’t even know what all the effects of the drug are.

Fallacy 3: “I want data-driven results.”

Data-driven analysis is all the rage in Silicon Valley today, but few people I talk to even know what that means, or how futile it is in inventing new products.

Prerequisites would be:

  1. enough relevant data
  2. enough statistical and domain knowledge to model the data, design tests and interpret the results
  3. enough resources to do the data analysis without reducing ability to do product development or customer support. Google and Yahoo! can afford legitimate A/B testing. Most startups simply can’t.

“The best way to predict the future is to invent it.” Apple’s recent consumer products success is because of delivering and marketing products that nobody was asking for yet, not analytics.

Fallacy 4: “The economy is this or that. Unemployment is this or that.”

Economists are captive to their employers, whether government or private. Either they parrot what their employer wants, and what their “economic school” dictates, or they’re soon unemployed.

Regarding statistics on unemployment, they’re generally quietly restated a couple years after a recession. In the case of Silicon Valley, during and immediately after a recession the newspapers publish unemployment rates of 10%, then restated them as 25% to 30% two years later.

Posted in Business, Tech | Leave a comment

SVLUG: Puppet

At the Silicon Valley Linux Users Group tonite, James Taylor gave an intro on using Puppet.

Overall, he loves Puppet.

One of the few things he doesn’t like is the puppet-specific url of puppet:///, and another is user mgmt.

He said, “remember PCS.”

Thanks again to Symantec for hosting facilities.

Posted in API Programming, Cloud, Linux, Open Source, Tech, User Groups | Leave a comment

Jenkins User Conference US West 2014

Jenkins LogoI went to the Jenkins Users Conference, hosted at the very nice Hyatt Regency SFO in Burlingame, Calif. There were about 600 attendees.

There were 3 talk tracks, plus a misc. track. The talks were not technically difficult, but then using Jenkins is mostly about organizational process and installing plugins.

Kohsuke Kawaguchi, Jenkins Creator

  • X1K project to handle 1,000 build slaves (should be enough for any single corporation)
  • NIO for a few threads to manage humdreds of slaves
  • Puppet Enterprise consulting donated a week to setup puppet, vagrant and docker infra to replace ad hocness
  • 3D print your own Jenkins mascot.

Automation, Innovation and Continuous Delivery
Mario Cruz, Choose Digital @MarioCruz

  • private branded digital entertainment downloads for hotels and megacorps (Marriot, petro-canada, airlines)
  • moved from on-prem to aws
  • outsource everything that doesn’t make money
  • feature flags are great for syncing app with apple app store approvals
  • builds are automatically deployed to either prod or stage env using one Jenkins hosted by cloudbees
  • motivation for CD is to have 2-week vacations
  • manager guesses can only be trumped by data
  • programmers are on-call. Write better tests if you don’t want to get called
  • devs get 45 minutes to either fix or rollback a release, ends up being 30% rollback, 70% fix with frequent releases. Easy if a feature flag can be disabled.
  • QA has 45 min to look at stage before live in prod for consumer brands. API just goes out.
  • Slack is better than JIRA, but $6 more per developer
  • Hipchat, New Relic, Asana, Cloudbees, JIRA
  • Archaius from Netflix OSS
  • DBA is the real gate for Continuous Deployment. Buy him some lunches and explain what’s in it for him.

Advanced Workflows for Multi-Platform Mobile Development
Emanuele Zattin, Realm Inc. (Italy) @emanuelez

  • Nokia before, deployed Jenkins globally to replace custom build scripts
  • even if you’re a mobile library, you still have to provide sample apps, so resolution, UI, stores
  • make, Xcode, etc., valgrind!
  • Tip #1: don’t waste developer time and focus with a context switch. Don’t waste qa time.
  • build, test, coverage, todo length, artifact size growth
  • Extreme Feedback lamp, gamifies multiple teams!
  • Tip #2: fingerprint and store artifacts. You will need them later!
  • Tip #4: version control your build scripts, be careful with release scripts
  • mobile devices are connected to slaves
  • gradle is helpful for Android builds
  • custom interface for Apple iOS devices
  • Mac Pro with VMware, RAM disks crash
  • Ansible to provision slaves
  • Docker for Android builds
  • gcc and clang have valgrind-type address sanitizer option now that’s very fast, just run valgrind nightly


  • sandwiches, but nice mini subs with pasta

Exhibits Area

Jenkins Exhibitors
Jenkins Exhibitors

About a dozen booths.

Keynote Systems

  • metrics down to the end-user


  • awesome frog toy
  • database tracks artifacts, proxy reduces bandwidth, licence manager tracks usage


  • push button deploy after Jenkins builds
  • graphical UI for those who need that


  • Zend server


  • hosted Jenkins


  • recruiting
  • migrating from powerful but internally-developed build and release tools to popular Open Source tools

Ravello Systems

  • cloud management software that actually works, including a product similar to Ubuntu Juju .

    Unmasking the Jenkins DSL
    Matt Moore, Google

    • ex-compiler optimization guy
    • works with Jenkins at Google now
    • jobs plugin is crap. Needs knowledge of other 1,000 plugins to work, so far has 7, so 20 years behind already
    • hence structured form submission
    • YAML project type
    • !by-name shell
    • create a job, view yaml template
    • !kind matrix
    • “git flow” by Victor Driessen
    • config in code with latest git plugin
    • features.yaml, releases.yaml, master.yaml multi-branch
    • intended for git and mercurial, others unknown
    • Jenkins does most validation in form, so yaml-project doesn’t do much yet
    • there is also the autojobs plugin

    Operating Jenkins at Scale
    Ryan Campbell & Robert Sandell, Cloudbees, Inc.

    • congratulations, if you’re here, you own a large Jenkins installation! :)
    • monitoring
    • health, perf, business value
    • collectd, graphite, nagios
    • Jenkins Metrics Plugin DropWizard
    • Jenkins Operations Center (JOC) plugin. Uses ElasticSearch and Kibana
    • custom kibana tiles
    • who’s responsible?
    • how are backups, disk space, sameness, upgrades done?
    • choose plugins carefully
    • more masters: depts, plugin testing, etc.
    • bulk operations plugin for JOC

    Using Jenkins to Build Apache vhosts for Github Pull Requests on PHP Apps
    Sam Moffatt, Account Bouncer

    • previously at ebay, Sears
    • pull requests build an environment for developer or qa engineer
    • bind9 with mod_vhost_alias, comment out default vhost to avoid conflict
    • http://pr1.test, http://pr2.test, …
    • ensure if using Github Enterprise and custom SSL they are added to GE
    • ensure API setup correct
    • beware multiple pulls at same time, can cause confusing errors.

    Perfecto Mobile

    • USAA mobile app check scan, share feature awesome
    • emulators suck – false sense of confidence
    • 7 requirements
    • managing a device lab is hard – try 65 devices, different versions, can’t roll back
    • MobileCloud – Open and Integrated, real devices connected via USB (no jailbreaks)
    • will work on shake testing if there’s market demand

      Lightning Talk #1: FIFA Gameplay Automated Testing with Jenkins
      Stuart Rowe, Electronic Arts

      • old system was scripts with manual emails notifying developers, about 8 hours of busywork per week by an engineer
      • new system uses Jenkins
      • performance testing done on real devices.

      Lightning Talk #2: Supercharged Configuration as Code – Using Jenkins to Control Mass Updates of JOB DSL
      Alan Beale, Chicago Trading Company

      • bulk updating of 100 jobs?


      • Jenkins backup master. Master is just a directory, so copy files?

      Conference Suggestions

      • Advanced Track with more programming, like Google’s YAML-project talk
      • community bulletin board in hallway.

      Interesting phone charger tree with Post Office box-style, with combo locks:

      Knuckles Historical Sports Bar


Posted in API Programming, Business, Cloud, Conferences, Open Source, Oracle, Tech, Toys | Leave a comment

Lessons from “Air Crash Investigations” for Passengers

Youtube channel “Canadian Flight Centre” has the National Geographic series “Air Crash Investigations.”

It’s very educational for both pilots and passengers.

Airline travel is very safe in the United States and Europe, with Western airlines having about 1 major accident annually. That doesn’t apply to Asian and Eastern European airlines, and non-airline flights.

Following is a meta-study that summarizes passenger survivability over several accidents.

Lessons for Passengers

Before Flight – Planning Your Flight

  • don’t fly in typhoons, thunderstorms, volcanic eruptions. Avoid Surabaya on a rainy day. Avoid flights at airports immediately after icing closures because of delayed takeoffs which increases icing.
  • some airports don’t have adequate navigation aids for bad weather to land safely (chronic ILS inoperative at SFO)
  • the remarkable safety of airlines is due to having dispatchers and flying the same routes repeatedly. Don’t expect the same outcome from new airlines, maiden flights, charter flights or diversions
  • flying into airspace with different local customs is less safe: airspace like Russia and Brazil are examples, where language and military-style control can cause misunderstandings
  • wear jeans or cotton pants and a cotton hoodie – no hairspray or nylon
  • be observant – a passenger saw a fatigue crack on Aloha 243, but didn’t report it. Same with pax on Midlands 092 knew the wrong engine was shutdown after left engine caught fire, but didn’t say anything. ice, pitot covers, etc.
  • American regionals (which exist to shave costs) and discount Asian airlines are sketchy on maintenance and provide less pilot training than major airlines
  • Korean pilots can’t fly non-precision approaches (Guam KAL, SFO Asiana 214)
  • tail strikes are bad news on pressurized airplanes
  • safest place is rear of cabin, except for cargo door blowout
  • fires worst over wings. pilots rarely get burned
  • brown cigarette smoke on outside of fuselage indicates crack
  • choose airports with doppler weather radar if thunderstorms are possible
  • if your airline often runs late, choose flights earlier in the day to avoid rushing to make schedules and crew rest rules
  • for small planes, typically propeller, ensure operator weighs baggage before loading
  • original 747 rear cargo door latches have aluminum latches in steel mounts, so will eventually wear and fail (UA 811)
  • if 747 center fuel tank is not full and no nitrogen system, there is a vapor explosion risk. (TWA 800)
  • original ATRs can’t handle icing conditions
  • MD80 has non-redundant jackscrew in tail
  • original DC10 had non-redundant electrical equipment powered by engine #1.

Before Crash

  • an alert private pilot pax saved himself and seatmates on the Washington Air Florida Flight 90 accident (crash brace, expedited egress)
  • weightlifters could help with holding yoke/rudder pedals after flight controls problem to rest pilots
  • make a mental note of two exits and count floor lights
  • don’t inflate life vest before ditching: easier to swim, easier to exit door

During Impact

  • many survivors broke both ankles under seats, unless you’re really short

After Impact

  • if you can smell kerosene (Jet A), you’re alive
  • plug-type doors generally need to be opened and discarded through its doorway to not block passenger egress (USAir 1493)
  • 2 minutes to get out, one death every second after that
  • one breath of black smoke near ceiling makes your “lungs feel solid, 2 breaths kill you.” crawl. (Brit. Airtours)
  • smoke hoods would save lives, but slow down deplaning. crew may have smokehoods and oxygen bottles stashed in overhead bins (Brit. Airtours)
  • a survival strategy when crowded is to stay low to floor until other pax inhale near ceiling, however flashover will kill everybody (Brit. Airtours)
  • pax climb over seats if exits are blocked (exactly like a money race) (Brit. Airtours)
  • an engine fire will blowtorch the fuselage if positioned upwind (Brit. Airtours)
  • if emergency exit jammed, try pushing both top and bottom (Brit. Airtours)
  • burns can kill half of survivors due to post-accident organ failure
  • don’t jump into center of flames (USAir 1493.) if you’re on fire, drop and roll after stepping over jet fuel on ground
  • using axe on reinforced cockpit windows takes a long time, can break axe, best performed by uninjured person
  • “like being in a war. go go go” Little Rock, AK

After Egress

  • you can be on your own for an hour or more – days in the mountains – and freeze (AA 965.) For JAL 123, rescuers decided to wait until the next day to ascend the mountain, taking 14 hours to reach the accident site despite knowing the location. In France, 100 rescuers parked for 4 hours, waiting for “official” notification to search, so a journalist followed the smell of kerosene to the site.
  • “the first disaster was the crash. the second was being held in custody in a small room for hours.” (KAL Guam)
  • except in inverted dives into the ground, rescuers report that there are almost always initial survivors, so fight to get out of the airplane wreckage
  • don’t lay down in foam or in a vehicle route, or rescue vehicles can run over you (Asiana SFO).

Improving Airliner Safety

  • pilots need real-time camera views of control surfaces and engines in cockpit. Until then, cabin crew needs to relay engine fires, cabin smoke or damaged flight controls. (The locking cockpit door actually reduces safety during an emergency as it impedes communications amongst crew and creates a psychological barrier.) (Midlands 092 wrong engine shutdown, Chicago #1 engine fire, Alaska Air jackscrew failure, Schiphol El AL 747, etc.)
  • since pilots can’t see the aircraft wings or engines, cabin crew need to be trained how to observe and report on engine fires, fuel leaks and wing damage. They need to know airplane left from right, and what to look for even in low visibility (fuel leaks are nearly invisible at night.) They need to know time for useful consciousness and remind the flight crew every few minutes after a depressurization, as passenger oxygen only lasts 10 or 11 minutes. (For years, Qantas ground crews “accidently” connected nitrogen instead of oxygen.)

Powerlines Can’t Hide from Safe Flight Detector
Baker Aviation Offers “Hot Deal” on Fire Containment Bags
USAIG Offers ‘Never Events’ Maintenance Tech Program
FLYING THE ENGINE – How are you counting engine cycles?

Posted in Tech | Leave a comment

Bash Shellshock Fix for Old Macs

This chap has a shellshock fix for older Macs, including PPC and pre-Mountain Lion. I tested it on a Macbook Pro (Intel) with Leopard 10.5.8 and it works fine.

It is also an easy way to fix failed bash update attempts, as it’s a single file that you can copy over /bin/bash and /bin/sh. (If you borked bash and can’t open a Terminal window now, just type in the command “ksh”.)

The one good thing that has come out of shellshock is that the bash debugger runs on bash4, so that will be good for wider adoption.

Now if only Microsoft would get that Skype update out for older Macs. (Instead, they’re actively blocking old versions that used to work fine.)

SO: How unpack and pack pkg file?
Solving Snow Leopard Crashes on a 24″ iMac

Posted in Open Source, Retro, Tech | Leave a comment