ArXiv Paper on Using Additive Noise to Determine Cause and Effect

One doesn’t see many scientific papers that are immediately useful, but this one qualifies:

arXiv.org: Distinguishing Cause From Effect Using Observational Data: Methods And Benchmarks
Physics arXiv Blog: Cause And Effect: The Revolutionary New Statistical Test That Can Tease Them Apart

“They say the additive noise model is up to 80 per cent accurate in correctly determining cause-and-effect” … “in the very simple situation in which one variable causes the other.”

Posted in Tech | Leave a comment

HAProxy and SSL SNI Support

The HAProxy 1.5 branch has SSL support built-in, so you don’t need stunnel or other SSL-termination helpers now.

I tested SSL Server Name Indication (SNI) functionality with HAProxy 1.5.10, OpenSSL 1.0.2 and two SSL certificates (GeoTrust from Namecheap.com) on 3 Dell 1950 servers and it worked fine for me. HAProxy ran on one server and the others ran Apache HTTPD using virtual servers for each domain being load balanced.

SNI lets you use one IP address with multiple SSL certificates. For each site, you just create a single PEM file with key, crt and chain entries, in that exact order. Using SNI reduces the number of IP addresses you need, and also avoids having a separate stunnel process for each SSL certificate.

SNI works fine with most desktop browsers since 2003. However, custom client applications and embedded devices that use SSL may be confused with SNI. I noticed that the Nagios plugin cannot see the second certificate, even with -H hostname specified.

For GeoTrust certs for Apache+OpenSSL as of Feb. 15 2015, the correct installation of the 4 certificates is:

cat server.key server.crt rapidssl_cabundle.crt >server.pem

-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
INTERMEDIATE CA:
---------------------------------------
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

Troubleshooting:

  1. note that haproxy prints a general error message of “unable to load SSL private key from PEM file”, regardless if it’s a missing filename, incorrect file permissions or incorrectly formatted certificates, so check the filename and permissions first.
  2. ensure there’s no malformed header (dashed) lines and delete blank lines
  3. OpenSSL certs are in PEM format by default, so there’s no need to convert them. (Usually it’s Windows users who have to do PEM conversion.)
  4. After haproxy starts, it’s important to verify the certificate chain. Use sslchecker.com and use the Chain Details button to see the intermediate and root certificate names and dates.

A new section in haproxy.cfg is needed to listen on port 443:

frontend https-in
    bind *:443 ssl crt /etc/ssl/server1.pem crt /etc/ssl/server2.pem
    reqadd X-Forwarded-Proto:\ https
    default_backend application-backend

For CentOS 5 users, SNI requires you to build haproxy from source with a newer version of OpenSSL statically. The README tells you how to do that. Use the latest version of OpenSSL to avoid errors about missing function names.

cd openssl-1.0.2
export STATICLIBSSL=/tmp/staticlibssl
make clean
./config --prefix=$STATICLIBSSL no-shared
make && make test && make install
cd ../haproxy-1.5*
make clean
make TARGET=linux26 USE_OPENSSL=1 SSL_INC=$STATICLIBSSL/include SSL_LIB=$STATICLIBSSL/lib ADDLIB=-ldl
service haproxy stop
make install
service haproxy start

For those upgrading from previous versions of haproxy, old .cfg files should still work, but warnings are emitted for timeout settings, as they have been renamed in 1.5:

service haproxy start
[...]
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.

1.5 has only been GA since June 2014, so ensure you test it adequately for your requirements and keep an eye on the changelog.

SO: Configure multiple SSL certificates in Haproxy
HAProxy and SNI-based SSL offloading with intermediate CA
blog.haproxy.com: Enhanced SSL load-balancing with Server Name Indication (SNI) TLS extension
blog.haproxy.com: How to get SSL with HAProxy getting rid of stunnel, stud, nginx or pound

sslmate.com: Buy SSL certs from the command line

Posted in Linux, Open Source, Tech | Leave a comment

Perl FormBuilder and HTML::Template Cookbook

Just using FormBuilder, aka the Perl CGI::FormBuilder module, for the first time in a project.

FormBuilder with DBI and CGI::Session is similar to Ruby on Rails for Perl for CRUD apps. You get HTML form creation, input stickiness, javascript validation and server-side validation for “free.”

By itself, FormBuilder is suitable for rapidly prototyping web applications for internal use.

After adding HTML::Template files, you can do professionally-polished web applications.

Some things to keep in mind when combining FormBuilder with HTML::Template:

  • ensure you use FormBuilder’s form-start, js-head and form-submit tags in each template, or nothing will seem to work
  • the default select menu text is “–select–“. Pass the ‘selectname’ argument to change the text to something like ‘Choose Plan’.
  • Pass a hash reference to ‘select’ to specify both options and values. Use numbered values to control ordering.
  • Pass ‘value’ to checkbox to pre-select it.
  • Suppress checkbox text labels by passing a hash reference with empty strings for values.
  • The source code for field controls is in /usr/local/share/perl5/CGI/FormBuilder/Field/. Although it’s very terse Perl, I found it useful for understanding the select field control.
  • pretty much any CGI:: namespace module has a dependency on CGI.pm. CGI::FormBuilder does a ‘require CGI;’ for the param method (in cgi_param()), and CGI::Session uses the cookie and/or header method. If you want to remove the dependency on CGI.pm (to improve performance), then it should be possible to write your own header, cookie and param functions.
  • if your submit buttons are touching each other, add this to a stylesheet:
    .fb_button { margin-right:5px; }
  • to fix the trailing br tag when using linebreaks => 1 in radio groups, do this:
    Field/radio.pm:172
    
     $tag .= '<br />' if $self->linebreaks and $opt ne $opt[-1];
    
  • the way I conditionally show some form fields within the same form is to omit the field names from the new(require) call and instead use field(require) on each, and use a class called my_fb_hide in my template on those fields as appropriate.
    .my_fb_hide { display:none; visibility:hidden; }

The best online reference I’ve found is this page.

To evaluate FormBuilder for your needs, I’d recommend doing a simple form first, then trying to implement your most complicated form. That should reveal if it gives you the level of control you need.

The author recommends using FormBuilder 3.08 or higher, but my testing on 3.05 worked fine for me.

FormBuilder has minimal dependencies, mainly CGI.pm and Scalar::Util::weaken, so it is much easier to install than Catalyst.

You can subclass CGI::FormBuilder like this:

# Program: AppBuilder.pm
# Purpose: subclass CGI::FormBuilder's most common methods when
#   used with HTML::Template
# Copyright: James Briggs, California, USA.
# Licence: Perl5 Artistic Licence
# Date: 2015 01 30
# Env: Perl5

package AppBuilder;
use parent qw(CGI::FormBuilder);

use strict;

   my $DEBUG = 1;

   my %FB_defaults = (
      submit     => [ 'Update', 'Cancel', ],
      stylesheet => '../css/style_fb.css',
      smartness  => 2,
#     debug      => 2,
      method     =>'POST',
      header     => 0,
      javascript => 1,
   );

   my %HTML_Template_defaults = (
      type     => 'HTML',
      utf8     => 1,
      cache    => 1,
      # shared_cache => 1,
   );

# extend new to provide site-specific defaults.
# could also overrride here if necessary.
sub new {
   my $self=shift;

   my %in = (@_);

   print "info: my_new\n" if $DEBUG;

   for my $key (keys %HTML_Template_defaults) {
      $in{template}{$key} = $HTML_Template_defaults{$key}
         if not exists $in{template}{$key};
   }

   for my $key (keys %FB_defaults) {
      $in{$key} = $FB_defaults{$key} if not exists $in{$key};
   }

   return $self->SUPER::new(%in);
}

# extend render to force content type of utf-8
sub render {
   my $self=shift;

   print "Content-Type: text/html; charset=utf-8\n\n";
   print "info: my_render\n" if $DEBUG;
   return $self->SUPER::render(@_);
}

# extend tmpl_param to accept hash and array references
sub tmpl_param {
   my $self=shift;
   my $r = shift;

   print "info: my_tmpl_param\n" if $DEBUG;

   if (ref($r) eq 'HASH') {
      for my $key (keys %{$r}) {
         $self->SUPER::tmpl_param($key, $r->{$key});
      }
   }
   elsif (ref($r) eq 'ARRAY') {
      while (@{$r}) {
         $self->SUPER::tmpl_param(shift @$r, shift @$r);
      }
   }
   else {
      unshift @_, $r;
      return $self->SUPER::tmpl_param(@_);
   }
}

1;

which can be called like this:

# test.pl

use lib '../lib';
use AppBuilder;

   my @fields = qw();

   my $form = AppBuilder->new(
             action => 'test.pl',
             fields => \@fields,
             template => {
                        filename => 'test.tt',
             },
    );

   $form->tmpl_param( { var1 => 1, var2 => 2, });

   print $form->render();

i18n

FormBuilder has crude but improving support for i18n:

  • new has a messages option for either a lang, filename of message catalog, or auto-detect lang from browser accept header. You can read about the old messages file features in the Custom Messages tutorial
  • the message catalogs are managed in modules now, one per language. Most of the common European languages plus Turkish and Japanese are translated. You can read more about that in the Changelogs.
  • HTML::Template files can be post-processed with a filter like this:
    my $lang = param('lang');
    
    my $form = CGI::FormBuilder->new(
    [...]
    template => {
       filter => sub { my_filter(shift, $lang) },
    }
    messages => $lang,
    
  • the JavaScript validation strings still show the form field names, which you can fix with code like this:
Messages.pm:34:

my $my_lang = 'en';
sub get_lang {
   $my_lang;
}

Messages.pm:84:

            $my_lang = $_;
            last;

Field.pm:642:

$alertstr =~ s/"([^"]+?)"/
MyModule::my_gettext(uc "X_$1",
CGI::FormBuilder::Messages::get_lang())/egx;

Patch for Multiple Submit Buttons and Webkit

Posted in Open Source, Perl, Tech | Leave a comment

AirAsia Flight QZ8501’s Flight Recorders Are Being Processed in Jakarta

It’s interesting that AirAsia Flight QZ8501’s (Airbus A320) flight recorders are being processed in Jakarta. It certainly makes sense for a native speaker to translate the CVR, but I didn’t know there was a lab in SE Asia.

Sadly, that may be the result of so many recent accidents there.

Indonesia does work closely with other international investigators, including having received NTSB accident investigation training. Many of their military pilots receive C-130/F-16 training in the USA and proudly carry FAA civil commercial licenses granted while overseas. :)

One official made a statement that confused the press. Likely he was thinking the Indonesian word “putus”, which can mean both break apart (on impact) and explode. Unfortunately it was related as “explosion”.

2nd AirAsia Flight QZ8501 black box recovered; fuselage possibly located

wikipedia: Indonesia AirAsia Flight 8501

Posted in Tech | Leave a comment

Airliners Helped By Recent Jet Streams

I had a 175 mph jet stream tailwind flying from Manila to SFO on Dec. 31, and BA had a 200 mph assist from Heathrow to JFK on Jan. 7.

Although the article says the BA plane was approaching the speed of sound, that’s not strictly accurate.

Somewhere the speed of sound could be 761 mph, but it was only the ground speed that was high. The actual airspeed, relative to the jet stream, was still only around 500 mph.

Since currently-flying airliners are not designed to fly near or beyond the sound barrier, you wouldn’t want to even get close to the actual speed of sound in a jet stream, which can be very turbulent. (A delta-shaped wing with flight controls tested at trans-sonic and supersonic speeds are needed.)

I vaguely remember tailwinds higher than 200 mph on the same route before, but don’t recall the actual numbers (over 800 mph ground speed.) It’s common on that route to see short return trips. I should make a note next time, as that is near the highest recorded speed according to wikipedia.

It’s amazing to me that one can fly from Asia to USA in 11 to 12 hours, weather permitting.

cnn.com: Air rage and emergency exits: Two stormy weeks in Chinese aviation

Posted in Tech, Travel | Leave a comment