Using Sphinx Search with Perl

I had been aware of Sphinx Search since meeting Andrew Aksyonoff, the author, at the MySQL Conference a couple ago, but just got around to using it.

Sphinx is a full-text indexing and search engine written in C that is compact and fast and has support for the most common scripting languages.

I’d say that Apache/SOLR and Sphinx are the 2 leading Open Source full-text search engines currently.

I maintain an application that used MySQL Full Text search for several years, but as the index grew in size, insert performance became too variable for a good user experience, especially under heavy disk load.

Downloading Sphinx Search 0.9.9-rc2, installing, making the initial index and getting a sample program working only took about an hour.

The very well-written manual is about 100 pages, so it would take longer to read all the options available, such as tuning, stemming and stripping html.

Initially, I ported my app search UI program to use contrib/perlapi/Sphinx.pm, which worked ok. There’s an actively-maintained version of Sphinx.pm on CPAN called Sphinx::Search, so I switched to that later.

Sample Sphinx indexer process output:

Sphinx 0.9.9-rc2 (r1785)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file '/etc/sphinx.conf'...
indexing index 'job'...
collected 20521 docs, 145.9 MB
sorted 21.7 Mhits, 100.0% done
total 20521 docs, 145855157 bytes
total 28.207 sec, 5170877 bytes/sec, 727.51 docs/sec
total 23 reads, 0.083 sec, 2583.7 kb/call avg, 3.6 msec/call avg
total 145 writes, 0.639 sec, 948.2 kb/call avg, 4.4 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=15511).

Sphinx searchd returns 50 results in about one-tenth of a second, which is fine for this application.

Andrew is adding enhanced incremental indexing ability, which currently has to be done using 2 indexes, main and delta.

He gave a live demo at the Percona Performance Conference of adding Sphinx Search to the Mantis source code in 20 minutes – then wiped it out – to groans from the audience who wanted to deploy it.