sv.pm.org: Building a Web Crawler with Perl

Perl LogoTonite at Silicon Valley Perl Mongers Group (sv.pm.org), Lambert Lum gave a talk on “Building a Web Crawler with Perl.”

Largely he talked about the following Perl modules:

Joe McMahon, a CPAN maintainer of 30 modules including WWW::Mechanize::Plugin, was available to answer detailed questions.

Afterwards, several members made very interesting comments on their search projects, including using HTML Tidy to fix broken pages before parsing, and to pay attention to site maps.

There was a lot of interest in what to do with JavaScript in crawled web pages.

Thanks to Plug and Play Tech Center for hosting the meeting again.

Silicon Valley Perl Mongers Wiki
Sauce Labs

This entry was posted in Open Source, Perl, Tech, User Groups. Bookmark the permalink.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.