Perl & LWP: Fetching Web Pages, Parsing Html, Writing Spiders & More

Perl soared to popularity as a language for creating and managing web content, but with LWP (Library for WWW in Perl), Perl is equally adept at consuming information on the Web. LWP is a suite of modules for fetching and processing web pages. The Web is a vast data source that contains everything from stock prices to movie credits, and with LWP all that data is just a few lines of code away. Anything you do on the Web, whether it's buying or selling, reading or writing, uploading or downloading, news to e-commerce, can be controlled with Perl and LWP. You can automate Web-based purchase orders as easily as you can set up a program to download MP3 files from a web site. Perl & LWP covers:

• Understanding LWP and its design
• Fetching and analyzing URLs
• Extracting information from HTML using regular expressions and tokens
• Working with the structure of HTML documents using trees
• Setting and inspecting HTTP headers and response codes
• Managing cookies
• Accessing information that requires authentication
• Extracting links
• Cooperating with proxy caches
• Writing web spiders (also known as robots) in a safe fashion

Perl & LWP includes many step-by-step examples that show how to apply the various techniques. Programs to extract information from the web sites of BBC News, Altavista, ABEBooks.com, and the Weather Underground, to name just a few, are explained in detail, so that you understand how and why they work. Perl programmers who want to automate and mine the web can pick up this book and be immediately productive. Written by a contributor to LWP, and with a foreword by one of LWP's creators, Perl & LWP is the authoritative guide to this powerful and popular toolkit.

Perl & LWP sets out to unwrap the Library for the Web in Perl (LWP), which is a collection of modules that make it easier to access and pick apart Web pages (and FTP-accessible files, and outgoing e-mail messages) from within your Perl programs. The book succeeds wonderfully, not only in conveying the technical aspects of LWP programming, but in making clear the fun of doing work that's very well suited to Perl. Sean Burke assumes that his readers know something about Perl, albeit not much, and a similar amount about HTML. He does a great job of explaining how LWP functions fit into Perl programs, and how you can use them to make reference to Internet resources far more easily than before.

Burke's narrative takes the form of a guided tour in which he introduces his readers to aspects of the LWP modules one by one. His tone is generally straightforward (sharp commentary alternates with brief code listings, with occasional passages of reference material), but there's sometimes an undercurrent of exuberance that makes the reader want to get going with his or her own programming right away. Overall, the emphasis is on teaching both LWP and Perl itself to the extent necessary to do LWP work. Because of the concise and nicely indexed code modules, though, you'll find this book useful as a reference after you're under way with LWP. --David Wall

Topics covered: How to program with LWP and Perl itself. All of LWP's strong points--including HTML parsing (with tokens and trees as well as with regular expressions), HTML generation and modification, manipulation of HTML forms, and the operation of spiders--are covered. This book has more of a tutorial tone than any similar reference material on the Internet.