NAME

WWW::Leech::Parser - HTML Page parser used by WWW::Leech::Walker

SYNOPSIS

use WWW::Leech::Parser;

my $parser = new WWW::Leech::Parser({
  'item:link' => '//a[contains(@class,"item-link")]',
  'nextpage:link' => '//a[contains(@class,"next-page-link")]',
  'fields' => {
    'name' => '//h1',
    'images[]' => '//img/@src',
    'comments[]' =>{
      type => 'html',
      xpath => '//div[@class="comments"]/div',
      filter => sub{
        my $values = shift;
        my $field_defs = shift;

        # ....

        return $values;
      }

    }
    # ....
  }
});

my $html_string = '...';

my $links_and_next_page_url = $parser->parseList($html_string);

my $item = $parser->parse($html_string);

DESCRIPTION

WWW::Leech::Parser extracts certain information from web page using provided XPath expressions.

First of all it is used to get links to 'sub-pages' and links to 'next-page' from a links-list-page (e.g. search engine results). Also it extracts required data from given HTML using rules defined upon object creation.

DETAILS

new($rules)

$rules is a hashref with following keys:

item:link

XPath extracting links to sub-pages

nextpage:link

XPath extracting link to next links-list page

fields

Fields tell parser how to extract data. Can be provided as an arrayref:

$fields = [
  {
    name => 'fieldname1',
    xpath => '//somenode'
  },
  {
    name => 'fieldname2',
    xpath => '//othernode'
  }
]

Or a hashref:

$fields = 
  {
    fieldname1 => '//somenode',
    fieldname2 => {
      xpath => '//othernode'
    }
  }
]

By default parser uses first node found text as a value for the element. Appending '[]' sequence to key name switches parser to 'wantarray' mode. Parser will return an array of values in this case.

Every element can be provided in a simple or a complex form.

Simple form is just a key-value pair where key is a name of a field and value is an XPath expression.

In complex form a hashref determining details about the field must be provided. Following keys are recognized:

xpath

Required.

XPath expression for element data.

type

Optional.

text - gets text content only (default)
html - extracts all node content including node itself as is
int - not appliable in 'wantarray' mode - removes non numeric characters from text value
unique - only appliable in 'wantarray' mode - removes duplicates 

filter

Optional.

Coderef. Parser runs filter callback passing extracted value and field definitions. Field value is replaced with whatever callback returns.

parseList($html_string)

returns list-page links as a hashref:

{
  links => [...], # URL's array
  links_text => [...], # Text inside corresponding 'a' tags
  next_page => "/page/N" # next page URL
}

parse($html_string)

returns hashref with data extracted from page using 'fields' section from rules

AUTHOR

Dmitry Selverstov
CPAN ID: JAREDSPB
jaredspb@cpan.org

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

The full text of the license can be found in the LICENSE file included with this module.