/ 
GunghoX-FollowLinks-0.00006
River stage zero No dependents
/ GunghoX::FollowLinks

NAME

GunghoX::FollowLinks - Automatically Follow Links Within Responses

SYNOPSIS

follow_links:
  parsers:
    - module: HTML
      config:
        rules:
          - module: HTML::SelectedTags
            config:
              tags:
                - a
          - module: MIME
            config:
              types:
                - text/html
    - module: Text
      config:
        rules:
          - module: URI
            config:
              match:
                - host: ^example\.com
                  action: FOLLOW_ALLOW

package MyHandler;
sub handle_response
{
  my ($self, $c, $req, $res) = @_;
  $c->follow_links($res);
}

DESCRIPTION

The most common action that a crawler takes is to follow links on a page. This module helps you with that task.

METHODS

setup

Parses the given HTTP::Response/Gungho::Response object and dispatches the appropriate parser from its content-type.

For each URL found, Automatically dispatches the rules given to the parser, and if the rules match, the URL is sent to Gungho->send_request.

Returns the number of matches found.

AUTHOR

Copyright (c) 2007 Daisuke Maki <daisuke@endeworks.jp>

LICENSE

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

See http://www.perl.com/perl/misc/Artistic.html