NAME

WWW::RobotRules::Parser::MultiValue - Parse robots.txt

SYNOPSIS

use WWW::RobotRules::Parser::MultiValue;
use LWP::Simple qw(get);

my $url = 'http://example.com/robots.txt';
my $robots_txt = get $url;

my $rules = WWW::RobotRules::Parser::MultiValue->new(
    agent => 'TestBot/1.0',
);
$rules->parse($url, $robots_txt);

if ($rules->allows('http://example.com/some/path')) {
    my $delay = $rules->delay_for('http://example.com/');
    sleep $delay;
    ...
}

my $hash = $rules->rules_for('http://example.com/');
my @list_of_allowed_paths = $hash->get_all('allow');
my @list_of_custom_rule_value = $hash->get_all('some-rule');

DESCRIPTION

WWW::RobotRules::Parser::MultiValue is a parser for robots.txt.

Parsed rules for the specified user agent is stored as a Hash::MultiValue, where the key is a lower case rule name.

Request-rate rule is handled specially. It is normalized to Crawl-delay rule.

METHODS

new

$rules = WWW::RobotRules::Parser::MultiValue->new(
    aget => $user_agent
);
$rules = WWW::RobotRules::Parser::MultiValue->new(
    aget => $user_agent,
    ignore_default => 1,
);

Creates a new object to handle rules in robots.txt. The object parses rules match with $user_agent. The rules of User-agent: * always match and have a lower precedence than the rules explicitly matched with $user_agent. If ignore_default option is specified, rules of User-agent: * are simply ignored.

parse

$rules->parse($uri, $text);

Parses a text content $text whose URI is $uri.

match_ua

$rules->match_ua($pattern);

Test if the user agent matches with $pattern.

rules_for

$hash = $rules->rules_for($uri);

Returns a Hash::MultiValue, which describes the rules of the domain of $uri.

allows

$test = $rules->allows($uri);

Tests if the user agent is allowed to visit $uri. If there is 'Allow' rule for the path of $uri, then the $uri is allowed to visit. If there is 'Disallow' rule for the path of $uri, then the $uri is not allowed to visit. Otherwise, the $uri is allowed to visit.

delay_for

$delay = $rules->delay_for($uri);
$delay_in_milliseconds = $rules->delay_for($uri, 1000);

Calculate a crawl delay for the specified $uri. The value is determined by 'Crawl-delay' rule or 'Request-rate' rule. The second argument specifies the base of the return value.

SEE ALSO

Hash::MultiValue

LICENSE

Copyright (C) INA Lintaro

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

AUTHOR

INA Lintaro <tarao.gnn@gmail.com>