NAME

CrawlerCommons::RobotRules - the result of a parsed robots.txt

SYNOPSIS

use CrawlerCommons::RobotRules;
use CrawlerCommons::RobotRulesParser;

my $rules_parser = CrawlerCommons::RobotRulesParser->new;

my $content = "User-agent: *\r\nDisallow: *images";
my $content_type = "text/plain";
my $robot_names = "any-old-robot";
my $url = "http://domain.com/";

my $robot_rules =
  $rules_parser->parse_content($url, $content, $content_type, $robot_names);

# obtain the 'mode' of the robot rules object
say "Anything Goes!!!!" if $robot_rules->is_allow_all;
say "Nothing to see here!" if $robot_rules->is_allow_none;
say "Default robot rules mode..." if $robot_rules->is_allow_some;

# are we allowed to crawl a URL (returns 1 if so, 0 if not)
say "We're allowed to crawl the index :)"
 if $robot_rules->is_allowed( "https://www.domain.com/index.html");

say "Not allowed to crawl: $_" unless $robot_rules->is_allowed( $_ )
  for ("http://www.domain.com/images/some_file.png",
       "http://www.domain.com/images/another_file.png");

DESCRIPTION

This object is the result of parsing a single robots.txt file

VERSION

Version 0.03

METHODS

my $true_or_false = $robot_rules->is_allowed( $url )

Returns 1 if we're allowed to crawl the URL represented by $url and 0 otherwise. Will return 1 if the method is_allow_all() returns true, otherwise, if is_allow_none is false, returns 1 if there is an allow rule or no disallow rule for this URL.

• $url

  The URL whose path is used to search for a matching rule within the object for evaluation.

AUTHOR

Adam Robinson <akrobinson74@gmail.com>