NAME

HTTP::RobotsTag::Rules - A Representation Of Robots Exclusion Rules

SYNOPSIS

$rules = $p->parse_rules($html);
$bool  = $rules->can_index();
$bool  = $rules->can_follow();
$bool  = $rules->can_archive();
$bool  = $rules->can_snippet();
$bool  = $rules->is_available( $dt );

DESCRIPTION

HTTP::RobotsTag::Rules represents the robots exclusion policies that are described within HTTP headers.

METHODS

new

can_index

can_follow

can_archive

can_snippet

is_available($dt)

Returns true if the resource is available on $dt.