NAME
HTTP::RobotsTag::Rules - A Representation Of Robots Exclusion Rules
SYNOPSIS
$rules = $p->parse_rules($html);
$bool = $rules->can_index();
$bool = $rules->can_follow();
$bool = $rules->can_archive();
$bool = $rules->can_snippet();
$bool = $rules->is_available( $dt );
DESCRIPTION
HTTP::RobotsTag::Rules represents the robots exclusion policies that are described within HTTP headers.
METHODS
new
can_index
can_follow
can_archive
can_snippet
is_available($dt)
Returns true if the resource is available on $dt.