/ 
HTTP-OAI-3.01
River stage one • 2 direct dependents • 2 total dependents
/ HTTP::OAI::Harvester

NAME

HTTP::OAI::Harvester - Agent for harvesting from an Open Archives 1.0,1.1, 2.0 and static compatible repositories

DESCRIPTION

HTTP::OAI::Harvester provides the front-end to the OAI-PERL library for harvesting from OAI repositories. Direct use of the other OAI-PERL modules for harvesting should be avoided.

To harvest from an OAI-compliant repository first create the HTTP::OAI::Harvester interface using the baseURL option. It is recommended that you request an Identify from the Repository and use the repository method to update the Identify object used by the harvester.

When making OAI requests the underlying HTTP::OAI::UserAgent module will take care of automatic redirection (error code 302) and retry-after (error code 503).

OAI flow control (i.e. resumption tokens) is handled transparently by HTTP::OAI::Harvester.

Static Repository Support

Static repositories are automatically and transparently supported within the existing API. To harvest a static repository specify the repository XML file using the baseURL argument to HTTP::OAI::Harvester. An initial request is made that determines whether the base URL specifies a static repository or a normal OAI 1.x/2.0 CGI repository.

If a static repository is found the response is cached, and further requests are served by that cache. Static repositories do not support sets. You can determine whether the repository is static by checking the version ($ha->repository->version), which will be "2.0s" for static repositories.

FURTHER READING

You should refer to the Open Archives Protocol version 2.0 and other OAI documentation, available from http://www.openarchives.org/.

BEFORE USING EXAMPLES

In the examples I use arXiv.org's, and cogprints OAI interfaces. To avoid causing annoyance to their server administrators please contact them before performing testing or large downloads (or use other, less loaded, servers).

SYNOPSIS

use HTTP::OAI;

my $h = new HTTP::OAI::Harvester(baseURL=>'http://arXiv.org/oai2');
my $response = $h->repository($h->Identify)
if( $response->is_error ) {
	print "Error requesting Identify:\n",
		$response->code . " " . $response->message, "\n";
	exit;
}

# Note: repositoryVersion will always be 2.0, $r->version returns
# the actual version the repository is running
print "Repository supports protocol version ", $response->version, "\n";

# Version 1.x repositories don't support metadataPrefix,
# but OAI-PERL will drop the prefix automatically
# if an Identify was requested first (as above)
$response = $h->ListIdentifiers(
	metadataPrefix=>'oai_dc',
	from=>'2001-02-03',
	until=>'2001-04-10'
);

if( $response->is_error ) {
	die("Error harvesting: " . $response->message . "\n");
}

print "responseDate => ", $response->responseDate, "\n",
	"requestURL => ", $response->requestURL, "\n";

while( my $id = $response->next ) {
	if( $id->is_error ) {
		print "Error: ", $id->code, " (", $id->message, ")\n";
		last;
	}
	print "identifier => ", $id->identifier;
	# Only available from OAI 2.0 repositories
	print " (", $id->datestamp, ")" if $id->datestamp;
	print " (", $id->status, ")" if $id->status;
	print "\n";
	# Only available from OAI 2.0 repositories
	for( $id->setSpec ) {
		print "\t", $_, "\n";
	}
}

$response = $h->ListRecords(
	metadataPrefix=>'oai_dc',
	handlers=>{metadata=>'HTTP::OAI::Metadata::OAI_DC'},
);
if( $response->is_error ) {
	print "Error: ", $response->code,
		" (", $response->message, ")\n";
	exit();
}

while( my $rec = $response->next ) {
	if( $rec->is_error ) {
		die $rec->message;
	}
	print $rec->identifier, "\t",
		$rec->datestamp, "\n",
		$rec->metadata, "\n";
}

METHODS

$h = new HTTP::OAI::Harvester(baseURL=>'http://arXiv.org/oai1',repository=>new OAI::Identify(baseURL=>'http://cogprints.soton.ac.uk/perl/oai')[, resume=>0])

This constructor method returns a new instance of HTTP::OAI::Harvester. Requires either an HTTP::OAI::Identify object, which in turn must contain a baseURL, or a baseURL from which to construct an Identify object.

Any other options are passed to the HTTP::OAI::UserAgent module, and from there to the LWP::UserAgent module.

The resume argument controls whether resumptionToken flow control is handled internally. By default this is 1. If flow control is not handled by the library programs should check the resumptionToken method to establish whether there are more records.

$repo = $h->repository([repo])

Returns and optionally sets the HTTP::OAI::Identify data used by the Harvester agent.

$r = $h->GetRecord(identifier=>'oai:arXiv:hep-th/0001001',metadataPrefix=>'oai_dc')
$r = $h->Identify
$r = $h->ListIdentifiers(metadataPrefix=>'oai_dc',-from=>'2001-10-01',until=>'2001-10-31',set=>'physics:hep-th',resumptionToken=>'xxx')
$r = $h->ListMetadataFormats(identifier=>'oai:arXiv:hep-th/0001001')
$r = $h->ListRecords(from=>'2001-10-01',until=>'2001-10-01',set=>'physics:hep-th',metadataPrefix=>'oai_dc',resumptionToken=>'xxx')
$r = $h->ListSets(resumptionToken=>'xxx')

These methods perform an OAI request corresponding to their name. The options are specified in the OAI protocol document, with a prepended dash. resumptionToken is exclusive and will override any other options passed to the method.

These methods either return either a:

HTTP::Response

If there was a problem making the HTTP request.

Or, a module subclassed from HTTP::OAI::Response, corresponding to the method name (e.g. GetRecord will return HTTP::OAI::GetRecord).

Use $r->is_success to determine whether an error occurred.

Use $r->code and $r->message to obtain the error code and a human-readable message. OAI level errors can be retrieved using the $r->errors method.

If the response contained an HTTP::OAI::ResumptionToken this can be retrieved using the $r->resumptionToken method.

ABOUT

These modules have been written by Tim Brody <tdb01r@ecs.soton.ac.uk>.

You can find links to this and other OAI tools (perl, C++, java) at: http://www.openarchives.org/tools/tools.html.