/ 
HTML-Detergent-0.06
River stage one • 1 direct dependent • 1 total dependent
/ HTML::Detergent

NAME

HTML::Detergent - Clean the gunk off an HTML document

VERSION

Version 0.06

SYNOPSIS

use HTML::Detergent;

my $scrubber = HTML::Detergent->new($config);

# $input can be a string, GLOB reference, or XML::LibXML::Document

my $doc = $scrubber->process($input, $uri);

DESCRIPTION

HTML::Detergent is for isolating the main content of an HTML page, stripping it of navigation, visual design, and other ancillary content.

The main purpose of this module is to aid in the migration of web content from one content management system to another. It is also useful for preparing HTML resources for automated content inventories.

The module currently has no heuristics for determining the main content of a page. It works instead by assuming prior knowledge of the layout, given in the configuration by an XPath expression that uniquely isolates the container node. That node is then lifted into a new document, along with the contents of the <head>, and returned by the "process" method. To accommodate multiple layouts on a site, the module can be initialized to match multiple XPath expressions. If further processing is necessary, an expression can be associated with an XSLT stylesheet, which is assumed to produce an entire document, thus overriding the default behaviour.

After the new document is generated and before it is returned by "process", it is possible to inject <link> and <meta> elements into the <head>. This enables the inclusion of metadata and the re-association of the main content with links that represent aspects of the page which have been removed (e.g. navigation, copyright statement, etc.). In addition, if the page's URI is supplied to the "process" method, the <base> element is either added or rewritten to reflect it, and the URI attributes in the body are rewritten relative to the base. Otherwise they are left alone.

The document returned is an XML::LibXML::Document object using the XHTML namespace, http://www.w3.org/1999/xhtml, but does not profess to validate against any particular schema. If DTD declarations (including the empty <!DOCTYPE html> recommended in HTML5) are desired, they can be added on afterward. Likewise, the object can be converted from XML into HTML using "toStringHTML" in XML::LibXML::Document.

METHODS

new %CONFIG | \%CONFIG | $CONFIG

Initialize the processor, either with a list of configuration parameters, a HASH reference thereof, or an HTML::Detergent::Config object. Below are the valid parameters:

match

This is an ARRAY reference of XPath expressions to try against the document, in order of preference. Entries optionally may be two-element ARRAY references themselves, the second element being a URL where an XSLT stylesheet may be found.

match => [ '/some/xpath/expression',
           [ '/other/expr', '/url/of/transform.xsl' ],
         ],

This is a HASH reference where the keys correspond to rel attributes and the values to href attributes of <link> elements. If the values are ARRAY references, they will be processed in document order. rel attributes will be sorted lexically. If a callback is supplied instead, the caller expects a result of the same form.

link => { rel1 => 'href1', rel2 => [ qw(href2 href3) ] },

# or

link => \&_link_cb,
meta

This is a HASH reference where the keys correspond to name attributes and the values to content attributes of <meta> elements. If the values are ARRAY references, they will be processed in document order. name attributes will be sorted lexically. If a callback is supplied instead, the caller expects a result of the same form.

meta => { name1 => 'content1',
          name2 => [ qw(content2 content3) ] },

# or

meta => \&_meta_cb,
callback

These callbacks will be passed into the internal XML::LibXSLT processor. See XML::LibXML::InputCallback for details.

callback => [ \&_match_cb, \&_open_cb, \&_read_cb, \&_close_cb ],

# or

callback => $icb, # isa XML::LibXML::InputCallback

process $INPUT [, $URI, $CONFIG ]

Processes $INPUT, which may be a string, GLOB reference, or XML::LibXML::Document object. Returns an XML::LibXML::Document object with the changes mentioned in the "DESCRIPTION".

AUTHOR

Dorian Taylor, <dorian at cpan.org>

BUGS

Please report any bugs or feature requests to bug-html-detergent at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=HTML-Detergent. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

perldoc HTML::Detergent

You can also look for information at:

SEE ALSO

XML::LibXML
XML::LibXSLT
HTML::HTML5::Parser

LICENSE AND COPYRIGHT

Copyright 2013 Dorian Taylor.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.