/ 
HTML-Content-Extractor-0.16
River stage zero No dependents
/ HTML::Content::Extractor

NAME

HTML::Content::Extractor - Recieving a main text of publication from HTML page and main media content that is bound to the text

SYNOPSIS

 my $obj = HTML::Content::Extractor->new();
 $obj->analyze($html);
 
 my $main_text    = $obj->get_main_text();
 my $main_images  = $obj->get_main_images();
 
 my $raw_text     = $obj->get_raw_text();
 my $main_text_we = $obj->get_main_text_with_elements(1, ["a", "b", "br", "strike", ...]);
 
 print $main_text, "\n\n";
 
 print "Images:\n";
 foreach my $url (@$main_images) {
	print $url, "\n";
 }
 
 # html elements
 my $obj = HTML::Content::Extractor->new();
 
 $obj->build_tree($html);
 my $tree = $obj->get_tree();
 
 my $i = -1;
 while( my $element = $obj->get_element_by_name("div", ++$i) ) {
	print "<", $element->{name};
	
	foreach my $key (keys %{$element->{prop}}) {
		print " ", $key, '="', $element->{prop}->{$key}, '"';
	}
	
	print ">\n";
 }

DESCRIPTION

This module analyzes an HTML document and extracts the main text (for example front page article contents on the news site) and all related images.

METHODS

new

my $obj = HTML::Content::Extractor->new();

Creates and prepares the structure for the subsequent analysis and parsing HTML.

analyze

$obj->analyze($html);

Creates an HTML document tree and analyzes it.

get_main_text

# UTF-8
my $main_text = $obj->get_main_text(1);
# or not
my $main_text = $obj->get_main_text(0);
# default UTF-8 is on

Return plain text.

get_raw_text

# UTF-8
my $raw_text = $obj->get_raw_text(1);
# or not
my $raw_text = $obj->get_raw_text(0);
# default UTF-8 is on

Return the main text without post-processing (saving all html tags)

get_main_text_with_elements

# UTF-8
my $main_text_we = $obj->get_main_text_with_elements(1, ["span", ...]);
# or not
my $main_text_we = $obj->get_main_text_with_elements(0, ["span", ...]);
# default UTF-8 is on

Returns the main text while saving selected html tags. Post-processing is skipped

get_main_images

# UTF-8
my $main_images = $obj->get_main_images(1);
# or not
my $main_images = $obj->get_main_images(0);
# default UTF-8 is on

Returns ARRAY with pictures URL.

build_tree

my $res = $obj->build_tree($html);

Build flat html tree and returns 1

get_tree

my $res  = $obj->build_tree($html);
my $tree = $obj->get_tree(1);

Returns ARRAY with flat html tree

get_tree_by_element_id

my $element_tree = $obj->get_tree_by_element_id($element->{id}, 1);

Returns ARRAY with flat html tree by element id

get_element_by_name

my $element = $obj->get_element_by_name("div", 0);

Returns HASH or undef with element by tag name. ARGS: 1) tag name 2) offset

Structure of this element:

 $element = {
	id     => <number>,
	name   => <text>,
	tag_id => <number>,
	prop   => <HASH>,
	level  => <number>,
	start  => <number>,
	stop   => <number>,
	bstart => <number>,
	bstop  => <number>
 };

get_stat_by_element_id

my $element = $obj->get_stat_by_element_id($element->{id});

Returns HASH with element stats by element id. HASH included: count, all, words, AI_TEXT, AI_LINK, AI_IMG, all_AI_LINK, all_AI_LINK, all_AI_IMG

get_child

my $element = $obj->get_child(0);

get_parent

my $element = $obj->get_parent();

get_curr_element

my $element = $obj->get_curr_element();

get_prev_element

my $element = $obj->get_prev_element();

get_next_element_curr_level

my $element = $obj->get_next_element_curr_level();

get_prev_element_curr_level

my $element = $obj->get_prev_element_curr_level();

set_position

my $element = $obj->set_position($element);

Set position by element. Returns this element or undef if something is wrong

DESTROY

undef $obj;

Cleaning of all internal structures (HTML tree and other)

AUTHOR

Alexander Borisov <lex.borisov@gmail.com>

COPYRIGHT AND LICENSE

This software is copyright (c) 2013 by Alexander Borisov.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.