/ 
HTML-Content-Extractor-0.01
River stage zero No dependents
/ HTML::Content::HTMLTokenizer

NAME

HTML::Content::HTMLTokenizer - Perl module to tokenize HTML documents.

SYNOPSIS

use HTML::Content::HTMLTokenizer;

my $tokenizer = new HTML::Content::HTMLTokenizer('TAG','WORD');

open(HTML,"index.html");
my $doc = join("",<HTML>);
close(HTML);

my ($word_count_arr_ref,$tag_count_arr_ref,$token_type_arr_ref,$token_hash_ref) = $tokenizer->Tokenize($doc);

DESCRIPTION

HTML::Content::HTMLTokenizer has one main method, Tokenize, which tokenizes a HTML document into a sequence of 'TAG' and 'WORD' tokens.

Methods

  • my $tokenizer = new HTML::Content::HTMLTokenizer($tagMarker,$wordMarker)

    Initializes HTML::Content::HTMLTokenizer.

    $tagMarker - String that will represent tags in the token sequence returned from Tokenize.

    $wordMarker - String that will represent words in the token sequence returned from Tokenize.

  • my (\@WordCount,\@TokenCount,\@Sequence,\%Tokens) = $tokenizer->Tokenize(\$htmldocument);

    $WordCount[$i] is the number of word tokens before or at the ith token in the input HTML document.

    $TagCount[$i] is the number of tag tokens before or at the ith token in the input HTML document.

    $Sequence[$i] is the type of token at the ith spot in the input HTML document. Either $tagMarker or $wordMarker.

    $Tokens{$i} is the word at the ith spot in the input HTML document. This is defined only if there is a word at the ith spot in the document.

AUTHOR

Jean Tavernier (jj.tavernier@gmail.com)

COPYRIGHT

Copyright 2005 Jean Tavernier. All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

ContentExtractorDriver.pl (1), HTML::Content::ContentExtractor (3), HTML::WordTagRatio::Ratio (3),HTML::WordTagRatio::WeightedRatio (3), HTML::WordTagRatio::SmoothedRatio (3), HTML::WordTagRatio::RelativeRatio (3), HTML::WordTagRatio::ExponentialRatio (3), HTML::WordTagRatio::NormalizedRatio (3).