NAME
Text::TFIDF::Ngram - Compute the TF-IDF measure for ngram phrases
VERSION
version 0.0509
SYNOPSIS
use Text::TFIDF::Ngram;
my $t = Text::TFIDF::Ngram->new(
files => [qw( foo.txt bar.txt )],
size => 3,
);
my $w = $t->tf( 'foo.txt', 'foo bar baz' );
my $x = $t->idf('foo bar baz');
my $y = $t->tfidf( 'foo.txt', 'foo bar baz' );
printf "TF: %.3f, IDF: %.3f, TF-IDF: %.3f\n", $w, $x, $y;
my $z = $t->tfidf_by_file;
print Dumper $z;DESCRIPTION
This module computes the TF-IDF ("term frequency - inverse document frequency") measure for a corpus of text documents.
This module will only work when given more than one document. Because the idf method is computed based on all documents, a single document in the given corpus will return 0.
ATTRIBUTES
files
ArrayRef of filenames.
size
Integer ngram phrase size.
Default: 1
stopwords
Boolean indicating that phrases with stopwords will be ignored.
Default: 1
punctuation
Regular expression to be used to parse-out unwanted punctuation. Giving the constructor a value of '' or 0 will override this and not exclude any characters from the results.
Default: qr/(?!')[[:punct:]]/
Note that the default does not exclude the single quote.
lowercase
Boolean to render the ngrams in lowercase.
Default: 0
counts
HashRef of the ngram counts of each processed file.
This is a computed attribute. Providing it to the constructor will be ignored.
file_tfidf
HashRef of the TF-IDF values in each processed file.
This is a computed attribute. Providing it to the constructor will be ignored.
METHODS
new
$t = Text::TFIDF::Ngram->new(
files => \@files,
size => $size,
stopwords => $stopwords,
punctuation => $punctuation,
lowercase => $lowercase,
);Create a new Text::TFIDF::Ngram object. If the files argument is passed in, the ngrams of each file are stored in counts.
tf
$tf = $t->tf( $file, $phrase );Return the frequency of the given phrase in the document file. This is not the "raw count" of the phrase, but rather the percentage of times it is seen.
idf
$idf = $t->idf($phrase);Return the inverse document frequency of a phrase across all corpus documents.
If the phrase is not in the corpus, a warning is issued and undef is returned.
tfidf
$tfidf = $t->tfidf( $file, $phrase );Compute the TF-IDF weight for the given file and phrase.
tfidf_by_file
$tfidf = $t->tfidf_by_file;Construct a HashRef of all files with all phrases and their tfidf values.
SEE ALSO
The eg/* and t/01-methods.t files in this distribution
https://en.wikipedia.org/wiki/Tf%E2%80%93idf
AUTHOR
Gene Boggs <gene@cpan.org>
COPYRIGHT AND LICENSE
This software is copyright (c) 2022 by Gene Boggs.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.