The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.
 / 
Text-Similarity-0.04
River stage two • 3 direct dependents • 11 total dependents
/ Text::Similarity

NAME

Text::Similarity - measuring the pair-wise similarity of documents.

SYNOPSIS

      # this will return an un-normalized score that just gives the
      # number of overlaps

      use Text::Similarity::Overlaps;
      my $mod = Text::Similarity::Overlaps->new;
      defined $mod or die "Construction of Text::Similarity::Overlaps failed";

      # adjust file names to reflect true relative position
      # these paths are valid from lib/Text/Similarity
      my $text_file1 = 'Overlaps.pm';
      my $text_file2 = '../OverlapFinder.pm';

      my $score = $mod->getSimilarity ($text_file1, $text_file2);
      print "The similarity of $text_file1 and $text_file2 is : $score\n";

      # if you want to turn on the verbose option and provide a stoplist
      # you can pass those parameters to Overlaps.pm via hash arguments

      use Text::Similarity::Overlaps;
      my %options = ('verbose' => 1, 'stoplist' => '../../bin/stoplist.txt');

      my $mod = Text::Similarity::Overlaps->new (\%options);
      defined $mod or die "Construction of Text::Similarity::Overlaps failed";

      # adjust file names to reflect true relative position
      # these paths are valid from lib/Text/Similarity
      my $text_file1 = 'Overlaps.pm';
      my $text_file2 = '../OverlapFinder.pm';

      my $score = $mod->getSimilarity ($text_file1, $text_file2);
      print "The similarity of $text_file1 and $text_file2 is : $score\n";

DESCRIPTION

This module is a superclass for other modules and provides generic services such as stop word removal, compound identification, and text cleaning or sanitizing.

It's important to realize that additional methods of measuring similarity can be added to this package. Text::Similarity::Overlaps is just one possible way of measuring similarity, others can be added.

Subroutine sanitizeString carries out text cleaning. Briefly, it removes nearly all punctuation except for underscores and embedded apostrophes, converts all text to lower case, and collapes multiple white spaces to a single space.

This module is where compounds are identified (although currently disabled). When implemented it will check a list of compounds provided by the user, and then when a compound is found in the text it will be desigated via an underscore (e.g., white house might be converted to white_house).

Stop words are removed here. The length of the documents reported does not include the stop words. Overlaps are found after stopword removal. By including a word in the stoplist, you are saying that the word never existed in your input (in effect).

SEE ALSO

AUTHOR

Ted Pedersen, University of Minnesota, Duluth tpederse at d.umn.edu

Siddharth Patwardhan sidd at cs.utah.edu

Jason Michelizzi

Last modified by : $Id: Similarity.pm,v 1.17 2008/03/21 23:01:49 tpederse Exp $

COPYRIGHT AND LICENSE

Text::Similarity Copyright (C) 2004-2008, Ted Pedersen, Jason Michelizzi, and Siddharth Patwardhan

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA