NAME

Text::Identify::BoilerPlate - Remove repeated text

VERSION

Version 0.2

SYNOPSIS

Finds boilerplate text (lines that are repeated across documents) in a list of plain text files.

use Text::Identify::BoilerPlate;

my @files = ('file1', 'file2', 'file3');
rem_boilerplate(\@files, { min_dupl => 4, ignore_digits => 0 });

New files are written, containing everything but the boilerplate text.

FUNCTIONS

rem_boilerplate()

rem_boilerplate() takes two arguments: A reference to a list of files to be processed, and a reference to a hash of options.

The options are:

min_dupl - the minimum number of thimes a line has to occur to be considered boilerplate (default: 3)

ignore_digits - lines only seperated by differences in digits will be considered duplicates (default: yes)

suffix - added to the new files (default: 'content')

only_headers_and_footers - Only sets consecutive lines of duplicates at the start and end of documents are considered boilerplate (default: yes)

digest - lines will be replaced by a MD5 digest during duplicate compilation, saving memory (default: no)

log - name of the log file, where deleted lines are recorded; if set to false, no log will be created (default: './text-identify-boilerplate.log')

AUTHOR

Lars Nygaard, <lars.nygaard@inl.uio.no>

BUGS

The program needs extensive testing and tweaking before the simple algorithm can give consistently high-quality results.

Please report any bugs or feature requests to bug-text-identify-boilerplate@rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Text-Identify-BoilerPlate. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

ACKNOWLEDGEMENTS

COPYRIGHT & LICENSE

Copyright 2005 Lars Nygaard, all rights reserved.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.