NAME
huge-merge.pl - Merge the results of multiple huge-sort generated files into a single sorted file.
SYNOPSIS
count.pl --tokenlist input1.out input1 count.pl --tokenlist input2.out input2
huge-sort.pl input1.out huge-sort.pl input2.out
mv input1.out-sorted output-directory mv input2.out-sorted output-directory
huge-merge.pl output-directory
DESCRIPTION
Combine the sorted bigram files generated by huge-sort.pl efficiently.
This program is used internally by huge-count.pl.
USGAE
huge-merge.pl [OPTIONS] SOURCEDIR
INPUT
Required Arguments:
SOURCEDIR
Input to huge-merge.pl should be a single flat directory containing multiple plain text files generated by huge-sort.pl. The result file, merge.* (* is a number, the final result file has the maximum number), is in the source directory.
Optional Arguments:
--keep
Switches ON the --keep option will keep all the intermediate merging files.
--remove L
Bigrams with counts less than L in the entire SOURCE data are removed from the sample. The counts of the removed bigrams are not counted in any marginal totals. This has same effect as count.pl's --remove option.
--uremove L
Bigrams with counts more than L in the entire SOURCE data are removed from the sample. The counts of the removed bigrams are not counted in any marginal totals. This has same effect as count.pl's --uremove option.
--frequency F
Bigrams with counts less than F in the entire SOURCE are not displayed. The counts of the skipped bigrams ARE counted in the marginal totals. In other words, --frequency in huge-count.pl has same effect as the count.pl's --frequency option.
--ufrequency F
Bigrams with counts more than F in the entire SOURCE are not displayed. The counts of the skipped bigrams ARE counted in the marginal totals. In other words, --frequency in huge-count.pl has same effect as the count.pl's --ufrequency option.
Other Options:
--help
Displays the help information.
--version
Displays the version information.
AUTHOR
Ying Liu, University of Minnesota, Twin Cities. liux0395 at umn.edu
Ted Pedersen, University of Minnesota, Duluth. tpederse at umn.edu
COPYRIGHT
Copyright (C) 2009-2010, Ying Liu and Ted Pedersen
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.