/ 
Lingua-Align-0.04
River stage zero No dependents
/ treealign

NAME

treealign - training tree alignment classifiers and aligning syntactic trees

SYNOPSIS

treealign [OPTIONS]

# train a model from tree aligned data
treealign -n 100 -m treealign.model -a train-data.xml

# aligning a parallel treebank
treealign -m treealign.model -a parallel-treebank.xml > aligned.xml

DESCRIPTION

This script allows you to train a tree alignment model and to apply them to parallel treebanks. Tree alignment is based on local binary classification and rich feature sets.

Currently, training data has to be in Stockholm Tree Aligner format. The output format is the same format. Here is a short example of this format (taking from the output of the TreeAligner):

<?xml version="1.0" ?>
<treealign>
<head>
 <alignment-metadata>
   <date>Tue May  4 16:23:04 2010</date>
   <author>Lingua-Align</author>
 </alignment-metadata>
</head>
 <treebanks>
   <treebank filename="treebanks/en/smultron_en_sophie.xml" id="en"/>
   <treebank filename="treebanks/sv/smultron_sv_sophie.xml" id="sv"/>
 </treebanks>
 <alignments>
   <align author="Lingua-Align" prob="0.11502659612149206125" type="fuzzy">
     <node node_id="s105_17" type="t" treebank_id="en"/>
     <node node_id="s109_23" type="t" treebank_id="sv"/>
   </align>
   <align author="Lingua-Align" prob="0.45281832125339427364" type="fuzzy">
     <node node_id="s105_34" type="t" treebank_id="en"/>
     <node node_id="s109_15" type="t" treebank_id="sv"/>
   </align>
 </alignments>
</treealign>

OPTIONS

There is a number of options that can be specified on the command line.

Input options

-a parallel-treebank-file

Name of the file that contains the parallel treebank. Default format is Stockholm Tree Aligner format (where the sentence alignment is implicitely given by tree node alignments). To use a different format use the option -A

-A format

Format of the parallel treebank/corpus. Default is sta (Stockholm Tree Aligner format). Other options are, for example, 'opus' (CES XML format as it is used in the OPUS corpus)

-s source-treebank-file

Name of the files that contains the source language treebank. This is useful to sepcify a file that is different from the one that is specified in the 'parallel-treebank-file'. For example, sentence alignment files from OPUS usually refer to non-parsed XML files. With -s we can overwrite this and refer to the parsed corpus instead. However, be aware that the same sentences have to be covered in the same order and appropriate IDs of these sentences have to be found when reading through the treebank files.

-S format

Format of the source language treebank. Default is TigerXML (which is used in the Stockholm Tree Aligner)

-t target-treebank-file

Name of the target language treebank file (similar to -s but for the target language)

-T format

Format of the target language treebank (similar to -S)

-w

Swap alignment direction when reading through the parallel treebank

-i

Try to align index nodes as well (used in AlpinoXML)

Training options

Training will be enabled if a positive number of training sentences iss specified with the -n option OR the modelfile does not exist.

-n nr_sent

Specify how many sentence (tree) pairs will be used for training a new tree-aligner model.

-f features

Define features to be used in training. (For alignment, features are taken from the modelfile.feat file!!) 'features' is a string with feature types separated by ':'. There are various features that can be used and combined. For more details look at Lingua::Align::Trees::Features. The default is 'insideST2:insideTS2:outsideST2:outsideTS2'

-m model-file

Name of the file to store model parameters / read model parameters

-c classifier

Classifier to be used. Default is 'megam'. Another possiblity is 'clue' which refers to a noisy-or like classifier with independent precision-weighted features (requires probabilistic values for each feature and supports only positive features). Other classifiers may be supported in future releases of Lingua::Align.

-M moses-dir

Directory with the GIZA++ and Moses word alignment files that will be used for extracting certain features. Default is 'moses' and the treealigner expects to find files with the following names

<moses-dir>/model/lex.0-0.e2f
<moses-dir>/model/lex.0-0.f2e
<moses-dir>/giza.src-trg/src-trg.A3.final.gz
<moses-dir>/giza.trg-src/trg-src.A3.final.gz
<moses-dir>/model/aligned.intersect

An alterantive way of specifying the location of word alignment files is to use the options (-d -D -g -G -y), see below.

-d lexe2f

Path to the probabilistic source-to-target lexicon created by Moses from the word aligned corpus. Of course, it could be any kind of bilingual dictionary as long as it provides a score for each entry and it uses the same format as the one created by Moses. Default is moses/model/lex.0-0.e2f.

-D lexf2e

Similar to -d but for the target-to-source lexicon. Default is moses/model/lex.0-0.f2e

-g giza.e2f.A3

Path to the Viterbi word alignment (source-to-target) created by GIZA++ (or other word aligners producing aligments in the same format). Default is moses/giza.trg-src/trg-src.A3.final.gz.

-G giza.f2e.A3

Similar to -g but for the other alignment direction. Default = moses/giza.trg-src/trg-src.A3.final.gz

-y symal-file

Path to the symmetrized word alignment format created by Moses (or other tools). Default = moses/model/aligned.intersect

-I id-file

Name of the file that contains pairs of IDs for all sentences that have been word aligned with GIZA++/Moses. This is useful to match sentences when reading word alignment files for feature extraction (sometimes not all sentences are included in both, the parsed collection and the word aligned data!). Note that word aligments and parallel treebanks have still to be stored in the same order but sentences may be skipped if they do not appear in one of them. The format is like follows:

## source-file-name    target-file-name
src-id1   trg-id1
src-id2   trg-id2
....

The delimiter is one TAB character! n:m alignments are possible (IDs separated by spaces) but only 1:1 alignments will be used in the treealigner anyway.

-C

Switch on the linked-children feature (depending on the links between children nodes of the current node pair). This flag has to be specified in both, train and align mode!

-U

Switch on the linked-subtree-nodes feature (depending on the links between all descendent nodes of the current node pair). This flag has to be specified in both, train and align mode!

-P

Switch on the linked-parent feature (depending on the links between parent nodes of the current node pair). This flag has to be specified in both, train and align mode! This flag should NOT be used together with -U or -C!

-R iter

Use <iter> number of iterations for adaptive SEARN style learning. This is only useful in connection with (any of) the link depedency features from above (-C -U -P). Instead of learning from the given true link depedency feature extracted from the training data, this option will run the training several times and adjust these features acoording to the predicted link likelihoods from the previously trained classifier. This is currently very slow because it re-runs the feature extraction procedure (which should not be necessary when re-running the classifier). This should be improved later but the effect of SEARN seems to be very little anyway ....

-L

Align terminal nodes only (leaf nodes). It is possible to use this flag together with -N which then forces the aligner to align corresponding node types only (terminals with terminals and non-terminals with non-terminals)

-N

Align non-terminal nodes only. If specified together with -L: align corresponding nodes as explained above.

-1 weight

Training weight for good (sure) alignments Default = 3

-2 weight

Training weight for fuzzy (possible) alignments Default = 1

-3 weight

Training weight for negative examples (non-aligned nodes) Default = 1

-4 weight

Training weight for weak alignments (new category in our Europarl data) Default = 1

-k

Keep the feature file extracted for training which usually is removed to save storage space. The features are stored in __train.$$ (where $$ corresponds to the process ID)

Alignment options

-x threshold

Score threshold used for tree alignment. Node pairs obtaining scores below this threshold will not be considered in the alignment process.

-b strategy

Type of alignment strategy to be used. Default is 'inference' which refers to a two-step procedure with local classification in the first step and alignment inference in the second (see LinkSearch with argument -l). An alternative strategy is called 'bottom-up' in which the alignment is done in a greedy bottom-up fashion starting with leaf node pairs and going up to the root nodes. Nodes are linked immediately when the classification score (conditional link likelihood) exceeds the threshold (usually 0.5). Aligned nodes are removed from the search space. Therefore, only 1:1 links are returned. In a final step link likelihoods are used to align previously unlinked nodes with the selected alignment inference strategy in the same way as in the two-step procedure.

-l LinkSearch

Link strategy used to extract the node aligments after classification. Default strategy is 'greedy'. Other possible strategies are 'wellformed' (greedy + wellformedness criteria) and threshold (allow all links above the threshold score). You can also add the option 'final' (by adding the string '_final') to the selected strategy. In that case the aligner will first do the basic link search and then add links between nodes that obey the well-formedness criteria if either source or target language node is not linked yet. In other words, this final step makes 1:many links in the data that do not violate wellformedness. Yet another option is 'and' (which can be added as the string '_and' to the selected strategy, also in combination with '_final'). Using this option unlinked nodes (source and target) will be aligned in a last step in a greedy way even if they violate well-formedness. For example: 'wellformed_final_and' will force the aligner to, first, look for 1:1 links that are well-formed (multiple links are not allowed), then add well-formed links between nodes where one of them is already linked to another one, and, finally, adds links between still unlinked nodes.

-u

Switch to add-links mode (union). Existing links between nodes will be kept in the output file and new ones will be added. (In the default mode, existing links will be considered for evaluation only). This option is espcially useful if one wants to use a pipeline of alignments, for example, terminal node alignment first and non-terminal nodes in the next step.

-K

Similar to -u: switches to 'add-link' mode but now forces the aligner to use existing links to compete with the new ones. This means that the scores of existing links will be used in the link search algorithm applied for aligning tree nodes. This may also cause some existing links to disappear, for example, because they are not conform to the wellformedness criteria anymore.

Runtime and other options

-v

Verbose output

-O format

Output format (one of sta (=default) or dublin (= Dublin subtree aligner format)

SEE ALSO

Lingua::Align::Trees, Lingua::Align::Features, Lingua::Align::Corpus

AUTHOR

Joerg Tiedemann

COPYRIGHT AND LICENSE

Copyright (C) 2009 by Joerg Tiedemann

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.8 or, at your option, any later version of Perl 5 you may have available.

Copyright for MegaM by Hal Daume III see http://www.cs.utah.edu/~hal/megam/ for more information Paper: Notes on CG and LM-BFGS Optimization of Logistic Regression, 2004 http://www.cs.utah.edu/~hal/docs/daume04cg-bfgs.pdf