NAME

Uplug::PreProcess::Tokenizer

IMPLEMENTS

tokenize

load_prefixes

DESCRIPTION

This module heavily relies on the implementation of the tokenizer and detokenizer used in the Moses toolkit for SMT. All credits go to the original authors (Josh Schroeder and Philipp Koehn).