/ 
Text-Transliterator-1.04
River stage two • 2 direct dependents • 12 total dependents
/ Text::Transliterator::Unaccent

NAME

Text::Transliterator::Unaccent - Compile a transliterator from Unicode tables, to remove accents from text

SYNOPSIS

my $unaccenter = Text::Transliterator::Unaccent->new(script => 'Latin',
                                                     wide   => 0,
                                                     upper  => 0);
$unaccenter->($string);

my $map   = Text::Transliterator::Unaccent->char_map(script => 'Latin');

my $descr = Text::Transliterator::Unaccent->char_map_descr();

DESCRIPTION

This package compiles a transliteration function that will replace accented characters by unaccented characters. That function is fast, because it uses the builtin tr/.../.../ Perl operator; it is compact, because it only treats the Unicode subset that you need for your language; and it is complete, because it relies on the builtin Unicode character tables shipped with your Perl installation.

The algorithm for detecting accented characters is derived from the notion of compositions in Unicode; that notion is explained in perluniintro. Characters considered "accented" are the precomposed characters for which the Unicode canonical decomposition contains more than one codepoint; for such decompositions, the first codepoint is the unaccented character that will be mapped to the accented one. This definition seems to work well for the Latin script; I presume that it also makes sense for other scripts as well, but I'm not able to test.

METHODS

new

my $unaccenter = Text::Transliterator::Unaccent->new(@range_description);
# or
my $unaccenter = Text::Transliterator::Unaccent->new(); # script => 'Latin'

Compiles a new 'unaccenter' function. The @range_description argument specifies which ranges of characters will be handled, and is comprised of pairs of shape :

script => $unicode_script

$unicode_script is the name of a Unicode script, such as 'Latin', 'Greek' or 'Cyrillic'. For a complete list of unicode scripts, see

perl -MUnicode::UCD=charscripts -e "print join ', ', keys %{charscripts()}"
block => $unicode_block

$unicode_block is the name of a Unicode block. For a complete list of Unicode blocks, see

perl -MUnicode::UCD=charblocks -e "print join ', ', keys %{charblocks()}"
range => \@codepoint_ranges

@codepoint_ranges is a list of arrayrefs that contain start-of-range, end-of-range code point pairs.

wide => $bool

Decides if wide characters (i.e. characters with code points above 255) are kept or not within the map. The default is true.

upper => $bool

Decides if uppercase characters are kept or not within the map. The default is true.

lower => $bool

Decides if lowercase characters are kept or not within the map. The default is true.

The @range_description may contain a list of several scripts, blocks and/or ranges; all will get concatenated into a single correspondance map. If the list is empty, the default range is script => 'Latin'.

The return value from that new method is actually a reference to a function, not an object. That function is called as

$unaccenter->(@strings);

and modifies every member of @strings in place, like the tr/.../.../ operator. The return value is the number of transliterated characters in the last member of @strings.

char_map

my $map = Text::Transliterator::Unaccent->char_map(@range_description);

Utility class method that returns a hashref of the accented characters in @range_description, mapped to their unaccented corresponding characters, according to the algorithm described in the introduction. The @range_description format is exactly like for the new() method.

char_map_descr

my $descr = Text::Transliterator::Unaccent->char_map_descr(@range_descr);

Utility class method that returns a textual description of the map generated by @range_descr.

SEE ALSO

Text::Unaccent is another unaccenter module, with a C and a Pure Perl version. It is based on iconv instead of Perl's internal Unicode tables, and therefore may produce slighthly different results. According to some experimental benchmarks, the C version of Text::Unaccent is faster than Text::Transliterator::Unaccent on short strings and on small number of calls, and slower on long strings or high number of calls (but this may be a side-effect of the fact that it returns a copy of the string instead of replacing characters in-place); however I am not able to give a predictable rule about which module is faster in which circumstances.

Text::StripAccents is a Pure Perl module. In only handles Latin1, and is several orders of magnitude slower because it does an internal split and join of the whole string.

Search::Tokenizer uses the present module for building an unaccent tokenizer.

AUTHOR

Laurent Dami, <dami@cpan.org>

BUGS

Please report any bugs or feature requests to bug-text-transliterator at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Text-Transliterator. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

perldoc Text::Transliterator::Unaccent

You can also look for information at:

LICENSE AND COPYRIGHT

Copyright 2010, 2017, 2023 Laurent Dami.

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.