NAME
Lingua::Identify - Language identification
SYNOPSIS
use Lingua::Identify qw(:language_identification);
$a = langof($textstring); # gives the most probable language
@a = langof($textstring); # gives pairs of languages / probabilities
# sorted from most to least probable
%a = langof($textstring); # gives a hash of language / probability
# or the hard (expert) way (see section OPTIONS, under HOW TO PERFORM
# IDENTIFICATION)
$a = langof({ method => [qw/smallwords prefix2 suffix2/] },$textstring);DESCRIPTION
Lingua::Identify identifies the language a given string or file is written in.
See section WHY LINGUA::IDENTIFY for a list of Lingua::Identify's strong points.
See section KNOWN LANGUAGES for a list of available languages and HOW TO PERFORM IDENTIFICATION to know how to really use this module.
If you're in a hurry, jump to section EXAMPLES, way down below.
Also, don't forget to read the following section, IMPORTANT WARNING.
IMPORTANT WARNING
Take a word that exists in two different languages, take a good look at it and answer this question: "What language does this word belong to?".
You can't give an answer like "Language X", right? You can only say it looks like any of a set of languages.
Similarly, it isn't always easy to identify the language of a text if the only two active languages are very similar.
Now that we've taken out of the way the warning that language identification is not 100% accurate, please keep reading the documentation.
WHY LINGUA::IDENTIFY
You might be wondering why you should use Lingua::Identify instead of any other tool for language identification.
Here's a list of Lingua::Identify's strong points:
it's free and it's open-source;
it's portable (it's Perl, which means it will work in lots of different platforms);
26 languages and growing;
4 different methods of language identification and growing (see METHODS OF LANGUAGE IDENTIFICATION for more details on this one);
it's a module, which means you can easily write your own application (be it CGI, TK, whatever) around it;
it comes with langident, which means you don't actually need to write your own application;
it's flexible (you can actually chose the methods to use and their relevance, and pretty soon you'll be able to chose some other things)
it's easy to deal with languages (you can activate and deactivate the ones you chose whenever you want to, which can improve your times and accuracy);
it's being maintained.
HOW TO PERFORM IDENTIFICATION
langof
To identify the language a given text is written in, use the langof function. To get a single value, do:
$language = langof($text);To get the most probable language and also the percentage of its probability, do:
($language, $probability) = langof($text);If you want a hash where each active language is mapped into its percentage, use this:
%languages = langof($text);OPTIONS
langof can also be given some configuration parameters, in this way:
$language = langof(\%config, $text);These parameters are detailed here:
method
You can chose which method or methods to use, and also the relevance of each of them.
To chose a single method to use:
langof( {method => 'smallwords' }, $text);To chose several methods:
langof( {method => [qw/prefixes2 suffixes2/]}, $text);To chose several methods and give them different weight:
langof( {method => {smallwords => 0.5, ngrams3 => 1.5} }, $text);To see the list of available methods, see section METHODS OF LANGUAGE IDENTIFICATION.
If no method is specified, the configuration for this parameter is the following (this might change in the future):
method => { smallwords => 0.5, prefixes2 => 1, suffixes3 => 1, ngrams3 => 1.3 };
confidence
After getting the results into an array, its first element is the most probable language. That doesn't mean it is very probable or not.
You can find more about the likeliness of the results to be accurate by computing its confidence level.
use Lingua::Identify qw/:language_identification/;
my @results = langof($text);
my $confidence_level = confidence(@results);
# $confidence_level now return a value between 0 and 1; the higher that
# value, the more accurate the results seem to beget_all_methods
Returns a list comprised of all the available methods for language identification.
LANGUAGE IDENTIFICATION IN GENERAL
Language identification is based in patterns.
In order to identify the language a given text is written in, we repeat a given process for each active language (see section LANGUAGES MANIPULATION); in that process, we look for common patterns of that language. Those patterns can be prefixes, suffixes, common words, ngrams or even sequences of words.
After repeating the process for each language, the total score for each of them is then used to compute the probability (in percentage) for each language to be the one of that text.
METHODS OF LANGUAGE IDENTIFICATION
Lingua::Identify currently comprises four different ways for language identification, in a total of thirteen variations of those.
The available methods are the following: smallwords, prefixes1, prefixes2, prefixes3, prefixes4, suffixes1, suffixes2, suffixes3, suffixes4, ngrams1, ngrams2, ngrams3 and ngrams4.
Here's a more detailed explanation of each of those ways and those methods
Small Word Technique - smallwords
The "Small Word Technique" searches the text for the most common words of each active language. These words are usually articles, pronouns, etc, which happen to be (usually) the shortest words of the language; hence, the method name.
This is usually a good method for big texts, especially if you happen to have few languages active.
Prefix Analysis - prefixes1, prefixes2, prefixes3, prefixes4
This method analyses text for the common prefixes of each active language.
The methods are, respectively, for prefixes of size 1, 2, 3 and 4.
Suffix Analysis - suffixes1, suffixes2, suffixes3, suffixes4
Similar to the Prefix Analysis (see above), but instead analysing common suffixes.
The methods are, respectively, for suffixes of size 1, 2, 3 and 4.
Ngram Categorization - ngrams1, ngrams2, ngrams3, ngrams4
Ngrams are sequences of tokens. You can think of them as syllables, but they are also more than that, as they are not only comprised by characters, but also by spaces (delimiting or separating words).
Ngrams are a very good way for identifying languages, given that the most common ones of each language are not generally very common in others.
This is usually the best method for small amounts of text or too many active languages.
The methods are, respectively, for ngrams of size 1, 2, 3 and 4.
LANGUAGE MANIPULATION
When trying to perform language identification, Lingua::Identify works not with all available languages, but instead with the ones that are active.
By default, all available languages are active, but that can be changed by the user.
For your convenience, several methods regarding language manipulation were created. In order to use them, load the module with the tag :language_manipulation.
These methods work with the two letters code for languages.
- activate_language
-
Activate a language
activate_language('en'); # or activate_language($_) for get_all_languages(); - activate_all_languages
-
Activates all languages
activate_all_languages(); - deactivate_language
-
Deactivates a language
deactivate_language('en'); - deactivate_all_languages
-
Deactivates all languages
deactivate_all_languages(); - get_all_languages
-
Returns the names of all available languages
my @all_languages = get_all_languages(); - get_active_languages
-
Returns the names of all active languages
my @active_languages = get_active_languages(); - get_inactive_languages
-
Returns the names of all inactive languages
my @active_languages = get_inactive_languages(); - is_active
-
Returns the name of the language if it is active, an empty list otherwise
if (is_active('en')) { # YOUR CODE HERE } - is_valid_language
-
Returns the name of the language if it exists, an empty list otherwise
if (is_valid_language('en')) { # YOUR CODE HERE } - set_active_languages
-
Sets the active languages
set_active_languages('en', 'pt'); # or set_active_languages(get_all_languages()); - name_of
-
Given the two letter tag of a language, returns its name
my $language_name = name_of('pt');
KNOWN LANGUAGES
Currently, Lingua::Identify knows the following languages (26 total):
- AF - Afrikaans
- BG - Bulgarian
- BR - Breton
- BS - Bosnian
- CY - Welsh
- DA - Danish
- DE - German
- EN - English
- EO - Esperanto
- ES - Spanish
- FI - Finnish
- FR - French
- FY - Frisian
- GA - Irish
- HR - Croatian
- HU - Hungarian
- IS - Icelandic
- IT - Italian
- LA - Latin
- NL - Dutch
- NO - Norwegian
- PL - Polish
- PT - Portuguese
- SQ - Albanian
- SV - Swedish
- TR - Turkish
EXAMPLES
THE BASIC EXAMPLE
Check the language a given text file is written in:
use Lingua::Identify qw/langof/;
my $text = join "\n", <>;
# identify the language by letting the module decide on the best way
# to do so
my $language = langof($text);IDENTIFYING BETWEEN TWO LANGUAGES
Check the language a given text file is written in, supposing you happen to know it's either Portuguese or English:
use Lingua::Identify qw/langof activate_language/;
activate_language(qw/pt en/);
my $text = join "\n", <>;
# identify the language by letting the module decide on the best way
# to do so
my $language = langof($text);TO DO
Add more examples in the documentation;
Add examples of the values returned;
Configuration parameter to let the user chose which part(s) of the text to use;
Configuration parameter to let the user chose a maximum size of text to deal with;
WordNgrams based methods;
Easy way to learn new languages;
More languages;
File recognition and treatment;
Create sets of languages and permit their activation/deactivation;
What happens when { method => [] } ?
SEE ALSO
langident(1), Text::ExtractWords(3), Text::Ngram(3), Text::Affixes(3).
A linguist and/or a shrink.
The latest CVS version of Lingua::Identify can be attained at http://natura.di.uminho.pt/natura/viewcvs.cgi/Lingua/Identify/
AUTHOR
Jose Castro, <cog@cpan.org>
COPYRIGHT & LICENSE
Copyright 2004 Jose Castro, All Rights Reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.