NAME

Lingua::EN::Sentence - Module for splitting text into sentences.

SYNOPSIS

use Lingua::EN::Sentence qw( get_sentences add_acronyms );

add_acronyms('lt','gen');		## adding support for 'Lt. Gen.'
my $sentences=get_sentences($text);	## Get the sentences.
foreach my $sentence (@$sentences) {
	## do something with $sentence
}

DESCRIPTION

The Lingua::EN::Sentence module contains the function get_sentences, which splits text into its constituent sentences, based on a regular expression and a list of abbreviations (built in and given).

Certain well know exceptions, such as abreviations, may cause incorrect segmentations. But some of them are already integrated into this code and are being taken care of. Still, if you see that there are words causing the get_sentences() to fail, you can add those to the module, so it notices them.

ALGORITHM

Basically, I use a 'brute' regular expression to split the text into sentences. (Well, nothing is yet split - I just mark the end-of-sentence). Then I look into a set of rules which decide when an end-of-sentence is justified and when it's a mistake. In case of a mistake, the end-of-sentence mark is removed.

What are such mistakes? Cases of abbreviations, for example. I have a list of such abbreviations (Please see `Acronym/Abbreviations list' section), and more general rules (for example, the abbreviations 'i.e.' and '.e.g.' need not to be in the list as a special rule takes care of all single letter abbreviations).

FUNCTIONS

All functions used should be requested in the 'use' clause. None is exported by default.

get_sentences( $text )

The get sentences function takes a scalar containing ascii text as an argument and returns a reference to an array of sentences that the text has been split into. Returned sentences will be trimmed (beginning and end of sentence) of white-spaces. Strings with no alpha-numeric characters in them, won't be returned as sentences.

add_acronyms( @acronyms )

This function is used for adding acronyms not supported by this code. Please see `Acronym/Abbreviations list' section for the abbreviations already supported by this module.

get_acronyms( )

This function will return the defined list of acronyms.

set_acronyms( @my_acronyms )

This function replaces the predefined acroym list with the given list.

get_EOS( )

This function returns the value of the string used to mark the end of sentence. You might want to see what it is, and to make sure your text doesn't contain it. You can use set_EOS() to alter the end-of-sentence string to whatever you desire.

set_EOS( $new_EOS_string )

This function alters the end-of-sentence string used to mark the end of sentences.

Acronym/Abbreviations list

Currently supported acronym lists are:

PEOPLE ( 'jr', 'mr', 'mrs', 'ms', 'dr', 'prof', 'sr', 'sen', 'rep', 'gov' )
ARMY ( 'col','gen', 'lt', 'cmdr', 'adm' )
INSTITUTES ( 'dept', 'univ' )
COMPANIES ( 'inc', 'ltd', 'co', 'corp' )
PLACES = ( 'arc', 'al', 'ave', "blv?d", 'cl', 'ct', 'cres', 'dr', "expy?",
	"fw?y", "hwa?y", 'la', "pde?", 'pl', 'plz', 'rd', 'st', 'tce',
	'Ala' , 'Ariz', 'Ark', 'Cal', 'Calif', 'Col', 'Colo', 'Conn',
	'Del', 'Fed' , 'Fla', 'Ga', 'Ida', 'Id', 'Ill', 'Ind', 'Ia',
	'Kan', 'Kans', 'Ken', 'Ky' , 'La', 'Me', 'Md', 'Is', 'Mass', 
	'Mich', 'Minn', 'Miss', 'Mo', 'Mont', 'Neb', 'Nebr' , 'Nev',
	'Mex', 'Okla', 'Ok', 'Ore', 'Oreg', 'Penna', 'Penn', 'Pa'  , 'Dak',
	'Tenn', 'Tex', 'Ut', 'Vt', 'Va', 'Wash', 'Wis', 'Wisc', 'Wy',
	'Wyo', 'USAFA', 'Alta' , 'Man', 'Ont', 'Qué', 'Sask', 'Yuk')
MONTHS = ('jan','feb','mar','apr','may','jun','jul','aug','sep','oct','nov','dec', 'sept')
MISC ( 'vs', 'etc' )

If I come across a good general-purpose list - I'll incorporate it into this module. Feel free to suggest such lists.

SEE ALSO

Text::Sentence

AUTHOR

Shlomo Yona <Shlomo.Yona@Siftology.com>

COPYRIGHT

Copyright (c) 2001 Siftology Inc.. All rights reserved.

This library is free software. You can redistribute it and/or modify it under the same terms as Perl itself.