/ 
Algorithm-NGram-0.1
River stage zero No dependents
/ Algorithm::NGram

NAME

Algorithm::NGram

SYNPOSIS

use Algorithm::NGram;
my $ng = Algorithm::NGram->new(token_size => 3); # use trigrams
$ng->add_text($text1); # analyze $text1
$ng->add_text($text2); # analyze $text2

# feed in arbitrary sequence of tokens
$ng->add_start_token;
$ng->add_tokens(qw/token1 token2 token3/);
$ng->add_end_token;

my $output = $ng->generate_text;

DESCRIPTION

This is a module for analyzing token sequences with n-grams. You can use it to parse a block of text, or feed in your own tokens. It can generate new sequences of tokens from what has been fed in.

VERSION

Version 0.01

EXPORT

None.

METHODS

new

Create a new n-gram analyzer instance.

Options:

token_count

This is the "window size" of how many tokens the analyzer will keep track of. A token_count of two will make a bigram, a token count of three will make a trigram, etc...

token_count

Returns token window size (e.g. the "n" in n-gram)

add_text

Splits a block of text up by whitespace and processes each word as a token. Automatically calls add_start_token() at the beginning of the text and add_end_token() at the end.

add_tokens

Adds an arbitrary list of tokens.

add_start_token

Adds the "start token." This is useful because you often will want to mark the beginnings and ends of a token sequence so that when generating your output the generator will know what tokens start a sequence and when to end.

add_end_token

Adds the "end token." See add_start_token().

analyze

Generates an n-gram frequency table. Returns a hashref of N->tokens->count, where N is the number of tokens (will be from 2 to token_size). You will not normally need to call this unless you want to get the n-gram frequency table.

generate_text

After feeding in text tokens, this will return a new block of text based on whatever text was added.

generate

Generates a new sequence of tokens based on whatever tokens have previously been fed in.

next_tok

Given a list of tokens, will pick a possible token to come next.

token_lookup

Returns a hashref of the counts of tokens that follow a sequence of tokens.

token_key

Serializes a sequence of tokens for use as a key into the n-gram table. You will not normally need to call this.

SEE ALSO

Text::Ngram, Text::Ngrams

AUTHOR

Mischa Spiegelmock, <mspiegelmock@gmail.com>

COPYRIGHT AND LICENSE

Copyright 2007 by Mischa Spiegelmock

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.