NAME

Text::TokenStream - lexer to break text up into user-defined tokens

SYNOPSIS

my $lexer = Text::TokenStream::Lexer->new(
    whitespace => [qr/\s+/],
    rules => [
        word => qr/\w+/,
        sym => qr/[^\w\s]+/,
    ],
);

my $stream = Text::TokenStream->new(
    lexer => $lexer,
    input => "foo *",
);

my $tok1 = $stream->next; # --> "word" token containing "foo"
my $tok2 = $stream->next; # --> "sym" token containing "*"

DESCRIPTION

This class is part of a collection of classes that act together to lex (aka scan) an input text into a stream of tokens.

This token stream class provides the stream interface, along with a notion of the "current position" in the input text, and position-aware error reporting. It composes Text::TokenStream::Role::Stream; that role lists the methods this class provides (so that you can easily write a parser class that has a token stream which in turn handles the tokenizer methods).

The basic lexer machinery is found in Text::TokenStream::Lexer; it is separated out from the token stream so that it can be reused across many inputs.

Tokens are instances of a class, Text::TokenStream::Token by default.

CONSTRUCTOR

This class uses Moo, and inherits the standard new constructor.

ATTRIBUTES

lexer

An instance of Text::TokenStream::Lexer; required; read-only. Will be used to find tokens in the input.

input

Str; required; read-only. The text that will be lexed into a stream of tokens.

input_name

A Maybe[Path]; read-only. Can be coerced from a string. If a defined value is present, it should contain the name of the file that the input was read from, and that name will be used in any error messages.

token_class

The name of a class that inherits from Text::TokenStream::Token; defaults to Text::TokenStream::Token itself; read-only. Tokens found in the input will be constructed as instances of this class.

OTHER METHODS

collect_all

Takes no arguments. Returns a list of all remaining tokens found in the input.

In the current implementation, this method is provided by Text::TokenStream::Role::Stream.

collect_upto

Takes a single argument indicating a token to match, as with Text::TokenStream::Token#matches. Scans through the input until it finds a token that matches the argument, and returns a list of all tokens before the matching one. If no remaining token in the input matches the argument, behaves as "collect_all".

In the current implementation, this method is provided by Text::TokenStream::Role::Stream.

create_token

Takes a listified hash of token attributes, and creates a token instance. The token object is created by calling:

$self->token_class->new(%data);

If you have particularly complex needs, you may wish to override this method in a subclass.

current_position

Takes no arguments. Returns the 0-based position of the first input character that hasn't yet been returned by "next".

err

Takes multiple arguments, that are concatenated into an error message. (If no arguments are supplied, acts as if you'd supplied the string "Something's wrong".) Throws an exception, reporting the locus of the error as the current input position (using 1-based line and column numbers).

fill

Takes a single positive-integer argument. Attempts to fill an internal buffer of already-lexed tokens so that it contains that many tokens. Returns a boolean that is true iff there were enough tokens to do that.

looking_at

Takes zero or more arguments, each of which indicates a token to match, as with Text::TokenStream::Token#matches. Returns a boolean that is true iff there's at least one more token in the input, and it matches the argument.

next

Takes no arguments. Returns the next token found in the input, and advances the current position past it; if no tokens remain, returns undef. The token instance is created by "create_token".

next_of

Takes a single argument indicating a token to match, as with Text::TokenStream::Token#matches, and an optional string argument describing the current position (for example, "in expression", or "after keyword"). If there are no more tokens in the input, reports an error at the current position, using "err". Otherwise, if the next token doesn't match the argument, reports an error at the position of that token, using "token_err". Otherwise, the next token matches what is being looked for, so that token is returned.

peek

Takes no arguments. Returns the next token that would be returned by "next", but doesn't advance the current input position, and a subsequent "next" call will return the same token.

An internal buffer is used to ensure that every token is lexed only once.

skip_optional

Takes a single argument indicating a token to match, as with Text::TokenStream::Token#matches. If there are no more tokens in the input, or the next token doesn't match the argument, returns false; otherwise, advances past the next token, and returns true.

token_err

Takes a token as an argument, followed by multiple arguments that are concatenated into an error message. (If no non-token arguments are supplied, acts as if you'd supplied the string "Something's wrong".) Throws an exception, reporting the locus of the error as the position of the token (using 1-based line and column numbers).

AUTHOR

Aaron Crane, <arc@cpan.org>

COPYRIGHT

Copyright 2021 Aaron Crane.

LICENCE

This library is free software and may be distributed under the same terms as perl itself. See http://dev.perl.org/licenses/.