NAME
File::Stream - Regular expression delimited records from streams
SYNOPSIS
use File::Stream;
my $stream = File::Stream->new($filehandle);
$/ = qr/\s*,\s*/;
print "$_\n" while <$stream>;
# or:
($handler, $stream) = File::Stream->new(
$filehandle,
read_length => 1024,
separator => qr{to_be_used_instead_of_$/},
);
while(<$stream>) {...}
my $line = $stream->readline(); # similar
# extended usage:
use URI;
my $uri = URI->new('http://steffen-mueller.net');
my ($pre_match, $match) =
$handler->find('literal_string', qr/regex/, $uri);
# $match contains whichever argument to find() was found first.
# $pre_match contains all that was before the first token that was found.
# both the contents of $match and $pre_match have been removed from the
# data stream (buffer).
# Since version 2.10 of the module, you can use seek() and tell() on
# File::Stream objects:
my $position = tell($stream);
# ...
seek($stream, 0, $position); # rewindDESCRIPTION
Perl filehandles are streams, but sometimes they just aren't powerful enough. This module offers to have streams from filehandles searched with regexes and allows the global input record separator variable to contain regexes.
Thus, readline() and the <> operator can now return records delimited by regular expression matches.
There are some very important gripes with applying regular expressions to (possibly infinite) streams. Please read the CAVEATS section of this documentation carfully.
EXPORT
None.
new
The new() constructor takes a filehandle (or a glob reference) as first argument. The following arguments are interpreted as key/value pairs with the following parameters being defined:
- separator
-
This may be set to either a string, a compiled regular expression (qr//), or an object which is overloaded in stringification context to return a string.
If separator is set, its value will be used in calls to readline() (including the <> diamond operator which calls readline internally) instead of $/, the global input record separator.
String will be interpreted as literal strings, not regular expressions.
- read_length
-
Defaults to 1024 bytes. Sets the number of bytes to read into the internal buffer at once. Large values may speed up searching considerably, but increase memory usage.
- die_on_anchors
-
Defaults to true. If set, any call to
find()will die on regular expressions which contain anchors (^ and $). Usually, those do not make much sense in regular expressions applied to streams. If, however, you're using the//moption, these anchors match the start and end of each line respectively. In that case, you can suppress the fatal error by settingdie_on_anchors => 0.
The new() method returns a fresh File::Stream object that has been tied to be a filehandle and a filehandle. All the usual file operations should work on the filehandle and the File::Stream methods should work on the object.
readline
The readline method on a File::Stream object works just like the builtin except that it uses the objects record separator instead of $/ if it has been set via new() and honours regular expressions.
This is also internally used when readline() is called on the tied filehandle.
find
Finds the first occurrance one of its arguments in the stream. For example,
$stream_handler->find('a', 'b');finds the first character 'a' or 'b' in the stream whichever comes first. Returns two strings: The data read from the stream before the match and the match itself. The arguments to find() may be regular expressions, but please see the CAVEATS section of this documentation about that. If any of the arguments is an object, it will be evaluated in stringification context and the result of that will be matched literally, ie. not as a regular expression.
As with readline(), this is a method on the stream handler object.
fill_buffer
It is unlikely that you will need to call this method directly. Reads more data from the internal filehandle into the buffer. First argument may be the number of bytes to read, otherwise the 'read_length' attribute is used.
Again, call this on the handler object, not the file handle.
CAVEATS
There are several important issues to keep in mind when using this module. First, setting $/ to a regular expression will most certainly break badly when $/ is used on filehandles that are not File::Stream object. Please consider setting the "separator" attribute of the File::Stream object instead for a more robust solution.
In a later version of this module, either $/ may be tied to do magic that only applies the regex to File::Stream objects, or CORE::readline() might be overridden to use the builtin readline() whenever the handle at hand is not a File::Stream. It is currently unclear which would be less bad.
Most importantly, however, there are some inherent problems with regular expressions applied to (possibly infinite) streams. The implementation of Perl's regular expression engine requires that the string you apply a regular expression to be in memory completely. That means applying a regular expression that matches infinitely long strings (like .*) to a stream will lead to the module reading in the whole file, or worse yet, an infinite string. Anchors like ^ or $ don't make sense with streams either, but since version 1.11 of File::Stream, the module throws a fatal error when finding an anchor in a regular expression. Using infinitely long matches on infinite streams may still result in your machine running out of memory. So don't do that!
Since version 1.10, the buffer is extended whenever the regex reaches its end. That means it has to tokenize the regex and insert weird constructs in many places. This is a rather slow and fragile process.
AUTHOR
Steffen Mueller, <stream-module at steffen-mueller dot net>
Many thanks to Simon Cozens for his advice and the original idea, Autrijus Tang for much help with the fiendish regexes I couldn't handle, and Ben Tilly for suggesting the use of the ${} regex construct.
Furthermore, since version 2.10, File::Stream includes a patch implementing seek() and tell() for File::Stream objects. The idea and code were kindly supplied by Phil Whineray.
COPYRIGHT AND LICENSE
Copyright (C) 2003-2011 by Steffen Mueller
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.6 or, at your option, any later version of Perl 5 you may have available.