NAME
MySQL::Slurp - Use PIPEs to import a file into MySQL table.
CAVEAT
MySQL::Slurp only works on systems that support FIFOs and
does not support Windows ... yet.VERSION
0.20
SYNOPSIS
use MySQL::Slurp;
# NEW OBJECTS
my $slurper= MySQL::Slurp->new(
database => 'test' ,
table => 'table_1' ,
buffer => 10000 ,
);
$slurper->open;
# OR,
my $slurper->new( database => 'test', table => 'table_1' )->open;
# IMPORT METHODS
$slurper->slurp(); # slurp from <STDIN>
# RECOMMENDED METHOD TO WRITE TO A TABLE
# implements buffer and locks
$slurper->write( @records );
# WRITE DIRECTLY TO TABLE WITHOUT BUFFER AND LOCKS
print {$slurper->writer} "Fred\tFlinstone\n";
print { $slurper->{writer} } "Fred\tFlinstone\n";
$slurper->close;
# In cordinated environents
$slurper->write( @a ); # In thread 1.
$slurper->write( @b ); # In thread 2.DESCRIPTION
The command-line tool, mysqlimport, is the fastest way to import data into MySQL especially using --use-threads. Unfortunately, mysqlimport does not read from <STDIN>. IN fact, mysqlimport only reads from files that have the same name as the target table. This is often inconvenient.
MySQL::Slurp has the speed of mysqlimport but permits loading from <STDIN> or provides a GlobRef file handle for writing directly to a MySQL table. This is very handy for large ETL jobs.
This module simply wraps and mysqlimport and creates the necessary FIFO. As such, catching (data) errors is relegated to mysqlimport. Unike using DBI for trapping errors, catching errors with mysqlimport can be troublesome with inconsitent data. It is recommended that you check you data before writing to the MySQL::Slurp handle or use a suitable DBI method. inconsistent.
METHODS
new
Creates a new MySQL::Slurp object.
- database
-
name of database (required)
- table
-
Name of table to import (required)
- tmp
-
Name of temporary directory (optional)
- buffer
-
Maximum number of records that are stored in the buffer before locking the fifo and flushing to the table. By default, there is no buffering, buffer = 1.
- args
-
Options to pass to mysqlimport. args is an array ref and should appear exactly as it does in the command line invocation of mysqlimport
- method
-
Method to use...presently not implemented, uses mysqlimport.
open
Opens a connection to the MySQL table through a temporary FIFO. Returns a GlobRef that can be directly written to.
write
Writes arguments to the MySQL database. Buffering is on by default, see the buffer attribute.
close
Closes and removes the pipe and temporary table.
slurp
Write <STDIN> to the database table.
THREAD SAFE
MySQL::Slurp is believed to be thread safe if using the 'write' method. Directly accessing the IO::File pipe is not considered Thread safe.
EXPORT
None.
TODO
- use MooseX::Attribute::Defaults::GNU for object attributes
- remove reliance on installation of mysqlimport, by XS wrapping the C libraries.
- Better error catching than mysqlimport
- create a version to run on windows with named pipes(?)
- alias attribute tmp as temp.
- allow options for slurp to change the MySQL::Slurp object's attributes
SEE ALSO
MySQL::Slurp relies on the Moose metaobject package.
mysqlimport at http://mysql.com, currently http://dev.mysql.com/doc/refman/5.1/en/mysqlimport.html
AUTHOR
Christopher Brown, <ctbrown@cpan.org<gt>
COPYRIGHT AND LICENSE
Copyright (C) 2008 by Open Data
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.8 or, at your option, any later version of Perl 5 you may have available.