/ 
Hadoop-IO-0.003
River stage one • 1 direct dependent • 1 total dependent
/ Hadoop::IO::SequenceFile::HDFSWriter

NAME

Hadoop::IO::SequenceFile::HDFSWriter

VERSION

version 0.003

DESCRIPTION

This is a simple wrapper around WebHDFS providing a more file-like interface.

NAME

Hadoop::IO::SequenceFile::HDFSWriter - buffering HDFS file writer.

METHODS

new
$class->new($webhdfs, $path, %options)

Construct new HDFSFile writer. Arguments:

$webhdfs

An instance of Net::Hadoop::WebHDFS.

$path

HDFS path to the destination file. If it doesn't exist it will be created. Can not point at a directory.

%opts

Other options:

truncate

If true, file will be truncated to zero length. If not set or false, do nothing here, and write calls just append to the existing file.

chunk_size

Buffer this many bytes before actually writing to HDFS. Default is 8 Mb.

on_write

This coderef will be called without arguments before each request to WebHDFS service.

write
$writer->write($data)

Append $data to the file. Buffers up to chunk_size (see "new") bytes before flushing out to HDFS.

Call "flush" after last write to ensure all data reaches the destination file.

flush
$writer->flush()

Flush buffered data out to HDFS. Normally you only need to call this once at the end of all writes.

rename
$writer->rename($new_name)

Rename destination file to a new name. If $new_name contains '/' character, path is treated as absolute and used as is, otherwise only last component of the path is updated, ie file is renamed but stays in the same directory.

AUTHORS

  • Philippe Bruhat

  • Sabbir Ahmed

  • Somesh Malviya

  • Vikentiy Fesunov

COPYRIGHT AND LICENSE

This software is copyright (c) 2023 by Booking.com.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.