NAME
File::DigestStore - Digested hierarchical storage of files
VERSION
version 1.007
SYNOPSIS
my $store = File::DigestStore->new( root => '/var/lib/digeststore' );
# stores the file and returns a short-ish ID
my $id = $store->store_path('/etc/motd');
# Will output a hex string like '110fe...'
print "$id\n";
# returns a filename that has the same contents as the stored file
my $path = $store->fetch_file($id);
# Will return something like '/var/lib/digeststore/1/2/110fe...'
print "$path\n";DESCRIPTION
This module is used to take large files (or strings) and store them on disk with a name based on the hashed file contents, returning said hash. This hash is much shorter than the data and is much more easily stored in a database than the original file. Because of the hashing, only a single copy of the data will be stored on disk no matter how many times one calls store_path() or store_string().
BACKEND STORAGE
The backend data store should be considered opaque as far as your Perl program is concerned, but it actually consists of the files hashed and then stored in a multi-level directory structure for fast access. Files are never moved around the tree so if the stash is particularly large, you can place subtrees on their own filesystem if required. Directories are created on-demand and so do not need to be pre-created.
The file's name is just the hash of the file's contents, and the file's directory is the result of applying the nhash algorithm to the hash. Thus, replacing the nhash object with another class that provides a nhash() method allows you to fine-tune the directory layout.
MOOSE FIELDS
root (required)
The base directory that is used to store the files. This will be created if it does not exist, and the stashed files stored underneath it.
levels (optional, default "8,256")
The number of directory entries in each level of the tree. For example, "8,256" means that the top-level will have eight directories (called "0" through "7") and each of those directories will have 256 sub-directories. The stashed data files appear under those.
algorithm (optional, default "SHA-512")
The digest algorithm used to hash the files. This is passed to Digest->new(). The file's content is hashed and then stored using that name, so you should select an algorithm that does not generate collisions.
dir_mask (optional, default 0777)
The directory creation mask for the stash directories. This is merged with your umask so the default is usually fine.
As a special case, this will also treat strings starting with a zero as an octal number. This is helpful when you are using Catalyst::Model::Adaptor on this class and wish to change the mask in the application configuration file.
file_mask (optional, default 0666)
The file creation mask for the stashed files. This is merged with your umask setting so the default is usually fine.
This has the same special-casing for strings as dir_mask.
layers (optional, default ":raw")
The PerlIO layer to use when storing and retrieving data. Note that while this could be used to set the encoding for string I/O (e.g. with store_string()), the Digest algorithms expect your strings to be sequences of octets. This is mainly here for if you wish to use a layer that performs transparent compression.
PRIVATE MOOSE FIELDS
nhash (optional, builds an Algorithm::Nhash based on levels)
This is the internal Algorithm::Nhash object used to convert a file's hash into subdirectories.
METHODS
new
my $store = File::DigestStore->new( root => '/var/lib/digeststore' );This creates a handle to a new digested storage area. Arguments are given to it to define the layout of the storage area. See "MOOSE FIELDS" above for the available options.
store_path
my $id = $store->store_path('/etc/motd');
my ($id, $size) = $store->store_path('/etc/passwd');This copies the file's contents into the stash. In scalar context it returns the file's ID. In list context it returns an (ID, file size) tuple. (The latter saves you having to stat() your file.)
store_string
my $id = $store->store_string('Hello, world');This copies the string's contents into the stash. In scalar context it returns the file's ID. In list context it returns an (ID, string length) tuple.
fetch_path
my $path = $store->fetch_path($id);Given an ID, will return the path to the stashed copy of the file, or undef if no file with that ID has ever been stored. Note that the path refers to the master copy of the file within the stash and you must not modify it.
fetch_string
my $string = $store->fetch_string($id);Given an ID, will return the string which was previously stashed to that ID or undef if no string with that ID has ever been stored.
exists
if($store->exists($id)) {
# ...
}Returns true if anything is stashed with the given ID, otherwise false.
delete (new in 1.007)
$store->delete($id);Removes the data stashed with the given ID and returns true if it existed or false if it did not.
DEPRECATED METHODS
fetch_file
fetch_path was originally called this, but the name is inappropriate since it implies that it fetches the file rather than just the file's name.
store_file
store_path was originally called this, but the name is inappropriate since it implies that the parameter was the file rather than the file's name.
PRIVATE METHODS
These methods are private and not for end-users; the API may change without notice. If you use them and stuff breaks, it's your own fault.
_build_nhash
_digest2path
_readfile
_writefile
BUGS
This does not provide any means to check for hash collisions.
You cannot provide a hashing algorithm that is not a Digest::* derivative.
SEE ALSO
File::HStore implements a similar idea.
AUTHOR
Peter Corlett <abuse@cabal.org.uk>
COPYRIGHT AND LICENSE
This software is copyright (c) 2011 by Peter Corlett.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.