/ 
Data-AnyXfer-0.1
River stage zero No dependents
/ Data::AnyXfer::Elastic::Test::Import

NAME

Data::AnyXfer::Elastic::Test::Import - es package for importing test data

SYNOPSIS

my $import = Data::AnyXfer::Elastic::Test::Import->new;


# play a datafile

$import->import_test_data( file => 't/data/import.interiors-20140910160216.datafile' );


# the same for a directory

$import->import_test_data(
    dir => 't/data',
    index_info => 'Interiors::IndexInfo' );


# or, if you had mixed data to import (going to multiple indexes)
# you could use import_test_data_all

$import->import_test_data_all(
    {
        dir => 't/data/property',
        index_info => 'Property::ES::IndexInfo'
    },
    {
        dir => 't/data/landreg',
        index_info => 'HeatmapData::Landreg::IndexInfo'
    },
    {
        dir => 't/data/interiors',
        index_info => 'Interiors::IndexInfo'
    },
);


# You may need test the contents of an Elasticsearch index...

my $index = Interiors::IndexInfo->new->get_index;
my $expected_documents = [{1},{2},{3},{4}];

$import->index_contains($index, $expected_data,
    'Index contains expected documents');

$import->index_contains_exact($index, $expected_data,
    'Index contains *only* the expected documents');


# Or, you may need to test the contents of a datafile

my $datafile = $import->datafile( file => 't/data/import.interiors-20140910160216.datafile' );
my $expected_documents = [{1},{2},{3},{4}];

$import->datafile_contains($datafile, $expected_documents,
    'Datafile contains expected documents');

$import->datafile_contains_exact ($datafile, $expected_documents,
    'Datafile contains *only* the expected documents');

DESCRIPTION

This package can be used to create and play datafiles containing test data for use with tests. The operations provided by this test will usually be used / performed in 00-setup.t.

All routines in this module may be used as either instance or package methods.

SEE ALSO

Data::AnyXfer::Elastic::Import::DataFile, Data::AnyXfer::Elastic::Importer

METHODS

datafile

my $datafile = Data::AnyXfer::Elastic::Test::Import
    ->datafile(
        dir => 't/data/landreg',
        index_info => 'HeatmapData::Landreg::IndexInfo'
    );

Creates or reads an existing Data::AnyXfer::Elastic::Import::DataFile, to push test data to or read from.

Takes the same arguments as the Data::AnyXfer::Elastic::Import::DataFile constructor, but modfifies the datafile specifically to be run in test environments.

Returns the new DataFile instance.

import_test_data_all

$import->import_test_data_all(
    {
        dir => 't/data/property',
        index_info => 'Property::ES::IndexInfo'
    },
    {
        dir => 't/data/landreg',
        index_info => 'HeatmapData::Landreg::IndexInfo'
    },
    {
        dir => 't/data/interiors',
        index_info => 'Interiors::IndexInfo'
    },
);

Convenience batch import method which takes multiple import configuration HASH refs and performs them in sequence.

If you had mixed data to import (going to multiple indexes) you could use import_test_data_all.

See "import_test_data" for supported hash arguments, and implementation details.

Test output will be the output provided by "import_test_data".

Does not return anything.

import_test_data

$import->import_test_data( file => 't/data/import.interiors-20140910160216.datafile' );

#
# play a datafile, overriding the original index information
# test naming schemes are applied as previously
# but the mappings and type information etc. will be used
# from the current code-base and environment

$import->import_test_data(
    file => 't/data/import.interiors-20140910160216.datafile',
    index_info => 'Interiors::IndexInfo' );


# the same for a directory

$import->import_test_data(
    dir => 't/data',
    index_info => 'Interiors::IndexInfo' );

"Plays" a datafile using Data::AnyXfer::Elastic::Importer generating TAP output for each index created and finalised.

Takes a key-value list of arguments.

Requires at least one of arguments file or dir, which may be paths, or Path::Class::File and Path::Class::Dir objects respectively.

  • If a file attribute is supplied, this file will be imported.

  • If a dir attribute is supplied, all files matching *.datafile will be imported in the target directory.

Any additional arguments will be passed directly to "execute" in Data::AnyXfer::Elastic::Importer.

Produces test output directly.

build_test_index

my $index_info = Data::AnyXfer::Elastic::Test::Import->build_test_index(
    name      => 'some_stuff',
    documents => [
        { title => "This is document 1" },
        { title => "This is document 2" }
    ]
);

Builds a one-off test index containing the specified data, for quick mocking of data.

Takes 2 arguments which are both required. name, which can be any identifier (the alias etc. are generated off of this), and documents, which must be an array of document HASH refs.

datafile_contains

The same as "datafile_contains_exact" in ., except the datafile is allowed to contain additional records not in the expected set.

This is simply a wrapper around "datafile_contains_exact" in ., using "supersetof" in Test::Deep.

index_contains

The same as "index_contains_exact" in ., except the index is allowed to contain additional records not in the expected set.

This is simply a wrapper around "index_contains_exact" in ., using "supersetof" in Test::Deep.

datafile_contains_exact

my $datafile = $import->datafile( file => 't/data/mytestdata.datafile' );
my $expected = [{ document => 123, tags => qw( test entry sample ) }];

$import->datafile_contains_exact($datafile, $expected,
    'Test datafile contains the required documents')

Compares the contents of a datafile to the supplied document ARRAY, supplied as a reference.

Takes 3 arguments. The first argument must be a Data::AnyXfer::Elastic::Import::DataFile instance, the second argument must be a reference to an ARRAY of expected document / data structures, and the third argument is the test name.

Produces test output directly.

Note: Do not run this method on full datasets as all of the data must be loaded into memory (x2)

index_contains_exact

# get an index object / connection
# (here we use the test datafile to create an ad-hoc connection)
my $index = $datafile->export_index_info->get_index;

# build a list of expected documents
my $expected = [{ document => 123, tags => qw( test entry sample ) }];

# test that they match
$import->index_contains_exact($index, $expected,
    'Test index contains the required documents');

Compares the contents of an elasticsearch index to the supplied document ARRAY, supplied as a reference.

Takes 3 arguments. The first argument must be a Data::AnyXfer::Elastic::Index instance, the second argument must be a reference to an ARRAY of expected document / data structures, and the third argument is the test name.

Produces test output directly. The target index for comparison may contain no more than 10,000 documents.

Note: Do not run this method on full datasets as all of the data must be loaded into memory (x2)

import_latest_test_data

$import->import_latest_test_data( 'Property::Search::IndexInfo' );
$import->import_latest_test_data( Property::Search::IndexInfo->new );

Creates a test index from the latest live datafile. Play a datafile, exactly as originally generated, but will apply the test alias and index naming schemes.

e.g. [index] interiors_20141201 -> HOSTN_USER_PKG_interiors_20141201
     [alias] interiors          -> HOSTN_USER_PKG_interiors

find_latest_live_data

my ($datafile, $index_info) =
    $import->find_latest_live_data('POI');

Finds the latest live datafile for a given IndexInfo class. It does this by knowing where datafile exports are stored on the SAN, and using the information on index naming in the IndexInfo class supplied.

The datafile is returned as a path class object.

COPYRIGHT

This software is copyright (c) 2019, Anthony Lucas.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.