NAME

Search::Brick::RAM::Query - Perl interface for the RAMBrick search thingie https://github.com/jackdoe/brick/tree/master/bricks/RAMBrick

SYNOPSIS

use Search::Brick::RAM::Query qw(query true)
my @results = query(host    => ['127.0.0.1:9000'],
                    settings => { log_slower_than => 100, explain => true }, # log queries slower than 100ms
                    request  => { term => { "title" => "book","author" => "john" } },
                    brick    => 'RAM',
                    action   => 'search',
                    timeout  => 0.5 # in seconds - this is total timeout
                                    # connect time + write time + read time
              );

#shortcuts are available using the search object:
my $s = Search::Brick::RAM::Query->new(host => '127.0.0.1:9000', index => '__test__');
my @results = $s->search({ term => { author => 'jack' } }, { size => 20, explain => true, log_slower_than => 5 });
$s->delete(); # deletes the __test__ index

DESCRIPTION

minimalistic interface to RAMBrick (a minimalistic Lucene wrapper)

FUNCTION: Search::Brick::RAM::Query::query( $args :Hash ) :Array

Search::Brick::RAM::Query::query has the following parameters:

host    => ['127.0.0.1:9000'],
request => { term => { "title" => "book","author" => "john" } },
brick   => 'RAM',
action  => 'search', # (search|store|alias|load)
index   => '...',    # name of your index
timeout => 0.5,      # in seconds
settings => {}       # used by different actions to get context on the request

timeout: timeout is the whole round trip timeout: (connect time + write time + read time).
brick: brick is the brick's name (in this case 'RAM')
action: action is the requested action (search|store|stat)
request: request must be reference to array of hashrefs
index: index is the action argument, provides context to the request (usually index name)
settings: settings RAMBrick requires settings to be sent (by default they are empty) things like size or items_per_group like: { size => 5, explain => true }
host: host string or arrayref of strings - the same request will be sent to all hosts in the list (the whole thing is async, so will be as slow as the slowest host) and un-ordered array of results is returned.

If the result is not ref() or there is any kind of error(including timeout), the request will die.

Query object

you can also use the module by creating a Query object, such as:

my $s = Search::Brick::RAM::Query->new(host => '127.0.0.1:9000', index => '__test__');
my @results = $s->search({ term => { author => 'jack' } }, { size => 20, explain => true, log_slower_than => 5 });

Simple wrapper making query() calls, but without having to pass index, brick, host every time.

search($query,$settings,$timeout || DEFAULT_TIMEOUT)
index($data,$settings,$timeout || DEFAULT_TIMEOUT)
alias($settings,$timeout || DEFAULT_TIMEOUT)
stat($timeout || DEFAULT_TIMEOUT)
delete($timeout || DEFAULT_TIMEOUT)

all of the above are just making query() calls with the selected parameters, but also passing brick => 'RAM'>, index => __test(in this example) and host => [ '127.0.0.1:9000' ]

multiple hosts

each request can be sent to more than 1 host, the exact same copy of the request is sent asynchronously to all of them, after all copies have been sent, we asynchronously wait for the results. So the whole request is as slow as the slowest host.

result

the results are un-sorted, so if the request is for host 127.0.0.1:9000, 10.0.0.2:9000 the result array might be ([arrayref of the result of 10.0.0.2:9000],[arrayref of 127.0.0.1:9000]).

Every result object is always array, even if the request has only 1 host.

timeout

the timeout is total timeout, includint connect time + send time + rect time for all the hosts combined. So when you say 0.5 (half a second) and give it 20 hosts, the moment it reaches 0.5 seconds, it will die, regardless of how much data it received, or how much data it is about to send.

error

there is no error object, the module dies on every error (including timeouts, query syntax errors etc..).

SEARCH

my $s = Search::Brick::RAM::Query->new(host => '127.0.0.1:9000', index => '__test__');
my @results = $s->search({ term => { author => 'jack' } }, { size => 20, explain => true, log_slower_than => 5 });

QUERY SYNTAX

boosting

every query except term and lucene supports boost parameter like:

bool => { must [ { term => { author => 'jack' } } ], boost => 4.3 }

term

creates http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/TermQuery.html

syntax:

term => { field => value }

example:

my @results = $s->search({ term => { author => 'jack' } });

since RAMBrick does not do any query rewrites (like ElasticSearch's match query) and it also does not do any kind of analysis on the query string, the term and lucene queries are the only queries that can be used to match specific documents.

lucene

creates http://lucene.apache.org/core/4_6_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description

syntax:

lucene => "author:john"

example:

my @result = $s->search({ lucene => "ConstantScore(author:john)^8273" });

you can do pretty much everything with it like in this example it creates a constant score query over a term query

dis_max

creates https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/DisjunctionMaxQuery.html

syntax:

dis_max => {
    queries => [ 
        { term => { author => 'jack' } },
        { term => { category => 'comedy' } } 
    ],
    boost => 1.0,
    tie_breaker => 0.3
}

bool

creates: https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/BooleanQuery.html

syntax:

bool => {
    must => [
        { term => { author => 'jack' } },
    ],
    should => [
        { term => { category => 'drama' } },
        { term => { category => 'comedy' } }
    ],
    must_not => [
        { term => { deleted => 'true' } }
    ],
    minimum_should_match => 1,
    boost => 1.0
}

constant_score

creates https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/ConstantScoreQuery.html

syntax:

constant_score => { query => ..., boost => 4.0 }

filtered

creates https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/QueryWrapperFilter.html or https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/CachingWrapperFilter.html

syntax:

filtered => { query => ..., filter => { query.. } }
filtered => { query => ..., cached_filter => { query.. } }

example:

$s->search(
{
       filtered => { 
                query => { match_all => {} },
                cached_filter => { term => { author => "jack" } }
       } 
})

match_all

creates http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/MatchAllDocsQuery.html

syntax:

match_all => {}

custom_score

syntax:

{ 
   custom_score => { 
       query => { 
           term => { author => "jack" } 
       }, 
       class => 'bz.brick.RAMBrick.BoosterQuery', 
       params => {} 
   }
}

will create an instance of BoosterQuery, with "Map<String,Map<String,String>>" params

look at https://github.com/jackdoe/brick/blob/master/bricks/RAMBrick/queries/BoosterQuery.java for simple example

span_first

creates http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/spans/SpanFirstQuery.html

synax:

{ 
   span_first => { 
       match => { 
           span_term => { author => "doe" }
       }, 
       end => 1 
   }
}

matches the term "doe" in the first "end" positions of the field more detailed info on the span queries: http://searchhub.org/2009/07/18/the-spanquery/

span_near

creates http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/spans/SpanNearQuery.html

syntax: { span_near => { clauses => [ { span_term => { author => 'jack' } }, { span_term => { author => 'doe' } } ], slop => 0, in_order => true() } }

more detailed info on the span queries: http://searchhub.org/2009/07/18/the-spanquery/ example:

{
   span_near => {
       clauses => [
           {
               span_near => {
                   clauses => [
                       { span_term => { author => 'jack' } },
                       { span_term => { author => 'doe' } }
                    ],
                    slop => 1
               },
           },
           {
               span_term => { author => 'go!' }
           }
       ],
       slop => 0
   }
}

span_term

creates http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/spans/SpanTermQuery.html

syntax: span_term => { field => value }

span_term queries are the building block of all span queries

QUERY SETTINGS

log_slower_than

syntax:

log_slower_than => 5

example:

my @result = $s->search({ term => { author => 'jack' } },{ log_slower_than => 5 });

if the query takes more than 5 milliseconds, in the return object there will be a query key which will contain the actual query. in this case it will look like this:

{
  hits => [],
  took => 6,
  query => "author:jack"
}

explain

syntax: explain => true()

example: my @result = $s->search({ term => { author => 'jack' } },{ explain => true() });

will fill '__explain' field in each document, like this:

{
  hits => [
 {
...
'__score' => '4.19903135299683',
'__explain' => '4.1990314 = (MATCH) weight(author:jack in 19) [BoostSimilarity], result of:
 4.1990314 = <3.2> value of f_boost field + subscorer
   0.9990314 = score(doc=19,freq=1.0 = termFreq=1.0), product of:
     0.9995156 = queryWeight, product of:
       0.9995156 = idf(docFreq=2064, maxDocs=2064)
       1.0 = queryNorm
     0.9995156 = fieldWeight in 19, product of:
       1.0 = tf(freq=1.0), with freq of:
         1.0 = termFreq=1.0
       0.9995156 = idf(docFreq=2064, maxDocs=2064)
       1.0 = fieldNorm(doc=19)
',
'author' => 'jack',
...
}
  ],
  took => 6,
}

dump_query

syntax:

dump_query => true()

example:

my @result = $s->search({ term => { author => 'jack' } },{ dump_query => true() });

will return the actual query.toString() in the result structure:

[
 {
   'took' => 109,
   'query' => 'author:jack',
   'hits' => [ {},{}... ]
 }
]

INDEX

all the indexes are created from messagepack'ed streams, following the same protocol:

mapping + settings
data

example:

my $settings = {
   mapping => {
       author => {
           type  => "string", # "string|int|long|double|float",
           index =>  true(),
           store =>  true(),
           omit_norms => false(),
           store_term_vector_offsets => false(),
           store_term_vector_positions => false(),
           store_term_vector_payloads => false(),
           store_term_vectors => false(),
           tokenized => true()
       },
       group_by => {
           type  => "string",
           index =>  true(),
           store =>  false()
       }
   },
   settings => {
       expected => 2, # number of expected documents from the data-part
       shards => 4,   # how many shards will be created 
                      # (in this example each shard will have 1/4th of the data)
       similarity => "bz.brick.RAMBrick.IgnoreIDFSimilarity", # can use:
                                                              # org.apache.lucene.search.similarities.BM25Similarity
                                                              # org.apache.lucene.search.similarities.DefaultSimilarity
                                                              # etc..
       expect_array => true(),
       store => "/var/lib/brick/ram/primary_index_20022014" # it will actually create lucene index there
                                                            # and next time it tries to autoload the file
                                                            # it will just check if number of documents
                                                            # match.
   }
};

check out the field type options from: https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/document/FieldType.html similarity information: https://lucene.apache.org/core/4_6_0/core/org/apache/lucene/search/similarities/Similarity.html

the data can be in array format, or just concatinated documnts joined by '' (depending on the expect_array setting)

the store option tells RAMBrick to store the lucene indexes on disk(instead of using http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/store/RAMDirectory.html), just specify the directory name in the index's settings (it MUST be somewhere within the RAMBRICK_AUTO_LOAD_ROOT).

the structure in our $settings example will look like:

/var/lib/brick/ram/primary_index_20022014/SHARD_0
/var/lib/brick/ram/primary_index_20022014/SHARD_1
/var/lib/brick/ram/primary_index_20022014/SHARD_2
/var/lib/brick/ram/primary_index_20022014/SHARD_3

each of those directories will contain the lucene index (using http://lucene.apache.org/core/4_6_0/core/org/apache/lucene/store/NIOFSDirectory.html)

In case the number of expected documents does not match the number of documents indexed, it will not create the index.

file backed

You can create .msgpack files by concatinating the settings with the data, and just putting it in the RAMBRICK_AUTO_LOAD_ROOT (by default /var/lib/brick/ram/) directory (start brick with RAMBRICK_AUTO_LOAD_ROOT env variable set to wherever you want). Those indexes can also be stored as lucene index (if the store setting points to a directory within the RAMBRIKC_AUTO_LOAD_ROOT)

online

my $s = Search::Brick::RAM::Query->new(host => '127.0.0.1:9000', index => '__test__');
$s->index([{ author => 'jack', group_by => "23" },{ author => 'jack', group_by => "24" }],$settings);

this will just send one blob of data to ram:store:__test__, which will be rerouted to RAMBrick.store('__test__',unpacker) and the next portion of data will be in the format <settings><data...>, this index will be lost after the brick server is restarted.

delete

when you delete an index, it will delete the autoload .msgpack file + the stored directory (containing all the lucene indexes)

my $s = Search::Brick::RAM::Query->new(host => '127.0.0.1:9000', index => '__test__');
$s->delete();

ALIAS

The aliases state is kept in a small metadata file named $RAMBRICK_AUTO_LOAD_ROOT/alias.metadata, every time alias is modified/created/deleted it will update this file. This file is loaded into the alias hash at init time.

my $s = Search::Brick::RAM::Query->new(host => '127.0.0.1:9000', index => '__test__');
$s->alias({
  add => [ { "some_alias_name" => "some_index_name" } ],
  delete => [ "some_alias","some_other_alias" ]
});

aliases are atomic, meaning that the whole request will be executed at one go. the order of the add and delete operation is undefined, but the order within the operations is honored.

STAT

my $s = Search::Brick::RAM::Query->new(host => '127.0.0.1:9000', index => '__test__');
print Dumper([$s->stat()]);

will produce:

[
 {
  'index.default.groups' => 20
  'index.default.searches' => 4,
  'index.default.documents' => 9999,
  'index.default.last_query_stamp' => 1391416081,
  'java.non_heap.usage.used' => 12678192,
  'java.heap.usage.init' => 31138368,
  'java.heap.usage.committed' => 91226112,
  'java.heap.usage.used' => 47775336,
  'java.non_heap.usage.init' => 24576000,
  'java.heap.usage.max' => 620756992,
  'java.non_heap.usage.max' => 224395264,
  'java.non_heap.usage.committed' => 24576000,
  'main.connection_pool.size' => 12,
  'main.connection_pool.active' => 1,
  'brick.search_pool.size' => 15,
  'brick.search_pool.active' => 0,
  'brick.time_indexing' => 3,
  'brick.uptime' => 212,
  'brick.time_searching' => 0,
  'brick.searches' => 8,
 }
]

EXAMPLES:

at the moment it looks like this: sister queries are joined by BooleanQuery with a MUST clause for example:

{ 
    term => { "author" => "john" },
    dis_max => { 
       queries => [ 
           { term => { "category" => "horror" } },
           { term => { "category" => "comedy" } } 
       ], 
       tie_breaker => 0.2 
    }
},

will generate: +(category:horror | category:comedy)~0.2 +author:john. different example:

query(
 host => ['127.0.0.1:9000','127.0.0.1:9000'],
 request => { 
    term => { "author" => "john" } ,
    dis_max => { 
       queries => [ 
        { term => { "category" => "horror" } },
        { term => { "category" => "comedy" } } 
       ], 
       tie_breaker => 0.2 
    },
    bool => {
        must => [
           { lucene => "isbn:74623783 AND year:[20021201 TO 200402302]" }
        ],
        should => [
           { lucene => "cover:(red OR blue)" },
           { term => { old => "0" } }
        ],
        must_not => [
           { term => { deleted => "1" } }
        ]
    }
 },
 brick => 'RAM',
 timeout => 10,
 settings => { "dump_query" => "true" });

generates: +((+(+isbn:74623783 +year:[20021201 TO 200402302]) -deleted:1 (cover:red cover:blue) old:0)~1) +(category:horror | category:comedy)~0.2 +author:john

another example:

my $b = Search::Brick::RAM::Query->new(index => '__test__');
$b->delete();
my $settings = {
   mapping => {
       author => {
           type  => "string",
           index =>  Data::MessagePack::true(),
           store =>  Data::MessagePack::true(),
       },
       f_boost => {
           type  =>  "float",
           index =>  Data::MessagePack::true(),
           store =>  Data::MessagePack::true(),
       },
       group_by => {
           type  => "string",
           index =>  Data::MessagePack::true(),
           store =>  Data::MessagePack::true()
       }
   },
   settings => {
       expected => 2,
       shards => 1
   }
};

$b->index([
    { author => 'jack', group_by => "23", f_boost => 0.5 },
    { author => 'john', group_by => "24",f_boost => 0.5 }
  ],$settings);
my @result = $b->search({ lucene => "ConstantScore(author:john)^8273" },
                        { log_slower_than => "1",explain => "true" } );
$VAR1 = [{
 'took' => 4,
 'hits' => [
   {
     '__score' => '8273.5',
     'f_boost' => '0.5',
     '__group_index' => '0',
     'group_by' => '24',
     '__explain' => '8273.5 = (MATCH) sum of:
 8273.5 = (MATCH) weight(author:john^8273.0 in 1) [BoostSimilarity], result of:
   8273.5 = <0.5> value of f_boost field + subscorer
     8273.0 = score(doc=1,freq=1.0 = termFreq=1.0), product of:
       8273.0 = queryWeight, product of:
         8273.0 = boost
         1.0 = idf(docFreq=1, maxDocs=2)
         1.0 = queryNorm
       1.0 = fieldWeight in 1, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         1.0 = idf(docFreq=1, maxDocs=2)
         1.0 = fieldNorm(doc=1)
',
     'author' => 'john'
   }
 ],
 'query' => '__no_default_field__:ConstantScore author:john^8273.0'
}];

as you can see the return structure is [{},{},{}] one result per request (for example if we do $b = Search::Brick::RAM::Query->new(host => [ '127.0.0.1:900','127.0.0.1:900]) there will be [{hits => []},{hits => []}] in the output)

AUTHOR

Borislav Nikolov, <jack@sofialondonmoskva.com>

COPYRIGHT AND LICENSE

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.18.2 or, at your option, any later version of Perl 5 you may have available.

1 POD Error

The following errors were encountered while parsing the POD:

Around line 276:: Can't have a 0 in =over 0

To install Search::Brick::RAM::Query, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Search::Brick::RAM::Query

CPAN shell

perl -MCPAN -e shell
install Search::Brick::RAM::Query

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

FUNCTION: Search::Brick::RAM::Query::query( $args :Hash ) :Array

Query object

multiple hosts

result

timeout

error

SEARCH

QUERY SYNTAX

boosting

term

lucene

dis_max

bool

constant_score

filtered

match_all

custom_score

span_first

span_near

span_term

QUERY SETTINGS

log_slower_than

explain

dump_query

INDEX

ALIAS

STAT

EXAMPLES:

SEE ALSO

AUTHOR

COPYRIGHT AND LICENSE

Module Install Instructions