NAME

Elastic::Manual::Attributes - Fine-tuning how your attributes are indexed

VERSION

version 0.52

SYNOPSIS

package MyApp::User;

use Elastic::Doc;

has 'email' => (
    isa       => 'Str',
    is        => 'rw',
    analyzer  => 'english',
    multi => {
        untouched => { index    => 'not_analyzed' },
        ngrams    => { analyzer => 'edge_ngrams' }
    }
);

no Elastic::Doc;

INTRODUCTION

Elastic::Model uses Moose's type constraints to figure out how each of the attributes in your classes should be indexed. This is a good start, but it won't be long before you want to fine tune the indexing process.

For instance, if you have attribute title which isa => 'Str', then Elastic::Model::TypeMap::Moose will map the attribute as a string, whose value will be analyzed by the standard analyzer.

But perhaps you want the words in your title to be stemmed (eg "fox" will match "fox", "foxes" or "foxy"), or perhaps your title is in Spanish and you want Spanish stemming rules to apply. Or perhaps you want to use as-you-type auto-completion on the title attribute.

This fine-tuning is easy to do, using the attribute keywords that are added to your attributes when you include the line use Elastic::Doc; in your class. These keywords come from Elastic::Model::Trait::Field.

TYPES

Each attribute/field must have one of the following types:

string
integer, long, float, double, short, byte
date
boolean
binary
ip (IPv4 addresses)
geo_point
object
nested (a specialized form of object)
multi_field (index the same field in different ways)

type

The type keyword allows you to override the default type that is generated by the typemap. For instance:

has 'epoch_milliseconds' => (
    isa     => 'Int',
    type    => 'date'
);

Although the type constraint Int would normally generate type long, we have, overridden the type to be indexed as a date field.

Arrays

A notable omission from the above list is array. That's because a field of any type can store and index multiple values. For example:

has 'author'  => ( isa => 'Str'           );
has 'authors' => ( isa => 'ArrayRef[Str]' );

Both of these fields would be mapped as { type => 'string' }.

Note: the values contained in an array must be of a single type in order to be indexable. You can't mix types in an array.

Also see "NESTED FIELDS" for a more detailed explanation, specifically about how to deal with arrays of objects.

Unknown types

If the typemap does not know how to deal with the type constraint on an attribute, and you haven't specified a "type", then the field is mapped as:

{
    type    => 'object',
    enabled => 0
}

This means that the value in the field will be stored in Elasticsearch, but will not be queryable. Also, no attempt is made to inflate or deflate the value - it is just passed through unchanged. If it is a value that JSON::XS cannot handle natively, (eg a blessed ref), then Elastic::Model will be unable to deal with it automatically.

See "CUSTOM MAPPING, INFLATION AND DEFLATION" for details of how to handle this situation.

GENERAL KEYWORDS

There are a number of keywords that apply to (almost) all of the field types listed below:

exclude

has 'cache_key' => (
    is      => 'ro',
    exclude => 1,
    builder => '_generate_cache_key',
    lazy    => 1,
);

If exclude is true then the attribute will not be stored in Elasticsearch. This is only useful for generated or temporary attributes. If you want to store the attribute but make it not searchable, then you should use the "index" keyword instead.

index

has 'tag' => (
    is      => 'rw',
    isa     => 'Str',
    index   => 'not_analyzed'
);

The index keyword controls how Elasticsearch will index your attribute. It accepts 3 values:

no: This attribute will not be indexed, and will thus not be searchable.
not_analyzed: This attribute will be indexed using exactly the value that you pass in, eg FoO will be stored (and searchable) as FoO. Also see "Keyword" in Elastic::Model::TypeMap::ES.
analyzed: This attribute will be analyzed. In other words, the text value will be passed through the specified (or default) "analyzer" before being indexed. The analyzer will tokenize and pre-process the text to produce terms. For example FoO BAR would (depending on the analyzer) be stored as the terms foo and bar. This is the default for string fields (except for enums).

include_in_all

has 'secret' => (
    is              => 'ro',
    isa             => 'Str',
    include_in_all  => 0
);

By default, all attributes (except those with index => 'no') are also indexed in the special _all field. This is intended to make it easy to search for documents that contain a value in any field. If you would like to exclude a particular attribute from the _all field, then specify { include_in_all => 0 }.

Note:

The _all field has its own "analyzer" - so the tokens that are stored in the _all field may be different from the tokens stored in the attribute itself.
When include_in_all is set on a field of type object, its value will propagate down to all attributes within the object.

You can disable the _all field completely using "has_mapping" in Elastic::Doc:

package MyApp::User;

use Elastic::Doc;

has_mapping {
    _all    => { enabled => 0 }
};

boost [DEPRECATED]

has 'title' => (
    is      => 'rw',
    isa     => 'Str',
    boost   => 2
);

NOTE: This attribute is deprecated in Elasticsearch and will be removed in a future version.

A boost makes a value "more relevant". For instance, the words in the title field of a blog post are probably a better indicator of the topic than the words in the content field. You can boost a field at search time and at index time. The benefit of boosting at search time, is that your boost is not fixed. The benefit of boosting at index time is that the boost value is carried over to the _all field.

Also see "omit_norms".

multi

has 'name' => (
    is      => 'ro',
    isa     => 'Str',
    multi   => {
        sorting     => { index    => 'not_analyzed' },
        partial     => { analyzer => 'ngrams'       }
    }
);

It is a common requirement to be able to use a single field in different ways. For instance, with the name field example above, we may want to:

Do a full text search for Joe Bloggs
Do a partial match on all names beginning with Blo
Sort results alphabetically by name.

A single field definition is insufficient in this case: The standard analyzer won't allow partial matching, and because it generates multiple terms/tokens, it can't be used for sorting. (You can only sort on a single value).

This is where multi_fields are useful. The same value can be indexed and queried in multiple ways. When you specify a multi mapping, each "sub-field" inherits the type of the main field, but you can override this. The "sub-fields" can be referred to as eg name.partial or name.sorting and the "main field" name can also be referred to as name.name.

Another benefit of multi-fields is that they can be added without reindexing all of your data.

index_name [DEPRECATED]

has 'foo' => (
    is          => 'rw',
    isa         => 'Str',
    index_name  => 'bar'
);

NOTE: This attribute is deprecated in Elasticsearch and will be removed in a future version.

Elasticsearch uses dot-notation to refer to nested hashes. For instance, with this data structure:

{
    foo => {
        bar => {
            baz => 'xxx'
        }
    }
}

... you could refer to the baz value as baz or as foo.bar.baz.

Sometimes, you may want to specify a different name for a field. For instance:

{
    street => {
        name    => 'Oxford Street',
        number  => 1
    },
    town => {
        name    => 'London'
    }
}

You can use the index_name to distinguish town_name from street_name.

unique_key

has 'email' => (
    is          => 'ro',
    isa         => 'Str',
    unique_key  => 'myapp_email'
);

The "unique_key" keyword allows you to ensure that an attribute's value is unique, eg to ensure that only one user account has a particular email address. See Elastic::Manual::Attributes::Unique for a fuller explanation.

store

has 'big_field' => (
    is          => 'ro',
    isa         => 'Str',
    store       => 'yes'
);

Individual fields can be stored (ie have their original value stored on disk). This is not the same as whether the value is indexed or not (see "index"). It just means that this individual value can be retrieved separately from the others. stored defaults to 'no' but can be set to 'yes'.

You almost never need this. The _source field (which is stored by default) contains the hashref representing your whole object, and is returned by default when you get or search for a document. This means a single disk seek to load the_source field, rather than a disk seek (think 5ms) for every stored field! It is much more efficient.

null_value

has 'foo' => (
    is          => 'rw',
    isa         => 'Str',
    null_value  => 'none'
);

If the attribute's value is undef then the null_value will be indexed instead. This option is included for completeness, but isn't very useful. Rather just leave the value as undef and use the exists and missing filters when you need to consider undef values.

STRING FIELDS

String fields fall into two broad categories:

Text, like this paragraph, which needs to be searchable via flexible full text queries
Terms, for instance tags, postcodes, enums, which should be stored and queried exactly (eg Perl is different from PERL)

The simplest way to distinguish between text and terms is to set the "index" keyword to analyzed (the default) for text or to not_analyzed for terms. See Elastic::Manual::Analysis for a more detailed discussion.

The keywords below apply only to fields of "type" string. You can also use "index", "include_in_all", "boost", "multi", "index_name", "unique_key", "store" and "null_value".

analyzer

has 'email' => (
    is          => 'ro',
    isa         => 'Str',
    analyzer    => 'my_email_analyzer'
);

Specify which analyzer (built-in or custom) to use at index time and at search time. This is the equivalent of setting "index_analyzer" and "search_analyzer" to the same value.

Also see Elastic::Manual::Analysis for an explanation.

index_analyzer

has 'email' => (
    is              => 'ro',
    isa             => 'Str',
    index_analyzer  => 'my_email_analyzer'
);

Sets the "analyzer" to use at index time only.

search_analyzer

has 'email' => (
    is              => 'ro',
    isa             => 'Str',
    search_analyzer => 'my_email_analyzer'
);

Sets the "analyzer" to use at search time only.

search_quote_analyzer

has 'email' => (
    is                    => 'ro',
    isa                   => 'Str',
    search_analyzer       => 'my_email_analyzer',
    search_quote_analyzer => 'my_quoted_email_analyzer'
);

Sets the "analyzer" to use in a Query-String query when the search phrase includes quotes (""). If not set, then it falls back to the "search_analyzer" or the "analyzer".

term_vector

has 'text' => (
    is          => 'ro',
    isa         => 'Str',
    store       => 'yes',
    term_vector => 'with_positions_offsets',
);

The full functionality of term vectors is not exposed via Elasticsearch, so the only real value for now is for highlighting snippets. Allowed values are: no (the default), yes, with_offsets, with_positions and with_positions_offsets.

Fast snippet highlighting

There are two highlighters available for highlighting matching snippets in text fields: the highlighter, which can be used on any analyzed text field without any preparation, and the fast-vector-highlighter which is faster (better for large text fields which require frequent highlighting), but uses more disk space, and the field needs to be setup correctly before use:

has 'big_field' => (
    is          => 'ro',
    isa         => 'Str',
    term_vector => 'with_positions_offsets'
);

NUMERIC FIELDS

Numeric fields can have any of the following types:

byte: Range: -128 to 127
short: Range: -32,768 to 32,767
integer: Range: -2,147,483,648 to 2,147,483,647
long: Range: -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
float: Single-precision 32-bit IEEE 754 floating point
double: Double-precision 64-bit IEEE 754 floating point

(See http://docs.oracle.com/javase/specs/jls/se7/html/jls-4.html#jls-4.2.3 for more on single or double value ranges.)

There are no numeric-specific keywords, but the general keywords apply: "index", "include_in_all", "boost", "multi", "index_name", "unique_key", "store" and "null_value".

DATE FIELDS

Dates in Elasticsearch are stored internally as long values containing milliseconds since the epoch. The maximum date range is 292269055 BC to 292278994 AD.

The following keyword applies only to fields of "type" date. You can also use "index", "include_in_all", "boost", "multi", "index_name", "unique_key", "store" and "null_value".

format

has 'year_week' => (
    isa     => 'Str',
    type    => 'date',
    format  => 'basic_week_date'
);

Date fields by default can parse (1) milliseconds since epoch (2) yyyy/MM/dd HH:mm:ss Z or (3) yyyy/MM/dd Z.

If you would like to specify a different format, you can use one of the built-in formats or a custom format.

BOOLEAN FIELDS

Boolean fields accept 0, "" and "false" as false values, and any other value as true.

There are no boolean-specific keywords, but the general keywords apply: "index", "include_in_all", "boost", "multi", "index_name", "store" and "null_value".

IMPORTANT: Moose and Perl consider undef to be an acceptable false value. However, an undef in Perl is converted to a null in JSON, which Elasticsearch would understand as meaning "this field has no value" or "is missing". For this reason we distinguish fields the type constraint of Bool from the type constraint of Maybe[Bool] (although functionally they are identical in Moose).

For the purposes of querying, Elastic::Model treats Bool fields set to undef as false, and Maybe[Bool] set to undef as NULL, "missing" or "not set".

Value           Bool            Maybe[Bool]
---------------------------------------------
1               True            True
0               False           False
Empty string    False           False
Undef           False           NULL / missing
No value        NULL / missing  NULL / missing

This is done by mapping Maybe[Bool] fields as:

{ type => 'boolean' }

and mapping Bool fields as:

{ type => 'boolean', null_value => 0 }

Note: the `null_value` setting only affects how a boolean field is indexed by Elasticseach - it doesn't change the actual value of the field in the object.

BINARY FIELDS

Binary data can be stored in Elasticsearch in Base64 encoding. The easiest way to do this is to use "Binary" in Elastic::Model::Types, which will handle the Base64 conversion for you:

use Elastic::Model::Types qw(Binary);

has 'binary_field' => (
    is      => 'ro',
    isa     => Binary
);

The field is always not indexed.

There are no binary-specific keywords, but you can use: "store" (defaults to yes)and "index_name".

IP FIELDS

Fields of type ip can be used to index IPv4 addresses in numeric form.

There are no ip-specific keywords, but the general keywords apply: "index", "include_in_all", "boost", "multi", "index_name", "unique_key", "store" and "null_value".

GEO_POINT FIELDS

geo_point fields are used to index latitude/longitude points. The easiest way to use them is to use "GeoPoint" in Elastic::Model::Types:

use Elastic::Model::Types qw(GeoPoint);

has 'point' => (
    is      => 'ro',
    isa     => GeoPoint,
    coerce  => 1
);

See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html for more information about geo_point fields.

The following keywords apply only to fields of "type" geo_point. You can also use "multi", "index_name" and "store".

lat_lon

has 'point' => (
    is      => 'ro',
    isa     => GeoPoint,
    lat_lon => 1
);

By default, a geo-point is indexed as a lat-lon combination. To index the lat and lon fields as numeric fields as well, which is considered good practice, as both the geo distance and bounding box filters can either be executed using in memory checks, or using the indexed lat lon values. Note: indexed lat lon only makes sense when there is a single geo point value for the field, and not multiple values.

geohash

has 'point' => (
    is      => 'ro',
    isa     => GeoPoint,
    geohash => 1,
);

Set geohash to true to index the geohash value as well.

This value is queryable via (eg) the point.geohash field.

geohash_precision

has 'point' => (
    is                  => 'ro',
    isa                 => GeoPoint,
    geohash             => 1,
    geohash_precision   => 8,
);

The geohash_precision determines how accurate the geohash will be - defaults to 12.

OBJECT FIELDS

Hashrefs (and objects which have been serialised to hashrefs) are considered to be "objects", as in JSON objects. Your doc class is serialized to a JSON object/hash, which is known as the root_object. The mapping for the root object can be configured with "has_mapping" in Elastic::Doc.

Your doc class may have attributes which are hash-refs, or objects, which may themselves contain hash-refs or objects. Multi-level data structures are allowed.

The following keywords apply only to fields of "type" object and nested. You can also use "include_in_all", "include_attrs", "exclude_attrs".

The mapping for these structures should be automatically generated, but these keywords give you some extra control:

enabled

has 'foo' => (
    is      => 'ro',
    isa     => 'HashRef',
    enabled => 0
);

Setting enabled to false disables the indexing of any value in the object. Defaults to false.

dynamic

has 'foo' => (
    is      => 'ro',
    isa     => 'HashRef',
    dynamic => 1
);

Elasticsearch defaults to trying to detect field types dynamically, but this can lead to mistakes, eg is "123" a string or a long? Elastic::Model turns off this dynamic detection, and instead uses Moose's type constraints to determine what type each field should have.

If you know what you're doing, you can set dynamic to 1 (auto-detect new field types), 0 (ignore new fields) or 'strict' (throw an error if an unknown field is included).

path [DEPRECATED]

NOTE: This attribute is deprecated in Elasticsearch and will be removed in a future version.

The path keyword controls how the attribute names of an object are flattened into field names in Elasticsearch. It can be set to full (the default) or just_path.

For instance, given the following example:

package MyApp::Types;

use MooseX::Types -declare 'FullName';

use MooseX::Types::Moose qw(Str);
use MooseX::Types::Structured qw(Dict);

subtype Fullname,
as Dict[
    first   => Str,
    last    => Str
];


package MyApp::Couple;

use Moose;
use MyApp::Types qw(FullName);

has 'husband' => (
    is      => 'ro',
    isa     => FullName,
);

has 'wife' => (
    is      => 'ro',
    isa     => FullName,
);

A MyApp::Couple object may look like this:

{
    husband => { first => 'John', last => 'Smith' },
    wife    => { first => 'Mary', last => 'Smith' }
}

By default, this data would be flattened and stored as follows:

FIELD NAME        |  VALUES
------------------|---------
husband.first     |  john
husband.last      |  smith
wife.first        |  mary
wife.last         |  smith

These field names, or "paths", can also have the type name prepended, so husband.first could also be referred to as couple.husband.first.

The path keyword can be used to control the construction of this path. For instance, if MyApp::Couple was defined as:

package MyApp::Couple;

use Moose;
use MyApp::Types qw(FullName);

has 'husband' => (
    is      => 'ro',
    isa     => FullName,
    path    => 'just_name'
);

has 'wife' => (
    is      => 'ro',
    isa     => FullName,
    path    => 'just_name'
);

then the values would be indexed as:

FIELD NAME        |  VALUES
------------------|---------------
first             |  john, mary
last              |  smith, smith

... also accessible as couple.first and couple.last.

The path keyword can also be combined with the "index_name" keyword.

NESTED FIELDS

nested fields are a sub-class of object fields, that are useful when your attribute can contain multiple values.

First an explanation. Consider this data structure:

{
    person  => [
        { first => 'John', last => 'Smith' },
        { first => 'Mary', last => 'Smith' },
        { first => 'Mary', last => 'Jones' }
    ]
}

If the person field is of type object, then the above data structure is flattened into something more like this:

{
    'person.first' => ['John','Mary','Mary'],
    'person.last'  => ['Smith','Smith','Jones']
}

With this structure it is impossible to run queries that depend on matching on attributes of a SINGLE person object. For instance, a query asking for docs that have a person who has (first == John and last == Jones) would incorrectly match this document.

Nested objects are the solution to this problem. When an attribute is marked as "type" nested, then Elasticsearch creates each object as a separate-but-related hidden document. These nested objects can be queried with the nested query and the nested filter.

The following keywords apply only to fields of "type" nested. You can also use "include_in_all", "path" "dynamic", "include_attrs" and "exclude_attrs".

include_in_parent

has 'person' => (
    is                  => 'ro',
    isa                 => ArrayRef[Person],
    type                => 'nested',
    include_in_parent   => 1,
);

If you would also like the data from the nested objects to be indexed in their containing object (as in the first data structure above), then set include_in_parent to true.

include_in_root

has 'person' => (
    is                  => 'ro',
    isa                 => ArrayRef[Person],
    type                => 'nested',
    include_in_root     => 1,
);

Objects can be nested inside objects which are nested inside objects etc. The include_in_root keyword does the same as the "include_in_parent" keyword, but refers to the top-most document, rather than the direct parent.

ELASTIC::DOC FIELDS

You can have attributes in one class that refer to another Elastic::Doc class. For instance, a MyApp::Post object could have, as an attribute, the MyApp::User object to whom the post belongs.

You may want to store just the Elastic::Model::UID of the user object, or you may want to include the user's name and email address, so that you can search for posts by a user named "Joe Bloggs". You can't do joins in a NoSQL database, so you need to denormalize your data.

By default, all attributes in an object are included. You can change the list with "include_attrs" and "exclude_attrs". You can use these with Moose classes too, but any attributes that are excluded, won't be stored and it won't be possible to retrieve them.

include_attrs

has 'user' => (
    is            => 'ro',
    isa           => 'MyApp::User',
    include_attrs => ['name','email']
);

The above declaration will index the user object's UID, plus the name and email attributes. If include_attrs is not specified, then all the attributes from the user object will be indexed. If include_attrs is set to an empty array ref [] then no attributes other than the UID will be indexed.

exclude_attrs

has 'user' => (
    is            => 'ro',
    isa           => 'MyApp::User',
    exclude_attrs => ['secret']
);

The above declaration will index all the user attributes, except for the attribute secret.

CUSTOM MAPPING, INFLATION AND DEFLATION

The preferred way to specify the mapping and how to deflate and inflate an attribute is by specifying an isa type constraint and adding a typemap entry.

However, you can provide custom values with the following:

mapping

has 'foo' => (
    is      => 'ro',
    type    => 'string',
    mapping => { index => 'no' },
);

You can specify a custom mapping directly in the attribute, which will be used instead of the typemap entry that would be generated. Any other keywords that you specify (eg "type") will be added to your mapping.

deflator

has 'foo' => (
    is       => 'ro',
    type     => 'string',
    deflator => sub { my $val = shift; return my_deflator($val) },
);

You can specify a custom deflator directly in the attribute. It should return undef, a string, or an unblessed data structure that can be converted to JSON.

inflator

has 'foo' => (
    is       => 'ro',
    type     => 'string',
    inflator => sub { my $val = shift; return my_inflator($val) },
);

You can specify a custom inflator directly in the attribute. It should be able to reinflate the original value from the plain data structure that is stored in Elasticsearch.

AUTHOR

Clinton Gormley <drtech@cpan.org>

COPYRIGHT AND LICENSE

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

To install Elastic::Model, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Elastic::Model

CPAN shell

perl -MCPAN -e shell
install Elastic::Model

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

SYNOPSIS

INTRODUCTION

TYPES

type

Arrays

Unknown types

GENERAL KEYWORDS

exclude

index

include_in_all

boost [DEPRECATED]

multi

index_name [DEPRECATED]

unique_key

store

null_value

STRING FIELDS

analyzer

index_analyzer

search_analyzer

search_quote_analyzer

term_vector

Fast snippet highlighting

NUMERIC FIELDS

DATE FIELDS

format

BOOLEAN FIELDS

BINARY FIELDS

IP FIELDS

GEO_POINT FIELDS

lat_lon

geohash

geohash_precision

OBJECT FIELDS

enabled

dynamic

path [DEPRECATED]

NESTED FIELDS

include_in_parent

include_in_root

ELASTIC::DOC FIELDS

include_attrs

exclude_attrs

CUSTOM MAPPING, INFLATION AND DEFLATION

mapping

deflator

inflator

AUTHOR

COPYRIGHT AND LICENSE

Module Install Instructions