NAME
BuzzSaw::Importer - Imports log entries of interest from data sources
VERSION
This documentation refers to BuzzSaw::Importer version 0.10.4
SYNOPSIS
use BuzzSaw::Importer;
use BuzzSaw::DataSource::Files;
my $source = BuzzSaw::DataSource::Files->new(
parser => "RFC3339",
names => [qr/^.*\.log(-\d+)?$/],
directories => [@ARGV],
recursive => 1,
);
my $importer = BuzzSaw::Importer->new(
sources => [$source],
filters => ["Kernel"],
);
$importer->import_events;DESCRIPTION
The BuzzSaw project provides a suite of tools for processing log file entries. Entries in files are parsed and filtered into a set of events of interest which are stored in a database. A report generation framework is also available which makes it easy to generate regular reports regarding the events discovered.
ATTRIBUTES
- sources
-
This is a reference to a list of objects which implement the BuzzSaw::DataSource role. You must specify at least one data source. These sources will be queried for entries until all sources are exhausted.
Each element in the list can be expressed as an array reference where the first element is the name of the class of object to be created. Any subsequent elements are then passed in as arguments for the object creation. This makes it possible to do something like this:
my $importer = BuzzSaw::Importer->new( sources => [ [ Files => { names => ["*.log"], directories => ["/var/log"], recursive => 0 } ], ], );This is primarily useful for saving the complete configuration into a file for later use with the
new_with_configmethod. - filters
-
This is a reference to a list of objects which implement the BuzzSaw::Filter role. Each filter is called in sequence for each log entry to find events of interest. If any filter expresses interest in the event then it will be stored into the database. Note that it is possible to do fairly complex filtering by careful sequencing of the filter order, this allows a filter to rely on the results of those earlier in the stack.
There are 3 possible scenarios for results returned by a filter: (1) If the returned value is positive the entry and tags will be stored. (2) If the returned value is zero the entry will not be stored unless another filter in the stack expresses interest, any tags returned will be totally ignored. (3) If the returned value is negative then the entry will not be stored unless another filter in the stack expresses interest BUT the tags will be retained and stored if the final decision is to store the entry. This makes it possible to do additional post-processing which does not alter the results from the previous filters. For instance, the UserClassifier filter adds a user type tag for any filter which sets the
useridfield (e.g. SSH and Cosign).If you do not specify any filters then ALL events will be automatically accepted and the entire data set will be stored into the database.
If an element in the list passed in is a string then it is considered to be a class name in the BuzzSaw::Filter namespace. Short names are allowed, e.g. passing in
Kernelwould result in a new BuzzSaw::Filter::Kernel object being created. - db
-
This is a reference to the BuzzSaw::DB object which will be used to store any events of interest. It will also be passed into the various data source objects so that it can be used to register parsed log sources, check for previously seen sources, etc.
It is possible to specify this as a string in which case that will be considered to be a configuration file name and it will be handed off to the
new_with_configmethod for the BuzzSaw::DB class.If you do not specify the BuzzSaw::DB object then a new one will be created by calling the
new_with_configmethod (which will use the default configuration file name for that class). - readall
-
This is a boolean value which controls whether or not all files should be read. If it is set to
true(i.e. a value of 1 - one) then the code which attempts to avoid re-reading previously seen files will not be used. The default value isfalse(i.e. a value of 0 - zero). When this value is set to true it will be set to true for all the data sources, this makes it easier to override globally. When it is false then the specific setting for the data source will be used.
SUBROUTINES/METHODS
This class has the following methods:
- $importer = BuzzSaw::Importer->new()
-
This will create a new BuzzSaw::Importer object. You will need to specify, at least, a data source.
- $importer = BuzzSaw::Importer->new_with_config()
-
This will create a new BuzzSaw::Impoter object using the attribute values stored in the configuration file. A filename maybe be specified, if not the default value will be used. The value for any attribute can be overridden.
- $importer->import_events
-
This is the method which does all the work. It works through the streams of entries from each data source. Firstly the SHA-256 digest is calculated for each event, any event which has previously been seen will then be ignored. For new events they are parsed into their constituent parts using the relevant BuzzSaw::Parser. The parsed event is then passed through all the specified BuzzSaw::Filter objects. If the event is of interest it is then stored using the BuzzSaw::DB object.
DEPENDENCIES
This module is powered by Moose. You will also need MooseX::Types, MooseX::Log::Log4perl and MooseX::SimpleConfig.
This module also requires the DateTime and Readonly modules.
SEE ALSO
BuzzSaw, BuzzSaw::DB, BuzzSaw::DataSource, BuzzSaw::Parser, BuzzSaw::Filter
PLATFORMS
This is the list of platforms on which we have tested this software. We expect this software to work on any Unix-like platform which is supported by Perl.
ScientificLinux6
BUGS AND LIMITATIONS
Please report any bugs or problems (or praise!) to bugs@lcfg.org, feedback and patches are also always very welcome.
AUTHOR
Stephen Quinney <squinney@inf.ed.ac.uk>LICENSE AND COPYRIGHT
Copyright (C) 2012-2013 University of Edinburgh. All rights reserved.This library is free software; you can redistribute it and/or modify it under the terms of the GPL, version 2 or later.