NAME
ODG::Record - Perl extension for efficient and simple manipulation of row based records.
SYNOPSIS
use ODG::Record;
my $record = ODG::Record->new(
{
_data_ => [1..25] ,
_layout_ => ODG::Layout->new( .. )
}
);
# Data can then be accessed with the _data_ l-value accessor
$record->_data_ = [ 26..50 ] ; DESCRIPTION
ODG::Record is an extensible class for efficiently and simply working with row based records. In short, this module provides two functions: lvalue accesors to the record (for simplicity) and manipultation data using an ArrayRefs for (for Performance).
lvalue accessors allow you to be terse with your code.
$record->first_name = "Frank" head1 DETAILS
Data and layout information are separate concerns existing in seperate slots (_data_ and _layout_ ) within the ODG::Record object.
Since the _layout_ (i.e. metadata) does not change between records, separating the _data_ from _layout_ allows for greater efficient when processing row based records via object recycling. Rather than creating a new object for each record, the new data is placed in the _data_ slot of an existing ODG::Record object. Since data is stored as an ArrayRef, this is a huge performance win.
ODG::Records require an ODG::Layout object ( a _data_ ArrayRef is optional ) for instantiation. During object construction, name-based accessors are built for each recod field. By default, the accessors permit lvalue assignment.
Object Model
ODG::Record
|
|- slot: _data_
|
|- slot: _layout_ (container for ODG::Metadata objects )
| |
| |- slot: _metadata_
|
|
|- slot: _metadata_ ( ref to ODG::Record::_layout_::_metadata_DISCUSSION
Object Recycling
This module is designed for efficient streaming, i.e. accessing only one record at a time. By repopulating the data slot, i.e. recycling the record object, we do not incur the expensive of object instantiation for each record. A huge win.
The downside is that the checking / validation in new object creation might be lost. This may or not be acceptable depending on the situation. Generally, when the records are well-defined and well-maintained, i.e. from a database, this is not an issue. The data from one record to the next is fairly consistent.
Encapsulation may also be lost. Again, whether this is acceptable depends on the situation. There are several methods of object recycling ( as opposed to creating a new object). They are:
* using a standard accessor, * using an lvalue accessor (not officially supported by Moose -and- may break encapsulation. ( Since this is Moose, other Moose techniques can be used for validation, e.g. after method) * direct access (breaks encapsulation).
_data_ assignment performance
Here is a comparison of methods for placing new data in the record object on an Intel(R) Xeon(TM) CPU 3.06GHz processor:
Data has 5 elements: 1..5
Rate new object moose accessor lvalue moose direct access
new object 1268/s -- -99% -100% -100%
moose accessor 222222/s 17428% -- -56% -78%
lvalue moose 500000/s 39337% 125% -- -50%
direct access 1000000/s 78775% 350% 100% --
Data has 25 elements: 1..25
Rate new object moose accessor lvalue moose direct access
new object 1243/s -- -99% -100% -100%
moose accessor 166667/s 13308% -- -37% -54%
lvalue moose 266667/s 21353% 60% -- -27%
direct access 363636/s 29155% 118% 36% --
new object : ODG::Record->new( { _data_ => qw( [ 1..5] ) } );
moose accessor : $record->data( [ 1..5 ] )
lvalue moose : $record->data = [ 1..5 ]
direct access : $record->_data_ = [ 1.5 ]In real situations, there is probably not much of a difference between the last three techniques, other bottleneck are likely to occur in the the code such as I/O ability to return > 100k/s.
Note to self: What is the comparison to fully encapsulate inside-out objects. Since inside out objects uses references, we would expect them to behave similar to the moose accessors.
RecordSet
This idea can be extended to a RecordSet where multiple records are placed in the data slot. This may be advantageous when batching is more appropriate. Some reasons for this might be related to:
* I/O constraints, especially latency provide for more
efficient batch processing
* Processing requires batch methods, look-backs e.g.
* Processing benefits from batch methods. i.e. records
are sorted in order and one event needs to be triggered
for all records of one type. An example of a RecordSet is demonstrated at:
http://code2.0beta.co.uk/moose/svn/Moose/trunk/t/200_examples/008_record_set_iterator.tThis demonstration is not efficient since it seems that each record object requires instantiation (costly).
Mixing of attributes, data and metadata
It is bad design to store object attributes, data and metadata in the same construct, ie all as attributes. A conflict arises when the field names of data conflict with the field names of the metadata. These should be seperated.
The interface should be designed such that the user has access principally to the data, via object->field syntax. And it seems sloppy to save the metadata with a alternate naming convention. Some alternative might be:
A special slot for attributes and metadata. These can be called _data_ and _layout_, e.g.
$record->field_1
sub field1 :lvalue {
$_[0]->{_data_}->[ $_[0]->{_layout_}->{field1}->{pos} ]Use $record->meta already exists, so this seems like a bad place to store information.
Absent a better methodology we stick to _name_.
METHODS
new
Object constructor. Creates and returns a ODG::Record object. It takes the following options.
_layout_ A ODG::Layout object containing the metadata for record. By convention, the first position has an index of 0.
_data_
L-value object accessor to the record data. Data is stroed internally as an array reference, so data This is the very fast accessor for the _data_,
# Getter
$record->_data_ # Retrieve entire array ref
$record->_data_->[ $index ] # Get a specific field
# Setter
$record->_data_( [ .. ] )
$record->_data_ = [ .. ]
$record->_data_->[ $index ] = $value _layout_
READ-ONLY accessor to the layout object.
_metadata_
Convenience method for accessing the _layout_->_metadata_ object.
EXPORT
None by default.
SEE ALSO
ODG::Metadata, Moose
THANKS
Steven Little, author of Moose
Paul Driver for suggesting to place the accessor methods in the instance rather than the class.
Members of moose@perl.org.
AUTHOR
Christopher Brown, <http://www.opendatagroup,com>
COPYRIGHT AND LICENSE
Copyright (C) 2008 by Open Data
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.10.0 or, at your option, any later version of Perl 5 you may have available.