NAME
Data::CTable - Read, write, manipulate tabular data
SYNOPSIS
## Read some data files in various tabular formats
use Data::CTable;
my $People = Data::CTable->new("people.merge.mac.txt");
my $Stats = Data::CTable->new("stats.tabs.unix.txt");
## Clean stray whitespace in fields
$People->clean_ws();
$Stats ->clean_ws();
## Retrieve columns
my $First = $People->col('FirstName');
my $Last = $People->col('LastName' );
## Calculate a new column based on two others
my $Full = [map {"$First->[$_] $Last->[$_]"} @{$People->all()}];
## Add new column to the table
$People->col(FullName => $Full);
## Another way to calculate a new column
$People->col('Key');
$People->calc(sub {no strict 'vars'; $Key = "$Last,$First";});
## "Left join" records matching Stats:PersonID to People:Key
$Stats->join($People, PersonID => 'Key');
## Find certain records
$Stats->select_all();
$Stats->select(Department => sub {/Sale/i }); ## Sales depts
$Stats->omit (Department => sub {/Resale/i}); ## not Resales
$Stats->select(UsageIndex => sub {$_ > 20.0}); ## high usage
## Sort the found records
$Stats->sortspec('DeptNum' , {SortType => 'Integer'});
$Stats->sortspec('UsageIndex', {SortType => 'Number' });
$Stats->sort([qw(DeptNum UsageIndex Last First)]);
## Make copy of table with only found/sorted data, in order
my $Report = $Stats->snapshot();
## Write an output file
$Report->write(_FileName => "Rept.txt", _LineEnding => "mac");
## Print a final progress message.
$Stats->progress("Done!");
## Dozens more methods and parameters available...OVERVIEW
Data::CTable is a comprehensive utility for reading, writing, manipulating, cleaning and otherwise transforming tabular data. The distribution includes several illustrative subclasses and utility scripts.
A Columnar Table represents a table as a hash of data columns, making it easy to do data cleanup, formatting, searching, calculations, joins, or other complex operations.
The object's hash keys are the field names and the hash values hold the data columns (as array references).
Tables also store a "selection" -- a list of selected / sorted record numbers, and a "field list" -- an ordered list of all or some fields to be operated on. Select() and sort() methods manipulate the selection list. Later, you can optionally rewrite the table in memory or on disk to reflect changes in the selection list or field list.
Data::CTable reads and writes any tabular text file format including Merge, CSV, Tab-delimited, and variants. It transparently detects, reads, and preserves Unix, Mac, and/or DOS line endings and tab or comma field delimiters -- regardless of the runtime platform.
In addition to reading data files, CTable is a good way to gather, store, and operate on tabular data in memory, and to export data to delimited text files to be read by other programs or interactive productivity applications.
To achieve extremely fast data loading, CTable caches data file contents using the Storable module. This can be helpful in CGI environments or when operating on very large data files. CTable can read an entire cached table of about 120 megabytes into memory in about 10 seconds on an average mid-range computer.
For simple data-driven applications needing to store and quickly retrieve simple tabular data sets, CTable provides a credible alternative to DBM files or SQL.
For data hygiene applications, CTable forms the foundation for writing utility scripts or compilers to transfer data from external sources, such as FileMaker, Excel, Access, personal organizers, etc. into compiled or validated formats -- or even as a gateway to loading data into SQL databases or other destinations. You can easily write short, repeatable scripts in Perl to do reporting, error checking, analysis, or validation that would be hard to duplicate in less-flexible application environments.
The data representation is simple and open so you can directly access the data in the object if you feel like it -- or you can use accessors to request "clean" structures containing only the data or copies of it. Or you can build your own columns in memory and then when you're ready, turn them into a table object using the very flexible new() method.
The highly factored interface and implementation allow fine-grained subclassing so you can easily create useful lightweight subclasses. Several subclasses are included with the distribution.
Most defaults and parameters can be customized by subclassing, overridden at the instance level (avoiding the need to subclass too often), and further overridden via optional named-parameter arguments to most major method calls.
Similar / related modules on CPAN
The Data::Table module by Yingyao Zhou & Guangzhou Zou offers similar functionality, but uses a different underlying data representation (2-dimensional array), and has a somewhat different feature set. Check it out. Maybe you will prefer it for your application.
http://search.cpan.org/search?mode=module&query=Data::TableThe Data::ShowTable module renders tables in various viewable formats. CTable relies on ShowTable's ShowBoxTable method to implement its own format() and out() methods.
http://search.cpan.org/search?mode=module&query=Data::ShowTablePrerequisites
The CTable documentation, source code, and examples assume familiarity with large nested data structures, object-oriented syntax and terminology, and comfort working with array and hash references and array and hash slice syntax.
See the perlref man page for more on these topics.
How to learn more
Dozens more methods, parameters, and examples are described below.
See the full source code in CTable.pm.
Or, after installing, read the man page using:
man Data::CTable
perldoc Data::CTableSee the eg/ (examples) folder in the Data::CTable distribution and the test.pl script for scripts demonstrating every CTable method.
For latest version and other news, check the Data::CTable home page:
http://christhorman.com/projects/perl/Data-CTable/Or search CPAN:
http://search.cpan.org/search?mode=module&query=Data::CTableINSTALLATION
Using CPAN module:
perl -MCPAN -e 'install Data::CTable'Or manually:
tar xzvf Data-CTable*gz
cd Data-CTable-?.??
perl Makefile.PL
make
make test
make installINCLUDED SUBCLASSES AND UTILITIES
In addition to the module itself, there are a number of subclasses and simple utilities included with the Data::CTable distribution.
Subclases
The Data::CTable distribution includes these example subclasses. Each is installed in your Perl environment along with the main module, and so may be used by your scripts. Each has its own man/perldoc page containing more detail.
Data::CTable::ProgressLogger is a subclass that logs all progress messages to a list within the object itself rather than (merely) echoing them to STDERR. Later, you may retrieve and examine the list.
Data::CTable::Script is a virtual subclass that includes class and object methods that make it easy to write a simple interactive command-line program that parses options and outputs a table.
Data::CTable::Listing is a very useful subclass of Data::CTable::Script that implements a souped-up Unix-like "ls" (file listing) command -- it first gets an optionally-recursive listing of any number of files and/or directories, builds a list of their full absolute or relative paths, then build a Data::CTable::Listing object that contains all the paths, plus about 25+ other pieces of useful information about each file or directory.
The "tls" utility, below, is simply a command-line cover for this class, but you could use this class in your own scripts in order to get detailed file listings.
Utilities
Each of these utilities is provided mainly so you can see mature examples of how to use Data::CTable in real-world scripts.
But each is also genuinely useful, too, and you may enjoy adding them to your regular bag of tricks or using them as an easily-modifiable basis for scripts of your own.
On most systems, these will be installed in an appropriate directory in your path when you install CTable, and hence will be executable just by typing their file name. (On Windows, they'll be wrapped by a .bat file and installed in C:\Perl\bin or equivalent. On *nix, they'll be in /usr/bin/ or equivalent.)
tls is a command-line utility that wraps Data::CTable::Listing to implement a variant on the classic Unix "ls" command in which an internal CTable object is used to hold and calculate a very large amount of meta-data about each file and directory and then output that data in formatted tables, or as any kind of delimited text file, and with much more flexibility and control over included data, and sort and sub-sort order than with ls.
tshow is a command-line utility that reads each of its arguments as a Data::CTable file and then calls the out() method to display its entire contents. (Warning: out() is slow with very large data sets.)
getweather is a command-line utility that takes a US zip code, grabs the local weather report from a popular weather web site and uses a CTable object to store, process, and clean, and present the table of weather data that results in a simple text format.
CREATING TABLE OBJECTS
## Create an object / read file(s) / override params
use Data::CTable;
$t = Data::CTable->new()
$t = Data::CTable->new($File)
$t = Data::CTable->new($File1, $File2....)
$t = Data::CTable->new($Params)
$t = Data::CTable->new($Params, $File1)
$t = Data::CTable->new($Params, $File1, $File2....)
## Internal initializer (subclassable): called for you by new()
$t->initialize()If the first argument to new() is a hash ref, it is the $Params hash of initial parameters and/or data columns which, if supplied will form the starting point for the object being created. Any non-hash and/or further arguments to new() are treated as file names to be opened.
If supplied, data in the $Params hash will be shallowly copied -- the original hash object passed will not be used, but any sub-structures within it will now "belong" to the resulting new object which will feel free to manipulate them or discard them.
Then, any parameters not supplied (usually most of them) will be defaulted for you because new() will call the internal method initialize() before the object is finished and returned.
See the PARAMETER REFERENCE section below for the parameters you can choose to supply or have defaulted for you.
Any file name arguments will be read and appended into a single object. If any of the files fails to be read, then new() will fail (return a false value) and no object will be created.
initialize() makes sure there is a legal and consistent value for every internal parameter in the object. Generally, initialize() leaves alone any parameters you supplied to new(), and simply sets default values for any that were not yet supplied.
You should never need to call initialize() directly.
After calling initialize(), new() then calls the append_files_new() method to process the filename arguments. This method then calls read() on the first filename, and then append_file on all subsequent file names, appending them to the first, in sequence.
See append() for an explanation of how the data from multiple files is combined into a single table.
Advanced: Using a template object
## Calling new() with a template object
$v = $t->new()
$v = $t->new($File)
$v = $t->new($File1, $File2....)
$v = $t->new($Params)
$v = $t->new($Params, $File1)
$v = $t->new($Params, $File1, $File2....)You can also call $t->new() to use an existing object, $t, as a template for the new object. $t->new() will create a new object of the same class or subclass as the template object. Furthermore, the template object, if provided, will be used as a starting point for the resulting object -- in fact, it will initially share shallow copies of all data columns, if any, and all internal parameters and data structures, if any.
This advanced shared-data technique could be used to create two separate table objects that share and operate on the same underlying data columns in memory but have different custom field lists, custom selections, sort behavior, etc. But don't do this unless you're sure you understand what you're doing, because changing data in one table would change it in the other.
PARAMETER REFERENCE
The parameters listed here are recognized by new(), initialize(), and by many functions that use the named-parameter calling convention, such as read(), write(), sort(), and others.
Any parameter listed may be specified when using new().
Most parameters should not be directly accessed or manipulated once the object has been created, except those that have appropriate accessor methods described throughout in this documentation.
Each parameter in the lists below is listed along with its defaulting logic as performed by new() via initialize().
Custom field list and custom selection (record list)
- _FieldList ||= undef;
-
This is the ordered list (array reference) of columns / fields present in the object. It is set by read() to reflect the names and order of the fields encountered or actually read from in the incoming data file, if any. Initially this list is undefined, and if removed or left undefined, then the de-facto field list will be a list of all columns present in the table object, in alphabetical order (see fieldlist() and fieldlist_all()).
Normally, all fields present in the table object would be listed in the field list.
However, this parameter may be set in the object (or overridden in named-parameter function calls like read(), write(), etc.) to cause a subset of fields to be read, written or otherwise used. If a subset field list is specified before reading a data file, then ONLY fields listed will be read -- this is a way to read just certain fields from a very large file, but may not always be what you want.
Specifying the field list before calling read() is required if the data file has no header row giving names to its fields -- the names you specify in the field list, in order, will be applied to the data file being read. See the _HeaderRow parameter, below.
If a subset field list is used after columns are already loaded in memory, the columns not listed in the field list will still be present in the object (and can be listed by calling fieldlist_all()), but they will be omitted from most operations that iterate over fields.
- _Selection ||= undef;
-
This is a list of the record numbers of "selected" records in the table, possibly indicating sorted order.
If absent, then all records are considered to be selected, in "natural" order -- i.e. the order they occur in the file.
You can create and set your own selection list or get and modify an existing one. Deleting it resets the selection (for example, by calling select_all()).
Calling sort() will create a _Selection if none existed. Otherwise it operates by modifying the existing _Selection, which may be a subset of all record numbers.
Cache behavior controls
See sections related to Cacheing, below.
- _CacheOnRead = 1 unless exists
-
Boolean: whether data files read by the read() method should be cached after reading. Once cached, the data will be read from the cache instead of the original file the NEXT TIME READ() IS CALLED, but only if: 1) the cache file is found, and 2) its date is later than the original. Otherwise, the cache file is ignored or re-written. Cacheing can be up to 10x faster than parsing the file, so it's almost always worth doing in any situation where you'll be reading a data file more often than writing it.
This parameter defaults to true.
- _CacheOnWrite = 0 unless exists
-
Boolean: whether tables written by the write() method should be cached after writing. This defaults to false on the assumption that the program won't need to re-read a file it just wrote. However, this behavior would be useful if a later step in your program or another program will be reading the file that was written and would benefit from having the cacheing already done. Cacheing a file after writing is quite fast since the data is already in memory and of course it speeds up subsequent read() operations by up to 10x.
- _CacheExtension = ".cache" unless exists
-
This is the file name extension that is added to a file's name to determine the name of its corresponding cache file. (First, any existing extension, if any, is removed.) If this extension is empty, then the cache file will be named the same as the original assuming the cache file is being stored in a different directory. (See next setting.)
- _CacheSubDir = "cache" unless exists
-
This is the absolute or relative path to the subdirectory that should be used to store the cache files. The default value is the relative path to a directory called "cache". Relative paths will be appended to the directory path containing the original file being read.
Absolute cache paths (such as /tmp or c:\temp\) can also be used.
Override _CacheExtension and _CacheSubdir in a subclass, in each object, or in each call to read() or write() in order to have the cache files stored elsewhere. But remember: unless you use the same cache settings next time you read the same file, the cache files will be orphaned.
Progress routine / setting
- _Progress = undef unless exists
-
The _Progress setting controls the routing of diagnostic messages. Four possible settings are recognized:
undef (default) The class's progress settings are used. subroutine reference Your own custom progress routine. true Built-in progress_default() method used. 0/false No progress messages for this object.See the PROGRESS section for a description of the interface of custom progress routines and for details on how the builtin one works.
File format settings
- _LineEnding ||= undef;
-
_LineEnding indicates the line ending string or setting to be used to read a file, the setting that actually was used to read a file, and/or the line ending that will be used to write a file.
Set this parameter to force a particular encoding to be used.
Otherwise, leave it undef. The program will Do What You Mean.
If _LineEnding is undef when read() is called, read() will try to guess the line ending type by inspecting the first file it reads. Then it will set this setting for you. It can detect DOS, Unix, and Mac line endings.
If _LineEnding is undef when write() is called, write() will use
"\n", which yields different strings depending on the current runtime platform: \x0A on Unix; \x0D in MacPerl, \x0D\x0A on DOS.Otherwise, write() uses the value defined in _LineEnding, which would match the value filled in by read() if this object's data originally had been read from a file. So if you read a file and then later write it out, the line endings in the written file will match the format of original unless you override _LineEnding specifically.
Since Data::CTable supports reading and writing all common endings, base your decision on line ending format during write() on the needs of other programs you might be using.
For example: FileMaker Pro and Excel crash / hang if Unix line endings are used, so be sure to use the ending format that matches the needs of the other programs you plan to use.
As a convenience, you may specify and retrieve the _LineEnding setting using the mnemonic symbols "mac", "dos" and "unix." These special values are converted to the string values shown in this chart:
symbol string value chars decimal octal control ------------------------------------------------------------- dos "\x0D\x0A" CR/LF 13,10 "\015\012" ^M^J mac "\x0D" CR 13 "\015" ^M unix "\x0A" LF 10 "\012" ^JSee the section LINE ENDINGS, below, for accessor methods and conversion utilities that help you get/set this parameter in either symbolic format or string format as you prefer.
- _FDelimiter ||= undef;
-
_FDelimiter is the field delimiter between field names in the header row (if any) and also between fields in the body of the file. If undef, read() will try to guess whether it is tab
"\t"or comma <",">, and set this parameter accordingly. If there is only one field in the file, then comma is assumed by read() and will be used by write().To guess the delimiter, the program looks for the first comma or tab character in the header row (if present) or in the first record. Whichever character is found first is assumed to be the delimiter.
If you don't want the program to guess, or you have a data file format that uses a custom delimiter, specify the delimiter explicitly in the object or when calling read() or make a subclass that initializes this value differently. On write(), this will default to comma if it is empty or undef.
- _QuoteFields = undef unless exists
-
_QuoteFields controls how field values are quoted by write() when writing the table to a delimited text file.
An undef value (the default) means "auto" -- each field is checked individually and if it contains either the _FDelimiter character or a double-quote character, the field value will be surrounded by double-quotes as it is written to the file. This method is slower to write but faster to read, and may make the output easier for humans to read.
A true value means always put double-quotes around every field value. This mode is faster to write but slower to read.
A zero value means never to use double-quotes around field values and not to check for the need to use them. This method is the fastest to read and write. You may use it when you are certain that your data can't contain any special characters. However, if you're wrong, this mode will produce a corrupted file in the event that one of the fields does contain the active delimiter (such as comma or tab) or a quote.
- _HeaderRow = 1 unless exists
-
_HeaderRow is a boolean that says whether to expect a header row in data files. The default is true: a header row is required. If false, _FieldList MUST be present before calling read() or an error will be generated. In this latter case, _FieldList will be assumed to give the correct names of the fields in the file, in order, before the file is read. In other words, the object expects that either a) it can get the field names from the file's header row or b) you will supply them before read() opens the file.
Encoding of return characters within fields
- _ReturnMap = 1 unless exists
-
_ReturnMap says that returns embedded in fields should be decoded on read() and encoded again on write(). The industry-standard encoding for embedded returns is ^K (ascii 11 -- but see next setting to change it). This defaults to true but can be turned off if you want data untouched by read(). This setting has no effect on data files where no fields contain embedded returns. However, it is vital to leave this option ON when writing any data file whose fields could contain embedded returns -- if you have such data and call write() with _ReturnMap turned off, the resulting file will be an invalid Merge/CSV file and might not be re-readable.
When these fields are decoded on read(), encoded returns are converted to
"\n"in memory, whatever its interpretation may be on the current platform (\x0A on Unix or DOS; \x0D on MacPerl).IMPORTANT NOTE: When these fields are encoded by write(), any occurrence of the current _LineEnding being used to write the file is searched and encoded FIRST, and THEN, any occurrence of "\n" is also searched and encoded. For example, if using mac line endings (^M) to write a file on a Unix machine, any ^M characters in fields will be encoded, and then any "\n" (^J) characters will ALSO be encoded. This may not be what you want, so be sure you know how your data is encoded in cases where your field values might contain any ^J and/or ^M characters.
IMPORTANT NOTE: If you turn _ReturnMap off, fields with returns in them will still be double-quoted correctly. Some parsers of tab- or comma-delimited files are able to support reading such files. HOWEVER, the parser in this module's read() method DOES NOT currently support reading files in which a single field value appears to span multiple lines in the file. If you have a need to read such a file, you may need to write your own parser as a subclass of this module.
- _ReturnEncoding ||= "\x0B";
-
This is the default encoding to assume when embedding return characters within fields. The industry standard is "\x0B" (ascii 11 / octal \013 / ^K) so you should probably not ever change this setting.
When fields are encoded on write(),
"\n"is converted to this value. Note that different platforms use different ascii values for"\n", which is another good reason to leave the ReturnEncoding feature enabled when calling write().To summarize: this module likes to assume, and you should too, that returns in data files on disk are encoded as "\x0B", but once loaded into memory, they are encoded as the current platform's value of
"\n". - _MacRomanMap = undef unless exists
-
Data::CTable assumes by default that you want field data in memory to be in the ISO 8859-1 character set (the standard for Latin 1 Roman characters on Unix and Windows in the English and Western European languages -- and also the default encoding for HTML Web pages).
_MacRomanMap controls the module's optional mapping of Roman characters from Mac format on disk to ISO format in memory when reading and writing data files. These settings are recognized:
undef ## Auto: Read/write Mac chars if using Mac line endings 1 ## On: Assume Mac char set in all fields 0 ## Off: Don't do any character mapping at allThe default setting is undef, which enables "Auto" mode: files found to contain Mac line endings will be assumed to contain Mac upper-ASCII characters and will be mapped to ISO on read(); and files to be written with Mac line endings will mapped back from ISO to Mac format on write().
If your data uses any non-Latin-1 character sets, or binary data, or you really want Mac upper-ASCII characters in memory, or you just don't want this module messing with your encodings, set this option to 0 (Off) or make a subclass that always sets it to 0.
See also the clean() methods that can help you translate just the columns you want after reading a file or before writing it, which may be faster for you if only a few fields might contain high-ASCII characters.
- _FileName ||= undef;
-
This is the name of the file that should be read from or WAS read from. (read() will set _FileName to the value it used to read the file, even if _FileName was only supplied as a named parameter.)
This name will also be used, unless overridden, to re-write the file again, but with an optional extension added. (See next setting.)
- _WriteExtension = ".out" unless exists
-
The _WriteExtension is provided so that CTable won't overwrite your input data file unless you tell it to.
_WriteExtension will be added to the object's _FileName setting to create a new, related file name, before writing.... UNLESS _FileName is supplied as an direct or named parameter when calling write().
In the latter case, write() uses the file name you supply and adds no extension, even if this would mean overwriting the original data file.
To add _WriteExtension, write() places it prior to any existing final extension in the _FileName:
_FileName default file name used by write() -------------------------------------------------------------- People.merge.txt People.merge.out.txt People People.outIf you want to always overwrite the original file without having to supply _FileName each time, simply set _WriteExtension to undef in a subclass or in each instance.
If _CacheOnWrite is true, then the _WriteExtension logic is applied first to arrive at the actual name of the file to be written, and then the _CacheExtension logic is applied to that name to arrive at the name of the cache file to be written.
Sorting-related parameters
- _SortOrder ||= undef;
-
_SortOrder is the list of fields which should be used as primary, secondary, etc. sort keys when sort() is called. Like other parameters, it may be initialized by a subclass, stored in the object, or provided as a named parameter on each call to sort().
If _SortOrder is empty or undefined, then sort() sorts the records by record number (i.e. they are returned to their "natural" order).
- _SortSpecs ||= {};
-
_SortSpecs is a hash of specifications for the SortType and SortDirection of fields on which sorting may be done. For any field missing a sort spec or the SortType or SortDirection components of its sort spec, the _DefaultSortType and _DefaultSortDirection settings will be used. So, for example, if all fields are of type String and you want them to sort Ascending, then you don't need to worry about _SortSpecs. You only need to provide specs for fields that don't take the default settings.
_SortSpecs might look like this:
{Age => {SortType => 'Integer'}, NameKey => {SortType => 'Text', SortDirection => -1}} - _SRoutines ||= {};
-
_SRoutines is a hash mapping any new SortTypes invented by you to your custom subroutines for sorting that type of data. (See the section on sort routines, below, for a full discussion.)
Sorting defaults
- _DefaultSortType ||= 'String';
-
If you sort using a field with no sort spec supplied, or whose sort spec omits the SortType, it will get its SortType from this parameter.
See the sections below on SORT TYPES and SORT ROUTINES.
- _DefaultSortDirection ||= 1;
-
If you sort using a field with no sort spec supplied, or whose sort spec omits the SortDirection, it will get its SortDirection from this parameter.
Legal sort directions are: 1 (Ascending) or -1 (Descending).
See the section below on DEFAULT SORT DIRECTION.
Miscellaneous parameters
- _ErrorMsg ||= "";
-
This parameter is set by read() or write() methods that encounter an error (usually a parameter error or file-system error) that prevents them from completing. If those methods or any methods that call them return a false value indicating failure, then _ErrorMsg will contain a string explaining the problem. The message will also have been passed to the progress() method for possible console feedback.
- _Subset
-
This parameter is set to 1 (true) by read() if the last call to read() brought in a subset of the fields available in the file; 0 otherwise.
The object uses this field internally so it knows to abandon any cache files that might not contain all requested fields upon read().
SUBCLASSING
Most subclasses will override initialize() to set default values for the parameters of the parent class and then they may provide default values for other subclass-specific parameters. Then, the subclass's initialize() should call SUPER::initialize() to let the parent class(es) take care of the remaining ones.
Every initialize() method should always allow for parameters to have already been provided by the $Params hash or template object. It should not overwrite any valid values that already exist.
The following sample subclass changes the default setting of the _Progress parameter from undef to 1 and then overrides the progress_default() method to log all progress messages into a new "_ProgrLog" (progress log) parameter stored in the object.
BEGIN
{ ## Data::CTable::ProgressLogger: store messages in the object
package Data::CTable::ProgressLogger;
use vars qw(@ISA); @ISA=qw(Data::CTable);
sub initialize ## Add a new param; change one default
{
my $this = shift;
$this->{_Progress} = 1 unless exists($this->{_Progress});
$this->{_ProgrLog} ||= [];
$this->SUPER::initialize();
}
sub progress_default ## Log message to object's ProgMsgs list
{
my $this = shift;
my ($msg) = @_;
chomp $msg;
push @{$this->{_ProgrLog}}, localtime() . " $msg";
return(1);
}
sub show_log ## Use Dumper to spit out the log list
{
my $this = shift;
$this->dump($this->{_ProgrLog});
}
}
## Later...
my $Table = Data::CTable::ProgressLogger->new("mydata.txt");
# ... do stuff...
$Table->write();
$Table->show_log();FIELD LIST
## Getting / setting the object's _FieldList
$t->fieldlist() ## Get _FieldList or fieldlist_all()
$t->fieldlist_get()
$t->fieldlist_hash() ## Get fieldlist() as keys in a hash
$t->fieldlist_all() ## Get all fields (ignore _FieldList)
$t->fieldlist($MyList) ## Set field list (_FieldList param)
$t->fieldlist_set($MyList)
$t->fieldlist(0) ## Remove field list (use default)
$t->fieldlist_set()
$t->fieldlist_force($MyList)## Set list; remove non-matching cols
$t->fieldlist_truncate() ## Just remove nonmatching cols
$t->fieldlist_default() ## Default field list (alpha-sorted)
$t->fieldlist_add($MyName) ## Append new name to custom list.
$t->fieldlist_delete($MyName) ## Delete name from custom list.A CTable object can optionally have a custom field list. The custom field list can store both the ORDER of the fields (which otherwise would be unordered since they are stored as keys in a hash), and also can be a subset of the fields actually in the object, allowing you to temporarily ignore certain effectively-hidden fields for the benefit of certain operations. The custom field list can be changed or removed at any time.
The custom field list is stored in the private _FieldList parameter.
fieldlist() always returns a list (reference). The list is either the same list as _FieldList, if present, or it is the result of calling fieldlist_default(). In CTable, fieldlist_default() in turn calls fieldlist_all() -- hence fieldlist() would yield an auto-generated list of all fields in alphabetical order.
fieldlist_all() can be called directly to get a list of all fields present regardless of the presence of a _FieldList parameter. The list is an alphabetical case-insensitively sorted list of all hash keys whose names do not begin with an underscore.
You could override this method if you want a different behavior. Or, you could create your own custom field list by calling fieldlist_all() and removing fields or ordering them differently.
To set a custom field list (in _FieldList), call fieldlist() or fieldlist_set() with a list (reference). The list must be a list of strings (field names) that do not begin with underscore. The object owns the list you supply.
To remove a custom field list (and let the default be used), call fieldlist(0) or fieldlist_set() with no arguments (these will return the fieldlist that was deleted, if any).
fieldlist_freeze() "freezes" the fieldlist in its current state. This is equivalent to the following:
$t->fieldlist_set($t->fieldlist());... which would force the fieldlist to $t->fieldlist_all() if and only if there is not already a custom _FieldList present.
IMPORTANT NOTE ABOUT PARTIAL FIELD LISTS: When setting a field list, the object ensures that all fields (columns) mentioned in the list are present in the object -- it creates empty columns of the correct length as necessary. However, it does NOT delete any fields not mentioned in the field list. This allows you to manipulate the field list in order to have certain fields be temporarily ignored by all other methods, then alter, restore, or remove it (allow it to revert to default) and they will be effectively unhidden again. Some methods (such as cols(), write(), etc.) also allow you to specify a custom field list that will override any other list just during the execution of that method call but will not modify the object itself.
Call fieldlist_force() to set the list AND have any non-listed fields also deleted at the same time (by calling fieldlist_truncate() internally). You can also just delete individual columns one-by-one, of course, using the column-manipulation methods and the custom fieldlist, if any, will be appropriately updated for you.
fieldlist_truncate() deletes any fields found in the table but not currently present in _FieldList. A hash of the deleted columns is returned to the caller. If there is no _FieldList, then this method does nothing.
fieldlist_default() just calls fieldlist_all() in this implementation, but could be changed in subclasses.
fieldlist_add() is the internal method that adds a new field name to the custom field list (if present) and if the field name was not already on the list. It is called by other methods any time a new column is added to the table. Don't call it directly unless you know what you're doing because the corresponding column won't be created. (Instead, use col().) The field name is appended to the end of the existing custom field list. If there is no custom field list, nothing is done.
fieldlist_delete() is the internal method that deletes a field name from the custom field list (if present). It is called by other methods when columns are deleted, but it does not actually delete the columns themselves, so use with caution: deleting a field from the custom field list effectively hides the field. This method has no effect, however, if there is no custom field list present. So don't call this method directly unless you know what you're doing.
DATA COLUMNS (FIELD DATA)
## Getting or setting data in entire columns
$t->{$ColName} ## Get a column you know exists
$t->col($ColName) ## Get a column or make empty one.
$t->col_get($ColName)
$t->col($ColName, $ListRef) ## Set all of a column all at once.
$t->col_set($ColName, $ListRef)
$t->col_force($ColName, $ListRef) ## Add but don't check size or
## add to custom field list
$t->col_set($ColName, undef) ## Delete a column completely
$t->col_delete($ColName)
$t->col_empty() ## An empty col presized for table
$t->col_empty(22) ## An empty col of another length
$t->col_empty($Col) ## An empty col sized to match another
$t->col_default() ## Default if req. column not found.
$t->col_exists($Field) ## Check existence of column
$t->col_active($Field) ## Restrict check to fieldlist()
$t->cols($ColList) ## Get list of multiple named columns
$t->cols_hash($ColList) ## Get hash " " "
$t->col_rename($Old => $New) ## Change name of columns
$t->col_rename($Old1 => $New1, $Old2 => $New2) ## Change severalA "column" is a field in the table and all its data. The column's field name is a key in the object itself, and may also optionally be listed in a custom field list if present. The column's data is the key's value in the hash and is an array ref of values presumed to be of the same data type (e. g. string, integer, etc.)
Sometimes the terms "column" and "field" are used interchangeably in this documentation.
If you already know that a column exists (because you got it from the fieldlist() method and you've not previously manipulated _FieldList directly but instead carefully used the method calls available for that), then you can safely get the column by just looking it up in the object itself.
The col() method does the same thing, but forces the column to spring into existence if it did not already (which can also have the potentially unwanted side-effect of hiding coding errors in which you retreive mis-named columns: so beware). Columns brought into existence this way will automatically be pre-sized (i.e. they will will be created and set to whatever col_default() returns).
The col() or col_set() methods can also be used to set a column. When the column is set, the list you pass is automatically sized (lengthened or truncated) to match the current length of the table. If this is not what you want, then call col_force() which will not check whether the new column matches the size of the others.
No matter how you set it, the object now "owns" the list you gave it.
As a convenience, col(), col_set() and col_force() return the column that was set. They silently discard any previous column.
All three methods of column setting will append the column to the custom field list if one is present and the column name is not already listed there (by calling fieldlist_add()). They will also call the extend() method to ensure all columns have the same length (either others will be extended to match the length of the new one, or the new one will be extended to match the length of the others).
col_delete() deletes a column.
col_empty() returns an anonymous list reference that is pre-sized to the length of the table (by default). You could use it to get an empty column that you intend to fill up and then later insert into the table or use to hold the results of an operation on other columns. If you want a different length, specify it as a number or as an array ref whose length should be matched.
col_default() is the internal method that implements the "springing into existence" of missing columns. Currently it just calls col_empty(). Other subclasses might want to have it return undef or a string like "NO_SUCH_COLUMN" in order to help track programming errors where nonexistent columns are requested.
cols($FieldList) returns an ordered list of the requested column names. If no list is given, then fieldlist() is used.
cols_hash($FieldList) does the same as cols(), but the result is a hash whose keys are the field names and whose values are the columns -- much like the original object itself, but not blessed into the class. The resulting hash, however, could be used as the prototype for a new Data::CTable object (by calling the new() method). However, be warned that both objects will think they "own" the resulting shared so be careful what you do..... which brings us to this:
IMPORTANT NOTE ABOUT GETTING COLUMNS: The columns you retrieve from a table are still "owned" by the table object as long as it lives. If you modify them, you are modifying the table's data. If you change their length, then you may be invalidating the table's own expectations that all its columns have the same length. So beware.
Just make yourself a copy of the data if that isn't what you want. For example, instead of this:
my $Foo = $Table->col('Foo'); ## Reference to actual columnDo this:
my $Foo = [@{$Table->col('Foo')}]; ## Shallow copy of the columnCLEANUP AND VALIDATION
## Performing your own cleanups or validations
$t->clean($Sub) ## Clean with custom subroutine
$t->clean($Sub, $Fields) ## Clean specified columns only
## Cleaning whitespace
$t->clean_ws() ## Clean whitespace in fieldlist() cols
$t->clean_ws($Fields) ## Clean whitespace in specified cols
## Cleaning methods that map character sets
$t->clean_mac_to_iso8859()
$t->clean_mac_to_iso8859($Fields)
$t->clean_iso8859_to_mac()
$t->clean_iso8859_to_mac($Fields)
## Character mapping utilities (not methods)
use Data::CTable qw(
ISORoman8859_1ToMacRoman
MacRomanToISORoman8859_1
ISORoman8859_1ToMacRoman_clean
MacRomanToISORoman8859_1_clean
);
&ISORoman8859_1ToMacRoman(\ $Str) ## Pass pointer to buffer
&MacRomanToISORoman8859_1(\ $Str) ## Pass pointer to buffer
&ISORoman8859_1ToMacRoman_clean() ## Operates on $_
&MacRomanToISORoman8859_1_clean() ## Operates on $_One of the most important things you can do with your data once it's been placed in a table in Perl is to use the power of Perl to scrub it like crazy.
The built-in clean_ws() method applies a standard white-space cleanup to selected records in every field in the fieldlist() or other list of fields you optionally supply (such as fieldlist_all()).
It does the following cleanups that are deemed correct for the majority of data out there:
- Remove all leading whitespace, including returns (\n)
- Remove all trailing whitespace, including returns (\n)
- Convert runs of spaces to a single space
- Convert empty string values back to undef to save spaceOf course, depending on your data, clean_ws() might just be the first thing you do in your cleanup pass. There might be many more cleanups you'd like to apply.
clean() is like clean_ws() except you supply as the first argument your own cleaning subroutine (code reference). It should do its work by modifying $_.
Both clean_ws() and clean() apply cleaning ONLY to selected records. If this isn't what you want, then select_all() before cleaning.
Since a cleanup subroutine can do ANY modifications to a field that it likes, you can imagine some cleanup routines that also supply default values and do other validations.
For example, a cleanup routine could convert every value in each field to an integer, or apply minimum or maximum numerical limits:
sub {$_ = int($_) }
sub {$_ = max(int($_), 0) }
sub {$_ = min(int($_), 200)}Or your cleanup routine could use regular expressions to do capitalizations or other regularizations of data:
sub Capitalize {/\b([a-z])([a-z]+)\b)/\U$1\E$2/g}
$t->clean(\ &Capitalize , ['FirstName', 'LastName']);
$t->clean(\ &PhoneFormat, ['Phone', 'Fax' ]);
$t->clean(\ &LegalUSZip, ['HomeZip', 'WorkZip' ]);... and so on. Cleanups are easy to write and quick and easy to apply with Data::CTable. Do them early! Do them often!
Hints for writing cleanup routines
If your cleanup routine may be used to clean up fields that could be empty/undef and empty/undef is a legal value, it should not touch any undef values (unintentionally converting them to strings).
Finally, instead of setting any values to the empty string, it should set them to undef instead. This includes any values it might have left empty during cleanup. (Using undef instead of empty string to represent empty values is one way that Data::CTable likes to save memory in tables that may have lots of those.)
For an example of a well-behaved cleanup routine, consider the following implementation of the builtin CleanWhitespace behavior:
sub CleanWhitespace
{
return unless defined; ## Empty/undef values stay that way
s/ \s+$//sx; ## Remove trailing whitespace
s/^\s+ //sx; ## Remove leading whitespace
s/ +/ /g; ## Runs of spaces to single space
$_ = undef unless length; ## (Newly?) empty strings to undef
}Roman character set mapping
The character set mapping cleanup routines can be used to convert upper-ASCII characters bidirectionally between two popular Roman Character sets -- Mac Roman 1 and ISO 8859-1 (also sometimes called ISO Latin 1) -- i.e. the Western European Roman character sets.
By default, read() converts all incoming data fields in data files with Mac line endings to ISO format when reading in. Conversely, write() does the reverse mapping (ISO to Mac) when writing a file with Mac line endings.
However, you may wish to turn off these default behaviors and instead apply the mappings manually, possibly just to certain fields.
For example, if a table contains fields with non-Roman character sets, you would definitely not want to apply these mappings, and instead might want to apply some different ones that you create yourself.
Utility routines for character mapping
This module can optionally export four utility routines for mapping character Latin 1 character sets. Always be sure to map the correct direction -- otherwise you'll end up with garbage! Be careful to only pass Western Roman strings -- not double-byte strings or strings encoded in any single-byte Eastern European Roman or non-Roman character set.
&ISORoman8859_1ToMacRoman(\ $Str) ## Pass pointer to buffer
&MacRomanToISORoman8859_1(\ $Str) ## Pass pointer to bufferThese routines translate characters whose values are 128-255 from one Western Roman encoding to another. The argument is a string buffer of any size passed by reference.
The functions return a count of the number of characters that were mapped (zero or undef if none were).
&ISORoman8859_1ToMacRoman_clean() ## Operates on $_
&MacRomanToISORoman8859_1_clean() ## Operates on $_These routines are variants of the above, but they're versions that are compatible with clean() -- they operate on $_ and will take care to leave undefined values undefined. They do not have return values.
More advanced cleaning and validation
Unfortunately, clean() only lets you operate on a single field value at a time -- and there's no way to know the record number or other useful information inside the cleaning routine.
For really powerful cleaning and validation involving access to all fields of a record as well as record numbers, see the discussion of the calc() method and other methods for doing complex field calculations in the next section.
CALCULATIONS USING calc()
## Calculate a new field's values based on two others
$t->calc($Sub) ## Run $Sub for each row, with
## fields bound to local vars
$t->calc($Sub, $Sel) ## Use these row nums
$t->calc($Sub, undef, $Fields) ## Use only these fields
$t->calc($Sub, $Sel, $Fields) ## Use custom rows, fields
my $Col = $t->calc($Sub) ## Gather return vals in vector
## Example 1: Overwrite values in an existing column.
$t->calc(sub{no strict 'vars'; $Size = (stat($Path))[7]});
## Example 2: Create empty column; fill fields 1 by 1
$t->col('PersonID');
$t->calc(sub{no strict 'vars'; $PersonID = "$Last$First"});
## Example 3: Calculate values; put into to table if desired
$PersonID = $t->calc(sub{no strict 'vars'; "$Last$First"});
$t->sel('PersonID', $PersonID);
## Example 4: Using fully-qualified variable names
$t->calc(sub{$main::PersonID = "$main::Last$main::First"});calc() runs your custom calculation subroutine $Sub once for every row in the current selection() or other list of row numbers that you specify in the optional $Sel argument.
This lets you apply a complex calculation to every record in a table in a single statement, storing the results in one or more columns, or retrieving them as a list.
Your custom subroutine may refer to the value in any field in the current row by using a global variable with the field's name:
For example, if the table has fields First, Last, and Age, then $Sub may use, modify, and set the variables $First, $Last, $Age. (Also known as $main::First, $main::Last, $main::Age).
Modifying any of these specially-bound variables actually modifies the data in the correct record and field within the table.
By default, the fields available to $Sub are all fields in the table. calc() must bind all the field names for you for each row, which can be time-consuming for tables with very large numbers of fields.
You can speed up the operation of calc() by listing only the fields your $Sub needs in the optional parameter $Fields. Any field names you don't mention won't be available to $Sub. Conversely, calc() will run faster because it can bind only the fields you actually need.
If you include non-existent fields in your custom $Fields list, calc() creates them for you before $Sub runs the first time. Then your $Sub can store field values into the new column, referring to it by name.
Variables in $Sub are in the "main" package. So you should set $Sub to use pacakge "main" in case the rest of your code is not in "main".
Similarly, if you "use strict", Perl will complain about global variables in $Sub. So you may need to assert "no strict 'vars'".
{ package Foo; use strict;
$t = ...;
$t->calc(sub {package main; no strict 'vars';
$Age = int($Age)});
}
## Or this:
{ package Foo; use strict;
$t = ...;
{ package main; no strict 'vars';
my $Sub = sub {$Age = int($Age)};
}
$t->calc($Sub);
}You may be able to get around both problems more easily by prefixing each variable reference in $Sub with "main::". This takes care of the package name issue and bypasses "use strict" at the same time, at the slight cost of making the calculation itself a bit harder to read.
$t->calc(sub {$main::Age = int($main::Age)}); ## OK in any packageIn addition to the field names, the following three values are defined during each invocation of $Sub:
$_r ($main::_r) -- the row number in the entire table
$_s ($main::_s) -- the item number in selection or $Recs
$_t ($main::_t) -- the table object itselfYou could use these values to print diagnostic information or to access any of the data, parameters, or methods of the table itself from within $Sub. Or you could even calculate field values using $_r or $_s.
For example, after searching & sorting, you could make a field which preserves the resulting sort order for future reference:
$t->col('Ranking'); ## Create the empty column first.
$t->calc(sub{$main::Ranking = $main::_s});This last example is equivalent to:
$t->sel(Ranking => [0 .. $#{$t->selection()}]); ## See sel() below"MANUAL" CALCULATIONS
calc() (see previous secion) is the briefer, more elegant way to do batch calculations on entire columns in a table, but it can be slightly slower than doing the calculations yourself.
If you have extremely large tables, and you notice the processing time for your calculations taking more than a second, you might want to rewrite your calculations to use the more efficient techniques shown here.
You will often need to create new calculated columns based on one or more existing ones, and then either insert the columns back in the tables or use them for further calculations or indexing.
Examples 1a and 1b create a new field 'NameOK' containing either the string "OK" or undef (empty) depending on whether the field 'Name' is empty. Just use map() to iterate over the existing values in the other column, binding $_ to each value in turn.
### Example 1a: Calculation based on one other column
## Retrieve column
my $Name = $t->col('Name');
## Make new column
my $NameOK = [map {!!length && 'OK'} @$Name];
## Insert column back into table:
$t->col(NameOK => $NameOK);
### Example 1b: Same calculation, in a single statement:
$t->col(NameOK => [map {!!length && 'OK'} @{$t->col('Name')}]);In order to iterate over MULTIPLE columns at once, you need a list of the row numbers generated by $t->all() so you can index the two columns in tandem. Then, you use map to bind $_ to each row number, and then use the expression $t->col($ColName)->[$_] to retreive each value.
Examples 2a and 2b demonstrate this method. They create a new field 'FullName' which is a string joining the values in the 'First' and 'Last' columns with a space between.
### Example 2a: Calculation based on multiple columns
## Retrieve columns
my $First = $t->col('First');
my $Last = $t->col('Last' );
## Retreive row nums
my $Nums = $t->all();
## Calculate a new column based on two others
my $Full = [map {"$First->[$_] $Last->[$_]"} @$Nums];
## Add new column to the table
$t->col(FullName => $Full);
### Example 2b: Same calculation, in a single statement:
$t->col(FullName =>
[map {"$t->col('First')->[$_] t->col('Last')->[$_]"}
@{$t->all()}]);In examples 1 and 2, you create entirely new columns and then add or replace them in the table.
Using the approach in Examples 3a and 3b, you can assign calculated results directly into each value of an existing column as you go.
## Example 3a: Calculate by assigning directly into fields...
my $A = $t->col->('A'); ## This column will be modified
my $B = $t->col->('B');
my $C = $t->col->('C');
foreach @($t->all()) {$A->[$_] = $B->[$_] + $C->[$_];}
## Example 3b: Same calculation, in a single statement:
foreach @($t->all()) {($t->col('A')->[$_] =
$t->col('B')->[$_] +
$t->col('C')->[$_])};Before writing your code, think about which calculation paradigms best suit your needs and your data set.
Just as Perl Hackers know: There's More Than One Way To Do It!
INDEXES
## Make indexes of columns or just selected data in columns
my $Index1 = $t->index_all($Key); ## entire column
my $Index2 = $t->index_sel($Key); ## selected data only
## Make hashes of 2 columns or just selected data in columns
my $Index1 = $t->hash_all($KeyFld, $ValFld); ## entire column
my $Index2 = $t->hash_sel($KeyFld, $ValFld); ## selected dataindex_all() creates and returns a hash (reference) that maps keys found in the column called $Key to corresponding record numbers.
Ideally, values in $Key would be unique (that's up to you).
If any values in $Key are NOT unique, then later values (higher record numbers) will be ignored.
index_sel() creates and returns a hash (ref) that maps keys found in the SELECTED RECORDS of column $Key to corresponding record numbers.
Any keys in unselected records are ignored. Otherwise, the behavior is equivalent to index_all().
hash_all() and hash_sel() are similar, except they create and return hashes whose keys are taken from column $KeyFld, but whose values are from $ValFld in the corresponding records.
So, for example, imagine you have a tab-delimited file on disk with just a single tab per line (2 fields) and no header row. The entries on the left side of the tab on each line are keys and the right side are values. You could convert that file into a hash in memory like this:
my $t = Data::CTable->new({_HeaderRow=>0, _FieldList=>[qw(F1 F2)]},
"DeptLookup.txt");
my $DeptLookup = $t->hash_all(qw(F1 F2));Reverse indexes
If you'd like an index mapping record number to key, just get $t->col($Key). That's what the data columns in Data::CTable are.
DATA ROWS (RECORDS)
## Getting or setting rows / records
$t->row($Num) ## Get a row or make empty one.
$t->row_get($Num)
$t->row($Num, $HashRef) ## Set all of a row all at once.
$t->row_set($Num, $HashRef)
$t->row_set($Num, undef) ## Delete a row completely
$t->row_delete($Num)
$t->row_delete($Beg, $End)## Delete a range of rows
$t->row_move($Old, $New) ## Move a row to before $New
$t->row_empty() ## An empty hash
$t->row_exists($Num) ## True if $Num < $t->length()
$t->rows($RowList) ## Get list of multiple row nums
$t->row_list($Num) ## Get row vals as a list
$t->row_list($Num, $Fields) ## Get row vals: specified fields
$t->row_list_set($Num, undef, $Vals) ## Set row vals as a list
$t->row_list_set($Num, $Fields, $Vals) ## Set row vals as a list
$t->row_list_set($Num, $Fields) ## Set vals to empty/undefUsually, when working with Data::CTable objects, you are operating on entire columns or tables at a time (after all: any transformation you do on one record you almost always want to do on all records or all selected ones).
You should very rarely need to access data by retrieving rows or setting rows, moving them around individually, and so on. (It's much cleaner, and much more efficient to manipulate the selection() (list of selected row numbers) instead -- just delete a row number from the selection, for example, and then for most operations it's almost as if the row is gone from the table, except the data is really still there.)
However, if on rare occasions you really do need direct row operations, you're reading the right section.
A row is generally accessed as a hash. The hash you provide or get back is not saved by the object in any way. Data values are always copied in or out of it, so you always "own" the hash.
Rows are specified by $Num -- the row number with in the unsorted columns (the raw data in the table). These numbers are just array indices into the data columns, and so their legal range is:
[0 .. ($t->length() - 1)] ## (Zero-based row numbering.)The row hash (or "record") has keys that are field names and values that are copies of the scalar values stored in the data columns within the table.
row() always copies only the fields in fieldlist(), except for row_list() which allows you to specify an optional $Fields parameter which can override the current fieldlist().
If the fieldlist happens to be a subset of all fields, but you really want to get all fields in your record, then call fieldlist_set(0) first to permanently or temporarily delete it.
row() and row_get() always return a hash.
row($Num, $Hash), row_set() take a hash and set just the fields you specify in the hash (in the given row of course). Any non-existent field names in the hash are created, so be careful.
In fact, in general with either getting or setting rows, any non-existent fields mentioned will be created for you (by internally calling col()). So you could build a whole table of 100 rows by starting with an empty, new, table and setting row 99 from a hash that gives the field names.
Setting a row number higher than any existing row number with row(), row_set() or row_force() will automatically set the new length of the entire table to match (extending all the columns with empty rows as necessary).
IMPORTANT: IF YOU SIMPLY MUST ADD ROWS SEQUENTIALLY, do not let the table auto-extend by one with each row you set. This is slow and gets rapidly slower if there's lots of data because the arrays holding the data columns will keep getting reallocated on every insert. Instead, first pre-extend the table to your highest row number by calling length($Len), and then set your rows. Or easier: if convenient just set your rows starting with the highest-numbered one first. If you don't know how many you'll have, guess or estimate and pre-extend to the estimated number and then cut back later. This will be faster than extending all columns by one each time.
row_delete() removes a row or range of rows completely from the table. Any rows above the deleted ones will move down and the table's length() will decrease. If the data columns are very large, this could be a bit slow because a lot of data could be moved around. The low and high row numbers will be limited for you to 0 and length() - 1, respectively. Null ranges are OK and are silently ignored. The range is inclusive, so to delete just row 99, call row_delete(99) or row_delete(99,99).
EFFICIENCY NOTE: Don't call row_delete() to remove lots of individual rows. Instead, select those row numbers by setting the selection (if not already selected), and then invert the selection using selection_invert(), so the undesired rows are deselected, and then use the cull() method to rewrite the entire table at once. The deselected rows will be omitted very efficiently this way.
row_move() moves a row from its $Old row number to the position before the row currently in row $New (specify $New = length() to move the row to the end). Again, in shuffling data in columns, lots of data could get moved around by this operation, so expect it to be slow. If as with row_delete(), if you will be doing several moves, consider building an appropriate selection() first, and then using cull() instead.
Using row_delete() and row_move() to shift records around changes the record numbers of the affected records and many others in the table. The record numbers in the custom selection, if any, are updated to reflect these changes, so the records that were selected before will still be selected after the move (except those that were deleted of course). If you had a private copy of the selection, your copy will likely become outdated after these operations. You should get it again by calling selection().
row_empty() returns a hash whose keys are the entries in fieldlist() and whose values are undef. (You could use it to fill in values before calling row_set()). Note: in this class, row_empty() does exactly the same thing as fieldlist_hash() when the latter is called with no arguments.
row_exists() returns true if (($Num >= 0) && ($Num < $t->length())).
rows() calls row() for each row num in a list and returns a list of the resulting hashes.
row_list() gets row values as a list instead of a hash. They appear in the order specified in fieldlist() unless you supply an optional $Fields parameter listing the fields you want to get.
row_list_set() sets row values as a list instead of a hash. Pass your own $Fields list or undef and fieldlist() will be used. $Values should be a list with the same number of values as fields expected; any shortage will result in undef/empty values being set.
ROW / RECORD COUNT (TABLE LENGTH)
## Getting or setting table length
$t->length() ## Get length
$t->length_get()
$t->length(22) ## Set length (truncate or pre-extend)
$t->length_set(22)
$t->extend() ## Set length of all columns to match longestThe length* methods assume the table already has columns of equal length. So the length of the table is the length of any field taken at random. We choose the first one in the field list.
Setting the length will truncate or pre-extend every column in the table to a given length as required.
(Pre-extending means setting each column's length via $# so that it has the correct number of entries already allocated (and filled with undef) so that operations that fill up the table can be done much more quickly than with push().
However, if a new column has been added directly, or a table has been constructed out of columns whose length may not initially match, the extend() method may be (should be) called to inspect all columns and extend them all to match the longest one. Note that extend() operates on all fields in the object, ignoring the custom _FieldList if any.
The length of a table with no columns is zero.
SELECTIONS
## Getting or setting the custom selection list itself (_Selection)
$t->selection() ## Get sel if any; else all()
$t->selection_get()
$t->selection($List) ## Set sel (list of rec nums)
$t->selection_set($List)
$t->selection(0) ## Remove sel (select all)
$t->selection_set(undef)
$t->selection_delete()
$t->select_all()
$t->selection_inverse() ## Get inverse copy of selection
$t->select_inverse() ## Invert the selection
$t->selection_validate() ## Remove invalid #s from sel
## List of all rec nums present (regardless of selection)
$t->all()
## Getting or setting just selected fields in columns
## (as contrasted with col() and friends).
$t->sel($ColName) ## Get col but only records in sel
$t->sel_get($ColName)
$t->sel($ColName, $ListRef) ## Set selected fields in col...
$t->sel_set($ColName, $ListRef) ##... in selection order
$t->sel_set($ColName) ## Set selected fields to undef
$t->sel_clear($ColName)
$t->sels($ColList) ## Like cols, but selected fields
$t->sels_hash($ColList) ## " " cols_hash()... " " " "
## Finding out size of selection (number of rows)
$t->sel_len() ## Get number of selected rows.A selection is an ordered list of record numbers. The record numbers in the selection may be a subset of available records. Furthermore, they may be in non-record-number order, indicating that the records have been sorted.
Record numbers are numeric array indices into the columns in the table. It is an error for any selection list to contain an index less than zero or greater than (length() - 1), so if you set a selection explicitly, be careful.
Any selection list you get or set belongs to the object. Be careful of modifying its contents.
The custom selection, if any, is stored internally in the _Selection parameter. If this parameter is absent, the selection defaults to all() -- i.e. a list of all record numbers, in order: [0..($this->length() - 1)] (which becomes [] if length() is 0).
REMEMBER: length() is one-based, but record numbers are zero-based.
Removing the selection (that is, removing the LIST itself of which records are selected), is the same as selecting all records. consequently, selection(0), selection_delete(), and select_all() are all synonymous.
selection_validate() removes any entries from the current _Selection list (if any) that are not valid record numbers -- i.e. it removes any record whose integer value is < 0 or greater than length() - 1. This routine is mainly used by other methods that might delete records, such as length_set().
Getting or setting just selected data from columns
Sometimes, you don't want to get/set entire columns, you instead want to get or set data in just the selected fields in a column.
The sel(), sel_get(), sel_set(), sels() and sels_hash() methods are analagous to the corresponding col(), ... cols_hash() methods except in these two ways:
- the 'sels' variants get or set just selected data, as determined by the current selection(), which gives an ordered list of the selected / sorted records.
- the 'sels' variants all make COPIES of the data you request or supply -- the data is copied out of or into the correspnding column. So, you "own" any vector you pass or receive in reply.
So, for example, imagine you have just set selection() to only list record numbers where the LastName field is not empty. Then you have called sort() to sort those record numbers by the LastName field. You could then call $t->sel('LastName') to get a sorted list of all non-empty last names.
It might be helpful to think of "sel" as short for "selected". So $t->sel('LastName') would mean "get the selected field values from the LastName field".
SEARCHING / SELECTING RECORDS
## Modifying the table's custom selection (_Selection)
$t->select_all() ## Set _Selection = $t->all() or undef
$t->select_none() ## Set _Selection = []
$t->select_inverse() ## Invert the curr. sel. (and get it)
## Specific searches: "the select() methods"
$t->select($Field1=>$Sub1, ## Del nonmatching recs from sel.
$Field2=>$Sub2, ## i.e. narrow sel. to match
...);
$t->omit ($Field1=>$Sub1, ## Del matching recs from sel.
$Field2=>$Sub2,
...);
$t->add ($Field1=>$Sub1, ## Add matching recs to sel.
$Field2=>$Sub2,
...);
$t->but ($Field1=>$Sub1, ## Add nonmatching recs to sel.
$Field2=>$Sub2,
...);
## Getting useful lists of record numbers...
$t->all() ## Get "full" sel. (all record #s)
$t->selection() ## Get current selection
$t->selection_inverse() ## Get inverse copy of curr. sel.
## Example 1: Refine a selection by narrowing down...
$t->select_all()
$t->select(Field1 => sub {$_});
$t->select(Field2 => sub {$_});
$t->select(Field3 => sub {$_});
## Example 2: Manually refine and set the selection...
$Sel = [grep {$t->col($Field1)->[$_]} @{$t->all ()}];
$Sel = [grep {$t->col($Field2)->[$_]} @$Sel];
$Sel = [grep {$t->col($Field3)->[$_]} @$Sel];
$t->selection($Sel); ## Set the selection when done.
## Example 3: Complex manual search using calculated value
my $A = $t->col('A');
my $B = $t->col('B');
my $S = [grep
{my $X = $A->[$_] + $B->[$_]; ($X > 100 && $X < 200);}
@{$t->all()}]; ## Or could start with $t->selection().
$t->selection($S); ## Set the selection when done.
## Example 4: Refine a selection by building up...
$t->select_none()
$t->add(Field1 => sub {$_});
$t->add(Field2 => sub {$_});
$t->add(Field3 => sub {$_});
## Example 5: Combining the select() methods to build a query...
$t->select_all()
$t->select(Status => sub {/prime/i });
$t->omit (DueDate => sub {$_ > $Today});
$t->add (Force => sub {$_ });select() and its friends omit(), add(), and but(), known collectively as "the select() methods," all work similarly: they take a series of one or more pairs indicating matches to be done, where each match is specified as (FieldName => Subroutine).
In addition to the field names already present in the table, the FieldName in any Spec may also be one of these two special pseudo-fields:
- _RecNum
-
the record number of the record being compared
- _SelNum
-
the numerical position of the record being compared within the previous selection (only usable with select() and omit() since add() and but() by definition operate on non-selected records).
For example:
## Match 2nd 100 rec numbers
$t->select(_RecNum => sub {$_ >= 100 && $_ <= 199});
## Match 2nd 100 currently selected/sorted items
$t->select(_SelNum => sub {$_ >= 100 && $_ <= 199});Be careful when using _SelNum in a search. In the above _SelNum search example, since the selection itself will be modified by select(), the items that were formerly selection items 100 - 199 will now be _SelNum 0 - 99 in the new selection.
The Subroutine is an anonymous grep-style predicate that operates on $_ and should return true/false to indicate a match with an element of the field FieldName.
The order of multiple matches in a single method call is significant only in that the searches can be faster if the field that will match the fewest records is listed first.
A given FieldName may be listed in the specs more than once if it has multiple search criteria that you prefer to execute as multiple subroutines (though it would be more efficient on very large tables to combine their logic into one subroutine joined with "&&").
Each field match will be applied (with an implied AND joining them) to determine whether the record itself matches. Then, based on whether the record itself matches, it will either be added or deleted from the selection based on which method is being called:
method... operates on... action....
------------------------------------------------------------------
select() selected records Keep only recs that DO match
omit() selected records Keep only recs that DO NOT match
add() non-selected recs Add recs that DO match
but() non-selected recs Add recs that DO NOT matchHere's how to think about what's going on:
methods... think...
------------------------------------------------------------------
select() "SELECT things matching this"...
omit() "... then OMIT those matching this."
select() "SELECT things matching this"...
add() "... and ADD any others matching this."
select() "SELECT things matching this"...
but() "... and add any others BUT those matching this."select() and omit() both NARROW the selection.
add() and but() both INCREASE the selection.
IMPORTANT: You DO NOT need to use these select() routines to work with selections. It may be much easier for you to clarify your logic, or more efficient to express your search, using a single grep or series of grep operations as in Examples 2 or 3 above.
Building the selection manually is required if you want to filter based on any COMPLEX RELATIONSHIPS BETWEEN FIELDS. For example, if you want to add two fields and match or reject the record based on the sum of the fields.
In Example 3 above, we add the values in fields "A" and "B" and then match the record only if the SUM is between 100 and 199. By grepping to produce a subset of @{$t->all()}, you end up with a Selection -- a list of record numbers you want "selected". Then you call $t->selection() to put the selection you built into the object.
If you had instead wanted to narrow an existing selection in the above example, you would start with $t->selection() (which defaults to $t->all()) instead of starting with $t->all().
Each of the select() methods returns $this->selection() as a convenience.
The effects of modifying a sorted selection
Generally, you should sort AFTER finding, and you should not generally rely on sort order after doing a find. But in case you want to know, the following chart explains what happens to the sort order after the various select() commands are called (at least in the current implementation, which may change without notice):
method... effect on an existing sort order...
------------------------------------------------------------------
select() relative sort order is preserved (stay sorted)
omit() all selected recs restored to "natural" order (unsorted)
add() orig. recs preserved; new recs appended: "natural" order
but() orig. recs preserved; new recs appended: "natural" orderIn other words, you could sort() first and then call select() to narrow down the selection repeatedly without disrupting the sort order. However, any of the other methods will disrupt the sort order and you would need to re-sort. The preservation of order when using select(), and other sort order effects, are likely but not guaranteed to be preserved in future implementations.
Hints about Boolean logic
Consider the following example and the alternative below it. You might initially think these are equivalent, but they're not:
## Case 1:
my $Sel = $t->add(Force => sub {$_ == 1 });
my $Sel = $t->add(Status => sub {$_ eq 'Prime'});
## Case 2:
my $Sel = $t->add(Force => sub {$_ == 1 },
Status => sub {$_ eq 'Prime'});Case 1 extends the selection by adding all records where Force == 1, and then extends it again by adding all additional records where Status eq 'Prime'.
Case 2 adds only those records where: Force == 1 AND ALSO, IN THE SAME RECORD, Status eq 'Prime'.
One final note about logic. This is not SQL and these select() etc. routines are not meant to replace the full power of a programming language.
If you want full Boolean expressions, use the power of Perl to form your own arbitrarily complex query using grep as in Example 3 above.
Writing your own grep is also almost always faster than chaining the builtin select() methods or using multiple Field / Sub specifications, so keep that in mind when working with extremely large data sets.
With tables of only a few thousand records or so, you probably won't notice the difference in efficiency.
SORTING
## Sort the current table's _Selection
$t->sort() ## Use existing/default params
$t->sort([qw(Last First Phone)]) ## Specify _SortOrder (fields)
$t->sort( ## Named-parameter call:
_SortOrder => [...], ## override sort-related params.
_Selection => [...], ## (See param lists above).
_SortSpecs => {...},
_SRoutines => {...},
_DefaultSortType=>'Integer',
_DefaultSortDirection=>-1,
);The sort() method modifies the _Selection (creating one with all records if it was missing, undef, or not supplied by caller) so that the record numbers listed there are sorted according to the criteria implied by _SortOrder, _SortSpecs, _SRoutines, etc.
For example, before sorting, a table's "natural" order might be:
Rec# First Last Age State
0 Chris Zack 43 CA
1 Marco Bart 22 NV
2 Pearl Muth 15 HI... and its selection() method would yield: [0, 1, 2] -- which is a list of all the records, in order.
After calling $t->sort([Last]), selection() would yield [1, 2, 0]. So displaying the table in "selection" order would yield:
Rec# First Last Age State
1 Marco Bart 22 NV
2 Pearl Muth 15 HI
0 Chris Zack 43 CAIMPORTANT: sorting does not alter any data in the table. It merely alters the _Selection parameter (which you can then get and set using the selection() methods described above).
If you want to permanently alter the table's data in memory so that the new sorted order becomes the "natural" order, you can use the cull() method to modify the original object, the snapshot() method to make a new object, or use the write() method to write the data to disk in selected/sorted order and then read() it back again.
Using the Named-parameter calling convention with sort()
You may specify any combination of the parameters listed above when calling sort(). Any you specify will be used IN PLACE OF the corresponding parameters already found in the object.
If you specify _Selection using the named-parameter calling, the sort() method reserves the right to "own" the list you provide, and use it as the object's new _Selection, possibly discarding the previous _Selection, if any and modifying the one you provided. So don't make any assumptions about ownership of that list object after calling sort(). Luckily, you will rarely need to provide _Selection explicitly since generally you'll want to be sorting the selection() already inherent in the object.
sort() returns the _Selection list owned by the object (the same list that would be returned if you called the selection() method immediately after calling sort()).
See the next sections for complete descriptions of _SortOrder and other sorting parameters.
SORT ORDER
## Getting / Setting table's default _SortOrder
$t->sortorder() ## Get sortorder (default is [])
my $Order = [qw(Last First State Zip)];
$t->sortorder($Order) ## Set sortorder (use [] for none)
$t->sortorder_default() ## Get the object's default sort order ([])The sort order is an optional list of field names on which to sort and sub-sort the data when sorting is requested. The field names must be the names of actual columns in the table. The names in the sort order do not necessarily need to coincide with the custom fieldlist if any.
There is one special value that can be included: _RecNum. This sorts on the imaginary "record number" field. So for example, you could specify a sort order this way:
[qw(Last First _RecNum)](There is no point in putting _RecNum anywhere except at the end of the sort order because no two records will ever have the same record number so there will be no further need to disambiguate by referring to additional fields.)
Sorting by _RecNum adds a bit of computational overhead because sort() first builds a record number vector for use in sorting, so for very large tables, don't do it unless you really need it.
A sort order can be specified each time the object is sorted (see the sort() method for details).
Or, the object's sort order can be set once, and then sort() will use that sort order when no other sort order is specified.
If sorting is done when there is no sort order present in the object or specifed for the sort() method, the selection is sorted by record number (i.e. it is "unsorted" or returned to its "natural" order).
In order words, a sortorder that is undef or [] is considered the same as: [qw(_RecNum)]. This is sometimes called "unsorting".
In order to decide how values in each field should be compared, sort() is informed by SortSpecs (specifying SortType and SortDirection for each field) and by SortRoutines, each of which may similarly either be pre-set for the object or specified when calling sort() -- see below for further details.
SORT SPECIFICATIONS
## Getting / Setting table's default _SortSpecs
$t->sortspecs() ## Get sortspecs (default is {} -- none)
my $Specs = {Last => {SortType => 'String' ,
SortDirection => -1 },
Zip => {SortType => 'Integer' }};
$t->sortspecs($Specs) ## Set sortspecs
$t->sortspecs_default() ## Get the object's default sort specs ({})The sortspecs are an optional hash mapping field names to "sort specifications".
Each field's sort specification may specify zero or more of these values:
- SortType
-
the sort type to use (For example: String, Integer)
- SortDirection
-
the sort direction (1: ascending, -1: descending)
Sortspecs can be specified when calling the sort() routine, or, a set of specs can be placed beforehand into the object itself and those will be used by sort() if no other specs are given.
For any field listed in the sort order at the time of sorting, but lacking a sort spec or any component of the sort spec, the object's default sort type (see sorttype_default()) and default sort direction (see sortdirection_default()) will be used.
In addition to getting/setting sort specs as a whole, they may be gotten/set on a per-field basis, too:
sortspec($Field) ## Get sortspec for $Field or default spec
my $Spec = {SortType => 'Integer', SortDirection => -1};
sortspec('Zip', $Spec) ## Set sortspec
sortspec_default() ## Get a sortspec with all defaults filled inFor any $Field not found in the object's sortspecs, sortspec($Field) returns the same thing returned by sortspec_default(), which is a sortspec filled in with the default sort type and sort direction (see below).
For a list of available built-in SortTypes, and instructions for how to define your own, see SORT ROUTINES, below.
DEFAULT SORT DIRECTION
## Getting / Setting table's default _DefaultSortDirection
$t->sortdirection_default() ## Get default sort direction
$t->sortdirection_default(-1) ## Set default sort directionEach element in a sort specification can optionally specify a sort direction.
1 = ascending, -1 = descending
For any sort specs that don't specify a direction, the object's default sort direction will be used. Use these routines to get/set the default sort direction.
DEFAULT SORT TYPE
## Getting / Setting table's default _DefaultSortType
$t->sorttype_default() ## Get default sort type
$t->sorttype_default('Integer') ## Set default sort typeEach element in a sort specification can optionally specify a sort type. The sort type is a string (like 'String' or 'Integer' or 'Date') that selects from one or more sort routines. (See Sort Routines, below).
There are several sort routines built into the CTable object, and you can also add as many of your own routines (and hence Sort Types) as you like or need. This allows for very flexible sorting.
For any sort specs that don't specify a type, the object's default sort type will be used. Use these routines to get/set the default sort type, which initially is 'String'.
SORT ROUTINES: BUILTIN AND CUSTOM
## Getting / Setting table's custom sort routines (_SRoutines)
$t->sortroutine($Type) ## Get a sort routine for $Type
$t->sortroutine($Type, $Sub) ## Set a sort routine for $Type
$t->sortroutine($Type, 0 ) ## Remove sort routine for $Type
$t->sortroutine_set($Type)
$t->sortroutines() ## Get hash of any sort routines
$t->sortroutines_builtin() ## Get hash of builtin routinesEach SortType in the sortspecs should have a corresponding sort routine (any unrecognized type will be sorted using the 'String' sort routine).
The sort() command looks up the appropriate sort routine for each field it is asked to sort, based on the SortType for that field, as given in the sortspecs, as described above.
Builtin sort types, recognized and implemented natively by this module, are:
String ## Fastest case-sensitive compare (data is string)
Text ## Insensitive compare (lowercases, then compares)
Number ## Number works for floats or integers
Integer ## Faster than floats. Uses "use integer"
DateSecs ## Same as integer; assumes date in seconds
Boolean ## Treats item as a Perlish boolean (empty/undef = false)The above sort types are always recognized. Additional sort types may be added by subclasses (and could shadow the builtin implementations of the above types if desired) and/or may be added to instances (and again could shadow the above implementations), and/or may be specified when the sort() method is called, once again optionally shadowing any deeper definitions.
CUSTOM SORT ROUTINE INTERFACE
A custom sort routine is called with two arguments, each of which is a pointer to a scalar. The sort routine should dereference each pointer and compare the resulting scalars, returning -1 if the first scalar is smaller than the second, 1 if it is larger, and 0 if they are considered equal.
For example, here is the built-in comparison routine for 'String':
sub { $ {$_[0]} cmp $ {$_[1]} }NOTE: Your custom sort routines should NOT compare $a and $b as with Perl's builtin sort() command.
Examine the variable $BuiltinSortRoutines in this module's implementation to see some additional examples of sort routines.
Internally, sort() calls the sortroutines() method to get a hash that should consist of all builtin sort routines with the per-object sort routines, if any, overlaid. sortroutines() in turn calls the sortroutines_builtin() method to get a copy of the hash of all builtin sort routines for the object. (So a subclass could easily add additional SortTypes or reimplement them by just overriding sortroutines_builtin() and adding its own additional routines to the resulting hash.)
sortroutine() may be called to get or set a custom sort routine for a given type in the given object.
There is no way to directly manipulate the builtin sort routines for the entire class. To accomplish that, you should define and use a subclass that extends sortroutines_builtin() to add its own routines.
For example:
BEGIN
{ ## A subclass of Data::CTable with an INetAddr SortType.
package IATable; use vars qw(@ISA); @ISA = qw(Data::CTable);
sub sortroutines_builtin
{
my $this = shift;
my $CustomRoutines =
{INetAddr =>
sub {use integer; ip2int($ {$_[0]}) <=> ip2int($ {$_[1]})}};
my $AllRoutines =
{%{$this->SUPER::sortroutines_builtin()} %$CustomRoutines};
return($AllRoutines);
};
sub ip2int {.....} $# Could memoize & inline for efficiency
}
my $Table = IATable::new(......);The IATable class would then have all the same features of Data::CTable but would then also support the INetAddr SortType.
FREEZING SELECTION & FIELD LIST
## Freeze data layout: re-order columns; omit unused fields
$t->cull(...params...) ## Rebuild table in order
my $s = $t->snapshot(...params...) ## Make copy as if rebuiltThe cull() method re-writes all data in the table to be in the order indicated in _Selection (if present). This will cause any records not listed in _Selection to be omitted (unless selection is null in which case all records are retained in original order).
In addition, if there is a custom field list present, it removes any fields NOT mentioned in _FieldList.
The snapshot() method is similar, except instead of modifying the object itself, it makes a copy of the object that's equivalent to what cull() would have created, and returns that new object, leaving the original untouched. (All data structures are deep-copied from the old object to the new one, leaving the objects totally independent.)
cull() and snapshot() both take two optional named parameters: _FieldList and/or _Selection to be used in place of the corresponding parameters found in the object.
If only a single argument is supplied, it is assumed to be _Selection.
LINE ENDINGS
## Get current value
$t->lineending() ## Get actual setting: string or symbol
$t->lineending_symbol() ## Get setting's symbolic value if possible
$t->lineending_string() ## Get setting's string value if possible
## Set value
$t->lineending($Ending) ## Will be converted internally to symbol
## Convert a value to symbol or string form
$t->lineending_symbol($L) ## Convert string form to symbolic form
$t->lineending_string($L) ## Convert symbol form to string form
## Get internal conversion hash tables
$t->lineending_symbols() ## Hash ref mapping known strings to symbols
$t->lineending_strings() ## Hash ref mapping known symbols to stringsUse these accessor functions to get/set the _LineEnding parameter.
You can set the parameter in either string or symbol form as you wish. You can get it in its raw, as-stored, form, or, you can get it in string form or symbol form as desired.
Finally, some utility conversion calls allows you to convert a string you have on hand to a symbolic form. For example:
$L = "\x0D";
print ("This file uses " . $t->lineending_symbol($L) . " endings.");This would print:
This file uses mac endings.AUTOMATIC CACHEING
By default, Data::CTable makes cached versions of files it reads so it can read them much more quickly the next time. Optionally, it can also cache any file it writes for quicker re-reading later.
On Unix systems, cache files are always created with 0666 (world-write) permissions for easy cleanup.
When reading files, Data::CTable checks the _CacheOnRead parameter. If that parameter is true, which it is by default, the module tries to find an up-to-date cache file to read instead of the original. Reading a cache file can be 10x faster than reading and parsing the original text file.
In order to look for the cache file, it must first calculate the path where the cache file should be located, based on the _FileName of the file to be read.
The path of the cache file is calculated as follows:
If the _CacheSubDir parameter is a RELATIVE PATH, then it is appended to the directory component of _FileName to arrive at the directory to use to store the cache file. If it is an ABSOLUTE PATH, then _CacheSubDir is used by itself. (The trailing path separator is optional and an appropriate one will be added by Data::CTable if it is missing.)
The file name of the cache file is calculated as follows:
If the _CacheExtension parameter is specified, it is appended to the base file name component from the _FileName parameter. If you want the cached file name to be the same as the name of the original file, you can set _CacheExtension to "", which is not recommended.
Then, the cache path and cache file name are joined to arrive at the name of the cache file. If both _CacheSubDir and _CacheExtension were empty, then the cache file path will be the same as the _FileName, and Data::CTable will refuse to either read or write a cache file, so setting these fields both to empty is equivalent to setting _CacheOnRead to false.
The cache file contains a highly-efficient representation of all the following data that would otherwise have to be determined by reading and parsing the entire text file:
- All the data columns (field values)
- _FieldList: The list of fields, in order
- _HeaderRow: Whether a header row is / should be present
- _LineEnding: The line ending setting
- _FDelimiter: The field delimiter settingIf found prior to a read(), AND, the date of the cache file is LATER than the date of the original file, the cache file is used instead. (If the date is EARLIER, then the cache file is ignored because it can be presumed that the data inside the text file is newer.)
If cacheing is ON, then after successfully reading the text file (either because there was no cache file yet or the cache file was out of date or corrupted or otherwise unusable), read() will then try to create a cache file. This, of course, takes some time, but the time taken will be more than made up in the speedup of the next read() operation on the same file.
If creating the cache file fails (for example, because file permissions didn't allow the cache directory to be created or the cache file to be written), read() generates a warning explaining why cacheing failed, but the read() operation itself still succeeds.
No parameters in the object itself are set or modified to indicate the success or failure of writing the cache file.
Similarly, there is no way to tell whether a successful read() operation read from the cache or from the original data file. If you want to be SURE the reading was from the data file, either turn off _CacheOnRead, or call the read_file() method instead of read().
NOTE: because the name of the cache file to be used is calculated just before the read() is actually done, the cache file can only be found if the _CacheSubDir and _CacheExtension are the same as they were when the cache was last created. If you change these parameters after having previously cached a file, the older caches could be "orphaned" and just sit around wasting disk space.
Cacheing on write()
You may optionally set _CacheOnWrite (default = false) to true. If done, then a cache file will be saved for files written using the write() command. Read about write() below for more about why you might want to do this.
AUTOMATIC DIRECTORY CREATION
When Data::CTable needs to write a file, (a cache file or a data file), it automatically tries to create any directories or subdirectories you specify in the _FileName or _CacheSubDir parameters.
If it fails while writing a data file, write() will fail (and you will be warned). If it fails to create a directory while writing a cache file, a warning will be issued, but the overall read() or write() operation will still return a result indicating success.
Any directories created will have the permissions 0777 (world-write) for easy cleanup.
Generally, the only directory the module will have to create is a subdirectory to hold cache files.
However, since other directories could be created, be sure to exercise caution when allowing the module to create any directories for you on any system where security might be an issue.
Also, if the 0666 permissions on the cache files themselves are too liberal, you can either 1) turn off cacheing, or 2) call the prep_cache_file() method to get the name of the cache file that would have been written, if any, and then restrict its permissions:
chmod (0600, $this->prep_cache_file());READING DATA FILES
## Replacing data in table with data read from a file
$t->read($Path) ## Simple calling convention
$t->read( ## Named-parameter convention
## Params that override params in the object if supplied...
_FileName => $Path, ## Full or partial path of file to read
_FieldList => [...], ## Fields to read; others to be discarded
_HeaderRow => 0, ## No header row (_FieldList required!)
_LineEnding => undef, ## Text line ending (undef means guess)
_FDelimiter => undef, ## Field delimiter (undef means guess)
_ReturnMap => 1, ## Whether to decode internal returns
_ReturnEncoding=>"\x0B", ## How to decode returns.
_MacRomanMap => undef, ## Whether/when to read Mac char set
_CacheOnRead => 0, ## Enable/disable cacheing behavior
_CacheExtension=> ".x", ## Extension to add to cache file name
_CacheSubDir => "", ## (Sub-)dir, if any, for cache files
## Params specific to the read()/write() methods...
_MaxRecords => 200, ## Limit on how many records to read
)
$t->read_file() ## Internal: same as read(); ignores cacheingread() opens a Merge, CSV, or Tab-delimited file and reads in all or some fields, and all or some records, REPLACING ANY EXISTING DATA in the CTable object.
Using the simple calling convention, just pass it a file name. All other parameters will come from the object (or will be defaulted if absent). To specify additional parameters or override any parameters in the object while reading, use the named-parameter calling convention.
See the full PARAMETER LIST, above, or read on for some extra details:
_ReturnMap controls whether return characters encoded as ASCII 11 should be mapped back to real newlines ("\n") when read into memory. If false, they are left as ASCII 11 characters. (default is "true")
_ReturnEncoding controls the character that returns are encoded as, if different from ASCII 11.
_FieldList is an array (reference) listing the names of fields to import, in order (and will become the object's _FieldList upon successful completion of the read() operation). If not provided and not found in the object, or empty, then all fields found in the file are imported and the object's field list will be set from those found in the file, in the order found there. If _HeaderRow is false, then this parameter is required (either in the object or as a formal parameter) and is assumed to give the correct names for the fields as they actually occur in the file. If _HeaderRow is true and _FieldList is provided, then _FieldList specifies the (sub-)set of fields to be read from the file and others will be ignored.
_HeaderRow, which defaults to true, if set to false, tells read() to not expect a header row showing the field names in the file. Instead, it assumes that the _FieldList gives those (and _FieldList must therefore be specified either as a parameter or an existing parameter in the object).
_MaxRecords (optional) is an upper limit on the number of fields to import. If not specified, or zero, or undef, then there is no limit; all records will be imported or memory will be exhausted.
read() returns a Boolean "success" code.
If read() returns false, then it will also have set the _ErrorMsg parameter in the object. It may or may not have partially altered data in the object if an error is encountered.
After a successful read:
fieldlist() (the object's _FieldList parameter) tells which fields were actually read, in what order. It may omit any fields requested in _FieldList that were not actually found in the file for whatever reason.
length() tells how many fields were read.
The selection() is reset to no selection (all selected / unsorted)
The object's _FileName parameter contains the path to the file that was read. If the _FileName you specified did not have a path, then _FileName will be prepended with a path component indicating "current directory" (e.g. "./" on Unix).
_FDelimiter will contain the actual delimiter character that was used to read the file (either tab or comma if the delimiter was guessed, or whatever delimiter you specified).
_LineEnding will contain the actual line-ending setting used to read the file. This will be either "mac" ("\x0D"), "unix" ("\x0D"), or "dos" ("\x0D\x0A") if the line endings were guessed by read(). Otherwise it will be whatever _LineEnding you specified.
FILE FORMAT NOTES
As mentioned, read() allows the following flexibilities in reading text-based tabular data files:
You may specify the line endings (record delimiters), or it can guess them (mac, unix, dos are supported).
You may specify the field delimiters, or it can guess them (tab and comma are supported).
It can get field names from a header row, or, if there is no header row, you can tell it the field names, in order.
You can tell it whether or not to decode embedded returns in data fields, and if so, which character they were encoded as.
Beyond supporting the above flexible options, read() makes the following non-flexible assumptions:
Fields must NOT contain unencoded returns -- that is: whatever character sequence is specified for _LineEnding will NEVER occur inside a field in the text file; in addition, the current platform's definition of "\n" will NEVER occur; these characters if present in field data, MUST have been encoded to some safe character string before the file was created.
Each field may OPTIONALLY be surrounded with double-quote marks. However, if the field data itself contains either a double-quote character (") or the current file's field delimiter (such as tab or comma), then the field MUST be surrounded with double-quotes. (Currently, all data written by Data::CTable have all field values surrounded by double-quotes, but a more selective policy may be used in the future.)
If a field contains a double-quote character, then each double-quote character in the field must be encoded as "" -- i.e. each " in the original data becomes "" in the text file.
Data files may not mix line-ending types or field delimiter types. Once determined, the same endings and delimiters will be used to read the entire file.
The fields recognized on each line will either be determined by the header row or the _FieldList provided by the caller. Any extra fields on any given line will be ignored. Any missing fields will be treated as undef/empty.
If you are having trouble reading a delimited text file, check that all data in the file obeys these assumptions.
WRITING DATA FILES
## Writing some or all data from table into a data file
$t->write($Path) ## Simple calling convention
$t->write( ## Named-parameter convention
## Params that override params in the object if supplied...
_FileName => $Path, ## "Base path"; see _WriteExtension
_WriteExtension=> ".out",## Insert/append extension to _FileName
_FieldList => [...], ## Fields to write; others ignored
_Selection => [...], ## Record (#s) to write; others ignored
_HeaderRow => 0, ## Include header row in file
_LineEnding => undef, ## Record delimiter (default is "\n")
_FDelimiter => undef, ## Field delimiter (default is comma)
_ReturnMap => 1, ## Whether to encode internal returns
_ReturnEncoding=>"\x0B", ## How to encode returns
_MacRomanMap => undef, ## Whether/when to write Mac char set
_CacheOnWrite => 1, ## Enable saving cache after write()
_CacheExtension=> ".x", ## Extension to add to cache file name
_CacheSubDir => "", ## (Sub-)dir, if any, for cache files
## Params specific to the read()/write() methods...
_MaxRecords => 200, ## Limit on how many records to write
)
$t->write_file() ## Internal: same as write(); ignores cacheingwrite() writes a Merge, CSV, or Tab-delimited file.
It uses parameters as described above. Any parameters not supplied will be gotten from the object.
Using the simple calling convention, just pass it a path which will override the _FileName parameter in the object, if any.
All other parameters will come from the object (or will be defaulted if absent).
If no _FileName or path is specified, or it is the special string "-" (dash), then the file handle \ * STDOUT will be used by default (and you could redirect it to a file). You can supply any open file handle or IO::File object of your own for the _FileName parameter.
If write() is writing to a file handle by default or because you specified one, then no write-cacheing will occur.
To specify additional parameters or override any parameters in the object while reading, use the named-parameter calling convention.
If the object's data was previously filled in using new() or read(), then the file format parameters from the previous read() method will still be in the object, so the format of the written file will correspond as much as possible to the file that was read().
write() returns the path name of the file actually written, or the empty string if a supplied file handle or STDOUT was written to, or undef if there was a failure.
If write() returns undef, then it will also have set the _ErrorMsg parameter in the object.
write() never modifies any data in the object itself.
Consequently, if you specify a _FieldList or a _Selection, only those fields or records will be written, but the corresponding parameters in the object itself will be left untouched.
How write() calculates the Path
The _FileName parameter is shared with the read() method. This parameter is set by read() and may be overridden when calling write().
In the base implementation of Data::CTable, write() will try not to overwrite the same file that was read, which could possibly cause data loss.
To avoid this, it does not use the _FileName parameter directly. Instead, it starts with _FileName and inserts or appends the value of the _WriteExtension parameter (which defaults to ".out") into the file name before writing.
If the _FileName already has an extension at the end, write() will place the _WriteExtension BEFORE the final extension; otherwise the _WriteExtension will be placed at the end of the _FileName.
For example:
Foobar.txt ==> Foobar.out.txt
Foobar.merge.txt ==> Foobar.merge.out.txt
My_Merge_Data ==> My_Merge_Data.outIf you DON'T want write() to add a _WriteExtension to _FileName before it writes the file, then you must set _WriteExtension to empty/undef either in the object or when calling write(). Or, you could make a subclass that initializes _WriteExtension to be empty. If _WriteExtension is empty, then _FileName will be used exactly, which may result in overwriting the original data file.
Remember: write() returns the path name it actually used to successfully write the file. Just as with read(), if the _FileName you specified did not have a path, then write() will prepend a path component indicating "current directory" (e.g. "./" on Unix) and this will be part of the return value.
Cacheing with write()
By default, Data::CTable only creates a cached version of a file when it reads that file for the first time (on the assumption that it will need to read the file again more often than the file's data will change.)
But by default, it does not create a cached version of a file when writing it, on the assumption that the current program probably will not be re-reading the written file and any other program that wants to read it can cache it at that time.
However, if you want write() to create a cache for its output file, it is much faster to create it on write() than waiting for the next read() because the next read() will be able to use the cache the very first time.
To enable write-cacheing, set _CacheOnWrite to true. Then, after the write() successfully completes (and only if it does), the cached version will be written.
FORMATTED TABLES (Using Data::ShowTable)
## Get formatted data in memory
my $StringRef = $t->format(); ## Format same data as write()
my $StringRef = $t->format(10); ## Limit records to 10
my $StringRef = $t->format(...); ## Specify arbitrary params
print $$StringRef;
## Write formatted table to file or terminal
$t->out($Dest, ....);## $Dest as follows; other params to format()
$t->out($Dest, 10, ....) ## Limit recs to 10; params to format()
$t->out() ## print formatted data to STDOUT
$t->out(\*STDERR) ## print to STDERR (or any named handle)
$t->out("Foo.txt") ## print to any path (file to be overwritten)
$t->out($FileObj) ## print to any object with a print() methodout() takes a first argument specifying a destination for the output, then passes all other arguments to format() to create a nice-looking table designed to be human-readable; it takes the resulting buffer and print()s it to the destination you specified.
Sample output:
+-------+------+-----+-------+
| First | Last | Age | State |
+-------+------+-----+-------+
| Chris | Zack | 43 | CA |
| Marco | Bart | 22 | NV |
| Pearl | Muth | 15 | HI |
+-------+------+-----+-------+
(Note extra space character before each line.)The destination may be a file handle (default if undef is \*STDOUT), a string (treated as a path to be overwritten), or any object that has a print() method, especially an object of type IO::File.
The main purpose of out() is to give you a quick way to dump a table when debugging. out() calls format() to create the output, so read on...
format() produces a human-readable version of a table, in the form of a reference to a string buffer (which could be very large), and returns the buffer to you. Dereference the resulting string reference before using.
If format() is given one argument, that argument is the _MaxRecords parameter, which limits the length of the output.
Otherwise, format() takes the following named-parameter arguments, which can optionally override the corresponding parameters, if any, in the object:
_FieldList ## Fields to include in table
_Selection ## Records to be included, in order
_SortSpecs ## SortType controls number formatting
_DefaultSortType
_MaxRecords ## Limit number of records output
_MaxWidth ## Limit width of per-col. data in printoutformat() will obey _MaxRecords, if you'd like to limit the number of rows to be output. _MaxRecords can also be a single argument to format(), or a second argument to out() if no other parameters are passed.
format() also recognizes the _SortSpecs->{SortType} and _DefaultSortType parameters to help it determine the data types of the fields being formatted. Fields of type "Number" are output as right-justified floats; "Integer" or "Boolean" are output as right-justified integers, and all others (including the default: String) are output as left-justified strings.
In addition, there is one parameter uniquely supported by format() and out():
_MaxWidth specifies the maximum width of columns. If unspecifed, this will be 15; the minimum legal value is 2. Each column may actually take up 3 more characters than _MaxWidth due to divider characters.
The data to be output will be examined, and only the necessary width will be used for each column. _MaxWidth just limits the upper bound, not the lower.
Data values that are too wide to fit in _MaxWidth spaces will be truncated and the tilde character "~" will appear as the last character to indicate the truncation.
Data values with internal returns will have the return characters mapped to slashes for display.
format() and out() will NOT wrap entries onto a second line, like you may have seen Data::ShowTable::ShowBoxTable do in some cases. Each record will get exactly one line.
format() and out() ignore the _HeaderRow parameter. A header row showing the field names is always printed.
format() and out() make no attempt to map upper-ascii characters from or to any particular dataset. The encoding used in memory (generally ISO 8859-1 by default) is the encoding used in the output. If you want to manipulate the encoding, first call format(), then change the encoding, then format the resulting table.
APPENDING / MERGING / JOINING TABLES
## Append all records from a second table
$t->append($Other) ## Append records from $Other
$t->append_file($File, $Params) ## Append from new($Params, $File)
$t->append_files($Files, $Params) ## Call append_file for all files
$t->append_files_new($Files, $Params) ## Internal helper routine
## Combine all fields from a second table
$t->combine($Other) ## Combine fields from $Other
$t->combine_file($File, $Params) ## Combine new($Params, $File)
$t->combine_files($Files, $Params) ## combine_file on each file
## Left-join records from a second table (lookup field vals)
$t->join ($Other, $KeyField1, [$KeyField2, $Fields])
$t->join_file ($File, $Params, $KeyField1, [$KeyField2, $Fields])
$t->join_files($Files, $Params, $KeyField1, [$KeyField2, $Fields])The append() method concatenates all the records from two CTable objects together -- even if the two tables didn't start out with exactly the same fields (or even any of the same fields).
It takes all the data records from another CTable object and appends them into the present table. Any columns present in the $Other table but not in the first table, are created (and the corresponding field values in the first table will all be empty/undef). Similarly, any columns present in $t but not present in $Other will be extended to the correct new length as necessary and the field values in the original columns will be empty/undef. Columns present in both will, of course, have all the data from both the original sets of data.
All data from the second table is brought into the first one. No attempt whatsoever is made to eliminate any duplicate records that might result.
The number of records (length()) after this call is the sum of the length() of each of the tables before the operation.
IMPORTANT NOTE: The data from the $Other object is COPIED in memory into the new object. This could be hard on memory if $Other is big. Might want to be sure to discard $Other when you're done with it.
$Other is left untouched by the operation.
All columns from both tables are combined whether or not they are mentioned in the custom field list of either.
The custom field lists, if present in either table object, are concatenated into this object's custom field list, but with duplications eliminated, and order retained.
Any existing custom selections, custom sort order, sort specs, and/or sort routines are also combined appropriately, with settings from this object taking precedence over those from $Other anywhere the two have conflicting settings.
append_file() takes a file name and optional $Params hash. It uses those to create a new() object with data read from the file. Then, the new table is appended to $t using append() and then the new table is discarded.
append_files() is a convenience function that calls append_file() on each file in a list, using the same optional $Params for each.
append_files_new() is the internal routine that implements the processing done by new() on the optional list of files to be read. It does the following: It calls read() on the first file in the list. Then, it calls append_files() to read the remaining into their own new() objects of the same class as $t and using the same $Params to new() (if any were supplied). Then each of these is append()-ed in turn to $t and discarded. The final result will be that $t will hold a concatenation of all the data in all the files mentioned. However, consistent with the behavior of append(), the _FileName parameter and other read()-controlled settings will correspond to the first file read. The intermediate objects are discarded.
NOTE: As with new() and read(), if a non-empty _FieldList Param is specified, the read() methods called internally by the append_file*() methods will read only the fields mentioned and will ignore any other fields in the files.
Combining tables
combine() adds columns from a second table into the current one.
CAUTION: You should only use combine() when you have two tables where all the (possibly selected) records in the second table line up perfectly with all the (unselected) records in the first table -- in other words, each table before combine() should contain a few of the columns of the new table -- for example, maybe one table contains a column of file names, and the other contains columns of corresponding file sizes and modification times. If you don't understand the consequences of combine, don't use it or you could end up with some records whose field values don't refer to the same object. (Maybe you meant to use append() or join() instead.)
If the second table has a custom field list, only those columns are brought in.
If any column in the second table has the same name as one in the current table, the incoming column replaces the one by the same name.
All columns are COPIED from the second table, so the first table owns the new data exclusively.
If the second table has a selection, only those records are copied, in selection order. (select_all() first if that's not what you want.)
The selection in the first table, if any, is ignored during the combine. If this isn't what you want, then consider using cull() before combine().
Field list and sort order are concatenated (but retaining uniqueness: second mentions of a field in the combined lists are omitted).
Custom sort routines and sort specs are combined, with those in the first table taking precedence over any copied in with the same name.
The custom _Selection from the first table, if any, is retained. (It will initially omit any records added by extend()).
All other parameters from the first table are retained, and from the second table are ignored.
combine() calls extend() after combining to ensure that all columns have the same length: if either the older or newer columns were shorter, they will all be set to the length of the longest columns in the table -- creating some empty field values at the end of the lengthened columns.
combine_file() does the same as combine() except starting with a file name, first creating the $Other object by creating it using new($Params, $File), then discarding it after combining.
Joining tables (Looking up data from another table)
join() looks up field values from a second table, based on common values in key fields which may have different or the same names in each table. It adds columns to the current table if necessary to hold any new field values that must be brought in.
join() never adds any new or non-matching records to the table: records where the lookup fails will simply have empty/undef values in the corresponding columns.
## Example:
$t->join ($People, 'FullName', 'FirstAndLast'); ## or
$t->join_file("People.txt", {}, 'FullName', 'FirstAndLast');Here's how join() calculates the list of fields to bring in:
- Legal field names from the optional $Fields list, if supplied
- Otherwise, the fieldlist() from second table
- ... minus any fields with same name as $KeyField1 or $KeyField2Join starts by adding new empty columns in the first table for any field to be brought in from the second but not yet present in the first.
Here's how join() calculates the records eligible for lookup:
- Join only modifies the selected records in the first table
- Join only looks up values from selected records in second table(If you want all records to be used in both or either table, call the table's select_all() method before calling join().)
Then, for every selected record in $t (using the example above), join examines the FullName field ($KeyField1), and looks up a corresponding entry (must be 'eq') in the FirstAndLast field ($KeyField2) in the second table.
IMPORTANT NOTE ABOUT KEY LENGTH: To speed lookup, hash-based indices are made. The strings in $Key1 and $Key2 fields should not be so long that the hash lookups bog down or things could get ugly fast. There is no fixed limit to hash key length in Perl, but fewer than 128 characters in length is longer than customary for such things. (Many systems require text-based keys to be no longer than 31 characters.) So be judicious about the values in $Key1 and $Key2 fields.
The first record found in the second table's selection with a matching value in the key field is then copied over (but only the appropriate fields are copied, as explained above). Any field values being brought over will REPLACE corresponding field values in the first table, possibly overwriting any previous values if the field being looked up was already present in the first table and contained data.
The first table's _FieldList is updated to reflect new fields added.
Its _Selection is untouched.
Its _SortOrder is untouched.
Its _SortSpecs are augmented to include any entries from the second table that should be brought over due to the field additions.
Its _SRoutines are augmented to add new ones from the second table.
All other parameters of table 1 are untouched.
The second table is not modified. No data structures will be shared between the tables. Data is only copied.
join_file() calls join() after creating a seond table from your $File.
join_files() calls join_file() repeatedly for each file in a list, but it is important to note that each file in the list of files to be joined must have a $Key2 field -- AND, that any values looked up from the second file will overwrite any values of the same key found in the first file, and so on. You probably will not ever need join_files(). It is mainly here for completeness.
INVERTING A TABLE'S ROWS/COLUMNS
## Re-orient table's data using vals from $ColName as field names...
$t-invert($ColName)Sometimes a situation gives you a table that's initially organized with column data in rows, and field names in one of the columns, so you need to flip the table in order to be able to work meaningfully with it.
"Inverting" a table means to rewrite each row as a column. One row is designated to be used as the field names.
For example, consider this table:
F01 F02 F03 F04
------------------------
First Chris Agnes James
Last Bart Marco Nixon
Age 22 33 44Calling invert() using field names from "F01"...
$t->invert('F01');... would change the table to look like this:
First Last Age
----------------
Chris Bart 22
Agnes Marco 33
James Nixon 44The field F01 which formerly contained the field names, is now gone, and the remaining data columns have been converted from their old row orientation into a column orientation.
PROGRESS MESSAGES
## Printing a progress message....
$t->progress($Msg) ## Print a message per current settings
## Progress settings applying to this object only...
$t->progress_get() ## Get current progress setting
$t->progress_set(1) ## Use progress_default() method
$t->progress_set($Sub) ## Set a custom progress routine
$t->progress_set(0) ## Disable progress
$t->progress_set(undef) ## Use class's settings (default)...
## Class's settings (for instances with _Progress == undef)
$t->progress_class() ## Get current setting.
$t->progress_class(1) ## Use progress_default() method
$t->progress_class($Sub) ## Set shared custom prog routine
$t->progress_class(0) ## Disable class-default progress
Data::CTable->progress_class(..) ## Call without an object
## Call builtin default progress method regardless of settings
$t->progress_default($Msg) ## Default prog. routine for class
## Generate a warning (used internally by other methods)
$t->warn($Msg) ## In this class, calls progress_default()
## Timed progress: print msg to start, then at most once/2 sec
$t->progress_timed($Op, $Item) ## Re-print msg every 2 sec
$t->progress_timed($Op, $Item, $Pos, $Tot) ##... with % readout
$t->progress_timed($Op, $Item, $Pos, $Tot, $Wait) ## Not 1st x
$t->progress_timed_default($Msg) ## Called by progress_timedData::CTable is especially useful in creating batch-oriented applications for processing data. As such, routines that may perform time-consuming tasks will, by default, generate helpful progress messages. The progress mechanism is highly customizable, however, to suit the needs of applications that don't require this output, or that require the output to go somewhere other than STDERR or the console.
The default progress routine is one that prints a message with a date/time stamp to STDERR if and only if STDERR is an interactive terminal, and otherwise is silent.
You could write a custom progress routine that does something else or something in addition (e.g. logs to a file or syslog). The custom routine could either be implemented by overriding the progress_default() method in a subclass, or by calling progress_set() in any instance.
The custom progress routine, if any, is stored in the _Progress parameter of the object. But use progress_set() and progress_get() to access it.
The interface for your custom progress routine should be:
sub MyProgress {my ($Obj, $Message) = @_; chomp $Message; .....}In other words, the routine takes a single message which may or may not have a trailing newline. It should always chomp the newline if present, and then do its business... which generally will include printing or logging a message (usually with a newline added).
The default, built-in progress routine for Data::CTable is:
sub progress_default
{
my ($this, $msg) = @_;
chomp $msg;
print STDERR (localtime() . " $msg\n") if -t STDERR;
return(1); ## Indicate progress actually completed
}Of course, you are free to call this method directly at any time, and it will do its thing regardless of other progress-enabling settings. But the preferred way is to first set the settings and then call progress().
The warn() method always calls progress_default() -- i.e. warnings will display even if progress is otherwise disabled or overridden at the object or class level. However, you could create a subclass that changes warn()'s behavior if desired. (For example, it could just call perl's builtin warn function, or be even more forceful, generating warnings even if STDERR is not a terminal, for example.)
The progress_set() method may be used to override the progress routine for an individual object (set to 1/true for default behavior, or 0/undef/false to disable progress for that object entirely).
Call progress_class() to set similar values to control the global default behavior (e.g. turning on/off default progress behavior for all instances), but be cautious about using this approach in any environment where other programs might be accessing the same loaded class data, since the setting is stored in a class-owned global ($Data::CTable::DefaultProgress).
Manipulating the class-default settings is only recommended in batch or shell-script environments, not in mod_perl Web applications where the module stays loaded into the Perl environment across multiple invocations, for example.
If you want a particular method (e.g. read() but not write()) to be silent, you could make a subclass and could override that method with an implementation that first disables progress, calls the SUPER:: method, and then restores the progress setting to its original setting.
Timed progress
Timed progress is a way of printing periodically-recurring progress messages about potentially time-consuming processes to the terminal.
For example, consider the following messages which might appear every 2 seconds during a lengthy read() operation:
Reading... 0 (0%)
Reading... 2000 (4%)
...
Reading... 38000 (96%)
Reading... 40000 (100%)The progress_timed() method is called internally by potentially time-consuming processes (read(), write(), and sort()), and you may want to call it yourself from your own scripts, to produce weary-programmer-soothing visual output during otherwise panic-producing long delays.
Generally, progress_timed() is called with the $Wait parameter set to true, which delays the display of any messages until 2 seconds have passed, so no messages will be displayed unless the process actually does end up being slower than 2 seconds.
Parameters are:
$Op The string that identifies the "operation" taking place
$Item A milestone such as a number or datum to indicate progress
$Pos A numeric position against the (maybe estimated) baseline
$Tot The baseline. If estimated, don't re-estimate too often
$Wait If true, skip printing the first message for this $OpAll parameters except $Op are optional.
progress_timed() has a throttle that keeps it from re-triggering more often than every 2 seconds for any given sequence of the same $Op. The clock is restarted each time you call it with a different $Op or $Tot from the previous call (on the assumption that if the operation or the baseline changes then that fact should be noted).
The messages printed will start with "$Op... ".
If you supply $Item, which could be a number or a string, the messages will then show the $Item after the $Op.
If you supply BOTH $Pos and $Tot, then a percentage will be calculated and added to the readout; otherwise omitted.
If you supply $Wait, the first message (only) that uses this $Op will be skipped, and the next one won't appear for at least 2 seconds.
If using $Pos and $Tot to display percentages for your user, be sure to call progress_timed() one final time when $Pos == $Tot so your user sees the satisfying 100% milestone. This "completion" call will not be skipped even if 2 seconds have not passed since the previous timed progress message was printed.
Althought progress_timed() is designed to cut down on too much visual output when called often in a tight loop, remember that it still takes some processing time to call it and so if you call it too frequently, you're slowing down the very loop you wish were running faster.
So, you might want to call it every tenth or 100th or even 1000th time through a tight loop, instead of every time through, using the mod (%) operator:
$t->progress_timed(....) if ($LoopCount % 100) == 0;progress_timed_default() is the method called internally by progress_timed() to actually print the messages it has prepared. In this implementation, progress_timed_default() just calls progress_default(). That is, it ignores all other progress-inhibiting or -enhancing settings so delay-soothing messages will print on the terminal even if other messages are turned off.
This is because the author assumes that even if you don't want all those other progress messages, you might still want these ones that explain long delays. If you REALLY don't, then just make yourself a lightweight subclass where progress_timed_default() is a no-op, or maybe calls regular progress(). For example:
BEGIN {package Data::CTable::Silent; use vars qw(@ISA);
@ISA=qw(Data::CTable); sub progress_timed_default{}}
## Later...
my $t = Data::CTable::Silent->new(...);Rejecting or reporting on groups of records and continuing
Use utility methods omit_warn() and omit_note() to conditionally omit some records from a table and warn (or "note") if any were affected.
Use select_extract() to do the same thing but without actually removing the extracted records from the table, and restoring the original selection before select_extract was called.
If you supply a file name as the 4th argument, the omitted records will be extracted to a file for later reference.
If you supply a message prefix as the 5th argument, a string other than "WARNING" or "Note" may be specified.
# Reject with a progress message prefixed by "WARNING:"
$t->omit_warn(FirstName => sub{!length($_)}, "First name is empty");
Mon Aug 23 08:24:15 2004 WARNING: Omitting 2 of 78243 records (now 78241): First name is empty.
# Reject with a progress message prefixed by "Note:", with output to a file
$t->omit_note(FirstName => sub{!length($_)}, "First name is empty", "empty.names.txt");
Mon Aug 23 08:24:15 2004 Note: Omitting 2 of 78243 records (now 78241): First name is empty.
Mon Aug 23 08:24:15 2004 Writing bad.firstname.txt...
Mon Aug 23 08:24:15 2004 Wrote bad.firstname.txt.
# Extract some items, leaving original selection intact
$t->select_extract(FirstName => sub{!length($_)}, "First name is empty", "empty.names.txt");
Mon Aug 23 08:24:15 2004 Note: Extracting 2 of 78243 records: First name is empty.DEBUGGING / DUMPING
## Print some debugging output...
$t->out() ## Pretty-print $t using Data::ShowTable
$t->dump() ## Dump $t using Data::Dumper
$t->dump($x, $y) ## Dump anything else using Data::Dumper
## Print some debugging output and then die.
die $t->out() ## Same but die afterwards.
die $t->dump() ## Same but die afterwards.
die $t->dump($x, $y) ## Same but die afterwards.These two methods can be very helpful in debugging your scripts.
The out() method, which has many options, is described in complete detail in the section below titled "FORMATTED TABLES". In short, it prints a nicely-formatted diagram of $t, obeying the custom field list if any and custom selection if any.
The dump() method uses the Data::Dumper module to call &Dumper() on the table itself (by default) and prints the result to STDERR. If you specify any number of other values, those will be dumped instead using a single call to &Dumper (rather than individually).
Optional module dependencies
These methods require the otherwise-optional modules shown here:
out() Data::ShowTable
dump() Data::DumperYou'll get a warning at runtime if you try to call either method without the appropriate module installed on your system.
MISCELLANEOUS UTILITY METHODS
The following utilities are methods of the Data::CTable object. They may be called directly by clients, subclassed, or used by subclass implementations as needed.
## Get cache file path (all args optional: defaulted from $t)
$f = $t->prep_cache_file($FileName, $CacheExtension, $CacheSubDir)
## Verify all directories in a path, creating any needed ones.
$ok = $t->verify_or_create_path($DirPath, $Sep)
## Testing readability / writeability of a proposed file
$ok = $t->try_file_read ($Path); ## Opens for read; then closes
$ok = $t->try_file_write($Path); ## Opens for write; then deletes
## Getting parameters from object with optional overrides
$param = $t->getparam($Params, $Param)prep_cache_file() is the internal method used by both read() and write() to calculate the name of a cache file to be used for a given $FileName.
It calculates the path to the cache file that corresponds to the given $FileName (which may be a bare file name, a relative path, or a partial path, as long as it obeys the current platform's path format rules). All arguments are optional and if absent (undef), will be defaulted from the corresponding parameters in $t.
In addition to calculating the path and file name, it also prepends the "current directory" path if there was no path. Then it checks that all directories mentioned in the path actually exist. If not, it fails. Then, it checks that EITHER the file exists and is readable, OR it does not exist but would be writeable in that directory. If any of these directory creations or file checks fails, then undef is returned (and there would be no cache file).
You may call it with no arguments on a file that has been read() to find the path to the cache file that may have been used and/or created, if any.
You may call it with a file name that was written to, to see what the corresponding written cache file would be.
For example:
## Get name of cache file used or created by read and delete it.
$RCache = $t->prep_cache_file() and unlink($RCache);
## Cache on write() and get name of file and delete it.
$Written = $t->write(_CacheOnWrite=>1, _FileName=>"Foo.txt");
$WCache = $t->prep_cache_file($Written) and unlink($WCache);verify_or_create_path() is the internal routine used by read(), write(), and the cache-writing logic, that makes sure a requested file path exists (by creating it if necessary and possible) before any file is written by this module.
(If you don't like this module's tendency to try to create directories, make yourself a subclass in which this routine simply checks -d on its $Path argument and returns the result.)
It must be called with a full or partial path TO A DIRECTORY, NOT A FILE. You may supply $Sep, a platform-appropriate separator character (which defaults correctly for the runtime platform if you don't).
Returns true if the path verification and/or creation ultimately succeeded, false otherwise (meaning that, after this call, there is no such directory on the system and so you should not try to write a file there).
try_file_read() and try_file_write() are the internal methods called by prep_cache_file() as well as by read() and write() to preflight proposed file reading and writing locations.
try_file_read() opens a file for read and closes it again; returns true if the open was possible.
try_file_write() opens a file for write and closes it again, deleting it if successful. Returns true if the open for write and the delete were successful. (Be aware that this call will actually delete any existing file by this name.)
The reason that failure to delete causes try_file_write() to fail is that successful cacheing depends on the ability to delete cache files as well as create them or write to them. A file in a location that couldn't be deleted will not be used for cacheing.
getparam() looks up a named parameter in a params hash if it exists there, otherwise looks it up in the object, thereby allowing $Params to shadow any parameters in $this.
This internal routine is used by any methods that allow overriding of parameters in the object when using a named-parameter calling interface. It should be used by any subclasses that also wish to use a named-parameter calling convention. For example:
my $this = shift;
my $Params = (@_ == 1 ? {_FieldList => $_[0]} : {@_});
my($FieldList, $Selection) = map {$this->getparam($Params, $_)}
qw(_FieldList _Selection);GENERAL-PURPOSE UTILITY FUNCTIONS
These general-purpose utility routines are defined in the Data::CTable module but are not method calls. You may optionally import them or call them by their fully-qualified name.
use Data::CTable qw(
guess_endings
guess_delimiter
path_info
path_is_absolute
min
max
);
## File-format guessing
my $E = &guess_endings($IOHandle) ## Guess txt file line endings
my $D = &guess_delimiter($String) ## Tab if found, else comma
## Cross-platform file path analysis
my $Info = path_info(); ## Hash: 3 of platform's path values:
my ($Sep, ## ... path separator ( / on Unix)
$Up, ## ... "up" component (../ on Unix)
$Cur) = ## ... curr. dir path ( ./ on Unix)
@$Info{qw(sep up cur)};
my $Abs = path_is_absolute($Path) ## Check path type
## Our old favorites min and max
$x = max($x, 0); ## Should have been part of Perl...
$x = min($x, 100);guess_endings() tries to figure out whether an open IO::File handle has DOS, Mac, or Unix file endings. It reads successively large blocks of the file until it either finds evidence of at least two separate line endings (of any type, but presumably they are the same), or until it reaches the end of the file. Then, it takes the resulting block and searches for the first qualifying line ending sequence it finds, if any. This sequence is then returned to the caller. If it returns undef, it was not able to find any evidence of line endings in the file.
guess_delimiter() takes a string buffer and returns a "," unless it finds a tab character before the first comma in the $String, if any, in which case a tab is returned.
path_info() returns a hash of three helpful strings for building and parsing paths on the current platform. Knows about Mac, Dos/Win, and otherwise defaults to Unix.
path_is_absolute($Path) returns true if it thinks the given path string is an absolute path on the current platform.
IMPLEMENTATION LIMITATIONS
- Column (field) names must not start with underscore
-
This object is implemented as a blessed hash reference. By convention, keys that do not start with underscore are data columns and the key is the field name. Keys that do start with underscore refer to parameters or other data structures stored in the object.
Consequently, no field names may start with underscore. When a file is read from disk, any field names that DO start with underscores will have the leading underscores stripped off. Strange things could then occur if the field names are then no longer unique. For example, field "A" and "_A" in the data file would be treated as the single field "A" after the file was read.
- Field values are always read as strings
-
Field values when written to a file are necessarily converted to strings. When read back in, they are read as strings, regardless of original format. The sole exception is the empty string which is read back in as undef for efficiency.
An exception is when the _CacheOnWrite feature is used: field values stored internally as integers or other scalar types may be saved and later restored as such. However, you should not rely on this behavior.
- Undef vs. empty
-
Empty field values are stored as "undef" for efficiency. This means that programs should generally not rely on any differences between "" and undef in field values. However, when working with large but sparse tables, programs should take care not to convert undef values to empty strings unnecessarily since the separate string objects consume considerably more memory than undef.
CONTRIBUTIONS
Corrections, bug reports, bug fixes, or feature additions are encouraged. Please send additions or patches with a clear explanation of their purpose. Consider making additions in the form of a subclass if possible.
I'm committed to bundling useful subclasses contributed by myself or others with this main distribution.
So, if you've got a subclass of Data::CTable (which should have a name like Data::CTable::YourClassName) and you would like it included in the main distribution, please send it along with a test script and I'll review the code and add it (at my discretion).
If you've got a module that uses, augments, or complements this one, let me know that, too, and I'll make appropriate mention of it.
SEE ALSO
The Data::CTable home page:
http://christhorman.com/projects/perl/Data-CTable/The implementation in CTable.pm.
The test.pl script, other subclasses, and examples.
The Data::ShowTable module.
The Data::Table module by Yingyao Zhou & Guangzhou Zou.
The perlref manual page.
AUTHOR
Chris Thorman <chthorman@cpan.org>
Copyright (c) 1995-2002 Chris Thorman. All rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.