NAME
record_consolidator.pl - consolidate old RTG data to reduce database size.
VERSION
1.0.0
SYNOPSIS
record_consolidatorOptions for controlling consolidation are found in rtgreport.conf.
DESCRIPTION
record_consolidator.pl - Consolidates RTG data into summary records. We monitor tens of thousands of network ports and have hundreds of gigabytes of RTG data. When disk space or performance becomes an issue, the RTG supplied tool is a pruning script that simply deletes old data. Rather than deleting that data, I wanted to condense it, preserving the essence of the data. I borrowed a concept from Tobias Otiker and his RRD databases--rather than discard old data, reduce its granularity.
In testing, I discovered that 15 minute averages still work perfectly with unaltered versions of the RTG CGI application. Any more than 15 minute averages and graphs stop rendering. Dropping from 5 to 15 minute averages reduces the disk space required by old data by 66%.
How you use the consolidator script will be dictated by your business needs. I suggest using the consolidator script with two sets of settings. Old data that is no longer needed for graph publication should be pruned to 60 minute averages. A setting of 190/60 will do this. Then have another tier of pruning where you consolidate data older than 2-3 months to 15 minute averages. A setting of 90/15 would do that.
CONFIGURATION
The consolidator script is controlled by two settings in the rtgreport.conf config file:
[Pruning]
days = 190
interval = 60Days controls what data you want to alter. A setting of 190 as shown will only condense data older than 190 days. Interval controls what interval the consolidated records will be. A setting of 15 will condense the data to 15 minute averages (66% reduction). A setting of 60 as shown will reduce the data to 60 minute averages for a 91% reduction in disk space.
DEPENDENCIES
Config::Std
Date::Calc
Date::Format
DBIx::SimpleAUTHOR
Matt Simerson <msimerson@cpan.org>
CHANGES
LICENSE AND COPYRIGHT
Copyright (c) 2008 Layered Technologies, Inc. All rights reserved.
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.