NAME

savelogs - save/rotate/delete log files nicely

SYNOPSIS

savelogs saves your log files in a nice way (by default).

savelogs --postmovehook='/usr/local/bin/restart_apache' \
         --apacheconf=/www/conf/httpd.conf /var/log/messages

savelogs `cat list_of_logs_to_process.txt`

savelogs --loglevel=2 /var/log/maillog /var/log/cronlog \
         /var/log/messages

savelogs --config=/etc/savelogs.conf

savelogs --period=15 /var/log/messages

savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com

DESCRIPTION

savelogs is a flexible and robust log file archival system. Its logic is simple: move (rename) the log file, filter data from the log file, store the log file in an archive (via tar or gtar), and compress the archive (via gzip or compress). After successful compression, the original log file is deleted.

All of the above phases are optional. This means that you may simply delete files if you wish. Or you may simply compress existing log files. Or you may move files and add them to a tar file but leave the tar file uncompressed, etc. You pick ;o)

(If you just want to cut to the chase and don't care how savelogs works, see the "EXAMPLES" section near the bottom of this document.)

Savelogs Phases

The processing order may be abbreviated into these five phases:

move -> filter -> archive -> compress -> delete

any of which may be subtracted from the process order. In addition to these phases are some intermediate 'hooks' where you may supply an external program or shell command to invoke. This is useful if you're rotating web server log files and you need to HUP (restart) your web server after moving the logs (for example).

Subtracting phases is done in one of two possible ways. The first way is to specify it in the configuration file:

Process          move,archive,delete

which will move log files and archive them (but not filter or compress them). After successful archival, the original log files will be deleted.

The second way is to specify it on the command-line:

--process=compress,delete

which will simply compress log files (but not move, filter, or archive them).

In addition to the five phase processing options above, you may also employ the following abbreviations:

(no option specified): If you specify no process option, the default is move,compress.
none: Do none of the phases. This isn't a very useful option.
all: Do all of the phases.

An Overview

A typical savelogs session might begin by typing this command:

savelogs /var/log/messages

After which the following phases will execute:

move

The log file is renamed:

/var/log/messages --> /var/log/messages.010523

compress

The log file is compressed

/var/log/messages.010523 --> /var/log/messages.010523.gz

A Word About Paths

All paths you specify to savelogs should be relative to your home directory (if you're the root user--uid 0--your home directory is set to '/'). You do not need to use a tilde (~). You may assume that savelogs runs from your home directory and knows how to handle absolute paths.

If my real home directory were located in /usr/home/joe and I wanted to rotate the log file located in /usr/home/joe/var/log/messages, I would do something like this:

savelogs /var/log/messages

and savelogs would Do What I Mean.

The only exception to this are external commands given to postmovehook, postfilterhook and other such options. Paths you specify here really are full paths.

CONFIGURATION

Configuration file option format

savelogs will read its configuration options from a configuration file (which you must supply) or from the command-line. Creating a configuration file is easy: it is a simple Apache-style plaintext file that has options specified in this format:

Option          value

where Option is one of the options below and value is either a true/false; yes/no; on/off combination or some string value, such as a pathname and file (depending on the nature of the option).

Your distribution of savelogs may have included a sample configuration file for you to edit as you wish in ~/etc/savelogs.conf.sample.

Option processing order

Configuration options are first read from savelogs internal defaults, which are sprinkled throughout this document. Next savelogs reads its configuration file, if any is specified. Lastly, savelogs uses options from the command-line.

For example, the default value for the period directive is 10. If you ran savelogs like this:

savelogs --period /var/log/messages

every day for 10 days, you would have 10 archived log files.

If you have in your configuration file:

Period            5

and ran the same command above every day for 10 days, you'd only have 5 archived log files because the configuration file overrides savelogs internal defaults. Finally, if you had a configuration file with the previously mentioned value for period and ran this command:

savelogs --period=7 /var/log/messages

every day for 10 days, you would have 7 archived log files because command-line options override configuration file options (which override internal default values).

Available options

All options you may specify on the command-line you may also sepcify in a configuration file (except the configuration file directive itself). For example, if you had a cronjob that did this:

savelogs --process=delete \
--postmovehook="/usr/local/apache/bin/apachectl graceful" \
--apacheconf=/www/conf/httpd.conf

(which deletes all apache logs) you could make a nice configuration file (call it ~/etc/savelogs1.conf) that would do the same thing:

Process         delete
PostMoveHook    /usr/local/apache/bin/apachectl graceful
ApacheConf      /www/conf/httpd.conf

and then invoke your cron like this:

savelogs --config=/etc/savelogs1.conf

Case-sensitivity

A sample configuration file may be located in ~/etc/savelogs.conf.sample which you may copy and edit as you wish. Configuration file directives are case-insensitive:

Process    move,compress
PROCESS    move,compress
process    move,compress
pROceSs    move,compress

are all the same directive to savelogs. Configuration file values ARE case-sensitive.

ApacheConf    /www/conf/httpd.conf
ApacheConf    /WWW/conf/httpd.conf
ApacheConf    /www/CONF/httpd.conf
ApacheConf    /www/conf/Httpd.conf

are four distinct files, depending on the case-sensitivity of your operating system.

Testing configurations

When you are testing new configuration directives, use the dry-run option and watch the log output using the loglevel and logfile directives. This will help you avoid losing data unnecessarily.

You may also specify the settings option which will show you all the current settings after defaults, configuration file, and command-line options have been processed.

OPTIONS

Options given below are configuration directives that may appear in a configuration file or on the command-line. Options are case-insensitive, i.e., ApacheLog is the same as apachelog, though the values associated with the options are often case-sensitive (e.g., paths, filenames, etc.)

Options specified on the command-line should be prefixed with two hyphens. Some options do not make sense in a configuration file or need to occur before the configuration file is parsed such as config, help, or home.

Running Savelogs

help

Shows a brief usage statement and exits.

Example:

savelogs --help

version

Shows the current version of savelogs and exits.

Example:

savelogs --version

settings

Shows the current settings of savelogs and exits.

Example:

savelogs --settings --apacheconf=/www/conf/httpd.conf \
         /var/log/messages

dry-run

When used with the logfile and loglevel settings, dry-run will show you what will happen if you were to run savelogs with the current settings without actually doing it. This is a useful option to specify if you want to see what effect some changes might have, or to see which files are going to get archived with the current settings.

Note that savelogs running under the dry-run directive will sometimes produce errors that wouldn't occur during normal use. This can happen for a variety of reasons, mostly related to savelogs looking for files that don't yet exist, or archives that don't yet exist because they weren't actually created. In this respect, dry-run doesn't give you precisely what will happen, but it does give you a good idea. Use it with a grain of salt.

Example:

savelogs --dry-run --loglevel=2 /var/log/foo

home

Changes the default base location of where savelogs runs from. This is mostly a debugging utility. Consider this an advanced feature which should probably be ignored. Defaults to the process owner's home directory (which is almost always what you want).

Example:

Home                /usr/home/joe/usr/home/bob

config

Changes the default configuration file savelogs reads at startup. This should be done from the command-line or it won't have any effect.

Example:

savelogs --config=/etc/my_other_savelogs.conf

process=[move],[filter],[archive],[compress],[delete]

Tells savelogs which phases to execute. If you just want to move (rename) logs, do this:

savelogs --process=move /var/log/messages

and ~/var/log/messages will become ~/var/log/messages.yymmdd (where yymmdd are today's date).

For just removing logs, specify the delete option. You can get fancy:

savelogs --process=move,compress /var/log/messages

which renames the log file ~/var/log/messages to ~/var/log/messages.yymmdd and then compresses it, not filtering, archiving (i.e. putting into a separate tar file), or deleting it (the compress option also renames the file so that delete becomes useless since the file as it was no longer exists).

move,compress is the default value for the process option, so the above directive could have also been specified:

savelogs /var/log/messages

You may also specify all or none as shortcuts for

savelogs --process=move,filter,archive,compress,delete

and

savelogs --process=

respectively.

Savelogs Logging

savelogs has an internal logging facility that helps you diagnose problems, or just see what's going on. By default, savelogs writes to stdout (i.e., your screen if you're running this from a tty).

loglevel=#

Determines how verbose savelogs is when it's writing its own internal messages. Valid values are between 0 and 5 inclusive.

Example:

LogLevel            3

The general rule of thumb for savelogs logging is this:

Level  What will be logged
=====  ===================
0      no output except fatal errors
1      Level 0 + start/finish stats, errors
2      Level 1 + warnings, logfiles to process
3      Level 2 + chdir, filter, phase completion
4      Level 3 + phase core actions, phase beginning
5      Level 4 + everything else

The first few times you run savelogs, try a higher loglevel value to see what's happening with your log files. Once you're comfortable with how savelogs works, you may turn it down a few notches (level 1 is usually fine) so at least you can check to see if your cronjob actually ran ;o)

logfile=[stdout|stderr|/path/to/log]

savelogs, depending on the loglevel you've specified, writes what it's doing, such as moving, archiving, or deleting, etc. The logfile directive tells savelogs where to write all these messages.

The default value for logfile is stdout which means that your output will go to the screen unless you've redirected stdout. You may also specify stderr or the path to a file where you'd like messages to be appended.

Example:

LogFile             /var/log/savelogs.log

How Savelogs Finds Logs to Process

savelogs processes the logs you specify on the command-line (items on the command-line that are not recognized as options are assumed to be log files to process).

If no logs are specified (either on the command-line or in a configuration file using the following directives), savelogs will complain and show a 'usage' statement. To turn off the usage statement, use the gripe directive (to gripe is the default behavior):

savelogs --nogripe /no/such/log

To save wear on your finger tips and phosphor in your monitor, we recommend liberal use of the following configuration directives.

Log=/path/to/log

This works just like adding a file on the command-line, but is included so that you can put log files you want processed in a configuration file. It will also work on the command-line:

Example:

savelogs --log=/var/log/messages

is the same as:

savelogs /var/log/messages

which is also equivalent to a config file named '~/etc/savelogs.conf' with this single line:

Log                  /var/log/messages

and invoked like this:

savelogs --config=/etc/savelogs.conf

The log directive also accepts any standard csh-ish wildcard (e.g., *, ?, [n-m], etc.) for globbing. Globbing is where you specify a wilcard pattern and the argument list is expanded to all filenames that match the pattern. In savelogs, this pattern implicitly excludes files whose names end in '.tar', '.gz', or '.tgz' (so you don't have to worry about compressing already-compressed files).

This is useful if you have log files that are created dynamically and whose names you may not know precisely. For example, say you have a list of files in a directory:

somelog.010909.gz
somelog.01090a.gz
somelog.01090b.gz
somelog.01090c.gz
somelog.01090d.gz
somelog.01090e.gz
somelog.01090f.gz
somelog.011001
somelog.011002
somelog.011003
somelog.011004
somelog.011005
somelog.011006
somelog.011007
somelog.011008
somelog.011009
somelog.01100a
somelog.01100b
somelog.01100c

To compress these files (the ones that have not already been compressed) you can simply do this:

savelogs --log='/path/to/somelog.*'

The files that end in '.gz' are skipped (savelogs skips them internally).

Be sure to protect the asterisk (*) with quotes so that the current shell doesn't try to expand them. You could also do this in a configuration file:

Log             /path/to/somelog.*

NoLog=/path/to/log

This is the compliment to the Log directive: it removes logs from the list of logs to process. This is useful if you have a log or set of logs that is handled by a separate rotation program or needs special treatment at another time.

For example, if you have many log files listed in your Apache configuration file, you'll want to take advantage of the ApacheConf directive (see below). This will make your savelogs configuration file small and easy to understand:

## rename and compress all logs found in httpd.conf
ApacheConf       /www/conf/httpd.conf

This is great, except that there's this one log that you don't want savelogs to process. Before savelogs version 1.40, the only option you had was to list each log individually with the Log directive (i.e., you couldn't use the ApacheConf directive at all in such cases). Now, however, you can use the NoLog directive to exclude logs that have already been added to the list:

## rename and compress all logs found in httpd.conf
ApacheConf       /www/conf/httpd.conf

## ... and exclude joe's logs (joe-*_log matches
## joe-access_log and joe-error_log)
NoLog            /www/logs/joe-*_log

You may use full paths or you may use shell wildcard patterns, just like the Log directive.

If you have both Log and NoLog directives, the NoLog directive is processed last. This means that:

Log        /var/log/messages
NoLog      /var/log/messages

is the same as:

NoLog      /var/log/messages
Log        /var/log/messages

and that /var/log/messages will not be processed in either case.

Gripe|NoGripe

NoGripe tells savelogs not to complain about not finding any log files to process. By default, savelogs gripes about not finding any log files: if you forget to specify any logs (or any directives such as ApacheConf that find logs for you) savelogs will complain. Also, if you specify log files to process but none of them exist, savelogs will similarly complain.

When you turn on NoGripe, the complaining is stopped and savelogs exits happily.

Example:

savelogs --nogripe

or in your configuration file:

Gripe      no

ApacheConf=/path/to/httpd.conf

If you specify this option, giving it a valid httpd.conf file, savelogs will parse your Apache configuration file looking for standard log file directives. Any files found will be processed.

Example:

savelogs --apacheconf=/www/conf/httpd.conf

or in your configuration file:

ApacheConf          /usr/local/etc/httpd/conf/httpd.conf

Using the ApacheConf directive will tell savelogs to search through httpd.conf looking for all files associated with TransferLog, ErrorLog, AgentLog, etc. (all those listed in the ApacheLog directive) and process them.

ApacheHost

This option tells savelogs which logs to select out of the Apache configuration file (as specified by the ApacheConf directive) based on the Apache ServerName directive in the VirtualHost block. If this option is set, only logs for matching hosts will be rotated (this applies only to logs found in the Apache configuration file; other logs specified in other ways (e.g., on the command-line or via the Log directive) will be processed as usual).

Example:

savelogs --apacheconf=/www/conf/httpd.conf --apachehost=foo.com

or in your configuration file:

ApacheConf /www/conf/httpd.conf
ApacheHost foo.com

The ApacheHost directive may be specified multiple times to process logs for multiple virtual hosts. If no logs are found in the Apache VirtualHost block, no logs will be rotated for that virtual host.

ApacheLog

This option allows you to tell savelogs which logs to process in the httpd.conf file specified by apacheconf. The default value for the apachelog directive is:

TransferLog|ErrorLog|AgentLog|RefererLog|CustomLog

You may do something clever like this:

savelogs --apacheconf=/www/conf/httpd.conf --apachelog=TransferLog

which would archive all of your access_log files. Then after running this command, you could do this:

savelogs --process=delete --apacheconf=/www/conf/httpd.conf \
         --apachelog=ErrorLog

which would delete your error_log files.

In general, the fewer values you specify in the apachelog directive the faster savelogs will find your log files (though the speedup really is negligible, it may also save you from rotating logs you didn't want to).

ApacheLog           TransferLog|ErrorLog

would be sufficient for most people using combined Apache logs.

ApacheLogExclude

For those who want somewhat finer control of which logs get processed in their Apache configuration file, the apachelogexclude directive allows you to specify a Perl regular expression (which may simply be a string like 'error') of log files to exclude when processing logs. This way you could do something like this:

savelogs --apacheconf=/www/conf/httpd.conf --apachelogexclude=logs/bob

which would process all logs found in your httpd.conf file except log files whose names contain the string 'logs/bob'. Maybe Bob likes to rotate his logs using another program or system (conceivable, though unlikely).

Multiple occurances of apachelogexclude are allowed:

ApacheLogExclude          /dev/null
ApacheLogExclude          \|
ApacheLogExclude          logs/bob

which would exclude log files whose names contained '/dev/null' or 'logs/bob' from being processed. Any valid Perl regular expression will work, so:

ApacheLogExclude          ^/dev/null$

is not the same as the previous example. This example will only match log files whose name is exactly /dev/null, no more, no less.

By default, savelogs uses the following patterns to determine logs to exclude:

^/dev/null$
\| (this is a literal pipe character)

This means that by default savelogs will not attempt to archive a log whose name is '/dev/null' or whose name contains a pipe (|). If for some bizarre reason you wish to remove these defaults when you run savelogs, you can give an empty apachelogexclude option on the command-line:

% savelogs --apachelogexclude= --config=/etc/savelogs.conf

Logs that Apache writes via a pipe must be specified separately using the Log directive or on the command-line.

ApacheInclude

When specified, this directive tells savelogs to read the Apache configuration file (httpd.conf) and follow Inlcude directives (see http://httpd.apache.org/docs/mod/core.html#include for details).

It will work just like Apache does: it will look in directories (when a directory is given as the Include option), it will expand path wildcards (e.g., httpd_[bcd].conf will expand to httpd_b.conf, httpd_c.conf, and httpd_d.conf), and it will work with simple include files as well (e.g., virtual_hosts.conf).

Log files found in Include'ed configuration files will also be processed. savelogs has internal consistency checks to ensure that logs are not processed twice, neither are configuration files read twice (thus avoiding those annoying infinite loops).

Example:

% savelogs --apacheconf=/www/conf/httpd.conf --apacheinclude

or in your configuration file:

ApacheConf    /www/conf/httpd.conf
ApacheInclude yes

Grouping Log File Directives

Beginning with savelogs version 1.90, you can apply directives to groups of log files with the <Group> directive in a savelogs configuration file (the directive has no command-line equivalent). The following directives apply in a Group block:

period, ext, sep, datefmt, hourly, touch, chown, chmod, clobber, apachehost, log, disabled

As of version 1.90, the following directives are not honored, but may be at some point in the future:

size, filter, stem, stemhook, stemlink, nolog, apachelog, apachelogexclude

Group settings override any other settings found in the file, but are applied only to log files found within the Group block.

Example:

ApacheConf     /www/conf/httpd.conf
PostMoveHook   apachectl restart
Period         30

<Group>
  ApacheHost  www.sample1.tld
  ApacheHost  www.sample2.tld
  Period      10
  Chown       roger:staff
</Group>

<Group>
  ApacheHost www.sample3.tld
  ApacheHost www.sample4.tld
  Chown      fonzie
</Group>

Explanation:

The first Group block will rotate Apache logs for sample1.tld and sample2.tld, but it will only rotate 10 days worth of logs, since the Period directive inside the block is set to 10.

The second Group block will rotate Apache logs for sample3.tld and sample4.tld, and the full 30 day Period setting will apply to them.

Both groups will search /www/conf/httpd.conf and restart with apachectl restart.

Group blocks have a special directive Disabled that when set to true, will skip this block. Griping is also disabled when Disabled is set to true:

## this group will be completely skipped.
<Group>
  Disabled         1
  ApacheHost       www.foo.com
  Chown            roberto
</Group>

The purpose of the Disabled directive is to prevent ApacheConf from reverting to its behavior in the absence of any ApacheHost directives (i.e., it will parse the entire configuration file and process all hosts found). If this is what you desire, then remove or comment out your Group blocks instead of Disabled'ing them.

Known bugs related to the Group directive as of 1.91:

If a log file name found inside of a Group block also matches a NoLog directive outside of the Group block, it will be skipped as if it were outside the block.

If a log referenced inside of a Group block is processed outside of it also (found by some other directive, for example), the first one found will have precedence and the settings inside the Group block may be ignored.

If there are many files being processed, it is possible that in rare circumstances (namely, when processing spans midnight) that one set of logs may be named using one date and another set using another date. This only applies to non-periodic rotation.

Moving (Renaming) Files

touch

Touches the original file, creating it if necessary. This is useful for programs that log in "append-only" mode and do not create the log file if it is missing. Once savelogs has renamed a log file, touch will create the file if it does not exist, or reset the timestamp if it does.

size=kbytes

Logs smaller than kbytes will not be included in any processing. To override a default setting, specify --size with no arguments on savelogs command-line.

Size                      5000

will skip all log files smaller than 5 megs, regardless of other settings.

datefmt=string

Allows you to change how dates are formatted using the standard strftime system call. See strftime(1) for format string options. The default string is '%y%m%d'.

## renames logs like this: access_log.02-12-25
## Merry Christmas!
DateFmt                 %y-%m-%d

Some popular options are:

## 20020626 (26 June 2002)
DateFmt                 %Y%m%d

## 1.Mar.2002
DateFmt                 %e.%b.%Y

ext=string

Set the filename extension to 'string'. When a file is moved, it is renamed to the original filename plus the extension you specify. If no extension is specified, today's date is used in 'yymmdd' format. Options include today and yesterday.

You may not use ext in a configuration file with a value in backticks (e.g., the line:

## this directive will not work: don't use it!
Ext                  `/bin/date`

in a configuration file will not work). Any other (static) value for ext in a configuration file will work.

See also hourly below for information on how to modify ext even further. See datefmt above if you want to format your dates differently. ext is provided chiefly for completeness; its usefulness is limited except in special circumstances where savelogs can't offer a reasonable name for your log.

sep=char

The separator character to use when moving files. By default this character is a dot ('.'). Other favorites are underscore ('_') and hyphen/minus sign ('-').

Example:

Use an underscore character:

savelogs --sep='_' --process=move /var/log/foo

will rename ~/var/log/foo to ~/var/log/foo_yymmdd (where yymmdd is today's date).

Use no separator (just concatenate the extension with the filename):

savelogs --sep= --process=move /var/log/foo

will rename ~/var/log/foo to ~/var/log/fooyymmdd.

hourly

Adds a letter of the alphabet to the back of the filename extension. This is useful if you are rotating logs several times a day. For example, if you specified your extension (via ext) as 'foo', any log files rotated in the 10am hour will be named 'log.fook'. At 11am, all logs will be called 'log.fool' and so forth.

Example:

savelogs /path/to/log_file

This will rename the log file to log_file.yymmdd where yymmdd are today's year, month, and day of month.

savelogs --hourly /path/to/log_file

If you specify the hourly option (or in your configuration file, the Hourly directive), the log will be renamed to log_file.yymmddz where the z represents the current hour as a letter of the alphabet (0 = a, 1 = b, 2 = c, etc.).

You could also use the datefmt directive to get similar results (with much more flexibility to boot). If you rotate more often than once and hour, use the datefmt directive or use period to rotate logs with a unique number as the extension. period, discussed below, renames log files with a simple integer: log becomes log.0, the former log.0 becomes log.1 and so forth.

period[=count]

Renames the file based on a period, which is how frequently you run savelogs. If you specify period you may also optionally specify a 'count', which is how many log files to save using the period option:

savelogs --period=8 /var/log/messages

You may also use the count option, which is deprecated for backward compatibility and some possible future enhancements:

savelogs --period --count=8 /var/log/messages

If you do not specify a count value (either in period or count), a count of 10 is assumed.

The period option will rename the current log to logfile.0, the log that was previously named logfile.0 to logfile.1 and so on, much like newsyslog(8).

When you specify the period option, the process phases move and compress are assumed. If you also specify a process phase of filter, that will be honored also.

The period option will override any other sep and ext options specified, using the default dot ('.') for the separator and an integer for the extension.

The author also recommends you don't try to mix period named log files with other log files in the same directory, since savelogs may not be able to tell which logs are oldest based on the filename extension and destroy the wrong files. You can safely use any default savelogs extensions (e.g., the default 'today' or 'yesterday' extensions) or your own extension if your own extension contain at least one non-digit (0-9) or your own extension has 5 or more digits. If you meet either of these criteria in your own extension, you may feel confident about mixing logs.

For those familiar with Unix system adminstration, period works like newsyslog(8) with the B option specified in the newsyslog configuration file (the B option tells newsyslog to treat logs as binary files and not append the status message to the log).

An example use of the period option:

savelogs --touch --period=15 /var/log/maillog /var/log/messages

will move any existing ~/var/log/maillog and ~/var/log/messages to ~/var/log/maillog.0 and ~/var/log/messages.0 and compress them. By specifying the touch option, the original ~/var/log/maillog and ~/var/log/messages will be 'touched', recreating the files. When this command is run again, ~/var/log/maillog.0 is moved to ~/var/log/maillog.1 and ~/var/log/maillog is moved to ~/var/log/maillog.0.

count

Limits the number of logs saved using the period option. The internal default value for count is 10.

If you are using the period option, as of version 1.21 you may now simply specify the count as part of the period option:

savelogs --period=5 /var/log/messages

postmovehook=command

Runs an arbitrary system command you specify after moving files. If you are rotating Apache log files, you should use a command that will tell your web server to close its log file descriptors and re-open them (e.g., 'restart_apache').

Example:

savelogs --apacheconf=/www/conf/httpd.conf \
         --postmovehook='/usr/local/bin/restart_apache'

or even nicer in your configuration file:

PostMoveHook        /usr/local/bin/restart_apache

Paths in postmovehook are NOT relative to your home directory, as most other paths are. You should specify the full path to the file if the file is not in your environment's $PATH. The exception to this rule is the $LOG macro which, specified, will automatically be replaced with the current log file path. See Variables below for details.

postmovehook is one of two ideal phases to analyze your data before it is archived away (the other phase is the postfilterhook phase).

Variables

Some internal savelogs variables are available during the postmovehook phase. These variables are automatically interpolated by savelogs during execution and are guaranteed to contain some useful value.

$APACHE_CONF

Contains the full path to the Apache configuration file as specified with the apacheconf option.

savelogs --apacheconf=/www/conf/httpd.conf \
         --postmovehook='touch $APACHE_CONF'

$HOME

Contains the path to your home directory.

savelogs --postmovehook='$HOME/bin/myprogram'

$LOG

Contains the current log file being processed. If you wanted to run a command on each log file after it is moved, you may enter that command here (on one line). If the line is really long, consider putting it into a shell script that "wraps" all of your options.

PostMoveHook    /usr/local/bin/do_something_with_every $LOG

force-pmh

Executes the postmovehook command even if there are no logs to process. By default, savelogs will not execute the postmovehook command if there are no logs.

chown=uid:gid

After savelogs has renamed your log(s), you may wish to chown them to a new user or group. Use the chown option for this. If you don't specify a user, the user will not change. If you don't specify a group, the group won't change.

## chown the logs to joe:joegroup
Chown    joe:joegroup

## make the log files owned by the group with gid 500, keep the
## current owner
Chown    :500

## change the owner to root, leave the group alone
Chown    root:

Notice that the colon is necessary at all times.

The chown option executes after the PostMoveHook phase has completed.

chmod

After savelogs has renamed your log(s), you may wish to change the permissions of the logfile. Use the chmod option for this. Permissions should be specified in octal.

## make readable only by the owner
Chmod    0600

The chmod option executes after the PostMoveHook phase has completed.

stem

Like ext except the stem is used in addition to ext. After the move and postmovehook phases have completed, savelogs checks to see if you have defined a stemhook. If a stemhook has been defined, a symbolic link to the log is made using stem.

As an example, say you were processing ~/var/log/messages. During the stem phase, savelogs would do this:

messages.today -> messages.<ext>

(where <ext> is the extension you specified or today's date by
default)

Once the stemhook command has executed, the symbolic link (or hard link or copy, as specified by stemlink) is removed.

The stem related options were added in version 1.28.

stemhook=command

A command to execute, much like postmovehook, except that this phase is suited for log file analysis tools that require a predictably named, dead (i.e., no logging is currently being done to it) log file. urchin and analog are good examples of programs that require such logs.

An example use for stemhook would be:

% savelogs --stemhook="$HOME/usr/local/urchin/urchin" \
  /www/logs/access_log /www/logs/error_log

urchin should be instructed to operate on /www/logs/access_log.today and /www/logs/error_log.today. urchin should also be instructed to do nothing to the log files since we're allowing savelogs to manage them for us. Any changes resulting from the stemhook command to the log file will occur in the original log file.

The same variables for postmovehook are available for stemhook.

stemlink=linktype

Specify the type of link that should be made during the stem phase. The default is symlink. Other options are hard which creates hard links and copy which creates a copy of the original file. hard links are useful for stemhook commands that cannot process symbolic links. If your stemhook command modifies the log file, you may wish to choose the copy option which will be discarded after the stemhook command is executed.

Filtering

A filter is simply a program that generates something on STDOUT. They may be pipelines or other programs.

filter=filter_command

If the filter process option is specified (via the process directive), you should supply a program to filter your log (via the filter command), such as egrep or perl (or a pipeline or a shell script containing your commands, etc.).

If no filter command is given, the filter phase will be skipped (even if you specify --process=filter as the process directive). Consider following filter command:

--filter='/usr/bin/egrep -v "/images/" \$LOG'

When savelogs gets to its filter phase, it will open a pipe to the above command. Output from this command will be saved to a temporary file, then the temporary file will be renamed to replace the original log file.

Notice the strange $LOG variable. This is an internal savelogs variable that refers to the location of the log file savelogs is currently working on. It is automatically replaced with the right file during execution.

If you are supplying a filter command on the command-line, the backslash (\) in front of $LOG is necessary to tell the shell to not interpolate <$LOG> as a shell variable, but instead pass it along untouched to savelogs. The backslash is not necessary if you are specifying a filter directive in the configuration file:

Filter          /usr/bin/egrep -v "/images/" $LOG

For the sake of completeness, you can also chain filters via a pipeline, like this:

Filter    egrep -v "/images/" $LOG | egrep -v "(root|cmd)\.exe" -

The final '-' tells egrep to use stdin from the previous pipe for its input.

postfilterhook=command

Runs an arbitrary system command you specify after filtering files. See postmovehook for examples, including the Variables section.

force-pfh

Like force-pmh, this forces execution of the postfilterhook command even if there are no logs (and assuming savelogs has reached this phase).

Archiving and Compressing

gtar

tar

Specifies the location of the tar program to use. This defaults to whatever it can find on your system. You usually don't need to modify this option unless your tar or gtar program is not in your path.

Example:

Gtar                 /usr/sbin/gtar

If both gtar and tar are specified, gtar will be used.

archive

Specifies the name of the archive to which files will be appended.

This directive is somewhat tricky to understand. Under normal use, savelogs uses the name of the file being archived as the archive name. For example, if you were archiving a file named ~/var/log/messages, the name of the archive would be ~/var/log/messages.tar.

If you are archiving multiple files (which is common), each file will be stored in its own archive by name in the directory where the file is located. For example, if you had several files located in ~/var/log, each file would be stored in its own archive named filename.tar in the ~/var/log directory.

If you want to lump together all files in a particular directory into one archive, use the archive directive without any path information:

savelogs --archive=system.tar \
         /var/log/messages /var/log/proftpd /var/log/foo \
         /www/logs/access_log /www/logs/error_log

This will archive ~/var/log/messages, ~/var/log/proftpd, and ~/var/log/foo in a single file named ~/var/log/system.tar (which may later be compressed if you've so specified). ~/www/logs/access_log and ~/www/logs/error_log will also be lumped together in a file called ~/www/logs/system.tar.

If you want to lump together all files for this savelogs session into one archive, use the archive directive and specify the full path to the archive:

savelogs --archive=/var/tmp/logs.tar /var/log/messages \
         --apacheconf=/www/conf/httpd.conf

This will archive ~/var/log/messages and all log files found in the Apache configuration file into a single archive named ~/var/tmp/logs.tar (which may later be compressed).

If you wish to place the archive in your home directory, you may be tempted to just do this:

savelogs --archive=/logs.tar

This won't work. Because of the way savelogs tries to simplify things relative to your home directory, the leading slash is dropped and savelogs doesn't find any path information (and therefore places the archive in where the files are).

To put files in your home directory, preceed the archive command with a dot-slash:

savelogs --archive=./logs.tar

This won't put the archive in your current working directory, as some are wont to assume, but in your home directory. savelogs has no notion of a current working directory because it is always changing directories from your home to the directory where the log files are and back.

full-path

Specifies whether files stored in archives are full paths relative to your home directory or relative paths relative to the directory in which the file is found.

If full-path is not specified, paths are stored in the tar file relative to their parent directory.

Example:

savelogs /var/log/messages

will create an archive with the following file in it:

messages

while

savelogs --full-path /var/log/messages

will create an archive with the following file in it:

var/log/messages

When you extract this file later on, the paths will be created for you if they don't exist, which may not be what you want (but, of course, it may be what you want which is why we have this directive).

gzip

compress

uncompress

Specifies the location of the gzip/compress/uncompress binaries for decompressing files (files ending with '.gz' or '.Z'). Use of compress and uncompress is deprecated. If your system has a gzip program in a directory that is not in your $PATH variable, specify its location with this directive. If gzip and compress are specified, gzip will be used.

By default (if none of the above options are specified), savelogs will search for a gzip binary in your path and use it.

clobber

If a compressed archive already exists along side a non-compressed archive (e.g., archive.tar and archive.tar.gz), and you've instructed savelogs to compress archive.tar, some compression programs (like gzip) will ask you for confirmation before overwriting existing files.

To get around this, savelogs by default enables the 'force' option on compression programs (usually -f). This way, if you're running savelogs from a cron job or another method where there is no controlling terminal, savelogs keeps running.

If you're running savelogs interactively (i.e., from a tty) and want savelogs to prompt you to overwrite existing compressed files, specify the noclobber option:

Example:

savelogs --noclobber

or in your configuration file:

Clobber             no

EXAMPLES

The author recommends liberal use of the dry-run option when testing these examples or when making big changes to your configuration file or command-line options. Doubly-so when you have the delete process option enabled. There's no 'undelete' for UNIX.

Archiving a single log file

savelogs /path/to/log_file

By default, savelogs will move, and compress a log file. This is its simplest use. If this command is run daily, the result will be a file name like the old file with a yymmdd extension. This file will be compressed.

Nuking logs

You can use savelogs (contrary to its name) to just wipe out log files and reclaim the disk space. If you've got a couple of files that from time to time just get too big and there's really no valuable information in them, do something like this:

savelogs --process=delete /path/to/log_file1 /path/to/log_file2

If you want to nuke all Apache log files, do something like this:

savelogs --process=delete --postmovehook=/usr/local/bin/restart_apache \
--apacheconf=/www/conf/httpd.conf

When you specify only the delete as a process option, no logs are moved, archived, or compressed. They're just deleted.

Compressing logs daily

Compressing logs daily is easy:

savelogs /var/log/messages

will make:

-rw-r--r--  1 test  vuser  751327 Jul  6 12:48 messages

become:

-rw-r--r--  1 test  vuser   84625 Jul  6 12:48 messages.010706.gz

Compressing logs daily if needed

savelogs --size=5000 /var/log/messages

will only compress the log if the size of the file is 5000 kilobytes (5 megabytes) or larger.

Using periodic log rotation

You want to save 3 days worth of Apache log files and 5 days worth of system log files. You might try the following lines in your crontab:

1 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \
--postmovehook=/usr/local/bin/restart_apache --period=3 \
--apacheconf=/www/conf/httpd.conf

5 0 * * * $HOME/usr/local/bin/savelogs --logfile=/var/log/savelogs \
--period=5 /var/log/messages /var/log/ftp.log

Most crontab files require that no lines wrap, so you'd need to make sure to keep everything on one line.

Your ~/www/logs directory may look something like this after a week:

access_log
access_log.0.gz
access_log.1.gz
access_log.2.gz
error_log
error_log.0.gz
error_log.1.gz
error_log.2.gz

and your system logs directory:

messages
messages.0.gz
messages.1.gz
messages.2.gz
messages.3.gz
messages.4.gz
ftp.log
ftp.log.0.gz
ftp.log.1.gz
ftp.log.2.gz
ftp.log.3.gz
ftp.log.4.gz

Archiving all logs

Most people want to group their log files in an archive. This makes storing them and retrieving them later for post-processing simple and efficient. Your directory tree might look like this:

usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/access_log
usr/local/etc/httpd/logs/bar/error_log
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/access_log
usr/local/etc/httpd/logs/baz/error_log
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/access_log
usr/local/etc/httpd/logs/biz/error_log
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/access_log
usr/local/etc/httpd/logs/buz/error_log
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/access_log
usr/local/etc/httpd/logs/foo/error_log

Issue this command:

savelogs --process=all --apacheconf=/www/conf/httpd.conf

and your directory tree now looks like this:

usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/access_log.tar.gz
usr/local/etc/httpd/logs/bar/error_log.tar.gz
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/access_log.tar.gz
usr/local/etc/httpd/logs/baz/error_log.tar.gz
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/access_log.tar.gz
usr/local/etc/httpd/logs/biz/error_log.tar.gz
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/access_log.tar.gz
usr/local/etc/httpd/logs/buz/error_log.tar.gz
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/access_log.tar.gz
usr/local/etc/httpd/logs/foo/error_log.tar.gz

Inside of each compressed archive is a single file:

access_log.010523

which is the old log renamed with today's date. When you run this command again (e.g., from a cron job) tomorrow, you'll see the same list of files above, except that inside each compressed archive is an additional file:

access_log.010523
access_log.010524

Archiving all logs per directory

Say you want to group together all logs in a single directory in one archive. Your directory tree might look like this:

usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/access_log
usr/local/etc/httpd/logs/bar/error_log
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/access_log
usr/local/etc/httpd/logs/baz/error_log
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/access_log
usr/local/etc/httpd/logs/biz/error_log
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/access_log
usr/local/etc/httpd/logs/buz/error_log
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/access_log
usr/local/etc/httpd/logs/foo/error_log

Try this:

savelogs --process=all --archive=logs.tar --apacheconf=/www/conf/httpd.conf

and your directory tree now looks like this:

usr/local/etc/httpd/logs
usr/local/etc/httpd/logs/bar
usr/local/etc/httpd/logs/bar/logs.tar.gz
usr/local/etc/httpd/logs/baz
usr/local/etc/httpd/logs/baz/logs.tar.gz
usr/local/etc/httpd/logs/biz
usr/local/etc/httpd/logs/biz/logs.tar.gz
usr/local/etc/httpd/logs/buz
usr/local/etc/httpd/logs/buz/logs.tar.gz
usr/local/etc/httpd/logs/foo
usr/local/etc/httpd/logs/foo/logs.tar.gz

Inside each 'logs.tar.gz' are two files:

access_log.010523
error_log.010523

Archiving all logs per session

Say you want to lump all logs in a savelogs session into a single archive. Specify the archive option with a full path, like this:

savelogs --process=all --archive=/tmp/all_logs.tar \
         --apacheconf=/www/conf/httpd.conf /var/log/messages

This will create a single file /tmp/all_logs.tar.gz that contains all logs found in httpd.conf as well as /var/log/messages.

Filtering logs in place

Maybe your log file fills up with garbage entries that you want to clean out daily. You can use the filter option to trim your log in place:

savelogs --process=filter \
         --filter='/usr/bin/egrep -v "(root|cmd)\.exe" \$LOG' \
         --postfilterhook='/usr/local/bin/restart_apache' \
         --apacheconf=/www/conf/httpd.conf

This will clean out some Windows worm requests from your Apache log files.

BUGS/CAVEATS

savelogs will not properly interpolate backticks inside the configuration file. Because backticks are shell operators, they must be visible to the shell (meaning they have to be done on the command-line). An example is the ext command where the command-line option might look like this:
```
savelogs --ext=`smalldate -m`
```
there is no corresponding configuration file entry.
You must obey shell escaping rules when you're doing filter commands on the command-line. Sometimes these can be tricky. For example, double quotes will do different things than single quotes (I recommend single quotes because you can then use the $LOG variable without big headaches). Using a config file sometimes helps.
You can't have multiline commands in the configuration file. This is on the "to do" list.

WISHLIST/TODO

Would be nice to use the VirtualHost 'User' directive for the chown option. Maybe a 'ApacheChown' option.
Optimize the ApacheHost directive to jump out of the Apache config parsing once all hosts have been found (can you have a VirtualHost block appear twice with the same ServerName directive?)

ACKNOWLEDGEMENTS

Thanks to Jeroen Latour (cpantester@calaquendi.net) for working with me on getting all the tests to run cleanly on his Cobalt box. It should now test cleanly on many other platforms because of his patience.

AUTHOR

Scott Wiersdorf, <scott@perlcode.org>

COPYRIGHT

1 POD Error

The following errors were encountered while parsing the POD:

Around line 2460:: =back doesn't take any parameters, but you said =back 4

To install savelogs, copy and paste the appropriate command in to your terminal.

cpanm

cpanm savelogs

CPAN shell

perl -MCPAN -e shell
install savelogs

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

Savelogs Phases

An Overview

A Word About Paths

CONFIGURATION

Configuration file option format

Option processing order

Available options

Case-sensitivity

Testing configurations

OPTIONS

Running Savelogs

Savelogs Logging

How Savelogs Finds Logs to Process

Grouping Log File Directives

Moving (Renaming) Files

Filtering

Archiving and Compressing

EXAMPLES

Archiving a single log file

Nuking logs

Compressing logs daily

Compressing logs daily if needed

Using periodic log rotation

Archiving all logs

Archiving all logs per directory

Archiving all logs per session

Filtering logs in place

BUGS/CAVEATS

WISHLIST/TODO

ACKNOWLEDGEMENTS

AUTHOR

COPYRIGHT

Module Install Instructions