/ 
Snapback2-0.3
River stage zero No dependents
/ snapback2

NAME

snapback2 -- rsync and hard-link backup script

SYNOPSIS

snapback2 [-c configfile] [-df] [-p PAT] [-P PAT] [configfile-base]

DESCRIPTION

Snapback2 does backup of systems via ssh and rsync. It creates rolling "snapshots" based on hourly, daily, weekly, and monthly rotations. When it runs for some period of time, you will end up with a target backup directory that looks like:

drwx--x--x   81 106      staff    4096 Jan  1 05:54 daily.0
drwx--x--x   81 106      staff    4096 Dec 31 05:55 daily.1
drwx--x--x   81 106      staff    4096 Dec 30 05:55 daily.2
drwx--x--x   81 106      staff    4096 Dec 29 05:54 daily.3
drwx--x--x   81 106      staff    4096 Dec 28 05:53 daily.4
drwx--x--x   81 106      staff    4096 Dec 27 05:53 daily.5
drwx--x--x   81 106      staff    4096 Dec 26 05:53 daily.5
drwx--x--x   81 106      staff    4096 Jan  1 05:54 hourly.0
drwx--x--x   81 106      staff    4096 Dec 31 17:23 hourly.1
drwx--x--x   81 106      staff    4096 Jan  1 05:54 monthly.0
drwx--x--x   81 106      staff    4096 Dec  1 05:54 monthly.1
drwx--x--x   81 106      staff    4096 Dec 28 05:53 weekly.0
drwx--x--x   81 106      staff    4096 Dec 21 05:53 weekly.1
drwx--x--x   81 106      staff    4096 Dec 14 05:53 weekly.2
drwx--x--x   81 106      staff    4096 Dec  7 05:53 weekly.3

You might think this would take up lots of space. However, snapback2 hard-links the files to create the images. If the file doesn't change, only a link is necessary, taking very little space. It is possible to create a complete yearly backup in just over 2x the actual storage space consumed by the image.

See www.http://www.mikerubel.org/computers/rsync_snapshots/ for detailed information on the principles used.

Apache-style configuration files are used. A configuration file for a basic backup might look like

Hourlies    4
Dailies     7
Weeklies    4
Monthlies  12
AutoTime   Yes

AdminEmail mikeh@perusion.com

LogFile    /var/log/snapback.log
Exclude *debug
Exclude core.*
SnapbackRoot /etc/snapback

Destination /mnt/backup1

<Backup mike.perusion.com>
	Destination /space
	Hourlies 2
	Directory /home/mike/
	Directory /etc/
	Directory /var/lib/mysql/
	Directory /var/lib/pgsql/
	<BackupDirectory /home/work/>
		Hourlies 4
	</BackupDirectory>
</Backup>

<Backup jean.perusion.com>
	Hourlies 2
	Directory /home/jean/
	Directory /var/mail/
</Backup>

The above configuration will be discussed in detail below.

Pre-requisites

This script is only tested on Linux at this point, but should operate on any UNIX-based computer with the following:

Gnu toolset, including cp, rm, and mv
rsync 2.5.7 or higher
ssh
Perl 5.8 or higher
Perl module Config::ApacheFormat

CONFIGURATION

The configuration directives use Apache format, thanks to the Config::ApacheFormat module.

Inheritance is on -- a sub-block inherits all configuration directives above it. You can override any configuration directive within the block container.

If not specified with the -c command line option, the following files are checked for existence in order and the first one found is used:

/etc/snapback2.conf
/etc/snapback/snapback2.conf
/etc/snapback.conf
/etc/snapback/snapback.conf

Block directives

There are two blocks supported:

Backup

This specifies the host computer which will be backed up, and it is given an internet address as a parameter (host name or IP address). Only one block can be specified per hostname per configuration file, but it is possible to make the parameter a pseudo-host by overriding the address with the BackupHost directive.

For example:

	<Backup pseudo>
		BackupHost jean.perusion.com
		Hourlies 2
		Directory /etc/
	</Backup>
 
	<Backup jean.perusion.com>
		Hourlies 4
		Directory /home/jean/
	</Backup>

Both backup configurations use jean.perusion.com as the target machine.

BackupDirectory

This is contained within a Backup block, and is an alternate method of specifying a Directory. The parameter is the directory name. The use for this is specifying different backup parameters for that directory only.

For example:

## directives are not case-sensitive
<backup perusion.com>
	Destination /mnt/backup1
	Hourlies 4
	Directory /etc/
	Directory /var/lib/mysql/
	<BackupDirectory /var/lib/catalogs/shop/orders>
		Destination /mnt/backup1/orders
		Hourlies 24
	</Backup>
</backup>

This allows a real hourly backup of a directory where frequent backups are needed, while cutting down the frequency of the main backup.

Other directives

The rest of the directives control various facets of the backup.

AdminEmail

Email address to mail errors (or results if AlwaysEmail is set). Default blank.

AlwaysEmail

Always email results even if there is not an error. Target address is set in AdminEmail.

AutoTime

If set to yes, which is the default, the time of the previous backup is checked and backup is only done if appropriate. The formula for checking appropriateness is:

(24 / Hourlies - 0.5) * 60 * 60 < now - mtime

where Hourlies is the value of the Hourlies directive, now is the current time in seconds since 1970, and mtime is the modification time in seconds since 1970.

For example, if Hourlies is set to 4 and the script is called every hour, it will only run the backup if the timestamp of the latest hourly backup is more than 5.5 hours old. Obviously this means your backup should take less than an hour to complete; if you have an extremely extensive backup setup that could exceed an hour you will want to break it up into separate runs or make the script call frequency greater.

ChargeFile

The file where byte counts are logged in the format

host:YYYYMMDD:N

where YYYYMMDD is the date in quasi-ISO format and N is the number of bytes read. This allows monitoring of bandwidth used for a particular backup host, possibly for a bandwidth-based charging mechnism.

Cp

Full path to the GNU cp program. Default is /bin/cp.

CreateDir

Controls whether Destination directories will be created automatically. A Boolean (Yes/No) directive. Default is Yes.

DailyDir

The root name of the daily backup directory, default daily. Not normally changed.

Debug

Sets debug output on. Equivalent to passing the -d option on the command line.

In the future, the debug level may vary with the number passed. At the moment, there is only one debug level.

Example:

Debug  4
DebugFile

Normally debug output goes to STDERR, but if you want it sent to a file specify the file with this directive.

Example:

DebugFile /tmp/snapback.debug
Destination

The destination directory for the backup. A subdirectory of the host address is created (providing CreateDir is yes, the default), and then the first part of the Directory is created. The hourly/daily/weekly directories are then maintained there.

For example, this configuration:

Destination /mnt/backup1
<Backup perusion.com>
	Directory /var/lib/mysql/
	Directory /home/mike/
	Directory /home/work/
</Backup>

will create the following directories on its first run:

/mnt/backup1/perusion.com/var/lib/mysql/hourly.0
/mnt/backup1/perusion.com/var/lib/mysql/daily.0
/mnt/backup1/perusion.com/home/mike/hourly.0
/mnt/backup1/perusion.com/home/mike/daily.0
/mnt/backup1/perusion.com/home/work/hourly.0
/mnt/backup1/perusion.com/home/work/daily.0

If the run was made on Sunday, a weekly.0 will be created. If the run was made on the first day of the month, a monthly.0 will be created.

DestinationList

A list of destinations that will be checked for the proper backup place. If this is in force, the Destination directive will be ignored.

Set to the places where you want backup to go, i.e.:

DestinationList  /mnt/backup1 /mnt/backup2

It checks the timestamp of the hourly.0 directory at each target, and selects the least-recently-used one for the target.

This allows spreading the backup over multiple disks for greater reliablility.

If you want to set a single destination in a Backup sub-block, overriding a global DestinationList, either set

DestinationList none
Destination     /real/destination

or just set the DestinationList directive to the single directory.

The number of Hourlies, Dailies, Weeklies, and Monthlies still applies at each target.

For example, this configuration:

DestinationList /mnt/backup1 /mnt/backup2
Hourlies 2
<Backup perusion.com>
	Directory /var/lib/mysql/
	Directory /home/mike/
</Backup>

will create the following directories on its first run:

/mnt/backup1/perusion.com/var/lib/mysql/hourly.0
/mnt/backup1/perusion.com/var/lib/mysql/daily.0
/mnt/backup1/perusion.com/home/mike/hourly.0
/mnt/backup1/perusion.com/home/mike/daily.0

this on its second:

/mnt/backup2/perusion.com/var/lib/mysql/hourly.0
/mnt/backup2/perusion.com/var/lib/mysql/daily.0
/mnt/backup2/perusion.com/home/mike/hourly.0
/mnt/backup2/perusion.com/home/mike/daily.0

and this on its third:

/mnt/backup1/perusion.com/var/lib/mysql/hourly.0
/mnt/backup1/perusion.com/var/lib/mysql/hourly.1
/mnt/backup1/perusion.com/var/lib/mysql/daily.0
/mnt/backup1/perusion.com/home/mike/hourly.0
/mnt/backup1/perusion.com/home/mike/hourly.1
/mnt/backup1/perusion.com/home/mike/daily.0

etc.

Directory

The directory to be backed up. It will be created on the Destination, and hourly.N, daily.N, weekly.N, and monthly.N directories will be maintained there. See also BackupDirectory.

Only valid within a <Backup host> block.

This directive is a multiple directive, and it can be set as many times as needed.

A trailing slash is always added if necessary unless LiteralDirectory is set to yes (which it should not be unless you are an rsync expert).

Exclude

File patterns to be excluded. Passed to rsync with the --exclude-pattern option. See the documentation for rsync.

It is normal to exclude core files, for example:

Exclude core
Exclude core.*

This directive is a multiple directive, and it can be set as many times as needed.

HourlyDir

The root name of the hourly backup directory, default hourly. Not normally changed.

Include

Specify a file or directory to include from. If the specification is a directory, it will include all files in the directory:

Include clients

That is the equivalent of "Include clients/*", though that syntax is not supported due to Config::ApacheFormat limitations.

To include only a single file:

Include clients/something.conf

The file specification is based in ServerRoot.

LiteralDirectory

Normally snapback automatically appends a / to the source directory name to make rsync work properly. If LiteralDirectory is set to Yes, then it will not do that, with unpredictable results.

Default is No, and you should think long and hard before changing it. It is possible to construct useful backups without a trailing slash, but you will have to be an rsync expert.

In other words, don't mess with this.

LogFile

The name of the file where backup runs are logged. Default is /var/log/snapback.log.

MonthlyDir

The root name of the monthly backup directory, default monthly. Not normally changed.

Mv

Full path to the GNU mv program. Default is /bin/mv.

MyHost

The name of the backup host itself, used only for reporting purposes. Default is the result of Sys::Hostname::hostname().

MustExceed

The amount of time the current time must exceed the previous backup modification time before a backup will be done, when AutoTime is on. Default is 5 minutes.

Rm

Full path to the GNU rm program. Default is /bin/rm.

Rsync

Full path to the rsync program. Default is rsync.

RsyncOpts

The options for the rsync program. Default is

-avz -e ssh --force --delete-excluded  --one-file-system --delete

Play with this at your own risk. 8-)

SendMail

Path to the sendmail program, used for emailing errors (or results with AlwaysEmail). Default is /usr/sbin/sendmail.

The program to use must accept the -t option.

SnapbackRoot

The root directory where any Include directives will be based. Default is /etc/snapback.

Example:

SnapbackRoot  /opt/snapback
WeeklyDir

The root name of the weekly backup directory, default weekly. Not normally changed.

OPTIONS

There are a few command line options that can be called:

-c configfile

The complete path to the configuration file to use. If not specified, defaults to:

/etc/snapback2.conf
/etc/snapback/snapback2.conf
/etc/snapback.conf
/etc/snapback/snapback.conf

Fails if one of those is not found.

-d

Turns on debug output. If DebugFile is set, will go there, otherwise goes to the standard error output.

-p PATTERN

A pattern to apply to the <Backup foo> block. Only hosts matching the pattern will be operated on.

To backup all hosts in the perusion.com domains, but not any others, do:

snapback2 -p perusion.com
-P PATTERN

A pattern to apply to any Directory or <BackupDirectory>. Only directories matching the pattern will be operated on.

To backup all /var/ diretories, do:

snapback2 -P /var/

Other directories will be ignored.

To backup /var/ directories in the perusion.com domains, but not any others, do:

snapback2 -p perusion.com -P /var/

AUTHOR AND CREDITS

Mike Heins, <mikeh@perusion.com>.

This script is heavily cribbed from the original snapback done by Art Mulder. Some of the routines, particularly do_rotate() and do_backup(), are much the same; the main program flow and configuration is completely redone.

The initial principles were elucidated by Mike Rubel.

1 POD Error

The following errors were encountered while parsing the POD:

Around line 204:

You forgot a '=back' before '=head2'