psinib - Perl Snapshot Is Not Incremental Backup

-= Download perl script and install it anywhere on your system. Includes this documentation page if you invoke it with perldoc psinib.pl.
psinib.pl 25 Kb
-= Optional subjects file for --email option
subjects.txt 539 bytes
-= Whole archive with ChangeLog, TODO and example mountscript
psinib-1.0-rc3.tar.gz 12 Kb
-=> Latest source is always available from Subversion repository

Index

NAME

Top

psinib - Perl Snapshot Is Not Incremental Backup

SYNOPSIS

Top

./psinib.pl [OPTION]... [mount script]

DESCRIPTION

Top

Backup samba or Windows shares using LinNeighborhood mount script as configuration

Option can be one of more of following:

--backupdest=dir

Specify backup destination directory (default is /backup/)

--noping

Don't use ping to check if host is up (default is to use TCP SYN to CIFS port)

--verbose -v

Increase verbosity level. Default is 1 which prints moderate amount of data on STDOUT and STDERR.

--quiet -q

Decrease verbosity level

--email=email@domain

Send e-mails instead of dumping errors to STDERR. Useful for cron jobs.

--max_share_size=50000

Specify maximum size of share to backup (in Kb). This example will skip backuping shares which have more than 50Mb. By default this option is disabled.

--max_file_size=5000

Specify maximum size of file to backup (in Kb). This example will limit file size to 5Mb. By default this option is disabled.

--min_free_space=0

Specify minumum free space on destination file system to start backup. If space is not availabe, backup will abort. By default, 100Mb of free space is required on destination. You can set this value to 0 to disable this check.

--diff

Show compact diff-like output for changed files

--only=part_of_share_name

Backup just shares which match part_of_share_name in this run.

--help -h

Display usage page.

Idea of this script is to produce tool which will be easy enough to use for any Linux administrator which has a task to backup Windows workstations or samba shares. Backup is done to central disk space on Linux server running psinib. It's organized in multiple directories named after:

* server which is sharing files to be backed up
* name of share on server
* dated directory named like standard ISO date format (YYYYMMDD).

In each dated directory you will find snapshot of all files on exported share on that particular date.

You can also use symlink latest which will lead you to last completed backup. After that you can use some other backup software to transfer snapshot to tape, CD-ROM or some other media, or just re-export it via samba so that users can do restore of files on specific date themselves.

Design considerations

Since taking of share snapshot every day requires a lot of disk space and network bandwidth, psinib uses several techniques to keep disk usage and network traffic at acceptable level:

- usage of hard-links to provide same files in each snapshot (as opposed to have multiple copies of same file)
- usage of file size, atime and mtime to find changes of files without transferring whole file over network (just share browsing is transfered over network)
- usage of .md5sum files (compatible with command-line utility md5sum) to keep file between snapshots hard-linked

CONFIGURATION

Top

Configuration file format is simple:

smbmount //WIN_BOX/data /home/dpavlin/mnt/WIN_BOX/data/ -o username=dpavlin%password,fmask=644,dmask=755,uid=1000,gid=1000,debug=0,workgroup=HOME

This file is produced by LinNeighborhood http://www.bnro.de/~schmidjo/ utility. So to make backup of your Windows share, it's enough just to mount it using LinNeighborhood, and use Export Mountscript to produce mountscript which is at the same time configuration file for psinib.

HACKS, TRICKS, BUGS and LIMITATIONS

Top

This chapter will have all content that doesn't fit anywhere else.

Can snapshots be more frequent than daily?

There is not real reason why you can't take snapshot more often than once a day. Actually, if you are using psinib to backup Windows workstations you already know that they tend to come-and-go during the day (reboots probably ;-), so running psinib several times a day increases your chance of having up-to-date backup (psinib will not make multiple snapshots for same day, nor will it update snapshot for current day if it already exists).

However, changing psinib to produce snapshots which are, for example, hourly is a simple change of $DIR_TIME_FMT which is currently set to '%Y%m%d' (see strftime documentation for explanation of that format). If you change that to '%Y%m%d-%H you can have hourly snapshots (if your network is fast enough, that is...). Also, some of messages in program will sound strange, but other than that it should work. You have been warned.

Do I really need to share every directory which I want to snapshot?

Actually, no. Due to usage of Filesys::SmbClient module, you can also specify sub-directory inside your share that you want to backup. This feature is most useful if you want to use administrative shares (but, have in mind that you have to enter your Win administrator password in unencrypted file on disk to do that) like this:

smbmount //server/c$/WinNT/fonts /mnt -o username=administrator%win

After that you will get directories with snapshots like:

server/c_WinNT_fonts/yyyymmdd/....

Won't I run out of disk space?

Of course you will... Snapshots and logfiles will eventually fill-up your disk. However, you can do two things to stop that:

Clean snapshot older than x days

You can add following command to your root crontab:

find /backup/isis_backup -type d -mindepth 3 -maxdepth 3 -mtime +11 -exec rm -Rf {} \;

I assume that /backup/isis_backup is directory in which are your snapshots and that you don't want to keep snapshots older than 11 days (that's -mtime +11 part of command).

Rotate your logs

I will leave that to you. I relay on GNU/Debian's logrotate to do it for me.

What are YYYYMMDD.partial directories?

If there isn't latest symlink in snapshot directory, it's pretty safe to assume that previous backup from that day failed. So, that directory will be renamed to YYYYMMDD.partial and snapshot will be performed again, linking same files (other alternative would be to erase that dir and find second-oldest directory, but this seemed like more correct approach).

I can't connect to any share

Please verify that nmblookup (which is part of samba package) is in /bin or /usr/bin. Also verify that nmblookup returns IP address for your server using:

$ nmblookup tvhouse querying tvhouse on 192.168.34.255 192.168.34.30 tvhouse<00>

If you don't get any output, your samba might not listen to correct interface (see interfaces in smb.conf).

Aren't backups boring?

No! If you have subjects.txt in same directory as psinib.pl you can get various funny subjects in your mail. They change over time as long as you ignore your backup.

Why all those limit on backup share and files?

Well, if share is too big or disk space on destionation directory is too small, this might create denial of service attack on your server. By default, only free space on destination directory is checked.

AUTHOR

Top

Dobrica Pavlinusic <dpavlin@rot13.org>

http://www.rot13.org/~dpavlin/

LICENSE

Top

This product is licensed under GNU Public License (GPL) v2 or later.

ChangeLog

2008-10-26 10:33:49 dpavlin r51

/trunk/psinib.pl: We never want to pull .md5sum from client back to server, because it should end up on client only if we restored our backup on it. And if it does, it's probably wrong...

2008-01-22 18:04:36 dpavlin r50

/trunk/psinib.pl: handle case where local copy of directory which is backuped is empty

2007-02-15 12:32:09 dpavlin r49

/trunk/psinib.pl: don't show ignored files when using --diff without --verbose

2007-02-14 15:59:40 dpavlin r48

/trunk/psinib.pl: added --only to backups just some shares, documented --diff

2007-02-14 02:00:23 dpavlin r47

/trunk/psinib.pl: don't re-create local directories over and over again

2007-02-14 02:00:06 dpavlin r46

/trunk/fdupes-ln.pl: minor fix

2007-02-14 01:55:36 dpavlin r45

/trunk/psinib.pl: make Perl::Critic not critical, fixed temporary name creation [1.0-rc3]

2007-02-14 01:42:48 dpavlin r44

/trunk/psinib.pl: added long forgot --diff option which output just compact changes

2007-02-14 01:30:09 dpavlin r43

/trunk/fdupes-ln.pl: make Perl::Critic happy

2007-02-14 01:29:56 dpavlin r42

/trunk/Makefile: critic target

2007-02-14 00:54:16 dpavlin r41

/trunk/fdupes-ln.pl: skip directories (fdupes would groak, so why let him?)

2007-02-13 23:19:53 dpavlin r40

/trunk/fdupes-ln.pl: proper pod docs

2007-02-13 23:19:40 dpavlin r39

/trunk/Makefile: use svk instead of svn

2007-02-13 23:12:27 dpavlin r38

/trunk/fdupes-ln.pl: helper script to recover from network errors (and this broken chain-of-hardlinks in some shares)

2007-02-13 22:22:11 dpavlin r37

/trunk/psinib.pl: be a bit more chatty at default verbosity 1, transfer files to temporary ones and rename them after succesfull transfer (instead of doing it in-place and possibly corrupting files because of network errors) [1.0-rc1]

2005-06-09 16:05:43 dpavlin r36

/trunk/psinib.pl: fix verbose

2004-07-08 13:00:57 dpavlin r35

/trunk/psinib.pl: checking for host_up is much more agressive now (so it will detect machines which have firewall enabled, but also allowed file sharing). It will also always use IP address when connecting to SMB share, with fallback to host if nmblookup didn't return IP

2004-06-30 13:23:01 dpavlin r34

/trunk/psinib.pl: There is a problem with List::Compare up to 0.30, so we require 0.30 or newer

2004-06-13 13:28:25 dpavlin r33

/trunk/psinib.pl: --backupdest now takes arguments (stupid bug) and typo fix

2004-06-13 13:24:31 dpavlin r32

/trunk/psinib.pl: bug fix: change file and share size to Kb before comparison

2004-06-11 23:26:43 dpavlin r31

/trunk/psinib.pl: Major improvement and fixups for version 0.9:
- added options --max_share_size, --max_file_size, --min_free_space and --help
- display all values in Kb
- enable warn and die before log is opened
- documentation fixes

2004-05-15 13:06:32 dpavlin r30

/trunk/psinib.pl: improved documentation

2004-05-14 10:02:40 dpavlin r29

/trunk/Makefile, /trunk/psinib.pl: release of version 0.8

2004-05-14 09:27:29 dpavlin r28

/trunk/psinib.pl: use rmtree instead of rmdir, and check if directory exists

2004-03-01 19:38:16 dpavlin r27

/trunk/psinib.pl: if home directory isn't writable (for e.g. if you are running psinib under backup user with Debian) it will fallback to /tmp directory (which is somewhat bad -- why whould you want to run under user without writable home dir?)

2003-11-08 01:46:53 dpavlin r26

/trunk/psinib.pl: don't warn on undef if there isn't subjects.txt file

2003-11-08 00:31:14 dpavlin r25

/trunk/subjects.txt, /trunk/psinib.pl: fixed division by zero if no valid mounts are found in mountscript added --email option to send e-mails (use Mail::Send) implemented cycling subjects (use Tie::File)

2003-10-27 19:07:32 dpavlin r24

/trunk/psinib.pl: remove non-valid latest links

2003-10-27 18:58:41 dpavlin r23

/trunk/psinib.pl: spit fatal errors on screen even if quiet (use -q -q to supress all output)

2003-10-26 14:04:17 dpavlin r22

/trunk/psinib.pl: verbose, quiet options and added documentation for command line options

2003-10-26 12:55:56 dpavlin r21

/trunk/psinib.pl: use tcp syn ping to check if host if up

2003-10-12 21:46:42 dpavlin r20

/trunk/psinib.pl: doc add-on

2003-10-12 18:47:35 dpavlin r19

/trunk/psinib.pl: command-line options

2003-10-12 17:44:21 dpavlin r18

/trunk/psinib.pl: remove stale entries in .md5sum

2003-10-12 16:13:38 dpavlin r17

/trunk/psinib.pl: better logging, report correct number of dirs, don't create local file if remote permission doesn't allow reading

2003-10-12 15:58:28 dpavlin r16

/trunk/psinib.pl: add support for ip=1.2.3.4 option which overrides nmblookup

2003-07-15 17:41:45 dpavlin r15

/trunk/psinib.pl: removed all debugging output

2003-07-15 17:40:32 dpavlin r14

/trunk/psinib.pl, /trunk/TODO: fixed bugs dealing with latest links and smb mounts which aren't available

2003-03-04 21:08:43 dpavlin r13

/trunk/psinib.pl, /trunk/TODO: some more warnings

2003-01-31 22:32:42 dpavlin r12

/trunk/TODO: allmost at version 0.9

2003-01-31 22:31:41 dpavlin r11

/trunk/psinib.pl: Lot of changes:
- ping using tcp connect to port 139
- save warn and die in log
- use newest directory as latest (or partial) if latest symlink doesn't exist
- actually CREATE latest symlink

2003-01-21 21:30:19 dpavlin r10

/trunk/psinib.pl: how to erase old snapshots

2003-01-21 19:50:47 dpavlin r9

/trunk/psinib.pl: append log entries to log

2003-01-04 15:59:14 dpavlin r8

/trunk/psinib.pl: creates .md5sum files (but still doesn't use them) better documentation

2003-01-04 13:34:38 dpavlin r7

/trunk/mountscript.example: example of mountscript used for configuration (created with LinNeighborhood)

2003-01-04 13:29:12 dpavlin r6

/trunk/TODO, /trunk/psinib.pl: start of POD documentation -- use 'perldoc psinib.pl' to read it

2003-01-04 12:14:54 dpavlin r5

/trunk/TODO, /trunk/psinib.pl: added locking using logfile

2004-05-14 09:02:52 unknown r4

/tags, /tags/r0: This commit was manufactured by cvs2svn to create tag 'r0'.

2003-01-04 11:43:18 dpavlin r3

: initial import of first working version

2003-01-04 11:43:18 unknown r2

/branches, /branches/DbP: This commit was manufactured by cvs2svn to create branch 'DbP'.

2003-01-04 11:42:55 dpavlin r1

/trunk/TODO, /trunk/psinib.pl, /trunk: Initial revision