Nightly Backup Script

Whether you are using Linux on the desktop or as a server, it's still good that you're using Linux. Linux related questions go here.

Moderator: General Moderators

Post Reply
User avatar
twinedev
Forum Regular
Posts: 984
Joined: Tue Sep 28, 2010 11:41 am
Location: Columbus, Ohio

Nightly Backup Script

Post by twinedev »

I had to set this up for a friend, and thought I would share it here. Note, his system has dual 250 gig drives, one dedicated for his /backup directory, and his nightly backup is just under 80 megs, so he opted to save copies for 6 months. This script does a full nightly back up for his CentOS system.

The main thing you need to change is the first 3 variables, a username and password for account that has permissions to backup the databases, and the path (no trailing slash) as to where to put the backups. MAKE SURE THIS IS NOT INSIDE A DIRECTORY TO BE BACKED UP!

Also, the last line, 4320 is how old of files to delete. (note, this will use the timestamp of the last time ACCESSED, not created or modified). this is in hours (24 hours * 180 days for his use)

If you have more you want to backup, duplicate the first tar command you see, change home.tgz to be the final compressed filename and the other home is the directory (relative to the root) you are backing up. (the line after it does selected backups of what I wanted from his /etc directory)

The mysql backup portion will loop through all databases (other than the "test" one installed on default).

Assuming you ran this script as is, here is how it lays things out. All you need defined before first running it will be the /backup directory, it creates all else.

/backup/current -> this is the latest copy backed up. This should be copied and taken off-site often
/backup/current/data -> this is the compressed dump of the databases.
/backup/current/files -> this holds a compressed copy of the files you told it to back up
/backup/current/*.txt -> filelists of what was backed up, and mysql check report

/backup/dated -> each time the script runs, "current" gets moved into here, under yesterday's date
/backup/dated/2010-10 -> all the files for the month of October, 2010
/backup/dated/2010-10/2010-10-28 -> what was in current on October 28, 2010

Here is the script, enjoy. I welcome comments for improving it.

Code: Select all

#!/bin/bash

dbuser="backup" # Give only SHOW DATABASES, SELECT, LOCK TABLES, and RELOAD permissions on all DB's 
dbpass="savemystuff"
buroot="/backup"

yfolder=`date -d yesterday +%Y-%m-%d`
ymfolder=`date -d yesterday +%Y-%m` 
tfolder=`date +%Y-%m-%d`
tprefix=`date +%Y-%m-%d_`
mfolder=`date +%Y-%m`

mkdir $buroot/
mkdir $buroot/current
mkdir $buroot/dated
mkdir $buroot/dated/$ymfolder
mkdir $buroot/dated/$ymfolder/$yfolder

mv $buroot/current/* $buroot/dated/$ymfolder/$yfolder

mkdir $buroot/current/files
mkdir $buroot/current/data

cd /
tar -czvf `echo $buroot/current/files/$tprefix home.tgz | sed 's/ //'` home > $buroot/current/list_home.txt
tar -czvf `echo $buroot/current/files/$tprefix etc.tgz | sed 's/ //'` etc/hosts etc/httpd* etc/php* etc/yum* > $buroot/current/list_etc.txt

/usr/bin/mysqlcheck --user=$dbuser --password=$dbpass --fast --all-databases > $buroot/current/check_db.txt

cd $buroot
for i in `echo show databases|mysql -u $dbuser -p$dbpass`
do
  if [ "$i" != "information_schema" -a "$i" != "Database" -a "$i" != "test" ]
  then
    mysqldump -u $dbuser -p$dbpass $i > "$tprefix$i"
    tar -czf "$buroot/current/data/$tprefix$i.tgz" "$tprefix$i"
    rm "$tprefix$i"
  fi
done

/usr/sbin/tmpwatch 4320 $buroot/dated
PS. I know the line in the tar files is messy with the echo piped to sed, but I couldn't figure out how to butt text up against a variable. I welcome how the heck to do that (with leaving the _ in the variable)
Last edited by twinedev on Sat Oct 30, 2010 11:39 am, edited 1 time in total.
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Nightly Backup Script

Post by VladSun »

A quick notice - use "mkdir -p" to create all parent directories in a single command.
I'll examine your script in details later.

PS: Put it into [ syntax=bash ] tags, please
There are 10 types of people in this world, those who understand binary and those who don't
User avatar
twinedev
Forum Regular
Posts: 984
Joined: Tue Sep 28, 2010 11:41 am
Location: Columbus, Ohio

Re: Nightly Backup Script

Post by twinedev »

Thanks for the heads up on the [ syntax ] tag.

The mkdir for the code I wrote could most likely be simplified, our IT guy at work had explained to me he originally wrote it with each mkdir separate so that it would change explicitly change the time stamp on the folders, so that each morning he could quickly scan the directory and see if any didn't run (we used to have some servers, that well, crapped out from load and bad programming, but this was just left in).

At the office, the script actually uses rsync to sync over to a dedicated backup server, using the following so only changed files are in the dated directory:

Code: Select all

rsync -ab -e "ssh -l root" --delete --backup-dir=../$nfolder /home 192.168.10.32:/media/WebSites/SERVER_NAME/current
($nfolder I called $yfolder in my script)

Works good for saving room on backup space (the backup server currently has over 900G stored on it), but gets more complicated to do a restore from a previous day, so for my friend we went with a full nightly backup.

This is my first main script I have written in years, so I do appreciate the feedback.

-Greg

PS, for those looking to backup their own systems, never forget to get your data offsite. At work we have a duplicate backup server in another data center and have a dedicated connection for nothing but the backups. Each morning after the backups are done to the local server, that server backs up to the remote.
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Nightly Backup Script

Post by VladSun »

twinedev wrote:The mkdir for the code I wrote could most likely be simplified, our IT guy at work had explained to me he originally wrote it with each mkdir separate so that it would change explicitly change the time stamp on the folders, so that each morning he could quickly scan the directory and see if any didn't run (we used to have some servers, that well, crapped out from load and bad programming, but this was just left in).
I don't see how

Code: Select all

mkdir -p /1/2/3/4
differs from

Code: Select all

mkdir /1
mkdir /1/2
mkdir /1/2/3
mkdir /1/2/3/4
There are 10 types of people in this world, those who understand binary and those who don't
User avatar
twinedev
Forum Regular
Posts: 984
Joined: Tue Sep 28, 2010 11:41 am
Location: Columbus, Ohio

Re: Nightly Backup Script

Post by twinedev »

I just tested it from the time stamp issue and it does touch the timestamp using the -p method.

I do know there were parts of the original script removed before I received it for some extra things it handles that I didn't need, perhaps there were steps in the middle between some of those.

I just worked and modified what I was given as a starting point, and now know that step can be cleaned up more. ;-)

Any thoughts on how to get rid of the `echo $buroot/current/files/$tprefix etc.tgz | sed 's/ //'`??

One thought was to modify the script to read in a list of directories to back up, then I could just use $tprefix$curdir.tgz

-Greg
Post Reply