VPS directories not to backup

[bowlesj]

Hi, I guess backups are part of security so I chose this section of the forum. I am about to start a regular weekly backup of my CentOS 5 Linux VPS system using tar then copying the output file down to my home machine. I created an exclude file of directories not to back up based upon Google searches (many asking the same question "what not to back up"). Some of the directories below are empty but I put them in with the * at the end because someone pointed out that this will create the directory again during the return even if it is empty and often you want that. In other words it backs up the directory but nothing in the directory. In the past I was a good Linux/Ksh shell script-er but I am not an expert on Linux administration. Does anyone have any suggestions? Thanks John.

List of Directories inside the tar exclude file: The database is being double backed up daily (phpMyAdmin, mysqldump+incremental)

/var/lib/mysql/MyDatabase/*
/var/www/html/*
/dev/*
/tmp/*
/proc/*
/lost+found
/mnt/*
/media/*
/run/*
/sys/*

I was reading up on the restore at this link. It mentions some of the above directories but has comments as to what are in the directories. It has VPS backup/restore tips as well.
http://serverfault.com/questions/479336 ... -for-a-vps

[bowlesj]

It is from a hosting company. It is a low cost package (did that on purpose to force myself to learn). They backup the VPS weekly but they are slow to do a restore and expensive if you want it fast.

[Weiry]

I would suggest that it comes down to how much you know about linux and how quickly you could re-deploy the OS that your systems are running on.

If they provide you with a way to do your own snapshots and download them, that would probably be the easiest option.

But it can really be as complex or as easy as you would like.

If your server is just a plain web server that runs a website with a MySQL database, i would say that the only real requirements for a backup would include:

• a) Copy of the site files, usually in /var/www/html if not configured differently.
• b) Copy of the MySQL database, use the command line program 'mysqldump' instead of tar'ing /var/lib/mysql/<database>. mysqldump will provide you with a .sql file that you can import straight away without having to worry about matching MySQL server versions and worrying about the MySQL schema.
• c) Copy of the apache/httpd virtualhost file (if configured) and main configuration file (apache2.conf / httpd.conf), usually found in /etc/apache2/sites-available/ (Ubuntu/Debian/etc.) or /etc/httpd/conf.d/ (CentOS/RedHat)
• d) Copy of PHP configuration, usually located in /etc/php5/apache2/php.ini (Ubuntu/Debian/etc.) or /etc/php.ini (CentOS/RedHat)

The above really only includes the absolute essentials to get up and running again and also assumes that you know your way around a linux system and how to set it up again if things go wrong. This also reduces your backup size by a huge amount as you aren't backing up the OS itself.

On the other hand, if there is a lot of stuff on the server and it needs to have everything backed up, i would check out this link Full Hard-Drive Backup with Linux Tar

This has a very simple to set up and use tar script which will backup your whole server and an explanation about each part of it as well.

Keep in mind also, that if you want to run your own backups automatically, to check out how to use crontab.
Cron and Crontab usage and examples
Crontab Code Generator

[bowlesj]

Thanks Weiry, I appreciate your response and some of it I will be looking at today. Some of what you say I have done and some I have not.

Yes, I am running a single website (never will run multiple websites) and I basically will only be running (apache, mysql, php, my website scripts and related .jpg files in a subdirectory). I will run any software I will need for hacker protection (something I need to tighten up). I may run some website monitoring software some day but currently I am not (not much traffic expected for a while).

I did install linux on my home machine about 15 years ago but I only learned (bash, sed, awk, grep) really well. The GNU CD rom install took care of the Linux install and I never touched it (no need to worry about hackers changing things, no need to worry about wanting to be always running for users, no need to understand the kernel and where it is on the PC, etc).

Your response "This also reduces your backup size by a huge amount as you aren't backing up the OS itself." is probably the best thing you could have said since it exposes my fear of the unknown about this VPS backup and restore process. I am not completely sure how a VPS works. From what you are saying it sounds like the whole Linux operating system is duplicated on the machine for each VPS that is running (that kind of makes sense compared to my prior thinking that only part of the operating system files are inside the VPS). So if I drop the tar backup file in the "/" directory (which is inside my VPS of course) and run the tar extract this tar run then sets everything back to the way it was at the point of time I took the tar backup (including the full Linux operating system inside my VPS assuming that Linux is stored inside some of the directories - much like Windows is stored in directories). That last sentence is basically what one web page said occurs in terms of clobbering everything. I was thinking I should use the tar "p" parameter on both backup and extract to set the permissions back to the point of time I took the snapshot.

My biggest question is related to any stuff that hackers might install that is not on the backup (stuff since the last backup). If I just extract the backup and clobber everything with that snapshot it will not remove anything that the hackers have put out there. I seem to remember one website saying I should clear my VPS with a remove command (rm) and run the tar extract to bring everything back (scary). Yes I have done this many times on my home pc with acronis image backup/restore but I have no idea if I can do a complete erase of all directories and recreate of everything inside a vps as it is running (lets repeat the scary word). So far I have not found a good tutorial to convince me it is okay to run a remove command to blow away every sub directory from "/" down and expect the system to run until I do my tar extract. My assumption is blowing away everything below "/" wipes out the "Linux" operating system. The other thing I was thinking is the host company could reset the VPS (to the day I got it effectively wiping out the hacker stuff) and I would run the tar extract.

So maybe someone can help me with this fear. Currently I am using 1.6 gig of the 15 gig I have available to me and my website could take a long time to approach the use of the remaining 13.6 gig so a big tar backup file is probably not a problem. The mysql differential file will be huge but I can probably not do that and take mysqldump backups every hour if needed (I will be doing daily disk usage analysis to see if this file will fill all space). Hopefully when it gets too large I have enough users to warrant getting more disk space.

[Celauran]

A VPS is basically just a VM sitting atop a hypervisor. Your VM will have a full Linux install to which you have root access. I agree with Weiry's backup strategy; back up only the things you know you need rather than starting with the entire OS and trying to ascertain what to remove. Keep your config files. Keep any website content that isn't otherwise under version control -- user-generated content, for example. Back up your databases -- and these can be bzipped down to a fraction of their size since they're just text. The OS itself can always be reinstalled, and pretty easily.

[bowlesj]

Thanks guys, knowing what a VPS is helps a lot and now I see the decision point. I basically agree with both of you except for this which Weiry I guess was thinking of. Knowing what to not back up versus knowing what to back up. Thinking about which of these two "knowing" I know best reminds me of something I have said to people over the years. "There is no way to know what you don't know until you take a test" or something very similar "There is no way to know what you have forgotten until you take the test". So considering practicalities "if I can blow away everything with (cd /) and (rm *) and return everything with a tar extract" This is something I know right now (assuming it can be done - LOL - Okay I know with 98% certainty it will work but then again I haven't taken the test have I! - LOL. Case in point: It took me several hours to get the tar backup to work until I figured out that I had to change the position of the exclusion -X parameter and the exclusion file to after the destination file. The better solution (which you are both suggesting) of only backing up the stuff I need and returning the operating system, I can't do fast at the moment but eventually I may get there - touch wood! I have decided to allocate a % of the day to marketing the website and a % of the day to increasing security (I need the test of bringing the site up to know if I know enough about security). It is just a matter of how much I will allocated to each. Currently most will go to increasing security until I feel safe to market it more heavily. Basically feeling better if 50 users wait for a week to get the system back rather than 1,000 users. In short, make those tests easier tests at first then increase the difficulty as you get a better feel for what you know.

I am readings Weiry's link about full backup now. Thanks for that. It is much better than the tutorials I found on the web before.

[bowlesj]

Not directly applicable to a VPS I don't think but I was looking up "bare metal" restore and I found this interesting link. Maybe it is of interest to some.
http://relax-and-recover.org/

[Weiry]

"There is no way to know what you have forgotten until you take the test"

This is very much true, however it is much easier than you might think to test this
If you ever feel that you need to test your backups to make sure that you have everything, i highly recommend using something like VirtualBox to set up your own minimal linux server on your own PC! You don't need a dedicated machine and you can tinker/blow away virtuals as needed. This is also part of the learning experience and making sure that what you're backing up is enough to restore your site/server back to a working copy.

"if I can blow away everything with (cd /) and (rm *) and return everything with a tar extract"

This is the quickest way to never seeing your VPS again... Think of it as going to C:\ and deleting the WINDOWS directory, your OS would crash.
I am not a sysadmin myself (although i do work with one), I am a web administrator that manages all aspects of our web server (partly security as our sysadmin deals with most things). I have never needed to destroy an entire VM due to a hacked website once you set up your security properly.

I have decided to allocate a % of the day to marketing the website and a % of the day to increasing security (I need the test of bringing the site up to know if I know enough about security)

This is a very wise thing to do, especially if you are the one managing the systems and are unsure about what you can do to improve things. Managing a server can be a little daunting especially when you don't know where to start. I would check out the following articles as the should help point you in the right direction:

• Linux: 25 PHP Security Best Practices For Sys Admins
• 13 Apache Web Server Security and Hardening Tips
• SecuringPHP

Particularly the last one, using php_admin_value open_basedir is basically a requirement in any of our configs and is probably one of the easiest ways to secure PHP from the rest of your server.

Basically feeling better if 50 users wait for a week to get the system back rather than 1,000 users.

Again, Virtualbox! All you need is the linux ISO you want to test with, load it up and you're ready to go.

Keep any website content that isn't otherwise under version control

This.
If you are running a single website on your VPS and have access to install things, i would HIGHLY recommend checking out version control (VCS) to help manage your site and server. I'm not going to tell you which VCS to use, but IMHO, Git is probably one of the easiest from a command line perspective. I use Git for keeping track of any internal projects and websites that we do for 2 reasons.

• Tracking changes to the code base
• Ease of deployment to servers

The way i explain the usage of VCS in terms of websites and security is: Imagine your website just got hacked. Files are everywhere, all your PHP files have got code injections and all your javascript files now contain XXS injections. That is a nightmare to clean up.

Now (using Git as an example): using a single command "$ git reset --hard" restoring the entire website prior to when it was hacked instantly.
Or say that you wanted to perform an investigation about the hack, you have a complete change list of uncommitted changes to any modified file or new files.

This is an extreme case, but it also shows the power of using the system. You could (after spending the time to set things up) have your database exported via mysqldump, then trigger your VCS to commit the latest database to version control along with your website. You could track your database changes as well as the website itself.
Then if you ever needed to deploy the website to another server, it could almost be as simple as 'git clone <repo url>'
Git Basics - Getting a Git Repository

Im probably simplifying a little, but it should give you an idea.

[Celauran]

i highly recommend using something like VirtualBox to set up your own minimal linux server on your own PC! You don't need a dedicated machine and you can tinker/blow away virtuals as needed. This is also part of the learning experience and making sure that what you're backing up is enough to restore your site/server back to a working copy.

This is definitely a worthwhile time investment. With Vagrant and a provisioner (Ansible, Chef, Puppet) it's a breeze.

[bowlesj]

Thanks guys, this is an absolutely great help at exactly the time I need it and I will be back to these posts and links again for sure. Weiry's virtual box idea for testing and returning backups is great. Celauran mentioned something like this before. I was planning to go back and look it up. I read about this Virtual Box product last night but did not connect this idea together with testing backups.

Here is something to give back (as best as I can). Maybe not for you guys but for others in my (heavy learning curve position).

I just took stock (in the form of a spread sheet) of what I have to learn about security and I figure it will take me six months to a year. It is a big task but I think back a year and I don't feel too bad. Last year at this time I did not know these even existed. (PHP, MySql, Apache, Javascript, jQuery, Domain-names, etc). Today my website with [110 pages + 110 help pages + 32 MySql tables + some PHP routines that are pretty complex matching records processing] went live and the website itself (without hackers to screw it up) is working great. Before my website went live, when I read about security, I would get sleepy. Now that the test is here I am wide awake about it and I am finding it very interesting actually. It is funny how pressure and timing work on the mind.

To deal with the complexity of learning this with material scattered everywhere with lots of duplication, I created a spreadsheet as a form of info gathering tool and todo list. It is basically a specialized record of what I have done and should do next. For anyone in my current position, here is the column format.

1/ Description of to-do (can add an excel comment if needed).
2/ Importance score (1 to 10) (helps me decide what is next) (can sort it).
3/ Done or not-done (+ small comments) (make it red if very high priority)
4/where I read it (description in the header and URL link in a 2nd header ")
Column #4 repeats across the sheet for every source (Often 10+ sources)

To make the picture of this spreadsheet more clear, lets say (row 5, column 1) was "Create Fire Wall (allow ports 22, 25 and 80)". Column #4 might repeat 10 times with "yes" at maybe three of the intersections. If the source is especially good I may put a different comment such as "Great! See bottom of page (or whatever)". Or maybe I could put "Great - see excel comment). If you don't know MS-Excel you hover over the cell to see the comment pop up.

So creating this spread sheet gives me an overview of what work I have ahead which in turn lets me set priorities for what to do in sequence (re-prioritizing as I go if needed) and a way to find out where I read whatever I read.

Right now I am not marketing the website since I now (with the overview's help) clearly see I have some high priority security stuff I need to get done first. Yesterday I backed up everything on the VPS (first weekly backup with date-time stamp) (the hosting company does weekly backups of the VPS but wants $75 to restore it with a delay of potentially a day). My Sql Backups I described above. Now I can clearly see I need to improve my minimum to be backing up with version control and the help of these tools you suggest. This gets priority 10.

Today is a much needed day off to celebrate the website with my daughter. It is the first day of Spring too (in Canada at least). Enjoy your first day of spring guys. Check out Google today to see the flowers coming up.

Thanks again,
John.