"There is no way to know what you have forgotten until you take the test"
This is very much true, however it is much easier than you might think to test this
If you ever feel that you need to test your backups to make sure that you have everything, i highly recommend using something like VirtualBox to set up your own minimal linux server on your own PC! You don't need a dedicated machine and you can tinker/blow away virtuals as needed. This is also part of the learning experience and making sure that what you're backing up is enough to restore your site/server back to a working copy.
"if I can blow away everything with (cd /) and (rm *) and return everything with a tar extract"
This is the quickest way to never seeing your VPS again... Think of it as going to C:\ and deleting the WINDOWS directory, your OS would crash.
I am not a sysadmin myself (although i do work with one), I am a web administrator that manages all aspects of our web server (partly security as our sysadmin deals with most things). I have never needed to destroy an entire VM due to a hacked website once you set up your security properly.
I have decided to allocate a % of the day to marketing the website and a % of the day to increasing security (I need the test of bringing the site up to know if I know enough about security)
This is a very wise thing to do, especially if you are the one managing the systems and are unsure about what you can do to improve things. Managing a server can be a little daunting especially when you don't know where to start. I would check out the following articles as the should help point you in the right direction:
Particularly the last one, using php_admin_value open_basedir is basically a requirement in any of our configs and is probably one of the easiest ways to secure PHP from the rest of your server.
Basically feeling better if 50 users wait for a week to get the system back rather than 1,000 users.
All you need is the linux ISO you want to test with, load it up and you're ready to go.
Keep any website content that isn't otherwise under version control
If you are running a single website on your VPS and have access to install things, i would HIGHLY recommend checking out version control (VCS) to help manage your site and server. I'm not going to tell you which VCS to use, but IMHO, Git is probably one of the easiest from a command line perspective. I use Git for keeping track of any internal projects and websites that we do for 2 reasons.
- Tracking changes to the code base
- Ease of deployment to servers
Now (using Git as an example): using a single command "$ git reset --hard" restoring the entire website prior to when it was hacked instantly.
Or say that you wanted to perform an investigation about the hack, you have a complete change list of uncommitted changes to any modified file or new files.
This is an extreme case, but it also shows the power of using the system. You could (after spending the time to set things up) have your database exported via mysqldump, then trigger your VCS to commit the latest database to version control along with your website. You could track your database changes as well as the website itself.
Then if you ever needed to deploy the website to another server, it could almost be as simple as 'git clone <repo url>'Git Basics - Getting a Git Repository
Im probably simplifying a little, but it should give you an idea.