Downtime and Recovery

Where we keep all the boring tidbits about the PHPDN site, the news, and what not.

Moderator: General Moderators

Why did the database go down?

Jason tripped over the plug at the PHPDN office party
13
30%
Twig wasn't paying attention and 'rm -rf' did its thing
5
12%
volka + vodka = vanishing vatabase
12
28%
McGruff just started learning the mysql_connect() and mysql_query() functions
3
7%
Sami got <span style='color:blue' title='I'm naughty, are you naughty?'>smurf</span> and started mouthing off at MySQL
1
2%
phice wanted to prove a point; what that point is, nobody knows
2
5%
JAM decided he had had enough Swedish jokes (Btw, there once was this Swede...)
1
2%
Bech100 would never do this; he was too busy writing Sasser
3
7%
patrikG got upset that I forgot him on this poll as an option
3
7%
 
Total votes: 43

jason
Site Admin
Posts: 1767
Joined: Thu Apr 18, 2002 3:14 pm
Location: Montreal, CA
Contact:

Downtime and Recovery

Post by jason »

Update: From the wonderful people at ImHosted:

"We managed to rescue your forum back to the date "Thu Apr 22 - 2004".
The database was (315Mb) and took well over 1 hour to restore.

I have added a cron job on your server that will make a backup of your
main database (phpdn forums) every week. This is a direct
dump save so we won't have any partial backup issues."

Let me just say thank you to the great people at ImHosted. You guys rock. Thank you for the effort, and for keeping this great community alive (literally!).

-----
First: Obviously we are back. Minus a few posts, and minus a few members that the backup hadn't grabbed, but it's better the losing everything. Honestly, I don't know who recovered the system, whether it was Twig or someone else with ssh rights, or if it was ImHosted. I am inclined to say it was ImHosted, but I can't be sure. No one has sent me an email telling me.

Second: I am going to put together a backup and recovery plan. I had several people email me offering ideas and suggestions, and I will be looking into it. I will also be reviewing the MySQL manual. Most likely, I will document what I find so everyone (that means you!) can learn from this. We don't want this to happen again.

Third: I would like to thank everyone's support, assistance, and patience through this. It was most appreciated, it did indeed help.

Anyways, I will let you know more as I get clarifications, and once we have a solid backup and recovery plan, I will tell you what it is!
Last edited by jason on Tue May 11, 2004 10:25 am, edited 5 times in total.
User avatar
JayBird
Admin
Posts: 4524
Joined: Wed Aug 13, 2003 7:02 am
Location: York, UK
Contact:

Post by JayBird »

Its good to be home :)

Mark
User avatar
JayBird
Admin
Posts: 4524
Joined: Wed Aug 13, 2003 7:02 am
Location: York, UK
Contact:

Post by JayBird »

I voted for the Jason thing, but why wasn't i invited to the office party. Maybe cos i was writing the Sasser worm ;)

Mark
User avatar
markl999
DevNet Resident
Posts: 1972
Joined: Thu Oct 16, 2003 5:49 pm
Location: Manchester (UK)

Post by markl999 »

Dumping a 315M db might take some time. I hope the site will be offline whilst it happens or the dumps could be corrupt/out of sync :o Might be an idea to have a 'site down for maintenance' type message whilst the dump takes place? *shrug*
User avatar
redhair
Forum Contributor
Posts: 300
Joined: Fri May 30, 2003 4:36 pm
Location: 53.23N-6.57E
Contact:

Post by redhair »

We can all learn from this, so please keep us informed.

Are you backing up the data via a 300mb .sql file?
qads
DevNet Resident
Posts: 1199
Joined: Tue Apr 23, 2002 10:02 am
Location: Brisbane

Post by qads »

yup, what redhair said :D
User avatar
redhair
Forum Contributor
Posts: 300
Joined: Fri May 30, 2003 4:36 pm
Location: 53.23N-6.57E
Contact:

Post by redhair »

http://www.phpclasses.org/browse/package/1560.html
MySQL is undoubtedly the most popular database used with PHP applications. Backup and restore a MySQL database is not always an easy task for owners of PHP sites, especially if they do not have shell access to the server machine or some other way to access the database server remotely,

This package uses several classes to provide an elegant Web interface for backup and restore MySQL databases to files.

The concern that the author had in generating compressed backup files to minimize the disk space requirements, demonstrates a certain level of maturity of this package.
User avatar
mychew
Forum Newbie
Posts: 2
Joined: Wed Mar 31, 2004 2:56 am

Post by mychew »

i wonder what had happen to the forum, i know it when i see e message saying that database server is doiwn...luckily i'm back agaiun....hiuh....
User avatar
phice
Moderator
Posts: 1416
Joined: Sat Apr 20, 2002 3:14 pm
Location: Dallas, TX
Contact:

Post by phice »

Just because I dont know what I'm talking about doesn't mean that I know what I mean when you guys try to answer the questions I tend to know.
Image Image
User avatar
tim
DevNet Resident
Posts: 1165
Joined: Thu Feb 12, 2004 7:19 pm
Location: ohio

Post by tim »

:lol: :lol: :lol: @ the poll options

wow, 300 MB! On my dial-up, by the time I restored that people would have forgotten/forget about this place

thank god the big boys have their big toys (ie, broadband, PUNKS!!!)
User avatar
nigma
DevNet Resident
Posts: 1094
Joined: Sat Jan 25, 2003 1:49 am

Post by nigma »

haha, Bech, I can't believe you'd let that poor teenage boy take the blame for sasser!

;)
User avatar
phice
Moderator
Posts: 1416
Joined: Sat Apr 20, 2002 3:14 pm
Location: Dallas, TX
Contact:

Post by phice »

nigma wrote:haha, Bech, I can't believe you'd let that poor teenage boy take the blame for sasser!

;)
haha :)
Image Image
randomblink
Forum Commoner
Posts: 51
Joined: Wed Jan 28, 2004 11:27 am
Location: Tulsa, Oklahoma, just this side of hell...
Contact:

IT'S BACK!

Post by randomblink »

Dear Sweet Lord...
I am just glad the damn site is back up! (he he, backup)
I about croaked the first day I popped back and tried to ask a question.

Is there a donation spot? I think next payday I will donate to say thanks... I didn't realize how much I rely on this site TILL I NEEDED IT AND IT WASN'T THERE!!!

You guys rock.
This is an informative site... Polite interactions (as much as should be expected of course)... Well structured... And it looks nice too...
Thanks!
jason
Site Admin
Posts: 1767
Joined: Thu Apr 18, 2002 3:14 pm
Location: Montreal, CA
Contact:

Post by jason »

Is there a donation spot? I think next payday I will donate to say thanks... I didn't realize how much I rely on this site TILL I NEEDED IT AND IT WASN'T THERE!!!
No. And for certain reasons, I want to avoid making any direct money, or receiving any money for the site, so their probably won't be. Not that we need it, ImHosted hosts us for no fee, and in all truth, with no expectation in advertising (The banners on the site are completely here because I put them there, and was never asked to put them up). However, I might consider setting up a fund or something, non-profitish, and use the money for good things, like helping people in developing countries get computers, etc. Hrm, maybe I should consider getting non-profit status for PHPDN. ImHosted would probably like that, as it would mean they probably get a tax-break as well. And people donating would like that as well...hrm?

But I am also considering putting up a PHPDN Wishlist on Amazon, and use the Wishlist as a way for people to buy prizes that will be awarded to users of the forum. That way, you could buy something, and then we could send it to a user.
User avatar
JayBird
Admin
Posts: 4524
Joined: Wed Aug 13, 2003 7:02 am
Location: York, UK
Contact:

Post by JayBird »

what happend to the MyVirtualCard link?

Mark
Post Reply