Server hiccups

Where we keep all the boring tidbits about the PHPDN site, the news, and what not.

Moderator: General Moderators

User avatar
seodevhead
Forum Regular
Posts: 705
Joined: Sat Oct 08, 2005 8:18 pm
Location: Windermere, FL

Re: Server hiccups

Post by seodevhead »

Are you running any control panel, like cPanel? It could be a service like ClamAV or the common pkgacct process that is known to hang for long periods of time and cause higher server loads.

Is there a spiked server load when the site slows down?

Have you considered that it may be a problem with your host? In other words, you may be on a bad vlan (do you share an uplink?), or they may be rebooting their switches due to constant ddos attacks, etc.. and some of these switches have bad table routing. The IP that devnet is on... is it shared with any other websites? If so, it could be a DNS issue... your DNS could be incorrectly cached for various points in various DNS servers. This likely isn't the issue if you use custom nameservers and host your own DNS zone files... but if your host manages DNS for you... they could be mucking it up using zone files that are automatically regenerated every X minutes on their DNS server.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Re: Server hiccups

Post by Chris Corbyn »

We found out the cause. A crap load of apache processes were exhausting our memory and also our swap. This causes all kinds of issues. The reason for such a high number of apache processes is still unknown, and perhaps a tad suspicious, but what we've done is:

a) Increase the amount of swap space
b) Reduce the number of allowed apache processes significantly (this means you'll get the occassional "hang" while the page is loading because your request is queued; but that's only when we're crazy busy)
c) Reduced PHP's memory limit. God knows why, but phpBB3 cannot run under 8MB of memory so we have to set it higher. Something to look into
d) Set up a squid reverse proxy. Instead of ~20 requests to load the "Post a reply" page for example, there will now only be one request hitting apache. The rest will come from Squid's cache.

All in all we seem to running a lot more smoothly. We get the occasional "hang" for 10-15 seconds, but it subsides gracefully and it's nothing out of the ordinary. We are still optimizing however.
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Re: Server hiccups

Post by RobertGonzalez »

The hang issue is actually a subset of a larger issue that we are looking for an appropriate window of time to correct. Right now, when the site hangs, there is a good chance that it will at some point pop back as we are now using a proxy to serve up pages. However there are times when things still go alittle wobbly and at times like this we usually have an admin around to get things back on track.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Server hiccups

Post by josh »

I'm unable to send PMs?
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Re: Server hiccups

Post by John Cartwright »

josh wrote:I'm unable to send PMs?
What happens?
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Server hiccups

Post by josh »

After I hit send it goes to a blank page. I'll test real quick again on your name
Edit: worked. Don't know? Im not cryin wolf! I tried it several times before.
Post Reply