FORUM Extremely SLOW

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

User avatar
s.dot
Tranquility In Moderation
Posts: 5001
Joined: Sun Feb 06, 2005 7:18 pm
Location: Indiana

Re: FORUM Extremely SLOW

Post by s.dot »

What about all the emails this board sends?
Perhaps that can be disabled for people who haven't logged in for a long time, or be turned off by default and turned on explicitly for those who want it.
Set Search Time - A google chrome extension. When you search only results from the past year (or set time period) are displayed. Helps tremendously when using new technologies to avoid outdated results.
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: FORUM Extremely SLOW

Post by VladSun »

josh wrote:... That can be done by decreasing the amount of buffering...
... As you can see below from this example report of a "typical" VPS. This dedicated server has 4GB of total memory, and is running default distro config files. What you get is a bunch of sleeping processes eating up memory. A couple "innocent" spare servers, and and a naively high maxRequestsPerChild means we're paying for 2GB of memory to sit around allocated to apache even when there is no traffic. The box is unnecessarily being held back from it's true potential. It doesn't matter if you pay for 250MB of ram or 64GB if your server is configured to allocate all the memory to the wrong processes. The amount under the "free" column should be at least 2/3rds of your memory as a really informal rule of thumb.
There are indeed no such general rules (at leas in Linux). Some say that "an unused RAM = a wasted RAM".

The important thing is how RAM is used. Buffers may or may not be a bad thing. A number of sleeping processes "eating RAM" may or may not be a bad thing.

As you said - it is important to have a properly configured server and services, but it should be configured to match our needs. Generalizations such as "decrease RAM usage and you'll get better results" are more or less taken from Windows philosophy and many people can't agree with them. Take a look at the swappiness kernel config item and ask yourself a question "why it does exist"?
There are 10 types of people in this world, those who understand binary and those who don't
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

VladSun wrote:Generalizations such as "decrease RAM usage and you'll get better results" are more or less taken from Windows philosophy and many people can't agree with them.
I said that using up all of our 256MB on sleeping httpd processes would be a bad idea or mysql buffering is a poor idea. I never said or implied the part in quotes. I specifically called it an informal rule of thumb, and we're specifically talking about running phpbb on a low memory environment (and even more specifically, a parallels based VPS). So I don't think my statements could be taken as generalizations.

When you have PHP scripts within apache doing complex work, it seems they allocate memory and do not free it. Setting maxRequestsPerChild to 1 is done to limit the memory apache allocates while idle. I don't see how letting it leak memory benefits performance in any way and I'm simply sharing my experiences (that when I don't configure that setting, it crashes every VPS I've ever used)
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: FORUM Extremely SLOW

Post by VladSun »

josh wrote:When you have PHP scripts within apache doing complex work, it seems they allocate memory and do not free it.
It should be tested if it's so (eg. using some benchmarking software like ab), not just to suppose it's this way. Memory leaks may appear in specific PHP functions, not in all of them.
josh wrote:Setting maxRequestsPerChild to 1 is done to limit the memory apache allocates while idle. I don't see how letting it leak memory benefits performance in any way and I'm simply sharing my experiences (that when I don't configure that setting, it crashes every VPS I've ever used)
Setting maxRequestsPerChild to 1 is not a good thing to do and it's one of these cases where "it may or may not be a bad thing". maxRequestsPerChild should be tuned after applying some statistics about content loading per page. It's OK that you want to get rid of potential memory leaks, but these leaks may appear only in dynamically generated content. In other words, an average PHPBB page loads (e.g.) one PHP generated content, 5 CSS files, 10 JS files, 100+ images, etc. By setting maxRequestsPerChild to 1 Apache will construct and deconstruct a spare server for *every* one of them. So we would have 100+ times a greater number of spawning/killing processes then the one really needed to protect against memory leaks.
Last edited by VladSun on Mon Oct 11, 2010 9:50 am, edited 1 time in total.
There are 10 types of people in this world, those who understand binary and those who don't
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: FORUM Extremely SLOW

Post by Benjamin »

s.dot wrote:What about all the emails this board sends?
Perhaps that can be disabled for people who haven't logged in for a long time, or be turned off by default and turned on explicitly for those who want it.
The load from sending emails is very minor, not enough to make a difference on page loads by any means.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

VladSun wrote: we would have 100+ times number of spawning/killing processes then the number really needed to protect against memory leaks.
Right now they said memory is the bottleneck, not the CPU, so in all fairness its not a bad idea to try it out.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

I've been look into the matter the last couple of days. We've tweaked the database server configuration somewhat, and now we want to take care of the search problem. phpBB stores matches between posts and search words for some reason - and it's the biggest table in the database by far (approx. 50% of the entire database size). We want to eliminate searching through phpBB and implement something else.
We though of using either google search or installing sphinx and transfering search capabilities (there's a plugin available for phpBB). Would like to hear your thoughts on the matter.
User avatar
Jonah Bron
DevNet Master
Posts: 2764
Joined: Thu Mar 15, 2007 6:28 pm
Location: Redding, California

Re: FORUM Extremely SLOW

Post by Jonah Bron »

Wow, 50%?

I would be good with Google search; that would apparently take a pretty good amount of load off of the server. Just as a workaround until you get it done, you could submit to Google with "site:forums.devnetwork.net" appended to the search term (I'm not telling you how to do it, just a suggestion to do it :) )
mikosiko
Forum Regular
Posts: 757
Joined: Wed Jan 13, 2010 7:22 pm

Re: FORUM Extremely SLOW

Post by mikosiko »

pytrin wrote: We want to eliminate searching through phpBB and implement something else.
We though of using either google search or installing sphinx and transfering search capabilities (there's a plugin available for phpBB). Would like to hear your thoughts on the matter.
I've been reading a lot about phpbb optimization and seems that what are you suggesting (sphinx) is widely preferred ... also some peoples suggest convert the busiest tables to Innodb. Many seems to concur that searching is an important issue in Phpbb
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

also some peoples suggest convert the busiest tables to Innodb
I've been considering this as well. One of the biggest problem is locking on some slow to update tables (one of those is the wordmatch table), since MyISAM uses table-level locking
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: FORUM Extremely SLOW

Post by Benjamin »

It may be good to convert the posts and search tables to InnoDB. We would need to take the forum down for at least 2 hours in order to complete the process. It could take MUCH longer though. It might be better to take down the site, export the tables to a fast machine, convert it and then reimport it. Even then though it would take time to build the indexes. Pytrin, do you want to test for any speed improvements with InnoDB on your box?
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

I ran some tests on suspicious queries from the slow query log, and made some optimizations. What I can't replicate without setting up a stress test is what happens when multiple users use the forum at the same time - creating locking issues since it is on MyISAM tables.
What we want to do now is replace the search with one of the options I suggested (anyone else has opinion on either option?), and convert all the tables to innodb. That should take care of the locking issues.
It is also possible we are moving to a different VPS with much more memory, which will surely help greatly.
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: FORUM Extremely SLOW

Post by VladSun »

I suspect there are either some network related (DNS?) issues, or some Apache MaxClients related issues. Next time the forum dies I'll try to diagnose it :)

There are some issues with proxy for sure.
There are 10 types of people in this world, those who understand binary and those who don't
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: FORUM Extremely SLOW

Post by Benjamin »

I was able to procure a powerful machine for a great price from a friend of mine who is a system administrator. We will probably be discussing moving the site fairly soon.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

5:45am - editing a post takes 5+ minutes to pull up the edit page, and 5+ minutes to save
5:50am - editing a post suddenly takes only 1 second.

Also thats if the pages want to even load at all. They keep flat out crashing during these times of slowness.

Code: Select all

ERROR
The requested URL could not be retrieved

While trying to retrieve the URL: http://forums.devnetwork.net/posting.php?

The following error was encountered:

    * Read Error 

The system returned:

    (104) Connection reset by peer

An error condition occurred while reading data from the network. Please retry your request.

Your cache administrator is admin@devnetwork.net.
Generated Wed, 13 Oct 2010 09:51:53 GMT by devnetwork.net (squid/2.6.STABLE17) 
Post Reply