Page 3 of 5

Re: FORUM Extremely SLOW

Posted: Sun Oct 10, 2010 11:35 am
by s.dot
What about all the emails this board sends?
Perhaps that can be disabled for people who haven't logged in for a long time, or be turned off by default and turned on explicitly for those who want it.

Re: FORUM Extremely SLOW

Posted: Sun Oct 10, 2010 1:19 pm
by VladSun
josh wrote:... That can be done by decreasing the amount of buffering...
... As you can see below from this example report of a "typical" VPS. This dedicated server has 4GB of total memory, and is running default distro config files. What you get is a bunch of sleeping processes eating up memory. A couple "innocent" spare servers, and and a naively high maxRequestsPerChild means we're paying for 2GB of memory to sit around allocated to apache even when there is no traffic. The box is unnecessarily being held back from it's true potential. It doesn't matter if you pay for 250MB of ram or 64GB if your server is configured to allocate all the memory to the wrong processes. The amount under the "free" column should be at least 2/3rds of your memory as a really informal rule of thumb.
There are indeed no such general rules (at leas in Linux). Some say that "an unused RAM = a wasted RAM".

The important thing is how RAM is used. Buffers may or may not be a bad thing. A number of sleeping processes "eating RAM" may or may not be a bad thing.

As you said - it is important to have a properly configured server and services, but it should be configured to match our needs. Generalizations such as "decrease RAM usage and you'll get better results" are more or less taken from Windows philosophy and many people can't agree with them. Take a look at the swappiness kernel config item and ask yourself a question "why it does exist"?

Re: FORUM Extremely SLOW

Posted: Sun Oct 10, 2010 2:51 pm
by josh
VladSun wrote:Generalizations such as "decrease RAM usage and you'll get better results" are more or less taken from Windows philosophy and many people can't agree with them.
I said that using up all of our 256MB on sleeping httpd processes would be a bad idea or mysql buffering is a poor idea. I never said or implied the part in quotes. I specifically called it an informal rule of thumb, and we're specifically talking about running phpbb on a low memory environment (and even more specifically, a parallels based VPS). So I don't think my statements could be taken as generalizations.

When you have PHP scripts within apache doing complex work, it seems they allocate memory and do not free it. Setting maxRequestsPerChild to 1 is done to limit the memory apache allocates while idle. I don't see how letting it leak memory benefits performance in any way and I'm simply sharing my experiences (that when I don't configure that setting, it crashes every VPS I've ever used)

Re: FORUM Extremely SLOW

Posted: Mon Oct 11, 2010 4:10 am
by VladSun
josh wrote:When you have PHP scripts within apache doing complex work, it seems they allocate memory and do not free it.
It should be tested if it's so (eg. using some benchmarking software like ab), not just to suppose it's this way. Memory leaks may appear in specific PHP functions, not in all of them.
josh wrote:Setting maxRequestsPerChild to 1 is done to limit the memory apache allocates while idle. I don't see how letting it leak memory benefits performance in any way and I'm simply sharing my experiences (that when I don't configure that setting, it crashes every VPS I've ever used)
Setting maxRequestsPerChild to 1 is not a good thing to do and it's one of these cases where "it may or may not be a bad thing". maxRequestsPerChild should be tuned after applying some statistics about content loading per page. It's OK that you want to get rid of potential memory leaks, but these leaks may appear only in dynamically generated content. In other words, an average PHPBB page loads (e.g.) one PHP generated content, 5 CSS files, 10 JS files, 100+ images, etc. By setting maxRequestsPerChild to 1 Apache will construct and deconstruct a spare server for *every* one of them. So we would have 100+ times a greater number of spawning/killing processes then the one really needed to protect against memory leaks.

Re: FORUM Extremely SLOW

Posted: Mon Oct 11, 2010 6:23 am
by Benjamin
s.dot wrote:What about all the emails this board sends?
Perhaps that can be disabled for people who haven't logged in for a long time, or be turned off by default and turned on explicitly for those who want it.
The load from sending emails is very minor, not enough to make a difference on page loads by any means.

Re: FORUM Extremely SLOW

Posted: Mon Oct 11, 2010 8:25 am
by josh
VladSun wrote: we would have 100+ times number of spawning/killing processes then the number really needed to protect against memory leaks.
Right now they said memory is the bottleneck, not the CPU, so in all fairness its not a bad idea to try it out.

Re: FORUM Extremely SLOW

Posted: Mon Oct 11, 2010 4:55 pm
by Eran
I've been look into the matter the last couple of days. We've tweaked the database server configuration somewhat, and now we want to take care of the search problem. phpBB stores matches between posts and search words for some reason - and it's the biggest table in the database by far (approx. 50% of the entire database size). We want to eliminate searching through phpBB and implement something else.
We though of using either google search or installing sphinx and transfering search capabilities (there's a plugin available for phpBB). Would like to hear your thoughts on the matter.

Re: FORUM Extremely SLOW

Posted: Mon Oct 11, 2010 6:39 pm
by Jonah Bron
Wow, 50%?

I would be good with Google search; that would apparently take a pretty good amount of load off of the server. Just as a workaround until you get it done, you could submit to Google with "site:forums.devnetwork.net" appended to the search term (I'm not telling you how to do it, just a suggestion to do it :) )

Re: FORUM Extremely SLOW

Posted: Mon Oct 11, 2010 7:51 pm
by mikosiko
pytrin wrote: We want to eliminate searching through phpBB and implement something else.
We though of using either google search or installing sphinx and transfering search capabilities (there's a plugin available for phpBB). Would like to hear your thoughts on the matter.
I've been reading a lot about phpbb optimization and seems that what are you suggesting (sphinx) is widely preferred ... also some peoples suggest convert the busiest tables to Innodb. Many seems to concur that searching is an important issue in Phpbb

Re: FORUM Extremely SLOW

Posted: Tue Oct 12, 2010 2:46 am
by Eran
also some peoples suggest convert the busiest tables to Innodb
I've been considering this as well. One of the biggest problem is locking on some slow to update tables (one of those is the wordmatch table), since MyISAM uses table-level locking

Re: FORUM Extremely SLOW

Posted: Tue Oct 12, 2010 3:20 am
by Benjamin
It may be good to convert the posts and search tables to InnoDB. We would need to take the forum down for at least 2 hours in order to complete the process. It could take MUCH longer though. It might be better to take down the site, export the tables to a fast machine, convert it and then reimport it. Even then though it would take time to build the indexes. Pytrin, do you want to test for any speed improvements with InnoDB on your box?

Re: FORUM Extremely SLOW

Posted: Tue Oct 12, 2010 4:34 pm
by Eran
I ran some tests on suspicious queries from the slow query log, and made some optimizations. What I can't replicate without setting up a stress test is what happens when multiple users use the forum at the same time - creating locking issues since it is on MyISAM tables.
What we want to do now is replace the search with one of the options I suggested (anyone else has opinion on either option?), and convert all the tables to innodb. That should take care of the locking issues.
It is also possible we are moving to a different VPS with much more memory, which will surely help greatly.

Re: FORUM Extremely SLOW

Posted: Wed Oct 13, 2010 2:23 am
by VladSun
I suspect there are either some network related (DNS?) issues, or some Apache MaxClients related issues. Next time the forum dies I'll try to diagnose it :)

There are some issues with proxy for sure.

Re: FORUM Extremely SLOW

Posted: Wed Oct 13, 2010 2:41 am
by Benjamin
I was able to procure a powerful machine for a great price from a friend of mine who is a system administrator. We will probably be discussing moving the site fairly soon.

Re: FORUM Extremely SLOW

Posted: Wed Oct 13, 2010 4:55 am
by josh
5:45am - editing a post takes 5+ minutes to pull up the edit page, and 5+ minutes to save
5:50am - editing a post suddenly takes only 1 second.

Also thats if the pages want to even load at all. They keep flat out crashing during these times of slowness.

Code: Select all

ERROR
The requested URL could not be retrieved

While trying to retrieve the URL: http://forums.devnetwork.net/posting.php?

The following error was encountered:

    * Read Error 

The system returned:

    (104) Connection reset by peer

An error condition occurred while reading data from the network. Please retry your request.

Your cache administrator is admin@devnetwork.net.
Generated Wed, 13 Oct 2010 09:51:53 GMT by devnetwork.net (squid/2.6.STABLE17)