FORUM Extremely SLOW

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: FORUM Extremely SLOW

Post by Benjamin »

I didn't set this up and I haven't researched how everything is configured. Honestly, I doubt there's a whole lot I can do insofar as increasing the resources available to MySQL, which is the bottleneck here. I do know that the server is using squid, and I'm not sure what purpose it serves because I don't think that reduces queries in any way. Robert is currently in charge of donations and paying for hosting fees.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

CPU or memory? Big difference. Can you post the top reports?
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: FORUM Extremely SLOW

Post by Benjamin »

Memory of course. Not publicly. If you want to help optimize it we can do so privately.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

Benjamin wrote:Memory of course. Not publicly. If you want to help optimize it we can do so privately.
Benjamin wrote:I doubt there's a whole lot I can do insofar as increasing the resources available to MySQL, which is the bottleneck here.
Sounds like mysql is not the bottleneck, no available memory for any of the system services sounds like the real bottleneck. The cause of the bottleneck may very well by mysql. Please feel free to PM me and I'd be glad to help. I agree the forums are terribly slow and there's no real reason for it. I think the real issue is not "increasing the resources" mysql is using, but rather "decreasing the resources" mysql uses.

That can be done by decreasing the amount of buffering. Without the top reports you're not helping me help you though, so please do PM them. There's not really any sensitive info in a top report anyways, you'd find out the user and pid of httpd. As you can see below from this example report of a "typical" VPS. This dedicated server has 4GB of total memory, and is running default distro config files. What you get is a bunch of sleeping processes eating up memory. A couple "innocent" spare servers, and and a naively high maxRequestsPerChild means we're paying for 2GB of memory to sit around allocated to apache even when there is no traffic. The box is unnecessarily being held back from it's true potential. It doesn't matter if you pay for 250MB of ram or 64GB if your server is configured to allocate all the memory to the wrong processes. The amount under the "free" column should be at least 2/3rds of your memory as a really informal rule of thumb. On this box they pay for 4GB, the machine is 99.98% idle with a load average of 0.01 and and there is only 1GB free 8O

Code: Select all

19627 nobody    15   0  343m 156m 6888 S  0.0  4.0   0:12.93 httpd
21448 nobody    15   0  342m 155m 6876 S  0.0  4.0   0:17.51 httpd
19635 nobody    15   0  342m 155m 6864 S  0.0  3.9   0:14.15 httpd
19629 nobody    15   0  342m 155m 6852 S  0.0  3.9   0:14.74 httpd
20167 nobody    15   0  342m 155m 6788 S  0.0  3.9   0:22.52 httpd
19632 nobody    15   0  341m 155m 6876 S  0.0  3.9   0:20.47 httpd
23716 nobody    15   0  341m 155m 6872 S  0.0  3.9   0:06.93 httpd
19634 nobody    18   0  341m 154m 6896 S  0.0  3.9   0:11.87 httpd
 2636 mysql     15   0  481m 148m 4764 S  0.0  3.8  32:50.07 mysqld
12637 root      18   0  174m 134m 5696 S  0.0  3.4   2:11.54 clamd
 2786 root      18   0  304m 119m 7672 S  0.0  3.0   0:42.08 httpd
19621 root      17   0  288m 110m  260 S  0.0  2.8   0:00.00 httpd
25502 root      15   0  110m  43m 2540 S  0.0  1.1   0:00.63 spamd
25515 root      15   0  111m  43m 1748 S  0.0  1.1   0:03.75 spamd
25516 root      18   0  110m  42m 1200 S  0.0  1.1   0:00.00 spamd

Now here is a report from my VPS that I consider "properly configured" (at least to not run out of memory and crash every hour). I pay for 384MB. Notice how 325MB of that is free. My machine is using ~50MB to idle, where as the badly configured dedicated server is using 3GB to idle.

Code: Select all

top - 04:00:45 up 38 days, 14:47,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  21 total,   1 running,  20 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    393216k total,    57332k used,   335884k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11704 mysql     18   0  136m  35m 5712 S  0.0  9.2   2:46.87 mysqld
 3400 root      15   0 25736  12m 3920 S  0.0  3.2   1:44.98 httpd
30641 apache    20   0 25736  10m 1652 S  0.0  2.6   0:00.00 httpd
31990 apache    16   0 25736  10m 1652 S  0.0  2.6   0:00.00 httpd
32196 apache    15   0 25736  10m 1652 S  0.0  2.6   0:00.00 httpd
32583 apache    15   0 25736  10m 1652 S  0.0  2.6   0:00.00 httpd
32762 apache    18   0 25736  10m 1652 S  0.0  2.6   0:00.00 httpd
 5561 root      15   0 10788 2820 2264 S  0.0  0.7   0:00.03 sshd
 3347 root      18   0  9292 1904  816 S  0.0  0.5   0:03.56 sendmail
 3358 smmsp     15   0  8244 1520  648 S  0.0  0.4   0:00.05 sendmail
 5756 root      18   0  4004 1508 1224 S  0.0  0.4   0:00.03 bash
11633 root      19   0  3868 1260 1096 S  0.0  0.3   0:00.00 mysqld_safe
 6016 nscd      15   0  107m 1160  828 S  0.0  0.3   0:44.11 nscd
 3417 root      18   0  4488 1108  568 S  0.0  0.3   0:00.83 crond
 3115 root      16   0  7176 1060  664 S  0.0  0.3   5:06.40 sshd
 4035 root      18   0  2452  972  784 R  0.0  0.2   0:00.21 top
 3143 root      18   0  2828  868  700 S  0.0  0.2   0:00.00 xinetd
 3431 root      18   0  5668  708  436 S  0.0  0.2   0:00.00 saslauthd
    1 root      18   0  2152  660  568 S  0.0  0.2   0:15.30 init
 1621 root      15  -4  2240  552  344 S  0.0  0.1   0:00.00 udevd
 3432 root      18   0  5668  424  152 S  0.0  0.1   0:00.00 saslauthd

Right now on phpbb there are "In total there are 9 users online :: 8 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)". Its taking >1 minute to load this thread, 1 minute to pull up login page. 1 minute to load to forum index. Then I click "view your posts" and it pops up in 1 second. It seems a LOT like we're just being too frugal with our minimal resources. Its not certain actions that cause it. Because sometimes those certain actions take 1 second and other times just loading the forum index seems to take 10s of minutes. I've literally linked people threads and they weren't able to read them until 10 minutes later.

No speed = no donations. No donations = no hardware - sorry gentlemen, catch 22 ;-)
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

To me it sounds like MySQL is the bottleneck. Once working set doesn't fit in memory, it starts making random reads from disk and performance drops by several factors. The forums slows down mainly when someone is performing a search on posts (including search for your own posts or some other user's posts), and MySQL needs to fetch from data that doesn't fit in memory. Of course, lack of memory affects other processes as well, but MySQL takes it the hardest
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

pytrin wrote: The forums slows down mainly when someone is performing a search on posts (including search for your own posts or some other user's posts), and MySQL needs to fetch from data that doesn't fit in memory.
Everyone keeps saying this, yet this page loads in 10 seconds tops for me. Yet other times *all* pages take 1+ minute to load. Lots of forums are much larger than us, run on similar hardware and don't have these issues ;-)

Things seem noticeably faster this morning. Sub second page loads even on searches. 7 Users online...
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

Lots of forums are much larger than us, run on similar hardware and don't have these issues
Can you name those forums and their hardware? or are you just guessing?
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

http://www.theadminzone.com/forums/show ... hp?t=56982
http://www.theadminzone.com/forums/show ... stcount=11
http://www.bokt.nl/forums/

This guy's phpBB has 60 Million posts and 500 users online. It loads faster than ours.
Current hardware consists of:
- Two dualcore Athon 64 X2 3800+, 2GB memory
- One dual dualcore Xeon 5130, 16GB memory, hardware RAID 10 (4x WD Raptor), hardware RAID5 (4x 500GB WD)
- One singlecore Athon which acts as fileserver (NFS :\) and proxy for static files (Squid)

This to handle 30M pageviews/month, 50GB database, 40M posts, 100k registered users, 1800 concurrent users. DB server is way overkill (except for the 16GB of memory), could do with 1 webserver except during some serious spikes in traffic.
Now.. we have less than 1 Million posts. That's nothing. Sure we could do with 512MB or even 1GB of ram but lots of people are doing this with a smaller post-count-to-hardware ratio than us.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

post-count-to-hardware ratio? first time I ever heard that. Is that a technical term?

Seriously though, they have much better hardware. When you have their kind of hardware you have a lot more room to optimize. Performance is not a factor of post-count-to-hardware-ratio. When you have much stronger CPU, RAID 10 array of Raptors - even when the working set doesn't fit in memory and someone fetches older data, you won't notice that much of a hit in performance. That is certainly not the case with this machine.

Not saying it can't be optimized further (I haven't seen the actual configuration), but there's a very hard limit to the benefits it would gain
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

pytrin wrote:RAID 10 array of Raptors - even when the working set doesn't fit in memory and someone fetches older data, you won't notice that much of a hit in performance. That is certainly not the case with this machine.
Most modern VPS are running RAID 10, like I pointed out a few posts back. If you go years without upgrading you could be running on the drives of 3yrs ago.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

An upgrade is necessary, question is who will sponsor it. Maybe it's time to start another donation round?
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: FORUM Extremely SLOW

Post by Benjamin »

I'm on the market for a good VPS anyway. I can throw the site up on there. Donations would be good, to help offset the costs. I'll need to speak with the other admins/mods.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: FORUM Extremely SLOW

Post by josh »

pytrin wrote:An upgrade is necessary
So we're just going to ignore my idea about minimizing the amount of memory it consumes while idle? IMO we should make sure its all configured ideally before upgrading.
mikosiko
Forum Regular
Posts: 757
Joined: Wed Jan 13, 2010 7:22 pm

Re: FORUM Extremely SLOW

Post by mikosiko »

Benjamin wrote:I'm on the market for a good VPS anyway. I can throw the site up on there. Donations would be good, to help offset the costs. I'll need to speak with the other admins/mods.
post the options/cost
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: FORUM Extremely SLOW

Post by Eran »

So we're just going to ignore my idea about minimizing the amount of memory it consumes while idle? IMO we should make sure its all configured ideally before upgrading.
I think it should go hand in hand. You should talk to Benjamin about getting access to private data, I'll be doing the same.

I think we have some discount at rackspace for all the clients we refer to them, so I'll see what kind of package we can get from them.
Post Reply