Attention nerds: Help fix the server again

[Dan Druff]

Going crazy trying to figure this one out.

Starting a few days ago, PHP started hogging a lot of server time.

This is NOT the same problem as last time, which involved a corrupt database which I eventually found on my own.

Here are what I believe to be the facts:

- Not a DOS attack

- Does not seem to be database corruption. mysqld is not using an unusual amount of CPU, PLUS this is happening on BOTH forums running on this server (PFA and one other), and they have separate databases

- php process is hogging CPU time. It's not one particular PHP call. Sometimes it's showthread.php, sometimes it's forumdisplay.php, sometimes it's others.

- Load average constantly around 2-3. Before this problem, it was well below 1.

- Did not install anything new recently to bring this on, though it's possible something auto-updated without my knowledge

- Dedicated VPS machine (i.e. can't blame on shared hosting)

Basically what would cause these php calls to suddenly get slow and hog CPU time, when last week with the identical setup, everything was fine?

Ideas?

[abrown83]

Going crazy trying to figure this one out.

Starting a few days ago, PHP started hogging a lot of server time.

This is NOT the same problem as last time, which involved a corrupt database which I eventually found on my own.

Here are what I believe to be the facts:

- Not a DOS attack

- Does not seem to be database corruption. mysqld is not using an unusual amount of CPU, PLUS this is happening on BOTH forums running on this server (PFA and one other), and they have separate databases

- php process is hogging CPU time. It's not one particular PHP call. Sometimes it's showthread.php, sometimes it's forumdisplay.php, sometimes it's others.

- Load average constantly around 2-3. Before this problem, it was well below 1.

- Did not install anything new recently to bring this on, though it's possible something auto-updated without my knowledge

- Dedicated VPS machine (i.e. can't blame on shared hosting)

Basically what would cause these php calls to suddenly get slow and hog CPU time, when last week with the identical setup, everything was fine?

Ideas?

Can you drop the Process Log here from WHM?

[sonatine]

Explain why we cant blame VPS on hosting please?

It feels like disk IO bottleneck maybe? If the hypervisor isnt sophisticated enough to shape IOPS then someone doing heavy reads/writes elsewhere in the array is going to impact you.

[Dan Druff]

Explain why we cant blame VPS on hosting please?

It feels like disk IO bottleneck maybe? If the hypervisor isnt sophisticated enough to shape IOPS then someone doing heavy reads/writes elsewhere in the array is going to impact you.

But would this affect php only?

It's like some php setting changed without my knowledge and is screwing everything up, or perhaps php itself went corrupt in some way.

[abrown83]

Logically one would say that almost everything on this site has a database call, so it could still be some kind of database issue.

Does vBulletin have ways to flush and rebuild indexes?

Might want to run Repairs on all your tables.

[abrown83]

Explain why we cant blame VPS on hosting please?

It feels like disk IO bottleneck maybe? If the hypervisor isnt sophisticated enough to shape IOPS then someone doing heavy reads/writes elsewhere in the array is going to impact you.

Actually to that point is traffic up? Considering all the Micon stuff?

[sonatine]

Explain why we cant blame VPS on hosting please?

It feels like disk IO bottleneck maybe? If the hypervisor isnt sophisticated enough to shape IOPS then someone doing heavy reads/writes elsewhere in the array is going to impact you.

But would this affect php only?

It's like some php setting changed without my knowledge and is screwing everything up, or perhaps php itself went corrupt in some way.

I feel like diagnosing VPS issues downstream of the hypervisor is sort of a lottery; if there is an IO issue at the hypervisor level then its going to manifest itself as processes usurping top in your sandbox. Because by far the most popular processes are php, youre going to see more php processes stacking up in the queue.

FWIW Ive never heard of php just getting corrupted. abrown's suggestion about tuning your DB sounds like the most pragmatic at this point tho, just in terms of next steps.

Id also throw in a reboot and patch VB (if needed) just to be thorough, you're not going to fix some bad malloc() someplace without a reboot to clear mem.

[garrett]

FEDs???

[Dan Druff]

Hang on... I think something in cpanel might be going nuts and hogging the memory. Maybe that's it.

[Pooh]

Have you tried jerking off on it?

[abrown83]

But would this affect php only?

It's like some php setting changed without my knowledge and is screwing everything up, or perhaps php itself went corrupt in some way.

I feel like diagnosing VPS issues downstream of the hypervisor is sort of a lottery; if there is an IO issue at the hypervisor level then its going to manifest itself as processes usurping top in your sandbox. Because by far the most popular processes are php, youre going to see more php processes stacking up in the queue.

FWIW Ive never heard of php just getting corrupted. abrown's suggestion about tuning your DB sounds like the most pragmatic at this point tho, just in terms of next steps.

Id also throw in a reboot and patch VB (if needed) just to be thorough, you're not going to fix some bad malloc() someplace without a reboot to clear mem.

Yea I haven't ever had PHP go bad, I have had times where I just do a hard reboot and see what happens, to see if they come back.

You could try and tail your PHP connections

I think it's the following command ( tail -f /usr/local/apache/domlogs/pokerfraudalert.com ), I think you might be able to press the up arrow and follow all the connections along as well.

Could check /var/logs/messages as well to see if there is anything in there?

And check for key errors in MySQL which would require a grep something like grep -i "incorrect key" /var/lib/mysql/*.err

(Note that sometimes providers change these default directories around, so you might need to poke around a bit)

Also you could check and see if you had a spike in 404 traffic or other traffic (like somebody brute attacking a port or login window).

[sonatine]

dmesg for segfaults or IO related issues, etc.

[Dan Druff]

Now I don't know what to say.

Definitely there was a weird issue with the analog stats from cPanel hogging up all the memory. So I figured maybe that had a side effect of slowing php.

I disabled it and now we are down to an 0.72 load.

php still showing CPU usage higher than I remember, but maybe I just never paid attention before to how much it uses.

So maybe crisis has been averted. If load avg stays under 1, I won't worry about it.

[4Dragons]

Have you checked the plugins like the chat bar? disable it and test? Maybe the pulls from setting up your WSOP thread? First instinct is that it's a hosting issue though.

[Hockey Guy]

Did you try unplugging it & then plugging it back in?

[abrown83]

Now I don't know what to say.

Definitely there was a weird issue with the analog stats from cPanel hogging up all the memory. So I figured maybe that had a side effect of slowing php.

I disabled it and now we are down to an 0.72 load.

php still showing CPU usage higher than I remember, but maybe I just never paid attention before to how much it uses.

So maybe crisis has been averted. If load avg stays under 1, I won't worry about it.

I have actually had issues similar don't know if files get to a certain size if it just freaks out.

Both of the analytic programs are useless pieces of shit.

Also just for general overhead any pho extensions that are installed and not necessary could be removed to help load and performance as well

[abrown83]

Since we are talking performance Druff:

Enabling mod_deflate or mod_gzip (Should get about a 75% compression in JS and CSS which would take load off the server)

You should swap the logo with this jpeg version that I attached it saves 16kb on every non cache page load (doesn't seem like a lot but will for sure add up).

(ugg I see the file uploader compresses the thing more, I can upload it somewhere if you want me to)

Compressing and removing any unneeded CSS would probably pick you up a bit of performance as well.