Server problems and my plan going forward

[Dan Druff]

For the past 3 weeks or so, PFA has suffered bad server issues.

I have tried very hard to figure out the cause of the problem, yet failed to do so. I enlisted the help of the usually-competent tech support/sysadmin department at this site's host, and they were also completely stumped.

Before I go into what's happening, I want to explain how PFA is hosted.

I use what is known as a "virtual server", provided by a third-party company. That is, they create a partition on a machine to pretty much act as its own separate system, and they give me full control of it. They pretty much load it with a basic setup, give me the login credentials, and it's mine to use and configure as I please. They provide very limited technical support, but the assumption is that I will administrate my own system. If I need their help for anything but very simple requests (or hardware issues, which they obviously handle for free), it costs me $75/hr. Fortunately, my past career as a programmer comes in handy here, and I am capable of running my own system. I am not an expert systems administrator, but I can do well enough except for times when very difficult problems come up, at which point I need some help.

I am basically happy with the host I've been using. It is not cheap, but they are reliable and knowledgeable, so I chose them over some bargain solutions. Their customer service is also pretty good (albeit sometimes flawed). They're also not super expensive. When it comes to virtual servers, I'd say they are mid-range when it comes to price.

Anyway, back to the present. Starting sometime in early-mid February, the server started to experience high "load averages", which basically means it's being overtaxed and everything runs slowly. This was strange to me, as I have not added anything significant here for the past 2 years, nor has traffic increased.

I found a corruption in the database and fixed it, but that only seemed to solve a small part of the problem. The bigger problem seemed to be that "httpd" requests -- basically whenever you're loading a webpage or picture from this server -- have gotten painfully slow and inefficient. I can't figure out why. It's gotten to the point to where basic browsing of the site is already moderately taxing the server (which it never did before), and things such as the radio chat room, which ran fine in the past, absolutely crush the server. It's like someone turned down the server's processing ability by a factor of 100.

Nobody can figure this out. The host support staff has spent a long time looking into this, but have only come back with settings tweak suggestions, none of which have helped. The bottom line is that this is a mystery.

It appears that there are only two possible solutions at this point:

1) Switch to a different server on the same host
-or-
2) Switch hosts

Tomorrow, I will negotiate with the sales department of this host to give me a free "trial" server, to where I will transfer PFA and see if it runs any better on there. If it does, I'll stay on the new one. If it doesn't, I'll have to decide to either move elsewhere or try an old version of PFA from a month-old backup and see if that helps matters. I would REALLY hate to do that (and lose content for the past month), but it's a last resort in case this is somehow a problem related to a corrupt PFA database or whatever.

I still think something is fucked up with this server in particular, and that things will get better once I move to a new machine.

It will be a huge pain in the ass to migrate all the data, but I will try it.

Last time I switched servers (to save money, moving within the same host company), I had them actually remove the hard drive and swap it into the new machine. I don't want to do that this time, in case the hard drive itself or something corrupt in the OS (or other pre-installed software) is the cause of the problem.


So what does this mean for everyone here?

PFA will experience some downtime this week while I do this. Radio will again probably NOT occur on Tuesday, as the site might be down that day. I don't expect the downtime to last long, but some of you might have a hard time reaching "new" PFA at first. If that happens, be patient, clear your cache, and try again. If you still can't get on, wait a few hours and try later. You will know that you are stuck on "old" PFA when you get a message telling you so.

I don't want to wait with this, because it interferes with the ability to do radio properly, and our chat room doesn't work.

Thanks for your patience during this process.

[Dan Druff]

Oh, and there has been speculation that the problems here are a result of DDoS attacks, by Jewdonk or someone else.

This is highly unlikely. There appear to be technical problems serving httpd requests, where even a relatively moderate number of them (such as about 20 people using the chat room) can cause the server to get overburdened. I have also carefully inspected traffic logs, and have not found any evidence of traffic spikes that are the hallmark of DDoS attacks.

Why is the server at its worst prior to the start of radio? Because everyone flocks into the chat room, which seems to overburden the server.

[abrown83]

Druff we had a similar issue with a GoDaddy VPS on Friday. Neither us or them could figure it out, just httpd requests taking like 1% of memory s piece. Eventually we decided to add more RAM, that caused us to move machines and it fixed the problem.

So short answer, moving servers seems like a solid plan.

[Dan Druff]

Druff we had a similar issue with a GoDaddy VPS on Friday. Neither us or them could figure it out, just httpd requests taking like 1% of memory s piece. Eventually we decided to add more RAM, that caused us to move machines and it fixed the problem.

So short answer, moving servers seems like a solid plan.

BTW I don't think RAM is the problem. I have kept a close eye on that, and the RAM is never being used up.

But yes, I think the overwhelming chance is that I will move servers and it will all be better.

[BeerAndPoker]

Hope whatever you do goes well.

Will this eliminate the spammers trying to get people to click on a polish website for cock enhancing supplements?

[NorCal Sports]

I have a server where load times spike up every few days when the /tmp folder fills up for some reason -at the same time the load on the server spikes to well above normal levels. I've talked to about 10 different tech support guys at Liquid Web and no one can really figure it out. They do some minor things but nothing really solves it long term. Not too surprising they can't pinpoint your issue either.

The only thing I recommend is a fresh copy of vbulletin on a new server. Run some scans on your DB but I wouldn't roll that back because any injection was probably inserted a long time ago. If you have issues after that, I would recommend seeking further help from a technician. In my experience, finding a guy directly on a freelance site is cheaper & as effective as paying $70/hr to someone else ... they usually just pay the russian on odesk $20 and bill you for $140.

Good luck, I know the feeling, but remember that your users understand - I've shipped $1,000's to apple, adobe, microsoft ... ect for software that glitches all the time. Not to mention adobe got hacked for their user data a few months ago. PFA is a deal and hasn't hardly had any hiccups in the few years I've been here.

[Belly Buster]

Druff - while virtual servers are all the rage - I would recommend a physical (dedicated) server. The reason being is that your virtual machine can be screwed over by what is happening on the other VMs, and you won't have any visibility of this except for when your stuff slows down.

A dedicated server does not cost that much more than a VM (of similar spec).

[sonatine]

I have a server where load times spike up every few days when the /tmp folder fills up for some reason -at the same time the load on the server spikes to well above normal levels. I've talked to about 10 different tech support guys at Liquid Web and no one can really figure it out. They do some minor things but nothing really solves it long term. Not too surprising they can't pinpoint your issue either.

The only thing I recommend is a fresh copy of vbulletin on a new server. Run some scans on your DB but I wouldn't roll that back because any injection was probably inserted a long time ago. If you have issues after that, I would recommend seeking further help from a technician. In my experience, finding a guy directly on a freelance site is cheaper & as effective as paying $70/hr to someone else ... they usually just pay the russian on odesk $20 and bill you for $140.

Good luck, I know the feeling, but remember that your users understand - I've shipped $1,000's to apple, adobe, microsoft ... ect for software that glitches all the time. Not to mention adobe got hacked for their user data a few months ago. PFA is a deal and hasn't hardly had any hiccups in the few years I've been here.

applications write various files to /tmp. load is a measure of applications stacked up in the run queue. when applications cannot write to /tmp, they end up hung (depending on what type of error handlers are written in of course) which results in more applications stacking up in the queue behind them, hence high load.

[Dan Druff]

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 2.0G 36M 1.9G 2% /tmp

So I don't think it's a problem with /tmp.

I can't install a fresh copy of vBulletin because that would wipe out all of the modifications I've made here (which are numerous at this point). I also can't install a fresh vBulletin database or it will wipe out all of the content here!

I have gotten the host to agree to the process I mentioned in the OP. Took about an hour to do, but they finally agreed (required a few departments to communicate), and this is going forward.

At some point very soon, I will shut off the forum, move everything to the new server, and will change the name server so it points to the new site. If you still reach the old one, you will get a message telling you so. Either clear your cache or just wait a few hours and try again.

Radio will not take place again until this is fixed, so tomorrow's show is postponed for at least a day.

[Hockey Guy]

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 2.0G 36M 1.9G 2% /tmp

So I don't think it's a problem with /tmp.

I can't install a fresh copy of vBulletin because that would wipe out all of the modifications I've made here (which are numerous at this point). I also can't install a fresh vBulletin database or it will wipe out all of the content here!

I have gotten the host to agree to the process I mentioned in the OP. Took about an hour to do, but they finally agreed (required a few departments to communicate), and this is going forward.

At some point very soon, I will shut off the forum, move everything to the new server, and will change the name server so it points to the new site. If you still reach the old one, you will get a message telling you so. Either clear your cache or just wait a few hours and try again.

Radio will not take place again until this is fixed, so tomorrow's show is postponed for at least a day.

So that means you'll be following the terms of our agreement on Wednesday then?

[sonatine]

Filesystem Size Used Avail Use% Mounted on
/dev/sda2 2.0G 36M 1.9G 2% /tmp

So I don't think it's a problem with /tmp.

I can't install a fresh copy of vBulletin because that would wipe out all of the modifications I've made here (which are numerous at this point). I also can't install a fresh vBulletin database or it will wipe out all of the content here!

I have gotten the host to agree to the process I mentioned in the OP. Took about an hour to do, but they finally agreed (required a few departments to communicate), and this is going forward.

At some point very soon, I will shut off the forum, move everything to the new server, and will change the name server so it points to the new site. If you still reach the old one, you will get a message telling you so. Either clear your cache or just wait a few hours and try again.

Radio will not take place again until this is fixed, so tomorrow's show is postponed for at least a day.

Next time the site is fucked, run:

top -b -n 1 > /tmp/diagnostics.top.txt

And

netstat -nae > /tmp/diagnostics.network.txt

You can post the diagnostics.top.txt here because its non-sensitive, the second file will have network addresses in it however and should not be pasted in public. However if someone is dos'ing, depending on how they are doing it, it might become obvious through that file.

90% chance tho you just find a java proc going piss wild in the top output.