Attention nerds: Any idea what could be wrong with this site?

Printable View

02-25-2015, 01:14 AM
Dan Druff

Attention nerds: Any idea what could be wrong with this site?

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

From examining the running processes, I saw that the server load was occurring mainly due to both http calls and mysql calls. While there are indeed a lot of web page requests (http) and database calls (mysql) here, nothing significant had changed, yet these calls were causing the difference between load averages of 0.2-0.8 in the past to ones like 40-100 in the then-present. I couldn't figure it out, especially because it didn't appear that one user or person was causing those high-load http/mysql stats.

I contacted the tech support people at the host company. They don't work for free, but they are better at sysadmin stuff than I am, so they looked into it. Three different techs came up blank. Nobody could figure it out. I actually got my money back for their services because everyone sheepishly admitted they were stumped.

I came up with my own idea that perhaps something had gone corrupt within the server that was very hard to diagnose/discover, and that switching to another one might be the answer. They dismissed my idea, but finally allowed me to give it a shot.

Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.

Now I'm starting to see something similar happen. Load averages keep spiking up intermittently, but tonight it was as high as 188 (!!) when radio was supposed to start. I don't believe it was a DOS or DDoS attack.

This time, however, it was all mysql calls that were the problem.

Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.

Thoughts?

I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.

:nerds
02-25-2015, 01:35 AM
khalwat

Quote:

Originally Posted by Dan Druff

Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.

How did you do the database backup and restore? mysqldump ?

Quote:

Originally Posted by Dan Druff

Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.

If your indexes keep becoming corrupt, it's possible. Assuming you rebuilt the database by just dumping the raw sql via mysqldump and restored it the same way, the indexes would be rebuilt when you did that.

Quote:

Originally Posted by Dan Druff

Thoughts?

I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.

:nerds

I'd suggest first monitoring it with mytop so you can get a look at what's actually happening:

http://jeremy.zawodny.com/mysql/mytop/mytop.html

You need more data. Also while the CPU goes through the roof, also check on the state of pageins in case the VM is thrashing.
02-25-2015, 01:38 AM
Dan Druff

Yes, I'm using mysqldump.

I will try dumping it and re-creating it. I will also get mytop.

Hopefully I can get this solved.

Thanks.
02-25-2015, 01:42 AM
sonatine

What is dmesg telling you?

Also consider bumping up file descriptors/ulimit, so on... man sysctl for that whole rabbit hole.
02-25-2015, 01:43 AM
sonatine

Tuning a linux host for database optimization is not easy but it can be rewarding to the point of essential once you start dealing with a certain volume of data.
02-25-2015, 01:49 AM
khalwat

Quote:

Originally Posted by Dan Druff

Yes, I'm using mysqldump.

I will try dumping it and re-creating it. I will also get mytop.

Hopefully I can get this solved.

Thanks.

Is that what you did the first time? Because you can't simply copy mysql databases over... lots of bad things can happen if you do that.

You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

Usually I just mysqldump from the original database, and create a new database via mysqldump and point the web app at the new database, to be safe, and give you an easy way to roll back.

Also consider absolutely starting from scratch, so you can reclaim the space from ibdata1 as per:

http://stackoverflow.com/questions/3...-file-in-mysql
02-25-2015, 02:26 AM
khalwat

Quote:

Originally Posted by Dan Druff

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

I'd just use iptables for this.

http://blog.bodhizazen.net/linux/pre...with-iptables/

You can also use it to prevent repeated login/hacking attempts:

Code:

iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

Then make sure you whitelist your own IP(s):

Code:

iptables -I INPUT -s your_ip_address_here -j ACCEPT
02-25-2015, 04:14 AM
Dan Druff

Quote:

Originally Posted by khalwat

Quote:

Originally Posted by Dan Druff

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

I'd just use iptables for this.

http://blog.bodhizazen.net/linux/pre...with-iptables/

You can also use it to prevent repeated login/hacking attempts:

Code:

iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

Then make sure you whitelist your own IP(s):

Code:

iptables -I INPUT -s your_ip_address_here -j ACCEPT

Actually I am always getting hit with BS like this:

Code:

PlcmSpIp ssh:notty 92.61.39.162 Wed Feb 25 04:14 - 04:14 (00:00) PlcmSpIp ssh:notty 92.61.39.162 Wed Feb 25 04:14 - 04:14 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00) root ssh:notty 80.12.68.142 Wed Feb 25 02:19 - 02:19 (00:00)

So your suggestion will help with that. Not really related to the problems I've been experiencing, though.

I've actually just blocked large ranges through hosts.deny when I see things like this.
02-25-2015, 04:20 AM
Dan Druff

Oh and I already am using something similar that I wrote myself involving iptables to prevent DDoSing.

Believe it or not, the real thing it's catching is accidental DOSing, where someone's system goes crazy and repeatedly requests the radio show archives, or something similar.
02-25-2015, 05:00 AM
4BET

Word is seal team 6 is behind this, They are very good at hiding where the attack is coming from
02-25-2015, 07:39 AM
Daly

Did you try and shut it off and restart it?
02-25-2015, 08:08 AM
hutmaster

Quote:

Originally Posted by Daly

Did you try and shut it off and restart it?

Have you considered moving the servers to Antigua?
02-25-2015, 08:46 AM
abrown83

Druff,

Does vBulletin use MyISAM or InnoDB for it's tables?

I would suggest rebuilding or repairing table by table to see if you can pinpoint which one is causing the issue.

Also ever thought about getting an account at someplace like LiquidWeb or something where they have fully Managed VPS hosting for a reasonable amount of money?
02-25-2015, 10:42 AM
gauchojake

Equation Group malware imo
02-25-2015, 10:47 AM
sonatine

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.
02-25-2015, 11:24 AM
abrown83

Quote:

Originally Posted by sonatine

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

Tine is obviously right, but he only gave you half the SysAdmin info you needed not to fuck yourself.

Make sure you open the new port on your firewall prior to closing 22.
02-25-2015, 11:35 AM
Dan Druff

Quote:

Originally Posted by sonatine

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.
02-25-2015, 11:39 AM
abrown83

Quote:

Originally Posted by Dan Druff

Quote:

Originally Posted by sonatine

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

The element of risk is fairly small on a site like this.

If you wanted to do it still you could always just Remote Desktop or TeamViewer into a home computer and make changes or whitelist a new ip so you could work locally again.
02-25-2015, 11:44 AM
sonatine

Quote:

Originally Posted by Dan Druff

Quote:

Originally Posted by sonatine

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

I hadnt considered the secret locations factor, touche.

And yeah if youre locking out tumblers, thats another huge plus.

But the trend right now is distributed cracking through botnets, which we always see after a huge password database leak. A few thousand hosts hammering port 22 is going to result in other processes stacking up in the queue and generally shite performance.

Food for thought.

Migrating off port 22 is a fairly standard play right now.
02-25-2015, 11:45 AM
abrown83

Druff,

Is it mostly http requests or mysql load on the server?

You haven't been fucking around with the Apache MPM Worker settings have you?

Sometimes people think by allowing more requests that they will improve performance, but usually that's not the case at all, and it negatively impacts performance and sometimes significantly.
02-25-2015, 11:48 AM
abrown83

Quote:

Originally Posted by sonatine

Quote:

Originally Posted by Dan Druff

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

I hadnt considered the secret locations factor, touche.

And yeah if youre locking out tumblers, thats another huge plus.

But the trend right now is distributed cracking through botnets, which we always see after a huge password database leak. A few thousand hosts hammering port 22 is going to result in other processes stacking up in the queue and generally shite performance.

Food for thought.

Migrating off port 22 is a fairly standard play right now.

A whole different issue comes into play if you are on cable internet since a lot of time cable internet providers have dynamic ips.

If he doesn't have a static ip, it introduces another fun element.

Seems like what you said about at least a basic migration to a different port makes since.

Druff in WHM it is something like /etc/ssh/ssh_config or sshconfig or ssh-config you should be able to Vi in and change it....make sure you update your Firewall first.
02-25-2015, 12:02 PM
monsterj

Quote:

Originally Posted by sonatine

Quote:

Originally Posted by Dan Druff

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

I hadnt considered the secret locations factor, touche.

And yeah if youre locking out tumblers, thats another huge plus.

But the trend right now is distributed cracking through botnets, which we always see after a huge password database leak. A few thousand hosts hammering port 22 is going to result in other processes stacking up in the queue and generally shite performance.

Food for thought.

Migrating off port 22 is a fairly standard play right now.

i use openVPN and a dedicated IP address server to log in via ssh. if you want secure, this is how you do it.
02-25-2015, 12:24 PM
simpdog

Are you running out of RAM easily and it's using swap? That can easily f things up.

Also it is annoying in the days of virtual servers. The pricks could have overloaded your current server (and even the one you move it over to) and you would never know.
02-25-2015, 12:26 PM
sonatine

Quote:

Originally Posted by monsterj

Quote:

Originally Posted by sonatine

I hadnt considered the secret locations factor, touche.

And yeah if youre locking out tumblers, thats another huge plus.

But the trend right now is distributed cracking through botnets, which we always see after a huge password database leak. A few thousand hosts hammering port 22 is going to result in other processes stacking up in the queue and generally shite performance.

Food for thought.

Migrating off port 22 is a fairly standard play right now.

i use openVPN and a dedicated IP address server to log in via ssh. if you want secure, this is how you do it.

yeah im just working with whats on the table, but this is correct.
02-25-2015, 06:05 PM
khalwat

Quote:

Originally Posted by Dan Druff

Quote:

Originally Posted by sonatine

Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.

The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

It's fine to have ssh on port 22 if you implement the iptables mentioned above, and you have very good passwords for the accounts on the box.

Obviously disable root ssh access, and sudo instead... and then also don't have an obvious login like "admin" or other standard things that the botnets try by default.

These days, unless you're a high value target, you're mainly concerned about the random botnet / script scans, not about a determined hacker attempting to exploit your box.
02-25-2015, 07:28 PM
Hockey Guy

Only thing I can see that's wrong with this site would be a bunch of shitty posters.

& oh yeah, try what Daly said.
02-25-2015, 08:45 PM
OSA

Do these problems coincide with Crackino's return to this forum? Just sayin.
02-25-2015, 09:27 PM
Lord of the Fraud

Yet to make sense of any post.

Yet I still read the whole thread..

Hardcore
02-25-2015, 09:56 PM
Dan Druff

Quote:

Originally Posted by khalwat

You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

Gonna do this tonight or tomorrow night. If you see PFA down, you know why.

I'll see how well it works, then I'll investigate other possible solutions. This one is the easiest but I think it might work.
02-25-2015, 10:16 PM
khalwat

Quote:

Originally Posted by Dan Druff

Quote:

Originally Posted by khalwat

You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

Gonna do this tonight or tomorrow night. If you see PFA down, you know why.

I'll see how well it works, then I'll investigate other possible solutions. This one is the easiest but I think it might work.

Cheat sheet for you:

http://www.thegeekstuff.com/2008/09/...ing-mysqldump/