Attention nerds: Any idea what could be wrong with this site?

[Dan Druff]

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

From examining the running processes, I saw that the server load was occurring mainly due to both http calls and mysql calls. While there are indeed a lot of web page requests (http) and database calls (mysql) here, nothing significant had changed, yet these calls were causing the difference between load averages of 0.2-0.8 in the past to ones like 40-100 in the then-present. I couldn't figure it out, especially because it didn't appear that one user or person was causing those high-load http/mysql stats.

I contacted the tech support people at the host company. They don't work for free, but they are better at sysadmin stuff than I am, so they looked into it. Three different techs came up blank. Nobody could figure it out. I actually got my money back for their services because everyone sheepishly admitted they were stumped.

I came up with my own idea that perhaps something had gone corrupt within the server that was very hard to diagnose/discover, and that switching to another one might be the answer. They dismissed my idea, but finally allowed me to give it a shot.

Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.

Now I'm starting to see something similar happen. Load averages keep spiking up intermittently, but tonight it was as high as 188 (!!) when radio was supposed to start. I don't believe it was a DOS or DDoS attack.

This time, however, it was all mysql calls that were the problem.

Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.

Thoughts?

I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.

[khalwat]

Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.

How did you do the database backup and restore? mysqldump ?

Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.

If your indexes keep becoming corrupt, it's possible. Assuming you rebuilt the database by just dumping the raw sql via mysqldump and restored it the same way, the indexes would be rebuilt when you did that.

Thoughts?

I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.

I'd suggest first monitoring it with mytop so you can get a look at what's actually happening:

http://jeremy.zawodny.com/mysql/mytop/mytop.html

You need more data. Also while the CPU goes through the roof, also check on the state of pageins in case the VM is thrashing.

[Dan Druff]

Yes, I'm using mysqldump.

I will try dumping it and re-creating it. I will also get mytop.

Hopefully I can get this solved.

Thanks.

[khalwat]

Yes, I'm using mysqldump.

I will try dumping it and re-creating it. I will also get mytop.

Hopefully I can get this solved.

Thanks.

Is that what you did the first time? Because you can't simply copy mysql databases over... lots of bad things can happen if you do that.

You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

Usually I just mysqldump from the original database, and create a new database via mysqldump and point the web app at the new database, to be safe, and give you an easy way to roll back.

Also consider absolutely starting from scratch, so you can reclaim the space from ibdata1 as per:

http://stackoverflow.com/questions/3...-file-in-mysql

[Dan Druff]

You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

Gonna do this tonight or tomorrow night. If you see PFA down, you know why.

I'll see how well it works, then I'll investigate other possible solutions. This one is the easiest but I think it might work.

[khalwat]

You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

Gonna do this tonight or tomorrow night. If you see PFA down, you know why.

I'll see how well it works, then I'll investigate other possible solutions. This one is the easiest but I think it might work.

Cheat sheet for you:

http://www.thegeekstuff.com/2008/09/...ing-mysqldump/

[sonatine]

What is dmesg telling you?

Also consider bumping up file descriptors/ulimit, so on... man sysctl for that whole rabbit hole.

[sonatine]

Tuning a linux host for database optimization is not easy but it can be rewarding to the point of essential once you start dealing with a certain volume of data.

[khalwat]

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

I'd just use iptables for this.

http://blog.bodhizazen.net/linux/pre...with-iptables/

You can also use it to prevent repeated login/hacking attempts:

iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource

iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited

iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

Then make sure you whitelist your own IP(s):

iptables -I INPUT -s your_ip_address_here -j ACCEPT

[Dan Druff]

Some of you might remember about a year and a half ago (I think) I switched servers.

PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

I'd just use iptables for this.

http://blog.bodhizazen.net/linux/pre...with-iptables/

You can also use it to prevent repeated login/hacking attempts:

iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource

iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited

iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

Then make sure you whitelist your own IP(s):

iptables -I INPUT -s your_ip_address_here -j ACCEPT

Actually I am always getting hit with BS like this:

PlcmSpIp ssh:notty    92.61.39.162     Wed Feb 25 04:14 - 04:14  (00:00)
PlcmSpIp ssh:notty    92.61.39.162     Wed Feb 25 04:14 - 04:14  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)

So your suggestion will help with that. Not really related to the problems I've been experiencing, though.

I've actually just blocked large ranges through hosts.deny when I see things like this.

[Dan Druff]

Oh and I already am using something similar that I wrote myself involving iptables to prevent DDoSing.

Believe it or not, the real thing it's catching is accidental DOSing, where someone's system goes crazy and repeatedly requests the radio show archives, or something similar.

[4BET]

Word is seal team 6 is behind this, They are very good at hiding where the attack is coming from

[Daly]

Did you try and shut it off and restart it?