Page 1 of 2 12 LastLast
Results 1 to 20 of 30

Thread: Attention nerds: Any idea what could be wrong with this site?

  1. #1
    Owner Dan Druff's Avatar
    Reputation
    10110
    Join Date
    Mar 2012
    Posts
    54,626
    Blog Entries
    2
    Load Metric
    65669030

    Attention nerds: Any idea what could be wrong with this site?

    Some of you might remember about a year and a half ago (I think) I switched servers.

    PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.

    From examining the running processes, I saw that the server load was occurring mainly due to both http calls and mysql calls. While there are indeed a lot of web page requests (http) and database calls (mysql) here, nothing significant had changed, yet these calls were causing the difference between load averages of 0.2-0.8 in the past to ones like 40-100 in the then-present. I couldn't figure it out, especially because it didn't appear that one user or person was causing those high-load http/mysql stats.

    I contacted the tech support people at the host company. They don't work for free, but they are better at sysadmin stuff than I am, so they looked into it. Three different techs came up blank. Nobody could figure it out. I actually got my money back for their services because everyone sheepishly admitted they were stumped.

    I came up with my own idea that perhaps something had gone corrupt within the server that was very hard to diagnose/discover, and that switching to another one might be the answer. They dismissed my idea, but finally allowed me to give it a shot.

    Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.

    Now I'm starting to see something similar happen. Load averages keep spiking up intermittently, but tonight it was as high as 188 (!!) when radio was supposed to start. I don't believe it was a DOS or DDoS attack.

    This time, however, it was all mysql calls that were the problem.

    Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.

    Thoughts?

    I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.


  2. #2
    PFA Radio Co-Host
    Reputation
    88
    Join Date
    May 2012
    Posts
    311
    Load Metric
    65669030
    Quote Originally Posted by Dan Druff View Post
    Despite the fact that I backed up the PFA software and database and restored an exact copy on the new server, the problem was fixed. Same traffic, same database calls, but suddenly the load average was back to under 1.00 and everything was cool.
    How did you do the database backup and restore? mysqldump ?

    Quote Originally Posted by Dan Druff View Post
    Question: Is it possible for a database to be corrupt, but then operate normally when it is backed up and restored on a new machine? I would think no, but perhaps I'm wrong on this one.
    If your indexes keep becoming corrupt, it's possible. Assuming you rebuilt the database by just dumping the raw sql via mysqldump and restored it the same way, the indexes would be rebuilt when you did that.

    Quote Originally Posted by Dan Druff View Post
    Thoughts?

    I really don't feel like switching again (it's a huge pain in the ass), but I can't keep having this problem with radio like I did this week.

    I'd suggest first monitoring it with mytop so you can get a look at what's actually happening:

    http://jeremy.zawodny.com/mysql/mytop/mytop.html

    You need more data. Also while the CPU goes through the roof, also check on the state of pageins in case the VM is thrashing.

     
    Comments
      
      sonatine: the man makes a good point
      
      DRK Star: ban him....clearly too smart to be here

  3. #3
    Owner Dan Druff's Avatar
    Reputation
    10110
    Join Date
    Mar 2012
    Posts
    54,626
    Blog Entries
    2
    Load Metric
    65669030
    Yes, I'm using mysqldump.

    I will try dumping it and re-creating it. I will also get mytop.

    Hopefully I can get this solved.

    Thanks.

  4. #4
    Plutonium sonatine's Avatar
    Reputation
    7369
    Join Date
    Mar 2012
    Posts
    33,371
    Load Metric
    65669030
    What is dmesg telling you?

    Also consider bumping up file descriptors/ulimit, so on... man sysctl for that whole rabbit hole.

     
    Comments
      
      OSA: was that english?
    "Birds born in a cage think flying is an illness." - Alejandro Jodorowsky

    "America is not so much a nightmare as a non-dream. The American non-dream is precisely a move to wipe the dream out of existence. The dream is a spontaneous happening and therefore dangerous to a control system set up by the non-dreamers." -- William S. Burroughs

  5. #5
    Plutonium sonatine's Avatar
    Reputation
    7369
    Join Date
    Mar 2012
    Posts
    33,371
    Load Metric
    65669030
    Tuning a linux host for database optimization is not easy but it can be rewarding to the point of essential once you start dealing with a certain volume of data.
    "Birds born in a cage think flying is an illness." - Alejandro Jodorowsky

    "America is not so much a nightmare as a non-dream. The American non-dream is precisely a move to wipe the dream out of existence. The dream is a spontaneous happening and therefore dangerous to a control system set up by the non-dreamers." -- William S. Burroughs

  6. #6
    PFA Radio Co-Host
    Reputation
    88
    Join Date
    May 2012
    Posts
    311
    Load Metric
    65669030
    Quote Originally Posted by Dan Druff View Post
    Yes, I'm using mysqldump.

    I will try dumping it and re-creating it. I will also get mytop.

    Hopefully I can get this solved.

    Thanks.
    Is that what you did the first time? Because you can't simply copy mysql databases over... lots of bad things can happen if you do that.

    You need to dump them via mysqldump and restore/recreate them the same way. Make sure you delete the database using DROP DATABASE xxxxx; in mysql> to properly delete it, otherwise you'll have stale MYD, MYI, etc. files.

    Usually I just mysqldump from the original database, and create a new database via mysqldump and point the web app at the new database, to be safe, and give you an easy way to roll back.

    Also consider absolutely starting from scratch, so you can reclaim the space from ibdata1 as per:

    http://stackoverflow.com/questions/3...-file-in-mysql
    Last edited by khalwat; 02-25-2015 at 12:55 AM.

  7. #7
    PFA Radio Co-Host
    Reputation
    88
    Join Date
    May 2012
    Posts
    311
    Load Metric
    65669030
    Quote Originally Posted by Dan Druff View Post
    Some of you might remember about a year and a half ago (I think) I switched servers.

    PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.
    I'd just use iptables for this.

    http://blog.bodhizazen.net/linux/pre...with-iptables/

    You can also use it to prevent repeated login/hacking attempts:

    Code:
    iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource
    
    iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited
    
    iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
    Then make sure you whitelist your own IP(s):

    Code:
    iptables -I INPUT -s your_ip_address_here -j ACCEPT

  8. #8
    Owner Dan Druff's Avatar
    Reputation
    10110
    Join Date
    Mar 2012
    Posts
    54,626
    Blog Entries
    2
    Load Metric
    65669030
    Quote Originally Posted by khalwat View Post
    Quote Originally Posted by Dan Druff View Post
    Some of you might remember about a year and a half ago (I think) I switched servers.

    PFA is on a dedicated server (actually a virtual private server, but I digress), and it was running quite well. Then it started having high server load issues. At first I thought it was a DOS attack, but it wasn't. In fact, I wrote a program to thwart most forms of DOS attacks, and it wasn't catching any.
    I'd just use iptables for this.

    http://blog.bodhizazen.net/linux/pre...with-iptables/

    You can also use it to prevent repeated login/hacking attempts:

    Code:
    iptables -A INPUT -p tcp -m tcp --dport 22 -m tcp -m state --state NEW -m recent --set --name SSH --rsource
    
    iptables -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 600 --hitcount 8 --rttl --name SSH --rsource -j REJECT --reject-with icmp-host-prohibited
    
    iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
    Then make sure you whitelist your own IP(s):

    Code:
    iptables -I INPUT -s your_ip_address_here -j ACCEPT
    Actually I am always getting hit with BS like this:

    Code:
    PlcmSpIp ssh:notty    92.61.39.162     Wed Feb 25 04:14 - 04:14  (00:00)
    PlcmSpIp ssh:notty    92.61.39.162     Wed Feb 25 04:14 - 04:14  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    root     ssh:notty    80.12.68.142     Wed Feb 25 02:19 - 02:19  (00:00)
    So your suggestion will help with that. Not really related to the problems I've been experiencing, though.

    I've actually just blocked large ranges through hosts.deny when I see things like this.

  9. #9
    Owner Dan Druff's Avatar
    Reputation
    10110
    Join Date
    Mar 2012
    Posts
    54,626
    Blog Entries
    2
    Load Metric
    65669030
    Oh and I already am using something similar that I wrote myself involving iptables to prevent DDoSing.

    Believe it or not, the real thing it's catching is accidental DOSing, where someone's system goes crazy and repeatedly requests the radio show archives, or something similar.

  10. #10
    Gold 4BET's Avatar
    Reputation
    94
    Join Date
    Mar 2012
    Location
    FLORIDA
    Posts
    1,617
    Load Metric
    65669030
    Word is seal team 6 is behind this, They are very good at hiding where the attack is coming from
    -Allergic to the struggle

  11. #11
    Master of Props Daly's Avatar
    Reputation
    2671
    Join Date
    Mar 2013
    Posts
    10,305
    Load Metric
    65669030
    Did you try and shut it off and restart it?

     
    Comments
      
      jsearles22: 60% of the time that works every time
      
      herbertstemple: del *.* works every time.

  12. #12
    Silver
    Reputation
    390
    Join Date
    Mar 2012
    Posts
    857
    Load Metric
    65669030
    Quote Originally Posted by Daly View Post
    Did you try and shut it off and restart it?
    Have you considered moving the servers to Antigua?

  13. #13
    Gold abrown83's Avatar
    Reputation
    430
    Join Date
    Mar 2012
    Posts
    1,972
    Load Metric
    65669030
    Druff,

    Does vBulletin use MyISAM or InnoDB for it's tables?

    I would suggest rebuilding or repairing table by table to see if you can pinpoint which one is causing the issue.

    Also ever thought about getting an account at someplace like LiquidWeb or something where they have fully Managed VPS hosting for a reasonable amount of money?

  14. #14
    Gold gauchojake's Avatar
    Reputation
    583
    Join Date
    Mar 2012
    Location
    Zipolite
    Posts
    2,447
    Load Metric
    65669030
    Equation Group malware imo

  15. #15
    Plutonium sonatine's Avatar
    Reputation
    7369
    Join Date
    Mar 2012
    Posts
    33,371
    Load Metric
    65669030
    Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.
    "Birds born in a cage think flying is an illness." - Alejandro Jodorowsky

    "America is not so much a nightmare as a non-dream. The American non-dream is precisely a move to wipe the dream out of existence. The dream is a spontaneous happening and therefore dangerous to a control system set up by the non-dreamers." -- William S. Burroughs

  16. #16
    Gold abrown83's Avatar
    Reputation
    430
    Join Date
    Mar 2012
    Posts
    1,972
    Load Metric
    65669030
    Quote Originally Posted by sonatine View Post
    Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.
    Tine is obviously right, but he only gave you half the SysAdmin info you needed not to fuck yourself.

    Make sure you open the new port on your firewall prior to closing 22.

     
    Comments
      
      garrett: amen

  17. #17
    Owner Dan Druff's Avatar
    Reputation
    10110
    Join Date
    Mar 2012
    Posts
    54,626
    Blog Entries
    2
    Load Metric
    65669030
    Quote Originally Posted by sonatine View Post
    Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.
    The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

    I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

  18. #18
    Gold abrown83's Avatar
    Reputation
    430
    Join Date
    Mar 2012
    Posts
    1,972
    Load Metric
    65669030
    Quote Originally Posted by Dan Druff View Post
    Quote Originally Posted by sonatine View Post
    Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.
    The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

    I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.
    The element of risk is fairly small on a site like this.

    If you wanted to do it still you could always just Remote Desktop or TeamViewer into a home computer and make changes or whitelist a new ip so you could work locally again.

  19. #19
    Plutonium sonatine's Avatar
    Reputation
    7369
    Join Date
    Mar 2012
    Posts
    33,371
    Load Metric
    65669030
    Quote Originally Posted by Dan Druff View Post
    Quote Originally Posted by sonatine View Post
    Move ssh off of port 22 immediately. Like really serious there. Also consider locking down access to your home IP only.
    The latter won't work for various reasons, including the fact that I go to secret locations and will want access.

    I'm not so bothered about having it on 22 as long as I shut out repeated attempts from the same IP like above. There have not been any distributed hack attempts so I'm not worried.

    I hadnt considered the secret locations factor, touche.

    And yeah if youre locking out tumblers, thats another huge plus.

    But the trend right now is distributed cracking through botnets, which we always see after a huge password database leak. A few thousand hosts hammering port 22 is going to result in other processes stacking up in the queue and generally shite performance.

    Food for thought.

    Migrating off port 22 is a fairly standard play right now.

     
    Comments
      
      garrett: Yes, you did!
    "Birds born in a cage think flying is an illness." - Alejandro Jodorowsky

    "America is not so much a nightmare as a non-dream. The American non-dream is precisely a move to wipe the dream out of existence. The dream is a spontaneous happening and therefore dangerous to a control system set up by the non-dreamers." -- William S. Burroughs

  20. #20
    Gold abrown83's Avatar
    Reputation
    430
    Join Date
    Mar 2012
    Posts
    1,972
    Load Metric
    65669030
    Druff,

    Is it mostly http requests or mysql load on the server?

    You haven't been fucking around with the Apache MPM Worker settings have you?

    Sometimes people think by allowing more requests that they will improve performance, but usually that's not the case at all, and it negatively impacts performance and sometimes significantly.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Computer nerds - need help
    By cmoney in forum Flying Stupidity
    Replies: 23
    Last Post: 10-25-2014, 12:37 AM
  2. Attention Site Owner
    By Drexel in forum Flying Stupidity
    Replies: 69
    Last Post: 07-23-2014, 09:18 PM
  3. Nerds, do we have a DJ Tanner problem?
    By SrslySirius in forum Flying Stupidity
    Replies: 6
    Last Post: 04-23-2014, 10:34 PM
  4. Idea: Prob bettors forum
    By Deal in forum Flying Stupidity
    Replies: 0
    Last Post: 06-13-2012, 03:03 PM

Tags for this Thread