June 13, 2010
.
My wife and I were just settling in for our week-long 10th anniversary trip in the Ozark mountains. We had a great week planned,with lots of hiking and site seeing, even a trip to an old car museumwhich I love to go to when we visit there. I woke up the morning afterwe arrived ready to get started, only to see that there was a voice mailon my phone from 6 a.m. It was Amin Motin, my helpdesk manager,informing me that a number of my sites had been hacked into…
As you may be aware of, three of my servers were hacked into last week, resulting in a number of my sites displaying the hacker's "look atwhat I can do" message as the home page. Fortunately no data was lostfrom any of the databases, and the sites were restored back to theirproper working order within a matter of hours after I reported the issueto my server host.
It would appear that my computer got a keylogger-style virus installed on it, despite the fact that I only use a Mozilla browser(which I didn't think was targeted by such things, but apparently due toits popularity growth this is no longer the case). I often scan the"black hat" and "software cracks" forums for references to my ownsoftware so I can close any loopholes those guys find to try and rip offmy software, and I can only imagine that the virus came from one ofthose sites. Shame on me for getting busy and not updating my virusprotection software.
Well, long story short, I had these 3 server's open in a shell (SSH) window, and next thing you know the hacker was logging in and doing histhing.
The virus has since been removed, the servers restored to their normal working condition and all passwords changed.
This is not the kind of thing I wanted to deal with while I was away on my 10th anniversary with my wife, and it's certainly not the kind ofthing she wanted me to have to deal with on such a special occasion.However, the matter was resolved within a few hours of my reporting itto my web host, which allowed my wife and I to enjoy the rest of ourtrip in peace. It could have been a whole lot worse, to say the least,had I not had such fantastic support from Amin, my customers and my webhost.
I wanted to share with you the lessons I have learned from this incident in case they will help you.
1. Having strong passwords is not enough.
It doesn't matter how "unguessable" your password is if a keylogger gets installed on your computer and the hacker gets the password sent tohim by the virus. It's also not enough to rely on your virusprotection software, because a brand new virus may not be detected.
That's why I've had my host block all shell (SSH) and FTP access from all IP addresses except my own. So unless the hacker manages to breakinto the data center where my servers are, or breaks into my house andsits at my desk, even if he has the passwords it won't do him any good.
Since I never do any SSH work away from home, this works just fine. But even if I did need to access the server via SSH from somewhere else,I can always log a ticket with my web host to get the IP addresstemporarily added to the "allow" list.
2. Having loyal customers is a beautiful thing.
Within an hour of the hack, there were a dozen emails sitting in my inbox from customers and colleagues informing me of the issue andoffering to do whatever they could to help. Within two hours there weretwo dozen emails.
I can't even being to describe how good it made me feel to know that my customers and associates all have my back, and were willing to go theextra mile to help me resolve the problem if they could do anything tohelp. That really lets me know that I'm running this business the way Ishould.
Thanks to everyone who alerted me and offered to help. It is VERY much appreciated.
3. Having a responsive host is vital!
It wouldn't matter much if I knew about the hack while out of town if my web host was slow in responding. That certainly was not the case!Within a few hours of my notifying my host, they had the problem cleanedup and everything set right.
I was informed later that two of their top-tier systems administrators were set on the task. It wasn't long before they knewexactly how the hack occurred, what to do to close the hole, and how totighten security to prevent any such incidents in the future.
I always knew that my web host's service was top notch, but you REALLY know you're in good hands when something like this happens andthey're on the ball. I am SO glad I switched hosts a couple of yearsago. My previous host was incredibly slow to respond no matter howurgent the matter.
In case you're wondering, my server host is:
I cannot recommend their services highly enough. Their response time even on trivial issues is incredible, and when it really matters,they're on top of things in a flash.
Disaster Averted
The situation certainly could have been a lot worse than it was. I cringe to think how things would have gone had I still been with my oldweb host (which shall remain nameless — but it was a BIG host, which youthink would have great support, but didn't). I could have beenstressing it out all week long waiting for things to get corrected IF Ihad even known about what happened. I might have been obliviouslyhiking through the mountains while a dozen of my sites were down for aweek. Just thinking about that grows gray hairs on my head!
But things didn't work that way. Everything was set right the day I reported it to my fantastic host, thanks to being informed by Amin andmy customers and associates. Because all of the links in the chain werestrong, disaster was averted, and the security of my servers are nowfar stronger than they were before.
I have certainly learned a lot from this situation, but the biggest lesson can be summed up in these smile-evoking words: I'm in good hands.
Please post your thoughts and questions in a comment below.
