Jan 192013
 

The short version: Hack attempts were made (like they are regularly) but no hack/penetration occurred. No user data was at risk, no security breached. Not quite a false alarm, but it checked out okay.

The long version:

I believe that if/when a hack appear possible take action: immediately (1) shut the server down, quickly (2) communicate with interested parities, then (3) diagnose (4) repair and learn lesson (before bringing the server back up) (5) communicate. I took my time on (3) and (4) and here is (5).

When I believed that a hack might have occurred I took steps (1) and (2) via blog post and tweet.

I remotely viewed the disk (without booting it) and investigated the 3 clues.

#1 Hacked links to www.china101.com

This was the first sign I saw. The front page of WildObs linked to china101.com instead of itself. Not right. I assumed this meant that the hacker had gained write access to the file system. They had NOT.

WildObs uses rails page caching on some pages, one being the index.html front page. I believe some hacker had forged an HTTP request to wildobs.com, sending it to the correct IP address but spoofing the DNS name in the query. The wildobs code (incorrectly) respected “it’s host” and write it into pages. Not sure why it wrote absolute links not relative links, but both these two things will be resolved in a future update.

Yes the file was written, yes the hacker managed to alter it’s contents, but there even a possibility the hacker never even knew. It was an obscure side effect, that gain the hacker nothing, just gave me one big scare.

#2 Unable to connect to WildObs.com

This is what sealed the deal for me that this was a “hack”; I was locked out of WildObs.com. I assumed, having read this account of a hack, that the hacker was “buying time”. This turned out to be nothing but a bad coincident. I used an awesome product called Little Snitch which has one small weakness in that it is (at times) verbose, and can pop-up  questions, and can react to unintended keystrokes to store a rule. A stray local rule coincidentally disconnected me from WildObs.com, not a hacker.

#3 Failure in the database

This might have been due to me  rebooting the system, but it cleared itself up. I don’t like not having a definitive explanation for some clue, but I cannot find a malicious purpose. I took the database down immediately, but when I brought it up again the error was not there. It was some obscure column missing of some obscure valueless table, it cannot have value. I’ll keep thinking about this one, but I feel it had to be more bad coincidence.

Diagnosis:

One small hack-let and a couple of coincidences, plus one over active imagination.

Prevention:

WildObs has always been as secure as I can make it, and very locked down. That said, I’ve read/researched more, and I’ve made a few more changes to tighten security further. I will keep vigilant and communicate if I ever suspect a hack.

Jan 132013
 

UPDATE 1/18/2013: NOT Hacked.

Tonight I noticed that links on wildobs.com stopped pointing to wildobs.com but to china101.com. Not right. I went to log in to the servers, but was unable to.

Assuming the domain or IP hasn’t been re-routed, then the WildObs servers have been compromised. The links above means that files on the server have been altered. The “lock out” means they’ve compromised processes/configuration.

I’ve taken the servers down to limit their ability to steal or corrupt data.

At this time I do not know what has been done or taken, so I plan on assuming the worst, and reacting accordingly. If you are a WildObs user, the sort of things you should be thinking about are:

- If you created a WildObs account, yet (mistakenly, be best practices) re-used a password you use on other website, then change that password on those sites.

- If you’ve authorized WildObs for you Twitter, Facebook, Google or Flickr account, then perhaps suspend those permissions.

- Have a high index of suspicion on any communication you receive from WildObs for now. WildObs would never ask you for passwords or other sensitive information, so never give it any.

Sorry for this inconvenience and I’ll keep this blog updated.