The short version: Hack attempts were made (like they are regularly) but no hack/penetration occurred. No user data was at risk, no security breached. Not quite a false alarm, but it checked out okay.
The long version:
I believe that if/when a hack appear possible take action: immediately (1) shut the server down, quickly (2) communicate with interested parities, then (3) diagnose (4) repair and learn lesson (before bringing the server back up) (5) communicate. I took my time on (3) and (4) and here is (5).
I remotely viewed the disk (without booting it) and investigated the 3 clues.
#1 Hacked links to www.china101.com
This was the first sign I saw. The front page of WildObs linked to china101.com instead of itself. Not right. I assumed this meant that the hacker had gained write access to the file system. They had NOT.
WildObs uses rails page caching on some pages, one being the index.html front page. I believe some hacker had forged an HTTP request to wildobs.com, sending it to the correct IP address but spoofing the DNS name in the query. The wildobs code (incorrectly) respected “it’s host” and write it into pages. Not sure why it wrote absolute links not relative links, but both these two things will be resolved in a future update.
Yes the file was written, yes the hacker managed to alter it’s contents, but there even a possibility the hacker never even knew. It was an obscure side effect, that gain the hacker nothing, just gave me one big scare.
#2 Unable to connect to WildObs.com
This is what sealed the deal for me that this was a “hack”; I was locked out of WildObs.com. I assumed, having read this account of a hack, that the hacker was “buying time”. This turned out to be nothing but a bad coincident. I used an awesome product called Little Snitch which has one small weakness in that it is (at times) verbose, and can pop-up questions, and can react to unintended keystrokes to store a rule. A stray local rule coincidentally disconnected me from WildObs.com, not a hacker.
#3 Failure in the database
This might have been due to me rebooting the system, but it cleared itself up. I don’t like not having a definitive explanation for some clue, but I cannot find a malicious purpose. I took the database down immediately, but when I brought it up again the error was not there. It was some obscure column missing of some obscure valueless table, it cannot have value. I’ll keep thinking about this one, but I feel it had to be more bad coincidence.
One small hack-let and a couple of coincidences, plus one over active imagination.
WildObs has always been as secure as I can make it, and very locked down. That said, I’ve read/researched more, and I’ve made a few more changes to tighten security further. I will keep vigilant and communicate if I ever suspect a hack.