The escapeFactory leaderboards and website were down from 5:56:07 PM EST until 8:20:50 PM EST tonight, which is pretty crap.

What Happened

At 5:58 EST, I got an email from our webhost telling me we were using too much CPU. As they do on shared hosting, they immediately disabled (what they perceived to be) the offending script, which in our case was the entire API directory. Turns out, doing that kinda breaks the entire site.

Ben got a tip an hour into the downtime and emailed me. I missed both emails cuz I had just headed out to dinner and had my phone tucked away. I got back around 8 and restored the API directory which put the site back online immediately and resumed recording scores.

To be clear, this only affected the web server — all escapeFactory games are designed to work regardless of the reachability of our Leaderboard API. The apps themselves retain scores locally, but unfortunately all attempts to send scores up *to* our server during this time period failed and were not recorded. (More than likely it exposed some dummy code LOLcat usernames, too.)

Why Did It Break?

As of today, Mole is free for the week of GDC. In Lunarpages’ world, that made us a victim of a “Sudden Burst of Popularity“, which in turn exposed my (ahem) “Badly Written Script or Plugin”.

Every API call runs a SQL query that was only a little costly at 1000 records but an utter deal-breaker at 200,000 records. It was a junior mistake on my part, and one hidden deep that would have stayed hidden except for the sudden popularity of “Mole” today. An unnecessary subquery was causing our response time to slowly creep up over time. By the time I got to it, we were averaging over 5 seconds per API call. No wonder our CPU was topping out our limit. I refactored quickly and response time per call is happily back sailing under 0.03 seconds.

Going Forward

At this point, everything appears to be under control. The API as a whole is really efficient, and I think we bought ourselves some time having smoked out that rude subquery.

We’re clearly on the verge of outgrowing our little experimental phase where shared hosting stops making sense, which is frankly a great problem to have.