The escapeFactory leaderboards and website were down from 5:56:07 PM EST until 8:20:50 PM EST tonight, which is pretty crap.

What Happened

At 5:58 EST, I got an email from our webhost telling me we were using too much CPU. As they do on shared hosting, they immediately disabled (what they perceived to be) the offending script, which in our case was the entire API directory. Turns out, doing that kinda breaks the entire site.

Ben got a tip an hour into the downtime and emailed me. I missed both emails cuz I had just headed out to dinner and had my phone tucked away. I got back around 8 and restored the API directory which put the site back online immediately and resumed recording scores.

To be clear, this only affected the web server — all escapeFactory games are designed to work regardless of the reachability of our Leaderboard API. The apps themselves retain scores locally, but unfortunately all attempts to send scores up *to* our server during this time period failed and were not recorded. (More than likely it exposed some dummy code LOLcat usernames, too.)

Why Did It Break?

As of today, Mole is free for the week of GDC. In Lunarpages’ world, that made us a victim of a “Sudden Burst of Popularity“, which in turn exposed my (ahem) “Badly Written Script or Plugin”.

Every API call runs a SQL query that was only a little costly at 1000 records but an utter deal-breaker at 200,000 records. It was a junior mistake on my part, and one hidden deep that would have stayed hidden except for the sudden popularity of “Mole” today. An unnecessary subquery was causing our response time to slowly creep up over time. By the time I got to it, we were averaging over 5 seconds per API call. No wonder our CPU was topping out our limit. I refactored quickly and response time per call is happily back sailing under 0.03 seconds.

Going Forward

At this point, everything appears to be under control. The API as a whole is really efficient, and I think we bought ourselves some time having smoked out that rude subquery.

We’re clearly on the verge of outgrowing our little experimental phase where shared hosting stops making sense, which is frankly a great problem to have.

Things have been quiet on the blog here, but there is plenty going on — and a few announcements are imminent, too. I thought I would get out ahead of those and give an update on the development of our website backend and custom leaderboard API.

History

The idea for this site grew out of the work Ben and I did to give Snowferno online leaderboards. Snowferno.com was both our promotional site for the game and also the online home of users’ leaderboard scores. I wrote a custom API that allowed Ben and I to securely transmit information between Snowferno and our webserver.

Why write a custom API when so many leaderboards existed already? Well, being our first iPhone app, we didn’t want to commit to any 3rd-party leaderboard frameworks yet. I’m pretty sure a bunch weren’t even compatible with Unity3D for iPhone at the time. So rather than complicate our development, we designed a system that required only simple Unity3D WWW calls. With that, we were able to register an app launch and transmit scoring data to and from our online database.

After Snowferno launched, Ben and the Lycette Bros. started on a few follow-up ‘Dude companions to Snow Dude. The original didn’t have an online leaderboard, but they wanted something implemented in the sequels. Ben suggested we broaden the scope of our Snowferno API into one that multiple apps could use. At the same time, all of us original escapeFactory devs — Ben, FatLab Music, the Lycette Bros., and 3000 Words — were starting this site, so it made sense that our new multi-app Snowferno API should also become a part of the escapeFactory.

Design Goals

• Cross-platform: The API must integrate into any development platform without the inclusion of external frameworks. The maximum requirement must be that it can call a web URL and receive a HTTP response.
• Backward-compatibile: The API must always receive API calls from and send responses to apps once they are released, translating if needed as the API matures.
• Autonomous: Front-facing pages may piggyback on a CMS/blog framework, but all ties between the API Core must be contained within a single host-compatible “wrapper” plugin. Whenever possible, the API should recognize and behave properly on all app release platforms rather than impose compatibility requirements on any specific platform.
• Secure: Only authorized apps may communicate with the API. All incoming data must be securely signed.
• Modular: The default front-facing pages must be customizable per-app. Scoring metrics must be extensible per-app to allow multiple dimensions.
• Promotional: API and Core should serve to encourage visibility across the whole escapeFactory family of apps.

Features (as of current release v0.6)

• Site-wide app settings:
  • Toggle site-wide visibility of an app with public/private switch
  • Date-sensitive app releases
  • Toggle visibility of leaderboard pages per-app with on/off switch
  • Centralized fields to store current app price, icon image source, screenshot image sources, and iTunes App Store ID
• Statistics:
  • Total app launches and total unique devices by version per day
  • Total user install base
  • Aggregate average & median scoring trends by level
  • Aggregate achievement progress
• Leaderboard template system:
  • MVC-style modular views to allow per-app customization of leaderboard pages
• WordPress-specific shortcodes:
  • Gives site editors access to output custom data within the site-building features of WordPress without having to expose PHP functions
  • Output an app store URL w/optional affiliate encoding
  • Output a uniform product page block of HTML displaying app icon, screenshots, price, and app store links — for use in a WordPress Page
  • Output a Unity Web Player to enable play of an online version of an app
• m.escfactory.com:
  • lightweight, touch-friendly directory of the escapeFactory family of apps. Intended to be displayed in-app as cross-promotional tool
• API calls:
  • App Launch: register an app launch and update the app with current online data relevant to that user
  • Scores: record online scores and other gameplay metrics and provide the app with realtime high-score updates
  • Achievements: record online when pre-defined tasks are achieved
  • Twitter: post tweets to a user’s account, with app hashtags and custom shortened url to user’s leaderboard pages
  • Account Management: associate a user’s device with the WordPress user registration system, manage Twitter OAuth tokens and other account settings

Tools

Web Frameworks & Services
WordPress – VibrantCMS by WooThemes – jQuery – galleria – jQTouch – Random.org HTTP Interface – Twitter-async

Promotion & Marketing
ShareThis – YOURLs URL shortener – Google Analytics

Coding & Project Management
Basecamp – NetBeans/TextMate/Transmit – Beanstalk

Going Forward

The Lycette Bros.’ upcoming ‘Dudes and the soon-to-be-released Mole from Roo Games have been solely responsible for pushing me to get the escapeFactory API and Core this far. Many more new features are already on my todo lists — some you’ll see and some only we’ll notice.

Mole introduced our first online, “Play Now” version of an app, and we have big plans for doing more with Unity Web Player. Also, look for a Snowferno update that will utilize the new escapeFactory API features (ahem, tweets). And when the time is right, we’ll be bringing its leaderboard pages over to this site as well.

And of course, more great new games. Stay tuned.

Get your copy of “Snowferno” now for a limited time for only $0.99 US!

Read more about Snowferno on snowferno.com or go to the app right now: