First, a disclaimer: I'm not - by any stretch of the imagination - an expert in these matters. It's quite possible that I could've got better results with more time invested. But I still think this is a useful comparison because, let's face it, not many people actually do have the time needed to invest in a proper online advertising strategy. Especially if you're just a one-man-team like me!

So with that out of the way, I want to describe my experience using both Google's AdMob advertising platform and Facebook's platform for advertising War Worlds. My goal with the campaign was basically to get more signups in the game, a metric which I can measure directly by looking at the number of new signups in the backend (it's not so easy to track where the signups come from [not impossible, just not easy], so I decided ran both campaigns separately).

Facebook

As I said, very unscientific, but the ad I ran on Facebook was the one on the left. The text for the ad was "A new 4X MMO for Android. Explore, Exploit, Expand, Exterminate!".

I decided to target it at people in the U.S. and Australia who "like Mobile phones, strategy games, Massively multiplayer online games, Strategy games or mobile phones."

I gave it a budget of $20 over three weeks (i.e. just over $1 per day).

AdMob

AdMob is Google's mobile ad network. You create an ad and it displays in any app which use AdMob's SDK. I used the ad on the left, and while the Facebook ad landed users on the War Worlds home page, this one landed users on the Play Store (this is a requirement of AdMob ads: I would have preferred clicks to go to the website here as well).

For this ad, I targetted only Android devices which support War Worlds (i.e. Android 2.3+). There's no fine-grained filters like Facebook ads (so I couldn't, for example, show it only to people interetested in strategy games). I set the budget to $1 per day.

Results

To my eyes, the results are conclusive: Facebook was a complete waste of money, while AdMod is making a noticeable difference in active installs.

The Facebook ad, over the course of about 3 weeks and $20, garnered a total of 61 clicks. According to the "full" report that Facebook provide, I got 94,173 impressions. That's a click-through-rate of 0.065% and cost per click around $0.24.

Looking in the game's backend, there was no bump in new signups to the game at all (really, even if every one who visited the site signed up, that's only 61 potential signups, which -- over three weeks -- is not even a blip). Even though the Facebook ad was seemingly targetted very specifically to people who were actually interested in my genre of game, the actual number of clicks and actual signups wasn't even noticeable. If I maybe had money to burn, I could double or triple the daily spend and see some results, but for a game like mine, where the budget is already pretty razor-thin, it just doesn't seem worth it.

The AdMob ad, on the other hand, for a similar per-day budget, had quite a noticeable impact. Below you can see the impressions I got over the last few days:

The impressions seem to peak while Google figures out the best number of impression to get the number of clicks I want. So the first two days were slightly above $1, but then it plateau'd and the following days all cost pretty much exactly my $1 budget. But you can see, 236 clicks over only 9 days. Click through rate is an order of magnitude higher at 0.32% and cost per click is only $0.04.

And in terms of actual results -- i.e. signups within the game -- well, I think the results speak for themselves:

The yellow line there is the number of new empires created on that day and you can see the big jump after the ad started running. The red line is "7-day active" empires (that is, users who have logged in to the game in the last 7 days) and the blue line is 1-day actives (i.e. the number of users who have logged in in the last 24 hours). You can see the obvious jump on the 11th, when the ad started running. It's only been running for 9 days now (compared to the Facebook campaigns 3 weeks), but if this trend continues, I seen no reason to stop this campaign after the three weeks.

Conclusion

The conclusion seems pretty obvious, at least for me, in my particular situation. Facebook ads do not work, and AdMod ads do work. Maybe I could have spent more time tweaking the Facebook campaign to get better results, but when "time is money", why would I spend time on that when I can spend the same time and get orders of magnitude better results with AdMob?

Update

So I've been informed that Facebook actually does support mobile-only ads. Putting in the URL to the app on the Play Store automatically triggers an "app install" mode which displays ads on mobile Facebook. I suspect this will be much more lucrative, so I'm going to give it the same $1 per day budget and give it a go for a few days. Expect a new blog post in a week or so with the results!

Today's update is a fairly big one in terms of adding to the game. I've added the ability to upgrade your fleets. Currently, there are three upgrades available, for three different ships. To access the upgrades, you visit the "Ships" tab of the build menu, where you will now see "existing" ships as well as "new" ships:

Tapping on one of your existing ships, you'll get a menu of the upgrades you can choose. Currently, each kind of ship only has one upgrade available, but in the future we may add some more. Choose the upgrade you want and click "Upgrade".

The build will proceed like an upgrade of a building. The main difference is that ships are available for upgrade from any colony (as long as it has a shipyard), so you can build a fleet of ships on one colony and upgrade them on another.

So what upgrades can you build?

Cloak

As you can see above, the scout has an upgrade called "Cloak". Scouts with this upgrade are invisible to enemy radar, but more importantly, they can sit on an enemy star undetected. If you set the stance of the scout to "netural" or "passive" (to stop it from trying to attack), then enemy ships will not be able to see it and you can send it to spy on your neighbours.

This is great for seeing what your enemy is up to, even on stars that are outside the range of your radar.

Boost

The boost upgrade is available for fighters and, when actived, halves the ETA of the fighter. So if you send a fighter to another star, and it initially says it will take 2 hours, use the "Boost" button to halve that to 1 hour.

Boost is a single-use function. Once the fighters arrive at the destination, the boost is lost and you'll have to upgrade the fleet again to get another boost.

Cryogenics

The final upgrade in today's update is called "Cyrogenics" and it applies to the colonyship. If you upgrade a colonyship with this upgrade, then instead of depositing 100 coloniests on a planet, it will deposit 400 colonists (or the planet's population congeniality, whichever is less). This'll give your new colony a bit of a boost when starting out.

I'm a sad panda today...

I just had my first real instance of a player causing quite a bit of strife for one particular alliance, stealing all their cash and booting out all of their players. Kind of ironic this happened right on Thanksgiving weekend!

In order to kick a player out of an alliance, you need to get 10 votes from existing members. But to join, you only need three votes. What this guy did was create two extra legitimate sounding empires, then ask to join the target alliance. Once he had three puppet accounts in, he created 7 more and used his puppets to vote those in as well.

Then he proceeded kick out all existing members, withdraw all of the cash from the alliance's bank and make everything a living hell for everybody.

So I quickly added a "ban" feature to the backend, which allows me to keep the empire in the database, but mark it as banned (which means he can't even sign up again using the same email address). I've also just posted an update to the app in the app store which will display a useful error message when you've been banned (if you keep using the old version of the app, all you'll see is you keep getting authentication errors but you won't know why).

So far, I've been quite lucky in that the game has been small enough that everybody got along pretty well. But I guess such things are inevitable when you get a big enough player base.

Ever since the Alliance feature was added to War Worlds, not a whole lot has been done to support them. We added alliance-private chat a couple of months back, but the basic management was largely unchanged.

That's all changing today as we introduce a better way to manager your alliance. Now, instead of everyone having the power to grant all requests (previously, the only request was "join alliance", but now there's more...) we've added the concept of voting. Different actions require different vote counts to pass. Joining an alliance requires 5 votes, kicking someone out required 10. But not all members are created equal: the "owner" of the alliance (basically, the person who created it) is given 10 votes (so he can pass any motion immediately), whereas "normal" members are given just 1 vote.

In a future update, I'll be introducing the concept of ranks within an alliance, so you can give certain privileged members additional voting power, or take voting power away from troublesome members. But for now, it's just the two levels.

Joining an alliance

So to see the new system in action, here's the new sequence for joining an alliance.

After an empire requests to join, you'll see the request in the "Requests" tab, like before. On the left, you can see "Hunters" has requested to join my empire. On the right, when I tap on the request, I get the option to vote "Aye" or "Nay". You can also see that only two votes are required here, this is because there's only two members of my alliance at the moment, so it would be impossible for me to get the 5 votes required...

The request already has +1 vote, so if I vote "Aye" the vote count will increase by 1 and the vote will pass. Were I to vote "Nay", the vote could would be reduced by 1 and drop back down to 0. If my alliance had more members, we could have some voting "Aye" and some voting "Nay", but the motion won't pass until the total vote count hits the required value.

Alliance Bank

Another big feature in this update is the alliance bank. This is a way for more established players to shared their cash with some of the more cash-strapped newer players.

Depositing cash can be done by anyone at any time (assuming you have enough cash to start with, of course). You can enter a reason if you like, but in any case, the cash appears instantly in the alliance bank account.

Withdrawing cash requires a vote, and 5 votes are requrired to pass a "withdraw cash" motion. As I mentioned, the person who created the alliance gets 10 votes, so they can pass a vote to withdraw cash immediately.

Kicking Empires

Also in this update is the ability to kick empires out of your alliance. From the empire details page, tap on the empire to bring up some details about it. Then tap "Kick" to initiate a vote to kick them. Kicking requires ten votes to pass.

Coming up

I know in my last update I said there would be more options for capturing planets, and that hasn't happened yet. It's actually turned out to be slightly more complicated than I was expecting, so it might be a little further off...

I'm going to continue tweaking things with the alliance stuff a little. As I mentioned above, one thing I want to add is the ability to define your own "ranks" of members, so you can give additional voting power to other members of your alliance if you like (or take voting powers away).

There's also some historical/auditing information that I can expose (for example, if some people have voted for a motion, you can't currently see who that was, but it would be nice to know). Similarly, a transaction history for the alliance bank would be a nice thing to show.

Finally, I'm also looking for some additional ways to monetize the game (in addition to just removing ads and renaming stars). One way is to allow you to customize your shield picture. Another suggestion I've had is the ability to "buy" abandoned empires (say, if someone doesn't log in for X days, their empire comes up for grabs to purchase and subsume into your own). I'm not entirely sure about that one, as it's coming close to "pay-to-win" and that's something I've wanted to avoid... so I'll have to think about that one a bit more.

It took about 3 weeks, but I've now completely re-written the backend for War Worlds and moved it off App Engine. I've been running for about a month now on Google Compute Engine and the results are in: it's fantastic (yes, I didn't move far from Google's Cloud... I'll post reasons as to why I chose Compute Engine over, say, EC2 or a VPS in a future blog post. Suffice to say, comparing cloud providers is actually very difficult).

The biggest win for me has been in terms of cost, and that's what I want to compare today. To begin with, a quick breakdown of how the game is designed, so you can see what the typical usage patterns are.

Backend Design

The War Worlds backend is basically a REST-like web application that provides an interface to a data store. The world is made up of "sectors" and each sector has a collection of stars. Each star is (mostly) a self-contained ecosystem. On a star you have multiple planets, on a planet (possibly) one colony, on a colony you have multiple buildings. Additionally, you can have fleets of ships around a star. Everything in the game takes a certain amount of "real-time" to complete. Ships take a few hours to travel between stars, buildings take a few hours to complete and so on. [The Getting Started guide has a good overview of the game from a user's point of view, if you're not familar].

If there is nobody actually playing the game, then the server sits (largely) idle. When you pull up a star in the game client, the server runs a simulation of everything that has happened on the star since the last simulation (calculating things like population growth, build progress and so on). When it runs a simulation, there are certain future events which may need to be triggered (such as when a build completes) that the game saves off to be triggered at the appropriate time.

Main Problems

There were a few things in the App Engine backend which were not ideal. Some were my own fault, some were due to limitations of App Engine itself.

The first was entirely my own fault. I had originally built the backend in Python, and the frontend - being an Android app - in Java. This posed a problem, because it made it impossible to share code between the client and server. I couldn't run the same simulation on the client as I did on the server, so in order to make things on the client feel responsive I basically had to duplicate the simulation code, once in Python and once in Java. This resulted in quite a few out-of-sync bugs where the client thought one thing was happening but the server thought another.

Additionally, I was using the Task Queue to schedule the future events that I mentioned above (such as build-complete, fleet-move-complete, etc). This was a problem because App Engine provides no (simple) way to cancel future tasks. So if you build something and it's queued for 2 hours from now, then you re-adjust your colony's focus so that the build should only take 1 hour, the game would re-queue the "build-complete" event in an 1 hour, and the one scheduled for two hours would still run, realise it had nothing to do and just exit.

Finally, App Engine has a hard 60 second limit on frontend requests. For the most part, this was fine. But certain requests started to take longer than 60 seconds, particularly when empires started getting larger. Collecting taxes requires simulating every star in your empire and when you've got 200 stars, that can take a while (more on this later, but App Engine seems rather slow for CPU-bound tasks). This mean I needed a complex system involving tasks and callbacks just to do something like collect taxes: the user would press the "Collect Taxes" button, which would fire off a task in the backend. When the task was complete, it would send a notification back to the client telling it to refresh the empire (with the new tax value). Having to wait 1 minute+ just to get the taxes from your colonies was quite annoying.

Costs Start Skyrocketing

For a while, the game was doing alright. It was costing me about $10 per week to run it and I didn't mind that I was losing money (it's always been a labour of love anyway). But around the end of March, usage started to climb. In a week, I had double the daily active users and by mid-April or so, I was seeing about 300 daily active users and about 2 requests per second to the server.

It doesn't seem like much, but my costs had skyrocketed. It went from about $10 per week to around $20 per day. This was becoming unsustainable, for an app that was making $100 per month, no labour of love was going to sustain those kinds of loses. I either had to dramatically increase revenue (I didn't want to do that) or drastically reduce costs. Looking at my billing data in App Engine, a couple of things stood out:

The three biggest costs were frontend instance hours, data store reads and data store writes. Unfortunatly, I couldn't see a way to reduce any of those. I was already using memcache to it's fullest, but in a game like this, around half of requests are mutate requests, which makes caching in general not that useful (it was still getting about 60% hitrate in memcache which I thought was pretty good).

Frontend instance hours is something I tried tweaking a few settings for. For eample, a larger instance costs more per hour, but because requests should complete faster, then theoretically fewer instances should be required. It ended up being a bit of a wash (and the smallest instances were too slow by far anyway). I also tried switched to reserved instances, which reduce costs by about 30% (from around $9 per day to around $6) but with usage still climbing, that saving was quickly wiped out again.

I had to do something, the longer I left it, the more money I was losing (again, it wasn't ever really breaking the bank, but a over a hundred dollars a week is something the wife starts to notice). Eventually, I decided that App Engine just wasn't worth the cost and I decided to port away.

New Design

I settled on two main requirements for the new design:

1. It should be portable from one provider to another, and
2. It should be written in Java, so that I could share code between the client and server

So the basic design I came up with was a server built on the Jetty servlet container, MySQL for the database and Nginx for the frontend (to handle HTTPS and the like). MySQL was really just chosen because I was familiar with it, and it has pretty good tooling. Communication between client and server was always based off protocol buffers and I decided the easiest way to build the new server would be to keep the exact same URL scheme and data interface, just switch out the "base" URL. I ended up adding a "realm" feature to the game client so that players could choose to join the "Alpha" realm (i.e. the old App Engine backend) or the "Beta" realm (the new one I developed) with only minimal changes required in the game client.

It took about three weeks of nights and weekends to get the server re-written in Java and running on Jetty (plus a few more weeks to iron out all the bugs). The changes to the client were relatively minimal, most of the code changes on the client were to allow it to switch between the "Alpha" and "Beta" servers. Re-writing the server in Java would also allow me, in time, to perform more of the simulation on the client, only simulating on the server when actually required, but for the initial version the new server did basically the same work as the old server did.

After running the new server for about a month, the user count is now approximately what it was on the App Engine backend, performance is much better, and costs are a fraction of what they were: around $230 per month, down from around $600. The new server is running on a single n1-highcpu-4-d instance of Google Compute Engine.

As you can see from the graphs above, we're doing approximately the same ~2 requests a second (slightly lower, which is likely due to the better way future events are queued). But CPU usage is barely above 1%, I/O and network (not shown above) are similarly around 1% of what the instance allows.

So I'm paying less than half as much as I was before, and I can easily sustain 10-20x times the usage without incurring any extra charges.

Comparison of Performance

As always, real-world performance comparisons are hard and this is not entirely apples-apples. The new server is written Java instead of Python, talks to a MySQL database instead of App Engine's HRD and so on. But from a user's point of view, both servers are exactly identical -- that is, they provide the same functionality and the same interface.

There are a few things which are (somewhat) directly comparable. Running the simulation of a single star is something that happens quite a lot, and is fairly easy to compare.

The above graph shows the difference in performance between simulating stars in the old server and the new server. The blue column shows the time taken (in milliseconds) just to simulate the star, without storing the result back in the data store. The red column is the time taken to simulate a star and store the results back in the data store (these times also include round-trip network time to the server, which is about 150ms for me, located as I am in Australia). In the old server, I was using the App Engine HRD which uses the Paxos algorithm to ensure data consistency in the event of failures. The problem with Paxos is it has inherit performance trade offs that make writes extremely expensive. In a game like War Worlds, where about half of all requests end up mutating data, you can imagine how this can hurt performance.

Above is the graph of SQL queries to the database. As you can see, it's about 2:1 SELECTs to UPDATEs, but I'd say that's mostly because I've done absolutely no caching at all (so for example, every request requires a query the "session"  table to get the user's empire ID and so on which could easily be cached to dramatically reduce the amount of SELECTs required).

What I miss

There's a couple of things I miss about App Engine. In particular, the ease-of-deployment and automatic data integrity management. The way you develop on the local instance, then run a single command to upload it to the server and switch to the new code with no downtime was great. I basically ended up writing an ant script that compiled the server, zipped it up and copied it to a directory on the server, then another bash script on the server which I could run that unzipped the files, and swapped out the live server with no (or almost no) downtime. It's not that hard, but on App Engine you just don't think about it.

Similarly with backups: on App Engine, data integrity is gauranteed so backups are really not required. I ended up coming up with a simple cron task that takes a full dump of the database every hour and copies it to a "permanent" storage device. In the event of catastrophic failure, I should be able to spin up a new instance, copy my image to it and get the last hour's backup copied over in fairly short order. The main issue is the fact that I sleep for ~8 hours a day and work for ~8 hours a day, meaning it could be some time between when a problem occurs and I'm aware of it.

Conclusion

At the end of the day, I still really like App Engine (and I will continue to use it for things like this blog). But it's clearly not well-suited to all workloads and unless you've got cash to burn, you do need to think about the typical workload of your application and whether it's actually suitable or not. If you have heavy CPU requirements or a high write:read ratio to the data store, then App Engine may not be the best choice.