War Worlds is looking for a new home

Posted by

So it's been almost exactly one year to the day that I initially announced "shutting" War Worlds down. There was a lot of support for the idea of keeping it running, and I've tried to do that -- the donations I received at the time allowed me to keep War Worlds running for several months, but the time has come that I can no longer keep the servers up for much longer.

But rather than shutting down completely, I'm looking for someone to keep the fires burning and host the servers. Development of War Worlds has slowed down, but we've got a dedicated player base and it would be sad to see that go.

So if you're interested in hosting the War Worlds server, here's what's required:

  • At least 2 CPUs
  • 4GB RAM
  • ~20GB of disk space (depending on how many local backups you want to store)
  • The ability to run a custom Java server + cron jobs (basically that means we need a VPS)

The game needs a reasonably beefy server to keep all of the stars simulated, which is what as contributed most to the cost for me.

If you have a server which fits the bill available, please contact me and we can get started on the migration.

I'm basically going to run out of money to continue paying the bills in around a month, so we're going to need to find a solution fairly soon.

Wormhole Disruptors and mega-improved Starfield view

Posted by

Today's update of War Worlds has two big improvements. The first is there's a new "tactical" view of the starfield, which should generally be much better performing. You can see an example of it here:

That's what you'll see when fully zoomed out, the circles will fade away and the labels/icons will fade back in as you zoom in. You'll also notice that scrolling has been completely fixed, and you don't get that weird jumping around that you used to get in previous release. Man, that one had been bugging me for ages!

Also fixed in this release is the purchase of shields and empire names. In previous versions, if a shield change failed to be applied due to an error on the server (recently I had an issue where the server had run out of space, so all uploads were failing) the client would still "consume" the purchase and you would have been charged. Obviously that's not a great experience, so now purchases are only consumed once the client gets confirmation from the server that the change actually applied. If you get an error, you can retry without being charged again. (I should note, if you continue to get errors, please email me with your empire name and the change you want to make... I'm quite happy to do these for free if the purchase fails)

Wormhole Disruptors

But by far the biggest change in this release is the addition of the Wormhole Disruptor building. This is a building you can build on your star, and it allows you to "disrupt" (either destroy or take over) wormholes nearby.

Note that because it's a building you build on a star, in order to take over an enemy wormhole there is actually quite a bit of work you need to first. You need to take over a star near the target wormhole, build a disruptor on it, then upgrade it so that has enough range to reach the wormhole. You can see the range of your wormhole disruptor in red on the starfield view:

Once your wormhole disruptor is upgraded enough, two new buttons will be enabled on the enemy's wormhole screen, "Take over" and "Destroy":

The two buttons do what you would expect, "Take over" allows you to take ownership of the wormhole yourself, you can then tune it to one of your alliance's wormholes (if it's already tuned when you take over, the tuning will be reset). Destroy will simply destroy the wormhole (and any ships that are currently at the wormhole will also be destroyed!)

Some other improvements have been made to wormholes as well. You can now see who owns a wormhole when you select it in the starfield view, and there's also a "Locate" button which allows you to jump straight to the wormhole it's tuned to.

There's a bunch of other fixes in this release, head over to the github issue tracker for the complete list.

OK, so we're not shutting just yet...

Posted by

After my last post, I got quite a bit of feedback from people that they didn't actually want to shut the game down. And in fact, I'm quite happy to keep it up and running and also act as a sort of "project manager" if people wish to contribute bug fixes and so on. So in that vein, War Worlds is not going to be shut down.

However, you will have noticed over the last few months that there basically hasn't been any updates to the game at all. That's kind of a shame, but I'm hoping I can continue to contribute to the game, though perhaps at a slower pace than before (that's probably a good thing, I wouldn't want to burn out all over again!). As mentioned in the previous post, the complete source code for the game has been published on Github, and I'm more than happy to accept patches from anybody willing to contibute.

If you're not a programmer or artist and you can't contibute materially to the game, then I'm also in quite a bit of need of financial assistence to keep the server up & running. I have considered in the past moving to a cheaper host, but in the end, I stick with Google for a few reasons:

  1. They're fast. War Worlds is surprisingly CPU-intensive, and I need at least a few cores available to have decent response times,
  2. They're extremely reliable. I haven't touched the servers once in the last two months, and they're still humming along quite nicely.

In any case, what this basically means is, I'm always in need of financial assistence, there's two ways you can help with that. One is by purchasing in-game items (rename stars, and so on). But that gives a 30% cut straight to Google. Another way that I've just introduced, is the PayPal "Donate" button you can see at the top of the page here. Here is the button again, in case you missed it:

Click it. Donate. It really helps :-)

War Worlds is shutting down

Posted by

It still seems like yesterday when I announced the initial alpha of War Worlds is available on the Play Store. But it's actually been over one and a half years! In fact, it's been over three years since I started the project back in 2011! But, sadly, I feel I can no longer give the project the time and attention it requires and we must close our doors.

It's been an awesome time, and I've certainly learned a lot. Some highlights for me:

  • My initial post about the game on the r/IndieGaming subreddit is where I got around 100 initial players, the game has since been downloaded from the Play Store over 45,000 times!
  • At it's peak, War Worlds had over 2,000 active users!
  • Some of my posts about the game reached the Hacker News front page, not once, not twicenot event thrice, but four times!
  • I came to the attention of some Facebook employee for that last post about advertising, and they helped me tweak my campaign a bit. In fact, my post about App Engine also came to the attention of some people within Google as well. All in all, it seems like getting on to the front page of Hacker News is a pretty big deal.
  • Perhaps a low-light, rather than a high-light, but getting hacked last month was actually kind of an honour. Perhaps that's when you know you've really gone mainstream...

Now, I still have about a week and a bit left of time before the billing cycle for my server resets, so I plan to keep the game running until 30th August, at which point, it will be shut down.

If you're interested, I've also made the complete source code for the game (both client and server) available on GitHub: https://github.com/codeka/wwmmo.

What's next for me? Well, I'm not really sure. I've got plenty of ideas for new games, but I think I'm going to take things easy for a month or two and take some time out with my family.

Thanks a lot to everyone who has made this game so special, it will always have a special place in my heart!

Negative Ships and Unlimited Money Postmortem

Posted by

I'm sure everyone who plays War Worlds regularly will have noticed that there was a bit of an "incident" over the last couple of weeks. This post is basically a quick postmortem of what went down, and what I've learnt from the experience (tl;dr: a lot!).


Before we get started, first a little background about how the game actually works. The server-side is basically just a web server, that receives HTTP requests from clients. Everything is stored in a PostgreSQL database and the web server provides a layer of business logic on top of that database. The actual request/response is encoded using Protocol Buffers, which are an efficient binary encoding scheme (at some point, I'm thinking of also switching to SPDY instead of HTTP as well, for extra efficiency).

So for example, when you go to build a fleet of ships, say, the details of the request (such as the design you want to build, number of ships and so on) are serialized in a protocol buffer, and the request is posted to https://game.war-worlds.com/realms/beta/buildqueue.

Now, anyone who's ever built a web form will know, it doesn't matter whether you validate all your inputs in JavaScript in the form, you still need to validate them again on the server-side, because anyone can build their own form to post to your server. Well, the same is true here. Even though all inputs are validated in the game client, they also need to be validated by the server as well.

Generally, I think I do a pretty reasonable job of that, but I've been a somewhat lucky to date, since it's slightly more difficult to directly post requests to the server outside of the game client compared with a web form. Slightly more difficult, though not impossible!

Enter Proto Baggins

Proto Baggins is a game developer himself, he works for a major game company. So reverse-engineering the protocol buffer-based protocol is a pretty straight-forward affair. Initially, he was just using it to automate some tasks (presumably builds and expansions).

In addition to the automation, he also built some interesting tools to export visualizations of the game. For example, here's a visulation of his empire:

This is actually a pretty interesting visualization, since it shows the poisson distribution of stars pretty well (that's the algorithm I use to ensure stars are placed randomly, but never too close or too far apart). I go into the details of that algorithm a bit in this old post from a few years ago.

Here's another visualization Proto made, this time showing the entire War Worlds universe:

This time, you can see distribution of empires is roughly circular. This is because when a new player signs up, I first try to slot him in a star somewhere in the middle (usually where another empire joined, but then abandoned). If there's nowhere in the middle to add them, then new signups get added to a piece of empty space as close as possible to the "centre", leading to a roughly circular distribution. Also interesting here you can see all of the early players clustered around the middle, with some of the newer player's large empires around the outside. There's roughly 1.7 million stars represented here.

Negative Ships

Now, somehow, during Proto's probing of the game (which, I will say, is something I am not necessarily opposed to), he managed to get a fleet with a very large negative number of ships. I'm not entirely sure how this happened, but I suspect there was an integer overflow somewhere. An integer overflow happens when you have a very large integer value (> 2 billion) and it wraps around to a very large negative number instead. The issue here is that now, instead of costing money to send ships to other stars, you would be earning money to send ships to other stars!

Now, this doesn't actually get us infinite money, but it's quite easy to generate as much as you'd like. However, there was another glitch which did allow you to get infinite money, which I'm dubbing the Infinite Money glitch.

Infinite Money

This glitch was actually much more serious, in that it it actually allowed you to just generate money on demand (by depositing it into an alliance, and then having someone else take it out). I'm also not entirely sure how it was triggered, but it seems to have something to do with adding upgrades to fleets with "0" ships in them. Accelerating these builds seems to have caused the empire's "cash" value to go to NaN, which is a rather special number that tends to "infect" every calculation you make with it.

What Went Down

The exploits of Proto Baggins is what actually tipped people off to something going awry. People noticed his negative ship count, and that was quite suspicious. Next, they noticed that players who had only been in the game for a relatively short time were somehow able to amass massive armies of millions of ships, which they would launch against unsuspecting empires, who then had to scramble to deflect it.

What I discovered is that two empires, Carnage and Fatcow240, had hit the "Infinite Money" glitch and were depositing vast sums of cash into the alliance "Central Bank". They would then let their friends join that alliance, withdraw cash, and the cycle would continue.

Here's an example of what players would've seen before possibly getting completely wiped out:


After I figured out what was going on, I decided to ban the players who were generating money. So that's Proto Baggins, Carnage and Fatcow240. Not everyone has been happy with that decision, but I felt it was warranted for two reasons:

  1. While there are glitches from time to time, and glitches can sometime last for a while before I am able to fix them, this particular glitch was esspecially bad in that it only available to a few players who were "lucky" enough to have triggered the edge condition needed. Other glitches, such as the one that allowed players to create thousands of wormhole generators, were available to everyone: nobody was unfairly disadvantanged (at least, not once they were made aware of the glitch's existence), and
  2. The advantage gained from the exploit was disproportionately large. Essentially infinite. The wormhole generator glitch was relatively harmless, in that once you've created the wormhole generators, there's not much you can do with them, since they're so expensive to move anyway. This exploit allowed a player to launch massive armadas of ships, which are basically impossible to defend against by anyone who was not also in control of an infinite supply of cash.

In addition, I went through the audit history of the "Central Bank" alliance, and anybody who withdrew cash from that alliance has had all their cash confiscated. I thought about also removing any ships that they accelerated the build of after getting their cash, but decided that it would be quite a lot of work to go through everybody's fleet list and remove the ones that seemed to be affected. I figure if you have no cash to actually move those ships, then it's not as bad.

What Went Well

There were a couple of things that helped resolve the situation.

Firstly, every single cash transaction in the game is stored in a special "audit" table. Every time you accelerate a build, move a fleet of ships, or deposit or withdraw to/from an alliance, a record is added to the "audit" table. This allowed me to easily go back and find out everyone who had withdrawn cash from the "Central Bank" alliance. It also means I could pinpoint when Carnage and Fatcow240 had managed to trigger the "infinite cash" bug, which then allowed me to narrow down the cause to an accelerate of a "0-ship" build.

Secondly, while my server-side code isn't perfect at filtering out bad requests, I think it's done a fairly decent job overall so that Proto Baggins, even though he was making requests directly to the server, wasn't able to do too much damage (apart from triggering the negative ship thing, as I noted above, which I consider to simply be a bug somewhere). This could still be somewhat improved, though.

The bugs themselves have now all been fixed, and I managed to fix a couple of other glitches which have been bugging people as well (such as the aforementioned wormhole generator bug, and another one which let you build boosters and such for cheap).

Finally, the players themselves have been quite supportive of some of the tough decisions I had to make, and also at reporting the issues. People sent me screenshots, snippets of conversations, and all sorts of details which greatly helped to track down the issues. Thanks to everyone who has reported an issue in the past!

What Went Wrong

Not everything was perfect, of course. The biggest problem is that I didn't realise anything was wrong until quite a long time after it started happening. This is mostly my own fault, because my email inbox receives quite a bit of mail from not just players, but also app store marketers (spammers, essentially), and all sorts of people, so things tend to slip through the cracks.

Also, my "audit" table is literally just a table in the database. So actually trawling through it look for evidence is a little time-consuming, concocting SQL queries and parsing the output into csv files and so on.

Finally, I'm just one person. I wish I was able to dedicate more time to community moderation and monitoring, but I generally only have a couple of hours per day that I am able to dedicate to War Worlds (usually after the kids are in bed), and I'd much rather be working on new features of fixing bugs than tracking down people exploiting bugs...

Action Items

Now, all of the actual issues should have already been fixed (i.e. the bugs that caused these glitches in the first place), but a couple of additional action items have come out of all this:

  1. Spend more time in the chat. Keeping up with all of the email I get is a little tricky, but simply jumping on the in-game chat every now and then seems like a great way to keep on top of things (I do play the game as well, but I usually don't have time throughout the day to read chat as well),
  2. Get a proper issue-tracking website. This one seems like a no-brainer, but some issues can linger for weeks or months in my inbox because I don't have a central way of tracking everything that people report. This has the added advantage that spammers marketers won't be contacting me there.
  3. Monitoring and alerts! I need better monitoring for suspicious activity. If I'd been monitoring for things like spikes in requests per second, 400-errors per second and so on, I would have easily caught Proto Baggins before he'd been able to amass his large army.

So look for some of these changes coming in the next few weeks!