Apologies for this wall of text :)

I've started work on a new backend for War Worlds, which I'm hoping will solve a number of problems in one swoop. The biggest issue that I am facing at the moment, other than regular bugs and so on, is the fact that the game costs quite a lot to host on App Engine. When War Worlds first started, and for the first few months, hosting was quite cheap and while income from ads & in-app purchases still wasn't quite covering the whole costs, it was manageable. But over the last week or so, there's been an influx of players (which is a great thing!) and costs have skyrocketed to $20-30 per day. It's not exactly breaking the bank, and there's no danger of me going broke just yet, but for the number daily active users (for the record, about 500) and QPS (2-3) I'm doing, that's about an order of magnitude more than I would like to be paying.

Around half my costs go to App Engine's "frontend instance hours", with most of the other half going to datastore reads. I already make quite heavy use of memcache, to reduce the amount of datastore reads required, but the nature of the game is that things are largely not static, which naturally limits the amount of benefit you can get from heavy caching.

So I have begun work on re-writing the backend. Instead of Python, this time it will be in Java, and I plan to host it on something other than App Engine (I haven't decided exactly what yet, but I am thinking of using EC2). With this, I'm hoping to solve a couple of problems in one fell swoop:

1. By using the Java on both the client and server, I hope to be able to share a lot of code. Currently, the logic to simulate stars lives mostly on the server, but some if it was re-written in Java so that the client can do it as well. However, we've been plagued with a few out-of-sync conditions because the client-side prediction is not perfect. By having both client and server in Java, I should be able to make the client-side prediction much more accurate which will hopefully make these issues a thing of the past.
2. I will be able to switch data stores (I haven't decided yet if I'll stick with a key/value-like store or switch entirely to relational) but App Engine's consistency gaurantees are extremely strong (probably stronger than I need) and it causes write performance to suffer -- in a fairly dynamic world like War Worlds, that makes performance overall suffer.
3. The infrastructure needed for a "smooth" transition to a new backend will pave the way for supporting multiple "realms" in the future.

Client-side prediction

One of the biggest performance improvements I've made to the game over the last couple of weeks involved moving a bunch of logic from the server to the client. Actually, the server still performs all the same logic as before, but the client no longer needs to wait for the server to complete it, since it can run a lot of the simulation itself.

However, because the server is currently written in Python and the client in Java, it means I basically had to translate the algorithm and I have two version of the simulation that I need to maintain in parallel. This has caused a few issues already where the client thinks one thing and the server thinks another, and these are quite subtle issues to solve.

So by moving the server to Java, I hope to be able to share the exact same code between client & server and these issues should disappear. This will also allow for more advanced prediction on the client (so like in the "Build Confirm" dialog, the expected build times are woefully inaccurate, but with the full simulation available on the client, I'll be able to make those numbers accurate).

Different Data Store

The other concern for performance has been the write speed of the App Engine data store. The App Engine datastore provides strong consistency gaurantees, but it does so at the expense of write performance. Writes to the data store typically take 100ms or so, which is huge. I've tried to structure the application as much as possible to reduce the number of writes per request, and for the most part things perform reasonably well. But there are certain requests which simply take far too long to complete (collecting taxes from all of your colonies is the most pertinent example).

As I said above, I haven't decided yet whether I will stick with a key/value style of data store, or whether I'll go for a full relational data store. They both have their pros and cons, but one thing is for sure: which ever I choose, the performance is definitely going to be better than App Engine's.

Additional Realms

I kind of like the idea that everybody is in the same universe together. It makes the game feel more expansive. But a couple of things have come to my attention that make me think that multiple "realms", that are separated off from each other, might not be an entirely bad idea.

The first is the multi-lingual issue. For a while, we only had English-speaking players in the game. But there's been a lot of new non-English speaking players join, and I've noticed some... shall we say, "tension" between the two groups. I hate the idea of "racial segregation", and I would have loved for everyone to just "get along" (and I do plan on adding language-based filters to the chat), but I also think it might also be good to have language-specific realms. This would also allow for cool things like star names in your native language (assuming I can get the random name generation algorithm ported to other languages), which would be quite neat.

Also, additional realms could allow for things like realms with slightly different rules, realms with particular end-game scenarios (vs. realms where this is no "end-game" as such and you just continue expanding forever) and so on.

Conclusion

In any event, what you will see is that in the next couple of weeks, there will be an update to the game which will allow you to join the "Alpha" realm or the "Beta" realm. The "Alpha" realm will be the App Engine backend that you're all familiar with today. The "Beta" realm will be the new backend. Eventually, the "Alpha" realm will be switched off (hopefully not before everyone has volunterally moved to the "Beta" realm because it's going to be so much more awesome!) and all new development will continue with the new backend.

In the meantime, I will continue to update the game and fix bugs, but I think it's important to get this transition done as quickly as possible so that I'm adding features and bug fixes to both versions for as short a time as possible.

Please send me your comments and feedback on this change (leave a comment below, post to our Google+ community, or send me an email directly)