Anson the Gnome

Scott already has the best picture of the vulture stalking us. We've seen him outside the office a couple of times, but this time it seemed like he really wanted us to know he was there.

Joe Ludwig has been talking about continuous deployment recently. The basic goals of continuous deployment are two-fold: minimize downtime, and minimize the time between writing code and finding out it's wrong (it could be wrong for a lot of reasons, not just bugs).

An interesting counterpoint came from Ben Ziegler a couple of weeks later (I'm just following the trend in writing my take on the subject a couple more weeks on), arguing that downtime - and bugs, and lag - just don't matter.

First, let me point out that this isn't a new discussion, by any means. As of this writing, Bug Free Doesn't Sell is a wiki article last edited almost three years ago... on this exact subject¹.

On the one hand, I think Ben is right: "Stable, fast, fun. In that order" is not a mantra for a great game. "Fun. probably stable, and fast enough" is a better mantra. It's hard to see the direct benefit to maximizing stability and minimizing downtime, and it's always possible to over-think technical challenges and lose focus on the game itself in solving them.

On the other hand, I think there are a lot of indirect benefits. Doing things right attracts people who are interested in solving new problems (or old problems better), it reduces operational overhead (manual steps and emergency downtime require hands on deck - costs I'm probably more aware of, having worked down the hall from NCsoft operational staff), and it's another defense against the motivation-sapping attitude "nothing works right around here."

Hell, to some extent, you don't even have to succeed at minimizing downtime to see the benefits (c.f. Blizzard's launch of World of Warcraft). Mistakes were made, but they came more from inexperience in MMOGs specifically than bad culture, lack of investment, or having bad programmers. Now, though, that doesn't really apply; we have to create technical failures in other ways. We know better than to make the same mistakes, we'll recognize the same old mistakes coming... and if we do nothing but let them wash over us, we'll see our motivation and investment in the game disappear.

And then the game will suck, and people will argue about whether the problems were technical or not.

---

1. Incidentally, any programmers who aren't familiar with the Portland Pattern Repository's Wiki - it might be the oldest web-based programmer community around, and is the original Wiki. It can be hard to read at times, but there is a ton of stuff and it's easy to get lost in there for hours.

http://arstechnica.com/science/news/2009/02/aaas-60tb-of-behavioral-data-the-everquest-2-server-logs.ars

Is going around all kinds of sites; I came across that article by way of Slashdot, Damion Schubert came across it on Massively, etc.

How do you process 60TB of data? I'd argue that the right approach is to spread the data out across many machines, and split the processing into a number of tasks to map it to the questions you're trying to answer. Then you'd probably need to reduce the results from the cluster into a single set of data, storing intermediate results in a giant key/value store. Which is to say, use Hadoop.

I wonder what Sony is using? I wonder what the guys in academia will be using?

I keep on trying to come up with something to say here about NCsoft and the layoffs. I dunno; I feel like most of what I want to say would break some trust placed in me. What's terrible is that for the most part, I'm not particularly surprised: it's astonishing what the (public) rationales are each time, it catches me off guard who the targets are each time, but that it's happening? Nah.

I hope the best for my friends still there, working on Dungeon Runners and Aion and scattered through out other parts of the company. I hope the ones who got laid off today, the ones who got laid off six months ago and are still looking for work, and the ones in between find jobs soon.

Other than that, I think Scott Jennings says it best: ENOUGH WITH THE GODDAMN LAYOFFS.

A friend pointed me to an interesting ACM Queue article about Darkstar, Sun's open source Java-based MMOG server technology. It's a few months old, but hey, I just saw it!

One immediate reaction is to pooh-pooh Java for games, or high-concurrency servers; but the truth is, both have been done in Java before (the Star Wars Galaxies server tech was reportedly a mix of Java and C++, with most of the game logic - I think - being in Java). And anyway, as evidenced in the big debate from late '07 about scripting languages, we've been debating whether a pure compiled C++ (or C, in the case of Cryptic) or a lower-performing, easier-to-develop hybrid of C++ and Python/Lua/whatever for quite some time - with different developers taking different stances.

So "Java is too slow! Nobody uses Java for games!" is a tempting, but not entirely accurate response.¹

Another reaction is one of indignation: I think linking MMOGs to stand-alone, single player games (or regular multi-player games like Unreal Tournament) ignores the strong historical connection from MUDs to MMOGs. Meridian 59, Ultima Online, EverQuest, the early games that helped define this genre are all MUD-inspired, featuring people who started as MUD players and wizards doing everything on the server. I don't particularly care for being referred to as "people who grew up on a single (PC) machine, running a single thread," for whom being asked "to master the intricacies of concurrent programming or distributed systems takes them away from their concentration on the game or world experience itself."

How Dr. Waldo describes us is, still, not a very compelling argument against Darkstar. Take a look at how some game developers talk about their players... the games still stand on their own merit, good or bad. One of the biggest problems I see, actually, is that a Java server and a C++ client (i.e., even a moderately demanding 3D client) will require two separate code bases. There will be a lot of duplicated code for game logic, network protocol, and so on, with a natural inclination to make the two look very similar (or possibly be generated from some master source, a significant engineering project on its own). I have no idea how SWG approached this problem, but they at least had the benefit of a hybrid C++/Java system where presumably the shared code stayed in C++.

I'm also very negatively inclined toward the data access layer. Everything is atomic? Collision detection and aggro calculation are just two of the things that come to mind that could lock the data for every player in a vicinity while they're accessed, and make the game stutter. Mix in the argument against geographic decomposition, and things get even worse - much, much faster than your ability to buy additional hardware.

I am pretty skeptical of Darkstar. At the same time, I expect the trend of which it's a part - making MMOG development more accessible to more people - to continue. Through a combination of middleware, new languages, and more web-centric development, MMOG clients and servers will continue to cover more of the spectrum between "EVE Online" and "Facebook."

Bleh. I should go to sleep. :-)

---

1. Part of me still resists: in a lot of ways I don't see Java as being particularly better than C++, especially as I've been exploring Boost further. In some ways, Java seems like the worst of two worlds: very little of the dynamic flexibility of proper scripting languages like Python or Lua or Ruby, and very little of the performance from C++ (never mind C). There are other perspectives, of course, like the fact that the learning curve on C++ is much steeper than any of the other languages; or that Java supports a view of good software engineering that the others - including C++, with its operator overloading and template specialization and so on - simply can't support the same way.