Anson the Gnome

The previous post about UI design vs. code architecture has been heavily edited. I try not to edit posts after they've gone out, but I got a lot of feedback that I was rambling and the subject of the post wasn't at all clear.

Plus nobody commented here, so I didn't feel like I was pulling the rug out from anyone!

I'm primarily a server programmer, but I still end up doing UI work every once in a while, and that's been true for the majority of the last three weeks. Not being an experienced old hand at UI design, I find myself falling back on first principles and decade-old books. I'm mostly happy with the results, but the code that results in implementing the interface... well, it's not the kind of code that I like to write.

One of the basic principles of interface design¹ is coherence:

Internal consistency means that the program's behaviors make "sense" with respect to other parts of the program.... One should strive towards the principle of "least surprise".

If one part of your interface shows that your sword gives you a skill bonus, everything that displays information on skills should incorporate that bonus.

Put another way, each individual part of the user interface that displays information will grow to depend on both the information it ostensibly displays, plus what every other part of the interface ever displays. Very little code is left loosely coupled to the interface: data hiding obstructs coherence, singletons and global variables spring up, even the code that takes actions based on user input gets more context-sensitive (making it hard to avoid keeping that code loosely coupled and non-repetitive), and so on.

Is this bad? Nah, not really. It's just one of the many pressures that (in the short term) pushes against loose coupling, DRY code, and good code architecture. A good UI is an important aspect of good software, including games. But sometimes... sometimes, the things I do to the code in the name of users just seems awful. :-)

EDIT: Edited for clarity based on early feedback.

---

1. Per A Summary of User Interface Design Principles. See? All my references are a decade old...

There's going to be a hopefully-brief outage later tonight while I update the system to OpenBSD 4.5.

EDIT: done. Aside from dovecot not gracefully upgrading, everything was fine.

I touched a little bit on continuous deployment in talking about Craftsmanship, but got sidetracked into a discussion of why downtime matters even if it "doesn't matter." There are two big technical hurdles to continuous deployment, in my opinion. The first is having sufficient automated test coverage that you're confident, every time you publish, that the bugs are in your design and not in the rest of your code. Let's ignore that one for a moment, because the second hurdle - constantly bringing servers down to update them, disrupting your service - seems to have further caught the fancy of one Bryant Durrell, former director of operations at Turbine.

He doesn't like the idea of continuous deployment, but he loves the idea of minimizing downtime to nothing (who doesn't?). On the other hand, I think the process he describes could use some work. There are a few things that really jump out at me, ways to minimize stress and downtime without even pushing that hard. The first is get rid of the deployment checklist, as quickly as you can. Replace that checklist with a button to push. If you miss doing some things when the button is pushed, if you discover corner cases the automated publish doesn't handle, you'll notice - and you'll notice your mistakes faster when your expectation is "push button, see publish" than when you are constantly having to re-evaluate in your head where you are in the process, what's left, and at what point you see results.

The other issue I see in Bryant's posts are the discussion of rollbacks. I'm not sure there's a single thing more heinous, from a player's perspective, than a rollback. It's a terrible error when a bug causes a crash before players' data can be saved, I don't think we should be planning, as developers, to instigate extra rollbacks. I don't have any specific advice that comes to mind here, but I think that if you're resolute in avoiding rollbacks, the way to get around having to rollback after a bad publish will open. Sorry that's not very helpful. :)

On to the meat of minimizing downtime... I think the biggest issue is really just a question of load balancing. All kinds of other networked systems are designed to cluster in such a way that individual machines can go down, without the entirety of the service being affected.

Game servers have more state than most, but on the other hand we're talking about planned downtimes here, not disaster recovery (although that's worth thinking about too). If you design the entire system around short-lived processes on a cluster, for example, then it should be possible to label an individual machine as being no longer in the pool of available machines while the process updates. Of course, there you probably run into an issue of data storage, but you can design around that as well - make it easy for an older version of the process to ignore new information in the serialized version of the player character, for example.

If you have longer-lived processes, a stronger hand-off procedure would be needed. However, given gigabit interconnects, it seems reasonable that a server about to be updated could contact a hot spare, convey its current state, and transfer data ownership in a very brief span of time. Once a server has relinquished control of any player state, it could be safely updated and restarted, ready to serve updated clients.

Another option seems to be faking "zero downtime" - go ahead and force everyone to restart their client and reconnect when the server they're on is updated, but do it as a rolling update (so the majority of your playerbase can continue to be online at any given moment), make it as easy as "I'm disconnected, now I reconnect and the game is updated."

I've been dragging my feet on getting the necessary approvals to go to LOGIN, and now I'm kind of glad: I don't have the exact date yet but I'll be attending a memorial service for my uncle around that time instead. It might end up being the week before LOGIN, but I don't think that makes a second week away from home particularly more appealing.

Maybe next year... and I look forward to seeing a lot of the same people (and possibly some of the same presentations!) here in Austin in the fall.