Anson the Gnome

I finally had a chance to sit down with our lead designer today and discuss all the research I'd done thus far, and start discussing options.

One of the last things we determined (but which is important enough that I mention first) is that in addition to developing a ranking system, we must gather enough data on individual matches that we can analyze the results of the rating system afterward, as well as use the data after the fact to test other potential rating systems.

Anyway, the approach we're taking initially is a modified Elo system. The first modification we're making (reasonably, we think) is that the Elo rating for a group (in terms of match-making, and in terms of rating adjustments made after the match) is simply the average of the ratings of the characters in the group. These averages are used to calculate the expected outcomes E_A and E_B, which are in turn used to calculate the changes in ratings K( S_A - E_A) and K( S_B - E_B) based on the actual result.

That change is being applied because in general, lopsided groups are well-balanced in terms of the damage they can do (the larger team's damage is reduced, the smaller team's damage is boosted), as mentioned previously. It is an open question whether grouping with other players of significantly different skill effects the overall skill of your team in a linear fashion the way a calculated mean models it, but this is one of the reasons we'll be tracking results independently of the aggregate rating.

The second modification we're making is that for each member on each team, the score adjustment will be scaled linearly based on the distance of that member's rating from the mean rating of the team: |R_A-R_A1| * K( S_A - E_A). I'm also keeping in mind scaling the rating delta using a Gaußian distribution as an alternative, but again, we're starting simple.

Other modifications under consideration include replacement of K with a function K(G_A, G_B) that varies the potential size of the delta based on the (average?) number of games played by each side — essentially, the system used by Days of Wonder for Gang of Four to encourage experienced players to play newbies, and to let newbies reach their proper rating more quickly.

I doubt that these changes are novel, or even — to be honest — the best ranking system we could use (even accepting the performance constraint of "use as little CPU as possible"). Instead, they are a first stab at the problem, and one that may prove sufficiently accurate that a more complicated approach simply isn't warranted. The point of maintaining the data, and doing periodic comparisons of the existing rating system with other alternatives, is to figure out how complex the system needs to be before ratings stabilize and player skill is accurately recognized.

I don't have time to work on proper PvP rankings yet (improved grouping, and guilds, come first - we're still shooting for February on both of those), but I've started thinking about it so that when I have the time, I'll be reasonably ready with ideas. Currently the closest we come to PvP rankings is we track the number of wins each player has. Not losses, not win/loss ratio, just wins.

Of course, with a free game, there are (at least) two ways to achieve a large number of wins: play well, and play against stooge accounts who don't fight back. So we need a relative ranking system, one that puts up road blocks to stop — or at least slow down — efforts to get to the top without winning against the best existing players. One solution to this problem has already been implemented in Dungeon Runners: pairings are random. Of course, when PvP traffic is low, a player can still log in with two accounts and game the system... we still need some kind of relative rank.

One system designed to measure relative rank is the chess Elo ranking system, but it was also designed for very different constraints: although it's possible to convince someone to throw a match, it's not possible to manufacture an array of opponents, all of whom are willing to throw a match. More tellingly for the easy grouping of Dungeon Runners PvP, Elo only handles two opponents. Guild Wars guild combat and World of Warcraft arena rankings use Elo, but only to rank the formed groups - that doesn't work at all with pick-up groups, or tracking your rating when playing with a common group and playing alone.

Glicko and Glicko-2 are expanded models moderately based on Elo ranking (with a secondary rating that amounts to the confidence of the system in the rating). It has the same drawbacks regarding stooge accounts and grouping that Elo has, but has been further expanded by Microsoft in the form of TrueSkill™ to manage the additional information (and uncertainty!) of ranking individuals in a team vs. team match.

One problem none of these rankings systems have dealt with, however, is lopsided groups. TrueSkill™, for instance, relies at least somewhat (I haven't yet checked how deep this assumption goes in the calculations) on teams of even size for the way it calculates the relative skill of each team. It would take two very good players to have the same cumulative score as three decent players, yet in Dungeon Runners PvP the two players would have a sizable advantage due to the way balance between lopsided teams.

A simple solution is to compare score averages instead of score sums, but it seems like that loses a lot of the additional information you keep in score confidence values. Maybe it's good enough, maybe not... I'm investigating it. :-)

Another great analysis of these problems was written back in 2006 by Christopher Allen & Shannon Appelcline: Collective Choice: Competitive Ranking Systems. It's an interesting read and has definitely given me more food for thought on the subject.

I've been taken to task for not making more public the existence of our character XML feeds, but instead publicizing the players who have taken that raw data and done something with it. It's also somewhat questionable that we don't have our own canonical character sheet.

So first of all, obviously I'm mentioning the XML feeds right now. Beyond that... at some point we'll have character sheets on our site. For us, the priority was to get the data out to players, and see what the community wanted to do with it and how they approached it - this is our general approach, to let players see what we're doing and where the game is going before the whole thing is finished and perfected and polished. The polish phase of character XML has already begun, and we're hoping we get most of the way there with our upcoming patch - at which point, naturally, we'll start looking at what else to put up on the web. First as pure data, probably, and later as fancy web pages with useful information. :-)

So, NCsoft and Spacetime Studios have parted ways, shortly before the game they were working on entered production. This doesn't impact me or Dungeon Runners directly, but I didn't want to remain silent while STS has to lay off 12 people. Good luck, guys.

We're the runner up for "Best Free/Micropayment Game 2007" - and if you only count games that are completely free to play, we win! :-)