I am quite impressed with what Turbine did on My LotRO, and I wanted to take a minute here to make a note of it.
They do a few things I consider odd, like using WordPress as (what looks like) the starting point of the whole thing, but overall it's impressive in its polish and thoroughness. I can't say I'm surprised they have so much data about their players - Darius Kazemi got his start in MMOG metrics at Turbine - but that's just one more thing to be jealous of. Kinship event calendars, event logs going back at least to March of '07 (that's the earliest I found in a brief perusal), lots of AJAXy interactive bits all over the page and (by-and-large) good clean URLs1.
Initially they chose (mistakenly, in my opinion) to show all the characters that were tied into a single account, but it looks like they've done a quick hack to hide that while they go back and reconsider it. Hopefully, they'll take it out of the site design completely. We often have a lot of different virtual identities, and it feels like an invasion of privacy to show the links between them. What if I'm avoiding letting an annoying friend know about the character I usually play as, but I keep around another character so we can play when I'm up to it? Busted. It may have also been a security hole, if the player names being displayed were the actual logins.
The journals are great, and it's clear they got a lot of benefit out of WordPress here; it looks like the event calendars are a WordPress plug-in as well, so that probably saved them quite a bit of effort on the web development side. Using WordPress also meant that the journals are easily exposed in RSS format, something I had to hand-code on Dungeon Runners for the character blogs2.
Still, I think it would be more awesome if players could add to their journal from in-game - I'm pretty sure this is not the case, but can someone correct me if I'm wrong? "Balrog has killed me three times in a row! Getting frustrated" would be a lot more likely to show up - and less likely to disengage the player - if they didn't have to log out to type it. Also, I'm curious about whether links are allowed in the journals? I haven't seen any links yet, but perhaps no one has had the urge yet. The ability to link to other players' journals, and comment on specific journal entries, might be enough to render official forums obsolete3.
And my final question... where's the raw data? It looks like character profiles, guild rosters, and so on wouldn't be too hard to screenscrape, so someone is probably already starting on it. Is Turbine working on providing this data?
1. Should something go terribly wrong with WordPress (and it's happened before), "replace it" is a reasonably valid option. Implementation independence is a big win!
2. As far as I know, this still isn't actually live in Dungeon Runners. I hope it is one day, but it's out of my hands now.
3. Maaaaaybe. I haven't really given it much thought, but it seems plausible and is worth discussion, I think.
Tuesday, December 2. 2008
Metrics: Planning
I haven't said much about metrics here for about eight months, but that doesn't mean I haven't had more to say. I've just been sorting it out. :)
As I mentioned last time, there are different kinds of metrics to meet different needs. Operational metrics can often be gathered into 5-minute averages as they come out, those can be aggregated into hour-long averages a few hours after they're generated, and a week or two after that they can be rolled into 24-hour data points - the amount of data you need to actually store is tiny. It's trivial to store this data in MySQL, never mind every other relational database on the planet. Most of the work is even already done; just feed it to things like Cacti or MRTG and you're done.
The data that is ultimately visible to players on the web is also a relatively small amount of data: keeping the last 1,000 character events around for a blog system, a full character sheet, PvP rankings, etc. could all be stored in a handful of GB. Any relational database you care to name can probably handle it, on a cheap server from whatever vendor you like, on any storage solution you like - the whole thing can even be backed up onto a $20 flash drive if you want backups. Again, this is not a big deal.
...And then there's the meaty metrics, the things that let you generate pretty graphs of "death by level and zone" for the past month of game activity. Turns out, that's quite a bit of data. Darius Kazemi identified storage to be one of the big hurdles in dealing with gameplay metrics, in his talk with Larry Mellon at Austin GDC: Wake Up and Smell the Metrics! From the informal survey I've performed, a terabyte of data per month is easy to generate with decent coverage and a decent number of players; the field of suitable databases suddenly narrows.
To handle a terabyte of data (assuming you can condense most of the data down to reports once it's a month old), you have to invest a bit more in your database server: a RAID of high end disks is the tip of the iceberg, 8GB RAM is a good starting point, and 4 cores is a minimum (CPU is almost definitely not your first bottleneck). You're probably shelling out big money for your database license now, and adding capacity is far from a linear exercise.
In game development, we're often inclined to point to these kinds of problems and proudly declare that we're working on problems no one else is, so we probably can't use any one else's expertise. It's quite often true in game AI programming, graphics, and so on, but in this case the opposite is true: we've got options on who to emulate. The approach I outlined above is close to a low-end enterprise data warehousing model; they probably use more Java and have bigger budgets, but I think it's recognizable. MMOG servers can resemble enterprise software so thoroughly at times that even Sun Microsystems has started getting involved!
...But it's not the only model to follow. As it turns out, web developers deal with very similar problems: the access log on web servers gathers usage data at the same kind of granularity as "every time a character swings their axe," and moderately popular sites probably see data quantities on par with most MMOGs. Facebook claims it's logging system handles tens of billions1 of messages per day, which is hundreds of thousands of messages per second.
And you know what? These guys aren't logging all data to a central server immediately, because it doesn't scale. They aren't even trying to parse and load into databases. They aren't buying really expensive machines to choke down all of that data. Instead, they're going distributed, and utilizing something like Google's MapReduce - often in the form of Hadoop.
For comparison to the aforementioned "baseline beefy database server," I took a look at what a small cluster of decent Dell machines could provide. $10,000 - what could be, by itself, the price of a database license - can build a small cluster with 8TB of triply redundant storage2. With a system like MapReduce, you can actually utilize all that distributed horsepower to chew through data analysis. What's more, storage and processing capabality scales somewhat linearly with cost!3
I haven't been able to put this talk to any serious test just yet, but I have some ideas. Time-to-level data should be easy to generate, deaths per level, and even the relative popularity of different zones should all be straight-forward. It's a very different paradigm from relational databases, but then SQL presents its own unique challenges too. It's not a full solution yet by any means, but on the other hand the ability to store that much data, and process it however you like... it has a lot of potential.
1. High Scalability has the article where I first heard the number, although their math is off by an order of magnitude. Some of my arguments here are echoes of the conclusions over there.
2. You can easily hit 16GB RAM total, 32 cores, and 24 1TB disks spread out across 4 machines. Each datum being stored exists on three of the four machines, so whatever kind of hardware failure you have - up to and including one of the machines completely failing - data isn't lost. You also have higher total I/O bandwidth.
3. Some things, like network bandwidth or data center footprint or heat output, cause plateaus where the next step up requires only a modicum of computer hardware cost, but significant other expenses. So it's not completely linear, but compare it to scaling up a database server :-)
EDIT: added a footnote, and re-worded a few things - nothing big. :-)
As I mentioned last time, there are different kinds of metrics to meet different needs. Operational metrics can often be gathered into 5-minute averages as they come out, those can be aggregated into hour-long averages a few hours after they're generated, and a week or two after that they can be rolled into 24-hour data points - the amount of data you need to actually store is tiny. It's trivial to store this data in MySQL, never mind every other relational database on the planet. Most of the work is even already done; just feed it to things like Cacti or MRTG and you're done.
The data that is ultimately visible to players on the web is also a relatively small amount of data: keeping the last 1,000 character events around for a blog system, a full character sheet, PvP rankings, etc. could all be stored in a handful of GB. Any relational database you care to name can probably handle it, on a cheap server from whatever vendor you like, on any storage solution you like - the whole thing can even be backed up onto a $20 flash drive if you want backups. Again, this is not a big deal.
...And then there's the meaty metrics, the things that let you generate pretty graphs of "death by level and zone" for the past month of game activity. Turns out, that's quite a bit of data. Darius Kazemi identified storage to be one of the big hurdles in dealing with gameplay metrics, in his talk with Larry Mellon at Austin GDC: Wake Up and Smell the Metrics! From the informal survey I've performed, a terabyte of data per month is easy to generate with decent coverage and a decent number of players; the field of suitable databases suddenly narrows.
To handle a terabyte of data (assuming you can condense most of the data down to reports once it's a month old), you have to invest a bit more in your database server: a RAID of high end disks is the tip of the iceberg, 8GB RAM is a good starting point, and 4 cores is a minimum (CPU is almost definitely not your first bottleneck). You're probably shelling out big money for your database license now, and adding capacity is far from a linear exercise.
In game development, we're often inclined to point to these kinds of problems and proudly declare that we're working on problems no one else is, so we probably can't use any one else's expertise. It's quite often true in game AI programming, graphics, and so on, but in this case the opposite is true: we've got options on who to emulate. The approach I outlined above is close to a low-end enterprise data warehousing model; they probably use more Java and have bigger budgets, but I think it's recognizable. MMOG servers can resemble enterprise software so thoroughly at times that even Sun Microsystems has started getting involved!
...But it's not the only model to follow. As it turns out, web developers deal with very similar problems: the access log on web servers gathers usage data at the same kind of granularity as "every time a character swings their axe," and moderately popular sites probably see data quantities on par with most MMOGs. Facebook claims it's logging system handles tens of billions1 of messages per day, which is hundreds of thousands of messages per second.
And you know what? These guys aren't logging all data to a central server immediately, because it doesn't scale. They aren't even trying to parse and load into databases. They aren't buying really expensive machines to choke down all of that data. Instead, they're going distributed, and utilizing something like Google's MapReduce - often in the form of Hadoop.
For comparison to the aforementioned "baseline beefy database server," I took a look at what a small cluster of decent Dell machines could provide. $10,000 - what could be, by itself, the price of a database license - can build a small cluster with 8TB of triply redundant storage2. With a system like MapReduce, you can actually utilize all that distributed horsepower to chew through data analysis. What's more, storage and processing capabality scales somewhat linearly with cost!3
I haven't been able to put this talk to any serious test just yet, but I have some ideas. Time-to-level data should be easy to generate, deaths per level, and even the relative popularity of different zones should all be straight-forward. It's a very different paradigm from relational databases, but then SQL presents its own unique challenges too. It's not a full solution yet by any means, but on the other hand the ability to store that much data, and process it however you like... it has a lot of potential.
1. High Scalability has the article where I first heard the number, although their math is off by an order of magnitude. Some of my arguments here are echoes of the conclusions over there.
2. You can easily hit 16GB RAM total, 32 cores, and 24 1TB disks spread out across 4 machines. Each datum being stored exists on three of the four machines, so whatever kind of hardware failure you have - up to and including one of the machines completely failing - data isn't lost. You also have higher total I/O bandwidth.
3. Some things, like network bandwidth or data center footprint or heat output, cause plateaus where the next step up requires only a modicum of computer hardware cost, but significant other expenses. So it's not completely linear, but compare it to scaling up a database server :-)
EDIT: added a footnote, and re-worded a few things - nothing big. :-)
(Page 1 of 1, totaling 2 entries)
