Scott Jennings has more details. I'm not sure if I can express this properly, but I was immensely grateful to Jeff for our online interactions. I started reading his blog when I first joined the Dungeon Runners team, mostly because he was (at the time) working for another studio in town being published by NCsoft. The more I interacted with him, the more I came to believe he was a genuinely wonderful person, as well as insightful and funny... and then he went and wildly praised one of my first big user-visible contributions.
The discussions that ensued, and his constant encouragement, had a huge impact on me and the direction I've gone since. I owe him a lot, yet all of our interaction constituted relatively little of his time; I can only imagine how many people he had that kind of effect on. My prayers go out to his family in this difficult time.
Friday, September 26. 2008
Object/Relational Impedance Mismatches
Bleh, what a boring subject. Sorry. I'm spending a lot of time looking at technology to decide where to go for the new project, very similar to Joe Ludwig's current focus. I'm also reading a lot of books, trying to get a feel for what I should be doing now that I didn't want to or couldn't do when I joined the last project.
After talking it over with the CTO, I think we're headed toward a Linux server: possibly cross-platform so that each dev can choose either Linux or Windows, possibly with a managed VMWare image, and possibly on a big shared dev server. It's crucial, in my opinion, to have at least some of us running Linux if that's our production platform, and the consensus seems to be that Linux should be our production platform.
Now I'm looking at "what database system to use?" The biggest problem I've experienced, in dealing with ORM, is: how do you get an entire object hierarchy with a single query? Examples are "an area, all of the map data for this area, all of the NPCs on the map for this area, all of the items the NPCs on the map for this area are holding" or "a character, all of the stuff the character is holding, all of the stuff contained in the stuff the character is holding." If you store the top-level objects as binary blobs and unpack in your game code, this problem is solved.
Otherwise, you're in the classic problem of recursively descending a tree in a query (this is discussed in depth in The Art of SQL, and Vadim Tropashko has also written a broad view of the subject with his own proposal). The latest versions of the SQL standard offer a way to do this ("recursive with"), and some RDBMS's implement it, but it's never very fast and it always looks a little ugly1.
Another problem frequently faced with common object/relational maps is figuring out which tables the data is in - this is particularly a problem when your object model uses inheritance, so you've got classes that share some data, but don't share other data. Again, this problem is largely solved when storing objects as binary blobs. You can a) put all the information for all possible objects in a particular inheritance hierarchy in a single table, or b) put all the information for each class in its own table (often repeating columns across tables that share a common ancestor), or c) split it up so that each class's table has just its new properties, plus a pointer to its parent class's table. All of these approaches have problems.
Picking the right database system involves picking what solutions to use for these problems.
SQL Server is a possibility, we have fairly recent experience with it and it can definitely work. It also supports recursive with syntax, so that you can recursively descend a tree in a single query. SQL Server also has the advantage that there are a lot of very good pre-existing tools to remotely manage a SQL Server database above and beyond what Windows itself offers for remote management. So, managing SQL Server machines is not as bad as other kinds of Windows-based servers, but there are still gaps.
Versant is interesting because of how completely it solves object/relational mappings and recursive tree traversal, but on the other hand ad-hoc queries against a Versant database are... less than optimal. It involves writing quite a bit more code, and quite a bit less work is done in the database kernel - so it has to be done after retrieving potentially enormous result sets. In a lot of ways, Versant has many of the same pros and cons as storing binary blobs in the database, and in my mind represents the state of the art in binary blobs.
PostgreSQL is interesting because it provides table inheritance out of the box; I'm still playing around with ways of turning that into a solid implementation of approach c), but I'm not there yet. Recursive with is currently only proposed for PostgreSQL 8.4, although there are community-supplied stored procedures to implement (opaque to the planner/optimizer) Oracle-style "connect by."
I haven't looked past those three yet. Oracle is used by some in the game industry, but I'm a little wary of their cost and baseline performance. MySQL... I'll be frank, I haven't seriously considerid MySQL for any task in quite some time.
1. There's also the materialized path and nested set models of handling tree structures. Nested sets are horrendously expensive for any data change (many, many rows have to be updated when a leaf node is added to or removed from the tree). This is simply not acceptable for a tree structure such as a character with a frequently-changing inventory.
Materialized paths are less expensive than nested sets on updates (you only need to deal with sibling nodes), but still significantly more expensive than the adjacency model afforded by recursive with or connect by. I suspect that with a large enough number of separate trees (characters, maps, etc.) a materialized path will also start choking.
Now I'm looking at "what database system to use?" The biggest problem I've experienced, in dealing with ORM, is: how do you get an entire object hierarchy with a single query? Examples are "an area, all of the map data for this area, all of the NPCs on the map for this area, all of the items the NPCs on the map for this area are holding" or "a character, all of the stuff the character is holding, all of the stuff contained in the stuff the character is holding." If you store the top-level objects as binary blobs and unpack in your game code, this problem is solved.
Otherwise, you're in the classic problem of recursively descending a tree in a query (this is discussed in depth in The Art of SQL, and Vadim Tropashko has also written a broad view of the subject with his own proposal). The latest versions of the SQL standard offer a way to do this ("recursive with"), and some RDBMS's implement it, but it's never very fast and it always looks a little ugly1.
Another problem frequently faced with common object/relational maps is figuring out which tables the data is in - this is particularly a problem when your object model uses inheritance, so you've got classes that share some data, but don't share other data. Again, this problem is largely solved when storing objects as binary blobs. You can a) put all the information for all possible objects in a particular inheritance hierarchy in a single table, or b) put all the information for each class in its own table (often repeating columns across tables that share a common ancestor), or c) split it up so that each class's table has just its new properties, plus a pointer to its parent class's table. All of these approaches have problems.
Picking the right database system involves picking what solutions to use for these problems.
SQL Server is a possibility, we have fairly recent experience with it and it can definitely work. It also supports recursive with syntax, so that you can recursively descend a tree in a single query. SQL Server also has the advantage that there are a lot of very good pre-existing tools to remotely manage a SQL Server database above and beyond what Windows itself offers for remote management. So, managing SQL Server machines is not as bad as other kinds of Windows-based servers, but there are still gaps.
Versant is interesting because of how completely it solves object/relational mappings and recursive tree traversal, but on the other hand ad-hoc queries against a Versant database are... less than optimal. It involves writing quite a bit more code, and quite a bit less work is done in the database kernel - so it has to be done after retrieving potentially enormous result sets. In a lot of ways, Versant has many of the same pros and cons as storing binary blobs in the database, and in my mind represents the state of the art in binary blobs.
PostgreSQL is interesting because it provides table inheritance out of the box; I'm still playing around with ways of turning that into a solid implementation of approach c), but I'm not there yet. Recursive with is currently only proposed for PostgreSQL 8.4, although there are community-supplied stored procedures to implement (opaque to the planner/optimizer) Oracle-style "connect by."
I haven't looked past those three yet. Oracle is used by some in the game industry, but I'm a little wary of their cost and baseline performance. MySQL... I'll be frank, I haven't seriously considerid MySQL for any task in quite some time.
1. There's also the materialized path and nested set models of handling tree structures. Nested sets are horrendously expensive for any data change (many, many rows have to be updated when a leaf node is added to or removed from the tree). This is simply not acceptable for a tree structure such as a character with a frequently-changing inventory.
Materialized paths are less expensive than nested sets on updates (you only need to deal with sibling nodes), but still significantly more expensive than the adjacency model afforded by recursive with or connect by. I suspect that with a large enough number of separate trees (characters, maps, etc.) a materialized path will also start choking.
Saturday, September 20. 2008
server platforms
Austin GDC made it apparent that where I work is not a well-kept secret - still, I'm obligated to not say anything just yet. We still don't have office space, but I've been looking into some of the decisions we'll need to make early on for server technology.
My impression, at this point, is that MMO servers are about 50/50 split between Windows and Linux. I certainly know of games, both live and in development, that have used both. Would folks who read the blog like to weigh in with their opinions on this?
One strong advantage to using a Windows-based server is how easy it is to do all of your development on a single machine. Run the database server, game server, game client, and any ancillary processes all on the same machine. With just 1-2 players, all the running services together shouldn't significantly strain a modern desktop.
I have mixed feelings about another argument that Halldor Guðjónsson made in his talk at AGDC on the server technology of EVE Online: buying all your technology from one vendor prevents all your various vendors from pointing to the left when you have a problem. That makes a certain amount of sense, but first you have to pay for support contracts on everything from the vendor, and then you have to have problems that rely on those support contracts for resolution. I don't think I ever saw that come up while working at NCsoft... it was always our own database/network/system administrators fixing problems. Investing in your own team's expertise that way makes a lot more sense to me when fixing problems on the server side (whether systems, database, or network problems) are so critical to your business.
A better argument presented by Guðjónsson - and this one implicit in his demos, rather than stated outright - was that managing the servers consisted of mostly talking HTTP to the game server processes themselves, and interacting with their built-in management interface. That requires the same investment regardless of platform, and overall looked like an excellent way of doing things. On the other hand, starting with a Unix platform means (again, IMO) that more management tools are available immediately, for integration with your own server, rather than needing to be developed from scratch.
Right now I'm investigating whether VMWare would be an adequate development solution. Performance is a particular concern, but also how easily I could make the normal management steps used during development (start, stop, reload data, verify data, etc.) straight-forward for our non-Linux staff. I think that would really have a huge impact on the viability of a Linux (or FreeBSD, or Solaris) server platform: how well it could be integrated into the independent iteration cycles of all programmers, designers, and artists.
My impression, at this point, is that MMO servers are about 50/50 split between Windows and Linux. I certainly know of games, both live and in development, that have used both. Would folks who read the blog like to weigh in with their opinions on this?
One strong advantage to using a Windows-based server is how easy it is to do all of your development on a single machine. Run the database server, game server, game client, and any ancillary processes all on the same machine. With just 1-2 players, all the running services together shouldn't significantly strain a modern desktop.
I have mixed feelings about another argument that Halldor Guðjónsson made in his talk at AGDC on the server technology of EVE Online: buying all your technology from one vendor prevents all your various vendors from pointing to the left when you have a problem. That makes a certain amount of sense, but first you have to pay for support contracts on everything from the vendor, and then you have to have problems that rely on those support contracts for resolution. I don't think I ever saw that come up while working at NCsoft... it was always our own database/network/system administrators fixing problems. Investing in your own team's expertise that way makes a lot more sense to me when fixing problems on the server side (whether systems, database, or network problems) are so critical to your business.
A better argument presented by Guðjónsson - and this one implicit in his demos, rather than stated outright - was that managing the servers consisted of mostly talking HTTP to the game server processes themselves, and interacting with their built-in management interface. That requires the same investment regardless of platform, and overall looked like an excellent way of doing things. On the other hand, starting with a Unix platform means (again, IMO) that more management tools are available immediately, for integration with your own server, rather than needing to be developed from scratch.
Right now I'm investigating whether VMWare would be an adequate development solution. Performance is a particular concern, but also how easily I could make the normal management steps used during development (start, stop, reload data, verify data, etc.) straight-forward for our non-Linux staff. I think that would really have a huge impact on the viability of a Linux (or FreeBSD, or Solaris) server platform: how well it could be integrated into the independent iteration cycles of all programmers, designers, and artists.
Sunday, September 14. 2008
Still no details
I was kind of hoping to have more to say this week, since I'll be attending Austin GDC and talking to a lot of people. I don't much like being the one who can't talk about what he's doing, but it happens.
On the other hand, I'm really looking forward to the conference, there are some really good talks scheduled.
On the other hand, I'm really looking forward to the conference, there are some really good talks scheduled.
Monday, September 8. 2008
Hot damn
I'm in the process of accepting a job offer. I'd rather not discuss the details just yet, but I'm pretty excited. Stay tuned for details...
