MMO Tidbits

Perspectives on MMO Design and Production

  • Categories

  • Archives

  • Meta

  • Copyright Notice

    Contents Copyright 2009-2010 Arnold Hendrick

Why MMOs Are Complicated

Posted by Arnold Hendrick on March 9, 2011

MMOs are notorious for exceeding schedules and budgets, then stagger through launch with missing features and fatal flaws. Many say making MMOs is complicated, but why? It’s been 12-14 years since Ultima Online (1997) and EverQuest (1999) launched. In my current MMO project, our operations chief has launched six different MMOs in his career. Most of the studio has worked on MMOs before. Yet we have difficulties, last-minute delays and surprises.

Why are MMOs still difficult to create?

 

The Gamer’s View

A gamer’s view of an MMO is simple and straightforward. The player installs the game (client) on his or her PC, then connects via the internet to a game server, as shown below.

Gamers understand that extra technology is needed to “split” the game between client and server. Nevertheless, why is it so hard to build and release MMOs?

 

The Developer’s View

MMO developers are generally aware of additional complexities.

Game server(s) must do a great  many things. Player-characters and their data must be tracked by the server. NPC enemies (mobs) are created and managed by the server, including their “arAI. This AI manages NPC movement, shooting and melee across the 3D terrain. Developers also know that databases are usually the weak link. The constant “writing” of new character status and possessions is exactly what is slowest in traditional relational databases. Better solutions require either costly hardware and software, or require exceptionally rare programming knowledge. Servers themselves are usually a group of specialized server boxes clustered to create one “conceptual” server or shard. The many different technical approaches to shard organization could inspire a series of different articles.

Game developers understand that the game needs an online presence to operate. This includes a website, login systems, authentication systems, downloaders and updaters. The game also needs good community management and customer service. Unhappy customers will soon migrate to a competitor. Finally, an MMO needs an operations team to install and maintain server hardware and software.

 

 

The Daunting Reality

A fully functional MMO has many more “moving parts” around the game than most developers imagine. This infrastructure for a MMO is its “platform.” A more realistic portrayal of this ugly underbelly in MMO development looks like this:

Notice that game and accounting web functions are separate. The “game” website needs constant updating of marketing and promotional information, while user account creation and management must be rock-solid to prevent hacking and fraud. In fact, the account system supports creating, renewing and cancelling both accounts and subscriptions. Most modern MMOs also include a microtransaction (MTX) system for purchasing add-ons, from character transfers to in-game virtual currency. Finally, the account needs to link or “hand off” to the game during login without allowing hacks or cheats.

Customer service systems aren’t any easier. Customer Service Representatives (CSRs) need to examine everything from player financial transactions to in-game databases and chat. They not only need tools to email customers, but also tools to “action” players inside the game. These “actions” range from compensation (“Here’s 50 platinum because our update destroyed everything in your in-game bank) to chat messages (“Cool it or you’ll be banned) to player removals (“You’re banned for the next 7 days). The CSR system needs internal checks and balances. More than one MMO has been hurt badly by CSRs doing everything from favoring certain players to accepting bribes from gold farmers.

Another platform function is game metrics. Game executives want to know many people are playing, when and how much money they’re spending. It is not humanly possible to play sufficiently on each server. Metrics are provide vital information about the game’s operation. Are the servers are crowded or empty? Are subscriptions are rising or falling? What microtransaction items don’t sell? In additional, the operations team needs a host of technical readouts related to server hardware and software health. If the server software crashes its hardware suffers a failure, the operations staff needs to respond immediately.

In recent years, game metrics have expanded to measure gameplay. Which quests do people abandon most? Which dungeons go unplayed? At what level are people most likely to quit the game? Does this vary by age? By gender? By in-game friends? By guild membership? Recording this level of information is potentially dangerous. The very act of doing it within the server cluster can slow down processing, fill up memory, and overwhelm databases. Metrics systems must be carefully crafted.

The bottom of the diagram has a subtle pink box, showing where the game development studio connects into the game’s operating structure. To insure smooth server operation, development studio engineers never program directly on the production servers. Instead, they write and test code on an in-house server cluster, then “promote” a version to various test clusters. Promotion to the “live” servers is the last step in a chain of activities lasting days to weeks. Managing this process, and deciding when exceptions are allowed, is yet another complicated process.

 

Underestimation & Solutions

Game developers frequently underestimate the effort and complexity of MMO platforms. In the early, dreamy concept days development teams say, “Hey, it’s just a website. A million games and companies have them. It’ll be quick and easy.” In my most recent project, the platform team of four to six engineers with two managers needed over a half year to build a platform. All were highly competent veterans. They used a variety of external systems vendors for major subcomponents, such as customer service, financial transactions, and game metrics analysis. Platforms are never easy.

The ideal solution would be a platform services company that provides a platform game developers could customize and “skin.” Unfortunately, I don’t know of any firms offering a well-tested solution currently used by larger-scale MMO operators. If someone does, please let me know!

Lacking the ideal, a first alternative is working with an established online game operator. This means the game will be presented within the operator’s platform. For example, Pirates of the Burning Sea (PotBS) arranged for Sony Online Entertainment (SOE) to be their operator. Today PotBS is so strongly associated with Sony that relatively few people remember the small, scrappy, independent studio that created it. Players must go through multiple Sony-branded website and software to reach the game.

In addition to losing “customer mindshare,” the game developer often needs to rewrite and rework server functionality to meet platform requirements. Successful operators do not rewrite their platform for each new game! The earlier in the project a developer can make a deal with an operator, the less rework is needed to integrate the game to the platform.

The second alternative is a developer making their own platform. Some difficulty can be avoided by using established companies to provide platform components. These companies do exist, such as Aria for subscription services, PlaySpan for microtransactions, Parature for customer service software, etc. However, these components are a far cry from a full platform A platform team is still needed to build the website and create the “back end” glue between systems. Platform development also means serious database work.

Ultimately, the only sure-fire way to minimize platform development effort is an agile approach. Only build the minimum necessary platform for successful operation. The surest way to incur extra cost and schedule delay is adding features because “it would be really nice if…” or because “our next game might need…” Accept that the platform, like the game, constantly evolves and improves throughout its lifecycle.

Advertisements

5 Responses to “Why MMOs Are Complicated”

  1. Mike said

    Great article…thanks :)

  2. […] You don’t have a seperate website and game team? I guess your designers never saw: https://mmotidbits.com/2011/03/09/why-mmos-are-complicated/‘s diagrams huh? Seriously, you should split it ASAP, if only so you can get a faster […]

  3. Sergio said

    Hello Mr. Hendrick,

    I have to say I am a huge fan of Darklands. A modern one at that(first played it about 6 years ago). There is no game like it and that is a shame. Thanks for developing it, I had a great time playing it and I will certainly play it again.

    Recently a lot of old game series have been making a comeback through http://www.kickstarter.com , games like “Wasteland” and “Shadowrun”(both contemporary to Darklands) got great success at achieving independent fan funding and will now have sequels developed.

    I was wondering if you would consider using Kickstarter to develop a new game in the vein of Darklands. Maybe a direct sequel like it was planned. Your game is a cult classic with many fans, I am sure a kickstarter would be a resounding success. Just in two forums I frequent(www.rpgwatch.com and http://www.rpgcodex.net) there must be at least a couple thousand people willing to pay for a game like Darklands. Please consider this if you haven’t done so, I know it is wishful thinking on my part to suggest it, but so many great rpg and adventure series and developers have been making a comeback through kickstarter… I simply couldn’t help but wonder. As soon as the Wasteland 2 kickstarter was announced I couldn’t help thinking about Darklands and how cool would it be if it also got a well deserved revival.

    Thanks again for developing the game, as a fan, I’m really hoping you read this e-mail and consider it. Really, fingers crossed!

    Cheers,

    Sergio Araujo

  4. Hello Arnold!

    Sorry my English at first.

    Making mmo’s is complicated and your samples are great. But what about level of complication in games? Means ui, menu and general game managing system (by user)?

    I’m playing Dayz, this is different stuff, but still mmo I think and this model is very interesting, not only for fps, here a lot of great ideas – traveling, growing your character, etc. And most important – old players are not nerfed. With some experience, you can beat with pistol only top equiped “old player” or if you are skilled, you can beat even team of “old players”.

    I wonder about the creation of game with this model but with much more simple environment, maybe just 2d. I have some skills for it, not all. However, is possible to do game like this alone, but better with some team. I know problems, it’s why for now I wonder only :)

    Btw, now I know, I am a big fan of your work. I remember a lot of titles where you are credited, great times!

    • Arnold Hendrick said

      Pawel, you are correct that game development itself is just as complex as MMO architectures. A great many people have written about developing games, but relatively few have tried to explain the software architecture behind multiplayer online games (MOBAs present most to all of the same problems as MMOs).

      PvP games definitely have different mechanics than PvE games. You are quite correct that in PvP, player skill needs to trump character attributes/equipment. If it doesn’t, unless the game is segmented into different regions for different character levels, the well-equipped players will drive away almost all newcomers. As live producer, I’ve watched this happen, and the game’s player population slowly fall as a result.

      Thanks also for the kind words about my earlier games.

Sorry, the comment form is closed at this time.