Why MMOs Are Complicated
Posted by Arnold Hendrick on March 9, 2011
MMOs are notorious for exceeding schedules and budgets, then stagger through launch with missing features and fatal flaws. Many say making MMOs is complicated, but why? It’s been 12-14 years since Ultima Online (1997) and EverQuest (1999) launched. In my current MMO project, our operations chief has launched six different MMOs in his career. Most of the studio has worked on MMOs before. Yet we have difficulties, last-minute delays and surprises.
Why are MMOs still difficult to create?
The Gamer’s View
A gamer’s view of an MMO is simple and straightforward. The player installs the game (client) on his or her PC, then connects via the internet to a game server, as shown below.
Gamers understand that extra technology is needed to “split” the game between client and server. Nevertheless, why is it so hard to build and release MMOs?
The Developer’s View
MMO developers are generally aware of additional complexities.
Game server(s) must do a great many things. Player-characters and their data must be tracked by the server. NPC enemies (mobs) are created and managed by the server, including their “arAI. This AI manages NPC movement, shooting and melee across the 3D terrain. Developers also know that databases are usually the weak link. The constant “writing” of new character status and possessions is exactly what is slowest in traditional relational databases. Better solutions require either costly hardware and software, or require exceptionally rare programming knowledge. Servers themselves are usually a group of specialized server boxes clustered to create one “conceptual” server or shard. The many different technical approaches to shard organization could inspire a series of different articles.
Game developers understand that the game needs an online presence to operate. This includes a website, login systems, authentication systems, downloaders and updaters. The game also needs good community management and customer service. Unhappy customers will soon migrate to a competitor. Finally, an MMO needs an operations team to install and maintain server hardware and software.
The Daunting Reality
A fully functional MMO has many more “moving parts” around the game than most developers imagine. This infrastructure for a MMO is its “platform.” A more realistic portrayal of this ugly underbelly in MMO development looks like this:
Notice that game and accounting web functions are separate. The “game” website needs constant updating of marketing and promotional information, while user account creation and management must be rock-solid to prevent hacking and fraud. In fact, the account system supports creating, renewing and cancelling both accounts and subscriptions. Most modern MMOs also include a microtransaction (MTX) system for purchasing add-ons, from character transfers to in-game virtual currency. Finally, the account needs to link or “hand off” to the game during login without allowing hacks or cheats.
Customer service systems aren’t any easier. Customer Service Representatives (CSRs) need to examine everything from player financial transactions to in-game databases and chat. They not only need tools to email customers, but also tools to “action” players inside the game. These “actions” range from compensation (“Here’s 50 platinum because our update destroyed everything in your in-game bank) to chat messages (“Cool it or you’ll be banned) to player removals (“You’re banned for the next 7 days). The CSR system needs internal checks and balances. More than one MMO has been hurt badly by CSRs doing everything from favoring certain players to accepting bribes from gold farmers.
Another platform function is game metrics. Game executives want to know many people are playing, when and how much money they’re spending. It is not humanly possible to play sufficiently on each server. Metrics are provide vital information about the game’s operation. Are the servers are crowded or empty? Are subscriptions are rising or falling? What microtransaction items don’t sell? In additional, the operations team needs a host of technical readouts related to server hardware and software health. If the server software crashes its hardware suffers a failure, the operations staff needs to respond immediately.
In recent years, game metrics have expanded to measure gameplay. Which quests do people abandon most? Which dungeons go unplayed? At what level are people most likely to quit the game? Does this vary by age? By gender? By in-game friends? By guild membership? Recording this level of information is potentially dangerous. The very act of doing it within the server cluster can slow down processing, fill up memory, and overwhelm databases. Metrics systems must be carefully crafted.
The bottom of the diagram has a subtle pink box, showing where the game development studio connects into the game’s operating structure. To insure smooth server operation, development studio engineers never program directly on the production servers. Instead, they write and test code on an in-house server cluster, then “promote” a version to various test clusters. Promotion to the “live” servers is the last step in a chain of activities lasting days to weeks. Managing this process, and deciding when exceptions are allowed, is yet another complicated process.
Underestimation & Solutions
Game developers frequently underestimate the effort and complexity of MMO platforms. In the early, dreamy concept days development teams say, “Hey, it’s just a website. A million games and companies have them. It’ll be quick and easy.” In my most recent project, the platform team of four to six engineers with two managers needed over a half year to build a platform. All were highly competent veterans. They used a variety of external systems vendors for major subcomponents, such as customer service, financial transactions, and game metrics analysis. Platforms are never easy.
The ideal solution would be a platform services company that provides a platform game developers could customize and “skin.” Unfortunately, I don’t know of any firms offering a well-tested solution currently used by larger-scale MMO operators. If someone does, please let me know!
Lacking the ideal, a first alternative is working with an established online game operator. This means the game will be presented within the operator’s platform. For example, Pirates of the Burning Sea (PotBS) arranged for Sony Online Entertainment (SOE) to be their operator. Today PotBS is so strongly associated with Sony that relatively few people remember the small, scrappy, independent studio that created it. Players must go through multiple Sony-branded website and software to reach the game.
In addition to losing “customer mindshare,” the game developer often needs to rewrite and rework server functionality to meet platform requirements. Successful operators do not rewrite their platform for each new game! The earlier in the project a developer can make a deal with an operator, the less rework is needed to integrate the game to the platform.
The second alternative is a developer making their own platform. Some difficulty can be avoided by using established companies to provide platform components. These companies do exist, such as Aria for subscription services, PlaySpan for microtransactions, Parature for customer service software, etc. However, these components are a far cry from a full platform A platform team is still needed to build the website and create the “back end” glue between systems. Platform development also means serious database work.
Ultimately, the only sure-fire way to minimize platform development effort is an agile approach. Only build the minimum necessary platform for successful operation. The surest way to incur extra cost and schedule delay is adding features because “it would be really nice if…” or because “our next game might need…” Accept that the platform, like the game, constantly evolves and improves throughout its lifecycle.
5 Responses to “Why MMOs Are Complicated”
Sorry, the comment form is closed at this time.