CloudPundit: Massive-Scale Computing

When I first started this blog, I intended to write more about virtual worlds, following the general theme of massive scalability. In this instance, though, I want to muse upon the balance between maximizing your revenues, and adhering to principle, especially when you’re a public company with shareholders to worry about. Also, this involves the unintended consequences of user-generated content, and there are lessons to be learned here if you’re looking at UGC, whether in your own enterprise or for consumers in general. Similarly, there are perils in any customer-controlled environment. Bear with me, though, because this is long.

Massively multiplayer online games (MMOGs), and MMO roleplaying games (MMORPGs) in particular, all have distinct communities, but each such community is always full of players with conflicting interests. The development studio has to balance their own vision, as well as the sometimes-warring interests of different types of players, and the commercial needs of the game (whether it’s paid for in subscriptions, real-money trade, or other, there has to be revenue), in order to maximize long-term profit. Communities are particularly fragile, and widespread changes can lead to mass exodus, as Sony Online Entertainment discovered with Star Wars: Galaxies, where a thorough and expensive revamp instead caused more than a 50% drop in subscriptions. Players who depart are not individuals — they are part of a community of family, friends, and online acquaintances, and when key players leave, there’s a domino effect.

Enter NCsoft (SEO:036570), and one of its veteran properties, five-year-old City of Heroes. CoH is relatively small fry for NCsoft — it peaked at around 200,000 subscribers, and now has something in the 150,000 range, paying a base of $15/month in subscription fees. NCsoft’s Lineage and Lineage II, by contrast, each have about a million subscribers; for anyone that isn’t Blizzard and the juggernaut that is World of Warcraft, these are impressive numbers, but they’re down hugely from their all-time highs.

CoH currently enjoys a position as the only superhero-themed MMOG out there. However, Champions Online comes out this summer, designed by the same folks who originally created CoH, creating an imminent competitive threat. Paragon Studios (the studio within NCsoft that’s responsible for CoH) chose to do something smart — introduce user-generated content, allowing players to create their own missions (scenarios), complete with fully custom enemies to fight. (As an on-and-off CoH player with what I hope is a creative streak, UGC is deeply welcome feature, and lots of people are using it to do very entertaining things.)

As one would expect, players immediately went diligently to work to find ways to hyperoptimize UGC in order to maximize rewards for a given amount of play time. The game’s EULA specifies you’re not allowed to use exploits, but the difficulty created was this: What is an exploit, versus merely unintended levels of reward? There are methods in the game that generate very high rewards per unit time, for instance; UGC simply allowed players to generate optimal situations for themselves. The game’s programmers rapidly closed down some methods, but left other methods live for almost a full month. The hyper-efficient methods were well-known and broadly used by the player base, but the studio was essentially silent, with no communication to customers, other than a request for feedback.

Usually, in a virtual world, when there’s an exploit, the exploiters are limited to a handful of people; players normally know a bug when they see one, like the ability to duplicate a valuable object. This particular case is unusual because it affects a sizable percentage of the player base, and it’s unclear what is and is not an exploit.

Consequently, players have been shocked to see NCsoft announce that they’ve decided to react harshly, stating that players who have “abused” the reward system may lose the rewards they’ve gained, including losing access to the characters used. Since CoH is an MMORPG, characters may represent hundreds, even thousands, of hours of investment, so this is a serious threat. The real-world cash value of optimized characters is significant, too, although such sales and transfers are against the EULA.

It’s an extraordinary choice on NCsoft’s part. Other than the instructions not to “exploit” the system, as well as explicit rules forbidding players from creating exploitative UGC, there was never any warning to customers not to play UGC that might be exploitative, although CoH‘s parent studio publicly communicates with customers on a daily basis through the game’s forums. NCsoft has recently been pushing sales of a new boxed set for new players, as well, leading to the high likelihood of inadvertent “abuse” by new players who would not necessarily know that these were exceptional levels of reward for the time.

Losing access to rewards and characters essentially represents nullifying the time investment of players, and the removal of avenues from which to have fun (the character represents the ability to access content). Thus, impacted customers, most of whom subscribe month-to-month, have a very high likelihood of cancelling. This represents a potential direct revenue hit at a time when the game is likely extremely vulnerable to competition, and the aforementioned domino effect of subscriber loss is real and must be considered. Yet, to not do anything is a compromise of principle, and potentially creates a whack-a-mole effect whereby players find new gray areas of high-reward generation and widely use them to gain rewards, while developers try to patch these as quickly as possible. Moreover, because virtual worlds have internal economies, exceptionally fast rewards create imbalances, so they have an impact beyond individual players. (This does not include the impact to “gold farmers” and “power-leveling services”, who offer in-game rewards and powerful characters in exchange for real money, a practice which is against nearly every MMOG’s terms of service, but is nonetheless a significant and growing business. Ironically, making it easier for players to gain quick rewards on their own devalues such services.)

NCsoft is facing the prospect of significant subscriber bleed due to the forthcoming Champions Online, so a decision that increases the likelihood of cancellations is an extraordinarily bold move. It’s unusual for public companies to be willing to choose principle over revenue. Implementing harsh penalties based on clear guidelines, possibly with an automated warning system (i.e., if a player has gotten more than X widgets per Y time, alert him to it), may be advisable, but retroactive imposition of penalties on one’s customer base is another matter. Creating “traps” for bad apples disguised as paying customers is certainly reasonable. Punishing ordinary customers for having done something gray, and which your company has failed to even suggest is black, may be a quick ticket to having to offer unpleasantly complex explanations to your shareholders. Industry-watchers may find the outcome of this to be instructive.

So here are the broader lessons:

A couple of months ago, I wrote about scaling and friendly failure. The same principle that applies here: It’s not what the limits are. It’s how well you communicate them to your customers in advance of enforcing them. It applies whether you’re a gaming company, a cloud computing company, a network services provider, or an entirely non-tech company.

If you are providing an environment with user-generated content, expect that it will be abused, sometimes in subtle ways. Even in a corporate environment, there are potentials for abuse, particularly if the company gives employees goals or bonuses to work towards for completing UGC. Human nature being what it is, people optimize; in the work world, they’re careful not to optimize so much that they think they could get fired over it, but again, the boundaries are gray and hazy. Clear communication of what is and isn’t acceptable, in advance, is necessary.