Most of us have long since overcome our fear of handing over our credit cards to Internet merchants. It’s become routine for most of us to simply do so. We buy stuff, we sign up for subscriptions, it’s just like handing over plastic anytime else. For that matter, most of us have never really thought about all that credit card data laying around in the hands of brick-and-mortar merchants with whom we do business, until the unfortunate times when that data gets mass-compromised.
Bad billing problems plague lots of organizations, but Electronic Arts (in the form of its Mythic Entertainment studio, which does the massively multiplayer online RPGs Dark Ages of Camelot and Warhammer Online) just had a major screw-up: a severe billing system error that, several days ago, repeatedly charged customers their subscription fees. Not just one extra charge, but, some users say, more than sixty. Worse still, the error reportedly affected not just current customers, but past customers. A month’s subscription is $15, but users can pre-pay for as much as a year. And these days, with credit cards so often actually being checking-account debit cards, that is often an immediate hit to the wallet. So you can imagine the impact on even users with decent bank balances, being hit by multiple charges. (Plenty of people with good-sized savings cushions only keep enough money in the checking account to cover expected bills, so you don’t have to be on the actual fiscal edge to get smacked with overdraft fees.) EA is scrambling to get this straightened out, of course, but this is every company’s worst billing nightmare, and it comes at a time when EA and its competitors are all scrambling to shift their business models online.
How many merchants that you don’t do business with any longer, but used to have recurring billing permission on your credit card, still have your credit card on file? As online commerce and micropayments proliferate, how many more merchants will store that data? (Or will PayPal, Apple’s storefronts, and other payment services rule the world?)
When I first started this blog, I intended to write more about virtual worlds, following the general theme of massive scalability. In this instance, though, I want to muse upon the balance between maximizing your revenues, and adhering to principle, especially when you’re a public company with shareholders to worry about. Also, this involves the unintended consequences of user-generated content, and there are lessons to be learned here if you’re looking at UGC, whether in your own enterprise or for consumers in general. Similarly, there are perils in any customer-controlled environment. Bear with me, though, because this is long.
Massively multiplayer online games (MMOGs), and MMO roleplaying games (MMORPGs) in particular, all have distinct communities, but each such community is always full of players with conflicting interests. The development studio has to balance their own vision, as well as the sometimes-warring interests of different types of players, and the commercial needs of the game (whether it’s paid for in subscriptions, real-money trade, or other, there has to be revenue), in order to maximize long-term profit. Communities are particularly fragile, and widespread changes can lead to mass exodus, as Sony Online Entertainment discovered with Star Wars: Galaxies, where a thorough and expensive revamp instead caused more than a 50% drop in subscriptions. Players who depart are not individuals — they are part of a community of family, friends, and online acquaintances, and when key players leave, there’s a domino effect.
Enter NCsoft (SEO:036570), and one of its veteran properties, five-year-old City of Heroes. CoH is relatively small fry for NCsoft — it peaked at around 200,000 subscribers, and now has something in the 150,000 range, paying a base of $15/month in subscription fees. NCsoft’s Lineage and Lineage II, by contrast, each have about a million subscribers; for anyone that isn’t Blizzard and the juggernaut that is World of Warcraft, these are impressive numbers, but they’re down hugely from their all-time highs.
CoH currently enjoys a position as the only superhero-themed MMOG out there. However, Champions Online comes out this summer, designed by the same folks who originally created CoH, creating an imminent competitive threat. Paragon Studios (the studio within NCsoft that’s responsible for CoH) chose to do something smart — introduce user-generated content, allowing players to create their own missions (scenarios), complete with fully custom enemies to fight. (As an on-and-off CoH player with what I hope is a creative streak, UGC is deeply welcome feature, and lots of people are using it to do very entertaining things.)
As one would expect, players immediately went diligently to work to find ways to hyperoptimize UGC in order to maximize rewards for a given amount of play time. The game’s EULA specifies you’re not allowed to use exploits, but the difficulty created was this: What is an exploit, versus merely unintended levels of reward? There are methods in the game that generate very high rewards per unit time, for instance; UGC simply allowed players to generate optimal situations for themselves. The game’s programmers rapidly closed down some methods, but left other methods live for almost a full month. The hyper-efficient methods were well-known and broadly used by the player base, but the studio was essentially silent, with no communication to customers, other than a request for feedback.
Usually, in a virtual world, when there’s an exploit, the exploiters are limited to a handful of people; players normally know a bug when they see one, like the ability to duplicate a valuable object. This particular case is unusual because it affects a sizable percentage of the player base, and it’s unclear what is and is not an exploit.
Consequently, players have been shocked to see NCsoft announce that they’ve decided to react harshly, stating that players who have “abused” the reward system may lose the rewards they’ve gained, including losing access to the characters used. Since CoH is an MMORPG, characters may represent hundreds, even thousands, of hours of investment, so this is a serious threat. The real-world cash value of optimized characters is significant, too, although such sales and transfers are against the EULA.
It’s an extraordinary choice on NCsoft’s part. Other than the instructions not to “exploit” the system, as well as explicit rules forbidding players from creating exploitative UGC, there was never any warning to customers not to play UGC that might be exploitative, although CoH‘s parent studio publicly communicates with customers on a daily basis through the game’s forums. NCsoft has recently been pushing sales of a new boxed set for new players, as well, leading to the high likelihood of inadvertent “abuse” by new players who would not necessarily know that these were exceptional levels of reward for the time.
Losing access to rewards and characters essentially represents nullifying the time investment of players, and the removal of avenues from which to have fun (the character represents the ability to access content). Thus, impacted customers, most of whom subscribe month-to-month, have a very high likelihood of cancelling. This represents a potential direct revenue hit at a time when the game is likely extremely vulnerable to competition, and the aforementioned domino effect of subscriber loss is real and must be considered. Yet, to not do anything is a compromise of principle, and potentially creates a whack-a-mole effect whereby players find new gray areas of high-reward generation and widely use them to gain rewards, while developers try to patch these as quickly as possible. Moreover, because virtual worlds have internal economies, exceptionally fast rewards create imbalances, so they have an impact beyond individual players. (This does not include the impact to “gold farmers” and “power-leveling services”, who offer in-game rewards and powerful characters in exchange for real money, a practice which is against nearly every MMOG’s terms of service, but is nonetheless a significant and growing business. Ironically, making it easier for players to gain quick rewards on their own devalues such services.)
NCsoft is facing the prospect of significant subscriber bleed due to the forthcoming Champions Online, so a decision that increases the likelihood of cancellations is an extraordinarily bold move. It’s unusual for public companies to be willing to choose principle over revenue. Implementing harsh penalties based on clear guidelines, possibly with an automated warning system (i.e., if a player has gotten more than X widgets per Y time, alert him to it), may be advisable, but retroactive imposition of penalties on one’s customer base is another matter. Creating “traps” for bad apples disguised as paying customers is certainly reasonable. Punishing ordinary customers for having done something gray, and which your company has failed to even suggest is black, may be a quick ticket to having to offer unpleasantly complex explanations to your shareholders. Industry-watchers may find the outcome of this to be instructive.
So here are the broader lessons:
A couple of months ago, I wrote about scaling and friendly failure. The same principle that applies here: It’s not what the limits are. It’s how well you communicate them to your customers in advance of enforcing them. It applies whether you’re a gaming company, a cloud computing company, a network services provider, or an entirely non-tech company.
If you are providing an environment with user-generated content, expect that it will be abused, sometimes in subtle ways. Even in a corporate environment, there are potentials for abuse, particularly if the company gives employees goals or bonuses to work towards for completing UGC. Human nature being what it is, people optimize; in the work world, they’re careful not to optimize so much that they think they could get fired over it, but again, the boundaries are gray and hazy. Clear communication of what is and isn’t acceptable, in advance, is necessary.
Nostalgia for the ’80s continues to reign. (Robot Chicken fans: Have you seen the Pac-Matrix?)
A company called Jolt Online Gaming has acquired the rights to produce a browser-based MMORPG called Legends of Zork. For those of you who have never had the experience of realizing it is dark and you may be eaten by a grue, this is probably not particularly meaningful to you, but for fans of the era of classic Infocom text-adventure games, it is both fascinating and bizarre to see that they’re going to try to turn the Great Underground Empire into a hack-and-slash online multiplayer RPG.
The market for browser-based massively-multiplayer games supports a cottage industry of small companies with a handful of developers, backed by an artist or two, who crank out a reasonably nice living for themselves without ever competing with the big-time. I wonder if we’ll eventually see a roll-up of these guys, or if they like being “lifestyle companies”.
MMOG developer and publisher Trion World Network just closed a $70 million Series C round, which brings its total raised since its inception in 2006 to over $100 million.
This might seem like a staggering amount of money for a company with two games in development but none published yet. It’s trading on the name of its founder, Jon Van Caneghem, of Might and Magic fame. But it’s not that much money if you realize that games are now being made on movie-sized budgets, and MMOGs are exceptionally expensive to develop.
Dan Hunter had an interesting piece on the Terra Nova blog last year regarding the financials of MMOG development, based off an Interplay prospectus for an MMOG based on Fallout. That cited a cost of $75m, including a launch budget of $30m, which presumably includes marketing, manufacturing, and server deployment.
MMOGs are not efficient beasts, and by their nature, they are also prone to flash crowds and highly variable capacity needs. Most scale in a highly unwieldy manner, compounding the basic inefficient utilization of computing capacity. Utility computing infrastructure has huge potential to reduce the overbuy of capacity, but colocation on their own hardware is nigh-universally the way that such companies deploy their games.
Nicholas Carr estimated back in 2006 that an avatar in Second Life has a carbon footprint equivalent to a Brazilian. Last year, I heard, from a source I’d consider to be pretty authoritative, that an avatar in Second Life actually has a carbon footprint larger than its typical real-person (usually an affluent American).
This is why Internet data center providers drool at MMOG companies.
Mythic has stated publicly that all of the US game servers are located in Virginia, near Mythic’s offices. A couple of traceroutes seem to indicate that they’re in Verizon, almost certainly in colocation (managed hosting is rare for MMOGs), and seem to have purely Verizon connectivity to the Internet. The webservers, on the other hand, look to be split between Verizon, and ThePlanet in Dallas. FileBurst (a single-location download hosting service) is used to serve images and cinematics.
During the beta, Mythic used BitTorrent to serve files. With the advent of full release, it doesn’t appear that they’re depending on peer-to-peer any longer — unlike Blizzard, for instance, which uses public P2P in the form of BitTorrent for its World of Warcraft updates, trading off cost with much higher levels of user frustration. MMO updates are probably an ideal case for P2P file distribution — Solid State Networks, a P2P CDN, has done well by that — and with hybrid CDNs (those combining a traditional distributed model with P2P) becoming more commonplace, I’d expect to see that model more often.
However, I’m not keen on either single data center locations or single-homing, for anything that wants to be reliable. I also believe that gaming — a performance-sensitive application — really ought to run in a multi-homed environment. My favorite “why you should use multiple ISPs, even if you’re using a premium ISP that you love” anecdote to my clients is an observation I made while playing World of Warcraft a few years ago. WoW originally used just AT&T’s network (in AT&T colocation). Latency was excellent — most of the time. Occasionally, you’d get a couple of seconds of network burp, where latency would spike hugely. If you’re websurfing, this doesn’t really impact your experience. If you’re playing an online game, you can end up dead. When WoW switched to Internap for the network piece (remaining in AT&T colo), overall latencies went up — but the latencies were still well below the threshold of problematic performance, and more importantly, the latencies were rock-solidly in a narrow window of variability. (This is the same reason multi-homed CDNs with lots of route diversity deliver better consistency of user experience than single-carrier CDNs.)
Companies like Fileburst, by the way, are going to be squarely in the crosshairs of the forthcoming Amazon CDN. Fileburst will do 5 TB of delivery at $0.80 per GB — $3,985/month. At the low end, they’ll do 100 GB or less at $1/GB. The first 100 MB of storage is free, then it’s $2/MB. They’ve got a delivery infrastructure at the Equinix IBX in Ashburn (Northern Virginia, near DC), extensive peering, but any other footprint is vague (they say they have a six-location CDN service, but it’s not clear whether it’s theirs or if they’re reselling).
If Amazon’s CDN pricing is anything like the S3 pricing, they’ll blow the doors off those prices. S3 is $0.15/GB for space and $0.17/GB for the first 10 TB of data transfer. So deliver 5 TB worth of content, out of a 1 GB store, would cost me $5,785/month with Fileburst, and about $850 with Amazon S3. Even if the CDN premium on data transfer is, say, 100%, that’d still be only $1,700 with Amazon.
Amazon has a key cloud trait — elasticity, basically defined as the ability to scale to zero (or near-zero) as easily as scaling to bogglosity. It’s that bottom end that’s really going to give them the potential to wipe out the zillion little CDNs that primarily have low-volume customers.