Blog Archives

Are B2B cloud service agreements safe?

I’m seeing various bits of angst around “Is it safe to use cloud services, if my business can be suspended or terminated at any time?” and I thought I’d take some time to explain how cloud providers (and other Internet ecosystem providers, collectively “service providers” [SPs] in this blog post) enforce their Terms of Service (ToS) and Acceptable Use Policies (AUPs).

The TL;DR: Service providers like money, and will strive to avoid terminating customers over policy violations. However, providers do routinely (and sometimes automatically) enforce these policies, although they vary in how much grace and assistance they offer with issues. You don’t have to be a “bad guy” to occasionally run afoul of the policies. But if your business is permanently unwilling or unable to comply with a particular provider’s policies, you cannot use that provider.

AUP enforcement actions are rarely publicized. The information in this post is drawn from personal experience running an ISP abuse department; 20 years of reviewing multiple ISP, hosting, CDN, and cloud IaaS contracts on a daily basis; many years of dialogue with Gartner clients about their experiences with these policies; and conversations with service providers about these topics. Note that I am not a lawyer, and this post is not legal advice. I would encourage you to work with your corporate counsel to understand service provider contract language and its implications for your business, whenever you contract for any form of digital service, whether cloud or noncloud.

The information in this post is intended to be factual; it is not advice and there is a minimum of opinion. Gartner clients interested in understanding how to negotiate terms of service with cloud providers are encouraged to consult our advice for negotiating with Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP), or with SaaS providers. My colleagues will cheerfully review your contracts and provide tailored advice in the context of client inquiry.

Click-thrus, negotiated contracts, ToS, and AUPs

Business-to-business (B2B) service provider agreements have taken two different forms for more than 20 years. There are “click-through agreements” (CTAs) that present you with a online contract that you click to sign; consequently, they are as-is, “take it or leave it” legal documents that tend to favor the provider in terms of business risk mitigation. Then there are negotiated contracts — “enterprise agreements” (EAs) that typically begin with more generous terms and conditions (T&Cs) that better balance the interests of the customer and the provider. EAs are typically negotiated between the provider’s lawyers and the customer’s procurement (sourcing & vendor management) team, as well as the customer’s lawyers (“counsel”).

Some service providers operate exclusively on either CTAs or EAs. But most cloud providers offer both. Not all customers may be eligible to sign an EA; that’s a business decision. Providers may set a minimum account size, minimum spend, minimum creditworthiness, etc., and these thresholds may be different in different countries. Providers are under no obligation to either publicize the circumstances under which an EA is offered, or to offer an EA to a particular customer (or prospective customer).

While in general, EAs would logically be negotiated by all customers who can qualify, providers do not necessarily proactively offer EAs. Furthermore, some companies — especially startups without cloud-knowledgeable sourcing managers — aren’t aware of the existence of EAs and therefore don’t pursue them. And many businesses that are new to cloud services don’t initially negotiate an EA, or take months to do that negotiation, operating on a CTA in the meantime. Therefore, there are certainly businesses that spend a lot of money with a provider, yet only have a CTA.

Terms of service are typically baked directly into both CTAs and EAs — they are one element of the T&Cs. As a result, a business on an EA benefits both from the greater “default” generosity of the EA’s T&Cs over the provider’s CTA (if the provider offers both), as well as whatever clauses they’ve been able to negotiate in their favor. In general, the bigger the deal, the more leverage the customer has to negotiate favorable T&Cs, which may include greater “cure time” for contract breaches, greater time to pay the bill, more notice of service changes, etc.

AUPs, on the other hand, are separate documents incorporated by reference into the T&Cs. They are usually a superset or expansion/clarification of the things mentioned directly in the ToS. For instance, the ToS may say “no illegal activity allowed”, and the AUP will give examples of prohibited activities (important since what is legal varies by country). AUPs routinely restrict valid use, even if such use is entirely legal. Service providers usually stipulate that an AUP can change with no notice (which essentially allows a provider to respond rapidly to a change in the regulatory or threat environment).

Unlike the EA T&Cs, an AUP is non-negotiable. However, an EA can be negotiated to clarify an AUP interpretation, especially if the customer is in a “grey area” that might be covered by the AUP even if the use is totally legitimate (i.e. a security vendor that performs penetration testing on other businesses at their request, may nevertheless ask for an explicit EA statement that such testing doesn’t violate the AUP).

Prospective customers of a service provider can’t safely make assumptions about the AUP intent. For example, some providers might not exclude even a fully white-hat pen-testing security vendor from the relevant portion of the AUP. Some providers with a gambling-excluding AUP may not choose to do business with an organization that has, for instance, anything to do with gambling, even if that gambling is not online (which can get into grey areas like, “Is running a state lottery a form of gambling?”). Some providers operating data centers in countries without full freedom of the press may be obliged to enforce restrictions on what content a media company can host in those regions. Anyone who could conceivably violate the AUP as part of the routine course of business would therefore want to gain clarity on interpretation up front — and get it in writing in an EA.

What does AUP enforcement look like?

If you’re not familiar with AUPs or why they exist and must be enforced, I encourage you to read my post “Terms of Service: from anti-spam to content takedown” first.

AUP enforcement is generally handled by a “fraud and abuse” department within a service provider, although in recent years, some service providers have adopted friendlier names, like “trust and safety”. When an enforcement action is taken, the customer is typically given a clear statement of what the violation is, any actions taken (or that will be taken within X amount of time if the violation isn’t fixed), and how to contact the provider regarding this issue. There is normally no ambiguity, although less technically-savvy customers can sometimes have difficulties understanding why what they did wrong — and in the case of automatic enforcement actions, the customer may be entirely puzzled by what they did to trigger this.

There is almost always a split in the way that enforcement is handled for customers on a CTA, vs customers on an EA. Because customers on a CTA undergo zero or minimal verification, there is no presumption that those customers are legit good actors. Indeed, some providers may assume, until proven otherwise, that such customers exist specifically to perpetuate fraud and/or abuse. Customers on an EA have effectively gone through more vetting — the account team has probably done the homework to figure out likely revenue opportunity, business model and drivers for the sale, etc. — and they also have better T&Cs, so they get the benefit of the doubt.

Consequently, CTA customers are often subject to more stringent policies and much harsher, immediate enforcement of those policies. Immediate suspension or termination is certainly possible, with relatively minimal communication. (To take a public GCP example: GCP would terminate without means to protest as recently as 2018, though that has changed. Its suspension guidelines and CTA restrictions offer clear statements of swift and significantly automated enforcement, including prevention of cryptocurrency mining for CTA customers who aren’t on invoicing, even though it’s perfectly legal.) The watchword for the cloud providers is “business risk management” when it comes to CTA customers.

Customers that are on a CTA but are spending a lot of money — and seem to be legit businesses with an established history on the platform — generally get a little more latitude in enforcement. (And if enforcement is automated, there may be a sliding threshold for automated actions based on spend history.) Similarly, customers on a CTA but who are actively negotiating an EA or engaged in the enterprise sales process may get more latitude.

Often-contrary to the handling of CTA customers, providers usually assume an EA customer who has breached the AUP has done so unintentionally. (For instance, one of the customer’s salespeople may have sent spam, or a customer VM may have been compromised and is now being used as part of a botnet.) Consequently, the provider tends to believe that what the customer needs is notification that something is wrong, education on why it’s problematic, and help in addressing the issue. EA customers are often completely spared from any automated form of policy enforcement. While business risk management is still a factor for the service provider, this is often couched politely as helping the customer avoid risk exposure for the customer’s own business.

Providers do, however, generally firmly hold the line on “the customer needs to deal with the problem”. For instance, I’ve encountered cloud customers who have said to me, “Well, my security breach isn’t so bad, and I don’t have time/resources to address this compromised VM, so I’d like more than 30 days grace to do so, how do I make my provider agree?” when the service provider has reasonably taken the stance that the breach potentially endangers others, and mandated that the customer promptly (immediately) address the breach. In many cases, the provider will offer technical assistance if necessary. Service providers vary in their response to this sort of recalcitrance and the extent of their enforcement actions. For instance, AWS normally takes actions against the narrowest feasible scope — i.e. against only the infrastructure involved in the policy violation. Broadly, cloud providers don’t punish customers for violations, but customers must do something about such violations.

Some providers have some form of variant of a “three strikes” policy, or escalating enforcement. For instance, if a customer has repeated issues — for example, it seems unable implement effective anti-spam compliance for itself, or it constantly fails to maintain effective security in a way that could impact other customers or the cloud provider’s services, or it can’t effectively moderate content it offers to the public, or it can’t prevent its employees from distributing illegally copied music using corporate cloud resources — then repeated warnings and limited enforcement actions can turn into suspensions or termination. Thus, even EA customers are essentially obliged treat every policy violation as something that they need to strive to ensure will not recur in the future. Resolution of a given violation is not evidence that the customer is in effective compliance with the agreement.

Bottom Line

It’s not unusual for entirely legitimate, well-intentioned businesses to breach the ToS or AUP, but this is normally rare; a business might do this once or twice over the course of many years. New cloud customers on a CTA may also innocently exhibit behaviors that trigger automated enforcement actions that use algorithms to look for usage patterns that may be indicative of fraud or abuse. Service providers will take enforcement actions based on the customer history, the contractual agreement, and other business-risk and customer-relationship factors.

Intent matters. Accidental breaches are likely to be treated with a great deal more kindness. If breaches recur, though, the provider is likely to want to see evidence that the business has an effective plan for preventing further such issues. Even if the customer is willing to absorb the technical, legal, or business risks associated with a violation, the service provider is likely to insist that the issue be addressed — to protect their own services, their own customers, and for the customer’s own good.

(Update: Gartner clients, I have published a research note: “What is the risk of actually losing your cloud provider?“)

Beware of vendors bearing transformation Turkish Delight

“It is a lovely place, my house,” said the Queen. “I am sure you would like it. There are whole rooms full of Turkish Delight, and what’s more, I have no children of my own. I want a nice boy whom I could bring up as a Prince and who would be King of Narnia when I am gone. While he was Prince he would wear a gold crown and eat Turkish Delight all day long; and you are much the cleverest and handsomest young man I’ve ever met. I think I would like to make you the Prince—some day, when you bring the others to visit me.” — The White Witch (C.S. Lewis; The Lion, The Witch, and the Wardrobe)

When most people read the Narnia novels as children, they have no idea what Turkish Delight is. Its obscurity in recent decades has allowed everyone to imagine it as an entirely wonderful substance, carrying all their hopes and dreams of the perfect candy.

So, too, do people pour all of their business hopes and dreams into a nebulously-defined future of “digital transformation”.

Because the cloud is such a key enabling technology for digital business, I have plenty of discussions with clients who have been promised grand “digital transformation” outcomes by cloud providers and cloud MSPs. But it certainly not a phenomenon limited to the cloud. Hardware vendors and ISVs, outsourcers, consultancies, etc. are all selling this dream. While I can think of vendors who are more guilty of this than others, it’s a cross-IT-industry phenomenon.

Beware all digital transformation promises. Especially the ones where the vendor promises to partner with you to change the future of your industry or reinvent/revolutionize/disrupt X (where X is what your industry does).

I’ve quietly watched a string of broken transformation promises over the last few years, gently privately warning clients in inquiry conversations that you generally can’t trust these sorts of vendor promises. These behaviors have become much more prominent recently, though. And a colleague recently told me about a conversation that seemed like just a bridge too far: a large tech vendor promising to partner with a small Midwestern industrial manufacturer (tech laggards not doing anything exciting) to create transformative products, as part of a sales negotiation. (This vendor had not previously exhibited such behavior, so it was doubly startling.)

Clients come to us with tales of vendors who, in the course of sales discussions, promises to partner with them — possibly even dangling the possibility of a joint venture — to launch a transformational digital business, revolutionize the future of their industry, or the like. (Note that this is distinct from companies that sell transformation consulting. They promise to help you figure out the future, not form a business partnership to create that future — i.e. McKinsey, Deloitte, etc.)

Usually, neither the customer nor the vendor have a concrete idea of what that looks like. Usually, the vendor refuses to put this partnership notion in writing as a formal contract. On the rare occasion that there is a contract, it is pretty vague, does not oblige the vendor to put forth any business ideas, and allows the vendor to refuse any business idea and investment. In other words, it has zero teeth. Because it’s  so open-ended, the customer can fill the void with all their Turkish Delight dreams.

Moreover, the vendor may sometimes dangle samples of transformation-oriented services and consulting during the sales process. The customer gobbles down these sweet nuggets, and then stares mournfully at the empty box of transformation candy. For the promise of more, they’ll cheerfully betray their enterprise procurement standards, while the sourcing managers stand on the sidelines frantically waving contract-related warnings.

Listen to your sourcing managers when they warn you that the proposed “partnership” is a fiction. The White Witch probably doesn’t have your best interests at heart. Good digital transformation promises — ones that are likely to actually be kept — have concrete outcomes. They specify what the partnership will be developing, together with timelines, budgets, and the legal entity (such as a JV) that will be used to deliver the products/services. Or they specify the specific consulting services that will be provided — workshops, deliverables from those workshops, work-for-hire agreements with specific costs (and discounts, if applicable), and so forth.

Without concrete contractual outcomes, the vendor can vanish the candy into thin air with no repercussions. Sure, in a concrete transformation proposal, the end result will probably not be your Turkish Delight dreams. It might resemble a bowl of ordinary M&Ms. Or maybe a tasty grab-bag of Lindt truffles. (You’d have to get particularly lucky for it get much beyond the realm of grocery-store candy, though.) But you’re much more likely to actually get a good outcome.

Off-hand, I can think of one public example where a prominent “change the industry” vendor partnership with an enterprise, seems to have resulted in a credible product: Microsoft’s Connected Vehicle Platform. There, Microsoft signed a deal with a collection of automakers, each of whom had specific outcomes they wished to achieve — outcomes which could be realistically achieved in a reasonable amount of time, and representing industry advancement but not anything truly revolutionary. Microsoft built upon those individual projects to deliver a platform which would move the industry forward, which was announced with a clear mission and a timeframe for launch. Sure, it didn’t “change the future of cars”, but it brought tangible benefits to the customers.

Vendors often try to sell to who you hope to be, rather than who you are now. Your aspirations aren’t bad. Just make sure that your aspirations are well-defined and there’s a realistic roadmap to achieve them. Hope is not a strategy. The vendor may have little incentive not to promise everything  you could dream of, in order to get you to sign a large purchase agreement.

Contracting in the cloud

There are plenty of cloud (or cloud-ish) companies that will sell you services on a credit card and a click-through agreement. But even when you can buy that way, it is unlikely to be maximally to your advantage to do so, if you have any volume to speak of. And if you do decide to take a contract (which might sometimes be for a zero-dollar-commit), it’s rarely to your advantage to simply agree to the vendor’s standard terms and conditions. This is just as true with the cloud as it is with any other procurement. Vendor T&Cs, whether click-through or contractual, are generally not optimal for the customer; they protect the vendor’s interests, not yours.

Do I believe that deviations from the norm hamper a cloud provider’s profitability, ability to scale, ability to innovate, and so forth? It’s potentially possible, if whatever contractual changes you’re asking for require custom engineering. But many contractual changes are simply things that protect a customer’s rights and shift risk back towards the vendor and away from the customer. And even in cases where custom engineering is necessary, there will be cloud providers who thrive on it, i.e., who find a way to allow customers to get what they need without destroying their own efficiencies. (Arguably, for instance, Salesforce.com has managed to do this with Force.com.)

But the brutal truth is also that as a customer, you don’t care about the vendor’s ability to squeeze out a bit more profit. You don’t want to negotiate a contract that’s so predatory that your success seriously hurts your vendor financially (as I’ve sometimes seen people do when negotiating with startups that badly need revenue or a big brand name to serve as a reference). But you’re not carrying out your fiduciary responsibilities unless you do try to ensure that you get the best deal that you can — which often means negotiating, and negotiating a lot.

Typical issues that customers negotiate include term of delivery of service (i.e., can this provide give you 30 days notice they’ve decided to stop offering the service and poof you’re done?), what happens in a change of control, what happens at the end of the contract (data retrieval and so on), data integrity and confidentiality, data retention, SLAs, pricing, and the conditions under which the T&Cs can change. This is by no means a comprehensive list — that’s just a start.

Yes, you can negotiate with Amazon, Google, Microsoft, etc. And even when vendors publish public pricing with specific volume discounts, customers can negotiate steeper discounts when they sign contracts.

My colleagues Alexa Bona and Frank Ridder, who are Gartner analysts who cover sourcing, have recently written a series of notes on contracting for cloud services, that I’d encourage you to check out:

(Sorry, above notes are clients only.)

Liability and the cloud

I saw an interesting article about cloud provider liability limits, including some quotes from my esteemed colleague Drue Reeves (via Gartner’s acquisition of Burton). A quote in an article about Eli Lilly and Amazon also caught my eye: Tanya Forsheit, founding partner at the Information Law Group, “advised companies to walk away from the business if the cloud provider is not willing to negotiate on liability.”

I frankly think that this advice is both foolish and unrealistic.

Let’s get something straight: Every single IT company out there takes measures to strongly limit its liability in the case something goes wrong. For service providers — data center outsourcers, Web hosting companies, and cloud providers among them — their contracts usually specify that the maximum that they’re liable for, regardless of what happens, is related in some way to the fees paid for service.

Liability is different from an SLA payout. The terms of service-level agreements and their financial penalties vary significantly from provider to provider. SLA payouts are usually limited to 100% of one month of service fees, and may be limited to less. Liability, on the other hand, in most service provider contracts, specifically refers to a limitation of liability clause, which basically states the maximum amount of damages that can be claimed in a lawsuit.

It’s important to note that liability is not a new issue in the cloud. It’s an issue for all outsourced services. Prior to the cloud, any service provider who had their contracts written by a lawyer would always have a limitation of liability clause in there. Nobody should be surprised that cloud providers are doing the same thing. Service providers have generally limited their liability to some multiple of service fees, and not to the actual damage to the customer’s business. This is usually semi-negotiable, i.e., it might be six months of service fees, twelve months of fees, some flat cap, etc., but it’s never unlimited.

For years, Gartner’s e-commerce clients have wanted service providers to be liable for things like revenues lost if the site is down. (American Eagle Outfitters no doubt wishes it could hold IBM’s feet to the fire with that, right now.) Service providers have steadfastly refused, though back a decade or so, the insurance industry had considered whether it was reasonable to offer insurance for this kind of thing.

Yes, you’re taking a risk by outsourcing. But you’re also taking risks when you insource. Contract clauses are not a substitute for trust in the provider, or any kind of proxy indicating quality. (Indeed, a few years back, small SaaS providers often gave away so much money in SLA refunds for outages that we advised clients not to use SLAs as a discounting mechanism!) You are trying to ensure that the provider has financial incentive to be responsible, but just as importantly, a culture and operational track record of responsibility, and that you are taking a reasonable risk. Unlimited liability does not change your personal accountability for the sourcing decision and the results thereof.

In practice, the likelihood that you’re going to sue your hosting provider is vanishingly tiny. The likelihood that it will actually go to trial, rather than being settled in arbitration, is just about nil. The liability limitation just doesn’t matter that much, especially when you take into account what you and the provider are going to be paying your lawyers.

Bottom line: There are better places to focus your contract-negotiating and due-diligence efforts with a cloud provider, than worrying about the limitation-of-liability clause. (I’ve got a detailed research note on cloud SLAs coming out in the future that will go into all of these issues; stay tuned.)

Bookmark and Share

Does Procurement know what you care about?

In many enterprises, IT folks decide what they want to buy and who they want to buy it from, but Procurement negotiates the contract, manages the relationship, and has significant influence on renewals. Right now, especially, purchasing folks have a lot of influence, because they’re often now the ones who go out and shop for alternatives that might be cheaper, forcing IT into the position of having to consider competitive bids.

A significant percentage of enterprise seatholders who use industry advisory firms have inquiry access for their Procurement group, so I routinely talk to people who work in purchasing. Even the ones who are dedicated to an IT procurement function tend not to have more than a minimal understanding of technology. Moreover, when it comes to renewals, they often have no thorough understanding of what exactly it is that the business is actually trying to buy.

Increasingly, though, procurement is self-educating via the Internet. I’ve been seeing this a bit in relationship to the cloud (although there, the big waves are being made by business leadership, especially the CEO and CFO, reading about cloud in the press and online, more so than Purchasing), and a whole lot in the CDN market, where things like Dan Rayburn’s blog posts on CDN pricing provide some open guidance on market pricing. Bereft of context, and armed with just enough knowledge to be dangerous, purchasing folks looking across a market for the cheapest place to source something, can arrive at incorrect conclusions about what IT is really trying to source, and misjudge how much negotiating leverage they’ll really have with a vendor.

The larger the organization gets, the greater the disconnect between IT decision-makers and the actual sourcing folks. In markets where commoditization is extant or in process, vendors have to keep that in mind, and IT buyers need to make sure that the actual procurement staff has enough information to make good negotiation decisions, especially if there are any non-commodity aspects that are important to the buyer.

Bookmark and Share

Vendor horror stories

Everyone has vendor horror stories. No matter how good a vendor normally is, there will be times that they screw up. Some customers will exacerbate a vendor’s tendency to screw up — for instance, they may be someone the vendor really shouldn’t have tried to serve in the first place (heavy customization, i.e., many one-offs from a vendor who emphasizes standardization), or they may just be unlucky and have a sub-par employee on their account team.

Competitors of a vendor, especially small, less-well-known ones, will often loudly trumpet, as part of a briefing, how they won such-and-such a customer from some more prominent vendor, and how that vendor did something particularly horrible to that customer. I often find myself annoyed at such stories. It’s fine to say that you frequently win business away from X company. It’s great to explain your points of differentiation from your rivals. I’m deeply interested in who you think your most significant competitors are. But it’s declasse’ to tell me how much your competitors suck. Also, I can often hear the horror-story anomalies in those tales, as well as detect the real reason — like the desire to shift from a lightly-managed environment to a entirely managed one, or the desire to go from managed to nearly entirely self-managed, etc. I’ll often ask a vendor point-blank about that, and get an admission that this was what really drove the sale. So why not be honest about that in the first place? Say something positive about what you do well.

I think, for the most part, that it doesn’t work on prospective customers any better than it works on analysts. Most decent people recoil somewhat at hearing others put down, whether they are individuals or competing vendors. Prospects often ask me about badmouthing; naturally, they wonder what’s behind the horror stories, but they also wonder why the vendor feels the need to badmouth a competitor in the first place.

I often find that it’s not really the massive, boneheaded incidences that tend to drive churn, anyway. They can be the flash point that precipitates a departure, but far more often, churn is the result of the accumulation of a pile of things that the customer perceives as slights. The vendor has failed to generate competence and/or caring. While sincerity is not a substitute for competence, it can be a temporary salve for it; conversely, competence without conveying that the customer is valued can also be negatively perceived. Human beings, it seems, like to feel important.

Horror stories can be useful to illustrate these patterns of weakness for a particular vendor — a vendor that has trouble planning ahead, a vendor whose proposed customer architectures have a tendency not to scale well, a vendor with a broken service delivery structure, a vendor that doesn’t take accountability, and so on. Interestingly, above-and-beyond stories about vendors can cut both ways — they can illustrate service that is consistently good but is sometimes outstanding, but they can also illustrate exceptions to a vendor’s normal pattern of mediocre service.

As an analyst, I tend to pay the most attention to what customers say about their routine interactions with the vendor. Crisis management is also an important vendor skill, and I like to know how well a vendor responds in a crisis; similarly, the ramp up to getting a renewal is also an important skill. However, it’s the day-to-day stuff that tends to most color people’s perceptions of the relationship.

Still, we all like to tell stories. I’m always looking for a good case study, especially one that illustrates the things that went wrong as well as the things that went right.

Bookmark and Share

Out clauses

I’m seeing an increasing number of IT buyers try to negotiate “out clauses” in their contracts — clauses that let them arbitrarily terminate their services, or which allow them to do so based on certain economy-related business conditions.

People are doing this because they’re afraid of the future. If, for instance, they launch a service and it fails, they don’t want to be stuck in a two-year contract for hosting that service (or colocating that service, or having CDN services for it, etc.). Similarly, if the condition of their business deteriorates, they have an eye on what they can cut in that event.

We’re not talking about businesses that are already on the chopping block — we’re talking about businesses that seem to currently be in good health, whose prospects for growth would seem good. (Businesses that are on the chopping block, or wavering dangerously near it, are behaving in different defensive ways.)

Providers who would previously have never agreed to such conditions are sometimes now willing to negotiate clauses that address these specific fears of businesses. But don’t expect to see such clauses to be common, especially if the service provider has an up-front capital expenditure (such as equipment for dedicated, non-utility hosting). If you’re trying to negotiate a clause like this, you’re much more likely to have success if you tie it to specific business outcomes that would result in you entirely shutting down whatever it is that you’re outsourcing, rather than trying to negotiate an arbitrary out.

Bookmark and Share

Scaling limits and friendly failure

I’m on vacation, and I’ve been playing World of Goo (possibly the single-best construction puzzle game since 1991’s Lemmings by Psygnosis). I was reading the company’s blog (2D Boy), when I came across an entry about BlueHost’s no-notice termination of 2D Boy’s hosting.

And that got me thinking about “unlimited” hosting plans, throttling, limits, and the other challenges of running mass-market hosting — all issues also directly applicable to cloud computing.

BlueHost is a large and reputable provider of mass-market shared hosting. Their accounts are “unlimited”, and their terms of service essentially says that you can consume resources until you negatively impact other customers.

Now, in practice there are limits, and customers are sort of expected to know whether or not their needs fit shared hosting. Most people plan accordingly — although there have been some spectacular failures to do so, such as Sk*rt, a female-focused Digg competitor launched using BlueHost, prompting vast wonder at what kind of utter lack of thought results in trying to launch a high-traffic social networking site on a $7 hosting plan. Unlike Sk*rt, though, it was reasonable for 2D Boy to expect that shared hosting would cover their needs — hosting a small corporate site and blog. They were two guys who were making an indie garage game getting a gradual traffic ramp thanks to word-of-mouth, not an Internet company doing a big launch.

Limits are necessary, but no-notice termination of a legitimate company is bad customer service, however you slice it. Moreover, it’s avoidable bad customer service. Whatever mechanism is used to throttle, suspend service, etc. ought to be adaptable to sending out a warning alert: the “hey, if you keep doing this, you will be in violation of our policies and we’ll have to terminate you” note. Maybe even a, “hey, we will continue to serve your traffic for $X extra, and you have Y time to find a new host or reduce your traffic to normal volumes”. BlueHost does not sell anything beyond its $7 plan, so it has no upsell path; a provider with an upgrade path would hopefully have tried to encourage a migration, rather than executing a cold-turkey cut-off. (By the way, I have been on the service provider side of this equation, so I have ample sympathy for the vendor’s position against a customer whose usage is excessive, but I also firmly believe that no-notice termination of legitimate businesses is not the way to go.)

Automated elastic scaling is the key feature of a cloud, and consequently, limits and the way that they’re enforced technically and managed from a customer service standpoint, will be one of the ways that cloud infrastructure providers differentiate their services.

A vendor’s approach to limits has to be tied to their business goals. Similarly, what a customer desires out of limits must also be tied to their business goals. The customer wants reliable service within a budget. The vendor wants to be fairly compensated and ensure that his infrastructure remains stable.

Ideally, on cloud infrastructure, a customer scales seamlessly and automatically until the point where he is in danger of exceeding his budget. At that point, the system should alert him automatically, allowing him to increase his budget. If he doesn’t want to pay more, he will experience degraded service; degradation should mean slower or lower-priority service, or an automatic switch to a “lite” site, rather than outright failure.

Perhaps when you get right down to it, it’s really about what the failure mode is. Fail friendly. A vendor has a lot more flexibility in imposing limits if it can manage that.

Bookmark and Share

How badly do you need to keep that revenue?

If a customer really wants to leave, you are probably best off letting them leave. Certainly, if they’ve reached the end of their contract, and you are actively engaged in dialogue with one another, you should note little things like, “Your contract auto-renews”.

What not to do: Not tell the customer about the auto-renewal, then essentially point and laugh when they complain that they’re stuck with you for the next several years. Yes, absolutely, it’s the customer’s fault when this happens, and they have only themselves to blame, but it is still a terrible way to do business. (For those of you wondering how this kind of thing happens: Many organizations don’t do a great job of contract management, and it’s the kind of thing that is often lost in the shuffle of a merger, acquisition, re-org, shift from distributed to centralized IT, etc.)

There are all kinds of variants on this — lengthy multi-year auto-renewals, auto-renewals where the price can reset to essentially arbitrary levels, and other forms of “if you auto-renew we get to screw you” clauses, sometimes coupled with massive termination penalties. We’re not talking about month-to-month extensions here, which are generally to the mutual benefit of provider and customer in instances where a new contract simply hasn’t been negotiated yet. We’re really talking about traps for the unwary.

Unhappy customers are no good, but they often sort of gloom along until the end of their contracts and quietly leave. Customers who were unhappy and that you’ve forced into a renewal now hate you. They’ll tell everyone that they can how you screwed them. (And if they tell an analyst, that analyst will probably tell anyone they ever talk to about you, how you screwed another client of theirs. We like objectivity, but we also love a good yarn.) It has a subtle long-term effect on your business that is probably not worth whatever revenue coup you feel you got to pull off. An angry customer can torpedo you to potential prospects as easily as a happy customer can bring you referrals.

Bookmark and Share