Eric Domage, an analyst over at IDC, is being quoted as saying, “The decision in the next year or two will only be about the private cloud. The bigger the company, the more they will consider the private cloud. The enterprise cloud is locked down and totally managed. It is the closest replication of virtualisation.” The same article goes on to quote Domage as cautioning against the dangers of the public cloud, and, quoting the article: “He urged delegates at the conference to ‘please consider more private cloud than public cloud.’”
I disagree entirely, and I think Domage’s comments ignore the reality of what is going on within enterprises, both in terms of their internal private cloud initiatives, as well as their adoption of public cloud IaaS. (I assume Domage’s commentary is intended to be specific to infrastructure, or it would be purely nonsensical.)
While not all IaaS providers build to the same security standards, nearly all build a high degree of security into their offering. Furthermore, end-to-end encryption, which Domage claims is unavailable in public cloud IaaS, is available in multiple offerings today, presuming that it refers to both end-to-end network encryption, along with encryption of both storage in motion and storage at rest. (Obviously, computation has to occur either on unencrypted data, or your app has to treat encrypted data like a passthrough.)
And for the truly paranoid, you can adopt something like Harris Trusted Cloud — private or public cloud IaaS built with security and compliance as the first priority, where each and every component is checked for validity. (Wyatt Starnes, the guiding brain behind this, founded Tripwire, so you can guess where the thinking comes from.) Find me an enterprise that takes security to that level today.
I’ve found that the bigger the company, the more likely they are to have already adopted public cloud IaaS. Yes, it’s tactical, but their businesses are moving faster than they can deploy private clouds, and the workloads they have in the public cloud are growing every day. Yes, they’ll also build a private cloud (or in many cases, already have), but they’ll use both.
The idea that the enterprise cloud is “locked down and totally managed” is a fantasy. Not only do many enterprises struggle with managing the security within private clouds, many of them have practically surrendered control to the restless natives (developers) who are deploying VMs within that environment. They’re struggling with basic governance, and often haven’t extended their enterprise IT operations management systems successfully into that private cloud. (Assuming, as the article seems to imply, that private cloud is being used to refer to self-service IaaS, not merely virtualized infrastructure.)
The head-in-the-sand “la la la public cloud is too insecure to adopt, only I can build something good enough” position will only make an enterprise IT manager sound clueless and out of touch both with reality and with the needs of the business.
Organizations certainly have to do their due diligence — hopefully before, and not after, the business is asking what cloud infrastructure solutions can be used right this instant. But the prudent thing to do is to build expertise with public cloud (or hosted private cloud), and for organizations which intend to continue running data centers long-term, simultaneously building out a private cloud.
I’ve been trialing cloud IaaS providers lately, and the frustration of getting through many of the sign-up processes has reminded me of some recurring conversations that I’ve had with service providers over the past few years.
Many cloud IaaS providers regard the fact that they don’t take online sign-ups as a point of pride — they’re not looking to serve single developers, they say. This is a business decision, which is worth examining separately (a future blog post, and I’ve already started writing a research note on why that attitude is problematic).
However, many cloud IaaS providers state their real reason for not taking online sign-ups, or of having long waiting periods to actually get an account provisioned (and silently dropping some sign-ups into a black hole, whether or not they’re actually legitimate), is that they’re trying to avoid the bad eggs — credit card fraud, botnets, scammers, spammers, whatever. Some cloud providers go so far as to insist that they have a “private cloud” because it’s not “open to the general public”. (I consider this lying to your prospects, by the way, and I think it’s unethical. “Marketing spin” shouldn’t be aimed at making prospects so dizzy they can’t figure out your double-talk. The industry uses NIST definitions, and customers assume NIST definitions, and “private” therefore implies “single-tenant”.)
But the thing that worries me is that cloud IaaS providers claim that vetting who signs up for their cloud, and ensuring that they’re “real businesses”, makes their public, multi-tenant cloud “safe”. It doesn’t. In fact, it can lure cloud providers into a false sense of complacency, assuming that there will be no bad actors within their cloud, which means that they do not take adequate measures to defend against bad actors who work for a customer — or against customer mistakes, and most importantly, against breaches of a customer’s security that result in bad eggs having access to their infrastructure.
Cloud providers tell me that folks like Amazon spend a ton of money and effort trying to deal with bad actors, since they get tons of them from online sign-ups, and that they themselves can’t do this, either for financial or technical reasons. Well, if you can’t do this, you are highly likely to also not have the appropriate alerting to see when your vaunted legitimate customers have been compromised by the bad guys and have gone rogue; and therefore to respond to it immediately and automatically to stop the behavior and thereby protect your infrastructure and customers; and to hopefully automatically, accurately, and consistently do the forensics for law enforcement afterwards. Because you don’t expect it to be a frequent problem, you don’t have the paranoid level of automatic and constant sweep-and-enforce that a provider like Amazon has to have.
And that should scare every enterprise customer who gets smugly told by a cloud provider that they’re safe, and no bad guys can get access to their platform because they don’t take credit-card sign-ups.
So if you’re a security-conscious company, considering use of multi-tenant cloud services, you should ask prospective service providers, “What are you doing to protect me from your other customers’ security problems, and what measures do you have in place to quickly and automatically detect and eliminate bad behavior?” — and don’t accept “we only accept upstanding citizens like yourself on our cloud, sir” as a valid answer.
Here’s a succinct correction:
To quote Wikipedia, ISO 27001 requires that management:
- Systematically examine the organization’s information security risks, taking account of the threats, vulnerabilities and impacts;
- Design and implement a coherent and comprehensive suite of information security controls and/or other forms of risk treatment (such as risk avoidance or risk transfer) to address those risks that are deemed unacceptable; and
- Adopt an overarching management process to ensure that the information security controls continue to meet the organization’s information security needs on an ongoing basis.
ISO 27002, which details the security best practices, is not required to be used in conjunction with 27001, although this is customary. I forgot this when I wrote my post (when I was reading docs written by my colleagues on our security team, which specifically recommend the 27001 approach, in the context of 27002).
In other words: 27002 is proscriptive in its controls; 27001 is not that specific.
So FlyingPenguin is right — without the 27002, we have no idea what security controls Amazon has actually implemented.
Greetings from Gartner’s Application Architecture, Development, and Integration Summit. There are around 900 people here, and the audience is heavy on enterprise architects and other application development leaders.
One of the common themes of my interaction here has been talking to an awful lot of people who are using or have used Amazon for IaaS. They’re a different audience than the typical clients I talk to about the cloud, who are generally IT Operations folks, IT executives, or Procurement folks. The audience here is involved in assessing the cloud, and in adopting the cloud in more skunkworks ways — but they are generally not ultimately the ones making the purchasing decisions. Consequently, they’ve got a level of enthusiasm about it that my usual clients don’t share (although it correlates with the reported enthusiasm they know their app dev folks have for it). Fun conversations.
So on the heels of Amazon’s ISO 27001 certification, I thought it’d be worth jotting down a few thoughts about Amazon and the enterprise.
To start with, SAS 70 Is Not Proof of Security, Continuity or Privacy Compliance (Gartner clients only). As my security colleagues Jay Heiser and French Caldwell put it, “The SAS 70 auditing report is widely misused by service providers that find it convenient to mischaracterize the program as being a form of security certification. Gartner considers this to be a deceptive and harmful practice.” It certainly is possible for a vendor to do a great SAS 70 certification — to hold themselves to best pratices and have the audit show that they follow them consistently — but SAS 70 itself doesn’t require adherence to security best practices. It just requires you to define a set of controls, and then demonstrate you follow them.
ISO 27001, on the other hand, is a security certification standard that examines the efficacy of risk management and an organization’s security posture, in the context of ISO 27002, which is a detailed security control framework. This certification actually means that you can be reasonably assured that an organization’s security controls are actually good, effective ones.
The 27001 cert — especially meaningful here because Amazon certified its actual infrastructure platform, not just its physical data centers — addresses two significant issues with assessing Amazon’s security to date. First, Amazon doesn’t allow enterprises to bring third-party auditors into its facilities and to peer into its operations, so customers have to depend on Amazon’s own audits (which Amazon does share under certain circumstances). Second, Amazon does a lot of security secret sauce, implementing things in ways different than is the norm — for instance, Amazon claims to provide network isolation between virtual machines, but unlike the rest of the world, it doesn’t use VLANs to achieve this. Getting something like ISO 27001, which is proscriptive, hopefully offers some assurance that Amazon’s stuff constitutes effective, auditable controls.
(Important correction: See my follow-up. The above statement is not true, because we have no guarantee Amazon follows 27002.)
A lot of people like to tell me, “Amazon will never be used by the enterprise!” Those people are wrong (and are almost always shocked to hear it). Amazon is already used by the enterprise — a lot. Not necessarily always in particularly “official” ways, but those unofficial ways can sometimes stack up to pretty impressive aggregate spend. (Some of my enterprise clients end up being shocked by how much they’re spending, once they total up all the credit cards.)
And here’s the important thing: The larger the enterprise, the more likely it is that they use Amazon, to judge from my client interactions. (Not necessarily as their only cloud IaaS provider, though.) Large enterprises have people who can be spared to go do thorough evaluations, and sit on committees that write recommendations, and decide that there are particular use cases that they allow, or actively recommend, Amazon for. These are companies that assess their risks, deal with those risks, and are clear on what risks they’re willing to take with what stuff in the cloud. These are organizations — some of the largest global companies in the world — for whom Amazon will become a part of their infrastructure portfolio, and they’re comfortable with that, even if their organizations are quite conservative.
Don’t underestimate the rate of change that’s taking place here. The world isn’t shifting overnight, and we’re going to be looking at internal data centers and private clouds for many years to come, but nobody can afford to sit around smugly and decide that public cloud is going to lose and that a vendor like Amazon is never going to be a significant player for “real businesses”.
One more thing, on the subject of “real businesses”: All of the service providers who keep telling me that your multi-tenant cloud isn’t actually “public” because you only allow “real businesses”, not just anyone who can put down a credit card? Get over it. (And get extra-negative points if you consider all Internet-centric companies to not be “real businesses”.) Not only isn’t it a differentiator, but customers aren’t actually fooled by this kind of circumlocution, and the guys who accept credit cards still vet their customers, albeit in more subtle ways. You’re multi-tenant, and your customers aren’t buying as a consortium or community? Then you’re a public cloud, and to claim otherwise is actively misleading.
Just because it’s in the cloud doesn’t make it magic. And it can be very, very dangerous to assume that it is.
I recently talked to an enterprise client who has a group of developers who decided to go out, develop, and run their application on Amazon EC2. Great. It’s working well, it’s inexpensive, and they’re happy. So Central IT is figuring out what to do next.
I asked curiously, “Who is managing the servers?”
The client said, well, Amazon, of course!
Except Amazon doesn’t manage guest operating systems and applications.
It turns out that these developers believed in the magical cloud — an environment where everything was somehow mysteriously being taken care of by Amazon, so they had no need to do the usual maintenance tasks, including worrying about security — and had convinced IT Operations of this, too.
Imagine running Windows. Installed as-is, and never updated since then. Without anti-virus, or any other security measures, other than Amazon’s default firewall (which luckily defaults to largely closed).
Plus, they also assumed that auto-scaling was going to make their app magically scale. It’s not designed to automagically scale horizontally. Somebody is going to be an unhappy camper.
Cautionary tale for IT shops: Make sure you know what the cloud is and isn’t getting you.
Cautionary tale for cloud providers: What you’re actually providing may bear no resemblance to what your customer thinks you’re providing.
Most of us have long since overcome our fear of handing over our credit cards to Internet merchants. It’s become routine for most of us to simply do so. We buy stuff, we sign up for subscriptions, it’s just like handing over plastic anytime else. For that matter, most of us have never really thought about all that credit card data laying around in the hands of brick-and-mortar merchants with whom we do business, until the unfortunate times when that data gets mass-compromised.
Bad billing problems plague lots of organizations, but Electronic Arts (in the form of its Mythic Entertainment studio, which does the massively multiplayer online RPGs Dark Ages of Camelot and Warhammer Online) just had a major screw-up: a severe billing system error that, several days ago, repeatedly charged customers their subscription fees. Not just one extra charge, but, some users say, more than sixty. Worse still, the error reportedly affected not just current customers, but past customers. A month’s subscription is $15, but users can pre-pay for as much as a year. And these days, with credit cards so often actually being checking-account debit cards, that is often an immediate hit to the wallet. So you can imagine the impact on even users with decent bank balances, being hit by multiple charges. (Plenty of people with good-sized savings cushions only keep enough money in the checking account to cover expected bills, so you don’t have to be on the actual fiscal edge to get smacked with overdraft fees.) EA is scrambling to get this straightened out, of course, but this is every company’s worst billing nightmare, and it comes at a time when EA and its competitors are all scrambling to shift their business models online.
How many merchants that you don’t do business with any longer, but used to have recurring billing permission on your credit card, still have your credit card on file? As online commerce and micropayments proliferate, how many more merchants will store that data? (Or will PayPal, Apple’s storefronts, and other payment services rule the world?)
Recent links of interest…
I’ve heard that no less than four memcached start-ups have been recently funded. GigaOM speculates interestingly on whether memcached is good or bad for MySQL. It seems to me that in the age of cloud and hyperscale, we’re willing to sacrifice ACID compliance in many our transactions. RAM is cheap, and simplicity and speed are king. But I’m not sure that the widespread use of memcached in Web 2.0 applications, as a method of scaling a database, reflects the strengths of memcache so much as they reflect the weaknesses of the underlying databases.
Column-oriented databases are picking up some buzz lately. Sybase has a new white paper out on high-performance analytics. MySQL is plugging Infobright, a column-oriented engine for MySQL (replacing MyISAM, InnoDB, etc., just like any other engine).
Brian Krebs, the security blogger for the Washington Post, has an excellent post called The Scrap Value of a Hacked PC. It’s an examination of the ways that hacked PCs can be put to criminal use, and it’s intended to be printed out and handed to end-users who don’t think that security is their personal responsibility.
My colleague Ray Valdes has some thoughts on intuition-based vs. evidence-based design. It’s a riff on the recent New York Times article, Data, Not Design, Is King in the Age of Google, and a departing designer’s blog post that provides a fascinating look at data-driven decision making in an environment where you can immediately test everything.
We are, it seems, in the midst of a wave of distributed denial of service attacks. The victims include:
- Neustar’s UltraDNS. (Problems with specific regional DNS clusters, with little customer-visible impact.)
- Register.com. (Severe impact on Web hosting and email customers.)
- GoGrid. (Severe impact on cloud hosting customers.)
- ThePlanet. (Attack on their DNS servers, with severe impact on customers.)
The attack on ThePlanet is unusual in that it received minimal attention in the press, despite the company being one of the largest Web hosters, and having Cisco Guard (DDoS mitigation) appliances in place. Also, the status updates were eventually issued via Twitter, rather than a more expected form of customer communication. Here’s the full text, aggregated off Twitter:
“Between 2:30am and 5:00am CDT on April 8, The Planet’s name servers were flooded again with a large brute force (DDoS) attack. Unlike the previous attack, this attack did not appear to be DNS-specific; instead, targeted resources indirectly supporting DNS services. Because the nature of this attack was different from the previous event, mirroring the response to the previous attack was ineffective. Once our investigation determined the nature of the attack, we applied filters throughout our DNS support system to alleviate the effects. The Planet’s network and DNS performance have been restored, and the attack originator has ceased actions. Any lingering issues may be indicative of a different problem that may have been exacerbated by the attack and should be resolved quickly. We are working on several projects to help mitigate similar attacks in the future. Once those plans are in order, we will update the DNS Status announcement thread in our community forums. We understand that other providers are experiencing similar events. We will reach out to them, pool our information and then work together to find consistencies between attacks. Our goal is to establish best practices as an industry to better respond to these recent events.“
Jose Nazario of Arbor Networks claims these attacks are not Conficker at work, which makes this wave of attacks even more interesting.
The takeaway from this: Customers understand if you get DDoS’d. They don’t put up with a lack of communication. It’s enormously difficult to communicate with customers in the midst of a crisis, especially one that takes down customer-facing infrastructure in a customer-impacting way, but it’s also incredibly critical. Clearly, not everyone in the company is out trying to troubleshoot the problem, so you can usefully put them to work reaching out to your customers, if you have the policies and procedures in place to do so successfully.
Something to think about today, no matter who you are and who you work for: What policies do you have in place for customer communications when a crisis hits your company? (Book recommendation: Eric Dezenhall’s Damage Control, which is a hard-edged, realistic look at communication in a crisis, including coping with competitors who are deliberately fanning the negative-PR flames.)
Tossing a few links out there…
In the weekend’s biggest cloud news, Microsoft’s Azure was down for 22 hours. It’s now back up, with no root cause known.
Geva Perry has posted a useful Zoho Sheet calculator for figuring out whether an Amazon EC2 reserved instance will save you money over an unreserved instance.
Craig Balding has posted a down-to-earth dissection of PCI compliance in the cloud, and the practical reality that cloud infrastructure providers tend to deal with PCI compliance by encouraging you to push the actual payment stuff off to third parties.
Anti-virus vendor Authentium is now offering its AV-scanning SDK to cloud providers.
Authentium, unlike most other AV vendors, has traditionally been focused at the gateway; they offer an SDK designed to be embedded in applications and appliances. (Notably, Authentium is the scanning engine used by Google’s Postini service.) So courting cloud providers is logical for them.
Anti-virus integration makes particular sense for cloud storage providers. Users of cloud storage upload millions of files a day. Many businesses that use cloud storage do so for user-generated content. AV-scanning a file as part of an upload could be just another API call — one that could be charged for on a per-operation basis, just like GET, PUT, and other cloud storage operations. That would turn AV scanning into a cloud Web service, making it trivially easy for developers to integrate AV scanning into their applications. It’d be a genuine value-add for using cloud storage — a reason to do so beyond “it’s cheap”.
More broadly, security vendors have become interested in offering scanning as a service, although most have desktop installed bases to defend, and thus are looking at it as a supplement as opposed to a replacement for traditional desktop AV products; see the past news on McAfee’s Project Artemis or Trend Micro’s Smart Protection Network for examples.