Monthly Archives: August 2009
Are multiple cloud APIs bad?
Rackspace has recently launched a community portal called Cloud Tools, showcasing third-party tools that support Rackspace’s cloud compute and storage services. The tools are divided into “featured” and “community”. Featured tools are ones that Rackspace has looked at and believes deserve highlighting; they’re not necessarily commercial projects, but Rackspace does have formal relationships with the developers. Community tools are fro any random joe out there who’d like to be listed. The featured tools get a lot more bells and whistles.
While this is a good move for Rackspace, it’s not ground-breaking stuff, although the portal is notable for a design that seems more consumer-friendly (by contrast with Amazon’s highly text-dense, spartan partner listings). Rather, what’s interesting is Rackspace’s ongoing (successful) efforts to encourage an ecosystem to develop around its cloud APIs, and the broader question of cloud API standardization, “de facto” standards, and similar issues.
There are no small number of cloud advocates out there that believe that rapid standardization in the industry would be advantageous, and that Amazon’s S3 and EC2 APIs, as the APIs with the greatest current adoption and broadest tools support, should be adopted as a de facto standard. Indeed, some cloud-enablement packages, like Eucalyptus, have adopted Amazon’s APIs — and will probably run into API dilemmas as they evolve, as private cloud implementations will be different than public ones, leading to inherent API differences, and a commitment to API compatibility means that you don’t fully control your own feature roadmap. There’s something to be said for compatibility, certainly. Compatibility drives commoditization, which would theoretically lower prices and deliver benefits to end-users.
However, I believe that it’s too early in the market to seek commoditization. Universal commitment to a particular API at this point clamps standardized functionality within a least-common-denominator range, and it restricts the implementation possibilities, to the detriment of innovation. As long as there is rapid innovation and the market continues to offer a slew of new features — something which I anticipate will continue at least through the end of 2011 and likely beyond — standardization is going to be of highly limited benefit.
Rackspace’s API is different than Amazon’s because Rackspace has taken some different fundamental approaches, especially with regard to the network. For another example of significant API differences, compare EMC’s Atmos API to Amazon’s S3 API. Storage is a pretty simple thing, but there are nevertheless meaningful differences in the APIs, reflecting EMC’s different philosophy and approach. (As a sideline, you might find William Vambenepe’s comparison of public cloud APIs in the context of REST, to be an interesting read.)
Everyone can agree on a certain set of core cloud concepts, and I expect that we’ll see libraries that provide unified API access to different underlying clouds; for instance, libcloud (for Python) is the beginning of one such effort. And, of course, third parties like RightScale specialize in providing unified interfaces to multiple clouds.
One thing to keep in mind: Most of the cloud APIs to date are really easy to work with. This means that if you have a tool that supports one API, it’s not terribly hard or time-consuming to make it support another API, assuming that you’re confining yourself to basic functionality.
There’s certainly something to be said in favor of other cloud providers offering an API compatibility layer for basic EC2 and S3 functionality, to satisfy customer demand for such. This also seems to be the kind of thing that’s readily executed as a third-party library, though.
Amazon VPC is not a private cloud
The various reactions to Amazon’s VPC announcement have been interesting to read.
Earlier today, I summarized what VPC is and isn’t, but I realize, after reading the other reactions, that I should have been clearer on one thing: Amazon VPC is not a private cloud offering. It is a connectivity option for a public cloud. If you have concerns about sharing infrastructure, they’re not going to be solved here. If you have concerns about Amazon’s back-end security, this is one more item you’re going to have to trust them on — all their technology for preventing VM-to-VM and VM-to-public-Internet communication is proprietary.
Almost every other public cloud compute provider already offers connectivity options beyond public Internet. Many other providers offer multiple types of Internet VPN (IPsec, SSL, PPTP, etc.), along with options to connect virtual servers in their clouds to colocated or dedicated equipment within the same data center, and options to connect those cloud servers to private, dedicated connectivity, such as an MPLS VPN connection or other private WAN access method (leased line, etc.).
All Amazon has done here is join the club — offering a service option that nearly all their competitors already offer. It’s not exactly shocking that customers want this; in fact, customers have been getting this from competitors for a long time now, bugging Amazon to offer an option, and generally not making a secret of their desires. (Gartner clients: Connectivity options are discussed in my How to Select a Cloud Computing Infrastructure Provider note, and its accompanying toolkit worksheet.)
Indeed, there’s likely a burgeoning market for Internet VPN termination gear of various sorts, specifically to serve the needs of cloud providers — it’s already commonplace to offer a VPN for administration, allowing cloud servers to be open to the Internet to serve Web hits, but only allow administrative logins via the backend VPN-accessed network.
What Amazon has done that’s special (other than being truly superb at public relations) is to be the only cloud compute provider that I know of to fully automate the process of dealing with an IPsec VPN tunnel, and to forego individual customer VLANs for their own layer 2 isolation method. You can expect that other providers will probably automate VPN set-up so in the future, but it’s possibly less of a priority on their road maps. Amazon is deeply committed to full automation, which is necessary at their scale. The smaller cloud providers can get away with some degree of manual provisioning for this sort of thing, still — and it should be pretty clear to equipment vendors (and their virtual appliance competitors) that automating this is a public cloud requirement, ensuring that the feature will show up across the industry within a reasonable timeframe.
Think of it this way: Amazon VPC does not isolate any resources for an individual customer’s use. It provides Internet VPN connectivity to a shared resource pool, rather than public Internet connectivity. It’s still the Internet — the same physical cables in Amazon’s data center and across the world, and the same logical Internet infrastructure, just with a Layer 3 IPsec encrypted tunnel on top of it. VPC is “virtual private” in the same sense that “virtual private” is used in VPN, not in the sense of “private cloud”.
Amazon VPC
Today, Amazon announced a new enhancement to its EC2 compute service, called Virtual Private Cloud (VPC). Amazon’s CTO, Werner Vogels, has, as usual, provided some useful thoughts on the release, accompanied by his thoughts on private clouds in general. And as always, the RightScale blog has a lucid explanation.
So what, exactly, is VPC?
VPC offers network isolation to instances (virtual servers) running in Amazon’s EC2 compute cloud. VPC instances do not have any connectivity to the public Internet. Instead, they only have Internet VPN connectivity (specifically, an IPsec VPN tunnel), allowing the instances to seem as if they’re part of the customer’s private network.
For the non-techies among my readers: Think about the way you connect your PC to a corporate VPN when you’re on the road. You’re on the general Internet at the hotel, but you run a VPN client on your laptop that creates a secure, encrypted tunnel over the Internet, between your laptop and your corporate network, so it seems like your laptop is on your corporate network, with an IP address that’s within your company’s internal address range.
That’s basically what’s happening here with VPC — the transport network is still the Internet, but now there’s a secure tunnel that “extends” the corporate network to an external set of devices. The virtual instances get corporate IP addresses (Amazon now even supports DHCP options), and although of course the traffic is still coming through your Internet gateway and you are experiencing Internet performance/latency/availability, devices on your corporate WAN “think” the instances are local.
To set this up, you use new features of the Amazon API that lets you create a VPC container (a logical construct for the concept of your private cloud), subnets, and gateways. When you actually activate the VPN, you begin paying 5 cents an hour to keep the tunnel up. You pay normal Amazon bandwidth charges on top of that (remember, your traffic is still going over the Internet, so the only extra expense to Amazon is the tunnel itself).
When you launch an EC2 instance, you can now specify that it belongs to a particular VPC subnet. A VPC-enabled instance is not physically isolated from the rest of EC2; it’s still part of the general shared pool of capacity. Rather, the virtual privacy is achieved via Amazon’s proprietary networking software, which they use to isolate virtual instances from one another. (It is not intra-VM firewalling per se; Amazon says this is layer 2 network isolation.)
At the moment, an instance can’t be both be part of a VPC and accessible to the general Internet, which means that this doesn’t solve a common use case — the desire to use a private network for back-end administration or data, but still have the server accessible to the Internet so that it can be customer-facing. Expect Amazon to offer this option in the future, though.
As it currently stands, with an EC2 instance with VPC limited to communicating with other instances within the VPC, as well as the corporate network, this solves the use case of customers who are using EC2 for purely internally-facing applications and are seeking a more isolated environment. While some customers are going to want to have genuinely private network connectivity (i.e., the ability to drop an MPLS VPN connection into the data center), a scenario that Amazon is unlikely to support, the VPC offering is likely to serve many needs.
Note, by the way, that the current limitation on communication also means that EC2 instances can’t reach other Amazon Web services, including S3. (However, EBS does work, as far as I know.) While monitoring is supported, load-balancing is not. Thus, auto-scaling functionality, one of the more attractive recent additions to the platform, is limited.
VPN connectivity for cloud servers is not a new thing in general, and part of what Amazon is addressing with this release is a higher-security option, for those customers who are uncomfortable with the fact that Amazon, unlike most of its competitors, does not offer a private VLAN to each customer. For EC2 specifically, there have been software-only approaches, like CohesiveFT’s VPN-Cubed. Other cloud compute service providers have offered VPN options, including GoGrid and SoftLayer. What distinguishes the Amazon offering is that the provisioning is fully automated, and the technology is proprietary.
This is an important step forward for Amazon, and it will probably cause some re-evaluations by prospective customers who previously rejected an Amazon solution because of the lack of connectivity options beyond public Internet only.
Cloud services are evolving with extraordinary rapidity. I always caution customers not to base deployment plans for one year out on the current state of the technology, because every vendor is evolving so rapidly that the feature that’s currently missing and that you really want has, assuming it’s not something wacky and unusual, a pretty high chance of being available when you’re actually ready to start using the service in a year’s time.
Hype cycles
I’ve recently contributed to a couple of our hype cycles.
Gartner’s very first Hype Cycle for Cloud Computing features a whole array of cloud-related technologies and services. One of the most interesting things about this hype cycle, I think, is the sheer number of concepts that we believe will hit the plateau of productivity in just two to five years. For a nascent technology, that’s pretty significant — we’re talking about a significant fundamental shift in the way that IT is delivered, in a very short time frame. However, a lot of the concepts in this hype cycle haven’t yet hit the peak of inflated expectations — you can expect plenty more hype to be coming your way. There’s a good chance that for the IaaS elements that I focus on, the crash down into the trough of disillusionment will be fairly brief and shallow, but I don’t think it can be avoided. Indeed, I can already tell you tales of clients who got caught up in the overhype and got themselves into trouble. But the “try it and see” aspect of cloud IaaS means that expectations and reality can get a much faster re-alignment than it can if you’re, say, spending a year deploying a new technology in your data center. With the cloud, you’re never far from actually being able to try something and see if it fits your needs.
My hype cycle profile for CDNs appears on our Media Industry Content hype cycle, as well as our brand-new TV-focused (digital distribution and monetization of video) Media Broadcasting hype cycle. Due to the deep volume discounts media companies receive from CDNs, the value proposition is and will remain highly compelling, although I do hear plenty of rumblings about both the desire to use excess origin capacity as well as the possibilities that the cloud offers for both delivery and media archival.
I was involved in, but am not a profile author on, the Hype Cycle for Data Center Power and Cooling Technologies. If you are a data center engineering geek, you’ll probably find it to be quite interesting. Ironically, in the midst of all this new technology, a lot of data center architecture and engineering companies still want to build data centers the way they always have — known designs, known costs, little risk to them… only you lose when that happens. (Colocation companies, who have to own and operate these data centers for the long haul, may be more innovative, but not always, especially since many of them don’t design and build themselves, relying on outside expertise for that.)
Cloud IaaS adoption survey
My colleagues and I are planning to field a survey about cloud computing adoption (specifically, infrastructure as a service), both to assess current attitudes towards cloud IaaS as well as ask people about their adoption plans. The target respondents for the survey will be IT buyers.
We have some questions that we know we want to ask (and that we know our clients, both end-users and vendors, are curious about), and some hypotheses that we want to test, but I’ll ask in this open forum, in an effort to try to ensure the survey is maximally useful: What are the cloud-adoption survey questions whose answers would cause you to change your cloud-related decision-making? (You can reply in a comment, send me email, or Twitter @cloudpundit.)
I expect survey data will help vendors alter their tactical priorities and may alter their strategic plans, and it may assist IT buyers in figuring out where they are relative to the “mainstream” plans (useful when talking to cautious business leadership worried about this newfangled cloud thing).
Somewhat peripherally: Following up on earlier confusion, a potshot was taken at the popularity of surveys at large analyst firms. I’ll note that I’m very much a fan of surveys, and if I had infinite budget to work with, I’d probably field a lot more of them. Surveys are (hopefully) not just blind firing of questions into the populace. Intelligent survey design is an art form (as is proper fielding of a survey). Asking the right questions — forming testable hypotheses whose implications are actionable by clients, and getting good information density out of the questions you ask (looking for patterns in the correlations, not just the individual answers) — is incredibly important if you’re going to get something maximally useful out of the money you spent. Data analysis can drive insights that you wouldn’t have otherwise been able to obtain and/or prove.
The Magic Quadrant, Amazon, and confusion
Despite my previous clarifying commentary on the Magic Quadrant for Web Hosting and Cloud Infrastructure Services (On Demand), posted when the MQ was published, and the text of the MQ itself, there continues to be confusion around the positioning of the vendors in the MQ. This is an attempt to clarify, in brief.
This MQ is not a pure cloud computing MQ. It is a hosting MQ. Titling it as such, and making it such, is not some feeble attempt to defend the traditional way of doing things. It is designed to help Gartner’s clients select a Web hoster, and it’s focused upon the things that enterprises care about. Today, our clients consider cloud players as well as traditional players during the selection process. Cloud has been highly disruptive to the hosting industry, introducing a pile of new entrants, revitalizing minor players and lifting them to a new level, and forcing successful traditional players to revise their approach to the business.
The most common question asked by outsiders who just look at the chart and nothing more is, “Why doesn’t Amazon score higher on vision and execution?”
The answer, simply, is that the hosting MQ scores five use cases — self-managed hosting, mainstream (low/mid-end) managed hosting, highly complex managed hosting, global solutions portfolio (ability to provide multiple types of service packages at multiple price points, globally, for very large multi-nationals seeking global hosting options), and enterprise applications hosting. The final rating is a weighted composite of these scores. Amazon scores extremely highly on self-managed hosting, but has a much more limited ability to support the other four scenarios.
Amazon lacks many capabilities that are important in the overall Web hosting market, like managed services, the ability to mix in dedicated equipment (important to anyone who wants to run things that don’t virtualize well, like large-scale Oracle databases, as well as colocate “black box” hardware appliances, like those used for transaction functions for some e-commerce sites), the ability to isolate the environment from the Internet and just use private network connectivity, etc. Their lack of these capabilities hurts their scores. (Note that some capabilities that were missing may have been disclosed to us as part of Amazon’s roadmap, which augmented their Vision score positively, but similarly, stances taken that would definitively shut out some features would be penalized.)
Clearly, we don’t think that Amazon sucks as a cloud provider; it’s just that they don’t play as broadly in the hosting space as the best of the traditional players, although they are certainly a competitor against the traditional players, and a disruptive entrant in general.
The same could be said for many of Amazon’s cloud competitors, although those with some background in traditional hosting may have fewer product-portfolio gaps. Original innovation is a component of Vision but it’s only part of the overall Vision score, so being a fast follower only hurts you so much.
We recognize the need for a “pure cloud compute” vendor rating, and have one in the works.
Cloudy inquiry trends
I haven’t been posting much lately, due to being overwhelmingly busy with client inquiries, and having a few medical issues that have taken me out of the action somewhat. So, this is something of a catch-up, state-of-the-universe-from-my-perspective, inquiry-trends post.
With the economy picking up a bit, and businesses starting to return to growth initiatives rather than just cost optimization, and the approach of the budget season, the flow of client inquiry around cloud strategy has accelerated dramatically, to the point where cloud inquiries are becoming the overwhelming majority of my inquiries. Even my colocation and data center leasing inquiries are frequently taking on a cloud flavor, i.e., “How long more should we plan to have this data center, rather than just putting everything in the cloud?”
Organizations have really absorbed the hype — they genuinely believe that shortly, the cloud will solve all of their infrastructure issues. Sometimes, they’ve even made promises to executive management that this will be the case. Unfortunately, in the short term (i.e., for 2010 and 2011 planning), this isn’t going to be the case for your typical mid-size and enterprise business. There’s just too much legacy burden. Also, traditional software licensing schemes simply don’t work in this brave new world of elastic capacity.
The enthusiasm, though, is vast, which means that there are tremendous opportunities out there, and I think it’s both entirely safe and mainstream to run cloud infrastructure pilot projects right now, including large-scale, mission-critical, production infrastructure pilots for a particular business need (as opposed to deciding to move your whole data center into the cloud, which is still bleeding-edge adopter stuff). Indeed, I think there’s a significant untapped potential for tools that ease this transition. (Certainly there are any number of outsourcers and consultants who would love to charge you vast amounts of money to help you migrate.)
We see the colocation and data center leasing markets shift with the economy, and the trends and the players shift with them, especially as strong new regionals and high-density players emerge. The cloud influence is also significant, as people try to evaluate what their real needs for space will be going forward; this is particularly true for anyone looking at long-term leases, and wondering what the state of IT will be like going out ten years. Followers of this space should check out SwitchNAP for a good example of the kind of impact that a new player can make in a very short time (they opened in December).
August has been a consistently quiet month for CDN contract inquiries, and this year is no exception, but the whole of last three months has really been hopping. The industry is continuing to shift in interesting ways, not just because of the dynamics of the companies involved, but because of changing buyer needs. Also, there was a very interesting new launch in July, in the application delivery network space, a company called Asankya, definitely worth checking out if you follow this space.
All in all, there’s a lot of activity, and it’s becoming more future-focused as people get ready to prep their budgets. This is good news for everyone, I think. Even though the fundamental economic shifts have driven companies to be more value-driven, I think there’s a valuable emphasis being placed on the right solutions at the right price, that do the right thing for the business.
CloudPundit: Massive-Scale Computing 