Blog Archives
Cloud ecosystems for small businesses
As I’ve been predicting for a while, Microsoft and Intuit have joined forces around Quickbooks and Azure: Microsoft and Intuit announced that Intuit would name Microsoft’s Windows Azure as the preferred platform for cloud app development on its Intuit Partner Platform. This is an eminently logical partnership. MSDN developers, are a critical channel for reaching the small business with applications, Azure is evolving to be well-suited to that community, and Intuit’s Quickbooks is a key anchor application for the small business. Think of this partnership as the equivalent of Force.com for the small business; arguably, Quickbooks is an even more compelling anchor application for a PaaS ecosystem than CRM is.
A lot of non-IT companies are thinking about cloud strategies these days. I get a great deal of inquiry from companies seeking to target the small business with cloud offerings, and the question that I keep having to ask is, “What natural value does your existing business bring when extended to the cloud?” An astounding number of strategy people at miscellaneous companies seem to believe that they ought to be cloud IaaS providers, or resellers of other people’s SaaS solutions for small businesses — without being natural places for small businesses to turn for either infrastructure or software.
Whatever your business is, if you want to create a cloud ecosystem, you need an anchor service. Take something that you do today, and leverage cloud precepts. Consider doing something like creating a data service around it, opening up an API, and the like. (Gartner clients: My colleague Eric Knipp has written a useful research note on this topic entitled Open RESTful APIs are Big Business.) Use that as the centerpiece for an ecosystem of related services from partners, and the community of users.
The last quarter in review
The end of 2009 was extraordinarily busy, and that’s meant that, shamefully, I haven’t posted to my blog in ages. I aim to try to return to near-daily posting in 2010, but this means creating time in my schedule to think and research and write, rather than being entirely consumed by client inquiry.
December was Gartner’s data center conference, where I spent most of a week in back-to-back meetings, punctuated by a cloud computing end-user roundtable, a cloud computing town hall, and my talk on getting data center space. Attendance at the conference is skewed heavily towards large enterprises, but one of the most fascinating bits that emerged out of the week was the number of people walking around with emails from their CEO saying that they had to investigate this cloud computing thing, and whose major goals for the conference included figuring out how the heck they were going to reply to that email.
My cloud computing webinar is now available for replay — it’s a lightweight introduction to the subject. Ironically, when I started working at Gartner, I was terrified of public speaking, and much more comfortable doing talks over the phone. Now, I’m used to having live audiences and public speaking is just another routine day on the job… but speaking into the dead silence of an ATC is a little unnerving. (I once spent ten minutes giving a presentation to dead air, not realizing that the phone bridge had gone dead.) There were tons of great questions asked by the audience, far more than could possibly be answered in the Q&A time, but I’m taking the input and using it to figure out how to decide what I should be writing this year.
Q4 2009, by and large, continued my Q3 inquiry trends. Tons of colocation inquiries — but colocation is often giving way to leasing, now, and local/regional players are prominent in nearly every deal (and winning a lot of the deals). Relatively quiet on the CDN front, but this has to be put in context — Gartner’s analysts took over 1300 inquiries on enterprise video during 2009, and these days I’m pretty likely to look at a client’s needs and tell them they need someone like Kontiki or Ignite, not a traditional Internet CDN. And cloud, cloud, cloud is very much on everyone’s radar screen, with Asia suddenly becoming hot. Traditional dedicated hosting is dying at a remarkable pace; it’s unusual to see new deals that aren’t virtualized.
I’ll be writing on all this and more in the new year.
Recent inquiry trends
It’s been mentioned to me that my “what are you hearing about from clients” posts are particularly interesting, so I’ll try to do a regular update of this sort. I have some limits on how much detail I can blog and stay within Gartner’s policies for analysts, so I can’t get too specific; if you want to drill into detail, you’ll need to make a client inquiry.
It’s shaping up into an extremely busy fall season, with people — IT users and vendors like — sounding relatively optimistic about the future. If you attended Gartner’s High-Tech Forum (a free event we recently did for tech vendors in Silicon Valley), you saw that we showed a graph of inquiry trends, indicating that “cost” is a declining search term, and “cloud” has rapidly increased in popularity. We’re forecasting a slow recovery, but at least it’s a recovery.
This is budget and strategic planning time, so I’m spending a lot of time with people discussing their 2010 cloud deployment plans, as well as their two- and five-year cloud strategies. There’s some planning stuff going around data centers, hosting, and CDN services, too, but the longer-term the planning, the more likely it is that it’s going to involve cloud. (I posted on cloud inquiry trends previously.)
There’s certainly purchasing going on right now, though, and I’m talking to clients across the whole of the planning cycle (planning, shortlisting, RFP review, evaluating RFP responses, contract review, re-evaluating existing vendors, etc.). Because pretty much everything that I cover is a recurring service, I don’t see the end-of-year rush to finish spending 2009’s budget, but this is the time of year when people start to work on the contracts they want to go for as soon as 2010’s budget hits.
My colo inquiries this year have undergone an interesting shift towards local (and regional) data centers, rather than national players, reflecting a shift in colocation from being primarily an Internet-centric model, to being one where it’s simply another method by which businesses can get data center space. Based on the planning discussions I’m hearing, I expect this is going to be the prevailing trend going forward, as well.
People are still talking about hosting, and there are still plenty of managed hosting deals out there, but very rarely do I see a hosting deal now that doesn’t have a cloud discussion attached. If you’re a hoster and you can’t offer capacity on demand, most of my clients will now simply take you off the table. It’s an extra kick in the teeth if you’ve got an on-demand offering but it’s not yet integrated with your managed services and/or dedicated offerings; now you’re competing as if you were two providers instead of one.
The CDN wars continue unabated, and competitive bidding is increasingly the norm, even in small deals. Limelight Networks fired a salvo into the fray yesterday, with an update to their delivery platform that they’ve termed “XD”. The bottom line on that is improved performance at a baseline for all Limelight customers, plus a higher-performance tier and enhanced control and reporting for customers who are willing to pay for it. I’ll form an opinion on its impact once I see some real-world performance data.
There’s a real need in the market for a company who can monitor actual end-user performance and that can do consulting assessments of multiple CDNs and origin configurations. (It’d be useful in the equipment world, too, for ADCs and WOCs.) Not everyone can or wants to deploy Keynote or Gomez or Webmetrics for this kind of thing, those companies aren’t necessarily eager to do a consultative engagement of this sort, and practically every CDN on the planet has figured out how to game their measurements to one extent or another. It doesn’t make them without value in such assessments, but real-world data from actual users (via JavaScript agents, video player instrumentation, download client instrumentation, etc.) is still vastly preferable. Practically every client I speak to wants to do performance trials, but the means available for doing so are still overly limited and very expensive.
All in all, things are really crazy busy. So busy, in fact, that I ended up letting a whole month go by without a blog post. I’ll try to get back into the habit of more frequent updates. There’s certainly no lack of interesting stuff to write about.
Speculating on Amazon’s capacity
How much capacity does Amazon EC2 have? And how much gets provisioned?
Given that it’s now clear that there are capacity constraints on EC2 (i.e., periods of time where provisioning errors out due to lack of capacity), this is something that’s of direct concern to users. And for all the cloud-watchers, it’s a fascinating study of IaaS adoption.
Randy Bias of CloudScaling has recently posted some interesting speculation on EC2 capacity.
Guy Rosen has done a nifty analysis of EC2 resource IDs, translated to an estimate of the number of instances provisioned on the platform in a day. Remember, when you look at provisioned instances (i.e., virtual servers), that many EC2 instances are short-lived. Auto-scaling can provision and de-provision servers frequently, and there’s significant use of EC2 for batch-computing applications.
Amazon’s unreserved-instance capacity is not unlimited, as people have discovered. There are additional availability zones, and for serious users of the platform, choosing the right zone has become minimal, since you don’t want to pay for cross-zone data transfers or absorb the latency impact, if you don’t have to.
We’re entering a time of year that’s traditionally a traffic ramp for Amazon, the fall leading into Christmas. It should be interesting to see how Amazon balances its own need for capacity (AWS is used for portions of the company’s retail site), reserved EC2 capacity, and unreserved EC2 capacity. I suspect that the nature of EC2’s usage makes it much more bursty than, say, a CDN.
Are multiple cloud APIs bad?
Rackspace has recently launched a community portal called Cloud Tools, showcasing third-party tools that support Rackspace’s cloud compute and storage services. The tools are divided into “featured” and “community”. Featured tools are ones that Rackspace has looked at and believes deserve highlighting; they’re not necessarily commercial projects, but Rackspace does have formal relationships with the developers. Community tools are fro any random joe out there who’d like to be listed. The featured tools get a lot more bells and whistles.
While this is a good move for Rackspace, it’s not ground-breaking stuff, although the portal is notable for a design that seems more consumer-friendly (by contrast with Amazon’s highly text-dense, spartan partner listings). Rather, what’s interesting is Rackspace’s ongoing (successful) efforts to encourage an ecosystem to develop around its cloud APIs, and the broader question of cloud API standardization, “de facto” standards, and similar issues.
There are no small number of cloud advocates out there that believe that rapid standardization in the industry would be advantageous, and that Amazon’s S3 and EC2 APIs, as the APIs with the greatest current adoption and broadest tools support, should be adopted as a de facto standard. Indeed, some cloud-enablement packages, like Eucalyptus, have adopted Amazon’s APIs — and will probably run into API dilemmas as they evolve, as private cloud implementations will be different than public ones, leading to inherent API differences, and a commitment to API compatibility means that you don’t fully control your own feature roadmap. There’s something to be said for compatibility, certainly. Compatibility drives commoditization, which would theoretically lower prices and deliver benefits to end-users.
However, I believe that it’s too early in the market to seek commoditization. Universal commitment to a particular API at this point clamps standardized functionality within a least-common-denominator range, and it restricts the implementation possibilities, to the detriment of innovation. As long as there is rapid innovation and the market continues to offer a slew of new features — something which I anticipate will continue at least through the end of 2011 and likely beyond — standardization is going to be of highly limited benefit.
Rackspace’s API is different than Amazon’s because Rackspace has taken some different fundamental approaches, especially with regard to the network. For another example of significant API differences, compare EMC’s Atmos API to Amazon’s S3 API. Storage is a pretty simple thing, but there are nevertheless meaningful differences in the APIs, reflecting EMC’s different philosophy and approach. (As a sideline, you might find William Vambenepe’s comparison of public cloud APIs in the context of REST, to be an interesting read.)
Everyone can agree on a certain set of core cloud concepts, and I expect that we’ll see libraries that provide unified API access to different underlying clouds; for instance, libcloud (for Python) is the beginning of one such effort. And, of course, third parties like RightScale specialize in providing unified interfaces to multiple clouds.
One thing to keep in mind: Most of the cloud APIs to date are really easy to work with. This means that if you have a tool that supports one API, it’s not terribly hard or time-consuming to make it support another API, assuming that you’re confining yourself to basic functionality.
There’s certainly something to be said in favor of other cloud providers offering an API compatibility layer for basic EC2 and S3 functionality, to satisfy customer demand for such. This also seems to be the kind of thing that’s readily executed as a third-party library, though.
Amazon VPC is not a private cloud
The various reactions to Amazon’s VPC announcement have been interesting to read.
Earlier today, I summarized what VPC is and isn’t, but I realize, after reading the other reactions, that I should have been clearer on one thing: Amazon VPC is not a private cloud offering. It is a connectivity option for a public cloud. If you have concerns about sharing infrastructure, they’re not going to be solved here. If you have concerns about Amazon’s back-end security, this is one more item you’re going to have to trust them on — all their technology for preventing VM-to-VM and VM-to-public-Internet communication is proprietary.
Almost every other public cloud compute provider already offers connectivity options beyond public Internet. Many other providers offer multiple types of Internet VPN (IPsec, SSL, PPTP, etc.), along with options to connect virtual servers in their clouds to colocated or dedicated equipment within the same data center, and options to connect those cloud servers to private, dedicated connectivity, such as an MPLS VPN connection or other private WAN access method (leased line, etc.).
All Amazon has done here is join the club — offering a service option that nearly all their competitors already offer. It’s not exactly shocking that customers want this; in fact, customers have been getting this from competitors for a long time now, bugging Amazon to offer an option, and generally not making a secret of their desires. (Gartner clients: Connectivity options are discussed in my How to Select a Cloud Computing Infrastructure Provider note, and its accompanying toolkit worksheet.)
Indeed, there’s likely a burgeoning market for Internet VPN termination gear of various sorts, specifically to serve the needs of cloud providers — it’s already commonplace to offer a VPN for administration, allowing cloud servers to be open to the Internet to serve Web hits, but only allow administrative logins via the backend VPN-accessed network.
What Amazon has done that’s special (other than being truly superb at public relations) is to be the only cloud compute provider that I know of to fully automate the process of dealing with an IPsec VPN tunnel, and to forego individual customer VLANs for their own layer 2 isolation method. You can expect that other providers will probably automate VPN set-up so in the future, but it’s possibly less of a priority on their road maps. Amazon is deeply committed to full automation, which is necessary at their scale. The smaller cloud providers can get away with some degree of manual provisioning for this sort of thing, still — and it should be pretty clear to equipment vendors (and their virtual appliance competitors) that automating this is a public cloud requirement, ensuring that the feature will show up across the industry within a reasonable timeframe.
Think of it this way: Amazon VPC does not isolate any resources for an individual customer’s use. It provides Internet VPN connectivity to a shared resource pool, rather than public Internet connectivity. It’s still the Internet — the same physical cables in Amazon’s data center and across the world, and the same logical Internet infrastructure, just with a Layer 3 IPsec encrypted tunnel on top of it. VPC is “virtual private” in the same sense that “virtual private” is used in VPN, not in the sense of “private cloud”.
Amazon VPC
Today, Amazon announced a new enhancement to its EC2 compute service, called Virtual Private Cloud (VPC). Amazon’s CTO, Werner Vogels, has, as usual, provided some useful thoughts on the release, accompanied by his thoughts on private clouds in general. And as always, the RightScale blog has a lucid explanation.
So what, exactly, is VPC?
VPC offers network isolation to instances (virtual servers) running in Amazon’s EC2 compute cloud. VPC instances do not have any connectivity to the public Internet. Instead, they only have Internet VPN connectivity (specifically, an IPsec VPN tunnel), allowing the instances to seem as if they’re part of the customer’s private network.
For the non-techies among my readers: Think about the way you connect your PC to a corporate VPN when you’re on the road. You’re on the general Internet at the hotel, but you run a VPN client on your laptop that creates a secure, encrypted tunnel over the Internet, between your laptop and your corporate network, so it seems like your laptop is on your corporate network, with an IP address that’s within your company’s internal address range.
That’s basically what’s happening here with VPC — the transport network is still the Internet, but now there’s a secure tunnel that “extends” the corporate network to an external set of devices. The virtual instances get corporate IP addresses (Amazon now even supports DHCP options), and although of course the traffic is still coming through your Internet gateway and you are experiencing Internet performance/latency/availability, devices on your corporate WAN “think” the instances are local.
To set this up, you use new features of the Amazon API that lets you create a VPC container (a logical construct for the concept of your private cloud), subnets, and gateways. When you actually activate the VPN, you begin paying 5 cents an hour to keep the tunnel up. You pay normal Amazon bandwidth charges on top of that (remember, your traffic is still going over the Internet, so the only extra expense to Amazon is the tunnel itself).
When you launch an EC2 instance, you can now specify that it belongs to a particular VPC subnet. A VPC-enabled instance is not physically isolated from the rest of EC2; it’s still part of the general shared pool of capacity. Rather, the virtual privacy is achieved via Amazon’s proprietary networking software, which they use to isolate virtual instances from one another. (It is not intra-VM firewalling per se; Amazon says this is layer 2 network isolation.)
At the moment, an instance can’t be both be part of a VPC and accessible to the general Internet, which means that this doesn’t solve a common use case — the desire to use a private network for back-end administration or data, but still have the server accessible to the Internet so that it can be customer-facing. Expect Amazon to offer this option in the future, though.
As it currently stands, with an EC2 instance with VPC limited to communicating with other instances within the VPC, as well as the corporate network, this solves the use case of customers who are using EC2 for purely internally-facing applications and are seeking a more isolated environment. While some customers are going to want to have genuinely private network connectivity (i.e., the ability to drop an MPLS VPN connection into the data center), a scenario that Amazon is unlikely to support, the VPC offering is likely to serve many needs.
Note, by the way, that the current limitation on communication also means that EC2 instances can’t reach other Amazon Web services, including S3. (However, EBS does work, as far as I know.) While monitoring is supported, load-balancing is not. Thus, auto-scaling functionality, one of the more attractive recent additions to the platform, is limited.
VPN connectivity for cloud servers is not a new thing in general, and part of what Amazon is addressing with this release is a higher-security option, for those customers who are uncomfortable with the fact that Amazon, unlike most of its competitors, does not offer a private VLAN to each customer. For EC2 specifically, there have been software-only approaches, like CohesiveFT’s VPN-Cubed. Other cloud compute service providers have offered VPN options, including GoGrid and SoftLayer. What distinguishes the Amazon offering is that the provisioning is fully automated, and the technology is proprietary.
This is an important step forward for Amazon, and it will probably cause some re-evaluations by prospective customers who previously rejected an Amazon solution because of the lack of connectivity options beyond public Internet only.
Cloud services are evolving with extraordinary rapidity. I always caution customers not to base deployment plans for one year out on the current state of the technology, because every vendor is evolving so rapidly that the feature that’s currently missing and that you really want has, assuming it’s not something wacky and unusual, a pretty high chance of being available when you’re actually ready to start using the service in a year’s time.
Hype cycles
I’ve recently contributed to a couple of our hype cycles.
Gartner’s very first Hype Cycle for Cloud Computing features a whole array of cloud-related technologies and services. One of the most interesting things about this hype cycle, I think, is the sheer number of concepts that we believe will hit the plateau of productivity in just two to five years. For a nascent technology, that’s pretty significant — we’re talking about a significant fundamental shift in the way that IT is delivered, in a very short time frame. However, a lot of the concepts in this hype cycle haven’t yet hit the peak of inflated expectations — you can expect plenty more hype to be coming your way. There’s a good chance that for the IaaS elements that I focus on, the crash down into the trough of disillusionment will be fairly brief and shallow, but I don’t think it can be avoided. Indeed, I can already tell you tales of clients who got caught up in the overhype and got themselves into trouble. But the “try it and see” aspect of cloud IaaS means that expectations and reality can get a much faster re-alignment than it can if you’re, say, spending a year deploying a new technology in your data center. With the cloud, you’re never far from actually being able to try something and see if it fits your needs.
My hype cycle profile for CDNs appears on our Media Industry Content hype cycle, as well as our brand-new TV-focused (digital distribution and monetization of video) Media Broadcasting hype cycle. Due to the deep volume discounts media companies receive from CDNs, the value proposition is and will remain highly compelling, although I do hear plenty of rumblings about both the desire to use excess origin capacity as well as the possibilities that the cloud offers for both delivery and media archival.
I was involved in, but am not a profile author on, the Hype Cycle for Data Center Power and Cooling Technologies. If you are a data center engineering geek, you’ll probably find it to be quite interesting. Ironically, in the midst of all this new technology, a lot of data center architecture and engineering companies still want to build data centers the way they always have — known designs, known costs, little risk to them… only you lose when that happens. (Colocation companies, who have to own and operate these data centers for the long haul, may be more innovative, but not always, especially since many of them don’t design and build themselves, relying on outside expertise for that.)
Cloud IaaS adoption survey
My colleagues and I are planning to field a survey about cloud computing adoption (specifically, infrastructure as a service), both to assess current attitudes towards cloud IaaS as well as ask people about their adoption plans. The target respondents for the survey will be IT buyers.
We have some questions that we know we want to ask (and that we know our clients, both end-users and vendors, are curious about), and some hypotheses that we want to test, but I’ll ask in this open forum, in an effort to try to ensure the survey is maximally useful: What are the cloud-adoption survey questions whose answers would cause you to change your cloud-related decision-making? (You can reply in a comment, send me email, or Twitter @cloudpundit.)
I expect survey data will help vendors alter their tactical priorities and may alter their strategic plans, and it may assist IT buyers in figuring out where they are relative to the “mainstream” plans (useful when talking to cautious business leadership worried about this newfangled cloud thing).
Somewhat peripherally: Following up on earlier confusion, a potshot was taken at the popularity of surveys at large analyst firms. I’ll note that I’m very much a fan of surveys, and if I had infinite budget to work with, I’d probably field a lot more of them. Surveys are (hopefully) not just blind firing of questions into the populace. Intelligent survey design is an art form (as is proper fielding of a survey). Asking the right questions — forming testable hypotheses whose implications are actionable by clients, and getting good information density out of the questions you ask (looking for patterns in the correlations, not just the individual answers) — is incredibly important if you’re going to get something maximally useful out of the money you spent. Data analysis can drive insights that you wouldn’t have otherwise been able to obtain and/or prove.
The Magic Quadrant, Amazon, and confusion
Despite my previous clarifying commentary on the Magic Quadrant for Web Hosting and Cloud Infrastructure Services (On Demand), posted when the MQ was published, and the text of the MQ itself, there continues to be confusion around the positioning of the vendors in the MQ. This is an attempt to clarify, in brief.
This MQ is not a pure cloud computing MQ. It is a hosting MQ. Titling it as such, and making it such, is not some feeble attempt to defend the traditional way of doing things. It is designed to help Gartner’s clients select a Web hoster, and it’s focused upon the things that enterprises care about. Today, our clients consider cloud players as well as traditional players during the selection process. Cloud has been highly disruptive to the hosting industry, introducing a pile of new entrants, revitalizing minor players and lifting them to a new level, and forcing successful traditional players to revise their approach to the business.
The most common question asked by outsiders who just look at the chart and nothing more is, “Why doesn’t Amazon score higher on vision and execution?”
The answer, simply, is that the hosting MQ scores five use cases — self-managed hosting, mainstream (low/mid-end) managed hosting, highly complex managed hosting, global solutions portfolio (ability to provide multiple types of service packages at multiple price points, globally, for very large multi-nationals seeking global hosting options), and enterprise applications hosting. The final rating is a weighted composite of these scores. Amazon scores extremely highly on self-managed hosting, but has a much more limited ability to support the other four scenarios.
Amazon lacks many capabilities that are important in the overall Web hosting market, like managed services, the ability to mix in dedicated equipment (important to anyone who wants to run things that don’t virtualize well, like large-scale Oracle databases, as well as colocate “black box” hardware appliances, like those used for transaction functions for some e-commerce sites), the ability to isolate the environment from the Internet and just use private network connectivity, etc. Their lack of these capabilities hurts their scores. (Note that some capabilities that were missing may have been disclosed to us as part of Amazon’s roadmap, which augmented their Vision score positively, but similarly, stances taken that would definitively shut out some features would be penalized.)
Clearly, we don’t think that Amazon sucks as a cloud provider; it’s just that they don’t play as broadly in the hosting space as the best of the traditional players, although they are certainly a competitor against the traditional players, and a disruptive entrant in general.
The same could be said for many of Amazon’s cloud competitors, although those with some background in traditional hosting may have fewer product-portfolio gaps. Original innovation is a component of Vision but it’s only part of the overall Vision score, so being a fast follower only hurts you so much.
We recognize the need for a “pure cloud compute” vendor rating, and have one in the works.
CloudPundit: Massive-Scale Computing 