Pondering the care and feeding of your multicloud gelatinous cube. (Which engulfs everything in its path, and digests everything organic.)
Most organizations end up multicloud, rather than intending to be multicloud in a deliberate and structured way. So typical tales go like this: The org started doing digital business-related new applications on AWS and now AWS has become the center of gravity for all new cloud-native apps and cloud-related skills. Then the org decided to migrate “boring” LOB Windows-based COTS to the cloud for cost-savings, and lifted-and-shifted them onto Azure (thereby not actually saving money, but that’s a post for another day). Now the org has a data science team that thinks that GCP is unbearably sexy. And there’s a floating island out there of Oracle business applications where OCI is being contemplated. And don’t forget about the division in China, that hosts on Alibaba Cloud…
Multicloud is inevitable in almost all organizations. Cloud IaaS+PaaS spans such a wide swathe of IT functions that it’s impractical and unrealistic to assume that the organization will be single-vendor over the long term. Just like the enterprise tends to have at least three of everything (if not ten of everything), the enterprise is similarly not going to resist the temptation of being multicloud, even if it’s complex and challenging to manage, and significantly increases management costs. It is a rare organization that both has diverse business needs, and can exercise the discipline to use a single provider.
Despite recognizing the giant ooze that we see squelching our way, along with our unavoidable doom, there are things we can do to prepare, govern, and ensure that we retain some of our sanity.
For starters, we can actively choose our multicloud strategy and stance. We can classify providers into tiers, decide what providers are approved for use and under what circumstances, and decide what providers are preferred and/or strategic.
We can then determine the level of support that the organization is going to have for each tier — decide, for instance, that we’ll provide full governance and operations for our primary strategic provider, a lighter-weight approach that leans on an MSP to support our secondary strategic provider, and less support (or no support beyond basic risk management) for other providers.
After that, we can build an explicit workload placement policy that has an algorithm that guides application owners/architects in deciding where particular applications live, based on integration affinities, good technical fit, etc.
Note that cost-based provider selection and cost-based long-term workload placement are both terrible ideas. This is a constant fight between cloud architects and procurement managers. It is rooted in the erroneous idea that IaaS is a commodity, and that provider pricing advantages are long-term rather than short-lived. Using cost-based placement often leads to higher long-term TCO, not to mention a grand mess with data gravity and thus data management, and fragile application integrations.
See my new research note, “Comparing Cloud Workload Placement Strategies” (Gartner paywall) for a guide to multicloud IaaS / IaaS+PaaS strategies (including when you should pursue a single-cloud approach). In a few weeks, you’ll see the follow-up doc “Designing a Cloud Workload Placement Policy” publish, which provides a guide to writing such policies, with an analysis of different placement factors and their priorities.
A nontrivial chunk of my client conversations are centered on the topic of cloud IaaS/PaaS self-service, and how to deal with development teams (and other technical end-user teams, i.e. data scientists, researchers, hardware engineers, etc.) that use these services. These teams, and the individuals within those teams, often have different levels of competence with the clouds, operations, security, etc. but pretty much all of them want unfettered access.
Responsible governance requires appropriate guidelines (policies) and guardrails, and some managers and architects feel that there should be one universal policy, and everyone — from the highly competent digital business team, to the data scientists with a bit of ad-hoc infrastructure knowledge — should be treated identically for the sake of “fairness”. This tends to be a point of particular sensitivity if there are numerous application development teams with similar needs, but different levels of cloud competence. In these situations, applying a single approach is deadly — either for agility or your crisis-induced ulcer.
Creating a structured, tiered approach, with different levels of self-service and associated governance guidelines and guardrails, is the most flexible approach. Furthermore, teams that deploy primarily using a CI/CD pipeline have different needs from teams working manually in the cloud provider portal, which in turn are different from teams that would benefit from having an easy-vend template that gets provisioned out of a ServiceNow request.
The degree to which each team can reasonably create its own configurations is related to the team’s competence with cloud solution architecture, cloud engineering, and cloud security. Not every person on the team may have a high level of competence; in fact, that will generally not be the case. However, the very least, for full self-service there needs to be at least one person with strong competencies in each of those areas, who has oversight responsibilities, acts an expert (provides assistance/mentorship within the team), and does any necessary code review.
If you use CI/CD, you also want automation of such review in your pipeline, that includes your infrastructure-as-code (IaC) and cloud configs, not just the app code; i.e. a tool like Concourse Labs). Even if your whole pipeline isn’t automated, review of IaC during the dev stage, and not just when it triggers a cloud security posture management tool (like Palo Alto’s Prisma Cloud or Turbot), whether in dev, test, or production.
Who determines “competence”? To avoid nasty internal politics, it’s best to set this standard objectively. Certifications are a reasonable approach, but if your org isn’t the sort that tends to pay for internal certifications or the external certifications (AWS/Azure Solution Architect, DevOps Engineer, Security Engineer, etc.) seem like too high a bar, you can develop an internal training course and certification. It’s not a bad idea for all of your coders (whether app developers, data scientists, etc.) that use the cloud to get some formal training on creating good and secure cloud configurations, anyway.
(For Gartner clients: I’m happy to have a deeper discussion in inquiry. And yes, a formal research note on this is currently going through our editing process and will be published soon.)
Building cloud expertise is hard. Building multicloud expertise is even harder. By “multicloud” in this context, I mean “adopting, within your organization, multiple cloud providers that do something similar” (such as adopting both AWS and Azure).
Integrated IaaS+PaaS providers are complex and differentiated entities, in both technical and business aspects. Add in their respective ecosystems — and the way that “multicloud” vendors, managed service providers (MSPs) etc. often deliver subtly (or obviously) different capabilities on different cloud providers — and you can basically end up with a multicloud katamari that picks up whatever capabilities it randomly rolls over. You can’t treat them like commodities (a topic I cover extensively in my research note on Managing Vendor Lock-In in Cloud IaaS).
For this reason, cloud-successful organizations that build a Cloud Center of Excellence (CCOE), or even just try to wrap their arms around some degree of formalized cloud operations and governance, almost always start by implementing a single cloud provider but plan for a multicloud future.
Successfully multicloud organizations have cloud architects that deeply educate themselves on a single provider, and their cloud team initially builds tools and processes around a single provider — but the cloud architects and engineers also develop some basic understanding of at least one additional provider in order to be able to make more informed decisions. Some basic groundwork is laid for a multicloud future, often in the form of frameworks, but the actual initial implementation is single-cloud.
Governance and support for a second strategic cloud provider is added at a later date, and might not necessarily be at the same level of depth as the primary strategic provider. Scenario-specific (use-case-specific or tactical) providers are handled on a case-by-case basis; the level of governance and support for such a provider may be quite limited, or may not be supported through central IT at all.
Individual cloud engineers may continue to have single-cloud rather than multicloud skills, especially because being highly expert in multiple cloud providers tend to boost market-rate salaries to levels that many enterprises and mid-market businesses consider untenable. (Forget using training-cost payback as a way to retain people; good cloud engineers can easily get a signing bonus more than large enough to deal with that.)
In other words: while more than 80% of organizations are multicloud, very few of them consider their multiple providers to be co-equal.
What sort of org structures work well for helping to drive successful cloud adoption? Every day I talk to businesses and public-sector entities about this topic. Some have been successful. Others are struggling. And the late-adopters are just starting out and want to get it right from the start.
Back in 2014, I started giving conference talks about an emerging industry best practice — the “Cloud Center of Excellence” (CCOE) concept. I published a research note at the start of 2019 distilling a whole bunch of advice on how to build a CCOE, and I’ve spent a significant chunk of the last year and a half talking to customers about it. Now I’ve revised that research, turning it into a hefty two-part note on How to Build a Cloud Center of Excellence: part 1 (organizational design) and part 2 (Year 1 tasks).
Gartner’s approach to the CCOE is fundamentally one that is rooted in the discipline of enterprise architecture and the role of EA in driving business success through the adoption of innovative technologies. We advocate a CCOE based on three core pillars — governance (cost management, risk management, etc.), brokerage (solution architecture and vendor management), and community (driving organizational collaboration, knowledge-sharing, and cloud best practices surfaced organically).
Note that it is vital for the CCOE to be focused on governance rather than on control. Organizations who remain focused on control are less likely to deliver effective self-service, or fully unlock key cloud benefits such as agility, flexibility and access to innovation. Indeed, IT organizations that attempt to tighten their grip on cloud control often face rebellion from the business that actually decreases the power of the CIO and the IT organization.
Also importantly, we do not think that the single-vendor CCOE approaches (which are currently heavily advocated by the professional services organizations of the hyperscalers) are the right long-term solution for most customers. A CCOE should ideally be vendor-neutral and span IaaS, PaaS, and SaaS in a multicloud world, with a focus on finding the right solutions to business problems (which may be cloud or noncloud). And a CCOE is not an IaaS/PaaS operations organization — cloud engineering/operations is a separate set of organizational decisions (I’ll have a research note out on that soon, too).
Please dive into the research (Gartner paywall) if you are interested in reading all the details. I have discussed this topic with literally thousands of clients over the last half-dozen years. If you’re a Gartner for Technical Professionals client, I’d be happy to talk to you about your own unique situation.
Preface added 20 November 2020: This post received a lot more attention than I expected. I must reiterate that it is not in any way an endorsement. Indeed, sparkly pink unicorns are, by their nature, fanciful. Caution must be exercised, as sparkly pink glitter can conceal deficiencies in the equine body.
Digging into my archive of past predictions… In a research note on the convergence of public and private cloud, published almost exactly eight years ago in July 2012, I predicted that the cloud IaaS market would eventually deliver a service that delivered a full public cloud experience as if it were private cloud — at the customer’s choice of data center, in a fully single-tenant fashion.
Since that time, there have been many attempts to introduce public-cloud-consistent private cloud offerings. Gartner now has a term, “distributed cloud”, to refer to the on-premises and edge services delivered by public cloud providers. AWS Outposts deliver, as a service, a subset of AWS’s incredibly rich product porfolio. Azure Stack (now Azure Stack Hub) delivers, as software, a set of “Azure-consistent” capabilities (meaning you can transfer your scripts, tooling, conceptual models, etc., but it only supports a core set of mostly infrastructure capabilities). Various cloud MSPs, notably Avanade, will deliver Azure Stack as a managed service. And folks like IBM and Google want you to take their container platform software to facilitate a hybrid IT model.
But no one has previously delivered what I think is what customers really want:
- Location of the customer’s choice
- Single-tenant; no other customer shares the hardware/service; data guaranteed to stay within the environment
- Isolated control plane and private self-service interfaces (portal, API endpoints); no tethering or dependence on the public cloud control plane, or Internet exposure of the self-service interfaces
- Delivered as a service with the same pricing model as the public cloud services; not significantly more expensive than public cloud as long as minimum commitment is met
- All of the provider’s services (IaaS+PaaS), identical to the way that they are exposed in the provider’s public cloud regions
Why do customers want that? Because customers like everything the public cloud has to offer — all the things, IaaS and PaaS — but there are still plenty of customers who want it on-premises and dedicated to them. They might need it somewhere that public cloud regions generally don’t live and may never live (small countries, small cities, edge locations, etc.), they might have regulatory requirements they believe they can only meet through isolation, they may have security (even “national security”) requirements that demand isolation, or they may have concerns about the potential to be cut off from the rest of the world (as the result of sanctions, for instance). And because when customers describe what they want, they inevitably ask for sparkly pink unicorns, they also want all that to be as cheap as a multi-tenant solution.
And now it’s here, and given that it’s 2020… the sparkly pink unicorn comes from Oracle. Specifically, the world now has Oracle Dedicated Regions Cloud @ Customer. (Which I’m going to shorthand as OCI-DR, even though you can buy Oracle SaaS hosted on this infrastructure) OCI’s region model, unlike its competitors, has always been all-services-in-all-regions, so the OCI-DR model continues that consistency.
In an OCI-DR deal, the customer basically provides colo (either their own data center or a third party colo) to Oracle, and Oracle delivers the same SLAs as it does in OCI public cloud. The commit is very modest — it’s $6 million a year, for a 3-year minimum, per OCI-DR Availability Zone (a region can have multiple AZs, and you can also buy multiple regions). There are plenty of cloud customers that easily meet that threshold. (The typical deal size we see for AWS contracts at Gartner is in the $5 to $15 million/year range, on 3+ year commitments.) And the pricing model and actual price for OCI-DR services is identical to OCI’s public regions.
The one common pink sparkly desire that OCI doesn’t meet is the ability to use your own hardware, which can help customers address capex vs. opex desires, may have perceived cost advantages, and may address secure supply chain requirements. OCI-DR uses some Oracle custom hardware, and the hardware is bundled as part of the service.
I predict that this will raise OCI’s profile as an alternative to the big hyperscalers, among enterprise customers and even among digital-native customers. Prior to today’s announcement, I’d already talked to Gartner clients who had been seriously engaged in sales discussions on OCI-DR; Oracle has quietly been actively engaged in selling this for some time. Oracle has made significant strides (surprisingly so) in expanding OCI’s capabilities over this last year, so when they say “all services” that’s now a pretty significant portfolio — likely enough for more customers to give OCI a serious look and decide whether access to private regions is worth dealing with the drawbacks (OCI’s more limited ecosystem and third-party tool support probably first and foremost).
As always, I’m happy to talk to Gartner clients who are interested in a deeper discussion. We’ve recently finished our Solution Scorecards (an in-depth assessment of 270 IaaS+PaaS capabilities), including our new assessment of OCI. The scores are summarized in a publicly-reprinted document. The full scorecard has been published, and the publicly-available summary says, “OCI’s overall solution score is 62 out of 100, making it a scenario-specific option for technical professionals responsible for cloud production deployments.”
We’ve just completed our 2019 evaluations of cloud IaaS providers, resulting in a new Magic Quadrant, Critical Capabilities, and six Solution Scorecards — one for each of the providers included in the Magic Quadrant. This process has also resulted in fresh benchmarking data within Gartner’s Cloud Decisions tool, a SaaS offering available to Gartner for Technical Professionals clients, which contains benchmarks and monitoring results for many cloud providers.
As part of this, we are pleased to introduce Gartner’s new Solution Scorecards, an updated document format for what we used to call In-Depth Assessments. Solution Scorecards assess an individual vendor solution against our recently-revised Solution Criteria (formerly branded Evaluation Criteria). They are highly detailed documents — typically 60 pages or so, assessing 265 individual capabilities as well as providing broader recommendations to Gartner clients.
The criteria are always divided into Required, Preferred, and Optional categories — essentially, things that everyone wants (and where they need to compensate/risk-mitigate if something is missing), things that most people want but can live without or work around readily, and things that are use case-specific. The Required, Preferred, and Optional criteria are weighted into a 4:2:1 ratio in order to calculate an overall Solution Score.
If you are a Gartner for Technical Professionals client, the scorecards are available to you today. You can access them from the links below (Gartner paywall):
- Amazon Web Services
- Microsoft Azure
- Google Cloud Platform
- IBM Cloud
- Oracle Cloud Infrastructure
- Alibaba Cloud International (English-language offerings available outside of China)
We will be providing a comparison of these vendors and their Solution Scorecards at the annual “Cloud Wars” presentation at the Gartner Catalyst conference — one of numerous great reasons to come to San Diego the week of August 11th (or Catalyst UK in London the week of September 15th)! Catalyst has tons of great content for cloud architects and other technical professionals involved in implementing cloud computing.
Note that we are specifically assessing just the integrated IaaS+PaaS offerings — everything offered through a single integrated self-service experience and on a single contract. Also, only cloud services count; capabilities offered as software, hosting, or a human-managed service do not count. Capabilities also have to be first-party.
Also note that this is not a full evaluation of a cloud provider’s entire portfolio. The scorecards have “IaaS” in the title, and the scope is specified clearly in the Solution Criteria. For the details of which specific provider services or products were or were not evaluated, please refer to each specific Scorecard document.
All the scores are current as of the end of March, and count only generally-available (GA) capabilities. Because it takes weeks to work with vendors for them to review and ensure accuracy, and time to edit and publish, some capabilities will have gone beta or GA since that time; because we only score what we’re able to test, the evaluation period has a cut-off date. After that, we update the document text for accuracy but we don’t change the numerical scores. We expect to update the Solution Scorecards approximately every 6 months, and working to increase our cadence for evaluation updates.
This year’s scores vs. last year’s
When you review the scores, you’ll see that broadly, the scores are lower than they were in 2018, even though all the providers have improved their capabilities. There are several reasons why the 2019 scores are lower than in previous years. (For a full explanation of the revision of the Solution Criteria in 2019, see the related blog post.)
First, for many feature-sets, several Required criteria were consolidated into a single multi-part criterion with “table stakes” functionality; missing any part of that criterion caused the vendor to receive a “No” score for that criterion (“Yes” is 1 point; “No” is zero points; there is no partial credit). The scorecard text explains how the vendor does or does not meet each portion of a criterion. The text also mentions if there is beta functionality, or if a feature was introduced after the evaluation period.
Second, many criteria that were Preferred in 2018 were promoted to Required in 2019, due to increasing customer expectations. Similarly, many criteria that were Optional in 2018 are now Preferred. We introduced some brand-new criteria to all three categories as well, but providers that might have done well primarily on table-stakes Required functionality in previous years may have scored lower this year due to the increased customer expectations reflected by revised and new criteria.
Customizing the scores
The solution criteria, with all of the criteria detail, is available to all Gartner for Technical Professionals clients, and comes with a spreadsheet that allows you to score any provider yourself; we also provide a filled-out spreadsheet with each Solution Scorecard so you can adapt the evaluation for your own needs. The Solution Scorecards are similarly transparent on which parts of a criterion are or aren’t met, and we link to documentation that provides evidence for each point (in some cases Gartner was provided with NDA information, in which case we tell you how to get that info from the provider).
This allows you to customize the scores as you see fit. Thus, if you decide that getting 3 out of 4 elements of a criteria is good enough for you, or you think that the thing they miss isn’t relevant to you, or you want to give the provider credit for newly-released capabilities, or you want to do region-specific scoring, you can modify the spreadsheet accordingly.
If you’re a Gartner client and are interested in discussing the solution criteria, assessment process, and the cloud providers, please schedule an inquiry or a 1-on-1 at Catalyst. We’d be happy to talk to you!
In February of this year, we revised the Evaluation Criteria for Cloud IaaS (Gartner paywall). The evaluation criteria (now rebranded Solution Criteria) are essentially the sort of criteria that prospective customers typically include in RFPs. They are highly detailed technical criteria, along with some objectively-verifiable business capabilities (such as elements in a technical support program, enterprise ISV partnerships, ability to support particular compliance requirements, etc.).
The Solution Criteria are intended to help cloud architects evaluate cloud IaaS providers (and integrated IaaS+PaaS providers such as the hyperscale cloud providers), whether public or private, or assess their own internal private cloud. We are about to publish Solution Scorecards (formerly branded In-Depth Assessments) for multiple providers; Gartner analysts assess these solutions hands-on and determine whether or not they have capabilities that meet the requirements of a criterion.
The TL;DR version
In summary, we revised the Solution Criteria extensively in 2019, and the results were as follows:
- The criteria have been updated to reflect the current IaaS+PaaS market.
- Expectations are significantly higher than in previous years.
- Expectations have been aligned to other Gartner research, taking into account customer wants and needs in the relevant market, not just in a cloud-specific context.
- Many capabilities have been consolidated and are now required.
- Most vendor scores in the Solution Scorecards have dropped dramatically since last year, and there is a much broader spread of vendor scores.
The Evolution of Customer Demands
The Evaluation Criteria (EC) for Cloud IaaS was first published in 2012. It received a significant update every other year (each even-numbered year) thereafter. When first written, the EC reflected the concerns of our clients at the time, many of whom were infrastructure and operations (I&O) professionals with VMware backgrounds. With each iteration, the EC evolved significantly, yet incrementally.
In the meantime, the market moved extremely quickly. The market evolution towards cloud integrated IaaS and PaaS (IaaS+PaaS) providers, and the market exit (or strategic de-investment) of many of the “commodity” providers, radically changed the structure and nature of the market over time. Cloud IaaS providers weren’t just expected to provide “hardware infrastructure”, but also “software infrastructure”, including all of the necessary management and automation. This essentially forced these providers into introducing services that compete in many IT markets and in an extraordinary number of software niches.
Furthermore, as the market matured, the roles and expectations of our clients also evolved significantly. The focus shifted to enterprise-wide initiatives, rather than project-based adoption. Digital business transformation elevated the importance of cloud-native workloads, while IT transformation emphasized the need for high-quality cloud migration of existing workloads. The notion that a cloud IaaS provider could successfully run all, or almost all, of a customer’s IT became part of the assumptions that needed to underpin the provider evaluation process.
Today’s cloud IaaS customers have high expectations. Experienced customers are becoming more sophisticated, but late adopters also have high expectations of a provider that have to be met to help the customer overcome barriers to adoption.
For 2019, we decided to take a look at the EC“from scratch”, in order to try to construct a list of criteria that are the most relevant to the initiatives of customers today. In many cases, our clients are trying to pick a primary strategic IaaS provider. In other cases, our clients already have a primary provider but are trying to pick a strategic secondary provider as they implement a multicloud strategy. Finally, some of our clients are choosing a provider for a tactical need, but still need to understand that provider’s capabilities in detail.
Constructing the Revision
The revision needed to keep a similar number of criteria (in order to keep the assessment time manageable and the assessment itself at a readable length) — we ended up with 265 for 2019.
In order to keep the total number of criteria down, we needed to consolidate closely-related criteria into a single criterion. Many criteria became multi-part as a result. We tried to consolidate the “table stakes” functionality that could be assumed to be a part of all (or almost all) cloud IaaS offerings, in order to make room for more differentiated capabilities.
We tried to be as vendor-neutral as possible. The evaluation criteria have evolved since the initial 2012 introduction; when we introduced new criteria in the past, we often ended up with criteria requirements that closely mirrored the feature-set of the first provider to offer a capability, since that provider shaped customer expectations. In this 2019 revision, we tried to go back to the core customer requirements, without concern as to whether cloud provider implementations fully aligned with those requirements — the criteria are intended to reflect what customers want and not what vendors offer. There are requirements that no vendors meet, but which we often hear our clients ask for; in such cases we tried to phrase those requirements in ways that are reasonable and implementable at scale, as it’s okay for the criteria to be somewhat aspirational for the market.
We tried to make sure that the criteria were worded using standard Gartner terms or general market terminology, avoiding vendor-specific terms. (Note that because vendors not-infrequently adopt Gartner terms, there were cases where providers had adopted terminology from earlier versions of EC, and we made no attempt to alter such terms.)
We tried to keep to requirements, without dictating implementation, where possible. However, we had to keep in mind that in cloud IaaS, where there are customers who want fine-grained visibility and control over the infrastructure, there still must be implementation specificity when the customer explicitly wants those elements exposed.
Defining the Criteria
During the process of determining the criteria, we sought input broadly within Gartner, both in terms of discussing the criteria with other analysts as well as incorporating things from existing Gartner written research. (And the criteria reflect, as much as possible, the discussions we’ve had with clients about what they’re looking for, and what they’re putting into their RFPs.)
In some cases, we needed input from specialists in a topic. In some areas of technology, clients who need to have deep-dive discussions on features may talk almost exclusively to analysts specialized in those areas. Those analysts are familiar with current requirements as well as the future of those technology areas, and are thus the best source for determining those needs. For example, areas such as machine learning and IoT are primarily covered by analysts with those specializations, even when the customers are implementing cloud solutions. There are also areas, such as Security, where we have detailed cloud recommendations from those teams. So we extensively incorporated their input..
We also looked at non-cloud capabilities when there were market gaps relative to customer desires. There are areas where either cloud providers do not currently have capabilities, or where those capabilities are relatively nascent. Thus, we needed to identify where customers are using on-premises solutions, and want cloud solutions. We also needed to determine what the “minimum viable product” should be for the purposes of constructing a criterion around it.
Feedback from non-cloud analysts was also important because it identified areas where clients were not using a cloud solution because of something that was missing. In many cases, these were not technology features, but issues around transparency, or the lack of solutions acceptable on a global basis.
Finally, the way that customers source solutions, build applications, and manage their data is changing. We tried to ensure that the new criteria aligned with these trends.
Because more and more of our clients are deploying cloud solutions globally, every criterion also had some requirements as to its global availability. These are used only for advisory purposes and are not part of scoring.
The vendors were allowed to give feedback on the criteria prior to publication. We wanted to check if the criteria were reasonable, and seemed fair. We incorporated feedback that constituted good, vendor-neutral suggestions that aligned to customer requirements.
The End Results
When you see the Solution Scorecards, you may be surprised by lower scores on the part of many of the providers. We’re being transparent about the Evaluation Criteria (Solution Criteria) revision in order to help you understand why the scores are lower.
The lower scores were an unintentional side-effect of the revision, but reflect, to some degree, the state of the market relative to the very high expectations of customers. Note that this year’s lower scores do not indicate that providers have “gone backwards” or removed capabilities; they just reflect the provider’s status against a raised bar of customer expectations.
We expect that when we update the scorecards in the second half of this year, scores will increase, as many of the vendors have since introduced missing capabilities, or will do so by the next update. We retain confidence that the solution criteria are a good reflection of a broad range of current customer expectations. Because many vendors are doing a good job of listening to what customers and prospects want, and planning accordingly, we think that the solution criteria will also be reflected in future vendor roadmaps and market development.
We discuss the Solution Scorecards and scores in a separate blog post.
The Critical Capabilities for Public Cloud IaaS, 2016 has now been published. The Critical Capabilities is a technical assessment of public cloud IaaS offerings against a set of use cases — cloud-native applications, general business applications, application development environments, batch computing, and (new for 2016) the Internet of Things. It’s part of our integrated series of cloud IaaS assessments and complements our Magic Quadrant for Cloud IaaS (Gartner clients: see interactive version).
We are now launching right back into the Magic Quadrant cycle for 2017, with the goal of publishing a new Magic Quadrant in April 2017, and a new Critical Capabilities shortly thereafter.
A lot has happened since the early-2016 research process for our 2016 Magic Quadrant and Critical Capabilities cycle for this market. Multiple providers have launched new offerings and are phasing out their previous offerings, and there are some important new market entrants. We want to make sure that our research notes offer current representations of provider capabilities. (Usefully, a shift to April publication also gets us back to a schedule that aligns with our infrastructure & operations conference season.)
In previous years, we’ve issued an open invitation for the pre-qualification survey to all cloud IaaS providers. This year, we are not doing so; instead, we have issued invitations only to providers who we believe are highly likely to qualify.
If you are a cloud IaaS provider that did not receive an invitation, but you believe you are highly likely to qualify for inclusion, please email me at Lydia dot Leong at Gartner dot com to discuss it.
Oracle has made multiple previous attempts to enter the cloud IaaS market — most recently (early this year), with the Oracle Compute Cloud. At Oracle OpenWorld this week, however, Oracle announced a brand-new cloud IaaS offering. Oracle hasn’t officially given this a real brand yet, so for the purposes of this blog post, I’ll call it their next-gen cloud.
News of this project leaked last year. Oracle has paid richly to hire an “A” team, so to speak — former long-time senior AWS engineers lead the project, and they’ve recruited heavily from all three hyperscale clud providers in Seattle (AWS, Microsoft Azure, Google Cloud Platform). These are credible product and engineering people who, in my opinion, understand what they need to build and the enormous challenges ahead of them.
The next-gen cloud currently consists of an SDN (capable of both Layer 2 and Layer 3 networking, which is a differentiator), block storage, object storage, and bare-metal servers (thus the initial moniker, “Oracle Bare Metal Cloud”). Virtual machines (VMs) are coming later this year, with containers to follow early next year. Based on a detailed engineering briefing that Oracle provided to myself and my colleagues, I would say that smart and scalable choices seem to have been made throughout. However, I would characterize this early offering as minimum viable product; it is the foundation of a future competitive offering, rather than a competitive offering today.
In the near term, Oracle’s next-gen cloud will be interesting primarily to a general audience in a bare-metal context. Here, Oracle will compete with Packet, and to some lesser degree, the bare-metal cloud offerings from CenturyLink and Rackspace (OnMetal). It is a true software-defined cloud IaaS offering, provisioned in minutes and billed by the hour. This sets it apart from more hosting-like bare-metal offerings such as IBM SoftLayer, Internap, and Cogeco Peer 1.
It is unlikely that Oracle’s announced price-point — 20% below AWS list prices — will be sufficient to move the needle in a market where AWS’s “real” prices are lowered up to 70% by reserved instances (plus AWS negotiates custom discounts), and where Google is already competing intensively on price (especially on negotiated deals) and has an offering substantially more featureful than what Oracle will have in the market in the next year. Good price-performance is table stakes here. This is not a commodity market; providers compete on their capabilities. This is also not about capital investment to build data centers; Oracle can use colocation until they reach a scale where building makes sense, though since such projects can take years, they’ll need to time that properly.
Bare metal, of course, significantly outperforms VMs in some cases — especially high I/O use cases. But bare metal should be thought of as part of a complete offering — a compute option for some of a customer’s workloads. Price-performance should always be considered in the context of the customer’s specific architecture. In the case of Oracle, bare metal and the layer 2 SDN features are important because they are needed for Oracle RAC and for better performance of Oracle application software. Oracle has built the core of their offering around off-box virtualization of networking and storage, which is important for allowing their cloud IaaS offering to smoothly interoperate with other Oracle hardware placed into the same environment, like Exadata appliances.
Overall, this should be seen as a positive move for Oracle, but one with many open questions about its future. As always, if anyone has more detailed questions, I am happy to answer them in the context of client inquiry, and I’ve set aside some time to speak with reporters during this OpenWorld week.
We’re pleased to announce that the 2016 Magic Quadrant for Cloud Infrastructure, Worldwide has been published. (Link requires a Gartner subscription. If you’re not a Gartner client, there are free reprints available through vendors, and various press articles, such as the Tech Republic analysis. Note that press articles do not always accurately reflect our opinions, though.)
Producing the Magic Quadrant is a huge team effort that involves many people across Gartner, including many analysts who aren’t credited as co-authors, administrative support staff, and people in our primary-research and benchmarking groups. The team effort also reflects the way that we produce an entire body of IaaS research as an integrated effort across Gartner’s research divisions. (The approach described below is specific to our IaaS research and may not apply to Gartner’s assessments in other markets.)
Whether you already have a cloud IaaS provider and are just looking for a competitive check-up, you’re thinking of adding one or more additional providers, or you’re just getting started with cloud IaaS, our work can help you find the providers that are right for you.
The TL;DR list of assessments:
- Magic Quadrant (market and technical evaluation)
- Evaluation Criteria (230+ technical and service traits to look for in a provider)
- In-Depth Assessments (detailed assessments of specific providers against the Evaluation Criteria)
- Critical Capabilities (use-case-based technical evaluations; 2016 update coming soon)
- CloudHarmony and Tech Planner Cloud Module (real-world stats and cost-performance comparisons)
- Peer Insights (IT leaders review providers)
(Note that not all of these might be available as part of your current Gartner client subscription.)
Gartner has produced a Magic Quadrant for Cloud IaaS since 2011. The MQ is our overall perspective on the market, looking at the provider solutions from both a technical and business angle. Gartner clients can use the interactive MQ tool to change the weightings of the criteria to suit their own evaluation priorities (if you read the detailed criteria descriptions, there’s an explanation of how each criterion maps to buyer priorities). The interactive MQ can also be used to get a multi-year historical perspective.
The MQ covers public, hosted private, and industrialized outsourced private cloud IaaS; it’s not just a public cloud MQ. We look at multi-tenant and single-tenant, located in either provider or customer premises, cloud IaaS offerings. We also look at the full range of compute options (VMs, bare-metal servers, containers) that are delivered in a cloud model (API-provisionable via automation, and metered by the hour or less), not just VMs. In addition, we consider some integrated PaaS-layer services (we call these cloud software infrastructure services, which include things like database as a service), but we have a separate enterprise application PaaS MQ for pure aPaaS. While we consider the provider’s overall value proposition in the context of cloud IaaS (including their ability to deliver managed services, network services, etc.), this isn’t a general cloud computing or outsourcing MQ.
2016 marks our sixth iteration of a pure cloud IaaS MQ. Previously, in 2009 and 2010, we included cloud IaaS in our hosting MQ, but by 2010, it was already clear that the hosting and cloud IaaS buyer wants and needs were distinctly different. Since 2011, we’ve produced a global cloud IaaS MQ, along with three regional hosting MQs (suitable for customers looking for dedicated servers or managed hosting on a monthly or annual basis), and three regional data center outsourcing MQs (which include customized private cloud services as part of a broader portfolio of infrastructure outsourcing capabiities). Not every infrastructure need can or should be met with cloud IaaS.
The core foundation of our assessment is our Evaluation Criteria for Cloud IaaS. Over the years, we’ve converged the technical-detail questionnaire that we ask providers to fill out during the Magic Quadrant research process, with the Gartner for Technical Professionals (GTP) document that we produce to guide buyers on evaluating providers. This has resulted in nearly 250 service traits that the Evaluation Criteria document categorizes as Required (almost all Gartner clients are likely to want these things and these have the potential to be showstoppers if missing), Preferred (many will want these things), and Optional (use-case-specific needs). This gives us a consistent set of formal definitions for service features — things you can put a clear yes/no to. As a buyer, you can use the Evaluation Criteria to score any cloud IaaS provider — and even score your own IT department’s private cloud.
In the course of doing this particular Magic Quadrant, providers fill out very detailed questionnaires that list these service features and capabilities (broken down even more granularly than in the Evaluation Criteria), indicating whether their service has those traits, and they’re also asked to provide evidence, like documentation. We also ask them to provide other information like the location of their data centers, languages supported across various aspects of service delivery (like portal localization and tech-support languages spoken), a copy of their standard contract and SLAs, and so forth. We score those questionnaires (and check service features against documentation, and with hands-on testing if need be). We also score things like the buyer-friendliness of contracts, based on the presence/absence of particular clauses. Those component scores are used in many different individual scoring categories within the Magic Quadrant.
We also produce a set of In-Depth Assessments for the providers that our clients are most interested in evaluating. The In-Depth Assessments are detailed documents that score an individual provider against the Evaluation Criteria; for every criteria, we explain how the provider does and doesn’t meet it, and we provide links to the corresponding documentation or other evidence. The results of our hands-on testing are noted, as well. For many buyers, this minimizes the need to conduct an RFP that dives into the technical solution; here we’ve done a very detailed fact-based analysis for you, and the provider has verified the accuracy of the information. (Buyer beware, though: Providers sometimes produce something that looks like one of these assessments, even quoting the Gartner definitions, but with their own more generous self-assessment rather than the stringent Gartner-produced assessment!)
Then, we produce Critical Capabilities for Public Cloud IaaS (2016 update still in progress). This technical assessment looks at a single public cloud IaaS offering from each of the providers included in the Magic Quadrant. The same technical traits used in the other assessments are used here, but they are divided into categories of capabilities, and those capabilities are weighted in a set of common use cases. You can also customize your own set of weightings. In addition to providing quantitative scores, we summarize, in a fair amount of detail, the technical capabilities of each evaluated provider. This allows you to get a sense of what providers are likely to be right for your needs, without having to go through the full deep-dive of reading the In-Depth Assessments. (Critical Capabilities are also available to all Gartner clients and reprints may be offered by providers on their websites, whereas the In-Depth Assessments are only available to GTP clients.)
Performance, and price-performance, is important to many buyers. Gartner provides hardware benchmarking via a SaaS offering called Tech Planner. We offer a Cloud Module within Tech Planner that uses technology that we derived from our acquisition of CloudHarmony. We conduct continuous automated testing on many cloud IaaS providers, including all providers in the Magic Quadrant. We benchmark compute performance for the full range of VMs and bare-metal cloud servers offered by the provider, along with storage performance and network performance; we use this to calculate price-performance metrics. We monitor the availability of their services across the globe. We track provisioning times. All this data is used as objective components to the scores within the Magic Quadrant. Much of this data is directly available to Tech Planner customers, who can use these tools to calculate performance-equivalencies as well as determine where workloads will be most cost-effective.
Finally, we collect end-user reviews of cloud IaaS providers, called Peer Insights. IT leaders (who do not need to be Gartner clients) can submit reviews of their providers; we verify that reviews are legitimate, and it’s one of the very few places where you’ll see enter senior IT executives and architects writing detailed reviews of their providers. We use this data, along with vendor-provided customer references, and the many thousands of clients conversations we have each year with cloud IaaS buyers, as part of the fact base for our Magic Quadrant scoring.
More than a dozen analysts are directly involved in all of these assessments, and many more analysts provide peer-review input into those assessments. It’s an enormous effort, involving a great deal of teamwork, to produce this body of interlinked research. We’re always trying to improve its quality, so we welcome your feedback!
You can DM me on Twitter at @cloudpundit or send email to lydia dot leong at gartner.com.