The process of a Magic Quadrant

There’ve been a number of ongoing dialogues on Twitter, on Quora, and various people’s blogs about the Magic Quadrant. I thought, however, that it would be helpful to talk about the process of doing a Magic Quadrant, before I write another blog post that explains how we came to our current cloud-and-hosting MQ. We lay this process out formally in the initiation letters sent to vendors when we start MQ research, so I’m not giving away any secrets here, just exposing a process that’s probably not well known to people outside of analyst relations roles.

A Magic Quadrant starts its life with a market definition and inclusion criteria. It’s proposed to our group of chief analysts (each sector we cover, like Software, has a chief), who are in charge of determining what markets are MQ-worthy, whether or not the market is defined in a reasonable way, and so forth. In other words, analysts can’t decide to arbitrarily write an MQ, and there’s oversight and a planning process, and an editorial calendar that lays out MQ publication schedules for the entire year.

The next thing that you do is to decide your evaluation criteria, and the weights for these criteria — in other words, how you are going to quantitatively score the MQ. These go out to the vendors near the beginning of the MQ process (usually about 3 months before the target publication date), and are also usually published well in advance in a research note describing the criteria in detail. (We didn’t do a separate criteria note for this past MQ for the simple reason that we were much too busy to do the writing necessary.) Gartner’s policy is to make analysts decide these things in advance for fairness — deciding your criteria and their weighting in advance makes it clear to vendors (hopefully) that you didn’t jigger things around to favor anyone.

In general, when you’re doing an MQ in a market, you are expected to already know the vendors well. The research process is useful for gathering metrics, letting the vendors tell you about small things that they might not have thought to brief you on previously, and getting the summary briefing of what the vendor thought were important business changes in the last year. Vendors get an hour to tell you what they think you need to know. We contact three to five reference customers provided by the vendor, but we also rely heavily upon what we’ve heard from our own clients. There should generally not be any surprises involved for either the analysts or the vendors, assuming that the vendors have done a decent job of analyst relations.

Client status and whatnot doesn’t make any difference whatsoever on the MQ. (Gartner gets 80% of its revenue from IT buyers who rely on us to be neutral evaluators. Nothing a vendor could pay us would ever be worth risking that revenue stream.) However, it generally helps vendors if they’ve been more transparent with us, over the previous year. That doesn’t require a client relationship, although I suspect most vendors are more comfortable being transparent if they have an NDA in place with us and can discuss these things in inquiry, rather than in the non-NDA context of a briefing (though we always keep things confidential if asked to). Ongoing contact tends to mean that we’re more likely to understand not just what a vendor has done, but why they’ve done it. Transparency also helps us to understand the vendor’s apparent problems and bad decisions, and the ways they’re working to overcome them. It leads to an evaluation that takes into account not just what the vendor is visibly doing, but also the thought process behind it.

Once the vendors have gone through their song and dance, we enter our numeric scores for the defined criteria into a tool that then produces the Magic Quadrant graph. We cannot arbitrarily move vendors around; you can’t say, well, gosh, that vendor seems like they ought to be a Leader / Challenger / Visionary / Niche Player, let’s put them in that box, or X vendor is superior to Y vendor and they should come out higher. The only way to change where a vendor is placed is to change their score on the criterion. We do decide the boundaries of the MQ (the scale of the overall graph compared to the whitespace in the graph) and thus where the axes fall, but since a good MQ is basically a scatterplot, any movement of axis placement alters the quadrant placement of not just one vendor but a bunch.

Once the authoring analysts get done with that, and have done all the write-up text, it goes into a peer review process. It’s formally presented in a research meeting, any analyst can comment, and we get challenged to defend the results. Content gets clarified, and in some cases, text as well as ratings get altered as people point out things that we might not have considered.

Every vendor then gets a fact-check review; they get a copy of the MQ graphic, plus the text we’ve written about them. They’re entitled to a phone call. They beg and plead, the ones who are clients call their account executives and make promises or threats. Vendors are also entitled to escalate into our management chain, and to the Ombudsman. We never change anything unless the vendor can demonstrate something is erroneous or unclear.

MQs also get management and methodologies review — ensuring that the process has been followed, basically, and that we haven’t done anything that we could get sued for. Then, and only then, does it go to editing and publication. Theoretically the process takes four months. It consumes an incredible amount of time and effort.

(Please note that per Gartner’s social media policies for analysts, I am posting this strictly as an individual; I am not speaking on behalf of the company.)

Bookmark and Share

About these ads

Posted on January 10, 2011, in Analyst Life and tagged , , . Bookmark the permalink. 23 Comments.

  1. Our company is infested with idiots who try to impress by utilizing pretentious jargon.
    If all of the economists were laid end to get rid of, they’d never reach a conclusion.

    Like

  2. Awesome site you have here but I was wanting to know
    if you knew of any forums that cover the same topics discussed here?
    I’d really like to be a part of community where I can get feed-back from other knowledgeable individuals that share
    the same interest. If you have any recommendations,
    please let me know. Thanks a lot!

    Like

  3. I want too to thank you for his verfy good read!!
    I certainly lovedd every little bit of it. I have got
    you saved as a favorite to look att new things you post…

    Like

  4. On the other hand, you have GTA Online, the multiplayer game
    that has been overshadowed by glitch-related controversies.
    These are all impressive innovations to artificial intelligence, and Grand Theft Auto 5 is a good example of the direction video games need to go in order to
    keep players interested in the video game worlds they spend a
    great deal of their lives living in. Detroit, on the other hand, would be a unique setting for a video game.

    Like

  5. It is apprpriate time to make some plns for the future and it’s time to be happy.
    I’ve read this putt up and if I may I desire to recommend you
    few interesting issues or tips.Perhaps you can write next articles
    relating to this article. I want tto learn even more things about it!

    Like

  6. Great weblog right here! Additionally your website a lot up fast!
    What host are you using? Can I am getting your associate
    link for your host? I desire my web site loaded up as fast as yours lol

    Like

  7. Link exchange is nothing else but it is just placing the other person’s weblog
    link on your page at suitable place and other person
    will also do similar in favor of you.

    Like

  8. I quite like reading through an article that can
    make people think. Also, many thanks for allowing for me
    to comment!

    Like

  9. It’s very effortless to find out any matter on web
    as compared to textbooks, as I found this piece
    of writing at this web page.

    Like

  10. Understanding both advantages and disadvantages are important in order to choose
    the best internet marketing strategy. They only want your money
    and they want as much of it as they can possibly get.
    The Unit is powered by the Main chip adopting the CORTEX A8 core Sam
    – Sung S5PV210,and CPU frequency up to 1GHz, Memory 512MB, internal 2G memory space for TV and Internet
    integration.

    Like

  11. Very good blog! Do you have any hints for aspiring writers?
    I’m hoping to start my own website soon but I’m a little lost on
    everything. Would you propose starting with a free platform
    like WordPress or go for a paid option? There are so many options
    out there that I’m totally overwhelmed .. Any suggestions?
    Thanks!

    Like

  12. Hello There. I found your blog using msn. This is
    an extremely well written article. I’ll make sure to bookmark it and return to read more of your
    useful info. Thanks for the post. I will definitely
    comeback.

    Like

  13. Pretty section of content. I simply stumbled upon your
    weblog and in accession capital to say thawt I acquire in fact loved account
    your blog posts. Any wway I will be subscribing for
    your augment and even I success you access persistently fast.

    Like

  14. This is very interesting, You are a very skilled blogger.
    I’ve joined your feed and look forward to seeking more of your fantastic post.
    Also, I have shared your website in my social networks!

    Like

  15. With havin so much content do you ever run into any issues of plagorism
    or copyright infringement? My website has a lot of unique content
    I’ve either created myself or outsourced but it seems a lot of it is
    popping it up all over the web without my agreement. Do you know any techniques to help reduce
    content from being stolen? I’d truly appreciate it.

    Like

  16. Do you have a spam issue on this site; I also am a blogger, and I was
    curious about your situation; we have developed some nice practices and we are looking to trade solutions with others, why not shoot me an email if interested.

    Like

  17. Hi there Dear, are you truly visiting this web page daily,
    iff so then you will dewfinitely obtain pleasant experience.

    Like

  1. Pingback: Tweets that mention The process of a Magic Quadrant « CloudPundit: Massive-Scale Computing -- Topsy.com

  2. Pingback: Interesting Gartner Analyst Blog posts « The Future is…Cloudy

  3. Pingback: The cloud and customized contracts « CloudPundit: Massive-Scale Computing

  4. Pingback: Recommended reading for 2014 Cloud IaaS and Managed Hosting Magic Quadrants | CloudPundit: Massive-Scale Computing

  5. Pingback: M-A-O-L » The process of a Magic Quadrant

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 9,845 other followers

%d bloggers like this: