How to Best Understand Gartner’s Magic Quadrant

Click a Link to Jump to That Section

Executive Summary

  • The Gartner Magic Quadrant reduce software selection efficiency through the deceptive way they are created.
  • Gartner promotes large vendors over small vendors as large vendors have the most to pay Gartner.
  • The problem with the broadness of the Gartner Magic Quadrant and how they promote software acquisitions.

Introduction to the Gartner Magic Quadrant

The Gartner Magic Quadrant is a highly popular “research product” that Gartner uses to shake down vendors for payments in order to score well. However, for how popular the Gartner Magic Quadrant is, it remains little understood in terms of how it is created. In this article, you will learn from our detailed research into the Gartner Magic Quadrant.

What is the Gartner Magic Quadrant?

The Gartner Magic Quadrant is a simple and well-known graphic and is the best-known research produced by Gartner. The Gartner Magic Quadrant places a software vendor on a matrix, with the completeness of vision on one axis and the ability to execute on the other.

A frequent question people have about the Magic Quadrant is how many of these graphics Gartner produces. The full list of Gartner Magic Quadrants can be found on Gartner’s website at the following link.

However, many of the Gartner Magic Quadrants produced by Gartner have been discontinued. Using Gartner’s spreadsheet I found one hundred and twenty-six magic quadrants. However, it’s clear this spreadsheet is missing several entries, as I am aware of Gartner Magic Quadrants that are on not the list. Furthermore, there are several Magic Quadrants that I have from the mid-2000s that Gartner used to produce but does not list as retired Magic Quadrants.

An example of a Gartner Magic Quadrant is shown below:

According to Gartner

“Magic Quadrants depict markets in the middle phases of their lifecycle….”

Depending upon how the software vendor rates, they are placed into one of the following four quadrants. These quadrant descriptions are from Gartner’s publication Inside Gartner Research.

Gartner’s Quadrants that Make up the Gartner Magic Quadrant

  1. Leaders: “Leaders provide mature offerings that meet today’s market demand. These providers also demonstrate the vision necessary to sustain their leading position as the market evolves. Leaders focus and invest in their offerings in ways that impact and influence the market’s overall direction.”
  2. Visionaries: “Visionaries align with the Gartner view of how a market will evolve, but they have less-proven capabilities to deliver against their vision. In new markets, this status is normal. But in more mature markets, it may reflect a competitive strategy for a smaller provider (such as selling innovation ahead of mainstream demand), or a larger provider trying to break out of a rut to differentiate.”
  3. Challengers: “Challengers are well positioned to execute, but may not have a strategy in place to maintain strong, up-to-date value propositions for new customers. larger providers in mature markets may often be positioned as challengers because they choose to minimize risk. Although challengers typically have significant human and financial resources, they may lack vision, innovation or overall understanding of where the market is headed. In some cases, challengers may offer products that dominate a large but shrinking segment of the market. Challengers have the opportunity to move into the leaders’ quadrant by expanding their vision.”
  4. Niche players: “Niche players do well in a specific market segment or they have limited ability to innovate or outperform other providers. This may be because they focus on a functionality or geographic region, or they are new entrants to the market. Assessing niche players is more challenging than assessing providers in other quadrants. While some niche players could make progress, others do not execute well and may not be able to keep pace with broader market demands.”

According to my interviews with Gartner, buyers should use the Magic Quadrant independently of Gartner’s analysts. Furthermore, buyers should not interpret a Magic Quadrant as a recommendation to concentrate on those vendors that appear in the Leader quadrant. Gartner agrees that there is a misimpression that being categorized in the Niche quadrant is a bad thing. In fact, a vendor in a non-leader quadrant might be the best vendor for a particular buyer.

The Gartner Magic Quadrant Methodology

As has been previously stated, Gartner publishes only a high-level overview as to how it determines the Gartner Magic Quadrant rankings. People who read Gartner’s analytical products, especially the most influential products, can only review a high-level graphic, which is the output of the research.  To find out more details and speak with an analyst, one must pay more, but Gartner does publish the criteria that count towards each axis. These criteria, quoted from Gartner’s article Magic Quadrants and MarketScopes: How Gartner Evaluates Vendors Within a Market, are listed below. The asterisks are used later to explain the categorization of some of the criteria.

The Completeness of Vision Criterion

  1. *“Market Understanding: The ability of a vendor to understand buyers’ needs and translate these needs into products and services. A vendor that shows the highest degree of vision listens and understands buyers’ wants and needs, which it can shape or enhance with its vision.
  2. Marketing Strategy: A clear, differentiated set of messages consistently communicated throughout the organization and publicized through the Web site, advertising, customer programs and positioning statements.
  3. Sales Strategy: A strategy for selling products that uses the appropriate network of direct and indirect sales, marketing, service and communication affiliates to extend the scope and depth of a vendor’s market reach, skills, expertise, technologies, services and customer base.
  4. *Offering (Product) Strategy: A vendor’s approach to product development and delivery that relate to current and future requirements.
  5. Business Model: The validity and logic of a vendor’s underlying business proposition.
  6. Vertical/Industry Strategy: A vendor’s strategy to direct resources, skills, and offerings to meet the needs of market segments, including vertical industries.
  7. **Innovation: Marshaling of resources, expertise or capital for competitive advantage, investment, consolidation or defense against the acquisition.
  8. Geographic Strategy: A vendor’s strategy to direct resources, skills and offerings to meet the needs of regions outside of the vendor’s ‘home’ or native area, directly or through partners, channels, and subsidiaries, as appropriate for that region and market.”

The Ability to Execute Criterion

  1. **“Product/Service: Core goods and services offered by the vendor that compete in and serve the market. This category includes product and service capabilities, quality, feature sets and skills, offered natively or through original equipment manufacturers, as defined in the market definition and detailed in sub-criteria.
  2. Overall Viability: Includes an assessment of the vendor’s overall financial health, the financial and practical success of the relevant business unit, and the likelihood of that business unit to continue to invest in and offer the product within the vendor’s product portfolio.
  3. Sales Execution/Pricing: The vendor’s capabilities in pre-sales activities and the structure that supports them. This criterion includes deal management, pricing and negotiation, presales support and the overall effectiveness of the sales channel.
  4. Market Responsiveness and Track Record: Ability to respond, change direction, be flexible and achieve competitive success as opportunities develop, competitors act, customer needs evolve and market dynamics change. This criterion also considers the vendor’s history of responsiveness.
  5. Marketing Execution: The clarity, quality, creativity, and efficacy of programs designed to deliver the vendor’s message to influence the market, promote it’s brand and business, increase awareness of its products and establish a positive identification with the product, brand or vendor with buyers. This ’mind share‘ can be driven by a combination of publicity, promotions, thought leadership, word of mouth and sales activities.
  6. *Customer Experience: Relationships, products, and services and programs that enable clients to succeed with the products evaluated. This criterion includes the ways customers receive technical support or account support. It can also include ancillary tools, customer support programs (and their quality), availability of user groups and service-level agreements.
  7. *Operations: The vendor’s ability to meet its goals and commitments. Factors include the quality of the organizational structure, such as skills, experiences, programs, systems and other vehicles, that enable the vendor to operate effectively and efficiently.”

The Orientation of the Gartner Magic Quadrant Methodology

Something that was surprising to me (and is surprising to many people the first time I tell them about these criteria) is that only six of the fifteen bullet points above (40%) are in any way related to the actual application offered by the vendor, which are the bullet points with an asterisk in front of them. Only two of the fifteen bullet points above (13.3%) are directly related to the application. However, these percentages are based simply upon an even weighting of the criteria. Gartner does not explain if they weight the criteria equally or not, so that is all I have to go by for this analysis. After reviewing many Magic Quadrants as part of the research for this book, it does appear as if the criteria are equally weighted.

Gartner is, of course, free to use any set of criteria that they like. However, I find that few people actually read what their criteria in fact are, and most software buyers are under the impression that Gartner’s ratings are much more product-focused than they actually are. In fact, most of Gartner’s criteria are really just proxies for the size of the software vendor. Therefore, I think that it would be helpful if each Gartner Magic Quadrant were published with something like the following disclaimer:

“*Gartner Magic Quadrants rate vendors on a variety of criteria, 40% of which are related generally to the application, and 13.3% are related directly to the application. For more details on the exact criteria used, see this link.”

Gartner Magic Quadrant Scenarios: Large Vendor Versus Small

The following comparison was created to simulate how a large vendor could score well, even if it had very weak software, against a smaller vendor with the best possible rating for its software. To simulate this, I rated all of the product-related criteria as 2 out of 10 for the larger vendor, and 10 out of 10 for the smaller vendor. I then rated the other criteria not related to the product such that the larger vendor would score better.

Now we can show how this rating per criteria would show on a Magic Quadrant graphic.

“When this hypothetical example is placed on Gartner’s Magic Quadrant, the larger vendor scores higher than the smaller vendor. They both would be rated as visionaries, with the larger vendor almost in the leader category.”

How can the Magic Quadrant be a correct representation of the scores listed on the previous graphic? A vendor with a score of 2 out of 10 is neither a Visionary nor a Leader. Secondly, a small vendor cannot be considered merely a Visionary simply because it lacks sophistication in sales and marketing. If they score at the top of all product-related criteria, they are the Leader in that product space.

The Impact of Low Scores in the Gartner Magic Quadrant on the Software Vendor

If we look at the scores from the investor’s perspective, the low scores outside of the product-related criteria would probably limit the vendor’s ability to grow. However, from the software buyer’s perspective, clearly, the small software vendor would be the preferred choice because their software will implement more easily and have a far higher likelihood of success. However, it does not show that way in the Gartner Magic Quadrant.

It should be clear from this Gartner Magic Quadrant that the criteria used by a software buyer do not match up very well with the criteria used by Gartner. However, let’s not stop the analysis here. Instead, let’s take another example, and in this one, the larger vendor has a more competitive product than in the last example.

In this example, we keep all the same values from the first matrix, except this time we make all of the product-related rankings equal,  setting them to 7 for both the small vendor and the large vendor.

When plotted we see that the large vendor is now in the Leader quadrant and the smaller vendor is a Niche player. This is true even though they both have the same rating on their product, and by extension would be true if both companies had the exact same product.

How Gartner Magic Quadrant Promotes Software Acquisitions 

Therefore, if a large vendor acquires a smaller vendor, its overall score in the category being analyzed should increase significantly. Is Gartner telling us that it would prefer to have all small software vendors acquired by larger software vendors? Should all software coalesce into mega software vendors? Given the history of larger vendors with software innovation, what would be the predictable outcomes for innovation if this were to occur? Something that should be obvious is that these are very appropriate criteria for rating a company as an investment vehicle, but are not the criteria that one would normally select if one were attempting to develop a matrix for software buyers.

Now let’s imagine a smaller vendor that does not spend much on marketing or selling but simply has a great product. Here is one more thought experiment with the Gartner Magic Quadrant: How would this smaller vendor with a great product fare in Gartner’s rating system?

In this example, we model a highly product-focused company which invests very little in sales or marketing. However, they have a great product. Criteria that are neither sales nor marketing nor product-focused, we assign an average value of 5. Now let’s see how the product-focused company would fare in a Gartner Magic Quadrant.

Here we see that the vendor ends up as a Niche player. In Gartner’s Magic Quadrant methodology, sales and marketing count for as much in the rating as product-related factors.

Now that we have reviewed sample Magic Quadrants, let’s review an actual Gartner Magic Quadrant created by Gartner and look at two vendors on opposite sides of the spectrum in terms of both product and size.

The Problems with the Broadness of the Gartner Magic Quadrants

Gartner does not produce Magic Quadrants for every software category or for every area. For instance, in my software category, there is no Magic Quadrant for production planning, demand planning or supply planning. These are all subcategories of the software category “supply chain planning,” and Gartner does have a Magic Quadrant for this broader category. However, when a Gartner Magic Quadrant does not exist for the subcategory that a buyer is interested in purchasing, it does not work to use a Gartner Magic Quadrant for the broader category. Actually, beyond this, I did not actually find articles (not only Magic Quadrants, but any articles whatsoever) on production planning or supply planning, which are major subcategories of software within the supply chain planning category. The best analogy that I can give is that—while Honda may be a very good brand—one cannot actually but a Honda. One must buy a Honda Accord or a Honda Odyssey. As good as Hondas are, people want to see the model compared against models from competing brands.

By the same extension, it makes little sense to try to determine which demand planning application scores highest in Gartner’s Magic Quadrant, as brought up in the following quote by Mark Madsen. Here he is talking about a different software category, but the same issue applies.

“How can someone take seriously ‘IBM’ or ‘Oracle’ in the quadrant? Which of the dozen or so products from a single vendor is implied for leadership? If it stated ‘Oracle Data Integration’ or ‘IBM’s DataMirror’ it would be more useful. Some vendor’s products are DOA while others are terrific. Since the evaluation is not the products but the vendors, it’s not possible to use the MQ to choose DI technology appropriate to the situation. When I’m with clients who are Gartner subscribers I have to guide them in their use of these results.”

When No Gartner Magic Quadrant Exists for Your Software Category

One of the biggest obstacles that a buyer or analyst can face is when there is no Gartner Magic Quadrant for the software category of interest. When Gartner decides to create a Gartner Magic Quadrant for a group of products or a software suite, it is impossible for a company, which only makes a single product to compete in that type of analysis. Secondly, it is inadvisable for a buyer to implement multiple products from one suite. Doing this means using a vendor’s weak products along with its stronger products. Also, once the analysis moves to a suite, the business tends to get the short end of the stick, because at that point it simply becomes more about integration, and the functionality becomes an afterthought.

Nevertheless, software suites have been a common approach to IT implementation for several decades. While the concept was that there would be fewer integration issues if software suites were implemented, this approach has meant that the business does not get the software it needs. Software selection based on software suites does not emphasize each application but instead emphasizes the suite.

Software Suites as Anticompetitive Mechanisms

As explained in this quote from Christopher Koch of CIO Magazine, software suites themselves are mechanisms that reduce the competition a vendor must face.

“Indeed, integration standards interfere with ERP vendors’ traditional ways of gaining and keeping customers and market share. Before the Web came along, your integration strategy was simple: Buy as many pre integrated applications from a single vendor as possible. That worked for you, and it worked extremely well for the vendor; integrated application suites fetched a high price and required long-term maintenance and support contracts that promised a steady, predictable stream of revenue from customers.” – ABCs of ERP

I have now seen many poor-performing applications implemented and a great deal of frustration on the part of business users and managers who were not delivered applications that ever had a chance of meeting their requirements.

The Usefulness of High Level Gartner Magic Quadrants?

Such high-level Magic Quadrants are useless to a company that is selecting an application which is a subcategory of that Gartner Magic Quadrant. Suites reduce the incentive of the larger vendor to offer the best products because they do not have to compete on the individual application but can compete on the basis of the integration between the applications in the suite.

If a company is looking to select one or multiple products from a suite, it’s difficult to see how a high-level Gartner Magic Quadrant can be adjusted. Some detail about specific applications can be provided in the vendor description section, but this is simply not enough detail to support an informed decision. At this point, the Gartner analyst must be contacted, and there is a little value that can be received from the Gartner Magic Quadrant.

Adjusting for Functionality and Maintainability

Gartner employs analysts with a strong tendency to look at the bigger (biggest?) possible picture. Gartner looks at so many factors outside of how good the application actually is and how maintainable the solution is, that these most important factors are under-emphasized. Corporate buyers should strive to obtain the software that closely matches their business requirements and has the very best functionality. Gartner essentially tells buyers that the application is really just one component to the decision, but the problem is that Gartner has too many factors pulling their ratings away from practical software implementation criteria. This can be addressed by checking reference accounts and by asking the Gartner analyst how the various vendors performed on the surveys for functionality and maintainability.

The Supply Chain Software Category

Getting back to the supply chain planning category, I can say that no vendor rates highly across all of the software subcategories of supply chain planning. Secondly, some software vendors work better than others for certain requirements, and some for certain industries over others. Making broad Magic Quadrants, such as the one for supply chain planning, will tend to drive potential buyers to review the larger software vendors that have the broadest range of software within the software category. For instance, here is a quotation from Gartner’s Magic Quadrant writeup on supply chain planning.

“SAP has a stronger focus that Oracle on business process and a well-established, capable platform in Advanced Planning and Optimization (APO), which covers all the functional areas of supply chain planning, according to Payne. But SAP APO lacks depth in inventory optimization and production scheduling.”

Is This Quote Correct?

I have written four books on modules in SAP’s supply chain planning suite, and one book on the overall suite. When looking at the above quote, it’s hard to see what it is attempting to accomplish, but the statement certainly benefits SAP. SAP’s supply chain planning suite (called APO) has roughly ten modules (depending upon how you count them)—some are mature, some are quite immature and barely implemented anywhere. Each module must be evaluated independently. There is no way to cover the suite with such a high level explanation of how it works. In fact, one of the worst things a company could do is to try to find a single vendor for all of its advanced planning needs as it would leave major areas of requirements unaddressed and would oversimplify the decision-making toward vendor-oriented criteria rather than application-oriented criteria.

This is where the analyst comes in; however, I question the point of even having a supply chain planning Magic Quadrant if it is so broad that it cannot be used for specific application purchase decisions, and if it promotes purchasing behavior that looks for suites—a bad value for buyers.

How to Adjust the Gartner Magic Quadrant

There is a lot to unpack and understand when using a Magic Quadrant. However, the first thing to remember is that the MQ is not simply an honest evaluation of software. The MQ has a series of biases that are not random or coincidental. They are due to Gartner’s financial biases around how Gartner maximizes profits versus emphasizing research. Magic Quadrants must be viewed through this lens to get any value from them and to keep from being manipulated by what are Gartner’s financial biases.

Gartner’s analysis cannot be used properly without adjusting the research output. By most analysts’ accounts, only about twenty percent of the research that Gartner performs actually gets into a report. While Gartner publishes reports, it prefers to deliver more information by having clients consult with analysts. The downside to this, of course, is that consulting is more expensive.

Much of this book was dedicated to providing the background necessary for the reader to understand why Gartner reports must be adjusted. Companies can, of course, contact Gartner and ask questions but at a cost. There are some things that can be adjusted without contacting Gartner, and some things that cannot. I have included some of the most common adjustments, and have listed those items that would require a company to contact Gartner to get more information.

By making these adjustments I believe the usability of Gartner’s research is improved considerably. Some of these rating adjustments are universal, while others depend on the company you work for.

Gartner Report Adjustment Rules

1. Never Use Just the Gartner Magic Quadrant to Make a Purchasing Decision

This point should be self-evident, especially if you have read the rest of the book up to this point. According to Louis Columbus, “IT buyers regrettably sometimes make their entire purchasing decision just on the Quadrant alone.” It is hard to know where to begin as there are so many reasons not to do this, but as I have explained, Gartner’s methodology for their Magic Quadrants includes factors that have nothing to do with the benefit to enterprise software buyers. Instead, the criteria are metrics that could be used by investors contemplating buying the software vendor’s stock. Secondly, even Gartner does not want buyers or investors making decisions based simply upon the Magic Quadrant or any other research they produce. One should consider the research produced by Gartner as a “starter kit” to start the ball rolling for more analysis, contracting with Gartner for more analyst services, etc.

2. Adjust the Rankings for the Vendor’s Size

Gartner prefers large software vendors. This large vendor orientation is clear, and it is difficult to see how a person doing a complete analysis on this topic could find that this bias does not exist. I hope that the analysis from Chapter 5: “The Magic Quadrant” has made that clear.

On the web there are several statements by Gartner analysts that Gartner does not favor large vendors. Other than managing perception, I can’t think of a single good reason for anyone to say this. To believe that Gartner does not favor large vendors, one would have to both not understand the various Gartner methodologies and not read the Gartner research output.

A vendor’s size is not a universal positive. Bigger vendors are less responsive, particularly if the buyer is not a large client. Because bigger vendors offer a broader set of products, they are constantly trying to capture more and more footprint within their existing clients, and they are not going to be happy selling a single application to a client. Larger vendors are less focused on their products, less focused on R&D, and more focused on marketing and on their partnerships with major consulting companies.

Working with Smaller Software Vendors

One of the reasons I enjoy keeping up with smaller best-of-breed vendors is because they have less bureaucracy, and they engage in far more innovation than the big vendors. I implement the software of the large vendors, and when I visit smaller vendors, I always feel as if I have traveled forward in time because the approaches we follow in SAP are so dated. I essentially implement approaches and technologies that are little changed from the mid-1990s. And of course, bigger vendors are more expensive (see the section on Adjusting for Price below).

So, the bigger vendors in any Magic Quadrant are simply overrated by Gartner. The best way to account for this is by reducing their rating although I cannot provide a specific percentage adjustment.

3. Inserting More Reality into Gartner’s Ratings

Generally speaking, Gartner analysts do not have hands-on experience with the applications they are rating. Most of the Gartner analysts have at least fifteen years of experience in their area, so it is not that they lack experience but they usually lack implementation experience. Gartner analysts do speak with people in companies that implement software, but they speak at the level of CIOs and vice presidents. This constant interaction with senior executives at companies that buy enterprise software is the main reason why working for Gartner is considered to be such a good place to create contacts for future careers.

However, these high-level people do not use the applications that they purchase; they passing along information to Gartner analysts that is already second-hand! While executives at buyers certainly know the problems that they face on implementations, overseeing implementations and actually working with the software are two different things.

4. Steps to Increase Reality of the Software Selection

Now that we have established that Gartner’s ratings require more reality, the question is “how.” An effective software selection will combine the high-level analysis provided by Gartner (provided either by reading research or by interacting with Gartner analysts) with more detailed analysis of:

  • How the application actually works
  • How the employees of the company will actually use the application
  • How it will be configured and fit in with the rest of the company’s footprint

Unfortunately, this is a lot more work and cannot be done by those who are the decision makers for the software selection. So, this more detailed work tends not to get done. However, it should be done—if not for all the applications under consideration then at the very least for the applications that the company finds the most intriguing. This applies to both buyers and investors, and is accomplished by having technical resources participate in the software selection process.

I have no idea why so many executives think they are qualified to analyze statements made by presales consultants, but usually, presentations are made to buyers with no one in the meeting from the buyer’s side who can validate the technical statements made by the vendor’s representatives.

5. Finding Technical Fact Checkers

Many times individuals with technical backgrounds can be pulled from the buyer’s IT department. However, this is not always the case, especially if there is no one familiar with the category of the application being analyzed. In Appendix A I include a section on hiring independent consultants who can provide expertise that may not be available inside your company.

Viewing demos that are performed by a vendor’s most skilled presales consultants is not a good way to truly understand an application. Rather than scripted demos, a more reality-based approach is to allow the reviewers to ask more questions and drive more of the demo. I cover this topic in more detail in the book Enterprise Software Selection.

6. Adjusting Down the Rate of Predicted Change

Gartner tends to overestimate the amount of change that occurs in any one software category, as well as the business developments in that category. Also, Gartner will describe industry trends as being more legitimate and permanent than they actually are. For instance, in my area, many companies have initiatives to move towards a make-to-order environment; however, because of the limitations of the approach most never will. This falls under the category of a “pipe dream.”

It’s easy to get the impression that because Gartner learns about these initiatives from senior executives within the buying companies, Gartner tends to take their word for it that most of these initiatives will succeed, when in fact a higher percentage may fail. This dynamic applies equally on both the buyer and the vendor side, as Gartner will frequently get onboard with a new technology initiative before a vendor and before that initiative has had the time to prove itself. For instance, in my field, Gartner has been a major proponent of something called SAP Hana.

Innovations in the Larger Vendors?

I have witnessed several innovations that no IT analyst firm has written anything about (one of which I covered in the book Supply Chain Forecasting Software). After a large firm has copied the technology, I assume I will read about how great it is in a Gartner report, maybe five years from now. In any case, it’s far too early for Gartner to have jumped on the Hana bandwagon as essentially it is still in development.

As I hope the example above demonstrates, Gartner is well-known for its Hype cycle analysis, but Gartner itself will help drive a Hype Cycle to the “Peak of Inflated Expectations” (using Gartner’s own Hype Cycle terminology). Often they are more swayed by how much “sense” a vendor’s explanation seems to be over how it actually works in practice. Unfortunately, a lot of things seem like a good idea or seem to make sense but don’t necessarily take flight. For example, SaaS was supposed to be the next great thing, but its adoption has been slow. Hypothetically, Gartner’s frequent interaction with buyers should enable them to keep from doing this. However, my research into their past reports indicates that this is a consistent theme of Gartner despite their access to buyers which would allow them to validate vendors’ statements.

Buying Evaporating Trends

Therefore, Gartner’s reports should be adjusted and deferred. Whatever Gartner is predicting, it almost certainly will not happen as quickly as predicted. It should be understood that while many buyers may be moving in a particular direction, many will not be successful. Many initiatives are simply trends, and when they peter out, they are renamed. In my area, there is an initiative referred to as “lean,” a rebranding of many identical initiatives of JIT (just in time) which mostly flame out. There are now roughly half a million books on “lean,” but the actual impact on inventory management? Negligible. The benefit to consulting revenues of consultancies promoting lean concepts? Priceless.

Likewise, vendors frequently bring out new ideas and products, but most of these developments tend to be transitory and some are really just rebranding of old concepts by sales and marketing. Secondly, beyond time-phasing Gartner’s predictions, many of their projections will never come to pass. When reviewing Gartner’s past predictions, I found this repeatedly. Gartner’s Hype Cycle proposes a consistent and eventual movement of technologies to the “Plateau of Productivity.” However, some technologies or ideas are never implemented broadly and many disappear altogether.

7. Adjusting for Price

Gartner does not account for the price in its rankings. Instead, they compare all products as if they are in the same cost category. This approach creates a bias towards more expensive products. In addition, because Gartner’s methodology ranks larger vendors higher, Gartner’s recommendation tend to push buyers to higher-priced products and products that result in higher consulting costs. (Larger vendors tend to have both higher acquisition costs and higher consulting costs.) Therefore, it makes the most sense to adjust Gartner’s ratings for the price. Unfortunately, this is not an easy task.

There are few software vendors like Arena Solutions and Demand Works that post the cost of their software on their websites. In fact, in many years working in the field of software implementation, I am rarely informed of the actual costs of the software being installed. Many vendors are cagey about their price; among a host of factors, the price depends upon the number of seats and how much the software vendor wants the account. Determining the actual costs is time-consuming and requires a significant amount of interaction with the software vendor so they can “understand their requirements.”

Gartner’s Large Vendor Bias

Gartner is perfectly positioned to provide some type of rough approximation of the cost of the software, which could allow buyers to put some vendors out of their cost range. They are also well-positioned to estimate the consulting/implementation and maintenance costs of the different vendors per software category. However, if a company does not want to use a Gartner analyst to answer these questions, the company can adjust for Gartner’s large vendor bias (and by extension their bias towards expensive solutions bias). They can push a smaller vendor up in consideration in order to account for lower cost and better value if of course, the vendor has other compelling factors. This leads to the next area of adjustment.

8. Adjusting for the Buying Company’s Size

Gartner’s bias towards larger vendors is related partially to Gartner’s belief that companies prefer (or should prefer) to buy from the largest vendors. However, while larger buyers may prefer to purchase from larger vendors, it is less true of smaller buyers.  Of course, Gartner’s largest customers are the biggest companies in the world.

I do not know if smaller buyers would prefer buying from larger buyers if they could afford to do so, but the fact is that smaller buyers often can’t. Generally, small innovative vendors offer point solutions, have small deal sizes, and sell to small buyers. As the vendor grows, it is able to increase the size of buyers to which it sells. Therefore, Gartner’s ratings fit the preferences (and checkbook) of large buyers far more than they do for smaller companies. Experience over time will let the smaller buyer know who they can afford to buy from, so for smaller buyers, it is a simple matter to remove all or the majority of the largest vendors from their list and to focus on the other vendors.

The Problem: Thinking that Gartner is Focused on What is True

Gartner is hired by companies who fundamentally don’t understand how Gartner functions. Gartner has virtually no first-hand experience in the technologies that they evaluate and get most of their information from speaking with executives at buyers or executives at vendors and consulting firms. Gartner is also not a research entity, they compare very poorly to real research entities once you dig into the details as we did in the article How Gartner’s Research Compares to Real Research Entities. Gartner serves to direct IT spending to the most expensive solutions as these are the companies that can afford to pay Gartner the most money. Gartner has enormously aggressive internal sales goals that place accuracy far below revenue growth in importance.

Being Part of the Solution: Fact-Checking Gartner

Gartner is quite expensive, and using them without independent verification ends up in a lot of money being wasted, both in terms of fees to Gartner and also in money spent on Gartner recommendations. We offer Gartner support services that make sure companies get a better value from Gartner, and keep companies from relying on advice where Gartner cannot substantiate the advice or where the advice is clearly in response to vendor pressure. This includes both covering how Gartner works, as well as reading and fact-checking the information they provide. Our clients don’t even let Gartner know that we are evaluating the information provided to them.

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Search Our Other Research Bias Content

References

Gartner Book

Gartner and the Magic Quadrant: A Guide for Buyers, Vendors, and Investors

Gartner is the most influential IT analyst firm in the world. Their approval can make or break a vendor in an application category, or at the very least control their growth. Gartner has been behind most of the major IT trends for decades. However, many people read Gartner reports without understanding how Gartner works, how it comes to its information, its orientation, or even the details of the methods it uses for its analytical products. All of this and more is explained in this book.

Table of Contents

  • Chapter 1: Introduction
  • Chapter 2: An Overview of Gartner
  • Chapter 3: How Gartner Makes Money
  • Chapter 4: Comparing Gartner to the RAND Corporation, and Academic Research
  • Chapter 5: The Magic Quadrant
  • Chapter 6: Other Analytical Products Offered by Gartner
  • Chapter 7: Gartner’s Future and Cloud Computing
  • Chapter 8: Adjusting the Magic Quadrant
  • Chapter 9: Is Gartner Worth the Investment?
  • Chapter 10: Conclusion
  • Appendix a: How to Use Independent Consultants for Software Selection
  • Appendix b: What Does the History of Media Tell Us About This Topic
  • Appendix c: Disclosure Statements and Code of Ethics