How Gartner Research to Compares Real Research Entities

Executive Summary

  • Gartner positions itself as a research entity. However, when compared with real research entities Gartner does not perform well.
  • We compared Gartner against academic research, RAND and Consumer Reports to see how well they performed. Most people who read Gartner’s output will be surprised how they scored.

Introduction

Gartner is often referred to as a research entity. However, what makes an entity a research entity, aside from domain expertise is the following of long-established rules regarding research. These rules are very well understood among those that perform research. For example, we know them, and we follow them. In this article, we will review whether Gartner follows the established rules of research.

How Gartner Research Compares to Consumer Reports

I decided to evaluate the research output of Gartner by comparing Gartner’s rules and business practices to respected research rules. A comparison of the research approach used by Gartner to that used by other analyst firms may seem like the long way around to understanding Gartner’s research output, but I believe this chapter will greatly improve one’s understanding of Gartner’s research. While I have heard various criticisms of Gartner, I have never seen Gartner compared against respected research entities on important criteria. That is what is accomplished in this chapter.

One of the comparisons made in this chapter is between Gartner and what is probably the best-known rating company in the world: Consumer Reports. The second comparison is with one of the most respected think tanks in the world—the RAND Corporation—which has established a reputation for impartiality and sometimes groundbreaking research going back six decades. The third comparison is not with a single entity, but between Gartner and the academic research system to which most research institutions in the US, Europe, and Australia/New Zealand subscribe. This chapter is important as it sets the stage for later chapters. My research for this book highlighted to me that far too frequently, Gartner is discussed in isolation without the broader context of generally-accepted research practice.

Gartner Versus Consumer Reports on Funding from Rated Entities

Consumer Reports is the most trusted consumer rating and product-testing agency in the US. Consumer Reports has been continually published since 1936. Consumer Reports has a very structured and well-designed set of rules that strictly limits the influence of those whose products and services are rated by them. Consumer Reports takes no advertising or other forms of payments from those it reviews; instead, it is wholly supported by their subscribers and is the largest subscription supported website in the US.[1] Compare this to Gartner, which takes money from those companies that it rates, actively solicits more business from current vendors who are customers, and solicits vendors who are not customers with what is generally known as aggressive sales tactics which stop just short of promising better results in their ratings.

The fact that Consumer Reports takes no advertising or other monies from those that they rate is a critical point. Their policy is different from JD Power and Associates, which is another well-known rating company. JD Power and Associates not only charges the vendors they rate but charges vendors to advertise the JD Power and Associates award that they “won.” They rate the winners in a category but do not provide a complete list of the contestants, which is clearly a nod to the vendors who did not perform well in that category. These factors, among a host of others, is why JD Power and Associates is not seen as a serious research entity, and their principle usage is not by consumers but instead as something that advertisers use in order to create the illusion of a good rating with consumers.

Gartner Versus Consumer Reports on Vendor Use of Ratings

Consumer Reports also has a non-commercial use policy, which means that companies that are rated well by Consumer Reports may not use the Consumer Reports rating in advertising or even on their website. Generally speaking, this is the opposite from IT analyst firms and also from Gartner. Gartner has some restrictions on how the ratings can be used, but the limitations are quite liberal. The last reference to the rules regarding how Gartner’s r

Software vendors frequently publish their Gartner ratings (if they are good) on their websites, “tweet” them, and put them in press releases. So software vendors advertise their Magic Quadrant rating quite aggressively if they have done well.

The Problem with Allowing Vendors to Use the Ratings in Advertisements (And Charging Them)

At first it may not seem obvious why it should matter if those companies that are rated use the rating in their advertising. The reason relates to the perception of endorsement that is made by the inclusion of a rating in such an advertisement, as well as the strong tendency a vendor has in spinning the rating to their ultimate advantage. Essentially, once vendors begin using rating results in advertising, Consumer Reports can no longer control the presentation of its results.  Consumer Reports explains their policy in the following way:

“At Consumer Reports, we believe that objective, impartial testing, reviews, and ratings are critically important for consumers. That is why we have a strict “No Commercial Use Policy” preventing the use of our name and information for any promotional or advertising purposes. The policy helps ensure we avoid even the appearance of endorsing a particular product or service for financial gain. The policy also guarantees that consumers have access to the full context of our information and are not hearing about our findings through the language of salesmanship.”

Gartner Versus Consumer Reports on Controlling for Sample Bias

Consumer Reports buys all of the products that they rate. This allows them to be confident that the products they rate are the same products consumers would receive. If Consumer Reports did not do this and instead accepted free samples, these samples would deviate from what was available to consumers, as the manufacturers would make sure that only the best was sent to be rated. Not only do IT analyst firms not buy the software they rate, they don’t actually test the software they rate.

Ratings come from discussions with the software vendor, reviewing demos and questionnaires filled out by buyers, evaluating the literature produced by the vendor, and talking to companies that have implemented the software.

Gartner’s Lack of Testing or Even “Touching” of the Products they Rate

Therefore, Gartner differs quite significantly from Consumer Reports in that they do not actually test the products they recommend, and when Gartner does see the product, it is a demo presented by the vendor in an artificial environment. Gartner uses a script during these analyst briefings, which means that the vendor must show what Gartner has on its script and may not deviate from the script. This is done to provide a consistent rating methodology to each vendor, but it has a disadvantage in that it prevents the presentation of what could be interesting and useful functionality. A number of individuals who have participated in these analyst briefings have informed me through interviews that the demos are in fact lighter versions of customer demos, which are themselves light compared to a demo that would be presented to a software-oriented person such as myself.

Gartner’ Statements on Improving Ratings Through Purchasing Advisory Services from Gartner

[1] Gartner’s sales group clearly implies to vendors that they will improve their rating if they purchase consulting services from Gartner; this fact has been independently verified in multiple interviews I have conducted with people who work in senior positions out of the marketing department at software vendors. My confidence in this statement is reinforced by the fact that the same phraseology used by Gartner was quoted to me by these individuals and the fact that none of these individuals knew one another.

The idea that one would pay money — for anything, to a normal research entity and there is the potential for the rankings in the research to improve would eliminate any study from being taken seriously in the academic realm. It is not done by Consumer Reports. The RAND Institute discloses its funding in each study that it performs, but it does not put itself in that type of conflicted position in the first place.

How Gartner Research Compares to The RAND Corporation

Depending upon who you ask, RAND is either the world’s pre-eminent think tank or one of the top think tanks. RAND was created over sixty years ago, before think tanks had acquired their poor reputation. In effect RAND is a “real” think tank, following—and in many cases exceeding—academic research standards. Although they are highly productive, they employ only roughly 1,700 people. Upon occasion, I have read RAND research and was most impressed with the quality of their work. They were, in fact, a major innovator in an area of software about which I have written a book (Inventory Optimization and Multi Echelon Planning Software), and their research was featured prominently in that book.

The RAND Corporation began its life primarily doing research for the Pentagon. Some of their best-known research has been in the area of war gaming and strategies, which ranged from Vietnam (and from where was leaked the infamous Pentagon Papers) to the US anti-Soviet nuclear missile strategy. However, in the past few decades they have diversified into a much broader research entity that performs research on everything from energy and the environment to health care. RAND also publishes an explanation of the standards they follow. I did not use all of the criteria published in this document because some of the criteria do not relate to Gartner; RAND serves a public service function, whereas Gartner does not. However, a number of RAND’s standards were universally applicable and they are discussed and compared to Gartner’s standards in the following paragraphs.

Gartner Versus RAND on Referencing Past Work

RAND communicates very clearly how its research is performed.

“Although internal discussions about research quality have always been an integral part of RAND culture, more than a decade ago, we decided to codify in writing the quality standards for all RAND research. We intend the written standards to serve both as a guide for those who conduct, manage, support, and evaluate the research activities at RAND and also as the set of principles by which our research units and programs shape their individual quality assurance processes.”

RAND’s Standards for High-Quality Research and Analysis publication makes the following statement regarding how its research references other work.

“A high-quality study cannot be done in intellectual isolation: It necessarily builds on and contributes to a body of research and analysis. The relationships between a given study and its predecessors should be rich and explicit. The study team’s understanding of past research should be evident in many aspects of its work, from the way in which the problem is formulated and approached to the discussion of the findings and their implications. The team should take particular care to explain the ways in which its study agrees, disagrees, or otherwise differs importantly from previous studies. Failure to demonstrate an understanding of previous research lowers the perceived quality of a study, despite any other good characteristics it may possess.”

Gartner’s research is very much encapsulated and is much like Consumer Reports in this regard. A Gartner analyst might say that of course, they don’t work in isolation; they talk to vendors and client executives every day. However, that is not the isolation to which this statement refers. This statement refers to other research and is unrelated to whether one accesses data sources. I write this as explicitly as I can, knowing full well that if a Gartner analyst does comment on this criticism in some public forum, that they will bring up the fact that they are not isolated and therefore I am mistaken and don’t understand how Gartner operates. However, I can only declare this several times and let the chips fall where they may.

Gartner Versus RAND on Transparency of Research Data

RAND’s position on the transparency of research data is taken from the same RAND publication as the previous quotation.

“Data and other information are key inputs to research and analysis. Data-generation methods and database fields should be clearly specified, and the data should be properly screened and manipulated. The research team should indicate limitations in the quality of available data. In addition, information presented as factual should be correct and verifiable.”

Gartner’s data is oftentimes not verifiable, for the main reason that frequently the data is not published. However, to meet RAND’s standards the data from research must always be published.

Gartner Versus RAND on the Use of Jargon

It was interesting to see RAND take this position on the use of jargon:

“Necessary technical terms should be defined and explained.”

Gartner tends to use jargon in its writing. I recall one report that stated that a vendor’s application performed “stochastic optimization.” I was curious as to how many people who read the report actually knew what the term “stochastic” meant! Gartner basically assumes that the readers know all the terms that they use, but I do not recall seeing definitions at the end of their research. Although, in this day of the Internet where terms can be easily searched, it is probably less of an issue than it was in the past.

Gartner Versus RAND on the use of Graphical Elements

RAND has this to say on the topic of graphical elements:

“To help explain complex and novel ideas, the documentation should augment textual exposition with graphical or pictorial elements.”

Gartner Versus the Academy on Peer Review

Gartner’s research is to a degree peer-reviewed prior to publication, but not with peers outside of Gartner. In academics, research is always peer reviewed outside of the institution. According to Wikipedia: “Publications that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals.”

A lack of peer review can come across as a researcher having a particular bias, or that a researcher was not confident that the findings could survive a peer review. One of the best examples of the mistakes that can occur when research is not peer-reviewed is the example of cold fusion. In 1989 Stanley Pons and Martin Fleischmann, along with the University of Utah, submitted a research paper to the journals Nature and TheJournal of Electroanalytical Chemistry —at the same time that they released a press release saying that they had discovered a way to create a nuclear reaction at room temperature. Pons and Fleishmann predicted to members of the media that:

“..cold fusion would solve environmental problems, and would provide a limitless inexhaustible source of clean energy, using only seawater as fuel. They said the results had been confirmed dozens of times and they had no doubts about them.”

The world was on its way to clean and limitless energy. There was only one problem: the results recorded by Pons and Fleischmann could not be replicated by other research teams that followed their methodology. The results, which led to the cold fusion rush, were eventually chalked up to poor research controls and to the conclusion that Pons and Fleischmann had measured their study incorrectly. This research, which had not been peer-reviewed, led to a number of negative consequences including many millions of dollars spent in the subsequent decade—mostly by private industry—to replicate the faulty results. However, if Pons and Fleischmann and the University of Utah had followed the normal academic protocol of waiting to announce their results until after their experiment had been peer-reviewed and attempts had been made to replicate the results, the term “cold fusion” would not be in the popular consciousness because there would have been nothing to report.

The Issue Gartner Would Face if its Research Peer Reviewed

Peer review works in the academic system because these institutions are part of a large ecology of universities that have worked this way for quite some time and it is the accepted practice. It would be very difficult for Gartner to have its research peer-reviewed. Who would review it: Forrester? These entities are all for-profit companies that do not share information with other companies— a major limitation with research conducted by private entities. The research of Consumer Reports is not peer-reviewed; however, the transparency with which it publishes its reports makes Consumer Reports auditable. The lack of peer review, along with an inability to audit Gartner’s results, is a problematic combination, which will be explored in the conclusion of this chapter.

Gartner Versus the Academy on Publishing the Methodology

The methodology used for research is such an important part of the disclosure of scientific papers that it is part of the IMRAD acronym, which is commonly discussed in academics. This stands for:

  1. Introduction
  2. Methods
  3. Results
  4. (and)
  5. Discussion

This is the normal order in which academic papers are presented.

Does Gartner Publish Its Methodology?

A number of headings on Gartner’s website use the term “methodology.”

Some can be found below:

Supply Chain 25 Methodology

Gartner Methodology

These pages do provide information about the methodology, as does the research. Generally, it is clear, at least to me, what each of Gartner’s research reports is measuring. However, as a person who has read quite a bit of research, I would classify Gartner’s disclosure on methodology as very light. Essentially Gartner is misleading many of its subscribers (who do not have a research background) into believing that this is a normal level of disclosure.

For instance, let’s look at the methodology for the Magic Quadrant. The criteria used and which make up each axis are declared; however, are they all given equal weight or are some weighted more than others? What Gartner considers as disclosure of its methodology would not be considered a full disclosure by any of the research entities that I have discussed up to this point. In fact, Gartner’s results, as published to subscribers, would never be published in any journal. Not all IT analysts follow this concealed approach. Forrester is one. Donald Ham explains how Forrester’s Wave (a competitor to Gartner’s Magic Quadrant) discloses much more to subscribers and allows subscribers to gain more value from the research.

“Forrester’s Wave product, their graphical vendor comparison tool, is similar in approach to the well-known Gartner Magic Quadrant.  However, the wave lets end users change relative weights of the comparison criteria by downloading and interacting with an Excel spreadsheet, resulting in a graphical display customized for the specific need­—a very cool enhancement.  Forrester is not as prolific with Wave reports as Gartner is with MQs, but where they exist for the products you’re investigating, points go to Forrester.”

Gartner’s Degree of Methodological Opacity Versus an IT Entity Like Forrester

The following table shows the criteria, the weights, and the data from Forrester’s Wave on BI Service Providers.

A good example of a far more complete disclosure of the methodology is available at the website for statistics for the Netherlands. I have included this link so that readers can gain an understanding of what I am describing.

The following argument is used by ZL Tech’s CEO Kon Leong, the company referred to previously which sued Gartner for damaging its business by unfairly ranking its products versus competitor’s products because of the company’s smaller size (its lawsuit was eventually rejected).

“The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores, so vendors know exactly where they are lacking and by how much and take corrective action.”

It is important to consider that Gartner’s target market is not researchers, but rather decision makers who are buyers, vendors and investors. Many (but not all) of the people in these groups will be more interested in simply seeing the results. In fact, there are many who question how many of the people who read and rely on Gartner’s reports actually read the entire report, much less understand the research methodology. This gets back to the main theme throughout this book—that Gartner’s research is not particularly useful without the context provided by Gartner analysts. However, because the privately provided information is not part of the public record, it is also not auditable.

How the Methodology Is Interpreted

The methodology is one of the most overlooked areas of Gartner by those who read its research. The criteria are often not what one would expect to be used, and if you speak with most people about Gartner’s methodology, it turns out they don’t actually know the methodology. I can say this with confidence because when I tell interviewees what makes up the criteria for different analytical products, I frequently receive the response: “Is that what is counted by Gartner?”

Gartner itself has stated that readers are too quick to review a single graphic and not read the entire report. On the other hand, the way Gartner comments on its research is also responsible for providing the wrong impression regarding their methodologies as I will demonstrate. I explain this in Chapter 5: “The Magic Quadrant.” However, for the sake of continuity let me provide another example: Gartner’s “Supply Chain Top 25.”

The Gartner Supply Chain Top 25

Gartner publishes a “Supply Chain Top 25,” which most people would assume is the twenty-five companies that have the highest-performing supply chains (that is, the highest performance from the metrics normally associated with supply chain management—things like forecast accuracy, inventory turnover, service level, and other similar measures). However, upon reading Gartner’s methodology for creating the Supply Chain Top 25, only one of these criterion are used. The criterions are instead:

  1. Return on Assets (ROA) — Net income / total assets
  2. Inventory Turns — Cost of goods sold / inventory
  3. Revenue Growth — Change in revenue from prior year
  4. Gartner’s Internal Voting
  5. Gartner’s Client’s Voting

Two of the criteria that determine the Supply Chain Top 25 do not have anything to do with supply chain at all, but have to do with financial performance! However, when I read Gartner’s explanation of the research to media outlets, I found it did not match up with the methodology. The quotation below is an example of this.

“At the heart of the Supply Chain Top 25 is the notion of demand-driven leadership,” said Debra Hofman, managing vice president at Gartner. “We’ve been researching and writing about demand-driven practices since 2003, highlighting the journey companies are taking: from the old ‘push’ model of supply chain to one that integrates demand, supply and product into a value network that orchestrates a profitable response to ever-shifting changes in demand.”

Debra Hofman’s explanation is not the primary methodology of the study. Instead, the methodology focuses on the financial performance of the overall firm combined with people voting for who they thought should be on the list. As I’ve explained, the heart of the study is the methodology, not the notion of “demand-driven leadership.” The only control over whether a company showed “demand-driven leadership” is the one criterion, which Gartner voted on. The other four criteria could not have been related to this.

Furthermore, the definition of “demand-driven” is quite hazy. I work in the field and am unsure as to whether I have heard the term used before. Furthermore, the term “demand driven” is not used a single time in the report itself. If it is such an important component to the report, why is it not part of the methodology and not part of the report?

The Overall Scores

In the tables below, I have organized the scores of Gartner and of each of the comparison entities. The tables have a few more criteria than included in this chapter. These criteria were excluded from the text to keep the chapter to a manageable length. The tables rank all entities against Gartner on every criterion:

Here you can see a description of how each entity performed in each criterion. Based upon this description I have assigned a score, which are shown in the tables below: 

How the Scoring Works

  1. The scoring methodology is that the higher the score the better. The entity receives a ten if the issue does not apply to them. For instance, Consumer Reports does not receive any funding, so the issue of reporting industry funding does not even come up.
  2. The zero and ten scores were simple to score; however, when the criteria was not binary, the scoring required some judgment or subjectivity. Of the four entities, academic research was the most difficult to score because, while there are generally-accepted research principles, there is considerable variability in their adherence. Not only is there the issue of bio-medical research, which operates quite differently from other types of academic research, but there is the issue of public versus private universities, among just a few of many variable factors.
  3. Given RAND’s excellent reputation for impartiality and transparency, I was surprised that I was not able to find any record of financial contributions or even ranking of donors from large to small. In all the research entities that I studied, it is concerning that it is so difficult to determine exactly how much money is contributed by each entity. Although there is no question of which entity funds each of RAND’s research reports and RAND lists its contributors (which is far more transparent than Gartner), this disclosure is still below what I would like to see. To see how transparent RAND is on this topic, I have included below an example of the statement incorporated on the front of each paper:

“This research is supported by the United States Air Force under Project RAND—Contract No. F44620-67-C-0045—monitored by the Directorate of Operational Requirements and Development Plans, Deputy Chief of Staff, Research and Development, Hq USAF. Views or conclusions contained in this study should not be interpreted as representing official opinion or policy of the United States Air Force.”Descriptions of The Computer Program for Metric – A Multi-Echelon Technique for Recoverable Item Control

This makes it clear who funded the study, even if we are not told how much funding was received.

Disclosure of Financial Contributions in the Academy

Academic research is far hazier in terms of listing contributors, as the university contributors are never actually listed on the reports. Instead, the money tends to go into a big pot for each university department. But it would be difficult to doubt its corrupting influence. One of the more amusing contributions from industry was the $100 million that Exxon Mobile paid Stanford to fund climate and energy research. One can imagine what type of “research” Stanford will be doing with this money, and how many times this grant will be mentioned on Exxon Mobile marketing material.[1]

Because Consumer Reports is subscriber-supported, no one in the comparison group could match Consumer Reports in their lack of conflicts of interest. I did not weigh the criteria as I would normally do in a software selection matrix but instead considered each criterion equal in weight. In my opinion, the criteria should not be equally weighted, but I wanted to keep the presentation of the table simple.

Reasons for Gartner’s Low Score

The scores were quite interesting; as you can see from the tables on the previous page, Gartner scored quite low when compared to the other three research entities.

Gartner lost points for a lack of transparency in several areas related to both data and conflicts of interest. I was hoping that at this point the reader would have figured out that this, in fact, is the type of report that Gartner would produce. Now, certainly, Gartner analysts will not agree with the table and its findings. They may criticize individual criterion as not representative of good research practices (not a very good argument), but more likely, they will declare the entire exercise meaningless because Gartner cannot be compared to the other research entities as they are so unique. (Actually, Gartner would not score very well against other IT analyst firms. They would lose points against, say, Forrester because Forrester discloses their research data.). At least these are the arguments that I would expect.

However, if we pause for a moment, it should be clear that what I have laid out in these tables has a number of advantages over the research presented by Gartner. Here is how my research differs from Gartner’s:

  1. I explained the methodology for how I would determine the description of each criterion per entity (this has been laid out in the comparison sections of this book).
  2. The methodology of the scoring has been explained: it is based upon the descriptions.
  3. The actual complete research data set was shown and anyone can see the scores for each criterion.
  4. Although I have not yet made this declaration, it should be relatively obvious that I am not selling consulting services to Consumer Reports, RAND or the general academic community, and I have no conflict of interest that could pollute my results.

Therefore, while Gartner analysts may criticize the research at their leisure, the research adheres to a standard which no research produced by Gartner can claim to match.

[1] One of the things that made US universities the respected research institutions that they were was that the public universities had a single contributor: the US government. The US government has a history of a long-term commitment to research without the necessity for a commercial outcome. Corporations cannot come close to matching this commitment. However, as the US university system increasingly is privately funded, conflicts of interest become more common. These quotations are from Cal State University website:

“At a recent debate before Stanford’s Academic Senate, respected law professor Hank Greely argued that the tobacco industry ‘has perverted academic research for its own ends in ways that have had horrific consequences.’

‘It hurts me that my university gives them cover and sustenance,’ said Greely, a co-author of the resolution. ‘They are using us to whitewash themselves.’

For decades, university-based research funding was a strategic element of the tobacco industry’s effort to whitewash its tarnished public image.

Its disingenuous research was a focal point of a landmark trial last summer. On Aug. 17, 2006, Judge Gladys Kessler of the Federal District Court for the District of Columbia ruled that the tobacco industry had engaged in a 40-year conspiracy to defraud smokers about the health risks of tobacco. As evidence, she cited industry-sponsored work by UCLA epidemiologist James Enstrom, who challenged the view that second-hand smoke poses a serious health risk.”

Search Our Other Gartner Content

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Gartner Information Contact Form

  • Interested in Research on Gartner?

    It is difficult for most companies to make improvements in using Gartner without outside advice. We offer remote unbiased multi-dimension Gartner support.

    • We have a better track record of being correct than any of the well-known brands.
    • If this type of accuracy interests you, contact us and we will be in touch.

References

Gartner Book

Gartner and the Magic Quadrant: A Guide for Buyers, Vendors, and Investors

Gartner is the most influential IT analyst firm in the world. Their approval can make or break a vendor in an application category, or at the very least control their growth. Gartner has been behind most of the major IT trends for decades. However, many people read Gartner reports without understanding how Gartner works, how it comes to its information, its orientation, or even the details of the methods it uses for its analytical products. All of this and more is explained in this book.

Table of Contents

  • Chapter 1: Introduction
  • Chapter 2: An Overview of Gartner
  • Chapter 3: How Gartner Makes Money
  • Chapter 4: Comparing Gartner to the RAND Corporation, and Academic Research
  • Chapter 5: The Magic Quadrant
  • Chapter 6: Other Analytical Products Offered by Gartner
  • Chapter 7: Gartner’s Future and Cloud Computing
  • Chapter 8: Adjusting the Magic Quadrant
  • Chapter 9: Is Gartner Worth the Investment?
  • Chapter 10: Conclusion
  • Appendix a: How to Use Independent Consultants for Software Selection
  • Appendix b: What Does the History of Media Tell Us About This Topic
  • Appendix c: Disclosure Statements and Code of Ethics