How Gartner Research Compares to Real Research Entities

 Ratings can be used can be found on the Office of the Ombudsman website.
Software vendors frequently publish their Gartner ratings (if they are good) on their websites, “tweet” them, and put them in press releases. So software vendors advertise their Magic Quadrant rating quite aggressively if they have done well.

The Problem with Allowing Vendors to Use the Ratings in Advertisements (And Charging Them)

At first, it may not seem obvious why it should matter if those rated use the rating in their advertising. The reason relates to the perception of endorsement made by the inclusion of a rating in such an advertisement and the strong tendency a vendor has in spinning the rating to their ultimate advantage. Essentially, once vendors begin using rating results in advertising, Consumer Reports can no longer control the presentation of its results.  Consumer Reports explains their policy in the following way:

“At Consumer Reports, we believe that objective, impartial testing, reviews, and ratings are critically important for consumers. That is why we have a strict “No Commercial Use Policy” preventing the use of our name and information for any promotional or advertising purposes. The policy helps ensure we avoid even the appearance of endorsing a particular product or service for financial gain. The policy also guarantees that consumers have access to the full context of our information and are not hearing about our findings through the language of salesmanship.”

Gartner Versus Consumer Reports on Controlling for Sample Bias

Consumer Reports buys all of the products that they rate. This allows them to be confident that the products they rate are the same products consumers would receive. If Consumer Reports did not do this and instead accepted free samples, these samples would deviate from what was available to consumers. The manufacturers would make sure that only the best were sent to be rated. Not only do IT analyst firms not buy the software they rate, but they also don’t test the software they rate.

Ratings come from discussions with the software vendor, reviewing demos and questionnaires filled out by buyers, evaluating the literature produced by the vendor, and talking to companies that have implemented the software.

Gartner’s Lack of Testing or Even “Touching” of the Products they Rate

Therefore, Gartner differs significantly from Consumer Reports in that they do not test the products they recommend. When Gartner does see the product, it is a demo presented by the vendor in an artificial environment. Gartner uses a script during these analyst briefings, which means that the vendor must show what Gartner has on its script and may not deviate from the script. This is done to provide a consistent rating methodology to each vendor. Still, it has a disadvantage because it prevents the presentation of what could be interesting and useful functionality. Several individuals who have participated in these analyst briefings have informed me through interviews that the demos are, in fact, lighter versions of customer demos, which are themselves light compared to a demo that would be presented to a software-oriented person such as myself.

Gartner’ Statements on Improving Ratings Through Purchasing Advisory Services from Gartner

[1] Gartner’s sales group implies to vendors that they will improve their rating if they purchase consulting services from Gartner; this fact has been independently verified in multiple interviews with people who work in senior positions out of the marketing department at software vendors. My confidence in this statement is reinforced by the fact that Gartner’s same phraseology was quoted to me by these individuals and the fact that none of these individuals knew one another.

The idea that one would pay money for anything to a normal research entity and the potential for the rankings in the research to improve would eliminate any study from being taken seriously in the academic realm. Consumer Reports do not do it. The RAND Institute discloses its funding in each study that it performs, but it does not put itself in that type of conflicted position in the first place.

How Gartner Research Compares to The RAND Corporation

RAND is either the world’s pre-eminent think tank or one of the top think tanks depending upon who you ask. RAND was created over sixty years ago before think tanks had acquired their poor reputation. In effect, RAND is a “real” think tank, following—and in many cases exceeding—academic research standards. Although they are highly productive, they employ only roughly 1,700 people. Upon occasion, I have read RAND’s research and was most impressed with their work quality. They were, in fact, a major innovator in an area of software about which I have written a book (Inventory Optimization and Multi-Echelon Planning Software), and their research was featured prominently in that book.

The RAND Corporation began its life primarily researching the Pentagon. Some of their best-known research has been in wargaming and strategies, which ranged from Vietnam (and from where was leaked the infamous Pentagon Papers) to the US anti-Soviet nuclear missile strategy. However, in the past few decades, they have diversified into a much broader research entity that performs research on everything from energy and the environment to health care. RAND also publishes an explanation of the standards they follow. I did not use all of the criteria published in this document because some of the criteria do not relate to Gartner; RAND serves a public service function, whereas Gartner does not. However, several of RAND’s standards were universally applicable, and they are discussed and compared to Gartner’s standards in the following paragraphs.

Gartner Versus RAND on Referencing Past Work

RAND communicates very clearly how its research is performed.

“Although internal discussions about research quality have always been an integral part of RAND culture, more than a decade ago, we decided to codify in writing the quality standards for all RAND research. We intend the written standards to serve both as a guide for those who conduct, manage, support, and evaluate the research activities at RAND and also as the set of principles by which our research units and programs shape their individual quality assurance processes.”

RAND’s Standards for High-Quality Research and Analysis publication makes the following statement regarding how its research references other work.

“A high-quality study cannot be done in intellectual isolation: It necessarily builds on and contributes to a body of research and analysis. The relationships between a given study and its predecessors should be rich and explicit. The study team’s understanding of past research should be evident in many aspects of its work, from the way in which the problem is formulated and approached to the discussion of the findings and their implications. The team should take particular care to explain the ways in which its study agrees, disagrees, or otherwise differs importantly from previous studies. Failure to demonstrate an understanding of previous research lowers the perceived quality of a study, despite any other good characteristics it may possess.”

Gartner’s research is very much encapsulated and is much like Consumer Reports in this regard. A Gartner analyst might say that, of course, they don’t work in isolation; they talk to vendors and client executives every day. However, that is not the isolation to which this statement refers. This statement refers to other research and is unrelated to whether one accesses data sources. I write this as explicitly as I can, knowing full well that a Gartner analyst does comment on this criticism in some public forum. They will also bring up the fact that they are not isolated and, therefore, I am mistaken and don’t understand how Gartner operates. However, I can only declare this several times and let the chips fall where they may.

Gartner Versus RAND on Transparency of Research Data

RAND’s position on the transparency of research data is taken from the same RAND publication as the previous quotation.

“Data and other information are key inputs to research and analysis. Data-generation methods and database fields should be clearly specified, and the data should be properly screened and manipulated. The research team should indicate limitations in the quality of available data. In addition, information presented as factual should be correct and verifiable.”

Gartner’s data is oftentimes not verifiable, for the main reason that frequently, the data is not published. However, to meet RAND’s standards, the data from research must always be published.

Gartner Versus RAND on the Use of Jargon

It was interesting to see RAND take this position on the use of jargon:

“Necessary technical terms should be defined and explained.”

Gartner tends to use jargon in its writing. I recall one report that stated that a vendor’s application performed “stochastic optimization.” I was curious about how many people who read the report actually knew what the term “stochastic” meant! Gartner basically assumes that the readers know all the terms they use, but I do not recall seeing definitions at the end of their research. Although terms can be easily searched on this day of the Internet, it is probably less of an issue than it was in the past.

Gartner Versus RAND on the use of Graphical Elements

RAND has this to say on the topic of graphical elements:

“To help explain complex and novel ideas, the documentation should augment textual exposition with graphical or pictorial elements.”

Gartner Versus the Academy on Peer Review

Gartner’s research is to a degree peer-reviewed before publication, but not with peers outside of Gartner. In academics, research is always peer-reviewed outside of the institution. According to Wikipedia: “Publications that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals.”

A lack of peer review can come across as a researcher having a particular bias or that a researcher was not confident that the findings could survive a peer review. One of the best examples of the mistakes that can occur when research is not peer-reviewed is the example of cold fusion. In 1989 Stanley Pons and Martin Fleischmann, along with the University of Utah, submitted a research paper to the journals Nature and TheJournal of Electroanalytical Chemistry —at the same time that they released a press release saying that they had discovered a way to create a nuclear reaction at room temperature. Pons and Fleishmann predicted to members of the media that:

“..cold fusion would solve environmental problems, and would provide a limitless inexhaustible source of clean energy, using only seawater as fuel. They said the results had been confirmed dozens of times and they had no doubts about them.”

The world was on its way to clean and limitless energy. There was only one problem: the results recorded by Pons and Fleischmann could not be replicated by other research teams that followed their methodology. The results, which led to the cold fusion rush, were eventually chalked up to poor research controls and concluded that Pons and Fleischmann had mismeasured their study. This research, which had not been peer-reviewed, led to several negative consequences, including many millions of dollars spent in the subsequent decade—mostly by private industry—to replicate the faulty results. However, if Pons and Fleischmann and the University of Utah had followed the normal academic protocol of waiting to announce their results until after their experiment had been peer-reviewed and attempts had been made to replicate the results, the term “cold fusion” would not be in the popular consciousness. This is because there would have been nothing to report.

The Issue Gartner Would Face if its Research Peer-Reviewed

Peer review works in the academic system because these institutions are part of a large ecology of universities that have worked this way for quite some time, and it is the accepted practice. It would be challenging for Gartner to have its research peer-reviewed. Who would review it: Forrester? These entities are all for-profit companies that do not share information with other companies— a major limitation with research conducted by private entities. The research of Consumer Reports is not peer-reviewed; however, its transparency publishes its reports makes Consumer Reports auditable. The lack of peer review, along with an inability to audit Gartner’s results, is a problematic combination, which will be explored in the conclusion of this chapter.

Gartner Versus the Academy on Publishing the Methodology

The methodology used for research is such an important part of the disclosure of scientific papers that it is part of the IMRAD acronym, which is commonly discussed in academics. This stands for:

1. Introduction
2. Methods
3. Results
4. (and)
5. Discussion

This is the standard order in which academic papers are presented.

Does Gartner Publish Its Methodology?

Several headings on Gartner’s website use the term “methodology.”

Some can be found below:

Supply Chain 25 Methodology

Gartner Methodology

These pages do provide information about the methodology, as does the research. Generally, it is clear, at least to me, what each of Gartner’s research reports is measuring. However, as a person who has read quite a bit of research, I would classify Gartner’s disclosure on methodology as very light. Essentially Gartner is misleading many of its subscribers (who do not have a research background), believing that this is a reasonable disclosure level.

For instance, let’s look at the methodology for the Magic Quadrant. The criteria used and which make up each axis are declared; however, are they all given equal weight, or are some weighted more than others? Gartner considers that disclosure of its methodology would not be considered a full disclosure by any of the research entities that I have discussed up to this point. Gartner’s results, as published to subscribers, would never be published in any journal. Not all IT analysts follow this concealed approach. Forrester is one. Donald Ham explains how Forrester’s Wave (a competitor to Gartner’s Magic Quadrant) discloses much more to subscribers and allows subscribers to gain more value from the research.

“Forrester’s Wave product, their graphical vendor comparison tool, is similar in approach to the well-known Gartner Magic Quadrant.  However, the wave lets end users change relative weights of the comparison criteria by downloading and interacting with an Excel spreadsheet, resulting in a graphical display customized for the specific need­—a very cool enhancement.  Forrester is not as prolific with Wave reports as Gartner is with MQs, but where they exist for the products you’re investigating, points go to Forrester.”

Gartner’s Degree of Methodological Opacity Versus an IT Entity Like Forrester

The following table shows the criteria, the weights, and the data from Forrester’s Wave on BI Service Providers.

A good example of far more complete disclosure of the methodology is available at the website for statistics for the Netherlands. I have included this link so that readers can gain an understanding of what I am describing.

ZL Tech’s CEO Kon Leong uses the following argument. The company referred to previously sued Gartner for damaging its business by unfairly ranking its products versus competitor’s products because of the company’s smaller size (its lawsuit was eventually rejected).

“The tech industry would benefit if Gartner were required to disclose more data in its evaluation process and disclose component scores, so vendors know exactly where they are lacking and by how much and take corrective action.”

It is important to consider that Gartner’s target market is not researchers but rather decision makers who are buyers, vendors, and investors. Many (but not all) of the people in these groups will be more interested in simply seeing the results. Many questions how many people who read and rely on Gartner’s reports read the entire report, much less understand the research methodology. This gets back to the main theme throughout this book—that Gartner’s research is not particularly useful without the context provided by Gartner analysts. However, because the privately provided information is not part of the public record, it is also not auditable.

How the Methodology Is Interpreted

The methodology is one of the most overlooked areas of Gartner by those who read its research. The criteria are often not what one would expect to be used. If you speak with most people about Gartner’s methodology, it turns out they don’t know the methodology. I can say this with confidence because when I tell interviewees what makes up the criteria for different analytical products, I frequently receive the response: “Is that what is counted by Gartner?”

Gartner has stated that readers are too quick to review a single graphic and not read the entire report. On the other hand, the way Gartner comments on its research is also responsible for providing the wrong impression regarding their methodologies, as I will demonstrate. I explain this in Chapter 5: “The Magic Quadrant.” However, for the sake of continuity, let me provide another example: Gartner’s “Supply Chain Top 25.”

The Gartner Supply Chain Top 25

Gartner publishes a “Supply Chain Top 25,” which most people would assume is the twenty-five companies with the highest-performing supply chains. This is the highest performance from the metrics normally associated with supply chain management. These are things like forecast accuracy, inventory turnover, service level, and other similar measures). However, upon reading Gartner’s methodology for creating the Supply Chain Top 25, only one of these criteria is used. The criteria are instead:

1. Return on Assets (ROA) — Net income / total assets
2. Inventory Turns — Cost of goods sold / inventory.
3. Revenue Growth — Change in revenue from the prior year.
4. Gartner’s Internal Voting
5. Gartner’s Client’s Voting

Two of the criteria that determine the Supply Chain Top 25 do not have anything to do with the supply chain but have to do with financial performance! However, when I read Gartner’s explanation of the research to media outlets, I found it did not match the methodology. The quotation below is an example of this.

“At the heart of the Supply Chain Top 25 is the notion of demand-driven leadership,” said Debra Hofman, managing vice president at Gartner. “We’ve been researching and writing about demand-driven practices since 2003, highlighting the journey companies are taking: from the old ‘push’ model of supply chain to one that integrates demand, supply and product into a value network that orchestrates a profitable response to ever-shifting changes in demand.”

Debra Hofman’s explanation is not the primary methodology of the study. Instead, the methodology focuses on the overall firm’s financial performance combined with people voting for who they thought should be on the list. As I’ve explained, the heart of the study is the methodology, not the notion of “demand-driven leadership.” The only control over whether a company showed “demand-driven leadership” is the one criterion Gartner voted on. The other four criteria could not have been related to this.

Furthermore, the definition of “demand-driven” is quite hazy. I work in the field and am unsure as to whether I have heard the term used before. Furthermore, the term “demand driven” is not used a single time in the report itself. If it is such an important component of the report, why is it not part of its methodology?

The Overall Scores

In the tables below, I have organized the scores of Gartner and each of the comparison entities. The tables have a few more criteria than included in this chapter. These criteria were excluded from the text to keep the chapter to a manageable length. The tables rank all entities against Gartner on every criterion:

Here you can see a description of how each entity performed in each criterion. Based upon this description, I have assigned a score, which is shown in the tables below: 

How the Scoring Works

1. The scoring methodology is that the higher the score, the better. The entity receives a ten if the issue does not apply to them. For instance, Consumer Reports does not receive any funding, so the issue of reporting industry funding does not even come up.
2. The zero and ten scores were simple to score; however, the scoring required some judgment or subjectivity when the criteria were not binary. Of the four entities, academic research was the most difficult to score because, while there are generally-accepted research principles, there is considerable variability in their adherence. Not only is there the issue of biomedical research, which operates quite differently from other types of academic research, but there is the issue of public versus private universities, among just a few of many variable factors.
3. Given RAND’s excellent reputation for impartiality and transparency, I was surprised that I could not find any record of financial contributions or even ranking of donors from large to small. In all the research entities that I studied, it is difficult to determine exactly how much money is contributed by each entity. Although there is no question of which entity funds each of RAND’s research reports and RAND lists its contributors (which is far more transparent than Gartner), this disclosure is still below what I would like to see. To see how transparent RAND is on this topic, I have included below an example of the statement incorporated on the front of each paper:

“This research is supported by the United States Air Force under Project RAND—Contract No. F44620-67-C-0045—monitored by the Directorate of Operational Requirements and Development Plans, Deputy Chief of Staff, Research and Development, Hq USAF. Views or conclusions contained in this study should not be interpreted as representing official opinion or policy of the United States Air Force.” – Descriptions of The Computer Program for Metric – A Multi-Echelon Technique for Recoverable Item Control

This clarifies who funded the study, even if we are not told how much funding was received.

Disclosure of Financial Contributions in the Academy

Academic research is far hazier in listing contributors, as the university contributors are never actually listed on the reports. Instead, the money tends to go into a big pot for each university department. But it would be difficult to doubt its corrupting influence. One of the more amusing contributions from the industry was the $100 million that Exxon Mobile paid Stanford to fund climate and energy research. One can imagine what type of “research” Stanford will be doing with this money and how many times this grant will be mentioned on Exxon Mobile marketing material.[1]

Because Consumer Reports is subscriber-supported, no one in the comparison group could match Consumer Reports in their lack of interest conflicts. I did not weigh the criteria as I would normally do in a software selection matrix but instead considered each criterion equal in weight. The criteria should not be equally weighted, but I wanted to keep the presentation of the table simple.

Reasons for Gartner’s Low Score

The scores were quite interesting; as you can see from the tables on the previous page, Gartner scored quite low compared to the other three research entities.

Gartner lost points for lack of transparency in several areas related to both data and conflicts of interest. I was hoping that the reader would have figured out that this is the type of report that Gartner would produce at this point. Now, certainly, Gartner analysts will not agree with the table and its findings. They may criticize individual criterion as not representative of good research practices (not a very good argument). More likely, they will declare the entire exercise meaningless because Gartner cannot be compared to the other research entities as they are so unique. (Actually, Gartner would not score very well against other IT analyst firms. They would lose points against, say, Forrester because Forrester discloses their research data.). At least, these are the arguments that I would expect.

However, if we pause for a moment, it should be clear that what I have laid out in these tables has several advantages over the research presented by Gartner. Here is how my research differs from Gartner’s:

1. I explained the methodology for determining the description of each criterion per entity (this has been laid out in the comparison sections of this book).
2. The methodology of the scoring has been explained: it is based upon the descriptions.
3. The actual complete research data set was shown, and anyone can see the scores for each criterion.
4. Although I have not yet made this declaration, it should be relatively obvious that I am not selling consulting services to Consumer Reports, RAND, or the general academic community. I have no conflict of interest that could pollute my results.

Therefore, while Gartner analysts may criticize the research at their leisure, the research adheres to a standard that no research produced by Gartner can claim to match.

[1] One of the things that made US universities the respected research institutions was that the public universities had a single contributor: the US government. The US government has a history of a long-term commitment to research without the necessity for a commercial outcome. Corporations cannot come close to matching this commitment. However, as the US university system is increasingly privately funded, conflicts of interest become more common. These quotations are from the Cal State University website:

“At a recent debate before Stanford’s Academic Senate, respected law professor Hank Greely argued that the tobacco industry ‘has perverted academic research for its own ends in ways that have had horrific consequences.’

‘It hurts me that my university gives them cover and sustenance,’ said Greely, a co-author of the resolution. ‘They are using us to whitewash themselves.’

For decades, university-based research funding was a strategic element of the tobacco industry’s effort to whitewash its tarnished public image.

Its disingenuous research was a focal point of a landmark trial last summer. On Aug. 17, 2006, Judge Gladys Kessler of the Federal District Court for the District of Columbia ruled that the tobacco industry had engaged in a 40-year conspiracy to defraud smokers about the health risks of tobacco. As evidence, she cited industry-sponsored work by UCLA epidemiologist James Enstrom, who challenged the view that second-hand smoke poses a serious health risk.”

The Problem: Thinking that Gartner is Focused on What is True

Gartner is hired by companies who fundamentally don’t understand how Gartner functions. Gartner has virtually no first-hand experience in the technologies they evaluate and get most of their information from executives at buyers or executives at vendors and consulting firms. Gartner is also not a research entity. They compare very poorly to real research entities once you dig into the details, as we did in the article How Gartner’s Research Compares to Real Research Entities. Gartner serves to direct IT spending to the most expensive solutions as these are the companies that can afford to pay Gartner the most money. Gartner has enormously aggressive internal sales goals that place accuracy far below revenue growth in importance.