How Gartner Research Compares to Real Research Entities

Executive Summary

  • Gartner positions itself as a research entity. However, when compared with real research entities, Gartner does not perform well.
  • We compared Gartner against academic research, RAND, and Consumer Reports to see how well they performed. Most people who read Gartner’s output will be surprised by how they scored.

Video Introduction: How Gartner Research Compares to Real Research Entities

Text Introduction (Skip if You Watched the Video)

Gartner is frequently referred to as a company that does research and is considered as such by those that purchase research and advisory services from them. However, aside from domain expertise, what makes an entity a research entity is the following long-established rules regarding research. These rules are very well understood among those that perform research. For example, at Brightwork Research, we know them, and we follow them. What often goes without mention is whether Gartner follows these research rules. You will learn how Gartner functions and how research, rather than commercial considerations, plays in how Gartner manages its business.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Lack of Financial Bias Notice: The vast majority of content available on the Internet about Gartner is marketing fiddle-faddle published by vendors who republish reports they paid Gartner to publish, or Magic Quadrants they paid Gartner to score well. The IT industry is petrified of Gartner and only publishes complementary information about them. The article below is very different.

  • First, it is published by a research entity.
  • Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services as a vendor or consulting firm that shares their ranking in some Gartner report. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. 

How Gartner Research Compares to Consumer Reports

I decided to evaluate Gartner’s research output by comparing Gartner’s rules and business practices to respected research rules. A comparison of Gartner’s research approach to that used by other analyst firms may seem like the long way around to understanding Gartner’s research output. Still, I believe this chapter will greatly improve one’s understanding of Gartner’s research. While I have heard Gartner’s various criticisms, I have never seen Gartner compared against respected research entities on essential criteria. That is what is accomplished in this chapter.

One of the comparisons made in this chapter is between Gartner and the best-known rating company in the world: Consumer Reports. The second comparison is with one of the most respected think tanks globally—the RAND Corporation—which has established a reputation for impartiality and sometimes groundbreaking research going back six decades. The third comparison is not with a single entity but between Gartner and the academic research system. Most research institutions in the US, Europe, and Australia/New Zealand subscribe. This chapter is essential as it sets the stage for later chapters. My research for this book highlighted that far too frequently, Gartner is discussed in isolation without the broader context of generally-accepted research practice.

Gartner Versus Consumer Reports on Funding from Rated Entities

Consumer Reports is the most trusted consumer rating and product-testing agency in the US. Consumer Reports has been continually published since 1936. Consumer Reports has a very structured and well-designed set of rules that strictly limits the influence of those whose products and services are rated by them. Consumer Reports takes no advertising or other forms of payments from those it reviews; instead, it is wholly supported by its subscribers and is the largest subscription-supported website in the US.[1] Compare this to Gartner, which takes money from those companies that it rates, actively solicits more business from current vendors who are customers. And solicits vendors who are not customers with what is generally known as aggressive sales tactics, which stop just short of promising better results in their ratings.

Consumer Reports takes no advertising or other monies from those that they rate is a critical point. Their policy is different from JD Power and Associates, which is another well-known rating company. JD Power and Associates not only charge the vendors they rate but charges vendors to advertise the JD Power and Associates award that they “won.” They rate the winners in a category but do not provide a complete list of the contestants, which is a nod to the vendors who did not perform well. Among a host of others, these factors are why JD Power and Associates is not seen as a serious research entity, and their principle usage is not by consumers but instead as something that advertisers use to create the illusion of a good rating with consumers.

Gartner Versus Consumer Reports on Vendor Use of Ratings

Consumer Reports also has a non-commercial use policy, which means that companies rated well by Consumer Reports may not use the Consumer Reports rating in advertising or even on their website. Generally speaking, this is the opposite of IT analyst firms and also from Gartner. Gartner has some restrictions on how the ratings can be used, but the limitations are quite liberal. The last reference to the rules regarding how Gartner’s r