Is Gartner Correct That They Are Unbiased?

Executive Summary

  • Gartner proposes it has no bias, but it takes billions in revenue from software vendors.
  • Gartner stacks the deck in favor of large vendors and has many dimensions of bias, all of which is undeclared.

Introduction

In 2006, InformationWeek wrote an article titled The Credibility of Analysts. In this article, IW brought up a very interesting point about Gartner.

Gartner and Bias

Gartner often proposes that it has no bias. The following quotation is a very typical example of this.

Bias is a nonissue at the company, CEO Hall insists. “We wouldn’t have a dollar of revenue from the user community if our objectivity and independence weren’t held in high regard,” he says. Gartner has policies in place meant to ensure objectivity. The ombudsman office reports to Gartner’s general counsel to ensure it’s free from pressure from other parts of the company. And Gartner analysts aren’t allowed to own stock in the companies they cover.

None of those policies are verifiable or auditable. The only policy that Gartner has in place is its ombudsman, which is covered in the article How to Best Understand the Gartner Ombudsman

The Problem with the Argument Based On Popularity

Secondly, Hall’s argument is the same used by the University of Phoenix which it used to defend itself against charges of poor outcomes for its students. Their logic was the following. Because students chose the University of Phoenix it must be good.

The argument used by Gartner is circular.

  • Gartner has no bias because it is popular
  • Because Gartner is popular, it must have no bias.

Let’s see how some other companies could use this same logic.

  • If McDonald’s did not offer healthy food, they would not get any people buying meals from them.
  • So many people with money would not have invested their money with Bernie Madoff if there was anything unusual going on.
  • Volkswagen would not have been able to sell its “clean diesel” to so many customers if their cleaning technology were a fraud.
  • If political parties were corrupt, no one would vote for them.

The Logical Fallacy of Popularity

This argument is so well known to be false that it is categorized as a logical fallacy. Specifically, it is the logical fallacy of “argument by number.” That is if a lot of people believe something to be to be true, it must, therefore, be true. It leaves out the very obvious alternative that those people might be either deluded or misinformed. It is explained by Wikipedia as the following:

“This type of argument is known by several names,[1] including appeal to the masses, appeal to belief, appeal to the majority, social justice, appeal to democracy, appeal to popularity, argument by consensus, consensus fallacy, authority of the many, bandwagon fallacy, vox populi,[2] and in Latin as argumentum ad numerum (“appeal to the number”), fickle crowd syndrome, and consensus gentium (“agreement of the clans”). It is also the basis of a number of social phenomena, including communal reinforcement and the bandwagon effect.”

This type of argument should be a concern to anyone using Gartner opinions (what Gartner calls its research in court documents) because the fallacy of argument by number is primarily used by those that have no evidence to provide.

The second problem with Hall’s statement begins before on even gets to the argument by number. And that is that many people, both in the user community, but particularly in the vendor community question Gartner’s objectivity. (therefore, Hall was able to cram two falsehoods into a single sentence which only had 22 words in it.) Gartner has been sued by vendors likes ZL Technologies and NetScout. Yet, in Hall’s statement, the concept is that Gartner’s objectivity is never questioned.

Did these lawsuits happen or didn’t they?

Gartner’s Objectivity is Unquestioned?

So the presumption that their objectivity is not questioned is false. Many vendors would publish this fact, however, they fear reprisal on the part of Gartner. Gartner is now so powerful, primarily through the combination of a corrupt business model along with an unending stream of acquisitions — it now functions as a monopoly in the IT analyst space.

Vendors know that Gartner is pay to play, and if they can afford to, they normally play. Vendors publish the latest Magic Quadrant results with aplomb (something they pay Gartner for the privilege of doing). Vendors are looking for any edge they can get in sales pursuits. But that does not necessarily mean that they take the MQ results seriously themselves internally.

The History of Media Bias

Historical media analysis tells us that media output tends to align with the interests of those who control it and pay for it. As an example, the primary reason that advertised television tends towards so much unchallenging programming is that challenging programming puts viewers in a critical mindset, making them less susceptible to advertising. The best programming to put viewers in the correct mindset for suggestibility is lighthearted fare.

Evidence that smoking advertisements, or they payments for them adjusted the media’s coverage of the dangers of smoking is now well-established. Smoking was at least suspected of causing cancer before the following ad ran in 1931, and yet ads like this, with doctors promoting certain brands, continued to run for decades after the industry knew the relationship by the late 1950.[1] Cigarette advertising was shown to relate directly to the number of anti-smoking stories. In effect, for decades cigarette advertising slowed the dissemination of information about the real risks of smoking.

The story repeats itself today as Exxon-Mobile has made contributions to the Heritage Foundation and the National Center for Policy Analysis (NCPA). Unsurprisingly, both entities have published “misleading and inaccurate information about climate change” according to Bob Ward, the policy director at the Grantham Research Institute on Climate Change and the Environment at the London School of Economics.

As pointed out by the Stanford School of Medicine,

“Unlike with celebrity and athlete endorsers, the doctors depicted were never specific individuals, because physicians who engaged in advertising would risk losing their license. It was contrary to accepted medical ethics at the time for doctors to advertise, but that did not deter tobacco companies from hiring handsome talent, dressing them up to look like throat specialists, and printing their photographs alongside health claims or spurious doctor survey results. These images always presented an idealized physician—wise, noble, and caring. This genre of ads regularly appeared in medical journals such as the Journal of the American Medical Association, an organization which for decades collaborated closely with the industry. The big push to document health hazards also did not appear until later.”

Evidence of Advertisers Interfering in Media Content

The paper What Do the Papers Sell? The Model of Advertising and Media Bias, as well as other papers on the influence of payments on media, demonstrates a consistent conclusion.

“The regulatory view grew out of evidence that some advertisers seriously interfere with media content. Baker (1994) and Bagdikian (2000) detailed accounts of the history of suppression of news on tobacco-related diseases. Complementing this evidence, Warner and Goldenhar (1989) statistically identify tobacco advertising as causing the reporting bias (for further evidence, see e.g., Kennedy and Bero, 1999). Another more recent case is misreporting on anthropogenic climate change. Boyko and Boyko (2004) demonstrate a clear bias in the US quality press over 1988-2002 (see Oreskes, 2004, on the scientific benchmark). Automotive advertising has been signaled as a key explanatory factor: in the US in 2006, automotive advertising alone accounted for $19.8 billion, of which nearly 40% went to newspapers and magazines.” – Advertising Age, 2007

So there are consistent findings in other media outlets that payments from subjects influence editorial content. More observations can be found in the well-known paper (in media analysis circles at least) Do Ads Influence Editors? Advertising and Bias in the Financial Media:

“For their part, media outlets tend to strongly deny that such a pro-adviser bias exists. For example, a 1996 article in Kiplinger’s Personal Finance printed statements from editors at a number of personal finance publications (including the three in our study) claiming that advertisers have no influence over published content. In this paper, we test for advertising bias within the financial media. Specifically, we study mutual fund recommendations published between January 1997 and December 2002 in five of the top six recipients of mutual fund advertising dollars. Controlling for observable fund characteristics and total family advertising expenditures, we document a positive correlation between a family’s lagged advertising expenditures and the probability that its funds are recommended in each of the personal finance publications in our sample

(Money Magazine , Kiplinger’s Personal Finance , and SmartMoney ). While we consider several alternative explanations below, the robustness of the correlation leads us to conclude that the most plausible explanation is the causal one, namely, that personal finance publications bias their recommendations—either consciously or subconsciously—to favor advertisers.”

Advertising affects editorial decisions; this is a clear and unequivocal finding.[1]

The Problem with Payments from Industry

This issue is not merely restricted to companies that create ratings, but to all forms of media. Media outlets that are completely independent financially from those that they report on have a much better ability —both in theory and in actuality—to stay independent from the industries that they cover. Prior to the stock market crash of 1929, many reporters for financial publications would take direct payments to write glowing reviews/predictions of stocks by financial manipulators. The manipulators would buy a number of shares at a low price and then sell their shares as soon as the article came out. This was so common that it had a specific name: “pump and dump.” Many similarly corrupt arrangements were behind many of the financial panics prior to and since 1929. The corruption of the general financial press and of rating agencies plays a decisive role in financial panics and in a small minority getting very wealthy. Having a prestigious brand name is no protection against corruption; in fact, the most prestigious brands have the greatest ability to both drive and benefit from financial bubbles.

The Example of Buying Financial Ratings

Anyone who thinks that firms that do this will eventually be made to pay for the “market,” only have to look as far as Moody’s, Standard and Poor’s, and Fitch — three rating agencies that rated billions of toxic assets as the highest investment grade. They are still doing very well, thank you very much. I have a side interest in economics, and what I have found is that the smaller media outlets that are less known and take in very little revenue consistently outperform the much larger media and better-known outlets that do take advertising. Very simply, money corrupts the media product.

[1] With some uncommon candor, a French TV executive opened up about the actual purpose of his television channel.

“[French TV channel] TF1’s job is to help a company like Coca-Cola sell its products. For a TV commercial’s message to get through, the viewer’s brain must be receptive. Our programs are there to make it receptive, that is to say to divert and relax viewers between two commercials. What we are selling to Coca-Cola is human brain time.” What is surprising to most people who buy magazines is that they are not the only “payer” for the product. The consumer buys and pays for the magazine at the newsstand; however, advertisers pay in effect to have the magazine produced. The percentages of overall revenues are shown in the following quotation. “Mainstream US newspapers generally earn over 50 and up to 80% of their revenue from advertising; in Europe, this percentage lies between 30 and 80%, e.g., averaging 40% in the UK (see e.g., Baker, 1994; Gabszewicz et al., 2001). Overall, advertising exceeds 2% of GDP in the US and a substantial fraction of this becomes media revenue: 17.7% to newspapers, 17.5% to broadcast TV, 7.4% to radio and 4.6% to consumer magazines.” What Do the Papers Sell? The Model of Advertising and Media Bias: Advertising Age, 2007.

Two, Not One Customer Base

In this way, many media outlets can be seen to have two customer bases: their audience and their advertisers. That is they receive income from both the buy and sell side. This is similar to Gartner. However, because advertisers are much more concentrated, advertisers have even more power than the percentage of revenue they contribute to the media outlet. For instance, one member of the audience cannot seek to influence the media product. In fact, even ten percent of the audience would not be effective in influencing the media output; audience members do not coordinate with one another so their influence is diffused. However, a single advertiser can influence the media product, and the larger the advertiser the more its ability to influence the media entity.

[1] “Evidence linking smoking and cancer appeared in the 1920s. Between 1920 and 1940, a chemist named Angel Honorio Roffo published several articles showing that cancers could be experimentally induced by exposure to tars from burned tobacco. Roffo et al. further showed that cancer could be induced by using nicotine-free tobacco, which means that tar, with or without nicotine, was carcinogenic. Research implicating smoking as a cause of cancer began to mount during the 1950s, with several landmark publications in leading medical journals. The first official U.S. government statement on smoking and health was issued by the Surgeon General Leroy Burney in a televised press conference in 1957, wherein he reported that the scientific evidence supported cigarette smoking as a causative factor in the etiology of lung cancer.”

Financial Bias

In 2006, InformationWeek wrote an article titled The Credibility of Analysts. In this article, IW brought up a very interesting point about Gartner and the overall analyst community.

“Research firm executives are well aware of the questions being raised about their business models, but don’t expect changes to be fast or wide-sweeping. The financial stakes are too high — and the incentives for change aren’t compelling enough.”

Yes, research firms make great money, and while some may question the absolute lack of transparency in the IT analyst industry, companies keep buying these biased reports that fail to even mention where the funding from vendors come from. So since not enough people are complaining, why follow normal research rules?

Bias Removal?

In Chapter 9 of the book Supply Chain Forecasting Software (which is a chapter dedicated to the study of bias), I analyze bias removal. This was a very interesting area of study, partially because of the enormous discrepancy between the reality of bias and the interpretation of bias. Humans have so many areas of bias, including any type of perception or forecasting.[1] For instance, humans have a well-known optimism bias. There is confirmation bias— the selective use of information to support what one already believes to be true and rejecting information that contradicts one’s hypothesis.

These are the unconscious biases that are part of the human condition. Then we get into social and institutional biases.  In studying the bias of financial analysts for the book Supply Chain Forecasting Software, I found a detailed explanation of how analysts biased their forecasts to achieve career advancement. A good example of forecast bias, which is produced by institutional financial incentives, is described below:

“Sell-side analysts are pressured to issue optimistic forecasts and recommendations for several reasons. First, their compensation is tied to the amount of trade they generate for their brokerage firms. Given widespread unwillingness or inability to sell short, more trade will result from a ‘buy’ than from a ‘sell’ recommendation. Second, a positive outlook improves the chances of analysts’ employers winning investment banking deals. Third, being optimistic has historically helped analysts obtain inside information from the firms they cover (underline added). While all these pressures introduce an optimistic bias to analysts’ views, the magnitude of the bias is held in check by reputational concerns. Ultimately, an analyst’s livelihood—the ability to generate trades and attract investment banking business—depends on her credibility.” – Anna Scherbina

“Analysts will set the optimistic bias at an optimal point that balances the benefit of being upbeat against the cost to their reputation.” – Anna Scherbina

The Pressure for Optimistic Forecasts

Here the case is made that financial forecasters must trade off pressures to create optimistic forecasts with their concerns for their reputations. In this way, the forecast of a financial analyst can be seen as less of a forecast and more of a balancing act; they attempt to develop numbers that garner favor with the powerful companies from whom the analysts’ investment banks gain business while keeping some semblance of credibility with investors. This “credibility” also determines whether “information channels” are kept open or closed and highlights how political factors can influence a financial analyst’s forecast.

We hear the term “unbiased” quite frequently. However, when one analyzes the output of individuals and institutions, we find bias to be universal. The best that can be hoped for is that financial bias is reduced—but even this is incredibly rare. Here I would like to borrow from noted philosopher and linguist Noam Chomsky,  who said,

Everyone has a bias, the honest people tell you what their bias is. People that are not honest say they have no bias.”

Gartner’ Self Ascribed Reputation for Objectivity?

I have had several conversations with people at Gartner (years before I decided to write this book) during which they repeatedly stated that they believe they have a reputation for being objective. However, this subject is much greyer than Gartner would have it according to my conversations with many people and from publicly-available information on the Internet. Questions of bias plague Gartner, but it should be understood that this conversation is held chiefly among those who are the most sophisticated in their understanding of Gartner —usually those who work in marketing within vendors. Among investors and software buyers, the controversy regarding Gartner’s bias is not prevalent. In fact, among the individuals who are executive decision makers in software buyers, I do not see the topic as even a minor point of conversation.

Gartner Defending Gartner

It would be good if Gartner had more defenders than current or ex-Gartner analysts only. I have observed that the more experienced and savvy the individual, the less he or she is convinced that Gartner is objective. Those who are most critical of Gartner’s objectivity are those who work for the best-of-breed vendors or smaller vendors.

I can say with confidence that this criticism is not sour grapes on the part of these individuals, who are with smaller vendors belonging to the software category in which I specialize. I have used the software of these vendors as well as the software of the larger software vendors, and in each case, the software of these smaller vendors is far superior to software offered by the larger vendors (which, as stated previously, is ranked higher than that from these smaller vendors). Vendors that provide “point solutions,” (something which is considered a negative but really should not be) have a legitimate complaint in that the Gartner methodologies favor both large vendors and vendors with broad suites. I will discuss this topic in detail in Chapter 5: “The Magic Quadrant.”

Gartner’s IT Bias

Gartner tends to tailor its writing in a way that suits the interests of IT. For some time, IT’s control of software selection decisions has been increasing. Gartner’s growth has coincided with a reinforced influence of IT over software selection. This is demonstrated by Gartner’s diminished focus on the application and amplified focus on things like integration, reducing the number of vendors from which purchases are made, etc. IT simply has different incentives than the business. When I discuss the inability of software (which the company has just purchased) to do the job, the business is all ears while IT does not want to hear about it. This point is brought out well in the following quote:

“They think it’s better to have fewer software contracts to manage than it is to have the best technology for the business problems they face. Companies should buy the best software for the job, not because it’s software from the vendor they already use. That’s just plain lazy and bad business.” – Christopher Koch, CIO Magazine

Gartner’s IT Versus Business Bias

Within buyers, Gartner’s reports also have multiple categories of customers, the two most prominent being IT and the business. IT tends to be pro-Gartner because IT does not like to deal with many vendors, many service contracts, etc. Therefore, IT has been one of the main proponents of purchasing software from fewer firms. This policy alleviates the pressure faced on integration applications (although much less than generally anticipated)[1], but has numerous downsides with respect to implementation success. When a company restricts its buying alternatives to fewer software vendors, the business loses because no single vendor offers the best solution for even a small fraction of the software categories. What makes a solution the “best solution” is not only the general functionality of an application but includes how buying from more software vendors makes available to the business more applications that are suited to its industry and to the requirements of the particular buying company.

Thus, a mix of competing interests makes decisions that are not necessarily based on evidence or logic, but upon which department or grouping has the most power. For whatever reason, during the last several decades, IT has tended to get its way more often than not in software selection.

How Gartner Pushes Buyers Towards More Expensive Solutions

Because Gartner prefers solutions from larger vendors, Gartner also tends to push buyers towards more expensive solutions. This higher expense is not only for software but also for services. For instance, I work as an SAP consultant, and SAP consultants are some of the most expensive resources in IT. One aspect of the cost is the hourly billing rate, which is high. Another aspect is that SAP software is complicated to install, so SAP projects tend to be quite long. The billing rate multiplied by the total number of hours is of course, the consulting cost. The extent to which the cost to implement SAP is higher than to implement “best-of-breed solutions” is shown in the articles below. In these articles (knowing that I would receive a great deal of negative feedback), I provided an advantage to SAP by setting their software costs to zero.[2]

In terms of the overall costs, the implementation consulting costs far exceed the cost to purchase the product. (Therefore, it makes little sense to simply focus on the cost of acquiring software. Instead of a total cost approach, which includes implementation and maintenance costs, makes the most sense.) These consulting resources can come from the software vendor or from a consulting company. The software that is selected in large part determines the cost of the consulting that will follow. The differences in the total costs between large vendors and small vendors are quite significant, which is why placing them in the same category is problematic. This would be as if Consumer Reports placed Lexus automobiles in the same category as Toyota’s automobiles.[3] Lexus would outscore all of the Toyota products because Lexus uses upgraded components, paint, engines, etc. However, a review of the highway will show that Toyota sells many more cars than Lexus because price is an important consideration in the decision to buy an automobile. This is intuitively obvious: differently priced items should be compared in different categories. If I am looking for an automobile and my budget is twenty thousand dollars, I am quite aware that a forty thousand dollar car is probably better. I wouldn’t care so much as that car is out of my price range. If the costs of acquisition—as well as implementation and maintenance—were included in some type of “value matrix,” the matrix would look quite a bit different and the larger vendors would drop significantly in the rankings.

While Gartner’s Magic Quadrant rankings do not account for cost, sometimes the vendor descriptions do. For instance, in the Magic Quadrant for Business Intelligence and Analytical Platforms for 2013, which must be one of the longest and most thorough Gartner reports ever written, costs are discussed in several vendor profiles. The following quotations are examples [emphasis is mine]:

  • “Licensing cost remains a concern. When references were asked what limits wider deployment, 37.78% indicated the cost of the software, compared with the industry average of 25.4% across all vendor references.
  • When compared with Actuate, Quiterian has some concerns, including cost, ease of use for business users, ease of use for developers, support quality and product quality, all ranked below the survey average.”

Understanding Gartner’s Elitist Bias

Gartner has an elitist orientation and this is demonstrated in several ways. The more I analyzed Gartner and compared it to other IT analysts, the more this characteristic became apparent. I have listed some of the ways in which Gartner is elitist below:

  1. Gartner’s analysts deal with the most senior members of buyer and vendor companies.
  2. Gartner shares very little of its research for free, even for promotional purposes. The one exception to this is their market updates and predictions, which are selectively released as press releases. Gartner is effective at preventing the free dissemination of its research. By comparison many companies, including many IT analyst firms, have a blog of some type, which distributes some free analysis. Gartner does not do this (see the example below from Gartner’s website). Unless part of Gartner’s research is republished through a partner, their work is tightly controlled. This subject is one of the most interesting parts of how Gartner operates and will be discussed in detail in several places in this book.
  3. Many IT analysts sell research by the article. For instance, you can search through Forrester’s website and if you find an article that you like, you can buy just that article. Gartner does not allow nonsubscribers to search through their research database or, in most cases, to buy single articles. Gartner’s model is to sell subscriptions as a starter or an introduction to their analyst consulting services.
  4. Gartner seems to have an unlimited number of levels of service which one can sign up for, each promising more access to analysts and to more inside information. This exclusivity seems to increase the perceived value of research in the eyes of some clients. Gartner is proficient at marketing its upgraded services. For instance, criticism on the part of clients regarding satisfaction with their current service is quickly deflected by Gartner; whatever the client is dissatisfied with or whatever the client seeks is simply offered at the next level in service. Gartner’s employees must be trained in this response because I have heard it used as a reflexive defense against criticisms that have nothing at all to do with the level of services purchased from Gartner.
  5. All of Gartner’s research, consulting and events are expensive. This means that not only does one have to pay to participate with Gartner, but the price of admission is quite high.
  6. Gartner’s broadcasting approach does not allow for user commentary, which also means that subscribers cannot read what other subscribers are thinking about the research areas. Most articles published today on the web (and this is particularly true of technology articles) allow comments. However, the communication on Gartner’s articles is a one-way affair.

Because much of Gartner’s advice to clients is not in a published form, only those with the budgets get to learn what Gartner analysts “really think.”

The Dimensions of Gartner’s Bias

Because they are so numerous, it is difficult to keep track of all of the dimensions of Gartner’s bias. For this reason, we created a graphic that lays out how Gartner’s bias impacts their output.

The Brightwork Graphic of Gartner’s Biases

How Gartner’s Writing Style is Meant to Appear Unbiased

All of the Gartner reports that I have read are well written, and much like The Economist, they keep a consistent tone and writing style despite the fact that a report could have been written by any one of Gartner’s analysts. The way to ensure this consistency is primarily to have good internal training for analysts and to make sure that a group of internal copyeditors go through all the reports written by the analysts. While the writing is consistent, reports from category to category and area to area vary a great deal in terms of their thoroughness and content. For instance, within the topic of Magic Quadrants, there is great variability in the amount of text dedicated to vendors based upon the particular Magic Quadrant in question. Secondly, some Magic Quadrant reports will quote survey results, while other Magic Quadrant reports will not.

Generally, Gartner reports are written for an executive audience. Most software-oriented people, either developers or implementers, are not the target audience even though the reports cover software in which these groups specialize. The people who spend the most time reading and discussing Gartner’s research are:

  1. Executives faced with purchasing decisions in companies that implement enterprise software
  2. Marketing, sales and executives in the vendor companies
  3. Investor analysts

Hiding the Actual Math

Gartner’s analytical products are unusual in that they use text to explain the research rather than graphics or numerical tables. Gartner tends to avoid using numerical tables—which would allow the user to see what Gartner is writing about in a comparative manner—and the raw data is rarely provided to the reader (something which will be covered in Chapter 4: “Comparing Gartner to Consumer Reports, the RAND Corporation, and Academic Research.”)  For instance, Gartner will say how a vendor performed in some survey area, but not declare how other vendors performed in that same survey area. This methodology is somewhat unique in this type of comparative research; in academic research or with Consumer Reports, tables are quite commonly used to compare all the data points— a generally-accepted practice that Gartner declines to follow. Gartner clearly prevents direct comparisons that normal publication guidelines for research would allow. And no wonder, as it is not in Gartner’s best interests to declare its findings in black and white because those companies that it rates are also its customers. Therefore, Gartner’s research feels more like a liberal arts paper than a research paper, as everything is interpreted for the reader rather than presented to the reader. As a consequence of verbose prose and the lack of anchoring comparative graphics, it is quite easy to get lost in a lengthy Gartner report, and it is quite natural for the reader to simply go back to the single comparison that is offered in the reports (for instance the Magic Quadrant graphic).

Gartner’s Political Sensitivity

A clear political sensitivity comes across in the writing style used in Gartner reports and how information is disclosed to the reader. The writing approach also changes depending upon the type of report. For example, the writing for a Magic Quadrant document is dispassionate, while the writing for other articles (such as related to the future outcomes of mergers or technology market predictions) is more opinionated. Gartner also does a very good job of writing in such a way that rarely promotes one vendor over another. Occasionally I have come across Gartner research reports that are really just thinly-disguised press releases from a software vendor, but this is not representative of the vast majority of their research reports.

Gartner’s Lack of Transparency

The transparency of the scoring of vendors depends upon the report category. For instance, in all of the Magic Quadrant reports I reviewed, I was never able to find one that actually showed the scores for the different criteria. Providing the criteria scores per vendor would be very useful, as the scores would allow buyers to adjust these reports to their needs and to better analyze the research. For example, if the actual scores per criteria were listed, buyers could alter the weighting of the criteria or eliminate the criteria that are not important to them altogether, which would be preferable over using the exact same criteria that are deemed relevant by Gartner. There are a couple of reasons as to why Gartner does not show the criteria scores. One: Gartner has to be careful what it writes in its most influential reports, because the vendors (which Gartner also counts as customers) review the analytical products as well. Two: the more oblique the reports, the more the customers must hire Gartner analysts for interpretation.

Alternatively, when the stakes are lower, such as in Gartner’s Top 25 Supply Chain Companies report, a table with the criteria and the scores for each company is published. However, why is the difference so stark? This difference in the case of the Top 25 Supply Chain Companies (which coincidentally are not large buyers and customers of Gartner), those companies that do not find themselves on the Top 25 list are not going to cut their subscriptions. With this less political report, Gartner is much more free to publish the scoring—and therefore they do.

Gartner’s Varying Degrees of Disclosure

At other research entities, a standard of disclosure is applied for all research publications. Gartner’s varying degrees of disclosure, which depend upon political considerations, would not be allowed at these other entities.

However, I do not want to give the impression that other research entities ignore political considerations when performing their research. More often than not, research from academics that is politically inconvenient is simply suppressed, or academic researchers self-censor and do not submit grants for certain types of research, or their research grants are rejected by the funding agency. Therefore, there are actually two issues:

  1. How much disclosure is there on published research?
  2. How is politically sensitive research filtered out (i.e. not funded) before ever being researched?[1]

In this case, I am discussing the first issue and not the second. The footnote explains why an exploration of the second point is infeasible for this book.

[1] The issue of research suppression is hugely important. There are multiple cases of academic research being suppressed. Some research is kept private under false pretenses, such as for “national security” reasons. For example, RAND led the research project generally known as the “Pentagon Papers,” which were not published because they contained the truth behind Vietnam. The papers were unknown outside of RAND and the Pentagon until they were leaked. The papers were kept secret not only from the public but from the President of the United States—who has the top security clearance and had every right to have access to them and be aware that a massive research project on the history of the US involvement in Vietnam was being undertaken. The reason for doing this was that if the President actually knew the real history of the US involvement in Vietnam, the Pentagon would be less able to control the interpretation of the conflict, of the US subterfuge of Vietnam and therefore less able to lead the President to the conclusions that the Pentagon wanted him lead towards.

Research suppression is more difficult to prove than differing disclosure standards between research. On the other side of the coin, a significant component of research has little to no benefit, as is pointed out by John P. A. Loannidis, a medical researcher who actually specializes in analyzing medical research.

“Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure.”

Research suppression is also a complicated topic, which is far beyond the scope of this book.

Gartner’s Conclusions

Gartner looks authoritative until you understand that they do not publish information based upon research, but instead can be seen as an entity that puts its readers last. That is why it is important to consider how Gartner actually comes to its conclusions.

The graphics and links are, not an exhaustive list. Most likely we will be adding to this list in the future.

Are you aware of other dimensions by which Gartner is corrupt? Reach out to us to report what you know, and we may add to the list if we believe the item qualifies.

Why the Gartner Ombudsman is a Ruse

The Ombudsman may report to Gartner’s general counsel, but it is impossible to ascertain the situation regarding pressure brought by the ombudsman on the process of remediating complaints by vendors. However, Gartner is either confused as to what an ombudsman is, or has deliberately misled many people as to what it is and how an ombudsman actually functions.

  • An ombudsman that is on Gartner’s payroll, as an employee, will have no incentive to side with a vendor over Gartner on a complaint.
  • In other cases where an ombudsman is used, there is some type of transparency to the process. That is there is a publication of the deliberations of the ombudsman. However, Gartner offers none of this. Therefore it is difficult to see how Gartner’s ombudsman is an ombudsman in anything but name.

Finally, there is no area of research is an ombudsman used. Academic research does not use an ombudsman. This is because they follow a method of disclosure and of conflict of interest rules (except for medical research, whereas with Gartner, research standards are frequently not followed).

Is Gartner Correct that Having an Ombudsman Removes the Disclosure Requirement?

Having an ombudsman does not remove Gartner from the responsibility of reporting its sales to software vendors. By Gartner rating companies they also take money from, it is a textbook case of research conflict of interest.

Gartner analysts may not be allowed to own stock in the companies they cover, but Gartner is. This is covered in the article Why Does Gartner Invest in a Hedge Fund that Invests in Technology?

I would like to say that Gartner’s behavior is uniquely dishonest in the IT research space. However, I can’t. IDC, the Yankee Group, Aberdeen, and others follow similar corrupt models, which is why what they do cannot be classified as research. Upon researching statements by the leaders in these organizations, it is obvious that all of them are taking advantage of the general population’s lack of understanding of research rules.

Here the head of IDC copies the hollow defense of their biased research offered by Gartner’s Hall.

Fact-Based Service as Opposed To?

Execs at other major research firms speak with similar conviction. “We provide fact-based advice,” says IDC CEO Kirk Campbell. IDC’s research methodology, with its emphasis on hard data, provides a built-in guard against analyst bias or favoritism, he argues. Campbell dismisses any suggestion that vendors have to be paying customers to get fair treatment in IDC reports. “We have an open-door policy,” he says.

This quote is from several years ago, however, it matches similar statements made by the White House Press Secretary Sarah Huckabee who states that the press reports opinions when they should be reporting facts. One important way of evaluating a statement is whether it can be reversed and whether it would ever apply.

So if we reverse Kirk Campbell’s statement we get the following:

“We provide non-fact-based advice.”

Who would say such a thing? The question is not whether IDC provides fact based advice. It is whether vendor money causes IDC to promote vendors who can pay more than vendors that don’t or can’t. And Campbell’s statement here does not address that issue.

The Yankee Group

“Some research execs concede that their firms must do a better job of educating customers and the public about their policies and procedures, given the influence they have over multimillion-dollar buying decisions. One idea is to develop industrywide standards for business practices. “It’s something we should look at,” says Yankee Group CEO Emily Green. “Even the perception of favoritism would hurt us.” Says Forrester’s Kardon, “It would help the whole industry if we had a common set of practices to keep everybody clean.”

The reason this is so hilarious is that the rules of research are very well established. For instance, one thing Yankee Group could do is publish all of its funding from vendors. A second thing it could do is make sure it publishes a complete method explanation.

The Aberdeen Group

“Aberdeen Group CEO Jamie Bedard last week claimed the high ground. In a progress report on Aberdeen’s business, Bedard wrote to customers, “We promised you that … our research integrity was not for sale.” In an interview, Bedard charged that too many research firms base their advice on just a few interviews–what he calls opinion-based research–rather than on detailed surveys of dozens or hundreds of companies. “I think the industry can do a better job of deep research,” he says.

This is unintentionally hilarious because it effectively comingles two topics:

  1. Integrity
  2. Depth of Analysis

But Jamie Bedard starts off with one topic, but then quickly moves to a second topic. But these topics don’t have anything to do with each other.

Non Disclosure

There’s an important bit of information that Campbell refuses to share: He won’t disclose how much money IDC takes from tech-vendor clients. (emphasis added)

And it is not difficult to guess why that might be.

Next up to bat, we have Forrester’s Brian Kardon who proposes how much integrity Forrester has.

“Forrester Research focuses on selling its services to users rather than vendors to ensure that most of its revenue doesn’t come from the subjects of its research, says Brian Kardon, chief strategy and marketing officer. That affords the company a lot of freedom. “We routinely slam vendors,” he says. Still, about a third of Forrester’s revenue comes from tech vendors.”

What if the vendors that Forrester “slams” won’t pay off Forrester? Gartner is well known to do this. Therefore slamming non-paying vendors is not evidence of a lack of bias. In fact, that would be part of the business model. Payment means the providing positive coverage, while non-payment means negative coverage. What better way to turn non-paying customers into paying customers?

The Importance of Disclosure for Any Research Entity

May software vendors pay software analysts, and this influences the ratings that they receive. SAP alone pays Gartner several million dollars per year, and the larger the vendor, they more they can afford to pay. However, Gartner does not disclose this information anywhere on their website.

  • Most IT analysts do not disclose or publish the fact that they are paid by vendors.
  • Their business model is similar to the financial rating agencies, except those, unlike rating agencies that are paid exclusively by those who want their products rated.
  • Both the vendors and the software buyers pay the analysts, so their income sources are more balanced.
  • While they present themselves as having one customer (those that buy their research).

They have two clients, the vendors being the second clients. Within these vendors, the biggest pay the most, so the research results are slanted in their direction.

Gartner Invests in Hedge Funds?

It turns out that Gartner invests in hedge funds that invest in technology that it rates, and this is explained in the following quotation.

“The firm (Gartner) invests in hedge funds that hold significant stakes in the companies it covers.(Emphasis added) One such investment is SI Ventures’ SI Venture Fund II. On its Web site, SI Ventures notes a “long-term relationship” with Gartner. SI Ventures helped launch Authentor Systems, which provides network security software. Gartner analysts have been quoted in press releases issued by Authentor supporting the company’s approach to security.”

This is the classic conflict of interest between the investment advisor and the investing banking sides of investment banking.

Now Gartner readers don’t know if Gartner gave praise to Authentor Systems because it is their actual view, or because they benefit financially if Authenticator Systems goes up in value.

Who Owns Gartner?

However, Gartner’s conflict of interest regarding investing goes a step further. One example of this is who owns Gartner. This is covered by the following quoted.

“Gartner also is partly owned by investment companies that have stakes in tech vendors upon which Gartner is supposed to be casting a neutral eye.

Silver Lake Partners, which owns 33% of Gartner, counts Michael Dell, Bill Gates, Larry Ellison, and other tech-industry shakers among its current or former investors. Hedge fund ValueAct Capital owns more than 16% of Gartner and has owned as much as 11% of MSC Software, which Gartner views as a “challenger” in the market for product life-cycle-management software.”

This gets complex. But if Silver Lake Partners or Value Act Capital were so predisposed, could they pressure or otherwise influence Gartner to give better ratings to a software vendor that either of the two are invested in?

These are problematic ties. Gartner simply should not have them.

Conclusion

Gartner’s evidence that they are unbiased is simple. Trust Gartner. Trust that an ombudsman will control all the analysts at Gartner. Trust that allowing Gartner to make money off of technology investments, while its analysts can’t lead to unbiased outcomes. It all comes down to blindly trusting Gartner.

And the Yankee Group, Aberdeen, Forrester, IDC and others follow the same approach. They are not as prominent as Gartner, but they all refuse to declare who pays them, while at the same time swearing that none of these payments affect how they write about the vendors or their products.

The Problem: Thinking that Gartner is Focused on What is True

Gartner is hired by companies who fundamentally don’t understand how Gartner functions. Gartner has virtually no first-hand experience in the technologies that they evaluate and get most of their information from speaking with executives at buyers or executives at vendors and consulting firms. Gartner is also not a research entity, they compare very poorly to real research entities once you dig into the details as we did in the article How Gartner’s Research Compares to Real Research Entities. Gartner serves to direct IT spending to the most expensive solutions as these are the companies that can afford to pay Gartner the most money. Gartner has enormously aggressive internal sales goals that place accuracy far below revenue growth in importance.

Being Part of the Solution: Fact-Checking Gartner

Gartner is quite expensive, and using them without independent verification ends up in a lot of money being wasted, both in terms of fees to Gartner and also in money spent on Gartner recommendations. We offer Gartner support services that make sure companies get a better value from Gartner, and keep companies from relying on advice where Gartner cannot substantiate the advice or where the advice is clearly in response to vendor pressure. This includes both covering how Gartner works, as well as reading and fact-checking the information they provide. Our clients don’t even let Gartner know that we are evaluating the information provided to them.

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Search Our Other Research Bias Content

References

https://www.informationweek.com/credibility-of-analysts/d/d-id/1040282?page_number=2

https://en.wikipedia.org/wiki/Argumentum_ad_populum

[1] The initial idea behind ERP systems was that it would combine many different applications into a single system, thus reducing application integration issues. However, after the major ERP vendors sold the ERP product into companies, they began to develop specialized products for things like supply chain planning, business intelligence, customer relationship management, etc. This was done for several reasons. First, there was simply no way that an ERP system with its elementary approach to all functionality (with the possible exception of finance and accounting) could meet all the needs of companies. Secondly, once ERP companies had sold their ERP applications, they needed to develop more applications in order to grow their sales. Once they had the ERP system implemented, they had the network effect on their side as the ERP system is the “mother ship application”—the application or set of applications to which all other applications must integrate. Thus they were in a competitive position to sell more software into these accounts. These applications all have their own platforms and have adapters to one another, but each are a separate application, with a different database and sitting on different hardware, meaning that companies are essentially back where they started before the move to ERP systems. Except, they now rely more on external application development through commercial software rather than internal application development. All of these factors undercut one of the primary arguments that were often used to sell ERP systems: that they would reduce costs. More on this topic can be read at this link.

[2] SAP is a very powerful influence in IT and has many people that make a very good living from working in SAP. Unless my research results are heavily skewed in favor of SAP, I can expect negative feedback on my articles.

[3] Anyone who reads Consumer Reports will know that luxury cars are in their own category.

Gartner Book

Gartner and the Magic Quadrant: A Guide for Buyers, Vendors, and Investors

Gartner is the most influential IT analyst firm in the world. Their approval can make or break a vendor in an application category, or at the very least control their growth. Gartner has been behind most of the major IT trends for decades. However, many people read Gartner reports without understanding how Gartner works, how it comes to its information, its orientation, or even the details of the methods it uses for its analytical products. All of this and more is explained in this book.

Table of Contents

  • Chapter 1: Introduction
  • Chapter 2: An Overview of Gartner
  • Chapter 3: How Gartner Makes Money
  • Chapter 4: Comparing Gartner to the RAND Corporation, and Academic Research
  • Chapter 5: The Magic Quadrant
  • Chapter 6: Other Analytical Products Offered by Gartner
  • Chapter 7: Gartner’s Future and Cloud Computing
  • Chapter 8: Adjusting the Magic Quadrant
  • Chapter 9: Is Gartner Worth the Investment?
  • Chapter 10: Conclusion
  • Appendix a: How to Use Independent Consultants for Software Selection
  • Appendix b: What Does the History of Media Tell Us About This Topic
  • Appendix c: Disclosure Statements and Code of Ethics