How to Best Understand Gartner’s Faux Ombudsman

Executive Summary

• Gartner states that their ombudsman protects against their massive financial bias.
• We evaluate the accuracy of Gartner’s claim.

Video Introduction: How to Best Understand Gartner’s Faux Ombudsman

Text Introduction (Skip if You Watched the Video)

Gartner is a compelling and controversial entity in enterprise software. Gartner created a position called the ombudsman to dissipate criticism of how fair or accurate its ratings are. This is an internal position — not independent from Gartner — that intermediates disputes that vendors believe they are not being treated fairly. In the article, The Problem with How Gartner Makes Money, I pointed out that while Gartner has a high requirement for disclosure because it receives income from vendors, it also rates. However, Gartner does not disclose which vendors or the amount of money it gets from different vendors.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: You are reading one of the only independent sources on Gartner. If you look at the information software vendors or consulting firms provide about Gartner, it is exclusively about using Gartner to help them sell software or consulting services. None of these sources care that Gartner is a faux research entity that makes up its findings and has massive financial conflicts. The IT industry is generally petrified of Gartner and only publishes complementary information about them. The article below is very different.

• First, it is published by a research entity, not an unreliable software vendor or consulting firm that has no idea what research is. 
• Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services as a vendor or consulting firm that shares their ranking in some Gartner report. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. 

The Existence of Ombudsman Versus Meeting Disclosure Requirements?

In my book Gartner and the Magic Quadrant, a Guide for Buyers, Vendors, and Investors, I point out that Gartner will often use the existence of its ombudsman to argue that they are objective and that no further controls are necessary. This includes disclosing financial conflicts of interest. However, within the community of those that publish research, this is a contention that is unique to Gartner.

It is essential to consider that the rules on the disclosure of income sources that create a conflict of interest are much larger than Gartner. They are known throughout academic research and private research (not necessarily always followed, of course). The rules are much bigger than Gartner and cannot be defined by Gartner. And the rules are that for research to be considered “research,” and to be taken seriously, financial contributions need to be disclosed, which is one of the criteria by which I demoted them when compared against academic research, the Rand Corporation Consumer Reports.

Whether Gartner has an ombudsman is a different subject entirely. If we are trying to get to the truth, it is essential not to be distracted by an argument instead of whether rated vendors have recourse.

The area that Gartner most often refers to as their check on the bias is the Office of the Ombudsman, which manages against conflicts of interest and ensures fairness. Before we get into what the Gartner Ombudsman is, let’s start by analyzing the terminology.

An “ombudsman” is generally appointed by a government body and provided with the independence to investigate rights violations. The term “ombudsman” is infrequently used in the modern business environment. An ombudsman is similar to an Inspector General, something that everyone has heard of concerning the United Nations. The definition of the Inspector General from the UN website is informative:

“UNHCR’s Geneva headquarters has three mandated functions: to assess the quality of UNHCR’s management, including identifying measures to prevent mismanagement and waste of resources; to investigate allegations of misconduct by anyone working for the agency, including non-permanent staff; and to conduct inquiries into violent attacks on UNHCR staff and operations, as well as other types of incidents that could affect the reputation of the organization.”

Therefore an ombudsman is often appointed to investigate some issue of importance. However, one of the essential parts of the ombudsman is their independence. This is addressed in the next quotation from Wikipedia.

“The major advantage of an ombudsman is that he or she examines complaints from outside the offending state institution, thus avoiding the conflicts of interest inherent in self-policing. However, the ombudsman system relies heavily on the selection of an appropriate individual for the office, and on the cooperation of at least some effective official from within the apparatus of the state.”

Gartner describes their Ombudsman as follows:

“The Gartner Office of the Ombudsman has been modeled after that of the news industry, in which the ombudsman takes action on issues of accuracy, fairness, tone and balance submitted by newspaper readers or radio listeners. As in the news industry, the Gartner ombudsman recommends approaches to address these issues via content or process updates.

At Gartner, these issues may be raised by clients, non-clients or employees. We don’t simply react to comments from these audiences—we invite these comments with a dedicated Web site and staff that address issues as they arise.

Given the nature of the current environment, we have formalized and publicized our commitment to objectivity to benefit both our associates and our clients. The ombudsman simply makes more public the ethics and standards we established when we were founded in 1979 and which continue to evolve.”

Sounds good, doesn’t it? The explanation must be examined within the context of how different Gartner is from most media outlets, particularly with the statement that…

“The Gartner Office of the Ombudsman has been modeled after that of the news industry….”

Modeled on the News Industry?

While it may be true that the office was “modeled” on the news industry, the position of Gartner vis-à-vis news industry firms is entirely different; news organizations do not see every entity about which they write an article as a potential customer. To understand why this is the case, let’s perform a thought experiment. Imagine that Time Magazine was receiving payments from many of the entities about which they wrote articles. Of course, in a way, Time Magazine does accept payments as they accept advertising. But, not every entity that Time reports on is a potential revenue source. However, over at Gartner, every entity they report on is either a current or a potential revenue source. Also, how much an advertiser advertises in a media outlet is a matter of public record. One can quickly tell by reviewing the magazine, newspaper, television channel, or website and checking who has advertised; this is why we know the relationship between financial contributions and the effect on media output. However, which software vendors pay Gartner—and how much they pay Gartner—is not a matter of public record.

The Value of the Ombudsman?

Now comes the exciting part of the thought experiment. Could Time Magazine assuage people’s concerns about reporting bias by merely saying, “It’s OK? We have an ombudsman?” I don’t think it could. Let’s take the example of the Swedish press Ombudsman.

“Pressombudsmannen (or press Ombudsman) is a person whose role in the Swedish print media is to determine whether the actions of a newspaper are in line with good journalistic practice. Complaints regarding the practices of print media can be reported by the general public to the Pressombudsmannen who determines whether a complaint should be brought before the Swedish Press Council (PON). The PON can issue fines of up to SEK 2,000 and publish a rejecting opinion.” – Wikipedia

However, notice that the Press Ombudsman can refer the matter to the Swedish Press Council, and a fine can be issued. There is no possible referral to anyone outside of the Gartner system. The actual complaints are not published, nor are the rulings and transparency of the Gartner Ombudsman is nil. While Gartner states that they have a dedicated website, it is used primarily for public relations to tell people about Gartner Ombudsman’s effectiveness. There is no publishing of information regarding current complaints.

Impartiality is Proven by Gartner How?

What is known about the ombudsman in terms of his or her impartiality? For instance, let’s imagine that a large vendor has a complaint regarding a rating’s fairness, while a smaller software company that does not pay Gartner (or pays a much smaller amount) has the same claim. Which complaint would be a higher priority for the ombudsman? Anyone can guess, but there is no way to know for sure. Research requires data points, and Gartner does not publish the complaints, nor does it show how the ombudsman ruled. In a court case brought against Gartner by ZL, which claimed that Gartner’s Magic Quadrant was “…misleading and favored large vendors,” Dennis Howlett, writing for ZDNet, had the following to say about the ombudsman.

“Gartner currently has an employee act as ombudsman to handle disagreements. The conflict of interest is self-evident in the way ZL’s concerns were summarily dismissed with little supporting evidence. There is a crying need to establish an impartial ombudsman similar to those found in public media, in order to ensure purchasers that they are receiving impartial analysis.”

The Ombudsman Called out as a Phoney and Biased Entity

So ZL proposed in action against Gartner that the Gartner Ombudsman is simply fascia and does not meet the ombudsman’s standards in other areas, particularly in print. In this regard, ZL must be considered to be correct. Gartner could do several things to improve the Office of the Ombudsman.

1. The Office of the Ombudsman could be placed outside of Gartner and funded by software buyers.
2. The Office of the Ombudsman could publish the complaints and the rulings on these complaints on a public website.

As a research organization, Gartner should understand that their statements regarding the impartiality of their “ombudsman” are insufficient. For the same reason, Gartner should not accept a vendor’s announcement that they have “the best software in the world” or have “one hundred percent customer satisfaction” without actually asking for evidence of these claims. If research were repeating what entities said about themselves, it would not be called research.

The Proposed Roll Versus the Actual Roll

The inconsistency with this is a quotation that represents the general function of an ombudsman and their behavior — is that I have never seen the Gartner ombudsman be critical of any Gartner action.

The Gartner ombudsman tends to behave in just an opposite fashion and as more as a public relations officer — at least externally.

I can’t say precisely how the ombudsman operates internally — because there is no transparency to the ombudsman’s rulings. However, if one reads online posts that complain about bias on Gartner’s part, quite a few of them have comments by the Gartner ombudsman — declaring that Gartner is entirely impartial and using the existence of their position as evidence of this. (once again, the presence of the ombudsman does not prove impartiality) Contradicting this proposal is that many software vendors have confided in me that they believe the Gartner ratings are very much tied to vendors’ financial contributions. However, they won’t say this in public for fear of Gartner. In fact, Gartner has been sued on this exact point. In the court case brought by ZL Technologies against Gartner. I quote from my book Gartner and the Magic Quadrant: A Guide for Buyers, Vendors, and Investors:

..ZL, which claimed that Gartner’s Magic Quadrant was “…misleading and favored large vendors,” Dennis Howlett, writing for ZDNet, had the following to say about the ombudsman.

“Gartner currently has an employee act as ombudsman to handle disagreements. The conflict of interest is self-evident in the way ZL’s concerns were summarily dismissed with little supporting evidence. There is a crying need to establish an impartial ombudsman similar to those found in public media, in order to ensure purchasers that they are receiving impartial analysis.”

ZL Technologies

ZL Technologies did not only disagree with its Gartner rating — it called into question the ombudsman’s impartiality. Interestingly, Gartner did not move towards proving the impartiality of the ombudsman but instead adopted the legal argument adopted by many investment banks after the 2008 crash that its ratings and published information were “opinions,” protected as free speech. Therefore they could not be held accountable for them.

Whoa? What Was that a Legal Defense?

This is extremely inconsistent with what Gartner tells clients. Just as with the investment banks, they sell their research because their results are based upon very serious analysis and are not merely opinions.

ZL Technologies lost its case against Gartner but not because it proved impartiality but because it argued it had a first amendment right to say whatever it wanted. However, the legal strategy adopted by Gartner should give anyone relying upon Gartner’s research pause. They essentially proposed not that they had an excellent or reliable process of moderating disagreements among vendors. The complaint brought by ZL Technologies, but that their ratings were not particularly substantive, were just opinions, and therefore should not be used against them.

Are Gartner’s Views Unsubstantiated Opinions?

Gartner’s views are simply opinions and don’t have any method to support them, at least according to Gartner.

It isn’t easy to see why the court did not analyze Gartner’s promotional statements about itself or accurately interpret Gartner’s website. They claim a high degree of knowledge and present themselves as providing authoritative information.

How Standard is the Gartner Ombudsman’s Behavior?

In addition to criticism of the Gartner ombudsman, the Gartner ombudsman’s behavior is inconsistent with that of what we come to expect from an ombudsman.

The Gartner ombudsman should limit their activity to hearing complaints and moderating them. Gartner states that they modeled their ombudsman on that of the news ombudsman. Still, the news ombudsman doesn’t behave as a PR function for the news outlet that employs them — as it undermines the credibility of the position. For instance, the Gartner ombudsman is continually using the ombudsman’s existence as an argument for why Gartner does not play favorites based upon which vendors pay them the most money. This is illogical as they are two different issues. I would question any individual who would commingle these issues as engaging in a deliberate evasion. Secondly, it is firmly established and generally accepted that any research entity must publish its list of funding sources. Entities that refuse to do this — like many think tanks (for example, the American Petroleum Institute or The Hoover Institute), are not considered substantial outfits and most often produce fake research — which is directly tied to the results their funding sources dictate.

The PR Function of the Ombudsman

Gartner’s description of the ombudsman function focuses on how the ombudsman rules on fairness complaints made to Gartner. However, this leaves out the ombudsman’s quite significant public relations function. For example, the ombudsman—currently Nancy Erksine—is active on Internet comment sites stating that Gartner’s ratings are never for sale. She also comments on the Ombudsman website, generally saying good things about Gartner and so on. The Ombudsman website also informs visitors that Gartner analysts are not allowed to own stock or have other types of vendors’ investments.

Would the Ombudsman Every Criticize Gartner?

The statements made by the ombudsman are designed to counter-arguments that are critical of Gartner’s objectivity. Gartner’s entirety of evidence regarding its objectivity primarily comes down to the ombudsman’s word that Gartner has several controls in place. Therefore its research can be trusted to be without inappropriate influences. These statements are not auditable except by looking at Gartner’s media output and interviewing several people at vendor and buyer organizations, much of which would contradict the ombudsman’s comments.

The Objectivity of the Gartner Ombudsman?

Another problem with the ombudsman’s statements is objectivity. An individual has objectivity when they can reasonably be expected to be indifferent about which way the results swing. For example, salespeople cannot be said to be objective between the vendor for which they work and the vendors that are competitive because they are paid to sell one vendor’s products. They could not hold the position that competing vendors’ software is superior and continue to be employed. The same is true for the Gartner ombudsman. If Nancy Erksine were to begin telling members of the media that Gartner’s vendor ratings can be purchased or that Gartner analysts take positions in vendors’ stock before the publication of Magic Quadrants, how much longer would Nancy Erksine be the Gartner Ombudsman or be employed at Gartner? Since the answer is “not long,” it is clear that the Ombudsman’s Office is not objective. Being objective means that it is at least possible for you to hold the opposite view from that which you currently own.

Gartner has borrowed the name of “ombudsman” for their mediating position but has not imbued the role with the required independence to meet the term’s definition as it is used, not only in the media but in other areas as well.

What the Ombudsman Means for Vendors

Vendors should view the Gartner Ombudsman as principally a public relations mechanism to make buyers comfortable with Gartner’s reports and advice. If a vendor does not rank as desired in a report, appealing to the ombudsman will do little. All vendors want to be rated superior to other vendors regardless of whether there is any excellent rating justification. Gartner analysts are continually hearing from vendors about how they think they should be rated higher than they were in the last review, so that analysts will tune these vendors out after some point. To improve one’s rating in the future, a far more effective use of time and energy would be to hire Gartner to understand their ratings better and then do what it takes to score higher in these ratings.

Conclusion

While Gartner says they have an ombudsman, one should not accept this at face value — particularly if the ombudsman has zero independence from Gartner and does not behave as an ombudsman – externally at least. This would be like a company declaring that they have a “real Santa Claus.” However, having a real Santa means something. It means that the individual lives in the North Pole, rides a gravity-defying sleigh, can negotiate dropping quickly down chimneys, and special delivers millions of presents in a single night. If the Santa that is being referred to can do none of these things, it is just a “Mall Santa.”

As I discussed in an earlier post, Gartner does not match up very well with what is considered real research entities because they don’t follow a research entity’s rules. I researched this and laid out the generally accepted practices in my book Gartner and the Magic Quadrant: A Guide for Buyers, Vendors, and Investors. Gartner does not do the things that are expected of a research entity to ensure impartiality, or beyond this, even to provide transparency into the research results. I can’t emphasize this enough; the way Gartner publishes its research would not be acceptable and would not allow its output to be published in any academic journal.

Gartner’s Ombudsman is Not Real

Like a fake Santa, anyone can put on a Santa hat, which does not make them the real thing. After analyzing Gartner’s Ombudsman in detail, the comments made about the Ombudsman are laughable.

The existence of a Gartner ombudsman would not address any of these issues. Still, the fact that the Gartner ombudsman is a “Mall Santa” should further heighten the concern that Gartner clients have that Gartner is doing anything more than playing by its own unique set of rules.

What is the Overall Implication of this to Using Gartner?

My view on this is that after having to peel back the layers on Gartner, those that use Gartner research without first understanding Gartner — its motivations, orientation, etc. — are taking a lot of risks that can negatively impact their enterprise software decisions.

Gartner tells clients that they can’t understand Gartner’s research without hiring them as advisors — but the problem with this is that Gartner will not tell clients the real story behind how Gartner works. The Gartner ratings must be adjusted, and there is a way — working backward from Gartner’s orientation and incentives to do just that.