Gartner

How Gartner Got IBM Watson so Wrong

Executive Summary

  • Gartner has a pattern of making bad predictions, which they demonstrated again by predicting IBM Watson would be a success.
  • How much are Gartner’s accuracy problems related to their income from vendors like IBM?

Introduction

Watson IBM is now broadly understood to be a failure. Yet, Gartner gave IBM Watson their seal of approval back in 2014.

See our references for this article and related articles at this link.

Gartner on IBM Watson

In the following quotations, there is an eerie similarity between comments from IBM and comments from Gartner regarding IBM Watson.

Watson Offers A Real Benefit or a Brand?

We begin by providing a quotation from Reuters that sets the stage before the Gartner prediction captured in the same article.

“Ed Harbour, vice president of Implementation at IBM Watson believes Watson is still unique in its field. “Are there other companies out there that offered AI-based systems and machine learning? Yes, there are,” he said. “However…I believe very strongly Watson is ahead of the competition and we’ve got to continue to push [to make Watson better]. No, I don’t think it’s something that anybody can just do.”

But according to Perlich, data scientists who want to create similar platforms as Watson could possibly pull from various offerings from the likes of Microsoft Azure, , or Data Ninja. But what those products don’t offer is the Watson branding. “And everybody’s very happy to claim to work with Watson,” Perlich said. “So I think right now Watson is monetizing primarily on the brand perception.””

That does not seem real. It appears to be a differentiation based on brand awareness. That is, Watson is used on many occasions is used because of its brand. This may have been true in 2014, but Watson no longer has a good brand. Its brand has been dramatically degraded because it has publicly failed to produce value. It is also widely accused of being oversold by IBM. Those with access to what occurred with Watson internally stated have told us that the following:

“There was a lack of medical knowledge on IBM’s technology side. They expected the AI to train itself from non curated medical records. Not only the data sets were not parsed to skip errors from manual input. But they also were not informative enough to derive the kind of insight they hoped.”

If IBM has invested at least a $ billion into Watson, why isn’t there a differentiation? According to Reuters, in 2014, this was the level of investment.

Gartner’s Watson Prediction

“Jamie Popkin, managing vice president at research firm Gartner, said IBM’s technology significantly improved how information can be used and managed. “I think they’ve developed something that takes us to the next step where information management needs to go,” said Popkin.

IBM said it decided to establish the unit because of strong demand for cognitive computing.

“We have reached the inflection point where the interest is overwhelming and we recognized we need to move faster,” said Stephen Gold, vice president of Watson Business.

Watson will be deployed on Softlayer, the cloud computing infrastructure business IBM bought last year.

According to Gartner, by next year there will likely be a large and growing market for Watson-derived smart advisors and it said that Crédit Agricole predicted that these systems will account for more than 12 percent of IBM’s total revenue in 2018.”

Curiously, none of this came to pass, and Gartner once again fails on another prediction. And one wonders if being paid by IBM may have influenced this accuracy level.

Conclusion

Gartner takes a large amount of money from IBM every year. This influences their predictions and makes them go to bat for IBM applications and services.

The Problem: Thinking that Gartner is Focused on What is True

Gartner is hired by companies who fundamentally don’t understand how Gartner functions. Gartner has virtually no first-hand experience in the technologies that they evaluate and get most of their information from speaking with executives at buyers or executives at vendors and consulting firms. Gartner is also not a research entity. They compare very poorly to real research entities once you dig into the details as we did in the article How Gartner’s Research Compares to Real Research Entities. Gartner serves to direct IT spending to the most expensive solutions as these are the companies that can afford to pay Gartner the most money. Gartner has enormously aggressive internal sales goals that place accuracy far below revenue growth in importance.