- IBM has become a major provider of false information around AI.
- Central to IBM’s AI sales strategy is to distract attention from its colossal Watson failure.
IBM has sold many AI projects after making enormous promises about Watson AI that never panned out.
Watson’s Failure at M.D Andersen
“We often call out overly optimistic news coverage of drugs and devices. But information technology is another healthcare arena where uncritical media narratives can cause harm by raising false hopes and allowing costly and unproven investments to proceed without scrutiny.
A case in point is the recent collapse of M.D. Anderson Cancer Center’s ambitious venture to use IBM’s Watson cognitive computing system to expedite clinical decision-making around the globe and match patients to clinical trials.
Launched in 2013, the project initially received glowing mainstream media coverage that suggested Watson was already being deployed to revolutionize cancer care–or soon would be.
But that was premature. By all accounts, the electronic brain was never used to treat patients at M.D. Anderson. A University of Texas audit reported the product doesn’t work with Anderson’s new electronic medical records system, and the cancer center is now seeking bids to find a new contractor.
IBM spun a story about how Watson could improve cancer treatment that was superficially plausible – there are thousands of research papers published every year and no doctor can read them all,” said David Howard, a faculty member in the Department of Health Policy and Management at Emory University, via email. “However, the problem is not that there is too much information, but rather there is too little. Only a handful of published articles are high-quality, randomized trials. In many cases, oncologists have to choose between drugs that have never been directly compared in a randomized trial.
Forbes ran a blog headlined “IBM’s Watson Now Tackles Clinical Trials At MD Anderson Cancer Center.” Forbes stated use in patient care “might come in early 2014.” It quoted an M.D. Anderson doctor saying: “It’s still in testing and not quite ready for the mainstream yet, but it has the infrastructure to potentially revolutionize oncology research.”
Likewise Scientific American asserted: “The University of Texas M.D. Anderson Cancer Center is using Watson to help doctors match patients with clinical trials, observe and fine-tune treatment plans, and assess risks as part of M. D. Anderson’s ‘Moon Shots’ mission to eliminate cancer.”
While IBM has entered into numerous deals to use its artificial intelligence system in healthcare, a company spokeswoman said there’s no published study linking the technology to improved outcomes for patients because “the implementation of the technology is not there yet.”
Artificial intelligence has been suffering from overhype since the 1970s and 80s,” said Steven Salzberg, a professor of biomedical engineering at the Johns Hopkins School of Medicine. “
Sixty-two million was spent on Watson by the University of Texas before the contract was canceled. All of the information we have obtained from other sources around Watson is that Watson does not add value and that IBM lies about what Watson can do.
This has had real impacts on the usage of Watson as the following quote from the Wall Street Journal explains.
“More than a dozen IBM partners and clients have halted or shrunk Watson’s oncology-related projects. Watson cancer applications have had limited impact on patients, according to dozens of interviews with medical centers, companies, and doctors who have used it, as well as documents reviewed by the Wall Street Journal.”
What Does Watson Tell Us About IBM’s Honesty on AI?
This introduction to Watson describes things that have not occurred and that Watson has not accomplished as if they have already been accomplished.
These following quotes are from Gizmodo in and article titled Why Everyone Is Hating on IBM Watson—Including the People Who Helped Make It.
Watson Offers A Real Benefit or a Brand?
“Ed Harbour, vice president of Implementation at IBM Watson believes Watson is still unique in its field. “Are there other companies out there that offered AI-based systems and machine learning? Yes, there are,” he said. “However…I believe very strongly Watson is ahead of the competition and we’ve got to continue to push [to make Watson better]. No, I don’t think it’s something that anybody can just do.”
But according to Perlich, data scientists who want to create similar platforms as Watson could possibly pull from various offerings from the likes of Microsoft Azure, Amazon Web Services, or Data Ninja. But what those products don’t offer is the Watson branding. “And everybody’s very happy to claim to work with Watson,” Perlich said. “So I think right now Watson is monetizing primarily on the brand perception.””
That does not seem real. This is not an argument for differentiation. If IBM has invested at least a $ billion into Watson, why isn’t there a differentiation? According to Reuters in 2014, this was the level of investment.
“Jamie Popkin, managing vice president at research firm Gartner, said IBM’s technology significantly improved how information can be used and managed. “I think they’ve developed something that takes us to the next step where information management needs to go,” said Popkin.
IBM said it decided to establish the unit because of strong demand for cognitive computing.
“We have reached the inflection point where the interest is overwhelming and we recognized we need to move faster,” said Stephen Gold, vice president of Watson Business.
Watson will be deployed on Softlayer, the cloud computing infrastructure business IBM bought last year.
According to Gartner, by next year there will likely be a large and growing market for Watson-derived smart advisors and it said that Crédit Agricole predicted that these systems will account for more than 12 percent of IBM’s total revenue in 2018.”
Curiously, none of this actually came to pass, and Gartner once again fails on another prediction. And one wonders if being paid by IBM may have influenced this accuracy level.
Watson as the Donald Trump of AI?
“IBM Watson is the Donald Trump of the AI industry—outlandish claims that aren’t backed by credible data,” said Oren Etzioni, CEO of the Allen Institute for AI and former computer science professor. “Everyone—journalists included—know[s] that the emperor has no clothes, but most are reluctant to say so.”
Etzioni, who helps research and develop new AI that is similar to some Watson APIs, said he respects the technology and people who work at Watson, “But their marketing and PR has run amok—to everyone’s detriment.”
This is a delicate way of saying that IBM has been habitually lying about Watson.
IBM’s Moonshot and Curing Cancer?
“The designer thinks that false hope came from the Watson ads. For instance, one commercial depicts two doctors in a rural hospital that can do genomic analysis thanks to an intelligent black box that advises the doctors. In another commercial a soon-to-be seven-year-old talks to a fictional square about how she’s not sick anymore. After Watson reads her health data she asks if Watson is a doctor. “No, I help doctors identify cancer treatments.” Watson responds, as the copy on the screen reads: “IBM Watson is helping doctors outthink cancer, one patient at a time.”
“Outthink cancer” is deceptively vague. Rometty was even more vague in a 2015 Wall Street Journal interview. “We will change the face of health care,” Rometty told writer Monica Langley. “If you think solving cancer is cool, then we’re cool.””
This sounds very deceptive. Has IBM Watson cured cancer? If it has, IBM should come out and say this. It is unclear what outthinking or solving means. Has IBM solved cancer?
Ethical Problems with IBM’s Claims Around Watson?
The experience meeting the hopeful patient made the designer view the company in an entirely different light. “I would not put money on Watson helping patients on a grand scale,” the designer said. “IBM needs to be held accountable for the image that it’s producing of its successes compared to what they’re actually able to deliver, because at a certain point it becomes an ethical issue…You’re telling cancer patients that they should have a higher feeling of hope about their outcome and then under-delivering on that—to me, that’s just dirty.”
It seems like this ethics would apply outside of medicine as well. It is odd that if a person’s health is not put at risk, lying is often considered harmless, but if health is put at risk, then lying is a serious problem. This seems to translate to ethics only applying to software sold to the medical industry.
“Another former employee who worked as a design researcher lead at Watson for Oncology also said they were uncomfortable with how commercials portrayed the platform. “You watch those commercials and you think it’s finding new ways to cure cancer,” the designer said. “Why confuse people and make them think it’s going to find something that a physician couldn’t possibly find?… Then you’ve moved into what strikes me as unethical territory when you’re potentially giving hope to people who should never have placed hope in that kind of a system because it’s not a magical box that does that stuff. It’s not a god.””
Did IBM Make AI Mainstream?
“Now, thanks largely to IBM, it is no longer a risk for tech companies to focus on AI. Rather, it is a risk to ignore it. But because IBM wanted consumers to take it seriously in the early days, the company came up with its own flashy, imprecise branding for the fantastic new technology. As other companies have started investing heavily in AI in a time when it’s safer to do so, IBM has stayed on the same course, and Watson is trapped in the same black box.”
Yes, but if IBM’s hyped the AI market based upon false claims, this wouldn’t this be a negative? They may have become more accepting of AI because they did not sufficiently analyze the claims made by IBM.
IBM has been lying about Watson AI for over ten years. It has spent billions on Watson and had not only problems understanding how to train Watson to solve medical research problems, but also failed to harmonize different data sets. The curious thing is that IBM continues to sell AI projects. IBM claims to have 20,000 AI projects ongoing. However, these projects have been sold on false promises. IBM does not possess any AI capabilities that other entities in the space do not possess, and the field of AI is filled with false claims. Even if a company does employ many people who are familiar with how to run the major AI/ML algorithms, there is little evidence that these algorithms work. There are further problems with formatting data as it turns out that data lakes are even more difficult to convert into a usable form than previously thought.
All of this occurs in an environment where far more proven methods of forecasting often languish due to a lack of funding, unable to match the promises and “sexiness” level of AI.
A Hypothesis No AI Company Wants To be Tested
Even significantly into the AI bubble, there is yet to be much evidence that AI meets the hype. Every time an example of AI failing is found, industry sources that make money on AI tell these observers that the failure is not relevant. IBM had over 10 years and enormous resources and was not able to make a useful product in the AI space. IBM has promised that its clients, that have far less to spend on such projects, will benefit immensely from hiring IBM to implement AI projects.
Why would these clients be able to accomplish this, if IBM itself went down in flames on its own internal AI project?
Previously winning our Golden Pinocchio Award for lying about Watson.
Search Our Other Forecasting Content
Brightwork Forecast Explorer for Monetized Error Calculation
Improving Your Forecast Error Management
How Functional is the forecast error measurement in your company? Does it help you focus on what products to improve the forecast? What if the forecast accuracy can be improved, by the product is an inexpensive item? We take a new approach in forecast error management. The Brightwork Explorer calculates no MAPE, but instead a monetized forecast error improvement from one forecast to another. We calculate that value for every product location combination and they can be any two forecasts you feed the system:
- The first forecast may be the constant or the naive forecast.
- The first forecast can be statistical forecast and the second the statistical + judgment forecast.
It’s up to you.
The Brightwork Forecast Explorer is free to use in the beginning. See by clicking the image below:
Comment from this article:
“Watson’s win on Jeopardy wasn’t as straightforward as everyone thinks. Contrary to public perception, Watson has never had a speech interface. So for Jeopardy the questions were submitted in written form to Watson. However, the way the game was played, Watson received the question as soon as Alex Trebek began reading the question to the other contestants. With the speed that computers process information this meant that Watson had something like an hour to contemplate the question before the other contestants had finished hearing the last words. With this type of advantage it’s no surprise that Watson won. And IBM’s marketing department has taken that golden ring and run with it ever since.”