- Vendors and consulting firms have been aggressively selling AI in forecasting software and AI projects.
- Customers are finding something curious about these ongoing projects.
We are now someway into the AI/ML bubble. What are AI projects finding to their dismay? A lack of data for running AI/ML.
Quotes from IBM on AI Projects
“Many ambitious artificial intelligence-backed projects never come to fruition due in large part to issues with data collection and cleaning, according to Arvind Krishna, PhD, IBM’s senior vice president of cloud and cognitive software.
During an interview with The Wall Street Journal earlier this month, Dr. Krishna noted that a common reason projects using IBM Watson AI often unravel is that companies are unprepared for the amount of time and money they must spend just collecting and preparing data. Those unglamorous yet crucial tasks, he said, make up approximately 80 percent of an entire project.
This quote is problematic from multiple dimensions.
Breaking the Watson Quote from IBM’s Overall AI Projects
Watson has been a failed product for IBM. It is AI directed at health care which is still essentially non-functional after billions and over a decade of investment. However, this article is not about Watson (we have quotes about the problems with IBM Watson in the references). This quotation is about AI writ large. But it is curious that with Watson, IBM apparently ran into its own data problems as the following quote describes.
“The employees said there was never clear agreement, for example, on how to merge data gathered by the three companies into a unified format that could be used by Watson. That made it more difficult to deliver insights to help hospitals target medical services to specific patients, cut costs, and improve the quality of care.
With this acquisition, IBM will be one of the world’s leading health data, analytics and insights companies, and the only one that can deliver the unique cognitive capabilities of the Watson platform,” Deborah DiSanzo, general manager for IBM Watson Health, said in a statement following the Truven acquisition.
But the deals presented the difficult task of harmonizing all that data – housed in different formats, and focused on different aspects of patient care – into a model that could be digested by Watson, a challenge that is not unique to IBM.” – STAT
Perhaps IBM is not the company to rely upon for “spiffing” up your data for your AI project, as it is now quite clear they were not able to figure out how to do it for their internal project, for which they had more resources than any one individual IBM project will likely ever match. IBM Watson is a specific health care focused AI solution. However, IBM appears to also call AI not related to that specific item Watson as well, which is of course confusing.
Having said that, let us review this portion of the quote from Dr. Krishna.
On IBM AI
“often unravel is that companies are unprepared for the amount of time and money they must spend just collecting and preparing data.”
When IBM sold the project, did they explain the level of effort this would take? This quote makes it sound like someone else, that IBM does not communicate with, is selling AI projects that IBM consulting then has to work. Is Dr. Krishna that his own IBM sales team is communicating with these same customers before the IBM AI project begins?
Dr. Krishna goes on…
“You run out of patience along the way, because you spend your first year just collecting and cleansing the data,” he said. “And you say, ‘Hey, wait a moment, where’s the AI? I’m not getting the benefit.’ And you kind of bail on it.”
Questions Related to this Quotation
|1||Setting Customer Expectation||Was the data effort explained by IBM to customers? Has IBM ever oversold the benefits of AI and undersold the work effort required to get the data so it is in a state that it can be used by AI algorithms?|
|2||How Long Until Data Begins to Be Usable?||Does the data availability appear after the first year, or is this just the starting point?|
|3||What is the Efficacy of the ML Algorithms?||What about IBM AI projects that are sold on a promise of AI providing great improvements in forecasting accuracy which then, after the algorithms are run, don't and it turns out the entire premise of the project was flawed?|
|4||Forecasting AI Project Benefits||If the data is not close to being ready to run AI/ML algorithms, on what basis is IBM forecasting AI benefits to specific customers?|
The question of underselling the data effort and overselling the benefits of AI is all important because IBM has routinely oversold its Watson solution as the following quotation attests.
“But it also earned ill will and skepticism by boasting of Watson’s abilities. “They came in with marketing first, product second, and got everybody excited,”” – Robert Wachter, chair of the department of medicine at the University of California, San Francisco
“Robert Burns, a professor of health care management at the University of Pennsylvania’s Wharton School, said the complexity of integrating mis-matched data sets has vexed hospitals and other health care entities for decades. It is folly, he said, for IBM, or any company outside the industry, to suggest the problem can quickly be solved to cure terminal diseases or dramatically improve health care delivery.” – STAT
And of course, this is in no way limited to IBM. It is difficult to find a consulting company in IT that is not making outrageous claims around AI. In fact, let us review several.
Getting Your AI From Wipro
Wipro, a firm not known for forecasting is now your one-stop shop for AI.
Getting Your AI From Infosys
Infosys is another AI expert. So many AI experts to choose from among the giant IT consulting firms. That man later married that robot.
Getting Your AI From Capgemini
This video from Cap Gemini is filled with inaccuracies but if it does not “jack you up on AI” it is unclear if anything will.
As with WiPro and Infosys, Cap Gemini is a non-entity in the forecasting space, but that does stop them from producing a killer video.
IBM’s AI Projects Tend to Fizzle Out?
Still, Dr. Krishna maintained that the fairly common occurrence of halted AI projects is “the nature of any early technology.” Even as so many fizzle out, IBM still has about 20,000 more ongoing AI projects, a number that he deemed indicative of overall success.”
There is a serious problem with Dr. Krishna’s statement here. This is because AI is not new. Is Dr. Krishna unaware of this fact?
AI has failed to produce results in at least two separate historical AI bubbles (in the 1960s and Early 1970s, the 1980s), each one of them followed by an “AI winter.” Many of the people working in data science/AI are not even aware of these previous bubbles. And how far back AI goes surprises most people we discuss this topic with.
“Many of them predicted that a machine as intelligent as a human being would exist in no more than a generation and they were given millions of dollars to make this vision come true.
Eventually, it became obvious that they had grossly underestimated the difficulty of the project. In 1973, in response to the criticism from James Lighthill and ongoing pressure from congress, the U.S. and British Governments stopped funding undirected research into artificial intelligence, and the difficult years that followed would later be known as an “AI winter“.” – Wikipedia
For those of you who have not tried SodaStream, you really should. It not only can add fizzle to new drinks, but it can give that “sparkling quality” to drinks that have gone flat. The problem? As of yet, there is no SodaStream for AI projects.
To review a portion of the quote from Dr. Krishna.
“Even as so many fizzle out, IBM still has about 20,000 more ongoing AI projects, a number that he deemed indicative of overall success.”
And when questioned about IBM’s success in AI, he responded defensively with the following quotation.
““I think 20,000 is not slow,” he said. “I think 20,000 projects is, what I would call, successful.””
This brings up the following questions
- How does IBM have 20,000 ongoing AI projects?
- Successful for whom, the customer of for IBM?
IBM certainly sees this as a success, but IBM only cares about billing hours on projects. By this definition even AI projects where hours are billed but not work is done is considered successful by the consulting company. However, IBM clients measure success, not by IBM’s metric. That is customers that invest in AI measure the benefit by how AI improves the accuracy of their various predictions.
The idea that IBM would have so many IA projects ongoing, and that there would be so little published about the benefits of AI received by companies is odd.
Another question is why is IBM placing data science resources on site and billing for them if the data is largely unavailable and if it may take a year or more to develop the data? Would IBM sell an automobile service plan for a customer that has yet to purchase an automobile? It seems like an elementary question to ask of what data the client has that can be used. Without this, IBM has no idea if their client can benefit from an AI project.
The AI Project Preparedness Matrix
This topic of data availability brings up the question of how common it is for companies that engage in AI projects have the necessary items to actually successful pull such projects off.
To evaluate this, below are the individual estimates of the author and three other experienced resources in forecasting and ML/AI.
The Implications of the Poll
If this poll is roughly representative, it means that AI projects are begun with a very small likelihood of success. AI projects have been ongoing for a number of years now, and given these estimates, it is easy to project very high rates of failure. When these failures do happen, they will be hidden by consulting firms and vendors. And it will take far longer to find out the real story about the outcomes of these projects.
The question arises — how can an entire bubble be based upon AI, if such a small percentage of companies have the ability to be successful with these projects?
What Happened to Data Lakes?
For the better part of a decade, companies were told to through large amounts of unstructured data into data lakes. The idea was that data was now accumulating to so quickly, that there was no time to organize it. NoSQL is hot, it’s happening it’s now. The point was to accumulate almost as much as you could. The data scientists would come by later and sort out everything after it was collected. Unstructured or semi-structured data was seen almost as a virtue
However, now it is taking years to assemble this data, and now that it comes time to use this data, it takes lengthy projects to make is usable. Was the projection about the benefits of just collecting data and worrying about organizing it later actually justified, or was this waste?
Companies like IBM love charging for data lake projects. It allows them to talk up the future potential that will be released from AI. However, if Dr. Krishna is correct, these data lakes may not be as valuable as they were first proposed.
This is attested to by the following quotation.
“Data lakes promised to be the next generation of data warehouses, a central place to dump all of a company’s data. Unlike the warehouse, however, data lakes allow companies to dump data into the lake without ordering it beforehand. The problem with this approach, however, is that it simply delays the inevitable need to make sense of that data.”
Dataversity stated that 2019 is the year when companies begin “draining the data lake.” Data lakes did not appear that long ago, and we are draining them already?
The quotation from Dr. Krishna is misleading. Let us review some of the many issues in just a few lines of quotations from Dr. Krishna.
Issues with Dr. Krishna/IBM's Quotes
|1||Misrepresentation of IBM Watson||IBM Watson is not a successful product. In fact Watson has failed quite heavily and left a litany of dissatisfied customers that IBM does not acknowledge. IBM failed at their own internal data integration project, leading in part to Watson's downfall.|
|2||Confusion or Commingling of Watson with IBM AI.||Watson is not the same as IBM AI, or an IBM AI project.|
|3||AI's Development||AI is not new. This leads to the natural question of why Dr. Krishna would state that it is new. Does Dr. Krishna and IBM sales mislead prospects by repeating that AI is new in order to minimize and deflect from AI's true history?|
|4||Responsibility for Setting Sales Expectations||Dr. Krishna describes a scenario where IBM has no responsibility for explaining the effort in investing in data development to IBM's AI customers. It is difficult to believe that IBM properly apprises customers of these difficulties. Therefore, it fits with Dr. Krishna's incentives to state that "customers don't seem aware," when IBM puts informing them secondary to selling AI projects.|
|5||Measuring AI Success||Dr. Krishna seems to measure AI success by how many IBM AI projects are ongoing, rather than how successful those projects are at delivery benefits.|
The Otherworldly Claims of AI
Consulting firms are making large and unsubstantiated claims around AI. Consulting firms with no background in either AI or forecasting are making world-changing claims about their AI capabilities, and the claims appear to be uniform.
- AI is being proposed to defeat other methods in an almost universal manner, all without evidence this is true.
- AI is becoming homogenized to improve just about everything. AI’s benefits are claimed to be so universal, that in short order it will be challenging to declare what is not an improved outcome of applying AI.
- Many companies that eventually do assemble their multivariate data will find that in a higher percentage of cases the AI/ML is not able to show benefit versus far simpler and less expensive forecasting techniques. Dr. Krishna states the following.
“In the world of IT in general, about 50% of projects run either late, over budget or get halted. I’m going to guess that AI is not dramatically different.”
Not all IT projects have the same success rate. This is something else that Dr. Krishna should know. AI projects, because they are so strongly based upon false claims will have a much higher failure rate than 50%. In fact, The AI Project Preparedness Matrix above indicate that most of the AI projects that are sold are sold into companies that don’t have the ability to successfully complete them.
Who Are the AI Poll Contributors?
- Shaun Snapp: Shaun is the article author and an experienced forecasting consultant and the author of four books on forecasting.
- Ahmed Azmi: Ahmed has many years of experience in the AI/ML space.
- Steve Morlidge: Steve is a long term forecasting consultant, author or forecasting journal publications and the author of several books on forecasting.
- Anonymous: The anonymous entry is someone from a software vendor with many years of industry forecasting experience and publications in the forecasting literature.
Search Our Other AI and Machine Learning Content
Brightwork Forecast Explorer for Monetized Error Calculation
Improving Your Forecast Error Management
How Functional is the forecast error measurement in your company? Does it help you focus on what products to improve the forecast? What if the forecast accuracy can be improved, by the product is an inexpensive item? We take a new approach in forecast error management. The Brightwork Explorer calculates no MAPE, but instead a monetized forecast error improvement from one forecast to another. We calculate that value for every product location combination and they can be any two forecasts you feed the system:
- The first forecast may be the constant or the naive forecast.
- The first forecast can be statistical forecast and the second the statistical + judgment forecast.
It’s up to you.
The Brightwork Forecast Explorer is free to use in the beginning. See by clicking the image below:
The Foresight Forecast Search Engine
Foresight is a top forecasting journal and our favorite for publishing and reading. Foresight combines both academic with practical articles. Foresight provides an amazing search engine that can allow anyone to see what article apply to their interest or research area. Select the image below to go to their search engine.
We have reached and AI bubble to the point where we have AI “fraud.”
“It’s hard to build a service powered by artificial intelligence. So hard, in fact, that some startups have worked out it’s cheaper and easier to get humans to behave like robots than it is to get machines to behave like humans.
“Using a human to do the job lets you skip over a load of technical and business development challenges. It doesn’t scale, obviously, but it allows you to build something and skip the hard part early on,” said Gregory Koberger, CEO of ReadMe, who says he has come across a lot of “pseudo-AIs”.
“It’s essentially prototyping the AI with human beings,” he said.
“Outside of corporate headquarters, however, IBM has discovered that its powerful technology is no match for the messy reality of today’s health care system. And in trying to apply Watson to cancer treatment, one of medicine’s biggest challenges, IBM encountered a fundamental mismatch between the way machines learn and the way doctors work.
IBM’s bold attempt to revolutionize health care began in 2011. The day after Watson thoroughly defeated two human champions in the game of Jeopardy!, IBM announced a new career path for its AI quiz-show winner: It would become an AI doctor. IBM would take the breakthrough technology it showed off on television—mainly, the ability to understand natural language—and apply it to medicine. Watson’s first commercial offerings for health care would be available in 18 to 24 months, the company promised.
In fact, the projects that IBM announced that first day did not yield commercial products. In the eight years since, IBM has trumpeted many more high-profile efforts to develop AI-powered medical technology—many of which have fizzled, and a few of which have failed spectacularly. The company spent billions on acquisitions to bolster its internal efforts, but insiders say the acquired companies haven’t yet contributed much. And the products that have emerged from IBM’s Watson Health division are nothing like the brilliant AI doctor that was once envisioned: They’re more like AI assistants that can perform certain routine tasks.
But it also earned ill will and skepticism by boasting of Watson’s abilities. “They came in with marketing first, product second, and got everybody excited,” he says. “Then the rubber hit the road. This is an incredibly hard set of problems, and IBM, by being first out, has demonstrated that for everyone else.””
“On the May 8th edition of Closing Bell on CNBC, venture capitalist Chamath Palihapitiya, founder and CEO of Social Capital, created quite a stir in enterprise artificial intelligence (AI) circles, when he took on IBMIBM +0% Watson, Big Blue’s AI platform.
“Watson is a joke, just to be completely honest,” Palihapitiya said. “I think what IBM is excellent at is using their sales and marketing infrastructure to convince people who have asymmetrically less knowledge to pay for something.””
This independent analyst was contradicted by an IBM partner.
“Not all bloggers sided with Palihapitiya, however. André M. König, Co-Founder at Opentopic (an IBM partner), added his two cents. “Well I agree that IBM is a formidable marketing machine, only to be outmatched by their corporate boldness and technological innovation,” König wrote. “If you call IBM Watson a joke you call the hundreds of companies and startups that have built on it a joke.””
The following addresses canceled Watson projects, a common feature of Watson.
“In February 2017, M.D. Anderson Cancer Center canceled a promising, but troubled contract with IBM for its Watson platform. “The breakup with M.D. Anderson seemed to show IBM choking on its own hype about Watson,” Freedman added. “The University of Texas, which runs M.D. Anderson, announced it had shuttered the project, leaving the medical center out $39 million in payments to IBM—for a project originally contracted at $2.4 million.
“After four years it had not produced a tool for use with patients that was ready to go beyond pilot tests.”
Moreover, despite significant progress, even state-of-the-art machine-learning algorithms often cannot deliver sufficient sensitivity, specificity, and precision (that is, positive predictive value) required for clinical decision making.”
Instead, IBM is ceding whatever AI leadership it purported to have to a new crop of far more innovative startups and other AI firms willing to reinvent themselves as the inexorable pace of innovation continues unabated – and that’s no joke.””
Which is the standard response, any partner of a vendor defends that vendor.