- This article by Gartner makes a number of fallacious statements about forecasting and Big Data.
- We analyze the problems in Gartner’s understanding of forecasting.
*Note, the article mentioned here was a reprint of Gartner’s content by a company called BlueRidge. The way the article was republished made it difficult to determine if the material was created by BlueRidge or by Gartner. It was clearly the objective of BlueRidge to make it seem that its content was Gartners‘. Therefore, the statements below are from BlueRidge not from Gartner. I would suggest that Gartner not allows companies to “republish” its research as it causes confusion as to what is Gartners’ and what is BlueRidge. It is also difficult to believe that Gartner was not at some point made aware of the additions by BlueRidge to Gartners’ content.
As probably most of you have, I have been a follower of Gartner for some time. They are so influential and so often mentioned within companies that I wanted to peek behind the facade to see how they “make the sausage.” I researched how they come up with their rankings, and how they do research for the book Gartner and the Magic Quadrant: A Guide for Buyers, Vendors, and Investors. This is the only book written by an outsider to Gartner. I did not have to worry about offending any contacts at Gartner, so I was able to objectively analyze their methodology, how their ombudsman works, and generally how they do business.
Gartner is a fascinating story and one of the great marketing companies to operate in the enterprise software space. In fact, I don’t know anyone in the space that rivals them in marketing prowess. Gartner, at a high level, works in the following way.
- Gartner is in its essence an information broker. They get paid talking to software vendors (and collect information from them), and then turn around and get paid to talk to software buyers (and gather information from them). Brilliant!
- Gartner performs analysis and then in part sell software information to buyers, and buyer information to software vendors.
- Gartner is a money machine, however, in breaking with fundamental research principles, they do not declare which and how much different software vendors pay them.
Gartner’s Statements on Forecasting
While I could go on about Gartner all day long, this article will focus on recent statements made by them in their most recent Magic Quadrant on Supply Chain Planning for 2016.
Gartner made predictions that I will address in this article that I am as close to positive as I can be are incorrect.
- Customer Forecasting: The first of the prediction is that Big Data will be used to switch forecasting from the product and more towards the client.
- Causal Forecasting: The second prediction is that Big Data will lead to far more causal forecasting.
So let us begin with the first topic by reviewing the direct quotation.
“Nearly everyone talks about the item’s demand, and building a forecast based on what the item is doing. But what if you’ve been forecasting the wrong thing? The fact is, items don’t really “do” anything. They don’t sell themselves. They don’t make decisions. They don’t really even have a history. It’s the customer who has history, who buys, who makes decisions, who influences your stock levels. Any statistics you have on the item does not derive from the item, but from the customer. The customer’s behavior is what you should be tracking. But until the era of Big Data and sophisticated analytics, tracking and predicting behavior was not possible. Big Data and the ability to analyze customer transactions have revolutionized the understanding of customer demand, providing visibility and precision on a whole new level.”
The best way to describe this line of thinking is that it is high concept, but the most accurate way to describe it is that it is nonsense.
The Big Data Bubble
Big Data has become the sexy new concept, and now it is seen the magic elixir for a wide variety of problems. All one has to do during a pause in a presales presentation is utter something non-specific like “it’s Big Data,” or “it’s all about the Big Data,” and is just so hard for the audience to not simply nod and approve. This marks the existence of a bubble.
Conferences are awash with energetic optimism regarding the potential benefits of Big Data. This has become a huge buzzword among CEOs, who may not even know what Big Data is. This allows a person to say things that don’t make any sense but seem visionary, one reason being there is so much other hype around Big Data that almost any comment can seem reasonable. I recently tried to push the bubble myself.
I stated to someone that “Big Data is going to revolutionize laundromats because it will bring deep analytics to who the customers are and it will all be happening in real time.” It seemed to work, they agreed or at least did not protest. You can try this experiment yourself.
Why Gartner is So Wrong on This Topic
So let’s get into the detail of why I believe Gartner’s statement regarding transitioning from forecasting products to gaining deeper insights into a customer is incorrect.
- People who often like to talk about changing the forecast so as to forecast at a different attribute will often lose sight of the fundamental requirement for supply chain forecasting.
- The company is stocking products and it, therefore, must (at some point) generate a product forecast for inventory management to work properly.
As with supply chain forecasting systems, supply planning systems produce a supply plan in the form of goods and quantities to be ordered. This means a product forecast at a location, the relevant factor being over replenishment lead time (let us hold off on this of forecasting lead time as it goes how the forecast accuracy is measured, the value to supply planning and generally into another layer of complexity).
Big Data forecasting does nothing to change this foundational rule of supply chain planning.
The Complication of Customer Forecasts
Forecasting by the customer is performed by Sales and Marketing. However, it does nothing for supply chain planning. No matter what hierarchy Sales and Marketing chooses to use to produce its forecast (by product group, by sales group, by region, etc…), the supply chain forecast must be of the product, and more specifically at the product location combination.
Eventually, the Sales/Marketing forecast must be disaggregated to a product location, (which I cover in quite a lot of detail in the book Sales and Statistical Forecasting Combined).
It should be acknowledged or at least understood that the primary reason that sales create forecasts by a customer is that they use their knowledge of customers to make manual adjustments to the forecast — not that in very many situations are the customer being used to generate a better “automated” forecast.
The Normalcy of Using Customer Data
Customer data and its association with products have been used in forecasting applications for quite some time.
Any forecasting application, which can apply attributes or hierarchy, can use the customer as an attribute to perform a top-down forecast, which can then be analyzed and then used to drive the influence of the customer down to the lowest level of the forecast hierarchy.
The effect can be tested against other attributes (color of the product, product group, etc..) and then the impact of each attribute can be measured.
- Big Data Forecasting’s Involvement? None of this has anything at all to do with Big Data forecasting as I do this work comparing customer attributes versus other attributes in a forecasting application without any involvement from Big Data (as explained in this article).
- The Customer Ship to? Forecasting at the client is the general term, but in many cases, a company has multiple ship-to-locations for a single customer, therefore forecasting at the customer ship-to-location is another option. (as explained in this article.) Sometimes people who have a poor grasp of supply chain forecasting will recommend switching the forecast from the product to the customer or customer ship-to-location. The company then begins thinking regarding forecasts at the customer or customer-ship-to, thinking that the change is improving forecast accuracy. This does not work and requires lots of adjustment to get the forecast eventually back to a product location level which is what is used by supply planning.
A Conspicuous Lack of Evidence for Gartner’s Claim for Big Data Forecasting
Gartner is making a very bold claim that Big Data is going to transform forecasting to be far more customer based, but no references or other evidence are provided that Big Data is currently being used to improve customer forecasts.
So, I performed a search on a comprehensive academic paper database, and I found 28 total results from the search Big Data Customer Forecasting. However, upon review, most of these documents were not real hits on the topic, but false positives. I have been researching issues for quite some time, and I can say it would be strange for there to be no or close to no articles on a topic in academics but for Gartner to be writing on the fact that this is either happening or very close to happening.
Therefore, in my view, Gartner’s prediction on customer forecasting with Big Data is off, but Gartner was not finished proposing how Big Data forecasting.
They next moved on to making another prediction regarding something called causal forecasting. This is very similar to the forecast we just covered in that it sounds very enticing, and particularly to people that are less familiar with forecasting. Let’s take a look of that because it shows a pattern of Gartner referencing Big Data forecasting improvement.
Gartner and their Prediction on Causal Forecasting
In the most recent Magic Quadrant on Supply Chain Planning for 2016, Gartner made the following peculiar statement on the nature of the opportunity regarding Big Data forecasting:
“In particular, identifying influencers that impact customer behavior can move you closer to the holy grail of predicting the customer. Every single customer transaction is influenced by causal factors. The truth is that causal simply identifies what causes, or influences a shopper to buy something. When you know “why,” you have a much better chance at accurately forecasting “what” “when” and “how much.”
Gartner Versus the History of Causal Forecasting
This paragraph is problematic from some dimensions. The first of which being that predicting the customer is not the Holy Grail of forecasting. But to get to the meat of it, here Gartner is proposing that Big Data forecasting will allow forecasts to be created which are causal, which is somehow new to forecasting.
So what is causal forecasting, to begin with? Well, causal forecasting is where there is an attempt to identify an independent variable which can predict the dependent variable.
Using Causal Forecasting
So if you run a lemonade stand, and notice that sunny days outsell overcast days, you can use the weather forecast to plan when you will work and when you will take the day off.
First, causal forecasting is not at all new to supply chain forecasting. Quite the contrary, the ability to create causal models has existed for many years in many supply chain forecasting applications, and there are a virtually unlimited number of academic papers written on the topic and the areas where causal forecasting is applied are much greater than merely supply chain forecasting.
In fact, in supply chain forecasting causal models are very rarely used regardless of the availability of causal forecasting functionality. And the implied assumption by Gartner that the limiting factor in using causal forecasting has been not having access to Big Data is not true. Here is why:
- Unrelated Limitations to Causal Forecasting: The definition of Big Data is where the data sets that are processed are so large or complex that traditional data processing techniques are inadequate. This means using applications like Hadoop, NoSQL and so on to manage enormous amounts of often unstructured data. However, being able to process massive quantities of evidence using non-traditional processing techniques has never been the limiting factor in creating causal models.
- Actual Limitations in Causal Forecasting: The limiting factors have ranged from not having access to causal factors (because the data is not maintained or is not of sufficient quality to use) to not having the time or expertise to build causal models.
- When Causal Forecasting Tends to be Used: Causal models are often used where the number of forecasted items is small, and the financial benefit (or assumed financial interest) to forecasting is enormous. A good example of this is forecasting in the financial services industry where investment banks have few forecasted products and big budgets. Big Data does nothing to address the limitations that have caused causal forecasting to be so infrequently used in supply chain forecasting.
Gartner’s Influence and The Reaction to Their Observations
Gartner is extremely influential so what they say matters. So it matters, but it is important not to confuse mattering with being true. Here Gartner is providing verifiably incorrect information to its clients, which has a strong potential to lead their customers down pathways that are resource and time wasting.
Many people will assume that because Gartner is saying it that it must be true.
Is Gartner Simply Pumping People up on The New Sexy (Big Data Forecasting)?
Gartner’s proposal is extremely unlikely to add to forecast accuracy, but is it only being said to be-be topical or do Gartner’s analysts believe what they are writing?
The unfortunate thing, in this case, is that companies significantly underuse the functionality that is available to them in the forecasting applications that they have already purchased, and furthermore that they so often end up with inappropriate forecasting software because they don’t know how to perform software selection which I covered years ago in this article.
Distractions Galore (Big Data Forecasting Being One)
- Companies already have major issues in implementing and maintaining their current forecasting applications.
- The investment made my businesses improve prediction accuracy is low, particularly in proportion to what random forecast error costs businesses in operational inefficiencies, waste, etc..
- Companies in most cases are not applying well established and tested approaches for improving forecasting accuracy.
- There are exceptions, but often the knowledge of fundamentals of forecasting in businesses is weak, making many of them particularly susceptible to the erroneous information of the type written, in this case by a credible source like Gartner.
- Focusing on Big Data forecasting will almost certainly disperse the limited attention, funding, focus they do have, and away from actually improving their forecasting applications or choosing better ones.
Following Gartner’s advice on investigating Big Data to improve forecasting accuracy moves companies away from focusing on the real solutions that are available to them and is a distraction that has an amazingly low probability of improving forecast accuracy.
The Impact of Listening to Gartner on These Topics
Those in the position to heed Gartner’s advice on big data forecasting should be wary.
The information Gartner is providing in these two quotations is incorrect, and Gartner may or not believe it themselves (I will cover why in a future article). But it is clear Gartner is making up quite a bit about big data forecasting.
Gartner’s proposal regarding forecasting at the customer will have the result of not only not improving the forecast; it will quite to the contrary, reduce its accuracy.
Search Our Other Forecasting Content
Brightwork Forecast Explorer
How to Figure Out How to Effectively User Gartner
Whether you are a software buyer, a large or small vendor, or are wondering how Gartner can help you make better investment decisions, this book will give you new insights to Gartner’s research. By studying the methodology behind such popular analytical tools like the Magic Quadrant, you will understand how a vendor earned its rating and whether or not the ratings are justified!
Understanding Gartner, It’s History, and It’s Incentives
Starting with the history of Gartner and how it compares to other IT analyst firms, this book gives a realistic assessment of the value of Gartner research to a company and provides ideas about other resources that could complement Gartner’s analysis. You will also have the tools to level the playing field between large, medium and small vendors when using Gartner’s analysis in selecting software.
- Chapter 1: Introduction
- Chapter 2: An Overview of Gartner
- Chapter 3: How Gartner Makes Money
- Chapter 4: Comparing Gartner to Consumer Reports, the RAND Corporation, and Academic Research
- Chapter 5: The Magic Quadrant
- Chapter 6: Other Analytical Products Offered by Gartner
- Chapter 7: Gartner’s Future and Cloud Computing
- Chapter 8: Adjusting the Magic Quadrant
- Chapter 9: Is Gartner Worth the Investment?
- Chapter 10: Conclusion
- Appendix a: How to Use Independent Consultants for Software Selection
- Appendix b: What Does the History of Media Tell Us About This Topic
- Appendix c: Disclosure Statements and Code of Ethics
Sales Forecasting Book
The Problems with Combining Forecasts
In most companies, the statistical and sales forecast are poorly integrated, and in fact, most companies do not know how to combine them. Strange questions are often asked such as “does the final forecast match the sales forecast?” without appropriate consideration to the accuracy of each input.
Effectively combining statistical and sales forecasting requires determining which input to the forecast have the most “right” to be represented – which comes down to those that best improve forecast accuracy.
Is Everyone Focused on Forecast Accuracy?
Statistical forecasts and sales forecasts come from different parts of the company, parts that have very different incentives. Forecast accuracy is not always on the top of the agenda for all parties involved in forecasting.
By reading this book you will:
- See the common misunderstandings that undermine being able to combine these different forecast types.
- Learn how to effectively measure the accuracy of the various inputs to the forecast.
- Learn how the concept of Forecast Value Add plays into the method of combining the two forecast types.
- Learn how to effectively run competitions between the best-fit statistical forecast, homegrown statistical models, the sales forecast, the consensus forecast, and how to find the winning approach per forecasted item.
- Learn how CRM supports (or does not support) the sales forecasting process.
- Learn the importance of the quality of statistical forecast in improving the creation and use of the sales forecast.
- Gain an understanding of both the business and the software perspective on how to combine statistical and sales forecasting.
- Chapter 1: Introduction
- Chapter 2 Where Demand Planning Fits within the Supply Chain Planning Footprint
- Chapter 3: The Common Problems with Statistical Forecasting
- Chapter 4: Introduction to Best Fit Forecasting
- Chapter 5: Comparing Best Fit to Home Grown Statistical Forecasting Methods
- Chapter 6: Sales Forecasting
- Chapter 7: Sales Forecasting and CRM
- Chapter 8: Conclusion