How to Best Understand Forecastability

Executive Summary

  • Forecastability is an important concept of forecasting.
  • We cover how the law of large numbers can be used to improve supply chain design.

Introduction to Forecastability

Forecatability is a critical concept in forecasting that can be used to help improve forecast accuracy. You will learn how forecastability can be best used as an analytical device.

What is Forecastability?

The first thing to understand is that the term forecastable does not refer to whether an item will or will not be forecasted. Rather the term applies to whether it makes sense to add extra effort to performing forecasting.

Forecastability is a measurement of how much likely benefit can be obtained by putting forecasting resources into a forecasted item. The measurement of forecastability moves the forecasted items into three potential categories. I quote from the book Supply Chain Forecasting Software.

“Many of the products that I have analyzed over the years from different companies are clearly unforecastable. There is a simple reason for this. Many difficult to forecast products have no discernable pattern in their demand history, and without this, no mathematical algorithm can create a good forecast. This is not generally understood. I believe that part of the reason misallocate effort on very hard to forecast products is due to a misimpression about when statistical forecasting can add value, and when it can’t.

Most companies actively increase the intermittency of their demand and reduce its forecastability by doing things like creating promotions and instituting end of quarter sales “pushes.” Customers respond to these behaviors by further batching their demand. It is well known that eventually customers become habituated to end of quarter price reductions, and postpone their buying in anticipation of the end of quarter. These are just a few examples, but there are a host of programs often initiated by companies, which make demand less forecastable than it ordinarily would be if natural or true demand were received.”

The testing for forecastability does not mean those elements that are declared as tough to forecast will not still receive a forecast.

What is the Law of Large Numbers?

In a previous article, I mentioned the Law of Large Numbers. Here we will relate it to demand forecasting ability or forecastability.

The Law of Large Numbers is described by Wikipedia like this.

“According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed. The LLN is important because it “guarantees” stable long-term results for the averages of some random events. For example, while a casino may lose money in a single spin of the roulette wheel, its earnings will tend towards a predictable percentage over a large number of spins. Any winning streak by a player will eventually be overcome by the parameters of the game.”

The LLN is critical for supply chain management because so much of the field is based upon predicting future demand. Understanding the LLN can help improve supply chain strategy. I was asked the following question by a reader on this topic.

“Regarding the law of large numbers, my understanding is this works when the probability is known and the number of test cases approach infinity.  How does it apply when the probability of being accurate is not known (like in forecasting)?”

The LLN does not only apply to a very extreme case — such as the one mentioned above. Items that have more volume are simply easier to forecast. For instance, if we take two distribution centers and consolidate them, the volume per SKU-L increases, and the SKU-L — which is now one instead of two increases in its forecast ability. If we look at the lowest forecast ability, it tends to be things like service parts.

The Example of Service Parts and Demand Forecasting

Service parts — if carried at a forward location, will often have intermittent demand, that is many periods will have zero demand. At a regional distribution center, the demand will be higher, and demand forecasting ability is greater. Think about it regarding counting the number of cars that come down a street in any time interval. What street will be more predictable or forecastable, a busy street in a populated center, or a street on the outskirts of town?

Creating an Example Showing the LLN in Forecastability

Creating an example is very easy. Here it is:

Here are 6 locations for the same product. I have the demand for three months. I have calculated the Mean and STDev for each product location. The lower the STDev/Mean, the higher it is forecast ability.

LLN

ProductDC LocationJanFebMarMeanSTDevSTDev/Mean
ASan Diego105108.332.3528%
ALos Angeles22531.4147%
ASan Jose15182017.662.0512%
ACupertino34126.334.0264%
AOceanside13185125.3445%
ASan Juan Capistrano57971.6323%
ATotal48546154.335.3110%

Notice that once all locations are aggregated to a single location forecast capacity increases. BTW, this is a very “back of the envelope” calculation. Other factors affect forecast ability, but lower variance always helps. A more comprehensive analysis of demand forecasting ability which takes into account more factors can be performed.

Reviewing the Example

In the example listed above, one could quantify the extra shipping costs of carrying SKUs at a central location versus the costs of lower forecast ability. These can be calculated based on excess inventory, obsolescence, the cost of redeployment (if this is done at the company) as well as other factors like service levels (based upon assumptions of how much can be carried at which location). High volume items can tend to be decentralized, but low volume items generally can’t. An analysis can tell companies which products make sense to be carried wherein the supply network.

Designing the Supply Chain for Demand Forecasting Ability

A lot of businesses don’t think about demand forecasting ability when they design their supply chain or their strategy. There is a misconception that you can set up your supply chain strategy, the number of SKU-Ls you carry, your promotion frequency, and that the software or a forecasting method will solve all the problems. This is not correct. The bigger opportunity is in setting the strategy so that the SKU-Ls have high forecast ability.

The LLN is an important rule which can be used to help make better decisions regarding the supply network design among other considerations such as how many SKU-Ls to maintain. Most supply chains have been conceived in response to designs proposed by those outside of supply chain. Marketing and Sales are two primary drivers within companies that tend to make decisions that impact the supply chain and in fact the overall sustainability of businesses without consideration of important rules like the LLN.

The Oracle of Delphi providing advice and forecasts, but without domain expertise.

In this article, and in my upcoming bookSupply Chain Forecasting Software,” I make the point that a portion of the product database should be identified as unforecast-able. In his book, Michael Gilliland makes a similar point when he describes “over-fitting,” which means either developing a custom forecasting model or using best-fit functionality in an application in a uni-focal manner only attempt to fit the model to the demand history:

“But fit to history should not be the sole consideration when choosing your forecasting model. Blindly choosing the best fitting model and assuming it is the most appropriate for forecasting can be a problem in some forecasting packages, or in the misuse of those packages. Remember again that our objective is to create good forecasts…The key point for selection should not be the fit of the model, but the appropriateness of the model to the nature of behavior you are trying to forecast.”

He then uses the example of attempting to predict the results of flipping a coin, which of course cannot actually be predicted.

“You could fit a sophisticated model to this pattern. You could even fit the pattern perfectly and project it into the next year. But this is not the right forecast. You would have over fit such a model to the randomness. The proper model in this case is a straight line at 50% heads. Even though its fit to the history is not great and it won’t forecast particularly well, 50% heads is still the most appropriate forecast to use. It will deliver the most accurate and unbiased forecast possible over time.”

Does This Example Make Any Sense

What I really like about this example is that Michael uses a ridiculous example, the forecasting of the unforecastable, coin flips. He essentially states that a very complex model can be used to perfectly model history of anything, but that modeling history is essentially easy, and the question that is attempting to be answered is the modeling of the future. For difficult to forecast items, the whole point is that a replica of the past will not forecast the future.

While Michael’s example is deliberately ridiculous, this is no joke. People attempt to forecast unforecastable things all the time, and to use a lot of math, or smoke and mirrors to cover up the fact that the item of interest is not forecastable. This is one of the reasons why Wall Street employs so many mathematicians.

Complicate math is, in essence, the new mysticism. For anything to be considered mystic, it must be incomprehensible but have an aura of legitimacy. This extends to Kings consulting mystics or oracles as to the results of battles, weather, the harvest, etc..

Across many centuries leaders from Athens and Rome would travel to Delphi in modern Greece to consult the Oracles (mystical women) to find out the answers to the future one the most important matters of state. Interestingly we still have a record of some of their predictions or recommendations. Today, going to a Greek mountaintop for projections is considered quaint, and something to be laughed at, and of course, we are a scientific society, which is why we go to a magical island for our fake forecasts.

Adjusting for Overfitting

Michael brings up a method that I have used in the past to control for overfitting, which I learned from Bill Tonetti at Demand Works. Demand Works Smoothie made this very easy to do because its adjustment to model options is so flexible. This is to perform a forecast for the most recent previous periods and then compare the fitted model to a naive forecast. I have used this approach several times in testing attributes, to see which would work best to be incorporated into the companies top-down forecasting solution. Michael describes this approach below:

“A better approach is to withhold some of the most recent historical data, build a model based on the earlier history, and then test the model performance over the holdout sample.”

Conclusion

Overfitting is something which is done all the time, and it results in a worse forecast than a moving average or naive forecast. Humans have a natural tendency to think they can control things and predict things that they can’t. One way which this is reinforced is by taking a group of forecasters, say in financial forecasting, and then choosing the top few who beat the market.

This happens quite frequently in the financial press, with those that beat the market on any occasion becoming “stars,” or gurus and this is described by William A. Sherdan.

“Given that there are thousands of stock market predictors, pure chance guarantees that at least one of them will make what seem to be remarkably accurate calls and attain guru status. Being a market guru, however, is a short-lived honor, because the likelihood of a repeat performance is remote. the odds of making a truly spectacular predictions in any year is one in a thousand, the odds of a repeat performance is one in a million…The eventual fall of a market guru is inevitable.”

This means that gurus’ success is due to chance, and this is made apparent by their inability to duplicate their prediction success. However, this does not stop the financial press from continuing to create gurus. The financial press simply denies this evaluation, because it simply brings up a new batch of gurus to replace the old.

Research Contact

  • Interested in Accessing Our Forecasting Research?

    The software space is controlled by vendors, consulting firms and IT analysts who often provide self-serving and incorrect advice at the top rates.

    • We have a better track record of being correct than any of the well-known brands.
    • If this type of accuracy interests you, contact us and we will be in touch.

Search Our Other Forecasting Content

Brightwork Forecast Explorer for Forecastability

Improving Your Understanding of Your Forecasting Database

Companies required an understanding of the forecastability of their datasets. Brightwork Research & Analysis offers the following software which provides the measure of forecastability.

It is free to use until it receives “serious use,” and is always free for students and academics. See by clicking the image below:

 

References

Forecasting Software Book

FORECASTING

Supply Chain Forecasting Software

Providing A Better Understanding of Forecasting Software

This book explains the critical aspects of supply chain forecasting. The book is designed to allow the reader to get more out of their current forecasting system, as well as explain some of the best functionality in forecasting, which may not be resident in the reader’s current system, but how they can be accessed at low-cost.

The book breaks down what is often taught as a complex subject into simple terms and provides information that can be immediately put to use by practitioners. One of the only books to have a variety of supply chain forecasting vendors showcased.

Getting the Leading Edge

The book also provides the reader with a look into the forefront of forecasting. Several concepts that are covered, while currently available in forecasting software, have yet to be widely implemented or even written about. The book moves smoothly between ideas to screen shots and descriptions of how the filters are configured and used. This provides the reader with some of the most intriguing areas of functionality within a variety of applications.

Chapters

  • Chapter 1: Introduction
  • Chapter 2: Where Forecasting Fits Within the Supply Chain Planning Footprint
  • Chapter 3: Statistical Forecasting Explained
  • Chapter 4: Why Attributes-based Forecasting is the Future of Statistical Forecasting
  • Chapter 5: The Statistical Forecasting Data Layer
  • Chapter 6: Removing Demand History and Outliers
  • Chapter 7: Consensus-based Forecasting Explained
  • Chapter 8: Collaborative Forecasting Explained
  • Chapter 9: Bias Removal
  • Chapter 10: Effective Forecast Error Management
  • Chapter 11: Lifecycle Planning
  • Chapter 12: Forecastable Versus Unforecastable Products
  • Chapter 13: Why Companies Select the Wrong Forecasting Software
  • Chapter 14: Conclusion
  • Appendix A:
  • Appendix B: Forecast Locking
  • Appendix C: The Lewandowski Algorithm.

The Law of Large Numbers. Wikipedia. July 4. 2014.

https://en.wikipedia.org/wiki/Law_of_large_numbers

“The Business Forecasting Deal: Exposing Myths, Eliminating Bad Practices, Providing Practical Solutions,” Michael Gilliland, (Wiley and SAS Business Series), 2010 “The Fortune Sellers,” William A. Sheriden, John Wiley & Sons, 1998

https://sacredsites.com/europe/greece/delphi.html https://en.wikipedia.org/wiki/ Delphi

https://en.wikipedia.org/wiki/Famous_Oracular_Statements_from_Delphi