How to Understand Forecasting Lumpy Demand

Executive Summary

  • Intermittent of lumpy demand is increasing because of specific decisions by marketing, sales and other areas within companies.
  • Trader Joe’s breaks with the trend of having higher percentages of the product database being intermittent.
  • Interestingly other companies don’t copy the simple and proven techniques used by Trader Joe’s to reduce low volume products.

Introduction to Lumpy Demand

Many people have questions about how lumpy demand, also referred to as intermittent demand, which is a very frequently discussed topic in forecasting. This intersects with a much less commonly discussed topic, which is forecastability. In this article, you will learn important topics related to lumpy or intermittent forecasting.

What is Intermittent or Lumpy Demand?

Intermittent—or “lumpy”—demand is one of the most common features of a product’s demand history that makes a product unforecastable. Services parts are the best-known example of a product with lumpy demand. However, I have come across intermittent demand in many different types of companies. For instance, one of my clients was a textbooks publisher. A large percentage of their product database had an intermittent demand history which would normally not be expected of this type of product. However, due to the fact that different US states buy textbooks in large volumes whenever funding comes through, the demand ends up being quite unpredictable for many books. A school system will not make any purchase for some time, and then will buy many textbooks all at once. For example, California is on a seven-year procurement cycle, which means they wait seven years between purchases.

And this is a very important distinction that explains why demand, which one would expect to be forecastable, is much less forecastable in reality. This is because many products have significant lags and batching between procurement and consumption, unrelated to EOQ-driven ordering. In the case of this textbook publisher, the intermittency was related to when state funding was approved for textbook purchases.

The more lumpy demand history is, the more difficult it is to forecast and the less sophisticated methods can make an improvement in forecast accuracy. 

Forecasting Lumpy Demand

Forecasting lumpy demand brings up an issue in that to improve forecast accuracy it means creating a highly variable forecast, as the following example taken from a real client demonstrates.
Notice how a more accurate forecast (in Orange) produces enormous variability compared to a 12-month moving average. 

Uneven Demand

Uneven demand is the feature of tending to have periods of very low or zero demand, and then spikes of demand. There can be many reasons for uneven demand patterns. Services parts of the best-known example of lumpy demand, yet, this is inherent demand. Service parts simply tend to have lumpy demand as they are based upon breakage. They are quite a few stocked items that have lumpy demand.

Textbooks also have lumpy demand, which is because different states and municipalities purchase different textbooks at different times, and they buy in a mass purchase. Thus a school system will go for some time making no purchases, and then will buy many textbooks. In California for instance, their school books are on a seven-year cycle, and thus they wait seven years between purchases. Many companies impose a lumpy demand on themselves with their policies. One which does not is Trader Joe’s, and why is a fascinating story about how forecast-ability can be improved by active strategic decisions. They are discussed in this article.

Where is Lumpiness is Increasing?

Many products that one would not think would be lumpy are made uneven by direct actions by the company, that they just refuse to stop doing. This “unnecessary lumpiness” is primarily caused by the following:

  1. Sales and marketing that offering promotions
  2. Sales and marketing are introducing new products, without culling the database for products that have poor demand.

Why Lumpiness is Increasing?

ToolsGroup, (a best of breed software vendor in demand and supply planning) in their excellent white paper “Mastering Lumpy Demand,” points out that lumpiness is in general increasing. One reason for this is some products that must be planned keep increasing. There are several grounds for this; one is a general trend in marketing to offer more variations. A second trend is online retailers like Amazon.com, Netflix, and eBay that are leveraging online storefronts and national distribution networks to more selection than ever before. All one has to do to immediately grasp this is considered the demand history of a Blockbuster Video, vs. a Netflix. Clearly, with so many titles, Netflix happens to have more lumpy demand for their less popular items. I cover this in detail in my book “Supply Chain Forecasting Software.”

It increasingly appears that the more variety that is offered, the more variety that is demanded by customers. As is pointed out by ToolsGroup, SKUs are increasing faster than sales, and it’s not in just one industry. Other factors which ToolsGroup attributes to more frequent replenishment and more granular forecasting, and to more collaboration along the supply chain. I find these second two reasons less compelling than the long tail argument which I will focus on in this article. Anyone who wants to understand lumpy demand management needs to get their hands on ToolsGroup’s excellent paper in this area. They lay it all out.

(once on the site, download the white paper to the left titled Mastering Lumpy Demand)

ToolsGroup’s White Paper Observations

The article also points out that even in consumer packaged goods, the long tail consumes 86% of the SKUs and nearly half the 46% of the revenue. And consumer packaged goods are not the only one. In an electronics company that ToolsGroup worked with, 44% of the revenue was in the long tail. ToolsGroup points out that with lumpy demand, a necessary implication is the many zero demand periods. Any attempt to improve forecast accuracy will likely be costly and useless, as it will not lead to any relevant reduction in demand variability. Instead, ToolsGroup recommends analyzing the full probability distribution and creating a reliable statistical description of how demand behaves.

Is Extra Complexity Simply Managed with Systems?

Most sales and marketing types, as well as many strategy consultants, propose (none of whom are supply chain experts by the way), that this should be no problem. And that the supply chain should simply be able to adapt to more product proliferation and all other complexities introduced by sales and marketing, and that advanced planning software, and optimizers, advanced forecasting algorithms can manage these issues. (Either that or they propose “Lean,” can do it, whatever the complexity, there is, according to them, always a special band-aid that can make an inefficient business model design all better.)

The costs of all of this complexity are never calculated. As is pointed out by The Pricing Journal blog…

“While many stores try to be everything to everyone, they do not realize the additional costs incurred by carrying low volume (and often negative margin) products.”

Interestingly, and completely unknown to most strategy consultants, the less forecastable a product, the less (not more) useful advanced methods become. Extremely lumpy products may as well be placed on reorder point planning, which is how planning was performed before MRP was introduced.

Companies are essentially putting themselves in a position of having poorly performing and high-cost supply chains with large amounts of inventory and inventory obsolescence. (reorder point planning is discussed in the article.)

Proliferation Par Excellence

There may be no better example of an industry that has gone to the extreme with unnecessary product proliferation as the grocery industry. One grocery chain takes a different path, and this is a significant reason they perform so much better than the industry average.

“Trader Joe’s has put itself in a better position. Fewer SKUs and fewer lower turning SKUs as a percentage of the database means that Trader Joe’s will be in a better position to have a lower forecast error, and therefore a more efficient management of their inventory than would a typical supermarket.”Supply Chain Forecasting Software

Going Against the Grain in Reducing Lumpy Demand

Trader Joe’s does something unusual in this day and age, they have operations with a place at the table, co-developing the strategy and policies with sales and marketing.

How is this possible?

Sales and marketing (primarily at the food companies, which control the theme and orientation of supermarkets) have turned most supermarkets into packed to the gills, plastic environments with massive product proliferation which are quite unappealing places in which shop, and that also have high costs of forecasting and high supply chain costs.

Recovering these costs means offering more varieties of processed foods, which have higher margins, and which can sit longer due to their elevated levels of preservatives. In fact, if your inventory turns are slow because you have proliferated your product database, you just cannot afford to keep a significant percentage of fresher food. Trader Joe’s on the other hand, by designing a simpler model finds itself in a positive feedback loop. It can keep its prices low (for a better product even) because its supply chain efficiency is so much higher than the industry average.

Trader Joe’s can offer a freshness level in its food that the significant supermarkets simply can’t.

Comparing Different Methods

A study by the Decision Sciences Institute examined the following common lumpy demand forecasting methods.

Simple Moving Average

Single Exponential Smoothing

Croston’s Method

SBA (Syntetos and Boylan) Method

This research found the SBA method to be superior to the other three methods. In the book Complex System Maintenance by Murthy and Kobbacy, also found that the SBA method was superior to the SMA, SES, Croston’s method. The SBA is a bias-corrected adaptation of Croston’s estimator. These were used to predict 3000 service parts from the automotive industry. Other publications question whether SBA is all that superior to Croston’s. One such article is “On the Bias of Croston’s Forecasting Method” by Ruud Teunter and Babangida Sani. They propose that the negative bias of SBA is greater than the positive bias of Croston’s.

MCA Solutions / Servigistics and Lumpy Demand

MCA uses a variant of the single exponential smoothing forecast that is tailored to sparse and intermittent demand. This method works unless there is a significant trend or seasonality in the demand history.

Bucketing Products

One of the most important steps in dealing with lumpy demand is segmenting the products into buckets that are lumpy and those that are more stable and consistent. This is because a forecasting algorithm that works very well for lumpy demand will almost never work well for stable demand, and while much of the product database may be lumpy, most likely not all of it is. Another issue is low demand items. Low demand items that are lumpy should go out on service parts forecasting algorithm.

Forecastability

Many products are simply not forecastable. That is no statistical forecasting procedure can defeat a naive forecast. I have developed a formula (with the assistance of a previous client) that can be applied to a product database to determine which products or product location combinations are forecastable.

Conclusion

There is weak evidence that more sophisticated methods beat simple methods at forecasting for products with lumpy demand. Secondly, more advanced methods consume much more time and effort from the company, so while the results more often than not, do not improve with complex methods, the maintenance effort always does increase with them. As a practical matter, it is hard for me to see why so much effort in the forecasting literature has been spent attempting to forecast products that have such erratic histories.

The central concept to statistical forecasting (as opposed to consensus prediction which is based on judgment and domain expertise), is that there is a pattern which can be used by the forecast algorithm that is selected. If a demand history does not have this, it makes little sense to keep trying to apply more sophisticated methods. This is a critical point in the determination of how to deploy forecasting resources.

Trader Joe’s follows a grand strategy, while it is infinitely logical it is a strategy that few companies inside or outside the grocery industry can copy. One major reason is that sales and marketing are too powerful at most companies to allow decisions which are rational from a supply chain perspective to be rolled out.

Similarly to Southwest Airlines (another company with sky-high customer satisfaction), robust sales and marketing departments in competing airlines cannot allow their airline to copy a successful approach as it would reduce their control over their company’s policy.

In the aviation industry as well as the grocery industry, operations must be kept in its place, and sales and marketing must be allowed to have their way in all decisions. Supply chain management is now, merely an order taker for sales, marketing, and finance in the vast majority of companies.

  • Lumpy demand is a function of stocking decisions that are primarily controlled by the sales strategy.
  • Lumpy demand is the most difficult to forecast and translates into carrying a high stock level per the mean demand.

Remote Forecasting Consulting

  • Questions About This Area?

    The software space is controlled by vendors, consulting firms and IT analysts who often provide self-serving and incorrect advice at the top rates.

    • We have a better track record of being correct than any of the well-known brands.
    • If this type of accuracy interests you, tell us your question below.

Brightwork Forecast Explorer for Monetized Error Calculation

Improving Your Forecast Error Management

How Functional is the forecast error measurement in your company? Does it help you focus on what products to improve the forecast? What if the forecast accuracy can be improved, by the product is an inexpensive item? We take a new approach in forecast error management. The Brightwork Explorer calculates no MAPE, but instead a monetized forecast error improvement from one forecast to another. We calculate that value for every product location combination and they can be any two forecasts you feed the system:

  • The first forecast may be the constant or the naive forecast.
  • The first forecast can be statistical forecast and the second the statistical + judgment forecast.

It’s up to you.

The Brightwork Forecast Explorer is free to use in the beginning. See by clicking the image below:

References

https://docs.google.com/viewer?url=https://www.decisionsciences.org/Proceedings/DSI2008/docs/465-6531.pdf&pli=1

Promotions increase the lumpiness of demand when it is not accounted for in the demand history.

The following is the only book that exclusively focuses on promotions.

Forecast Promotions Book

PROMOTE2

Promotions Forecasting: Forecast Adjustment Techniques in Software

The Constant Issues with Promotions Planning

Promotions keep increasing in companies, but the ability to manage promotions is simply nowhere near to keeping up.

Accurately accounting for promotions is the only way to guarantee the availability of the promoted product during the promotion period. Not only do in-house promotions change demand, but so do competitor promotions; being able to forecast the impact on your demand is vital to maintaining service levels.

Promotions Forecasting in Software

All promotions can be measured for their effect on sales, and the history of any promotion can be used to adjust the forecast to account for future promotions. In this book, through the use of graphics and screenshots, you will receive a tutorial on how software applications can be used to maintain this history and adjust forecasts.

By reading this book you will:

  • Gain an overview of the different categories and types of promotions.
  • Understand how forecast accuracy is impacted by promotions.
  • Learn how to create a database of historical changes, and then use this information to predict future changes in demand when similar promotions are run.
  • Appreciate promotions from the perspective of both Sales and Marketing, and Supply Chain.
  • Recognize common issues with forecasting for promotions.
  • Create strategies for communicating with Sales and Marketing about future promotions, and for translating this information to Supply Chain and Forecasting.

Chapters

  • Chapter 1: Introduction
  • Chapter 2 Where Demand Planning Fits within the Supply Chain Planning Footprint
  • Chapter 3: The Common Problems with Statistical Forecasting
  • Chapter 4: Introduction to Best Fit Forecasting
  • Chapter 5: Comparing Best Fit to Home Grown Statistical Forecasting Methods
  • Chapter 6: Sales Forecasting
  • Chapter 7: Sales Forecasting and CRM
  • Chapter 8: Conclusion

Software Ratings: Demand Planning

Software Ratings

Brightwork Research & Analysis offers the following free demand planning software analysis and ratings. See by clicking the image below:

software_ratings