- Forecast prototyping is performing forecast testing.
- Forecast prototyping is critical to understanding how to make changes to the forecasting system.
Companies make a major mistake when they select one forecasting system and just trust that everything in that system will work as advertised. Instead, forecasting should be prototyped in another competitive solution to make sure that the results match.
In this article, I describe having a prototyping environment which can allow a company to simulate different forecasting approaches, (hierarchical approaches, improve forecast accuracy, formulate planning business processes and test planner workloads and model design trade-offs.) before locking in on a design for a more significant, less flexible system like APO.
Troubleshooting a Live Forecasting System
I have decided to perform prototyping on a product called Demand Works Smoothie to troubleshoot a problematic SAP DP implementation. This is important because SAP DP is not a good prototyping environment, and the way the system is currently configured is not working properly. Thus I needed a reliable external system to get the forecasting down to where my client required it.
After reviewing some products, I picked Demand Works Smoothie (heretofore known as just “Smoothie”) to do this job because of its strong ability to do best-fit forecasting (called Expert selection in Smoothie). The fact that it has resident within it the forecasting models that I needed to test and the fact that DemandWorks has knowledgeable support. It is important to point out that DemandWorks can be used for a lot more than I am describing here. However, this is the need that I initially selected Smoothie for.
Data Storage and Manipulation
One of the most critical factors for a good prototyping environment is having a fast and easy to manipulate data back-end. This is where SAP DP fails. SAP DP’s data backend is so onerous that it takes entirely a lot of time to make changes to it, and in my view is one of the poorest areas of functionality in the entire SAP APO suite. (see details on this topic here.) Data management in Smoothie is straightforward. It read both databases and spreadsheets. Up until this time I have only connected it to spreadsheets for data uploads.
However, the spreadsheet that is uploaded to Smoothie is very easy to manage. Essentially, all the different items that are to be uploaded are different tabs in the spreadsheet. This allows a different model to be saved as a simple spreadsheet. One spreadsheet model can hold anywhere from 2 tabs (if only essential forecasting is performed) to quite a bit more than for different measures, or if supply planning is being done (Smoothie does both demand and supply planning, with the Smoothie SP being an extension or add-on to the Smoothie forecasting application).
The Smoothie manual describes its data management in the following way:
“Almost all data in Smoothie is sorted at the base of the model. The model’s base is the level that the data is imported into. You can easily identify the model’s base level by using the Tree driller and clicking on the plus sign next to the folder.
The dot indicates that you are working with a base level item. Folders indicate that you are working with an aggregated of items (i.e. a hierarchy)
Smoothie does things differently depending upon whether you are at the base level or working with an aggregation. For example forecast model changes made in Visual Forecasting are handled differently depending upon whether you are working with a detail item or an aggregation.” – Smoothie Help
The steps were pretty straightforward. I needed an extract of my customer’s data per SKU. As my client only forecasted to one location, that decreased the SKU number significantly. Secondly, I decided to remove the deficient demand items from the product database. Deficient demand was less than three items per year. That reduces the database somewhat.
Thirdly, I need to get only those items for which there were no manual overrides. I wanted to compare a strict statistical forecast in Smoothie to one in SAP DP. If I pulled products which have been manually overwritten, I would have commingled the forecasting effectiveness of individual forecasters with the actual statistical method used. I did not want to do that, so I excluded manually altered forecasts. This would now be an apple to apple comparison.
Next, I decided to forecast using Expert mode in Smoothie simply, and the system selected the forecasts.
I then exported from Smoothie, and this provided me with a file which showed the method chosen for every product. It also provided me with several different forecast error measurements. The question was whether Smoothie could produce forecasts that were of higher quality than the forecasts produced by DP. This should be relatively easy as the SAP DP implementation that I was evaluating was primarily going out on one model.
Flagging Unforecastable Items
It is an illusion to think that all items in a product database can be forecasted. Items of high variability and intermittency, at some point, cannot be forecasted, and need to be placed on consumption-based logic in supply planning. Thus these product types do not need to be in the planning engine (although they do need to be in the S&OP engine, and this can be accomplished by generating two feeds, one from the forecasting engine for forecastable products, and one from the supply planning engine providing non-forecastable products).
In determining which products should not have a forecast, I decided to use MAPE, MASE and the R-squared (which in Smoothie is a correlation between the statistical forecast and the naive of mean averaged over time.)
In this way, I was able to segment my client’s items into forecastable and unforecastable products. This was helpful in clearing out a lot of the product database from the forecasting engine that was simply not adding any value being there. To find out more about determining forecastability see this article.
Ease of Doing Things Versus the Ability to do Things: The Problem with Feature and Functionality Comparisons
One of the major misunderstandings about SAP DP specifically (although it applies to SAP generally) is that SAP tends to have a very large amount of functionality. In a “feature and functionality” comparison, SAP tends to win those comparisons. However, the truth is that software used does not come down to simply features and functionality. This is because an application with a more narrow set of functionality is often superior in usage to an application with a broader range of functionality because the functionality of more narrow applications tends to work better. In fact, the straight comparison of features and functionality is problematic as it promotes software vendors to add as much functionality as possible, rather than trying to improve the most important functionality or to invest in making the functionality unique in some dimension. If vendors think that they will win software selections by cramming not particularly useful functionality into their applications, then that is what they will do. Secondly, the entire feature/functionality “bake-off” orientation pushes vendors into developing capabilities in areas that they lack competence.
If we take the example of SAP DP and restrict the discussion to statistical forecasting, there are several features that make Smoothie far more effective and far easier to use and to maintain than DP.
- Best Fit Functionality: Smoothie runs best fit automatically. It runs the best fit procedure as the data is loaded. SAP DP requires best fit (called automodel selection) to be run interactively (with the user selecting to have it run) or in background, which must be set up by IT. There are many more details to this topic, but Smoothie runs the best fit procedure far more easily and far more quickly than SAP DP, requiring far fewer technical skills. So while both the applications “have”best-fit functionality, the difference between how the best fit functionality can be run is night and day.
- Grouping and Hierarchy: When it comes to the grouping and hierarchy management, again the difference is night and day. SAP DP’s complexity concerning hierarchies means that on most DP projects, the hierarchy is rarely changed. Even discussing hierarchies in DP means understanding a specific vocabulary to discuss making changes or how they work in the IT sense. Smoothie, on the other hand, can add attributes by simply adding them to a flat file and making the upload to Smoothie. (when using a database this is more involved, but Smoothie can be run with files with no database, SAP DP cannot). With Smoothie, the creation of the hierarchy is performed during the upload. As a user of the software, there is nothing else to do and the creation of this hierarchy is very fast, and very easy to change.
Therefore, the takeaway in this topic is that Smoothie does things much more easily, smoothly and with lower effort and maintenance than SAP DP.
The Issue with SAP DP Maintenance
We tune SAP DP systems professionally, and we simply do not encounter SAP DP systems that are properly maintained. This may be due to our exposure, but it is natural that systems that are difficult to maintain often will not get the level of maintenance they require. Forecasting departments do not have the bandwidth to optimize difficult to maintain systems and must have an easy to use and manage system if not for forecasting in production at the very least for forecast testing. We would argue that SAP DP has actually set back forecasting at the companies where it has been implemented. The reason is that keeping DP working technically, combined with the user overhead of simply using DP is so high that it consumes the bandwidth of planning department leaving too little left over to actually focus on the business side of forecasting.
Companies that do not have a system to use other than SAP DP where various hypotheses can be tested demonstrate a strong tendency to use guesswork to determine how to configure SAP DP, and they pay a heavy price in forecast accuracy.
The Logic for Using Both SAP DP and Demand Works Smoothie
Customers that have purchased and implemented SAP DP typically don’t appreciate the concept of removing DP. Secondly, time and money has gone into training resource both in demand planning but also in sales or marketing to use the SAP DP planning book (user interface).
However, it should not be the goal to use as few forecasting systems as possible. There is no reason, no financial reason, no user training reason and no functionality reason, not to use Smoothie alongside SAP DP. For forecast testing, one of the most underestimated areas of importance in forecasting, Smoothie is one of the best systems available. Companies that buy Smoothie can bring powerful forecasting testing capabilities into their company.
*This should not be interpreted as an endorsement for the idea that forecasting is like “self-service BI” performing this type of testing requires specialized skills and cannot be delegated to people that already have a full plate or significant operational responsibilities.
Smoothie can be used to validate results in SAP DP, and can, to the degree it is accepted by the company, even replace things that SAP DP does. We are focusing on testing in this section, but there are other benefits as well. For example, the company would also bring Smoothie’s excellent user interface into its company, enhancing the ability to provide forecast training to users. Smoothie is also one of the best applications for training people not only on software, but on forecasting concepts in general.
After many years working both in SAP DP and Smoothie, we conclude that a combined Smoothie/SAP DP solution would have to have to have a lower TCO and higher achieved forecast outcome than SAP DP alone. And the measurement should be a combination of the TCO along with the forecasting outcomes, not how few applications the company can get by.
SAP DP has been on the market for close to 20 years and is now a well-known quantity. We conclude that SAP DP cannot achieve the objectives that companies have for forecasting all by itself. This seems like a strong statement, and while it is hypothetically possible, it is not practically possible because no company is willing to staff at the level necessary to get SAP DP to this level of maintenance. Furthermore, doing so would be incredibly wasteful.
The best option for companies that have already invested in SAP DP is to find a second forecasting application, Smoothie being an excellent choice, and use Smoothie to fill in the holes in SAP DP. We have been doing this ourselves with good tested outcomes.
Brightwork Forecast Explorer for Monetized Error Calculation
Improving Your Forecast Error Management
How Functional is the forecast error measurement in your company? Does it help you focus on what products to improve the forecast? What if the forecast accuracy can be improved, by the product is an inexpensive item? We take a new approach in forecast error management. The Brightwork Explorer calculates no MAPE, but instead a monetized forecast error improvement from one forecast to another. We calculate that value for every product location combination and they can be any two forecasts you feed the system:
- The first forecast may be the constant or the naive forecast.
- The first forecast can be statistical forecast and the second the statistical + judgment forecast.
It’s up to you.
The Brightwork Forecast Explorer is free to use in the beginning. See by clicking the image below:
Forecasting Software Book
Providing A Better Understanding of Forecasting Software
This book explains the critical aspects of supply chain forecasting. The book is designed to allow the reader to get more out of their current forecasting system, as well as explain some of the best functionality in forecasting, which may not be resident in the reader’s current system, but how they can be accessed at low-cost.
The book breaks down what is often taught as a complex subject into simple terms and provides information that can be immediately put to use by practitioners. One of the only books to have a variety of supply chain forecasting vendors showcased.
Getting the Leading Edge
The book also provides the reader with a look into the forefront of forecasting. Several concepts that are covered, while currently available in forecasting software, have yet to be widely implemented or even written about. The book moves smoothly between ideas to screen shots and descriptions of how the filters are configured and used. This provides the reader with some of the most intriguing areas of functionality within a variety of applications.
- Chapter 1: Introduction
- Chapter 2: Where Forecasting Fits Within the Supply Chain Planning Footprint
- Chapter 3: Statistical Forecasting Explained
- Chapter 4: Why Attributes-based Forecasting is the Future of Statistical Forecasting
- Chapter 5: The Statistical Forecasting Data Layer
- Chapter 6: Removing Demand History and Outliers
- Chapter 7: Consensus-based Forecasting Explained
- Chapter 8: Collaborative Forecasting Explained
- Chapter 9: Bias Removal
- Chapter 10: Effective Forecast Error Management
- Chapter 11: Lifecycle Planning
- Chapter 12: Forecastable Versus Unforecastable Products
- Chapter 13: Why Companies Select the Wrong Forecasting Software
- Chapter 14: Conclusion
- Appendix A:
- Appendix B: Forecast Locking
- Appendix C: The Lewandowski Algorithm.