Last Updated on March 17, 2021 by Shaun Snapp
- This is the only independent research into SAP’s accuracy.
- This article documents that accuracy of SAP versus Brightwork Research & Analysis.
Introduction to the SAP Accuracy Study
In most research, the topic is introduced, and the background is provided, and the data is provided second. However, this research serves two purposes. One purpose is to serve as a long-term resource for people and companies interested in fact-checking SAP. For this reason, we have placed the data first and the explanation second.
Our References for This Article
If you want to see our references for this article and other related Brightwork articles, see this link.
Lack of Financial Bias Notice: The vast majority of content available on the Internet about SAP is marketing fiddle-faddle published by SAP, SAP partners, or media entities paid by SAP to run their marketing on the media website. Each one of these entities tries to hide its financial bias from readers. The article below is very different.
- First, it is published by a research entity.
- Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department.
- If you are here just for the fact-checking, the table you wish to query is below.
- You do not need to type in the entire term. Type in the first few letters of the word you seek, and the list will begin to filter immediately.
- If this is your first time seeing this research, please click the Jump to Research Explanation link below to be taken to the full explanation of the study.
A second purpose is for those with a more academic interest in SAP’s accuracy.
*Scan the list, or use the search box in the upper right above the table.
A Study into SAP Versus Brightwork Research & Analysis Accuracy
|SAP Topic Area (Prediction)||SAP||Brightw.||1st Made by SAP||1st Made by Brightw.||Link to Article with Full Explanation|
|SAP MDM||5%||95%||2005||2009||See Article Link|
|The PP/DS Optimizer Provides Good Production Planing and Scheduling Output||0%||100%||1999||2010||See Article Link|
|The SNP Optimizer Provides Good Supply Planning Output||0%||100%||1999||2012||See Article Link|
|Best Practices are Contained within SAP||0%||100%||1990's||2010||See Article Link|
|Design Thinking Speeds SAP Implementations||0%||100%||2013||2017||See Article Link|
|S/4HANA is Complete||0%||100%||2015||2017||See Article Link|
|SAP Rapid Deployment Solution Sped Implementations||0%||100%||2012||2017||See Article Link|
|SAP ASAP Methodology Sped Implementations||0%||100%||1997||2015||See Article Link|
|S/4HANA Has a Simplified Data Model||0%||100%||2015||2017||See Article Link|
|SAP to Introduce Revolutionary Health Application Based on HANA||0%||100%||2012||2017||See Article Link|
|Ariba is Useful for Direct Procurement||0%||100%||2012||2016||See Article Link|
|HANA Will Help Employees Work from 10 to 10,000 Times Faster||0%||N/A||2014||N/A||See Article Link|
|HANA Will Run 100,000x Faster than Any Other Technology||0%||100%||2012||2016||See Article Link|
|SAP Will Have a Billion Users by 2015||0%||N/A||2012||N/A||See Article Link|
|SAP SPP||5%||95%||2006||2010||See Article Link|
|Solution Manager Would Become "the" CMS for SAP Implementations||0%||100%||2001||2012||See Article Link|
|SAP PLM||10%||90%||2004||2009||See Article Link|
|SAP ByDesign Will Beat or Shock Netsuite with a 90 MPH Fastball||0%||N/A||2014||N/A||See Article Link|
|SAP CRM 7.0 is the Best Product in the Field||0%||N/A||2008||N/A||See Article Link|
|Apple and SAP Partnership||1%||99%||2016||2017||See Article Link|
|With SAP Fiori and SAP Screen Personas, SAP Made Fiori Widely Available..etc..||0%||100%||2013||2017||See Article Link|
|Fiori is Commercially Successful, Inevitable and Broadly Used by SAP Customers||0%||100%||2014||2016||See Article Link|
|HANA is the Fastest Growing Product in the History of Software||0%||100%||2013||2017||See Article Link|
|SAP Mobility is Number One in Mobility Due to Sybase Acquisition||0%||N/A||2010||N/A||See Article Link|
|You Can Run S/4HANA on a Smartwatch||0%||100%||2015||2017||See Article Link|
|HANA Will Make Run MRP Faster than On Any Other Database||0%||100%||2015||2016||See Article Link|
|HANA Will be an Perfect Fit for ERP Systems?||0%||100%||2011||2017||See Article Link|
|SAP's Run Simple Marketing Program Reflects the Reality of SAP||0%||100%||2015||2016||See Article Link|
|SAP to Host its Own Products and not Rely on AWS||0%||N/A||2015||N/A||See Article Link|
|Netweaver to Become Major Infrastructure "Product."||0%||100%||2003||2010||See Article Link|
|In Memory Computing (via HANA) is Something New and Different||0%||100%||2011||2017||See Article Link|
|How Accurate Was SAP in Projecting Qualtrics as a Strategic Acquisition?||0%||100%||2018||2018||See Article Link|
|SAP Will Extend ECC Support Past the 2025 Deadline||0%||100%||2019||2020||See Article Link|
|Sybase Database Will be a Good Acquisition||0%||100%||2010||2012||See Article Link|
|Fiori Will Not Survive (To be replaced by new UI initiative)||N/A||TBD||N/A||2017||See Article Link|
|Is it SAP Leonardo that Ensures Frozen Ice Cream Delivery?||0%||100%||2017 (est)||2018||See Article Link|
|All Non HANA DBs are Legacy||0%||100%||2017 (est)||2018||See Article Link|
|Only SAP HANA is an ACID Database||0%||100%||2017 (est)||2018||See Article Link|
|S/4HANA in the Cloud Can be Easily Extended||0%||100%||2016||2018||See Article Link|
|SAP on Alliance with Google||0%||100%||2015||2018||See Article Link|
|Parallel Processing Works with the SNP Optimizer||0%||100%||2000||2013||See Article Link|
|Shelf Life Planning Works in APO SNP||0%||100%||2000||2010||See Article Link|
|HANA Has a Lower TCO||0%||100%||2014||2017||See Article Link|
|Hasso Plattner and his PhDs Invented HANA||0%||100%||2011||2017||See Article Link|
|The Count of Fiori Apps is High||0%||100%||2015||2017||See Article Link|
|How Accurate Was SAP in Keeping S/4HANA Exclusive to HANA?||100%||0%||2015||2016||See Article Link|
|SAP Support is a Good Value||0%||100%||Since SAP's beginning||2017||See Article Link|
|Reengineering Business Processes to Match SAP Leads to Long Term Competitive Advantages||0%||100%||1980s||2017||See Article Link|
|SAP's Integrated Suite Offers a Lower TCO||0%||100%||Since SAP's beginning||2014||See Article Link|
|SAP Simplifies The Customer's IT Landscape||0%||100%||Since SAP's beginning||2014||See Article Link|
|S/4HANA is so Different from ECC That it is Not Covered as a Normal Free Upgrade||0%||100%||2014||2016||See Article Link|
|All Non HANA Databases are Legacy||0%||100%||2014||2017||See Article Link|
|All of an SAP Prospect's Existing Applications are Legacy||0%||100%||Since SAP's beginning||2017||See Article Link|
|SAP's ERP Would be The Only System a Company Would Ever Purchase||0%||100%||Since SAP's beginning||2014||See Article Link|
|ByDesign to Run on HANA||0%||100%||2017||2017||See Article Link|
|Fiori is the Best UI in Enterprise Software||0%||100%||2013||2017||See Article Link|
|Indirect Access (Type 2) is a Legitimate Claim||0%||100%||2014||2017||See Article Link|
|SAP's S/4HANA Implementation Numbers||20%||80%||2015||2017||See Article Link|
|SAP is Finished on Oracle||0%||100%||2015||2017||See Article Link|
*Any cell with N/A means that either SAP or Brightwork did not make a statement about the line item.
Overall Accuracy Calculation and Future Accuracy Forecasting
The following is a straight average of all of the items listed in the table. This is a record of statements and predictions made over decades. In statistical forecasting, the historical forecast error is typically used to estimate future forecast error.
Following this precedent, it should be considered a useful accuracy percentage to apply to future predictions by SAP.
What should be clear is that SAP has a long-term issue with accuracy. This is the only published study in SAP’s accuracy. It is not generally understood how low SAP’s accuracy is by those that work in SAP or even those vendors that compete with SAP.
A Recognized Advantage Versus SAP
Brightwork had the advantage of making observations years, in some cases decades after SAP initially made its statements/predictions. However, Brightwork performed well even for recent SAP projections. Brightwork Research & Analysis has made several predictions that contradict both SAP and the overall market. At this point, not enough time has passed to determine if these predictions can be considered to have come true. Therefore they are marked as TBD.
To be accurate, any entity must value accuracy over other objectives. It should be remembered that companies are generally said to maximize profits, not to maximize the accuracy of their statements to customers. SAP’s score illustrates the extremes this can take.
Is Accuracy SAP’s Concern?
SAP is trying to drive revenues and is very obviously not attempting to be accurate. Individuals within SAP may or may not believe what they are saying. The study is not engaging in analyzing what statements SAP representatives know as false when they say them. It only analyzes if the statement turned out to be true.
For obvious reasons, neither SAP nor the massive army of companies and people that make money from SAP and cover up for them is not fans of the research we do at Brightwork Research & Analysis.
We have been accused by SAP and SAP IT departments of the following:
- Having an ax to grind (which we responded to in this article).
- Of biting the hand that fed us (which is the opposite of the previous argument, which is having an ax to grind). This argument (biting the hand that fed us) we agree with does not make our research untrue, merely disloyal. We guarantee no loyalty to any software vendor or any consulting company or entity, whether we have financially benefited from them or not.
SAP’s inaccuracies are repeated through a network of entities that do not care about accuracy much more than SAP. This topic is the subject of future research, with IT media entities also being scored for their accuracy on SAP.
Partners in Inaccuracy?
What SAP and other SAP proponents are careful not to do is address the conclusions of our research and analysis. SAP prefers to direct people to sources of information like
- CIO Magazine
- Gartner and other media entities and IT analysts that SAP pays.
- Or to Deloitte and Accenture, and other SAP consulting partners, entities with huge businesses built around billing for SAP consulting.
Therefore, according to who SAP recommends that their customers listen to, the universal feature is that the sources in some way have a financial tie to SAP, which we covered in Which Sources Does SAP Consider Credible?
SAP seems to have a precise definition of what is credible. The source seems to line up very carefully with sources that unthinkingly repeat what they say and where the money is involved for the entity that obeys SAP.
So how is a person trying to get a straight story to determine who is credible?
Determining Source Quality
In our view, a compelling way to determine source quality is to check their track record. Therefore, in this article, we will compare SAP’s track record to our own to see which is more reliable.
Background on Accuracy Measurement
We provide a relative accuracy measurement as part of our Honest Vendor Ratings.
We think that this type of measurement of accuracy is essential to decision making in IT. Our book Rethinking Enterprise Software Risk is highly based on evaluating the accuracy of sources.
However, this is the first time we measured, calculated, and documented a vendor’s accuracy in anything like this level of detail. It is a lot of work, and we don’t know anyone who does it. First of all, to do something like this, you would need to have financial independence from the source, which rules out virtually all media entities from doing anything like this. Secondly, there is no way of monetizing something like this. Media and IT analysts primarily make their money by promoting software vendors. One can’t collect money from vendors by challenging their assertions. It also tends to make a lot of people angry at you to boot.
Tracking Predictions and Outcomes
SAP’s and our predictions can be easily tracked through the published articles where they make the prediction. The determination of whether that prediction is accurate is a bit more tricky. But in most cases, we have informative articles that explain what the eventual outcome was.
Another trick in measurement is that some predictions, for instance, that Brightwork or SAP made, have not come true yet. But it does not mean they will not come true. In those instances, we don’t give a complete rating but instead, explain the prediction’s current state.
Finally, some predictions cannot be declared as true or false; they are not binary. Some predictions worked out to some degree. In many cases, it makes the most sense to provide a percentage.
Here is how we scored the outcomes of predictions.
- A zero percent rating would mean that the prediction did not come true, or the exact opposite happened.
- A 75% rating is that the prediction mostly came true.
- A 100% rating means that the prediction entirely came true.
This is not a “perfect science,” but (unless you work for SAP or an SAP surrogate), we think the readers will find that this is entirely a beneficial exercise. One might question the entity rating itself against SAP and the primary entities that provide information about SAP. However, the results were so noticeable that we have every incentive to grade ourselves harshly.
Something we also have planned in this area is to measure the accuracy of the statements of IT media entities concerning SAP.
There you have it. So given this comparison, Brightwork Research & Analysis beats SAP by a substantial margin in predicting SAP.
The first observation is that SAP has a deficient accuracy level for its statements. Almost none of the projections that SAP makes come true.
How are we able to have such a high accuracy level versus SAP?
Well, first of all, SAP is a very aggressive company. Secondly, most of the SAP executives that make these claims are not attempting to be accurate; they are trying to hit sales goals.
We often receive negative feedback from SAP proponents on the research that we do. SAP proponents don’t seem to have an issue with SAP’s statement or prediction accuracy, but they seem to have a problem with an independent entity that checks what SAP says.
In forecasting, there is a technical definition of bias. Bias is when the forecast is consistently positive or negative. Bias is not the same thing as forecast error. You can be wrong 100% of the time but still not prefer if your mistakes are both positive and negative. (We cover the topic of forecast bias in the article How to Best Understand Forecast Bias.
SAP’s Constant Flow of Inaccuracy
From the observations in this study, SAP has very close to the highest possible error and the highest possible bias. That is, they nearly always overstate or have a positive spin on their projections.
Interestingly, we have been accused of being biased by SAP proponents. However, upon performing this research, not only do we not display any bias, but we also barely show any error. You can’t have a preference if you have almost an error.
Forecast bias is an appealing criticism because it is a charge that can be leveled without bringing evidence. When we level the charge of bias, we provide the reason. The most common reason we provide is financial bias or, in some cases, both financial bias and contractual bias. Such as in the case with SAP partners that are both financially incentivized to support SAP and contractually limited in what they can write about SAP as per SAP’s partnership contract, as we cover in the article The Control on Display in the SAP Partnership Contract.
Judgment or Prejudgement (Bias)?
Secondly, in general argumentation such as on LinkedIn forms, the term bias is often misused to mean a view that is not in agreement with the other debater. Also, the term bias is applied inaccurately as the term prejudiced. Being prejudiced is to prejudge something. It is not to judge something. The term prejudiced is inadequately applied when used as an ad hominem to describe when a person dislikes or disagrees with something they have sufficiently tested or experienced to render a judgment. For example, if one writes a movie review without seeing the movie, they prejudged the movie. But if they saw the entire film and then wrote a critical review, it cannot be said that they prejudged the movie. That is not an accurate criticism. They judged the movie.
Overall, SAP proponents have a different definition of bias than the actual meaning. The definition of bias, according to most SAP proponents, is whether you agree with SAP. If you agree with SAP, then you have no bias. If you disagree with SAP, then you have a bias.