A Study into the Accuracy of SAP

Executive Summary

  • This is the only independent research into SAP’s accuracy.
  • This article documents that accuracy of SAP versus Brightwork Research & Analysis.

Introduction to the SAP Accuracy Study

In most research, the topic is introduced, and the background is provided, and the data is provided second. However, this research serves two purposes. One purpose is to serve as a long-term resource for people and companies interested in fact-checking SAP. For this reason, we have placed the data first and the explanation second.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: We have no financial ties to SAP or any other entity mentioned in this article.

  • This is published by a research entity, not some lowbrow entity that is part of the SAP ecosystem. 
  • Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. As you are reading this article, consider how rare this is. The vast majority of information on the Internet on SAP is provided by SAP, which is filled with false claims and sleazy consulting companies and SAP consultants who will tell any lie for personal benefit. Furthermore, SAP pays off all IT analysts -- who have the same concern for accuracy as SAP. Not one of these entities will disclose their pro-SAP financial bias to their readers. 
  • If you are here just for the fact-checking, the table you wish to query is below.
  • You do not need to type in the entire term. Type in the first few letters of the word you seek, and the list will begin to filter immediately.
  • If this is your first time seeing this research, please click the Jump to Research Explanation link below to be taken to the full explanation of the study.

Jump to Research Explanation

A second purpose is for those with a more academic interest in SAP’s accuracy.

*Scan the list, or use the search box in the upper right above the table.

A Study into SAP Versus Brightwork Research & Analysis Accuracy

How SAP compares in accuracy.
SAP Topic Area (Prediction)SAPBrightw.1st Made by SAP1st Made by Brightw.Link to Article with Full Explanation
SAP MDM5%95%20052009See Article Link
The PP/DS Optimizer Provides Good Production Planing and Scheduling Output0%100%19992010See Article Link
The SNP Optimizer Provides Good Supply Planning Output0%100%19992012See Article Link
Best Practices are Contained within SAP0%100%1990's2010See Article Link
Design Thinking Speeds SAP Implementations0%100%20132017See Article Link
S/4HANA is Complete0%100%20152017See Article Link
SAP Rapid Deployment Solution Sped Implementations0%100%20122017See Article Link
SAP ASAP Methodology Sped Implementations0%100%19972015See Article Link
S/4HANA Has a Simplified Data Model0%100%20152017See Article Link
SAP to Introduce Revolutionary Health Application Based on HANA0%100%20122017See Article Link
Ariba is Useful for Direct Procurement0%100%20122016See Article Link
HANA Will Help Employees Work from 10 to 10,000 Times Faster0%N/A2014N/ASee Article Link
HANA Will Run 100,000x Faster than Any Other Technology0%100%20122016See Article Link
SAP Will Have a Billion Users by 20150%N/A2012N/ASee Article Link
SAP SPP5%95%20062010See Article Link
Solution Manager Would Become "the" CMS for SAP Implementations0%100%20012012See Article Link
SAP PLM10%90%20042009See Article Link
SAP ByDesign Will Beat or Shock Netsuite with a 90 MPH Fastball0%N/A2014N/ASee Article Link
SAP CRM 7.0 is the Best Product in the Field0%N/A2008N/ASee Article Link
Apple and SAP Partnership1%99%20162017See Article Link
With SAP Fiori and SAP Screen Personas, SAP Made Fiori Widely Available..etc..0%100%20132017See Article Link
Fiori is Commercially Successful, Inevitable and Broadly Used by SAP Customers0%100%20142016See Article Link
HANA is the Fastest Growing Product in the History of Software0%100%20132017See Article Link
SAP Mobility is Number One in Mobility Due to Sybase Acquisition0%N/A2010N/ASee Article Link
You Can Run S/4HANA on a Smartwatch0%100%20152017See Article Link
HANA Will Make Run MRP Faster than On Any Other Database0%100%20152016See Article Link
HANA Will be an Perfect Fit for ERP Systems?0%100%20112017See Article Link
SAP's Run Simple Marketing Program Reflects the Reality of SAP0%100%20152016See Article Link
SAP to Host its Own Products and not Rely on AWS0%N/A2015N/ASee Article Link
Netweaver to Become Major Infrastructure "Product." 0%100%20032010See Article Link
In Memory Computing (via HANA) is Something New and Different0%100%20112017See Article Link
How Accurate Was SAP in Projecting Qualtrics as a Strategic Acquisition?0%100%20182018See Article Link
SAP Will Extend ECC Support Past the 2025 Deadline0%100%20192020See Article Link
Sybase Database Will be a Good Acquisition0%100%20102012See Article Link
Fiori Will Not Survive (To be replaced by new UI initiative)N/ATBDN/A2017See Article Link
Is it SAP Leonardo that Ensures Frozen Ice Cream Delivery?0%100%2017 (est)2018See Article Link
All Non HANA DBs are Legacy0%100%2017 (est)2018See Article Link
Only SAP HANA is an ACID Database0%100%2017 (est)2018See Article Link
S/4HANA in the Cloud Can be Easily Extended 0%100%20162018See Article Link
SAP on Alliance with Google0%100%20152018See Article Link
Parallel Processing Works with the SNP Optimizer0%100%20002013See Article Link
Shelf Life Planning Works in APO SNP0%100%20002010See Article Link
HANA Has a Lower TCO0%100%20142017See Article Link
Hasso Plattner and his PhDs Invented HANA0%100%20112017See Article Link
The Count of Fiori Apps is High0%100%20152017See Article Link
How Accurate Was SAP in Keeping S/4HANA Exclusive to HANA?100%0%20152016See Article Link
SAP Support is a Good Value0%100%Since SAP's beginning2017See Article Link
Reengineering Business Processes to Match SAP Leads to Long Term Competitive Advantages0%100%1980s2017See Article Link
SAP's Integrated Suite Offers a Lower TCO0%100%Since SAP's beginning2014See Article Link
SAP Simplifies The Customer's IT Landscape0%100%Since SAP's beginning2014See Article Link
S/4HANA is so Different from ECC That it is Not Covered as a Normal Free Upgrade0%100%20142016See Article Link
All Non HANA Databases are Legacy0%100%20142017See Article Link
All of an SAP Prospect's Existing Applications are Legacy0%100%Since SAP's beginning2017See Article Link
SAP's ERP Would be The Only System a Company Would Ever Purchase0%100%Since SAP's beginning2014See Article Link
ByDesign to Run on HANA0%100%20172017See Article Link
Fiori is the Best UI in Enterprise Software0%100%20132017See Article Link
Indirect Access (Type 2) is a Legitimate Claim0%100%20142017See Article Link
SAP's S/4HANA Implementation Numbers20%80%20152017See Article Link
SAP is Finished on Oracle0%100%20152017See Article Link

*Any cell with N/A means that either SAP or Brightwork did not make a statement about the line item.

Overall Accuracy Calculation and Future Accuracy Forecasting

The following is a straight average of all of the items listed in the table. This is a record of statements and predictions made over decades. In statistical forecasting, the historical forecast error is typically used to estimate future forecast error.

Following this precedent, it should be considered a useful accuracy percentage to apply to future predictions by SAP.

What should be clear is that SAP has a long-term issue with accuracy. This is the only published study in SAP’s accuracy. It is not generally understood how low SAP’s accuracy is by those that work in SAP or even those vendors that compete with SAP. 

A Recognized Advantage Versus SAP

Brightwork had the advantage of making observations years, in some cases decades after SAP initially made its statements/predictions. However, Brightwork performed well even for recent SAP projections. Brightwork Research & Analysis has made several predictions that contradict both SAP and the overall market. At this point, not enough time has passed to determine if these predictions can be considered to have come true. Therefore they are marked as TBD.

To be accurate, any entity must value accuracy over other objectives. It should be remembered that companies are generally said to maximize profits, not to maximize the accuracy of their statements to customers. SAP’s score illustrates the extremes this can take. 

Is Accuracy SAP’s Concern?

SAP is trying to drive revenues and is very obviously not attempting to be accurate. Individuals within SAP may or may not believe what they are saying. The study is not engaging in analyzing what statements SAP representatives know as false when they say them. It only analyzes if the statement turned out to be true.

Research Background

For obvious reasons, neither SAP nor the massive army of companies and people that make money from SAP and cover up for them is not fans of the research we do at Brightwork Research & Analysis.

We have been accused by SAP and SAP IT departments of the following:

  1. Having an ax to grind (which we responded to in this article).
  2. Of biting the hand that fed us (which is the opposite of the previous argument, which is having an ax to grind). This argument (biting the hand that fed us) we agree with does not make our research untrue, merely disloyal. We guarantee no loyalty to any software vendor or any consulting company or entity, whether we have financially benefited from them or not.

SAP’s inaccuracies are repeated through a network of entities that do not care about accuracy much more than SAP. This topic is the subject of future research, with IT media entities also being scored for their accuracy on SAP. 

Partners in Inaccuracy?

What SAP and other SAP proponents are careful not to do is address the conclusions of our research and analysis. SAP prefers to direct people to sources of information like

  • ComputerWeekly
  • Forbes
  • CIO Magazine
  • Gartner and other media entities and IT analysts that SAP pays.
  • Or to Deloitte and Accenture, and other SAP consulting partners, entities with huge businesses built around billing for SAP consulting.

Therefore, according to who SAP recommends that their customers listen to, the universal feature is that the sources in some way have a financial tie to SAP, which we covered in Which Sources Does SAP Consider Credible?

SAP seems to have a precise definition of what is credible. The source seems to line up very carefully with sources that unthinkingly repeat what they say and where the money is involved for the entity that obeys SAP.

So how is a person trying to get a straight story to determine who is credible?

Determining Source Quality

In our view, a compelling way to determine source quality is to check their track record. Therefore, in this article, we will compare SAP’s track record to our own to see which is more reliable.

Background on Accuracy Measurement

We provide a relative accuracy measurement as part of our Honest Vendor Ratings.

We think that this type of measurement of accuracy is essential to decision making in IT. Our book Rethinking Enterprise Software Risk is highly based on evaluating the accuracy of sources.

However, this is the first time we measured, calculated, and documented a vendor’s accuracy in anything like this level of detail. It is a lot of work, and we don’t know anyone who does it. First of all, to do something like this, you would need to have financial independence from the source, which rules out virtually all media entities from doing anything like this. Secondly, there is no way of monetizing something like this. Media and IT analysts primarily make their money by promoting software vendors. One can’t collect money from vendors by challenging their assertions. It also tends to make a lot of people angry at you to boot.

Tracking Predictions and Outcomes

SAP’s and our predictions can be easily tracked through the published articles where they make the prediction. The determination of whether that prediction is accurate is a bit more tricky. But in most cases, we have informative articles that explain what the eventual outcome was.

Another trick in measurement is that some predictions, for instance, that Brightwork or SAP made, have not come true yet. But it does not mean they will not come true. In those instances, we don’t give a complete rating but instead, explain the prediction’s current state.

Finally, some predictions cannot be declared as true or false; they are not binary. Some predictions worked out to some degree. In many cases, it makes the most sense to provide a percentage.

Here is how we scored the outcomes of predictions.

  1. A zero percent rating would mean that the prediction did not come true, or the exact opposite happened.
  2. A 75% rating is that the prediction mostly came true.
  3. A 100% rating means that the prediction entirely came true.

This is not a “perfect science,” but (unless you work for SAP or an SAP surrogate), we think the readers will find that this is entirely a beneficial exercise. One might question the entity rating itself against SAP and the primary entities that provide information about SAP. However, the results were so noticeable that we have every incentive to grade ourselves harshly.

Something we also have planned in this area is to measure the accuracy of the statements of IT media entities concerning SAP.

Conclusion

There you have it. So given this comparison, Brightwork Research & Analysis beats SAP by a substantial margin in predicting SAP.

The first observation is that SAP has a deficient accuracy level for its statements. Almost none of the projections that SAP makes come true.

How are we able to have such a high accuracy level versus SAP?

Well, first of all, SAP is a very aggressive company. Secondly, most of the SAP executives that make these claims are not attempting to be accurate; they are trying to hit sales goals.

We often receive negative feedback from SAP proponents on the research that we do. SAP proponents don’t seem to have an issue with SAP’s statement or prediction accuracy, but they seem to have a problem with an independent entity that checks what SAP says.

In forecasting, there is a technical definition of bias. Bias is when the forecast is consistently positive or negative. Bias is not the same thing as forecast error. You can be wrong 100% of the time but still not prefer if your mistakes are both positive and negative. (We cover the topic of forecast bias in the article How to Best Understand Forecast Bias.

SAP’s Constant Flow of Inaccuracy

From the observations in this study, SAP has very close to the highest possible error and the highest possible bias. That is, they nearly always overstate or have a positive spin on their projections.

Interestingly, we have been accused of being biased by SAP proponents. However, upon performing this research, not only do we not display any bias, but we also barely show any error. You can’t have a preference if you have almost an error.

Forecast bias is an appealing criticism because it is a charge that can be leveled without bringing evidence. When we level the charge of bias, we provide the reason. The most common reason we provide is financial bias or, in some cases, both financial bias and contractual bias. Such as in the case with SAP partners that are both financially incentivized to support SAP and contractually limited in what they can write about SAP as per SAP’s partnership contract, as we cover in the article The Control on Display in the SAP Partnership Contract.

Judgment or Prejudgement (Bias)?

Secondly, in general argumentation such as on LinkedIn forms, the term bias is often misused to mean a view that is not in agreement with the other debater. Also, the term bias is applied inaccurately as the term prejudiced. Being prejudiced is to prejudge something. It is not to judge something. The term prejudiced is inadequately applied when used as an ad hominem to describe when a person dislikes or disagrees with something they have sufficiently tested or experienced to render a judgment. For example, if one writes a movie review without seeing the movie, they prejudged the movie. But if they saw the entire film and then wrote a critical review, it cannot be said that they prejudged the movie. That is not an accurate criticism. They judged the movie.

Overall, SAP proponents have a different definition of bias than the actual meaning. The definition of bias, according to most SAP proponents, is whether you agree with SAP. If you agree with SAP, then you have no bias. If you disagree with SAP, then you have a bias.

Jump Back to the Accuracy Table