How Credible is Brightwork Research & Analysis in SAP?

Executive Summary

• Credibility is critical to getting research accepted.
• We address the issue of our own credibility in SAP.

Introduction

In this article, we explore our own credibility in the SAP space.

A Small, But Highly Efficient Research Entity

While small, we have an outsized reach due to its popularity. We have a higher website volume than another IT research entity with 700 employees, which we cover in the article How Does Brightwork’s Web Influence Compare With Similar Sites?

Some SAP resources who dislike the conclusions in our articles, which do not help them sell SAP projects or bill SAP hours, have seized on Brightwork Research & Analysis’s small size to declare that it means BR&A is less credible. However, this assumes that the size of an entity is correlated with research accuracy. This is demonstrably untrue. We will get into measured accuracy on SAP by some of the largest entities in the next section.

The Research Approach

The articles at Brightwork are written without concern for hedging or trying to develop a politically correct result. We write exactly what we conclude without going through a filtration step. Our view is that when you begin to filter a research conclusion, you aren’t actually doing research but have moved into a different area that is closer to marketing. 

Analysis of articles written about SAP by Gartner, Forrester, and Constellation Research (as examples), demonstrate clear signs of the authors censoring themselves. The traditional IT analyst writes in a very neutral way. More specifically, the IT analyst writes their conclusions based upon, in great part, triangulating backward from what is acceptable to say rather than what is true. 

We cover in the book Gartner and the Magic Quadrant a Guide for Buyers, Vendors, and Investors that the IT analyst space does not stack up very well against real research entities, such as academic research, Consumer Reports, or The RAND Corporation. There are certain rules to research, and they mostly originate in the scientific method, which declares that.

1. One works from the evidence to a conclusion rather than the other way around.
2. Second, that hypothesis is tested and not merely accepted (the concept of skepticism)
3. ..and that the researcher removes himself or herself from conflict.

These rules are enumerated in several places. However, private research has a generally poor history of following these rules. In this sense, we refer to research that requires independence, not research BMW might perform to improve its automobiles.

SAP loyalists have previously attacked this lack of neutral tone as reducing the credibility of Brightwork Research & Analysis. One SAP resource we debated and who subsequently wrote an article critical of Brightwork Research & Analysis declared that the material presented in an article HANA was “fake news” — based on it having:

“Not the right tone.” 

However, when asked to support the claim that our research conclusion was false, they admitted that they did not know themselves but used the tone to conclude.

Brightwork does not have the major analyst firms staff that can employ six people to review articles and make sure every corner is rounded off. This is what Gartner and Forrester do, and they are not doing research. They are reverse engineering conclusions based upon what is profit-maximizing. But even if every corner is rounded off, SAP resources tend to live in an echo chamber and believe several biased assumptions such as SAP have the greatest technology in software, Fiori is the best user interface in software, HANA is the fastest database in the world (running 100,000x faster than any competing technology according to Bill McDermot) and that Hasso Plattner mostly designed it. 

We never triangulate conclusions based upon what will be accepted by the largest number of people. 

Brightwork’s Unmatched Accuracy on Predicting SAP

Curiously, it is rare for the research entity’s track record to discuss credibility in the conversation. For example, due to large sums of money paid to Gartner by client-server vendors, they provided inferior quality advice on mainframes, as we cover in the article How to Understand Gartner’s Disasterous Advice on Mainframes.

A primary reason that Gartner and others can get away with offering such financially rigged and poor quality advice is that the IT analyst market is very shallow. Gartner has made many acquisitions, which have greatly reduced competition in the space and provided a monoculture type to the IT analyst market. IDC is another major IT analyst, but they don’t really cover the same things as Gartner. Forrester is now so small about Gartner that Gartner has, in effect, a monopoly, particularly on the lucrative vendor comparison market. Websites like G2Crowd are popular, but they are a lead generation front end for vendors, and they are not in the same space as Gartner.

Neither Gartner, Forrester, nor Constellation Research has anything approaching Brightwork’s accuracy in predicting SAP. Rather than challenging SAP’s assertions, Gartner, Forrester, and Constellation Research tend to align themselves with SAP’s statements. Any IT analyst that does this will have poor accuracy.

Our accuracy can be reviewed in the article A Study Into SAP’s Accuracy.

What is clear by reviewing the list is that if companies followed Brightwork’s advice, they would vastly improve their SAP spend.

How We Attained the Best Accuracy in Enterprise Software on Predicting SAP

Brightwork can attain this accuracy because we use our experience of being on projects, not spending a lot of time talking to executives, and the fact that we are actually focused on doing good research rather than profit-maximizing the business.

Our view is that as soon as any research entity attempts to follow a profit maximization route, research is dead at that entity. Anyone wanting to leverage Brightwork’s research output should reference our accuracy. Our statement in response to “SAP homers” is the following. 

“If Brightwork Research & Analysis has an ax to grind with SAP, has a problem with Hasso Plattner and Bill McDermott, and (fill in the blank), why is Brightwork so increadibly accurate about SAP?

In the world of forecasting, a positive or negative bias results in the forecast being inaccurate — and consistently inaccurate.

A perfectly accurate or nearly perfectly accurate forecast cannot be claimed to be biased.

Allow us to illustrate.

This forecast has a negative bias. It has repeatedly produced a lower forecast than the actuals. 

This forecast is consistently aligned with the actuals. How can an accurate forecast be referred to as biased? It cannot. 

Moving From the Data to the Conclusion

We are not afraid to write whatever the data drives in terms of a conclusion. This is not done in the IT analyst community, which tends to do the reverse. The senior members tell IT analysts the conclusion they need to arrive at, and then the analysts rig the text to match that conclusion. 

None of the major or really nearly any of the smaller analyst firms are willing to do this. One analyst firm was known for writing material critical of SAP, as they had Oracle as a client. However, as soon as SAP bought them off, the SAP material was removed from their website. 

Brightwork lays claim to the most accurate track record in predicting HANA, and we can guarantee no other IT analyst firm will produce evidence to contest this assertion. In fact, all the major IT analyst firms’ track records are so poor that they don’t want the subject broached.

Furthermore, Brightwork is the only IT research entity that we are aware of to track accuracy. We have many articles at the site that compare the accuracy of IT analysts and IT media. 

Another article where we compared accuracy was our retrospective evaluation of Who Was Right and Who was Wrong about SAP HANA?

And we don’t only cover SAP or make predictions on SAP. We have predicted a coming bust in AI/ML.

Why?

Because the statements by vendors and consulting companies do not match these projects’ reality or capabilities.

Will Sharing Brightwork TCO Calculators be Interpreted as Flinging Mud at SAP?

This is a question we have received from several vendors. So we need to answer it here and provide the correct context, which may not be known by those that ask this question. Any material provided to a prospect comes brings up the question of the entity’s independence that produced it.

Our domain expertise around SAP is difficult to undermine. But ordinarily, SAP will respond that Brightwork is not an approved source of information. The only approved sources of information by SAP are IT analysts SAP pays. Explaining that to a client puts SAP in a difficult spot in terms of illustrating how self-centered is the policy of “approved analysts.” Imagine for a minute if Consumer Reports were declared “not an approved analyst by General Motors.” And that JD Powers & Associates, which GM does pay, is an approved analyst. How well would that go over? Furthermore, SAP history of accuracy on TCO regarding their products can be called into question as this article does on The Problem with Forrester’s TCO Study on S/4HANA.

This was an SAP funded study naturally, and when we have walked groups through the study, it has resulted in hearty laughs from the audience. The study is embarrassing, but neither Forrester nor SAP seem to appreciate why. SAP’s own funded studies can be used against them.

Why SAP Wants to Assert Lower TCO, Not Get Into the Details

SAP normally does not try to get into details because they know they will lose. Thus, the right response from a vendor would be, “let us get into the calculations and the assumptions.”

SAP Will Attempt to Move for Immediate Dismissal

If SAP can push the account to dismiss the calculation immediately, then, well, the calculation does not matter. But SAP’s can’t survive any scrutiny of their TCO claims. So outright dismissal is what needs to be stopped. A response can be,

“SAP makes very large claims for its lower TCO, how are these claims supported. And what independent entity agrees with SAP on these claims?”

The entire subject being broached is positive for any vendor competing against SAP because TCO for SAP is a house of cards. No IT analyst will tell the truth about SAP TCO because they seek to keep SAP or gain SAP as a customer. This is where Brightwork can offer remote presentation/support for the calculators, answer questions, etc. Or there can be remote questions answered by email, where Brightwork then points to supporting articles and can even create new ones to answer questions.  Brightwork does many TCO studies. See this article for the full list of The Public Brightwork TCO Calculators.

• All of the studies are done the same way.
• They are based upon regression formulas.
• They are adjustable, but only with a range. For example, a very large implementation requires a separate calculator from just a large implementation.

Credibility Versus the Vendor

Some vendors will ask about the credibility of Brightwork Research & Analysis without asking the question..

“Compared to what.”

Vendors sometimes reach out to us and ask us to help them create their own value estimation—this something we never do. But what is curious is that vendors who ask us to do this don’t appear to ask themselves how credible a TCO study is that a vendor produces. It should be self-evident that a study by an independent party is more credible than an entity trying to sell a product. However, the entity must also be truly independent. Forrester will take the vendor’s assumptions and produce a TCO study to the vendor’s specifications. JD Power and Associates create awards that deceive customers as to the quality of automobiles. Their primary customers are the lowest quality auto manufacturers. These are not examples of independent entities that follow research rules.

Conclusion

The credibility of Brightwork Research & Analysis in SAP is complex and dependent on the person in question. Any entity that does not parrot SAP marketing’s material lacks credibility for dyed in the wool SAP shops. But these aren’t very good potential customers in any case. On the other hand, we enjoy high credibility for many people who work on SAP projects and see the gap between SAP’s marketing department and SAP projects. We are a serious research entity and have the highest history of accurately predicting SAP. We have the studies to demonstrate that we are the most accurate source on SAP.