Paid Off Research

How to Understand Why IT Lacks Functioning Research Entities Covering SAP

Executive Summary

  • The IT industry lacks functioning research entities.
  • Much of this has to do with how little money there is in performing actual research.


Something occurred to us recently after we published the first independent TCO study on HANA. This is what we stated.

“And apparently, IT lacks functioning research bodies that focus on SAP. We say this because years after SAP made its TCO claims, no research entity, several with infinitely larger resources than Brightwork investigated this issue. In fact, the two most prominent IT analysts (Gartner and Forrester) released SAP supportive and inaccurate information on this topic.

The overall story on HANA never made any sense. The research shows there are foundational issues that make SAP’s assertion impossible. For one, how can an immature product that is priced higher than a mature product have a lower TCO? Before one gets into the calculations, the just first approximation would seem off to anyone experienced in system implementation.”

Before we go any further, we want to call out one research entity that does a very good job covering SAP. One is Bloor Research, which you can see in this article How to Understanding Bloor Research on HANA.

But there are just very few research entities to point to that cover SAP accurately, and that is what we will cover in the following article.

The Present Day Rules of IT Research

The rules of IT research are simple. IT analysts (I speak only of those that cover SAP along with other vendors) try to maximize their revenues and profits. To do this, they get as much money as they can from vendors. This means both the largest vendors have the most money to spend to get the most preferential treatment. Further, aside from giving money to analysts to buy better positioning, the larger vendors can afford the best analyst relationship teams. Constantly updating analysts, who are often switching positions or leaving the analyst companies, is a lot of work for the software vendors. And smaller vendors have less money for doing it. The more care and feeding and flattery the vendor can give the analysts, the better your outcome. That is some of it is how good the vendor can make the vendor feel about himself or herself. That is the sign of respect for their opinion.

So, right off the bat, the research entity will censor what they write about the largest vendors. And things brings up the question of HANA and its TCO claims. The one TCO study that was published happened to be published by Forrester. It was paid for by SAP. The study is completely unreliable and should be disregarded as it was engineered backward from the desired outcome. You can read our analysis in the article How Accurate was Forrester’s TCO Study for SAP HANA.

Academic Research into IT

Academic research into IT does exist, and we very frequently review the academic research and weave it into our own research as we did for the book The Real Story Behind ERP. However, academic research is very rarely referenced or read in IT.

How Consulting Companies Undermine Research

Consulting companies are supposedly knowledge-based entities. So what is their view and presentation of research?

We can recall working for a consulting company, and when we presented a reference to academic work, we were told the following.

“The client paid for our advice on the topic, not for some academic’s view on the topic.”

This comment was particularly ironic because we were subcontracting to this consulting company and offering them the knowledge they did not possess. They obtained the project contract, in part by claiming to possess capabilities that they did not. When on the project, we were instructed to use business cards they gave us and to pretend that we were part of the consulting company.

And this is the norm, where subcontractors are kept as secret as possible (note for customers, check each of the consultants on LinkedIn, and find out how many works for the consulting company and how many works for another company or are independent consultants).

On the topic, we were offering to the consulting company, the background, and experience of the consulting company was literally zero. All of it resided with us. And we have learned from project experience, researched by others, and book by others. In this area in question, we have written some books, but much of the information comes from other sources.

That is not the answer the consulting companies want to tell their customers. This is because many consulting companies approach their customers with the perspective that they are offering customers something can’t get anywhere else. Deloitte says their “methodology” distinguishes them from anyone else. Capgemini claims the same.

I have actually evaluated methodologies for SAP implementation from many consulting companies, as I covered in the article The Real Story on SAP Implementation Methodologies and did not find anything particularly different in any of them.

Consulting companies adopt concepts and methods from outside but tend to stamp their name on it. Consulting companies will tend, to the degree that they can, hide the origin of what they offer to customers. 

Example of Consulting Appropriation

One well-known vendor in application integration has consulting companies use its software without telling the customer thinks the software was created by the consulting company.

I can recall when I worked for one consulting company, I took an inventory formula from Plossl’s Inventory Production and Inventory Control book and built an inventory calculator. When I showed it to my manager, I was asked where I figured out the math for the calculator. I told him.

And he told me,

“No matter what you do don’t tell the client where you got this from.”

Consulting companies operate not on transparency, but on secrecy and keeping things proprietary. The benefits tend to be “stupendous.” But of course not validated. 

Once again, the presentation was to be that the formula was somehow proprietary to the consulting company. On the subject of SAP indirect access, a number of SAP negotiation companies have declared to me that they have the answer to managing indirect access issues. All the outcomes for clients are “fantastic. There is no way of validating any of these claims. This is all hidden and unpublished.

Therefore, there is a sizable contingent of entities in the IT space that are not only not positively disposed to research, but quite the contrary is actually hostile to research. Consulting companies by in large don’t do research (if any reader disagrees, I am happy to learn from you, please provide an example). For example, Deloitte’s research group reports directly to marketing. Not a single word is released without being evaluated by marketing as to whether it will help increase sales.

Consulting companies create the idea that they create all of this fantastic knowledge internally (to the degree that they can maintain the illusion).

What Would IT Look Like with Functioning Research Entities?

A functioning research entity that was not trying to maximize their income from vendors would be free to perform fact-checking. And if it existed, SAP’s HANA TCO claim would have been falsified many years ago.

None of the entities that cover SAP would meet any accepted definitions of research entities. They are profit-maximizing firms that sell faux research to people that do not know what the rules of research are.

Recently someone proposed something to me that I found quite interesting about Gartner.

I quote:

“A company like Gartner can make of break other companies. Taking money from parties looking for the truth and parties that want to become the new default in the market is a perverse incentive.

To conduct objective research and sell the outcomes, like you do, certainly sound like something that the market could use.

But is it really what they want? Are you sure that this market is about objective advice rather than buying off responsibility for decisions? Maybe both Gartners and their clients both know full well it is a scam, but since it serves both interests, everyone plays along.”

Know They are Buying a Sheild

And this brings up the question, do Gartner’s buyers also know that Gartner is corrupt and that they are not buying any real research but instead simply buying a shield again being held culpable for bad decision making?

I have thought about this, and when I have discussed Gartner with a wide variety of software buyers, I have often found that they have no idea what research rules Gartner follows. That was a reason I wrote the book on Gartner to get the word out. The people that generally know that Gartner is a scam are the software vendors. Although they normally also do not know how Gartner’s research works specifically. But they know something is fishy. This is because they have much more exposure to how the “sausage is made.” Gartner is very close to universally unpopular among all but the largest vendors.

All of this leads to remarkably uninformed decision making in IT. It also emboldens the marketing departments of many software vendors to make exaggerated claims knowing that they will not be exposed by these IT analysts. What entity can be expected to contradict claims made by software vendors if they rely upon these same vendors for a sizable portion of their overall revenues?


Brightwork Research & Analysis is a real functioning research entity. How can we prove it? Well, for one, we challenge prevailing concepts presented by software vendors, consulting companies, Wall Street analysts.

Research means challenging and testing hypotheses.

But our impact is tiny compared to the major brands in the analyst space. Furthermore, outside or Bloor and ourselves, none of the analyst firms either get into sufficient detail to be able to validate what SAP proposes from the technology. For example, Gartner tried to cover HANA from a technical perspective and fell right on their face.

The peculiar story of Gartner’s support for HANA and how the lead researcher for HANA was also the relationship manager for SAP is covered in the article How Gartner Got SAP so Wrong.

Verifying Statements on Technology

These analyst firms do not hire people who can verify statements on technology from SAP or from other vendors. Instead, their method is to simply talk to vendors and talk to (primarily) executives without any hands-on or technical verification.

The state of private (non-academic) IT research is not what anyone would expect from the outside. What we are essentially saying is that much like what we are going through politically in the US with the rise of Trump, that what is true does not matter. What matters is what illusion can be created. And we are back to one of my least favorite phrases that “perception is the reality.” And it follows that we have faux research entities that follow no research standards, and that do not and are not set up to fact check but serve as tools to repeat the messaging for the highest bidders.

New Entrants to Improve this Situation?

And perhaps most interestingly, there is no pathway out of this. New entities like G2Crowd have risen in the marketplace., but G2Crowd is a review aggregator they are not a research entity — and they serve as lead generation for vendors and are paid by vendors. None of this is declared by G2Crowd. So they follow the “Gartner” model of the disclosure. Furthermore, G2Crowd has an exploitive model where the owners of G2Crowd get rich while the people that write the reviews get nothing. And they don’t actually do any research into SAP. They have reviews of SAP products.

So much for the “new new” model.

The Problem: Thinking that Gartner is Focused on What is True

Gartner is hired by companies who fundamentally don’t understand how Gartner functions. Gartner has virtually no first-hand experience in the technologies that they evaluate and get most of their information from speaking with executives at buyers or executives at vendors and consulting firms. Gartner is also not a research entity. They compare very poorly to real research entities once you dig into the details as we did in the article How Gartner’s Research Compares to Real Research Entities. Gartner serves to direct IT spending to the most expensive solutions as these are the companies that can afford to pay Gartner the most money. Gartner has enormously aggressive internal sales goals that place accuracy far below revenue growth in importance.