How to Understand Forrester's Fake S/4HANA TCO Study

Table of Contents: Select a Link to be Taken to That Section

Executive Summary

Forrester created another TCO/ROI study that was paid for by SAP.
We review the accuracy of this Forrester SAP S4HANA study.

Video Introduction: How to Understand Forrester’s Fake S/4HANA TCO Study

Text Introduction (Skip if You Watched the Video)

Forrester has a history of taking money from SAP and producing exactly the outcome results that SAP would have wanted. Forrester will do this for any vendor and frequently does, as it is a major part of their business model. This study by Forrester is a completely fake TCO study that was designed to be used by SAP’s marketing and sales to mislead its prospects that S/4HANA has a low TCO. The study is hilarious and shows how the paid off research system works. You will see how Forrester concocted this faux study with its analysis of S/4HANA.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Lack of Financial Bias Notice: We have no financial ties to SAP or any other entity mentioned in this article.

This is published by a research entity.
Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department.

Forrester Gets to Talk to Three Companies?

According to Bill McDermott, S/4HANA is the fastest-growing application in the history of SAP. So why did SAP only make three customers available to Forrester? It does not take very much time to interview customers; as SAP was paying for the study, one would think they would make an effort to bring more customers to the table.

Forrester found the following financial benefits.

Three Year Benefits

These look interesting until you see how these numbers were determined. The logic for these numbers does not make sense, as will see from the quotations. Forrester did not find any of these cost savings, and they are all fake.

Quotations from the Study

Let us look through some of the impossible quotes from the article. These quotes were stated to Forrester from executives in the companies provided to Forrester by SAP, and Forrester did not bat an eye.

A potential reason for this is that a) Forrester is being paid here to look the other way, and b) it’s scarce for anyone who works at Forrester to have any implementation experience.

But to a person with implementation experience, these quotes make little sense.

Complex Supply Chains Lead to Lack of Inventory Visibility?

“Our supply chain from extraction to market is exceedingly complicated. We have over 157 different avenues for how we go to market. By the numbers, we have 60,000 unique materials, 6,000 vendors, 4,000 customers, and an additional 10,000 partners.”

S/4HANA’s supply chain functionality still lags ECC by a substantial margin. Secondly, a complex supply chain should not impact inventory management. No matter how many, each of these locations is a simple setup as a plant material combination connected to another plant material combination. I have personally had many customers with the numbers listed above, so it is not at all surprising or more complicated than what many other companies deal with and deal with using ECC. It is a simple goods issue and goods receipt in ECC. If this quotation were from 5 years ago, ECC would be considered what solved this issue. Pre-S/4HANA, if you were to have asked SAP if they can handle

“157 different avenues for how we go to market. By the numbers, we have 60,000 unique materials, 6,000 vendors, 4,000 customers, and an additional 10,000 partners.”

Their answer would have been that ECC is running the supply chains of the largest companies in the world, so the scenario above is nothing they have not done. The study is not precise that the company moved from ECC; they may have moved from another ERP system. But this study is trying to make the burning platform move to S/4HANA when almost all the vast supply chains using SAP in the world presently are running ECC.

S/4HANA Provides Real-Time Inventories?

“The most important change is that we now know the real inventories at our mine locations and operating plants. This gives us visibility from the extraction point to the final shipping to buyers. We can now measure real-time inventory across the entire company.”

All ERP systems have immediate visibility to inventory and have had this for many years now. The term real-time inventory is used. However, systems stopped advertising “perpetual inventory” many years ago as all systems that deal with inventories have this. That is, it ceased to be a differentiator, where at one time it was. Perpetual or real-time inventory was one of the original selling points of purchasing MRP systems back in the 1980s. We cover this in the article Whatever Happened to the Perpetual Inventory.

Note to executives; please stop saying things that reflect information technology crazy mouth, like your new system provides real-time inventory. Part of the entire logic for why you are paid so much more than employees is based upon the concept that you know more than the average person. When you say things like this, it ruins the illusion.

SAP and compliant customer executives have a habit of proposing that a new item newly solves a problem that was already supposed to have been solved with a previous version. If automobile manufacturers followed SAP’s lead in this regard, automobile ads might look something like the above.

Sales People No Longer Losing Deals Due to Faulty ATP Checks?

“Our salespeople were losing deals because we could not promise when the products would be delivered. We just didn’t have visibility into our supply chain to be able to make commitments when our competitors could and did make promises. A seemingly small system problem became a competitive disadvantage.”

It isn’t easy to see how the situation with S/4HANA will be much different. S/4HANA does not have superior ATP functionality over ECC. is this company having a problem with ATP because of functionality or other reasons like their production schedule changes too often? There can be many reasons why the available quantity can be incorrect. However, this is too much nuance, and Forrester has a preconceived conclusion they need to reach, so let’s move on.

All Companies Benefited from Reduced Operating Costs and Inventory Carrying Costs?

“Each of the companies that Forrester interviewed reported significant improvements in managing operating costs, such as inventory carrying costs. Using SAP S/4HANA enabled the composite organization to combine cost data with inventory data in a way that enabled employees to optimize flow and cost of inventory.”

This might be a good time for Forrester to re-emphasize that SAP did not allow Forrester to speak to any customers who bombed their S/4HANA implementations or even to the average S/4HANA customer. But our research supports the fact that most S/4HANA implementations (as covered in the Implementation Study of S/4HANA) have been unsuccessful. That is, the failure rate is far higher than 50%. This failure rate is currently covered up by both SAP and the giant SAP consulting firms. SAP also uses an NDA for its customers that implement S/4HANA, which means that no matter how bad the experience, customers can’t talk about it. Forrester did not see fit to report on the S/4HANA NDA.

This is the problem with Forrester posing as researching while its entire sample is unrepresentative of the population. What is the value of a completely rigged sample designed to lead to a predetermined outcome? The answer is zero…or, perhaps, negative.

The Company Reduced the Need for 600 Employees…And No One Was Fired?

“The financial model is based on a composite organization that was able to reduce the need for 600 employees. Although these staff will produce additional revenue in the future, the immediate benefit is measured by the reduction in employees required to do the same work. The future value is discussed in the Flexibility section.”

This is equivalent to having your cake and eating it too.

Cost Savings…..But With No Cost Savings!

It is doubtful that a company would make 600 employees redundant, but keep them on staff. Companies today are too bottom-line oriented. And every single one of these people is an excellent fit to be moved to a new job?

Hmmmmmm…… Some of those employees, if not most of them, would have been fired. However, this is how a company can claim cost savings while not admitting that they fired people. This is a political language that circumvents the discussion of what happens to employees supposedly reduced by systems. However, this is also quite overstated. None of the projections around the labor savings of ERP systems turned out to be exact — what did happen is companies were required to staff up IT.

Notice the following quotation as to the financial benefit of the moved employees.

Massive Revenue Unrealized Because of Lack of Employees

“By centralizing planning, production, and tracking functions, we eliminated the need for 600 employees. We redeployed those people into new locations to open new mines and refineries. It’s still too early to claim any revenue, but within a couple of years, we expect that these same employees will drive growth that will be hundreds of millions of dollars.”

Let us assume a future revenue of $250 million (they don’t specify how many hundreds of millions, only plural, so $250 million would be conservative). Divided by 600 employees, this is $416,000 per employee. If this type of opportunity were available to the company, they now look incompetent to have not hired them before the S/4HANA implementation freed up 600 people. Let us review what we know about this mining company.

They don’t understand that their inventory system has been real-time for decades.
They allow investment opportunities to wither on the vine instead of hiring people to drive hundreds of millions in revenue.

How badly run is this mining company?

S/4HANA Took Nine Months to Implement?

“The composite organization no longer needed to pay the annual subscription costs or maintenance fees for older applications. The company executives told Forrester that it took an average of nine months to implement SAP S/4HANA.”

Naturally, the implementation times reported are far less than the implementation for ECC, a one-year minimum, but often has implementations stretch to 2 or 3 years or more. This is true even though S/4HANA is far less mature than ECC, and immature products take longer than mature products to implement. Every single SAP reported implementation duration is shockingly fast.

However, when information that SAP does not control is reported, the timeline is quite long. In the case of the Lidl failure, it was a total of 7 years, with S/4HANA being a subcomponent of that as they transitioned to S/4HANA after beginning with ECC. Haribo, another failure, was also seven years. Yet the companies interviewed by Forrester averaged nine months? If we include Lidl and Haribo’s implementation duration, what happens to the average of Forrester’s sample? Forrester did not even have to interview these companies as the project’s duration, and the cost were published due to the projects failing.

Real Implementation Data Points Keep Rolling In that Contradict Forrester

It was later learned that the Schweizer Post S/4HANA implementation rose to $84 million.

Wait for a second; why did this implementation cost even more than $1 million? According to Forrester, a typical S/4HANA implementation is around $877,000 and runs around nine months, with only two consultants who stay on the project for six months. Forrester’s study of SAP provided sources showed almost no variance in this.

Now to hear of an $83 million implementation is very distressing. The 12X cost overrun seems odd, as this amounts to $83M/.87M 95X the typical cost of a S/4HANA implementation, according to Forrester. This cost will rise in the future. If we include just this implementation into the Forrester “study” we end up with (3 * $.877M + $84M) or $21M. Could there be some reason that the Schweizer Post case study was not provided to Forrester by SAP? I mean, isn’t SAP interested in providing the full context of data points to Forrester? It’s a study, right???

It is curious. We keep teaching statistics and talking about math’s importance, but we don’t seem to have research entities where you can be employed to use this math honestly. Forrester can publish a study with three rigged case studies, and it goes with little notice. Math + lying does not lead anywhere.

Sales Orders are Entered in Manually?

“We design complex systems for our customers, but then the information must be converted into orders and placed into SAP. Today this is a manual process, but we are looking into the idea of having the information convert directly over, which will reduce delays and avoid errors.”

This company has not yet figured out how to upload externally generated sales orders to S/4HANA? Interesting.

These implementation costs are ridiculous. ECC and S/4HANA implementations take many months of consulting support. In this case, the consultants rolled off three months before the go-live (or they were part-time).

Immature ERP System Implemented with Two “Swiss Army Knife” Consultants

S/4HANA is a risky implementation because so much of it is still immature. If only two consultants were used, how would that have worked? A minimum is to have a consultant for materials, finance/accounting, and sales. (it is unclear if production was implemented).

Which of the two consultants covered two areas?

And further, who covered HANA? S/4HANA must be implemented with HANA, a high overhead database. S/4HANA consultants are not database consultants. So one consultant implemented S/4HANA, and the other consultant implemented HANA?

It also means that there could not have been any variability in the number of consultants. Unless one project had one S/4HANA/HANA consultant, and another project had three.

Does this mean that one consultant implemented HANA while the other implemented S/4HANA? Or HANA took three months of consulting time to implement, and S/4HANA took 12 months of consulting time (summing to 9 * 2 = 18 total months of consulting)(that is for some of the time there were 2 S/4HANA consultants and sometimes not?)
So those two consultants are “Swiss Armied” to the extreme degree. They were able to run this lean on three different projects with three different sets of consultants?

Unrealistic Implementation Durations…..from Forrester

Every report by Forrester highly understates the implementation costs to customers. There are some clients with 70 to 100 SAP consultants on-site (for multiple products), and with immense monthly billings, every client Forrester seems to interview has a tiny amount of billings. However, if you approach Deloitte and ask them to help you implement S/4HANA with two consultants for six months, they will laugh at you.

For this claim, Forrester wins our highly coveted Golden Pinocchio Award. However, as the lie was funded by SAP, and SAP knowingly provided erroneous information and would have manipulated the customers into providing false information, the award is co-shared with SAP.

This estimate is even less believable because it is an average, which means that these minimum values were the three companies’ average. That is, this is not from a single company that had some minimal scope.

We can say with high confidence that this did not happen.

Either the three companies were instructed to lie to Forrester, or Forrester changed these values as directed by SAP.

Inverted Implementation Versus License Ratio?

Notice the license/subscription costs that serve as a significant part of the costs for S/4HANA. Something that should be a major red flag is that the license costs are far below the implementation costs.

In fact the ratio of implementation costs to license costs is $833,750 / $3,000,000 or 27%. That is never the case; the implementation expense on SAP projects is a multiple of the license cost, not a fraction of the license revenue. Even an SAP sales rep with a few months under the belt knows this (it comes up during discussions around implementation estimates for customers). And the fraction here is far less than 1. If multiple of 2 were used, that would mean that all three customers in the study 2/.27 = .135 or 13.5% of a typical implementation. Two is the number that many consulting companies use in the sales process, but it is far too low. Readers can see for themselves at the Brightwork ECC TCO Calculator (this is for less than 800 users, see the included link for more than 800 users. But even a simpler ECC implementation for, say, 500 users will run up more than $20 million in consulting fees.

However, SAP makes most of its money from licenses and support, so the unrealistic consulting costs are not SAP’s problem; they are Deloitte and Accenture, etc… ’s expectations to manage.

Forrester is Still Learning About Implementation Versus License Revenues

Is Forrester seriously unaware of the implementation/license multiple demonstrated through decades of SAP implementation experience? Let us look at the Lidl example again, which was 500 million Euro. ECC implementations are routinely $100 million and often much more. SAP even has multi-billion dollar failures on its resume. The Oracle ERP “legacy replacement” (it is well known that Oracle ERP is less in implementation costs), with the Airforce, cost $500 million before the program was stopped. Sub million dollar ERP implementations from commercial vendors generally, but particularly with SAP, do not exist. Forrester seems unconcerned that other people will also read this study and see how impossibly inaccurate it is.

And the errors get perhaps (even more?) or just as extreme with the next calculation for license costs.

Where are the Costs for HANA?

Forrester is quite unclear as to what version of S/4HANA was implemented. SAP is completely misleading about how different S/4HANA Cloud is from the on-premises version. S/4HANA Cloud can be implemented quickly, but its scope is minimal, and given the licenses, which show a $3,000,000 on-premises charge, these were on-premises S/4HANA versions.

And this brings up a related question. Where is the license charge for HANA? HANA is a costly database, and S/4HANA can only run on HANA, so it appears that Forrester entirely left out the cost for HANA.

These previous two points must be taken together. Not only is the implementation versus license multiple completely deflated, but the license numbers are missing the most expensive component of the overall BOM, which is the HANA database.

Forrester has absolutely no fear of making these types of errors (or deliberate underestimations), knowing full well that some people will read the entire study.

Analyzing The Vendor or Vendor Funded TCO Estimates

Vendors view TCO discussions within the context of making a software sale. Some vendors still produce TCO calculations, and we have analyzed a number of these TCO calculations. We have used some of these estimations for our estimations and have found some good and some bad in how they were put together.

Several of the most apparent issues are listed below:

No Complete or Full TCO

We have yet to come across a true “TCO” produced by a vendor or, of course, any vendor-sponsored study (which we view as the same). While a few have been accurate in several respects, they did not estimate the client’s cost of either implementing or maintaining the application. However, the TCO is literally that; it must include the entirety of costs to bring the application into the company, keep it running, and decommission the application. I have certainly appreciated the work put in by software vendors to estimate some of the costs, but it is confusing to see “TCO” in a publication’s title and then read the following.

“…does—NOT—include analyst costs, IT costs, local data management costs, etc.”

One could imagine the following riddle:

“When is a TCO not a TCO? When it does not include all costs.”

Incorrect Duration Estimation

Vendors have a strong tendency to underestimate the duration of the software implementation. For example, in the published S/4HANA case studies, the numbers published by SAP were far shorter than the known period of ECC implementations, even though ECC is far more mature than S/4HANA. Therefore, part of underestimating TCO is underestimating the time it takes to go live.

It may undoubtedly be true that the application could be installed as quickly as estimated in a laboratory environment, but companies are not laboratories. Some vendors (other than those like SAP and Oracle, both of which have business models of lengthy implementations due to the consulting companies’ need to outsource their consulting), will complain that the client is really what holds up the implementation. This is a bit like a doctor complaining that if only patients had medical degrees, they could see many more patients and get through the day faster. It’s simply not relevant to the question of duration estimation. The following statement is included in the TCO questionnaire that we give to vendors:

“Please use historical timelines rather than ‘optimal scenario’ timelines.”

Therefore, the client’s knowledge level and the time required for knowledge transfer must be estimated, along with all of the other realities of projects. Another example of this is integration. IT departments are slow to respond with extract creation.

Software Vendors Faking Enthusiasm for TCO

While most software vendors hide their pricing information, all software vendors still like to talk about TCO in the abstract. Being in favor of TCO is like saying you favor the American flag or apple pie; you’re in favor of goodness. You can’t find anyone who opposes TCO in principle. But the devil is in the details. As long as nothing is quantified or the TCO studies are rigged, all vendors can say they have the lowest TCO. Most software vendors are against publishing TCO estimations and are even against publishing pricing information for their applications on their websites.

It is certainly not difficult to find examples of software vendors creating false financial estimates for their clients to promote sales. I used to work for one of these companies and witnessed this firsthand (although I am happy to say I never participated in or supported this). The software vendor i2 Technologies offered something called a Strategic Opportunity Assessment (SOW), which estimated the financial implications of using its software.

“Another company, whose i2 deployment was two years behind schedule because of problems with i2 software, told Nucleus the following I2 did an ROI calculation for us during the sales process it was very positive. But based upon our experience now, we will [independently] evaluate ROI in the future.” – Nucleus Research

How any company could believe a TCO or ROI study developed by a software vendor and part of a vendor’s sales process, is well, strange. In a surreal turn, this company also priced their software based upon this value estimation. Talk about a conflict of interest! The following quote from Nucleus Research is educational:

“One customer, that is still cited on the i2 Web site (2003) as a success story but stopped using i2 more than a year ago, told Nucleus:

‘What killed the ROI was the fact that i2 priced their modules based on an ROI calculation that they themselves estimated during the sales process. (i2 stated) that this would lead to $40 million in benefits. (emphasis added) There are definitely benefis to the solution, but their pricing is too high. Sometime, I should take the time to contact i2 and say you really need to look at your pricing.’”

I worked with people out of this group at i2, which was called “Strategic Services.” Strategic Services would falsify any data to obtain a sale. Strategic Services was filled with MBA from the top schools. They were not technologists, did not know the software except at a superficial level. Their job was to perform financial estimation for prospects that would lead them into purchasing i2 applications. This was called an SOA or strategic opportunity assessment.

This is the value of TCO or ROI estimations from vendors or vendor-sponsored studies — zero or below zero if you include the damage of following the advice from such studies.

The Fictional Nature of ROI Estimates

When developing the estimation approach, I never even attempted to create an ROI estimator, as it never seemed to be a realistic goal. However, even if ROI could be calculated, it is doubtful that any software vendor or consulting company should ever be entrusted to perform the calculation. After all, they are attempting to justify software purchase, so they are most obviously a biased party. And this was precisely how SOAs worked at i2. Having seen several SOAs and worked on an SOA engagement providing technical support, I found that they produced SOAs that were utterly unrealistic. The SOA would propose massive cost savings with very little to support the cost savings.

Value-Added Selling

Things reached an extraordinary place at i2 when Strategic Services began pricing some deals not based up0n a typically negotiated software license fee but based upon a percentage of the cost savings that were to come from the software. As if i2 did not have enough bias in providing ROI estimates for its software, it further increased the bias by making sales based upon a percentage of the ROI estimate. This was an entirely fictional world in which Strategic Services lived, and for whatever reason, prospects often bought the unrealistic ROI estimations produced by Strategic Services.

This was an entirely fictional world in which Strategic Services lived, and for whatever reason, prospects often bought the unrealistic ROI estimations produced by Strategic Services.

The Impact of i2’s SOAs

The impact of the SOAs was to reduce the quality of decision making among many buyers. It convinced many buyers that their purchase would be essentially free as they would receive such an enormous payback on the software. But it was all a mirage.

Go to top

Conclusion

This TCO/ROI study gets off on bad footing by having SAP completely control the sample. The sample is rather tiny (3 companies) for what is supposed to be a product that many companies have implemented (according to SAP). SAP was not willing to bring very many customers to the table, which, of course, is a major red flag.

Forrester’s study gets worse with nonsensical quotations and then ultimately falls apart, showing its seams with the ridiculous average implementation time and the implementation costs that will never be experienced by any company implementing on-premises S/4HANA. And Forrester tops it off by excluding the cost of HANA, which would have added very significantly to the cost for each of the three customers that Forrester spoke to.

This study is a way for SAP and Forrester to say to readers, “we think you are fools.”

For vendors, the study illustrates that Forrester is “open for business.” If you want to say your applications have an infinite ROI or can be implemented without a database, Forrester is your go-to source to falsify a TCO/ROI study.

Something else to note, while Forrester was paid to make this fake study, Brightwork was not paid to analyze it and show how false it was. There is no vendor or consulting firm that will pay another entity to fact check their statements and ridicule them in public. Following a profit-maximizing model, one would never write an article like this. This is why most of the information providers that operate in enterprise software don’t do it. And it is why Forrester is so comfortable writing such a blatantly false article. Any experienced SAP consultant who reads this will see it is fake, but they will stay quiet as a mouse on LinkedIn. Most of them work for organizations that also publish false information, and the only information they are allowed to release is information that is promotional of SAP.

And it is not only Forrester that is open for business. ComputerWeekly, Forbes, etc…; the enterprise software market operates so that the entities that provide the most inaccurate information receive the bulk of the income available in the market.