How Accurate Was Forrester’s Study on the ROI of SuccessFactors?

Executive Summary

  • Forrester was paid by SAP to produce a report that showed SuccessFactors had a high ROI.
  • Numerous errors in the report demonstrate that the study was rigged to provide SAP’s intended outcome.


Forrester published a study named The Total Economic Impact™ Of SAP SuccessFactors HCM Suite Cost Savings And Business Benefits Enabled By SAP SuccessFactors HCM Solutions, September 2018. In this article, we will review the study for accuracy, eventually assigning an accuracy score

Forrester’s Quantified Financial Benefits

Forrester declares the following financial benefits from a sample of customers provided to Forrester by SAP.

“The composite Organization experienced the risk-adjusted present value (PV) quantified benefits of the following services totaling $11,005,505 over a three-year period (see the Financial Analysis section for more details):

Employee Central and Payroll: $1,189,136.

Recruiting and Onboarding: $2,921,945.

Performance, Goals, and Compensation: $6,194,531.

Learning, Succession, and Development: $702,893.”


Forrester declares the following costs from a sample of customers provided to Forrester by SAP.

“The Organization experienced the following costs totaling $5,906,920 with a risk-adjusted present value of $5,186,681 (see the Financial Analysis section for more details):

Internal labor to plan and deploy SAP SuccessFactors: $60,000.

Incremental hardware, database and operating system license, and maintenance: $0.* The Organization incurred none of these costs with the SAP SuccessFactors cloud solution.

SAP SuccessFactors (or partner) fees for professional services implementation assistance: $265,000.

SAP SuccessFactors subscription fees: $3,751,600.

SAP Preferred Success fees: $750,320.”

How Many Customers Were Used for the Sample?

Forrester stated the following with respect to the sample that it used in the study:

“For this study, Forrester conducted interviews with six SAP SuccessFactors HCM customers. Interviewed customers are described as follows (each requesting anonymity)”

However, when one reviewed the list, there were not six entities listed, there were seven. A company each in the following industries:

  1. Manufacturing
  2. Retail
  3. Local Government
  4. Insurance
  5. Transportation
  6. Mining
  7. Outsourcing Provider.

There is no other interpretation than Forrester miscounted the companies that were in the study. While that seems like a major error, it is in fact only a 14% error, but it just seems worse than it is because it is so obvious. As we will see, that is a high point in terms of accuracy for the study.

Sample Selection Bias

The second thing we noticed was the very obvious selection bias of the sample used by Forrester.

One of the foundational quality checks in research is determining if the sample is reflective of the population. Anyone who has taken statistics would know that this subject alone makes up a large component of the course. Before any descriptive statistics are used on a sample, the first question must be answered as to representativeness. It is like a chain, the first links in the chain have to be unbroken for the pulling on the rest of the chain to work.

Getting 30 Samples?

The rule of thumb (called the Central Limit Theorem) is that one requires 30 samples or that are randomly selected from a population to be relatively confident that the sample is reflective. This is a rule of thumb memorable to most people that have studied statistics, and the following limitations apply.

  1. This is the minimum sample size, not the optimum sample size. As explained by Christopher Rout “It’s not that “30 in a sample group should be enough” for a study. It’s that you need at least 30 before you can reasonably expect an analysis based upon the normal distribution (i.e. z test) to be valid. That is it represents a threshold above which the sample size is no longer considered “small 
  2. Many researchers consider 30 the minimum number of samples to conduct a test. When we performed our research into S/4HANA Implementations we had more than 30 samples. However, in many cases, this many samples are not available. But in this case as the SuccessFactors team had access to so many customers, one has to wonder why such a small number of samples was used. SuccessFactors is not like some SAP products where it is difficult to find pleased customers. But the budget available to perform the study cannot be underemphasized. Neither Forrester or SAP have any research orientation. SAP is trying to get marketing collateral at the lowest possible cost, and Forrester wants to make some money. This would have played a role in why so few customers were selected.
  3. The confidence interval is a statistical measurement for how likely it is that a sample is representative of the population from which it is drawn. The confidence interval is a measure of probability.

However, we are getting ahead of ourselves, because statistics can only be applied if the sample is random. If the sample is cherry-picked, then no statistics of any type can be applied. When cherry picked, the probability is 100% that the sample is not representative of the population.

This is sort of the amusing thing about statistics. One can take many courses in statistics, but for most of life and analysis, just the basic statistics are necessary to understand because in most cases people are not even applying basic statistical standards when evaluating information.

Now let us address why this sample is so biased.

What Customers Was SAP Likely to Provide to Forrester?

Right off the bat, as each of these companies was provided to Forrester from SAP to interview, a question of selection bias immediately comes to mind. The first question is “why these six companies?” versus the what must be over a thousand customers that SuccessFactors has.

  • Most likely SAP would have provided the six most successful implementations that were convenient for SAP to find and with which they had an excellent relationship.

This is not a random sample; this is an exceptional sample of SAP Success Factors customers. This means that companies that attempted a Success Factors implementation and failed were excluded. Average Success Factor implementations were excluded.

Yet this is nowhere in the disclosure on the first page. Under disclosures, it does say the following:

“The study was commissioned by SAP SuccessFactors and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.”

There are no other applications included in the study, so it is difficult to see what this paragraph means.

Further, it states.

“Forrester makes no assumptions as to the potential ROI that other organizations will receive.”

Well if the ROI estimate in the study is not usable by those that read the study, what was the point of publishing the study? Moreover, the disclosure is right, because the sample has been intensively cherry picked, the ROI is not usable — and that is if the math were right, which will be explained later in the article, it isn’t. And finally, when SAP publicized the study, they contradict Forrester’s disclosure. They clearly gave readers the impression that the study is reflective of a..

“a typical organization”

Now let us review how Forrester calculated (or represented the calculations from customers) the financial benefits of the SuccessFactors implementation.

The Financial Benefits Estimates

The estimates of financial benefits are broken into the following categories.

  • Employee Central And Payroll Benefits: Calculation Table is based upon an internal staff cost of $90,000 fully loaded.
  • Recruiting And Onboarding Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour.
  • Recruiting And Onboarding Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour.
  • Performance, Goals, And Compensation Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour and the cost of managers at $55 per hour.
  • Learning, Succession, And Development Benefits: Calculation Table is based upon a fully loaded cost of internal labor of $90,000 per year.

The savings in the study came from the reduction in labor costs.

These aren’t estimates; they are what the companies said happened. If this is true, then each of these companies would have had to have eliminated HR positions. One commentary might be that the positions were not eliminated, but they were merely “reallocated to other parts of the company.” Companies sometimes write things like this as they don’t like it having it pointed out that cost savings often result in resources being fired. But what if all of the HR employees can’t be fired? Well, that would impact the ROI.

The Implementation Costs

The implementation average for the seven sample companies is stated as $265,000 for consulting and $60,000 for internal resources over a three year period.

The problem here is there is no way this is true.

Jarret Pazahanick, a SAP and SuccessFactors consultant who has done 50 SAP and SuccessFactors implementations estimated that both the internal cost and the consulting cost would have been “5-10-15X.” This is modified to be 5x for the internal resources, and between 12x to 15x for the consulting resources. Jarret’s estimate is far higher in credibility than any estimate from SAP because Jarret is not being paid by SAP and is not trying to sell SuccessFactors, but works as an independent consultant. I face the same issue. As a long time SAP consultant, my estimates are always longer and more expensive than those produced by people with a vested interest in selling software. When I made these estimates, salespeople who had never worked on an SAP project would tell me they were confident my estimates were too long, and that they would interfere with the sale.

Jarret went on to state the following in a LinkedIn comment.

“It is SO beyond the realm of what a real implementation of SuccessFactors really take it is mind boggling. The backchannels are already start to buzz of upset partners and customers wondering why their implementations and quotes are so much more that what SuccessFactors provided Forrester as what the cost would be to get that great ROI (Obviously all made up).”

If the middle figure is taken from Jarret’s estimate, we end up 5x for the internal resources, and let us take a midpoint of between 12x and 15x for the consulting resources. This is 13.5x. 13x of 265,000 is $3,577,500 for consulting. 5x of 60,000 is 300,000.

If we redo the ROI using these estimates, we end up with the following.

The Overall ROI

Now we can contrast this with Forrester’s analysis of the ROI.

“Based on Forrester’s experience, the six-month payback period and 112% ROI numbers are strong when compared to other enterprise cloud implementations. In addition, the interviewed customers that migrated from SAP HCM on-premises software to the SAP SuccessFactors cloud solution had better benefit results.”

So we calculate 31.34% ROI. Even that would be a very high ROI for any IT implementation. However, the sample is cherry-picked so that ROI is meaningless. Furthermore, there are several problematic features of the benefits estimation.

  • There is no independent verification being performed by Forrester. Forrester asked the customers themselves to perform the estimates.
  • Customers have a bias to estimate their benefits higher and their costs lower as those that are queried were often part of the software selection. Therefore the responses they provide are a way of congratulating themselves on their decision-making skills.
  • Customers in my experience don’t typically know these things. This is because they are more focused on keeping fewer employees capable of performing any type of research or estimation. Now, there is nothing to stop companies from hiring these skills. However, they usually don’t as they don’t see them as pertinent for their objectives.

Therefore, even among the cherry-picked sample, the estimates are most likely extremely unreliable. As we just analyzed, the estimates of the internal and external costs are off by order of magnitude of somewhere around 10x, and the costs are the easiest areas to estimate! This is because you have receipts for what you paid for something. The problem is that Forrester received the consulting costs from SuccessFactors, and would not themselves know the cost of a SuccessFactors implementation.

Benefits are far more tricky to estimate. This is why at Brightwork Research & Analysis, we have online TCO calculators. Unlike Forrester, no one paid for these calculators to be produced, so they show a far higher TCO than any of the vendors would like to see published.

However, we did not attempt to estimate the ROI of the 53 different applications we estimated. By way of example, review our analysis of the problems faced with calculating lost sales and forecast error costs to gain an appreciation for the difficulty of estimating benefits from software implementations.

Curious Questions

Secondly, how did all seven companies end up measuring the benefits in the exact same way (that is with labor savings)? The answer is that there is no way they did. This means that the companies reported back on a survey to Forrester. That is Forrester setup the construct of the cost savings, and the cherry-picked customers responded to the survey. This increases the “fishiness” of the study because it means that even less thought was put into the estimates. For example, did every one of the customers agree with the benefit calculation approach? That would be odd if that were the case.


The study is interesting in that it illuminates Forrester’s process. However, the output of the study is not useful for anything for those seeking to evaluate if SuccessFactors is a good investment. SAP paid Forrester to produce a study showing that SF has a very high ROI. And Forrester produced the study that said what SAP wanted it to say. Observe how SAP marketing explained the study.

“SAP SE (NYSE: SAP) today announced the results of Forrester Consulting’s “Total Economic Impact™ of SAP SuccessFactors HCM Suite” study,* which found that a typical organization** that invests in the SAP SuccessFactors HCM Suite could realize potentially $11 million in total quantified benefits over a three-year period.”

Is that what the study found? No.

Also, are seven companies selected by SAP reflective of a typical SF implementation? That is as close as one can come to saying that the study was reflective of what a typical company should expect, which is the opposite of what the disclaimer on the Forrester study states. SAP has a history of misrepresenting the findings of a study they pay to have performed. SAP has no consideration for the reputation of the entities they buy research from. Their perspective is the same as the one they apply to customers, which is to extract the most from every possible transaction.

Also, notice the endorsement from the President of SuccessFactors.

“In our view, Forrester’s holistic study shows that SAP SuccessFactors solutions can drive not only better employee experiences but also significant cost savings,” said SAP SuccessFactors President Greg Tomb. “We are proud to be helping businesses better understand and serve their employees while simultaneously streamlining business processes and reaping significant cost benefits.”

Greg, from his history at Accenture and at SuccessFactors, would have known immediately upon reading the study that it was erroneous, but he says nothing but positive things.

Something else sort of incredible is that Forrester declares in the disclosure that they retained “full editorial control.” This seems to imply that SAP had no control over the content produced. Let us say for the sake of argument that the study was performed and the result was that SuccessFactors had a negative ROI. Would that study have been published? If a study can only find a positive outcome, then it is not a study. It is a marketing document. SAP could have produced the same “study” themselves, but they knew no one would buy the conclusions. So they hired Forrester to rig a study for them.

There is no way out of this, a report like that can’t get released in error. There are a bunch of different people working on it. And they can’t say “it got messed up when sent to the printer.” If you need “quality assurance” what is going on over there. I perform that type of research. And while I have typos, you don’t end up with a crazy number like that, because you would see it and correct it.

But what is funny is that while they manufactured the numbers to get the high ROI, the consequence is the implementation cost was ridiculous. So now partners are like

“Hey we are charging way way more than that, and our clients are going to come back on us!”

That is what happens when you are dishonest and backward engineer numbers. SAP is probably not happy either. SAP wants a rigged analysis, but they want one that looks credible. Forrester did this with their HANA analysis years ago which we covered in the article How Accurate Was Forrester on HANA TCO?. But it went under the radar.

Accuracy Rating

The study receives a 1 out of 10 for accuracy. This is not research; it is a backward engineering study to support a conclusion that was agreed to by Forrester and SAP.

Our observation is that it is impossible to take money from vendors and produce a good outcome. Vendors sometimes reach out to Brightwork Research & Analysis do some type of report. I have yet to meet a vendor who cared about research. What the want is a report that shows them as “The Best!” That is why Brightwork Research & Analysis does not take income from vendors to produce content. It always ends up with the same thing, which is an inaccurate outcome and marketing masquerading as research.

Financial Disclosure

Financial Bias Disclosure

This article and no other article on the Brightwork website is paid for by a software vendor, including Oracle and SAP. Brightwork does offer competitive intelligence work to vendors as part of its business, but no published research or articles are written with any financial consideration. As part of Brightwork’s commitment to publishing independent, unbiased research, the company’s business model is driven by consulting services; no paid media placements are accepted.

SAP Contact Form

  • Want More Infomation on SAP?

    Our unbiased experts can deliver accurate research and honest advice on SAP.

    This article is free, we do not answer questions for free. Filling out this form is for those that have a budget. If that describes you, just fill out the form below and we'll be in touch asap.



How SAP Customers Turn into SAP Conference Monkeys

What This Article Covers

  • Case Studies
  • What is a Conference Monkey
  • How Individuals Lie to For Personal Gain


In many conferences, it is customary to think that consulting companies and software vendors will tell lies in order to increase their sales. But what is surprising to many is that many customers also will lie. I can recall when I worked for Deloitte, I was in many meetings with individuals from my client Lockheed Martin. The Senior Manager from Deloitte was trying to get the members from Lockheed Martin to present at SAPPHIRE. I presented at SAPPHIRE that year as well. When the member from Lockheed Martin presented, I observed him completely mislead the audience about how much progress had been made on the project. This is interesting because the Lockheed Martin representative was one of the more honest people I have worked with on a project. But the pressure he was under was evidently too great. Much of this was while being pressured by the Senior Manager to do so. The Senior Manager needed to sell his own project if he was going to be promoted to Partner, and to do that effectively, he needed to get Lockheed Martin to lie about the progress of the SAP project.

Since that time I have witnessed all manner of people at conferences clearly lying about how quickly they implemented SAP software, what they implemented, etc.

Case Studies

In the Brightwork Research & Analysis study on S/4HANA Implementation Study, there were numerous public case studies where the customer representative clearly lied about what occurred on the S/4HANA implementation. The following are examples:

  • Florida Crystals: The CIO stated that Florida Crystals upgraded to S/4HANA in 3 weeks.
  • New York Life: New York Life presented the idea that they implemented S/4HANA in order to use S/4HANA’s ledger. The representative for New York Life stated that they moved to S/4HANA to consolidated many different ledgers. However, one could have consolidated the ledgers to a single ledger. However, that could have been done with ECC.
  • Hillary’s Blinds: The representative for Hillary’s Blinds has told all manner of incorrect statements, not only related to S/4HANA but to quite a few other applications, including pretending that moving CRM to HANA made a great improvement in CRM.

Conference Monkeys

SAP has a number of customers where the members of the customers have the incentive or receive the incentive to lie about the status of S/4HANA. This is a problem that extends to all SAP products.

A conference monkey is a high ranking individual from a customer that is willing to lie about the functionality, the speed of the implementation, etc. A conference monkey may receive the benefit from SAP of being placed at a new position and a better position by SAP when a position opens up. A conference monkey lies on command, with the expectation of receiving something in return. They are critical in providing misinformation to the market, and therefore reducing the quality of information in the market. Just about all the conference monkeys are CIOs or out of the IT department. This gets into the topic of to whom does the IT department owe its allegiance which is covered in the following article.

  • Sales people lie all the time to get deals.
  • Consulting companies lie all the time to get consulting contracts.
  • Senior members of companies lie on everything from the budgets to the successfulness of their implementations. In this case, they are simply lying for career reasons. This means that much of the information at SAP conferences is unreliable.

Financial Disclosure

Financial Bias Disclosure

This article and no other article on the Brightwork website is paid for by a software vendor, including Oracle and SAP. Brightwork does offer competitive intelligence work to vendors as part of its business, but no published research or articles are written with any financial consideration. As part of Brightwork’s commitment to publishing independent, unbiased research, the company’s business model is driven by consulting services; no paid media placements are accepted.

SAP Contact Form

  • Want More Infomation on SAP?

    Our unbiased experts can deliver accurate research and honest advice on SAP.

    This article is free, we do not answer questions for free. Filling out this form is for those that have a budget. If that describes you, just fill out the form below and we'll be in touch asap.


Enterprise Software Risk Book

Software RiskRethinking Enterprise Software Risk: Controlling the Main Risk Factors on IT Projects

Rethinking Enterprise Software Risk: Controlling the Main Risk Factors on IT Projects

Better Managing Software Risk

The software implementation is risky business and success is not a certainty. But you can reduce risk with the strategies in this book. Undertaking software selection and implementation without approximating the project’s risk is a poor way to make decisions about either projects or software. But that’s the way many companies do business, even though 50 percent of IT implementations are deemed failures.

Finding What Works and What Doesn’t

In this book, you will review the strategies commonly used by most companies for mitigating software project risk–and learn why these plans don’t work–and then acquire practical and realistic strategies that will help you to maximize success on your software implementation.


Chapter 1: Introduction
Chapter 2: Enterprise Software Risk Management
Chapter 3: The Basics of Enterprise Software Risk Management
Chapter 4: Understanding the Enterprise Software Market
Chapter 5: Software Sell-ability versus Implementability
Chapter 6: Selecting the Right IT Consultant
Chapter 7: How to Use the Reports of Analysts Like Gartner
Chapter 8: How to Interpret Vendor-Provided Information to Reduce Project Risk
Chapter 9: Evaluating Implementation Preparedness
Chapter 10: Using TCO for Decision Making
Chapter 11: The Software Decisions’ Risk Component Model