- Forrester was paid by SAP to produce a report that showed SuccessFactors had a high ROI.
- Numerous errors in the report demonstrate that the study was rigged to provide SAP’s intended outcome.
Forrester published a study named The Total Economic Impact™ Of SAP SuccessFactors HCM Suite Cost Savings And Business Benefits Enabled By SAP SuccessFactors HCM Solutions, September 2018. In this article, we will review the study for accuracy, eventually assigning an accuracy score
Forrester’s Quantified Financial Benefits
Forrester declares the following financial benefits from a sample of customers provided to Forrester by SAP.
“The composite Organization experienced the risk-adjusted present value (PV) quantified benefits of the following services totaling $11,005,505 over a three-year period (see the Financial Analysis section for more details):
Employee Central and Payroll: $1,189,136.
Recruiting and Onboarding: $2,921,945.
Performance, Goals, and Compensation: $6,194,531.
Learning, Succession, and Development: $702,893.”
Forrester declares the following costs from a sample of customers provided to Forrester by SAP.
“The Organization experienced the following costs totaling $5,906,920 with a risk-adjusted present value of $5,186,681 (see the Financial Analysis section for more details):
Internal labor to plan and deploy SAP SuccessFactors: $60,000.
Incremental hardware, database and operating system license, and maintenance: $0.* The Organization incurred none of these costs with the SAP SuccessFactors cloud solution.
SAP SuccessFactors (or partner) fees for professional services implementation assistance: $265,000.
SAP SuccessFactors subscription fees: $3,751,600.
SAP Preferred Success fees: $750,320.”
How Many Customers Were Used for the Sample?
Forrester stated the following with respect to the sample that it used in the study:
“For this study, Forrester conducted interviews with six SAP SuccessFactors HCM customers. Interviewed customers are described as follows (each requesting anonymity)”
However, when one reviewed the list, there were not six entities listed, there were seven. A company each in the following industries:
- Local Government
- Outsourcing Provider.
There is no other interpretation than Forrester miscounted the companies that were in the study. While that seems like a major error, it is in fact only a 14% error, but it just seems worse than it is because it is so obvious. As we will see, that is a high point in terms of accuracy for the study.
Sample Selection Bias
The second thing we noticed was the very obvious selection bias of the sample used by Forrester.
One of the foundational quality checks in research is determining if the sample is reflective of the population. Anyone who has taken statistics would know that this subject alone makes up a large component of the course. Before any descriptive statistics are used on a sample, the first question must be answered as to representativeness. It is like a chain, the first links in the chain have to be unbroken for the pulling on the rest of the chain to work.
Getting 30 Samples?
The rule of thumb (called the Central Limit Theorem) is that one requires 30 samples or that are randomly selected from a population to be relatively confident that the sample is reflective. This is a rule of thumb memorable to most people that have studied statistics, and the following limitations apply.
- This is the minimum sample size, not the optimum sample size. As explained by Christopher Rout “It’s not that “30 in a sample group should be enough” for a study. It’s that you need at least 30 before you can reasonably expect an analysis based upon the normal distribution (i.e. z test) to be valid. That is it represents a threshold above which the sample size is no longer considered “small“
- Many researchers consider 30 the minimum number of samples to conduct a test. When we performed our research into S/4HANA Implementations we had more than 30 samples. However, in many cases, this many samples are not available. But in this case as the SuccessFactors team had access to so many customers, one has to wonder why such a small number of samples was used. SuccessFactors is not like some SAP products where it is difficult to find pleased customers. But the budget available to perform the study cannot be underemphasized. Neither Forrester or SAP have any research orientation. SAP is trying to get marketing collateral at the lowest possible cost, and Forrester wants to make some money. This would have played a role in why so few customers were selected.
- The confidence interval is a statistical measurement for how likely it is that a sample is representative of the population from which it is drawn. The confidence interval is a measure of probability.
However, we are getting ahead of ourselves, because statistics can only be applied if the sample is random. If the sample is cherry-picked, then no statistics of any type can be applied. When cherry picked, the probability is 100% that the sample is not representative of the population.
This is sort of the amusing thing about statistics. One can take many courses in statistics, but for most of life and analysis, just the basic statistics are necessary to understand because in most cases people are not even applying basic statistical standards when evaluating information.
Now let us address why this sample is so biased.
What Customers Was SAP Likely to Provide to Forrester?
Right off the bat, as each of these companies was provided to Forrester from SAP to interview, a question of selection bias immediately comes to mind. The first question is “why these six companies?” versus the what must be over a thousand customers that SuccessFactors has.
- Most likely SAP would have provided the six most successful implementations that were convenient for SAP to find and with which they had an excellent relationship.
This is not a random sample; this is an exceptional sample of SAP Success Factors customers. This means that companies that attempted a Success Factors implementation and failed were excluded. Average Success Factor implementations were excluded.
Yet this is nowhere in the disclosure on the first page. Under disclosures, it does say the following:
“The study was commissioned by SAP SuccessFactors and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.”
There are no other applications included in the study, so it is difficult to see what this paragraph means.
Further, it states.
“Forrester makes no assumptions as to the potential ROI that other organizations will receive.”
Well if the ROI estimate in the study is not usable by those that read the study, what was the point of publishing the study? Moreover, the disclosure is right, because the sample has been intensively cherry picked, the ROI is not usable — and that is if the math were right, which will be explained later in the article, it isn’t. And finally, when SAP publicized the study, they contradict Forrester’s disclosure. They clearly gave readers the impression that the study is reflective of a..
“a typical organization”
Now let us review how Forrester calculated (or represented the calculations from customers) the financial benefits of the SuccessFactors implementation.
The Financial Benefits Estimates
The estimates of financial benefits are broken into the following categories.
- Employee Central And Payroll Benefits: Calculation Table is based upon an internal staff cost of $90,000 fully loaded.
- Recruiting And Onboarding Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour.
- Recruiting And Onboarding Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour.
- Performance, Goals, And Compensation Benefits: Calculation Table is based upon an hourly cost of internal recruiting staff of $40 per hour and the cost of managers at $55 per hour.
- Learning, Succession, And Development Benefits: Calculation Table is based upon a fully loaded cost of internal labor of $90,000 per year.
The savings in the study came from the reduction in labor costs.
These aren’t estimates; they are what the companies said happened. If this is true, then each of these companies would have had to have eliminated HR positions. One commentary might be that the positions were not eliminated, but they were merely “reallocated to other parts of the company.” Companies sometimes write things like this as they don’t like it having it pointed out that cost savings often result in resources being fired. But what if all of the HR employees can’t be fired? Well, that would impact the ROI.
The Implementation Costs
The implementation average for the seven sample companies is stated as $265,000 for consulting and $60,000 for internal resources over a three year period.
The problem here is there is no way this is true.
Jarret Pazahanick, a SAP and SuccessFactors consultant who has done 50 SAP and SuccessFactors implementations estimated that both the internal cost and the consulting cost would have been “5-10-15X.” This is modified to be 5x for the internal resources, and between 12x to 15x for the consulting resources. Jarret’s estimate is far higher in credibility than any estimate from SAP because Jarret is not being paid by SAP and is not trying to sell SuccessFactors, but works as an independent consultant. I face the same issue. As a long time SAP consultant, my estimates are always longer and more expensive than those produced by people with a vested interest in selling software. When I made these estimates, salespeople who had never worked on an SAP project would tell me they were confident my estimates were too long, and that they would interfere with the sale.
Jarret went on to state the following in a LinkedIn comment.
“It is SO beyond the realm of what a real implementation of SuccessFactors really take it is mind boggling. The backchannels are already start to buzz of upset partners and customers wondering why their implementations and quotes are so much more that what SuccessFactors provided Forrester as what the cost would be to get that great ROI (Obviously all made up).”
If the middle figure is taken from Jarret’s estimate, we end up 5x for the internal resources, and let us take a midpoint of between 12x and 15x for the consulting resources. This is 13.5x. 13x of 265,000 is $3,577,500 for consulting. 5x of 60,000 is 300,000.
If we redo the ROI using these estimates, we end up with the following.
The Overall ROI
Now we can contrast this with Forrester’s analysis of the ROI.
“Based on Forrester’s experience, the six-month payback period and 112% ROI numbers are strong when compared to other enterprise cloud implementations. In addition, the interviewed customers that migrated from SAP HCM on-premises software to the SAP SuccessFactors cloud solution had better benefit results.”
So we calculate 31.34% ROI. Even that would be a very high ROI for any IT implementation. However, the sample is cherry-picked so that ROI is meaningless. Furthermore, there are several problematic features of the benefits estimation.
- There is no independent verification being performed by Forrester. Forrester asked the customers themselves to perform the estimates.
- Customers have a bias to estimate their benefits higher and their costs lower as those that are queried were often part of the software selection. Therefore the responses they provide are a way of congratulating themselves on their decision-making skills.
- Customers in my experience don’t typically know these things. This is because they are more focused on keeping fewer employees capable of performing any type of research or estimation. Now, there is nothing to stop companies from hiring these skills. However, they usually don’t as they don’t see them as pertinent for their objectives.
Therefore, even among the cherry-picked sample, the estimates are most likely extremely unreliable. As we just analyzed, the estimates of the internal and external costs are off by order of magnitude of somewhere around 10x, and the costs are the easiest areas to estimate! This is because you have receipts for what you paid for something. The problem is that Forrester received the consulting costs from SuccessFactors, and would not themselves know the cost of a SuccessFactors implementation.
Benefits are far more tricky to estimate. This is why at Brightwork Research & Analysis, we have online TCO calculators. Unlike Forrester, no one paid for these calculators to be produced, so they show a far higher TCO than any of the vendors would like to see published.
However, we did not attempt to estimate the ROI of the 53 different applications we estimated. By way of example, review our analysis of the problems faced with calculating lost sales and forecast error costs to gain an appreciation for the difficulty of estimating benefits from software implementations.
Secondly, how did all seven companies end up measuring the benefits in the exact same way (that is with labor savings)? The answer is that there is no way they did. This means that the companies reported back on a survey to Forrester. That is Forrester setup the construct of the cost savings, and the cherry-picked customers responded to the survey. This increases the “fishiness” of the study because it means that even less thought was put into the estimates. For example, did every one of the customers agree with the benefit calculation approach? That would be odd if that were the case.
The study is interesting in that it illuminates Forrester’s process. However, the output of the study is not useful for anything for those seeking to evaluate if SuccessFactors is a good investment. SAP paid Forrester to produce a study showing that SF has a very high ROI. And Forrester produced the study that said what SAP wanted it to say. Observe how SAP marketing explained the study.
“SAP SE (NYSE: SAP) today announced the results of Forrester Consulting’s “Total Economic Impact™ of SAP SuccessFactors HCM Suite” study,* which found that a typical organization** that invests in the SAP SuccessFactors HCM Suite could realize potentially $11 million in total quantified benefits over a three-year period.”
Is that what the study found? No.
Also, are seven companies selected by SAP reflective of a typical SF implementation? That is as close as one can come to saying that the study was reflective of what a typical company should expect, which is the opposite of what the disclaimer on the Forrester study states. SAP has a history of misrepresenting the findings of a study they pay to have performed. SAP has no consideration for the reputation of the entities they buy research from. Their perspective is the same as the one they apply to customers, which is to extract the most from every possible transaction.
Also, notice the endorsement from the President of SuccessFactors.
“In our view, Forrester’s holistic study shows that SAP SuccessFactors solutions can drive not only better employee experiences but also significant cost savings,” said SAP SuccessFactors President Greg Tomb. “We are proud to be helping businesses better understand and serve their employees while simultaneously streamlining business processes and reaping significant cost benefits.”
Greg, from his history at Accenture and at SuccessFactors, would have known immediately upon reading the study that it was erroneous, but he says nothing but positive things.
Something else sort of incredible is that Forrester declares in the disclosure that they retained “full editorial control.” This seems to imply that SAP had no control over the content produced. Let us say for the sake of argument that the study was performed and the result was that SuccessFactors had a negative ROI. Would that study have been published? If a study can only find a positive outcome, then it is not a study. It is a marketing document. SAP could have produced the same “study” themselves, but they knew no one would buy the conclusions. So they hired Forrester to rig a study for them.
There is no way out of this, a report like that can’t get released in error. There are a bunch of different people working on it. And they can’t say “it got messed up when sent to the printer.” If you need “quality assurance” what is going on over there. I perform that type of research. And while I have typos, you don’t end up with a crazy number like that, because you would see it and correct it.
But what is funny is that while they manufactured the numbers to get the high ROI, the consequence is the implementation cost was ridiculous. So now partners are like
“Hey we are charging way way more than that, and our clients are going to come back on us!”
That is what happens when you are dishonest and backward engineer numbers. SAP is probably not happy either. SAP wants a rigged analysis, but they want one that looks credible. Forrester did this with their HANA analysis years ago which we covered in the article How Accurate Was Forrester on HANA TCO?. But it went under the radar.
The study receives a 1 out of 10 for accuracy. This is not research; it is a backward engineering study to support a conclusion that was agreed to by Forrester and SAP.
Our observation is that it is impossible to take money from vendors and produce a good outcome. Vendors sometimes reach out to Brightwork Research & Analysis do some type of report. I have yet to meet a vendor who cared about research. What the want is a report that shows them as “The Best!” That is why Brightwork Research & Analysis does not take income from vendors to produce content. It always ends up with the same thing, which is an inaccurate outcome and marketing masquerading as research.
Financial Bias Disclosure
This article and no other article on the Brightwork website is paid for by a software vendor, including Oracle and SAP. Brightwork does offer competitive intelligence work to vendors as part of its business, but no published research or articles are written with any financial consideration. As part of Brightwork’s commitment to publishing independent, unbiased research, the company’s business model is driven by consulting services; no paid media placements are accepted.