The Four Hidden Issues with SAP’s HANA Analytics Benchmark

Executive Summary

  • SAP developed a new benchmark to make HANA look good.
  • We cover the problems with creating a benchmark for BW.

Introduction

After HANA was released in 2011, SAP created the BW-EML benchmark (since renamed the BWAML) and the BWH benchmark. Both of these benchmarks were for SAP’s BW application. SAP published no HANA benchmarks for any other SAP application since 2011.

In this article, we will cover the hidden issues with SAP’s HANA benchmarks.

The Setup of the BW Benchmark for HANA

SAP describes the BW-EML benchmark as follows.

“To ensure that the database can efficiently use both InfoCubes and DataStore Objects (DSOs) for reporting, the data model for the BW-EML benchmark consists of three InfoCubes and seven DSOs, each of which contain the data produced in one specific year. The three InfoCubes contain the same data (from the last three years) as the corresponding DSOs. Both object types include the same set of fields. The InfoCubes include a full set of 16 dimensions, which comprise 63 characteristics, with cardinali-ties of up to 1 million values and one complex hierarchy. To simulate typical customer data models, the InfoCube is made up of 30 key figures, includ-ing those that require exception aggregation. In the data model of the DSOs, the high-cardinality characteristics are defined as key members, while other characteristics are modeled as part of the data members

The first problem with this benchmark is what is unsaid. This is brought up by Oracle.

“SAP is now promoting HANA as the database of choice for their applications and clearly has a conflict of interest when it comes to certifying benchmark results that show better performance than HANA. Of the 28 SAP standard application benchmarks, SAP has chosen to only publish results for HANA on the BW-EML benchmark (emphasis added).”

Hidden Issue #1: How About the Missing Benchmarks?

SAP simply does not mention that there are missing benchmarks, and after all the exaggerations on HANA, SAP has chosen to publish just one benchmark.

Why?

The one benchmark they can get HANA to perform well. SAP clearly has a policy of hiding any benchmark for HANA that it can’t perform well, which is why you don’t have the entity performing the benchmark with a horse in the benchmark race.

Hidden Issue #2: SAP Crowning HANA, i.e. Contestant + Judge = Unbiased Outcomes?

Yes, this should go without saying, but you cannot be both a contestant and be a judge.

What would happen if say Miss Hawaii was also the only judge in a beauty pageant? Who, under those circumstances, would be most likely to win the pageant? Is there perhaps some reason we don’t allow competitors to also judge competitions? Clearly, this requires much research with the best minds working on it. 

Yet note that SAP has a different view.

“To help the market easily and quickly make these judgments, SAP offers standard application benchmarks. When used consistently, these benchmarks provide impartial, measurement-based ratings of standard SAP applications in different configurations with regard to operating system, database, or hardware, for example. Decision makers trust these benchmarks to provide unbiased information about product performance.”

A Problem With Translating the Word “Unbiased” into German?

Interesting. SAP might want to look up the term “unbiased” in the dictionary, as it is apparently not translating properly into German. Either that or SAP is saying something quite inaccurate in this quote. But I looked up unbiased in Google translator to German and came up with the word.

“Unvoreingenommen”

I then found these synonyms in the German-English dictionary.

“dispassionately {adv} [impartially] impartial {adj}
candid {adj}
dispassionate {adj}
unprejudiced {adj}
detached {adj} [impartial] impartially {adv}
nonpartisan {adj}
unbiassed {adj} [spv., especially Br.] unjaundiced {adj}
fair-minded {adj}
open-minded {adj}
without bias {adj}”

So translation does not seem to be the problem.

This is just the first of the hidden issues with this benchmark.

But let us get to the second hidden issue, which is the inconsistency between InfoCubes or cubes and a column-oriented database.

Hidden Issue #3: Why Are InfoCubes Still Being Used for A Database with Column Oriented Capabilities?

I have been working on SAP DP projects for over a decade. DP uses the same data administration area as does BW. Except DP runs forecasting and has a forecasting front end on top of the data backend. HANA is supposed to eliminate the need for cubes, as cubes are aggregation devices use for performance based upon a row-oriented DB.

But in the BW-EML benchmark cubes are still used, as we can see from the quote above.

Why?

Because companies don’t want to decompose the cubes they already built for the pre-column oriented design? Quite possibly yes, as companies will still be using the cubes they built for many years. Actually, much of BW is made obsolete by putting it on top of a column-oriented design capable DB.

Nowhere in any of the BW-EML benchmark does it point out that a primary benefit of a column-oriented design the obsolescence of cubes.

Hidden Issue #4: The Problem with Benchmarking an Incompetent Application

How important is such benchmarking on BW in the first place? I ask because I perform forecast testing for full production data sets for clients on a laptop.

I have a best of breed forecasting application that handles hierarchies far better than DP, I can do things on the laptop with my inexpensive application that no customer I have ever seen can do in DP. Neither DP nor other forecasting applications do the type of forecast error measurement we want, so we created the Brightwork Explorer which we cover in How to Access Monetary Forecast Error Calculation. We put this on AWS and can apply any number of resources to it, making benchmarking studies like the BW-EML of little relevance.

  • The Brightwork “Hardware”: I have a decently powered laptop and it is all that I need to run the forecasting application. In fact, we would have liked to have purchased a more powerful one, but we were under time pressure as we were performing testing and an unfortunate Windows 10 install screwed up our previous laptop for a while. Therefore we went with a reasonably well-powered laptop that was available for purchase at a Costco across the street from our client at the time.
  • Why A Laptop is Just Fine: While I certainly could, I don’t even worry about buying a desktop and I perform repetitive testing with this setup. This means that I perform much more processing than a typical client because they normally do not perform testing but run the forecast on a weekly basis. However, I am performing forecast simulation (that is repeatedly performing forecasting jobs, but without passing them to a receiving system). This means that the load is far higher than the production server receives at my clients.

All of this illustrates the other problem with benchmarking. If the application is incompetently written and highly inefficient with how they manage resources like DP or BW, database benchmarking becomes a bit of a lost cause, because BW and DP will consume so much of the hardware and database processing capacity while it flails about. With these bad applications, one of the primary answers is to simply apply giant resources to them.

We have not once heard this topic raised, because neither SAP nor Oracle nor IBM has any interest in critiquing the application. Why? Well, their job is to sell databases to support the SAP application, the quality of the SAP application’s code is irrelevant to what they want to bring across. Customers have already made the decision to buy an awful application, now the only question is what database and hardware do you want to power your awful application.

I am not aware of what tricks the developer of the application I have used performed to make everything run so quickly and smoothly to make such flexible hierarchies, all I was told was that they put special attention to how the star schema was created, which obviously SAP did not, and which has been confirmed by conversations by other developers familiar with BW and DP.

Oh….this application I use was developed by a single developer. That has probably changed by now as the company has grown over time from when I first used it, but the application I used was developed by just one developer. And he ran circles around SAP’s large team of developers.

The BW-EML benchmark has since bee renamed to the BWAML. There are 17 benchmarks here, and the only database that is benchmarked is HANA. 

The second BW benchmark is called the BWH. There are 52 of these published at SAP’s benchmark site. The same issue applies, the only database that is benchmarked is HANA. The other database vendors have been excluded from this benchmark. 

BW is the only application that SAP has benchmarked HANA for. Both the BWAML and the BWH are BW benchmarks. SAP has refused to benchmark ECC on HANA or S/4HANA on HANA, which we cover in the article The Hidden Issue with the SD HANA Benchmarks.

Conclusion

Benchmarking can’t be interpreted in a vacuum, but it normally is. The issues specific to the BW-BML benchmark that we pointed out in this article are the following:

  • BW and DP are extremely poorly designed data warehouses (DP’s backend is BW) that consumes large amounts of computing resources.
  • Many decision-makers may read this benchmark without considering the fact that BW and DP are both inefficient resource consumers. If a more efficient data application were used, the database and hardware would not have to be so overpowered.
  • In testing against far less expensive applications, BW and DP lose, even when given far more resources to work with. Again, my comparisons have been using a consumer grade but reasonably powerful laptop, and beating a server that my clients were told by SAP that they needed to buy. The Brightwork “hardware” for forecast testing fits in a bag.
  • SAP serves as both a contestant and a judge in its own benchmarks, where HANA is set up as the winner before the competition begins.
  • None of the database vendors competing have any interest in the performance of the application versus other applications. They are there to sell databases.
  • It is highly unlikely that we could get SAP to certify our benchmarking that shows how inefficient BW and DP are versus other similar applications. SAP customers we have had as clients cannot be told that BW and DP are bad applications, so we are required to tiptoe around the issue to not make them feel bad about their poor investments. The primary benchmark in any IT environment is how good the IT department can be made to look. All other benchmarks are secondary to this primary benchmark.

The Broader Issues with Application and Database Benchmarking

There is no independent benchmarking entity for applications or for databases that exist in the enterprise software space. (Some might point to the TCP, but they are a benchmark specification setting entity, not a benchmarking entity).

  • Each participant runs and publishes benchmarks only to increase sales of their items.
  • Every entity that runs a benchmark, ends up, in a rather peculiar way, winning that benchmark. (surprise surprise)
  • Independent benchmarks are also dissuaded. Oracle demanded that an independent benchmarker be fired for publishing a benchmark that showed Oracle performing poorly. (The case of DeWitt — see footnote)
  • The commercial database vendors have clauses in their licenses that prevent independent companies from publishing benchmarks.
  • Open source databases do not have these clauses.

Overall, there are multiple dimensions to the presentation of the BW-EML/BWAML benchmark by SAP that hide information from the reader, such as the fact that SAP clearly did not release the benchmarks in which HANA was unable to perform well. HANA was supposed to perform 100,000 times faster than any competing database (McDermott) as we covered in How Accurate Was SAP About HANA Being 100,000x Faster Than Any Other Database. It was supposed to reduce the workday to (roughly six seconds) (Lucas) How Accurate Was SAP About HANA Enabling People to Work 10 to 10,000 Times. Yet when it came to proving these claims, SAP has had to rig its benchmarks to keep HANA from being compared to any other database. SAP often uses the term “AnyDB.” But perhaps the right explanation of SAP’s behavior is that SAP fears any objective comparison to “AnyDB,” or should just say the comparison to any DB.

 

“Coming Up with Solutions……Not Just Problems”

After publishing an article like this, readers sometimes ask that we come up with solutions rather than simply analyzing issues that are unpublished elsewhere.

Here the lesson should be straightforward enough.

IT departments should not take the word of SAP or SAP’s consulting ecosystem on the performance or other characteristics of HANA or any other item without evidence. The lesson for any business users that read this article is that IT departments that purchased and implemented HANA never looked for any evidence that HANA was able to meet the claims made HANA. SAP conveniently skirted the issue and rigged their benchmarks to specifically prevent HANA from being compared to any other database. No IT media or IT analyst ever called them out for this deception, and no company that purchased HANA ever bothered to check, preferring to base their purchase on the claims of SAP and their compliant consulting ecosystem. If these companies had done their research, it is unlikely they would have gone forward with a purchase of HANA. We say this repeatedly to clients that we advise on SAP. Whatever the SAP sales rep says is only a starting point. Everything stated by SAP must be fact-checked. And there is no reason to assume that something SAP says is true.

SAP’s Inaccurate Messaging on HANA as Communicated in SAP Videos

Fact-Checking SAP’s HANA Information

This video is filled with extensive falsehoods. We will address them in the sequence they are stated in this video.

SAP Video Accuracy Measurement

SAP's Statement
Accuracy
Brightwork Fact Check
Link to Analysis Article
HANA is a Platform
0%
HANA is not a platform, it is a database.How to Deflect You Were Wrong About HANA
HANA runs more "in-memory" than other databases.
10%
HANA uses a lot of memory, but the entire database is not loaded into memory.How to Understand the In-Memory Myth
S/4HANA Simplifies the Data Model
0%
HANA does not simplify the data model from ECC. There are significant questions as to the benefit of the S/4HANA data model over ECC.Does HANA Have a Simplified Data Model?
Databases that are not HANA are legacy.
0%
There is zero basis for SAP to call all databases that are not HANA legacy.SAP Calling All Non-HANA DBs Legacy.
Aggregates should be removed and replaced with real time recalculation.
0%
Aggregates are very valuable, and all RDBMS have them (including HANA) and they should not be removed or minimized in importance.Is Hasso Plattner Correct on Database Aggregates?
Reducing the number of tables reduces database complexity.
0%
Reducing the number of tables does not necessarily decrease the complexity of a database. The fewer tables in HANA are more complicated than the larger number of tables pre-HANA.Why Pressure SAP to Port S/4HANA to AnyDB?
HANA is 100% columnar tables.
0%
HANA does not run entirely with columnar tables. HANA has many row-oriented tables, as much as 1/3 of the database.Why Pressure SAP to Port S/4HANA to AnyDB?
S/4HANA eliminates reconciliation.
0%
S/4HANA does not eliminate reconciliation or reduce the time to perform reconciliation to any significant degree.Does HANA Have a Simplified Data Model and Faster Reconciliation?
HANA outperforms all other databases.
0%
Our research shows that not only can competing databases do more than HANA, but they are also a better fit for ERP systems.How to Understand the Mismatch Between HANA and S/4HANA and ECC.

The Problem: A Lack of Fact-Checking of HANA

There are two fundamental problems around HANA. The first is the exaggeration of HANA, which means that companies that purchased HANA end up getting far less than they were promised. The second is that the SAP consulting companies simply repeat whatever SAP says. This means that on virtually all accounts there is no independent entity that can contradict statements by SAP.

The Necessity of Fact Checking

We ask a question that anyone working in enterprise software should ask.

Should decisions be made based on sales information from 100% financially biased parties like consulting firms, IT analysts, and vendors to companies that do not specialize in fact-checking?

If the answer is “No,” then perhaps there should be a change to the present approach to IT decision making.

In a market where inaccurate information is commonplace, our conclusion from our research is that software project problems and failures correlate to a lack of fact checking of the claims made by vendors and consulting firms. If you are worried that you don’t have the real story from your current sources, we offer the solution.

Inaccurate Messaging on HANA as Communicated in SAP Consulting Firm Videos

For those interested in the accuracy level of information communicated by consulting firms on HANA, see our analysis of the following video by IBM. SAP consulting firms are unreliable sources of information about SAP and primarily serve to simply repeat what SAP says, without any concern for accuracy. The lying in this video is brazen and shows that as a matter of normal course, the consulting firms are happy to provide false information around SAP.

SAP Video Accuracy Measurement

SAP's Statement
Accuracy
Brightwork Fact Check
Link to Analysis Article
HANA runs more "in-memory" than other databases.
10%
HANA uses a lot of memory, but the entire database is not loaded into memory.How to Understand the In-Memory Myth
HANA is orders of magnitude faster than other databases.
0%
Our research shows that not only can competing databases do more than HANA, but they are also a better fit for ERP systems.How to Understand the Mismatch Between HANA and S/4HANA and ECC.
HANA runs faster because it does not use disks like other databases.
0%
Other databases also use SSDs in addition to disk.Why Did SAP Pivot the Explanation of HANA In Memory?
HANA holds "business data" and "UX data" and "mobile data" and "machine learning data" and "IoT data."
0%
HANA is not a unifying database. HANA is only a database that supports a particular application, it is not for supporting data lakes.
SRM and CRM are part of S/4HANA.
0%
SRM and CRM are not part of S/4HANA. They are separate and separately sold applications. SAP C/4HANA is not yet ready for sale. How Accurate Was Bluefin Solutions on C-4HANA?
Netweaver is critical as a platform and is related to HANA.
0%
Netweaver is not relevant for this discussion. Secondly Netweaver is not an efficient environment from which to develop.
HANA works with Business Objects
10%
It is very rare to even hear about HANA and Business Objects. There are few Buisness Objects implementations that use HANA.SAP Business Objects Rating
Leonardo is an important application on SAP accounts.
0%
Leonardo is dead, therefore its discussion here is both misleading and irrelevant.Our 2019 Observation: SAP Leonardo is Dead
IBM Watson is an important application on SAP accounts.
0%
Watson is dead, therefore its discussion here is both misleading and irrelevant.How IBM is Distracting from the Watson Failure to Sell More AI and Machine Learning
Digital Boardroom is an important application on SAP accounts.
0%
SAP Digital Boardroom is another SAP item that has never been implemented many places.

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Search Our Other HANA Performance Content

References

https://blogs.saphana.com/2015/03/19/behind-sap-bw-eml-benchmark/

https://www.springer.com/cda/content/document/cda…/9783319202327-c2.pdf

https://www.itconductor.com/blog/will-hana-dominate-in-sap-performance-over-oracle

https://www.glamour.com/story/miss-usa-is-still-a-beauty-pageant-but-not-the-one-it-used-to-be (image for beauty pageant)

https://dam.sap.com/mac/preview/a/67/mnPymWPAmmE7yyyXPglwXXl8OnyEAMlAXggXJlJlUDxlyPUv/41356_GB_40939_enUS.htm

https://www.linkedin.com/pulse/does-truth-matter-in-memory-benchmarks-sap-oracle-kuen-sang-lam/

https://blogs.oracle.com/oraclemagazine/the-undisputed-database-champ

http://www.tpc.org/tpcc/default.asp

https://www.brentozar.com/archive/2018/05/the-dewitt-clause-why-you-rarely-see-database-benchmarks/

https://www.sap.com/about/benchmark.html

AWS and Google Cloud Book

How to Leverage AWS and Google Cloud for SAP and Oracle Environments

Interested in how to use AWS and Google Cloud for on-premises environments, and why this is one of the primary ways to obtain more value from SAP and Oracle? See the link for an explanation of the book. This is a book that provides an overview that no one interested in the cloud for SAP and Oracle should go without reading.