The Problems with the Strange Lenovo HANA Benchmark

Executive Summary

  • Lenovo published a benchmark of ECC on AnyDB versus S/4HANA on HANA.
  • We cover the problems with this benchmark.

Introduction

On April 10, 2018 Lenovo published the technical paper, Lenovo SAP S/4HANA Scale out – Cycle 1.

This paper included a number of inaccuracies, which should not be surprising as Lenovo is an SAP partner and has a series of products they are trying to sell that are connected to HANA.

Notice the hardware specification below from the Lenovo paper.

The HANA box has a higher hardware specification, but it is hidden with the AnyDB/ECC server configuration not called out. 

Natural questions that arise.

  • Where is the hardware configuration listing on the AnyDB/ECC server?
  • How is the reader to know if the hardware is comparable? As we have pointed out in the article How Much of Performance is HANA?, many improvements that have been seen from HANA are due to the entity testing HANA against older hardware and previous versions of AnyDB.

Either Lenovo does not know how to list the specifications of the box, or more likely, Lenovo is excluding the AnyDB/ECC listings to obscure the fact that the hardware is not remotely comparable. How can this go unnoticed by those that read these studies?

Software Compared

Lenovo seems very interested in hiding the database that is being compared to ECC. Why the hesitation in identifying the database? It was not “AnyDB” it was a specific database. This gets again to the honesty of the study. Lenovo is intent on publishing benchmarking information in a way that offends the fewest possible parties. This is not an honest way to publish a benchmark, and Lenovo’s covering up of various aspects continues further on into the benchmark.

Compression of HANA

This contradicts all of the data points that we have on HANA, which shows a compression average of 30 to 40%. Of course, this value depends upon how much data is archived, which SAP attributes to compression. SAP has claimed compression of 97.5% which we covered in Is Hasso Plattner and SAP Correct About Database Aggregates?

This is suspicious because we have never encountered any compression even close to this.

Data Model Simplification

SAP has made many claims around the database or data models being simplified under S/4HANA. And here Lenovo holds to this line. However, while the data model is simplified in some ways, it is more complex in others, as we covered in the article How Accurate Was SAP About S/4HANA and a Simplified Data Model, this is a false claim. Furthermore, there is a great deal of work involved in switching to the new data model for existing ECC customers as we covered in the article Why It Is Important to Pull Forward S/4HANA Code Remediation. This is because in part all of the adapters and customizations have to be adjusted. 

This is another claim that calls into question the information that Lenovo is publishing. Second, you can’t measure data model complexity by simply counting objects.

Transaction Processing Performance

The lack of hardware listing for ECC/AnyDB has already ruined the study, but let us go forward to see what the study says.

First, let us look at the transaction processing.

Why are the exact comparisons redacted? Secondly, how is anyone to know how much of the improved transactions are from the HANA boxes higher hardware specification? And SAP promised massive improvements in performance across the board, so why are any of the transactions negative? 

The redaction is quite odd, as there is no reasonable explanation for why this should be.

Probably their best resources were put into the benchmark test…. and being a HW vendor, Lenovo have the best HW resources that money can buy and their best engineers working with the best SAP resources and the results is 21% of the existing transactions having slower performance.

Analytics on S/4HANA?

Now one could construct a scenario where a bunch of analytics is performed in ECC/S/4HANA. SAP has stated this as their vision. The concept is that all analytics would be performed inside the ERP system.

This is compared against an older version of most likely either Oracle or DB2, which did not have multimodel capability (that is column-oriented with row-oriented). However, why are any of the analytic scores slower than the older database versions?

  • Observations from the field show HANA underperforming all of the competitor databases, even SQL Server even in analytic workloads. This level of improvement, up to 2846% shows the hardware difference between the ECC box and the HANA box.
  • Once again we have redacted actual scores. What is being hidden here?

Conclusion

The Lenovo study is redacted and rigged to make HANA look good. It is extremely odd to find the hardware spec for the comparison system entirely lacking. Lenovo is apparently hoping that no one reads the study in much detail.

This means that the study cannot be used to say much of anything. The fact that Lenovo had to redact information even after not publishing the ECC/AnyDB hardware specification is another cause for concern.

Lenovo cannot publish a study on HANA because it has a financial bias to promote HANA.

How likely is it that Lenovo will publish any information that is not complimentary towards HANA that would negatively impact their hardware offering for HANA? 

SAP’s Inaccurate Messaging on HANA as Communicated in SAP Videos

Fact-Checking SAP’s HANA Information

This video is filled with extensive falsehoods. We will address them in the sequence they are stated in this video.

SAP Video Accuracy Measurement

SAP's Statement
Accuracy
Brightwork Fact Check
Link to Analysis Article
HANA is a Platform
0%
HANA is not a platform, it is a database.How to Deflect You Were Wrong About HANA
HANA runs more "in-memory" than other databases.
10%
HANA uses a lot of memory, but the entire database is not loaded into memory.How to Understand the In-Memory Myth
S/4HANA Simplifies the Data Model
0%
HANA does not simplify the data model from ECC. There are significant questions as to the benefit of the S/4HANA data model over ECC.Does HANA Have a Simplified Data Model?
Databases that are not HANA are legacy.
0%
There is zero basis for SAP to call all databases that are not HANA legacy.SAP Calling All Non-HANA DBs Legacy.
Aggregates should be removed and replaced with real time recalculation.
0%
Aggregates are very valuable, and all RDBMS have them (including HANA) and they should not be removed or minimized in importance.Is Hasso Plattner Correct on Database Aggregates?
Reducing the number of tables reduces database complexity.
0%
Reducing the number of tables does not necessarily decrease the complexity of a database. The fewer tables in HANA are more complicated than the larger number of tables pre-HANA.Why Pressure SAP to Port S/4HANA to AnyDB?
HANA is 100% columnar tables.
0%
HANA does not run entirely with columnar tables. HANA has many row-oriented tables, as much as 1/3 of the database.Why Pressure SAP to Port S/4HANA to AnyDB?
S/4HANA eliminates reconciliation.
0%
S/4HANA does not eliminate reconciliation or reduce the time to perform reconciliation to any significant degree.Does HANA Have a Simplified Data Model and Faster Reconciliation?
HANA outperforms all other databases.
0%
Our research shows that not only can competing databases do more than HANA, but they are also a better fit for ERP systems.How to Understand the Mismatch Between HANA and S/4HANA and ECC.

The Problem: A Lack of Fact-Checking of HANA

There are two fundamental problems around HANA. The first is the exaggeration of HANA, which means that companies that purchased HANA end up getting far less than they were promised. The second is that the SAP consulting companies simply repeat whatever SAP says. This means that on virtually all accounts there is no independent entity that can contradict statements by SAP.

Being Part of the Solution: What to Do About HANA

We can provide feedback from multiple HANA accounts that provide realistic information around HANA — and this reduces the dependence on biased entities like SAP and all of the large SAP consulting firms that parrot what SAP says. We offer fact-checking services that are entirely research-based and that can stop inaccurate information dead in its tracks. SAP and the consulting firms rely on providing information without any fact-checking entity to contradict the information they provide. This is how companies end up paying for a database which is exorbitantly priced, exorbitantly expensive to implement and exorbitantly expensive to maintain. When SAP or their consulting firm are asked to explain these discrepancies, we have found that they further lie to the customer/client and often turn the issue around on the account, as we covered in the article How SAP Will Gaslight You When Their Software Does Not Work as Promised.

If you need independent advice and fact-checking that is outside of the SAP and SAP consulting system, reach out to us with the form below or with the messenger to the bottom right of the page.

The major problem with companies that bought HANA is that they made the investment without seeking any entity independent of SAP. SAP does not pay Gartner and Forrester the amount of money that they do so these entities can be independent as we covered in the article How Accurate Was The Forrester HANA TCO Study?

If you need independent advice and fact-checking that is outside of the SAP and SAP consulting system, reach out to us with the form below or with the messenger to the bottom right of the page.

Inaccurate Messaging on HANA as Communicated in SAP Consulting Firm Videos

For those interested in the accuracy level of information communicated by consulting firms on HANA, see our analysis of the following video by IBM. SAP consulting firms are unreliable sources of information about SAP and primarily serve to simply repeat what SAP says, without any concern for accuracy. The lying in this video is brazen and shows that as a matter of normal course, the consulting firms are happy to provide false information around SAP.

SAP Video Accuracy Measurement

SAP's Statement
Accuracy
Brightwork Fact Check
Link to Analysis Article
HANA runs more "in-memory" than other databases.
10%
HANA uses a lot of memory, but the entire database is not loaded into memory.How to Understand the In-Memory Myth
HANA is orders of magnitude faster than other databases.
0%
Our research shows that not only can competing databases do more than HANA, but they are also a better fit for ERP systems.How to Understand the Mismatch Between HANA and S/4HANA and ECC.
HANA runs faster because it does not use disks like other databases.
0%
Other databases also use SSDs in addition to disk.Why Did SAP Pivot the Explanation of HANA In Memory?
HANA holds "business data" and "UX data" and "mobile data" and "machine learning data" and "IoT data."
0%
HANA is not a unifying database. HANA is only a database that supports a particular application, it is not for supporting data lakes.
SRM and CRM are part of S/4HANA.
0%
SRM and CRM are not part of S/4HANA. They are separate and separately sold applications. SAP C/4HANA is not yet ready for sale. How Accurate Was Bluefin Solutions on C-4HANA?
Netweaver is critical as a platform and is related to HANA.
0%
Netweaver is not relevant for this discussion. Secondly Netweaver is not an efficient environment from which to develop.
HANA works with Business Objects
10%
It is very rare to even hear about HANA and Business Objects. There are few Buisness Objects implementations that use HANA.SAP Business Objects Rating
Leonardo is an important application on SAP accounts.
0%
Leonardo is dead, therefore its discussion here is both misleading and irrelevant.Our 2019 Observation: SAP Leonardo is Dead
IBM Watson is an important application on SAP accounts.
0%
Watson is dead, therefore its discussion here is both misleading and irrelevant.How IBM is Distracting from the Watson Failure to Sell More AI and Machine Learning
Digital Boardroom is an important application on SAP accounts.
0%
SAP Digital Boardroom is another SAP item that has never been implemented many places.

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Search Our Other HANA Performance Content

References

https://en.resources.lenovo.com/whitepapers/lenovo-sap-s-4hana-scale-out

TCO Book

 

TCO3

Enterprise Software TCO: Calculating and Using Total Cost of Ownership for Decision Making

Getting to the Detail of TCO

One aspect of making a software purchasing decision is to compare the Total Cost of Ownership, or TCO, of the applications under consideration: what will the software cost you over its lifespan? But most companies don’t understand what dollar amounts to include in the TCO analysis or where to source these figures, or, if using TCO studies produced by consulting and IT analyst firms, how the TCO amounts were calculated and how to compare TCO across applications.

The Mechanics of TCO

Not only will this book help you appreciate the mechanics of TCO, but you will also gain insight as to the importance of TCO and understand how to strip away the biases and outside influences to make a real TCO comparison between applications.
By reading this book you will:
  • Understand why you need to look at TCO and not just ROI when making your purchasing decision.
  • Discover how an application, which at first glance may seem inexpensive when compared to its competition, could end up being more costly in the long run.
  • Gain an in-depth understanding of the cost, categories to include in an accurate and complete TCO analysis.
  • Learn why ERP systems are not a significant investment, based on their TCO.
  • Find out how to recognize and avoid superficial, incomplete or incorrect TCO analyses that could negatively impact your software purchase decision.
  • Appreciate the importance and cost-effectiveness of a TCO audit.
  • Learn how SCM Focus can provide you with unbiased and well-researched TCO analyses to assist you in your software selection.
Chapters
  • Chapter 1:  Introduction
  • Chapter 2:  The Basics of TCO
  • Chapter 3:  The State of Enterprise TCO
  • Chapter 4:  ERP: The Multi-Billion Dollar TCO Analysis Failure
  • Chapter 5:  The TCO Method Used by Software Decisions
  • Chapter 6:  Using TCO for Better Decision Making