The Problems with the Strange Lenovo HANA Benchmark

Executive Summary

  • Lenovo published a benchmark of ECC on AnyDB versus S/4HANA on HANA.
  • We cover the problems with this benchmark.

Introduction

On April 10, 2018, Lenovo published the technical paper, Lenovo SAP S/4HANA Scale out – Cycle 1.

This paper included several inaccuracies, which should not be surprising as Lenovo is an SAP partner and has a series of products they are trying to sell that are connected to HANA.

Notice the hardware specification below from the Lenovo paper.

The HANA box has a higher hardware specification, but it is hidden with the AnyDB/ECC server configuration not called out. 

Natural questions that arise.

  • Where is the hardware configuration listing on the AnyDB/ECC server?
  • How is the reader to know if the hardware is comparable? As we have pointed out in the article, How Much of Performance is HANA?, many improvements that have been seen from HANA are due to the entity testing HANA against older hardware and previous versions of AnyDB.

Either Lenovo does not know how to list the specifications of the box, or more likely, Lenovo is excluding the AnyDB/ECC listings to obscure the fact that the hardware is not remotely comparable. How can this go unnoticed by those that read these studies?

Software Compared

Lenovo seems very interested in hiding the database that is being compared to ECC. Why the hesitation in identifying the database? It was not “AnyDB” it was a specific database. This gets again to the honesty of the study. Lenovo is intent on publishing benchmarking information in a way that offends the fewest possible parties. This is not an honest way to publish a benchmark, and Lenovo’s covering up of various aspects continues further on into the benchmark.

Compression of HANA

This contradicts all of the data points that we have on HANA, which shows a compression average of 30 to 40%. Of course, this value depends upon how much data is archived, which SAP attributes to compression. SAP has claimed compression of 97.5%, which we covered in Is Hasso Plattner and SAP Correct About Database Aggregates?

This is suspicious because we have never encountered any compression even close to this.

Data Model Simplification

SAP has made many claims around the database or data models being simplified under S/4HANA. And here Lenovo holds to this line. However, while the data model is simplified in some ways, it is more complicated in others, as we covered in the article How Accurate Was SAP About S/4HANA and a Simplified Data Model, this is a false claim. Furthermore, there is a great deal of work involved in switching to the new data model for existing ECC customers, as we covered in the article Why It Is Important to Pull Forward S/4HANA Code Remediation. This is because, in part, all of the adapters and customizations have to be adjusted. 

This is another claim that calls into question the information that Lenovo is publishing. Second, you can’t measure data model complexity by only counting objects.

Transaction Processing Performance

The lack of hardware listing for ECC/AnyDB has already ruined the study, but let us go forward to see what the study says.

First, let us look at the transaction processing.

Why are the exact comparisons redacted? Secondly, how is anyone to know how much of the improved transactions are from the HANA boxes higher hardware specification? And SAP promised massive improvements in performance across the board, so why are any of the transactions negative? 

The redaction is quite odd, as there is no reasonable explanation for why this should be.

Probably their best resources were put into the benchmark test…. and being an HW vendor, Lenovo has the best HW resources that money can buy. And their best engineers working with the best SAP resources, and the result is 21% of the existing transactions having slower performance.

Analytics on S/4HANA?

Now one could construct a scenario where a bunch of analytics is performed in ECC/S/4HANA. SAP has stated this as their vision. The concept is that all analytics would be performed inside the ERP system.

This is compared against an older version of most likely either Oracle or DB2, which did not have the multimodel capability (that is column-oriented with row-oriented). However, why are any of the analytic scores slower than the older database versions?

  • Observations from the field show HANA underperforming all of the competitor databases, even SQL Server, even in analytic workloads. This level of improvement, up to 2846%, shows the hardware difference between the ECC box and the HANA box.
  • Once again, we have redacted actual scores. What is being hidden here?

Advice on Enjoying the Quiz

To see the full screen just select the lower right-hand corner and expand. Trust us, expanding makes the experience a whole lot more fun.

 

Conclusion

The Lenovo study is redacted and rigged to make HANA look good. It is incredibly odd to find the hardware spec for the comparison system entirely lacking. Lenovo is hoping that no one reads the study in much detail.

This means that the study cannot be used to say much of anything. The fact that Lenovo had to redact information even after not publishing the ECC/AnyDB hardware specification is another cause for concern.

Lenovo cannot publish a study on HANA because it has a financial bias to promote HANA.

How likely is it that Lenovo will publish any information that is not complimentary towards HANA that would negatively impact their hardware offering for HANA? 

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Getting the Most Accurate Information on SAP

Accessing Custom SAP Research

We provide custom research into SAP, as well as fact checking of the claims of multiple options. To find out more, contact us.

References

https://en.resources.lenovo.com/whitepapers/lenovo-sap-s-4hana-scale-out