The Hidden Issue with the SD HANA Benchmarks

Executive Summary

  • SAP has used the SD benchmarks for years for other databases.
  • Four years into S/4HANA, SAP has no benchmarks for S/HANA SD or for ECC SD on HANA.

Introduction

SAP has used the SD benchmarks for years to test for transaction processing performance, but something peculiar happened when SAP released HANA and the benchmarking was to be performed for S/4HANA.

In this article, we will cover the hidden issues with SD benchmarks.

The SD Benchmark for HANA

Here are the SD benchmarks listed on the SAP website for HANA.

This looks like an impressive list of benchmarks for different hardware and different operating systems and database releases. At the time we reviewed this data, we counted 1066 benchmarks.

The Databases

Now notice the databases that were used for the benchmarks.

Each benchmark typically is a single database version or variants of the version, and there are many database versions or variants. For instance, there is a separate benchmark for Oracle 10g, Oracle 10g with Real Application Cluster.

The databases are as follows.

  1. DB2: (41 Benchmarks)
  2. Adabas: (1 Benchmark)
  3. SQL Server: (17 Benchmarks)
  4. SAP ASE (4 Benchmarks)
  5. Informix: (7 Benchmarks)
  6. MaxDB: (4 Benchmarks)
  7. SAP DB: (4 Benchmarks)
  8. Oracle (26 Benchmarks)

The Operating Systems

  1. IBM AIX
  2. Red Hat Enterprise
  3. IBM OS
  4. Solaris
  5. SUSE Linux
  6. Windows 2000, .NET, NT, Enterprise Server, Enterprise Edition, etc..

The most benchmarks were run for Windows Enterprise Server 2003 with 135.

ECC/R/3 Versions

The benchmarks span 52 different versions of ECC. the most frequently benchmarked ECC/R/3 versions were the following.

  1. ECC EHP 5 for ERP 6.0: (242 Benchmarks)
  2. R/3 4.6C: (124 Benchmarks)
  3. SAP EHP 4 for ERP 6.0: (118 Benchmarks)

Hardware Environment

  1. Bare Metal: (984 Benchmarks)
  2. Cloud: (44 Benchmarks)
  3. Virtualized: (36 Benchmarks)

Obviously, this means that 92% and therefore the bulk of the benchmarks were run on bare metal and on premises.

Hardware Vendors

The benchmarks used hardware from 28 different hardware vendors and AWS (not a hardware vendor but a cloud services provider that uses its own open specified hardware as we covered in How the Open Compute Project Reduced the Power of Proprietary Vendors).

The largest number of benchmarks per hardware vendor was as follows:

  1. Dell: (86 Benchmarks)
  2. Fujitsu (57 Benchmarks)
  3. Fujitsu Siemens (106 Benchmarks)
  4. Hitachi (49 Benchmarks)
  5. HP/HPE (209 Benchmarks)
  6. IBM (201 Benchmarks)
  7. NEC (48 Benchmarks)
  8. Sun Microsystems (64 Benchmarks)
  9. Cisco Systems (37 Benchmarks)

AWS only had 14 benchmarks, but this list of SD benchmarks goes back for years. The first benchmark was performed in 1995, and goes all the way up to 2019. AWS does not get its first published benchmark until 2013.

As an example for recent years, there were 32 benchmarks performed in 2018.

The Natural Question That Arises

S/4HANA was introduced in 2015 and just recently had its 4 year anniversary. HANA is going on its 8 year anniversary. Within that context let us take note of the following facts about SAP’s benchmarking.

  • There is not a single benchmark for S/4HANA SD.
  • There is not a single benchmark for HANA for any ERP or other transaction processing system.
  • SAP has only published benchmarks for BW (both the BWH and the BWAML benchmarks)

The HANA benchmark could be run without S/4HANA as HANA works for ECC in something called suite on HANA. However, this setup was never benchmarked.

Conclusion

SAP made enormous claims for both HANA and S/4HANA. However, it has published exactly zero transaction processing benchmarks for HANA (with ECC) or HANA with S/4HANA. The only HANA benchmark that was published where there was a comparison was performed by Lenovo and its problems are covered in the article The Problems with the Strange Lenovo HANA Benchmark.

Information reported to us from the field, which we covered in the article HANA as a Mismatch for S/4HANA and ERP, illustrates that HANA is a weak performer in transaction processing. Our information shows that HANA underperforms previous versions of databases that were used for ECC in transaction processing, which is the dominant processing type in ERP systems.

It seems quite likely that SAP has published no ERP benchmarks for HANA because it would show that SAP’s statements about being able to master both OLTP and OLAP from one database are false.

Search Our Other HANA Content

Brightwork Disclosure

Financial Bias Disclosure

This article and no other article on the Brightwork website is paid for by a software vendor, including Oracle and SAP. Brightwork does offer competitive intelligence work to vendors as part of its business, but no published research or articles are written with any financial consideration. As part of Brightwork’s commitment to publishing independent, unbiased research, the company’s business model is driven by consulting services; no paid media placements are accepted.

HANA & S/4HANA Question Box

  • Have Questions About S/4HANA & HANA?

    It is difficult for most companies to make improvements in S/4HANA and HANA without outside advice. And it is close to impossible to get honest S/4HANA and HANA advice from large consulting companies. We offer remote unbiased multi-dimension S/4HANA and HANA support.

    Just fill out the form below and we'll be in touch.

References

https://www.sap.com/dmc/exp/2018-benchmark-directory/#/sd

https://www.sap.com/about/benchmark/measuring.html

“SAP Application Performance Standard (SAPS) is a hardware-independent unit of measurement that describes the performance of a system configuration in the SAP environment. It is derived from the Sales and Distribution (SD) benchmark, where 100 SAPS is defined as 2,000 fully business processed order line items per hour.

In technical terms, this throughput is achieved by processing 6,000 dialog steps (screen changes), 2,000 postings per hour in the SD Benchmark, or 2,400 SAP transactions.

In the SD benchmark, fully business processed means the full business process of an order line item: creating the order, creating a delivery note for the order, displaying the order, changing the delivery, posting a goods issue, listing orders, and creating an invoice.”

https://www.sap.com/about/benchmark.html

AWS and Google Cloud Book

How to Leverage AWS and Google Cloud for SAP and Oracle Environments

Interested in how to use AWS and Google Cloud for on-premises environments, and why this is one of the primary ways to obtain more value from SAP and Oracle? See the link for an explanation of the book. This is a book that provides an overview that no one interested in the cloud for SAP and Oracle should go without reading.