- SAP has used the SD benchmarks for years for other databases.
- Four years into S/4HANA, SAP has no benchmarks for S/HANA SD or ECC SD on HANA.
SAP has used the SD benchmarks for years to test for transaction processing performance, but something peculiar happened when SAP released HANA, and the benchmarking was to be performed for S/4HANA.
In this article, we will cover the hidden issues with SD benchmarks.
The SD Benchmark for HANA
Here are the SD benchmarks listed on the SAP website for HANA.
This looks like an impressive list of benchmarks for different hardware and different operating systems and database releases. At the time we reviewed this data, we counted 1066 benchmarks.
Now notice the databases that were used for the benchmarks.
Each benchmark typically is a single database version or variant of the version, and there are many database versions or variants. For instance, there is a separate benchmark for Oracle 10g, Oracle 10g with Real Application Cluster.
The databases are as follows.
- DB2: (41 Benchmarks)
- Adabas: (1 Benchmark)
- SQL Server: (17 Benchmarks)
- SAP ASE (4 Benchmarks)
- Informix: (7 Benchmarks)
- MaxDB: (4 Benchmarks)
- SAP DB: (4 Benchmarks)
- Oracle (26 Benchmarks)
The Operating Systems
- IBM AIX
- Red Hat Enterprise
- IBM OS
- SUSE Linux
- Windows, 2000, .NET, NT, Enterprise Server, Enterprise Edition, etc..
The most benchmarks were run for Windows Enterprise Server 2003 with 135.
The benchmarks span 52 different versions of ECC. The most frequently benchmarked ECC/R/3 versions were the following.
- ECC EHP 5 for ERP 6.0: (242 Benchmarks)
- R/3 4.6C: (124 Benchmarks)
- SAP EHP 4 for ERP 6.0: (118 Benchmarks)
- Bare Metal: (984 Benchmarks)
- Cloud: (44 Benchmarks)
- Virtualized: (36 Benchmarks)
This means that 92% and therefore the bulk of the benchmarks were run on bare metal and on premises.
The benchmarks used hardware from 28 different hardware vendors and AWS (not a hardware vendor but a cloud services provider that uses its own open specified hardware as we covered in How the Open Compute Project Reduced the Power of Proprietary Vendors).
The largest number of benchmarks per hardware vendor was as follows:
- Dell: (86 Benchmarks)
- Fujitsu (57 Benchmarks)
- Fujitsu Siemens (106 Benchmarks)
- Hitachi (49 Benchmarks)
- HP/HPE (209 Benchmarks)
- IBM (201 Benchmarks)
- NEC (48 Benchmarks)
- Sun Microsystems (64 Benchmarks)
- Cisco Systems (37 Benchmarks)
AWS only had 14 benchmarks, but this list of SD benchmarks goes back for years. The first benchmark was performed in 1995 and went all the way up to 2019. AWS does not get its first published benchmark until 2013.
As an example, in recent years, there were 32 benchmarks performed in 2018.
The Natural Question That Arises
S/4HANA was introduced in 2015 and just recently had its fourth anniversary. HANA is going on its eighth anniversary. Within that context, let us take note of the following facts about SAP’s benchmarking.
- There is not a single benchmark for S/4HANA SD.
- There is not a single benchmark for HANA for any ERP or other transaction processing systems.
- SAP has only published benchmarks for BW (both the BWH and the BWAML benchmarks)
The HANA benchmark could be run without S/4HANA as HANA works for ECC in something called suite on HANA. However, this setup was never benchmarked.
SAP made enormous claims for both HANA and S/4HANA. However, it has published exactly zero transaction processing benchmarks for HANA (with ECC) or HANA with S/4HANA. The only HANA benchmark that was published where there was a comparison was performed by Lenovo, and its problems are covered in the article The Problems with the Strange Lenovo HANA Benchmark.
Information reported to us from the field, which we covered in the article HANA as a Mismatch for S/4HANA and ERP, illustrates that HANA is a weak performer in transaction processing. Our information shows that HANA underperforms previous versions of databases that were used for ECC in transaction processing, which is the dominant processing type in ERP systems.
It seems quite likely that SAP has published no ERP benchmarks for HANA because it would show that SAP’s statements about being able to master both OLTP and OLAP from one database are false.
“SAP Application Performance Standard (SAPS) is a hardware-independent unit of measurement that describes the performance of a system configuration in the SAP environment. It is derived from the Sales and Distribution (SD) benchmark, where 100 SAPS is defined as 2,000 fully business processed order line items per hour.
In technical terms, this throughput is achieved by processing 6,000 dialog steps (screen changes), 2,000 postings per hour in the SD Benchmark, or 2,400 SAP transactions.
In the SD benchmark, fully business processed means the full business process of an order line item: creating the order, creating a delivery note for the order, displaying the order, changing the delivery, posting a goods issue, listing orders, and creating an invoice.”