How Accurate Was John Appleby on HANA Not Working Fast for OLTP?

Executive Summary

  • John Appleby made bold predictions on HANA.
  • We review how accurate he was in his article on HANA Not Working Fast for OLTP.

Introduction

John Appleby’s article on the SAP HANA blog was titled Who says HANA doesn’t work fast for OLTP? and was published on November 18, 2014.

The Quotations

HANA Performance with OLTP?

“Sometimes people think that because HANA is a columnar database, it doesn’t run fast for simple OLTP operations. I was just looking at a performance problem with class /IWBEP/CL_MGW_ABS_MODEL, method GET_LAST_MODIFIED.

This had some screwy ABAP, which is incidentally fixed in SAP Note 2023100 (thanks Oliver Rogers for finding the note), and it generated the following SQL:

SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c

FROM “REPOSRC”

WHERE “PROGNAME” LIKE ‘CL_ABAP_COMP_CLASS%’ AND “R3STATE” = ‘A’

ORDER BY “CDAT” ,”UDAT” ,“UTIME”

That SQL is pretty nasty, because it does a wildcard search on a big table. On the non-HANA system it was running in 20 seconds. I did the root cause analysis in the database and found that it was searching the primary clustered index, which was 98% fragmented.

Obviously I rebuilt the index – these are the results.

CPU time = 2750 ms,  elapsed time = 2746 ms.

CPU time = 2594 ms,  elapsed time = 2605 ms.

CPU time = 2750 ms,  elapsed time = 2764 ms.

I realized at this point this was some bad coding, so I found the fix thanks to Oli and we put the change in. That fixed the performance problem.

But then I thought… what happens if you run this bad query on a HANA system? This is just what custom code looks like a lot of the time…

Statement ‘SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c FROM “REPOSRC” WHERE “PROGNAME” LIKE …’

successfully executed in 12 ms 414 µs  (server processing time: 11 ms 613 µs)

Fetched 8 row(s) in 0 ms 68 µs (server processing time: 0 ms 0 µs)

Statement ‘SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c FROM “REPOSRC” WHERE “PROGNAME” LIKE …’

successfully executed in 9 ms 778 µs  (server processing time: 9 ms 136 µs)

Fetched 8 row(s) in 0 ms 64 µs (server processing time: 0 ms 0 µs)

Statement ‘SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c FROM “REPOSRC” WHERE “PROGNAME” LIKE …’

successfully executed in 12 ms 677 µs  (server processing time: 11 ms 830 µs)

Fetched 8 row(s) in 0 ms 56 µs (server processing time: 0 ms 0 µs)

So anyDB is averaging 2705ms, and HANA is averaging 10.86ms, an average speedup of 249x.

You may be saying… OK well that’s for poorly written SQL – what about when it was optimized. Sure, let’s test in that scenario. Here’s the SQL:

SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c

FROM “REPOSRC”

WHERE

“PROGNAME” IN (‘CL_ABAP_COMP_CLASS============CCDEF’, ‘CL_ABAP_COMP_CLASS============CCIMP’, ‘CL_ABAP_COMP_CLASS============CCMAC’, ‘CL_ABAP_COMP_CLASS============CI’, ‘CL_ABAP_COMP_CLASS============CO’, ‘CL_ABAP_COMP_CLASS============CP’, ‘CL_ABAP_COMP_CLASS============CT’, ‘CL_ABAP_COMP_CLASS============CU’)

AND “R3STATE” = ‘A’

ORDER BY “CDAT”, “UDAT”, “UTIME”

So ran it on anyDB, I couldn’t get accurate results from the SQL console so I had to use the ABAP trace to get the numbers. They were 5.504ms, 1.484ms, 4.605ms for an average of 3.86ms. Let’s see how HANA compares.

Statement ‘SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c FROM “REPOSRC” WHERE “PROGNAME” IN …’

successfully executed in 1 ms 977 µs  (server processing time: 1 ms 156 µs)

Fetched 8 row(s) in 0 ms 63 µs (server processing time: 0 ms 0 µs)

Statement ‘SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c FROM “REPOSRC” WHERE “PROGNAME” IN …’

successfully executed in 1 ms 946 µs  (server processing time: 1 ms 250 µs)

Fetched 8 row(s) in 0 ms 60 µs (server processing time: 0 ms 0 µs)

Statement ‘SELECT “PROGNAME”, “CDAT” AS c ,”UDAT” AS c ,”UTIME” AS c FROM “REPOSRC” WHERE “PROGNAME” IN …’

successfully executed in 2 ms 230 µs  (server processing time: 1 ms 127 µs)

Fetched 8 row(s) in 0 ms 59 µs (server processing time: 0 ms 0 µs)

With HANA then, we get an average of 1.18ms for an average speedup of 3.27x.

Yes, HANA does not run fast for OLTP. That is well established by this time in 2019 as we covered in HANA as a Mismatch for S/4HANA and ERP.

This overall explanation by Appleby for this test is illogical and including the detail that he did does not answer the question that he poses with the title of his article. And this is because the framework of the test is completely wrong. This particular example may be an example of a badly designed query, but HANA’s long term issues with transaction processing cannot be entirely due to bad queries. And secondly, this test actually has nothing to do with this as is pointed out by Ahmed Azmi.

“Query speed is NOT OLTP. OLTP means online TRANSACTION processing. You need to at least do a single transaction and perform an insert/update an a commit/rollback. This tests how your DB can do things like locking and redo log management.”

So why is Appleby using a query as an example of transaction processing?? A query is actually OLAP or analytics, so is Appleby unaware of the difference between OLAP and OLTP, because this article indicates that he is unaware of the distinction.

Shouldn’t the title of the article be how improving a query or rebuilding an index can improve query performance?

What on God’s Green Earth is Appleby’s Understanding of Testing?

“For poorly constructed OLTP queries at the database level, we can get enormous benefits of running HANA – up to 250x or more. With optimized SQL that hits database indexes on anyDB, that drops to around 3.27x, but SAP have only ever claimed a 2-3x increase of running ERP on HANA for transactional workloads.

And remember if you move to the sERP suite, you’ll see another 2-3x because the data structures are simpler. That’s going to mean response times of 5-10x faster than on anyDB.

I don’t know about, you, but that feels significant to me.

Yes, I know I didn’t do concurrency, inserts, updates and all that jazz. This was just a quick test run with 15 minutes of spare time. Hope it is informative. It’s also worth noting that with program changes, I was in this case able to get acceptable performance using anyDB for our timesheet app. The only area where performance is a problem is for the WBS element search, which is a wildcard search again.

For those searches, HANA rocks. For customer, product – anything with a free text search, HANA is going to kill anyDB.

Ok, where to begin with this quote.

First, this is a constant feature of how Appleby does tests. He has a very short attention span and writes up results as if he needs to be medicated for attention deficit disorder. No person with this type of attention span or lack of attention to detail can ever work in testing or in communicating testing results.

Something else which is apparent from reading Appleby’s tests or comparisons is that Appleby will always have an excuse for why he was unable to perform any type of thorough work. For example, in his “TCO” article that we covered in How Accurate Was John Appleby on HANA TCO for Cloud vs On-Premises?

He stated..

“Its an extremely simplistic TCO and doesn’t take into account the operational costs of running HANA”

Or the consulting costs for that matter!

That is, it isn’t a TCO at all, which means the article, which is titled “HANA TCO for Cloud vs On Premises” is completely misnamed. The title should have been “The License and Cloud Costs of HANA,” as he made zero attempts to calculate TCO. Appleby seems to think that he can get by without putting work into his analysis, and further that he can name analysis things that they are not. If I state I intend to estimate the weight of the earth, and then stop at estimating the weight of the oceans, I cannot say that my estimate was “quick and dirty” because I did not even attempt to complete the job.

In this article he states…

“This was just a quick test run with 15 minutes of spare time.”

Why is Appleby so pressed for time?

Appleby had people working for him at this time at Bluefin Solutions. There were plenty of more junior Bluefin consultants that he could have directed to perform this test, and he was one of the best-known sources on HANA at this time, so why the unwillingness to apply either his own mental effort and time or that of any of the other Bluefin resources to this task?

Appleby does not seem to realize that is not setting up a logical test. This test only shows how a query can be sped, it does not answer the question of why HANA is slow in transaction processing overall.

Notice in the next quote how he quickly inserts an evidence-free claim or hypothesis into the test analysis.

Why is the following sentence true?

“And remember if you move to the sERP suite, you’ll see another 2-3x because the data structures are simpler. That’s going to mean response times of 5-10x faster than on anyDB.”

What? What does that have to do with the test. The comment about simpler data structures is a hypothesis and one that has proven to not be true as we covered in the article How Accurate Was SAP About S/4HANA and a Simplified Data Model?

Data points from HANA projects have shown that these estimates are false and that HANA underperforms competing databases. Secondly, this hypothesis should not be included in this “test.”

Then he states the following.

“I don’t know about, you, but that feels significant to me.”

5-10x faster is significant…..but not demonstrated. This would be like saying that the moon is made of cheese, and

“I don’t know about you but that sounds like it would taste good to me.”

Yes, but the first step is to prove the claim, not to comment on the benefit of the claim. Or in Appleby’s world would we simply accept the claim and then move on to discuss how we will eat all of that cheese?

What is the most important question when the claim is made the moon is made of cheese? Is it…

  1. Think about the fact that that is a lot of cheese and that it probably would taste very good?
  2. Question whether it is true that the moon is made of cheese? 

Then Appleby goes on to make another claim.

“For those searches, HANA rocks. For customer, product – anything with a free text search, HANA is going to kill anyDB.”

Appleby is back to what is does best making claims. He is certainly not good at proving claims. Even his articles that propose to prove a claim do nothing of the sort. And SAP has produced no comparative benchmarks with HANA to back up their claims or Appelby’s claims as we covered in the article The Hidden Issue with the SD HANA Benchmark.

Comment from Shkelzen Hasanaj: The Issue of Comparable Hardware

“I understand that claims about speed increase are tested and I am not saying that it is not true, but I find your comparison in this case a little bit incomplete. From what information you provide, only the query is the same. We have no information about HW specifications etc. Moreover, I think it should be compared to its in-memory counterparts, not just any DB.

Yeah, HANA rocks but still, from this information, you can not pretend anything at all.”

I believe Shkelzen meant to say “you cannot pretend to know anything at all.” Which is true.

John Appleby’s Response

“This is a little ditty and not a large-scale comparison. I’d sure like to do such a thing, but only had 15 minutes to spare and thought the results were interesting enough to write about.

The hardware was certainly different. The MSSQL has a much faster CPU (Ivy Bridge vs Westmere) but HANA has a lot more cores and RAM.

250x is still significant whatever the hardware.”

And Appleby pivots from the question, by again stating the test is not thorough. And is that an understatement. Appleby only has 15 minutes to test his hypothesis? Interesting. If Appleby only had so little time to test his hypothesis perhaps he should have stopped making so many of them. In all the time I have spent reviewing research, I can’t ever recall a “15 minute time limitation” being used as an excuse divert from the question on the research.

Appleby’s sloppiness is inherent in how he brushes aside the hardware issue. This illustrates how Appleby’s imprecise thinking, and how little he understands or seem to care about testing.

Comment from Bill Ramos: Where Are the Benchmarks?

“If SAP HANA is so fast with OLTP, why has SAP yet to publish benchmark results for it’s own SD benchmark – http://global.sap.com/solutions/benchmark/sd2tier.epx? It’s getting old to hear that SAP HANA is “different” and enables a completely different way to thing about databases and that the old benchmarks no longer matter. I’d love to see a heads up OLTP type benchmark run where you can really contrast OLTP performance in a reproducible environment.

   If you could provide the data and code environment, I’d love to run SAP HANA against say SQL Server 2014 In-Memory OLTP on an identical AWS configuration. 🙂

Also, SAP HANA really needs to take a look at optimizing stored procedure execution. During some testing I did on build 76, stored procedures executed almost twice as slow as running the query that was part of the stored procedure. I had to get rid of all my stored procedures to get my app to run at top speed.

As John mentioned below, publishing benchmarks – even in a blog like this could land someone it hot water, but you can bet the vendors are always internally comparing each other. They just can’t publish results. This is why you have cross vendor “standards” like TPC and the SAP SD benchmark which require careful reviews of the results. Just as John suggests, SAP isn’t ready to throw HANA into the fray. At least SAP Sybase ASE shows up to the table.”

Bill is correct. SAP never publishing an OLTP benchmark when it introduced HANA. We hypothesize this was because publishing it would have illuminated the fact that HANA is deficient in transaction processing which we covered in the article What is the Actual HANA Performance?

Appleby’s Response

“You know, you get no-nonsense from me Bill.

I’ve not run the benchmark but I believe it’s because:

1) SD doesn’t run well on HANA

2) SD doesn’t accurately represent how customers actually use systems in 2014

3) HANA does run well for how customers really use systems in 2014

SAP are in the process of creating a new benchmark which I understand will include mixed-workload OLTAP queries.

There is a BW-EML benchmark which does mimic how customers use OLAP systems in 2014. Unfortunately it has some design flaws which mean it can be gamed (for instance you can run loads on one area and queries on the other, rather than forcing both at the same time).

Benchmarking is much better done on real customer data with real scenarios. I have done a lot of it, but none of it is possible to share in the public domain.

Also, the license agreements with Microsoft/IBM/Oracle/SAP specifically don’t allow benchmarking. That’s why I referred to the database above as anyDB, so it can’t be recognized.

What? So Appleby is proposing that SD transaction processing does not run well on HANA and that furthermore, SAP suppresses benchmarks that don’t make HANA look good? I mean we knew this, but Appleby is admitting this? Also, why would this be the case? HANA, according to Bill McDermott runs 100,000 times faster than any other competing technology. It should require no benchmarks to be hidden. Recall that S/4HANA was restricted to HANA under the logic that

“no other database could keep up with SAP’s innovation.”

What is this comment that HANA does not run SD well, but does run how for customers run SD? At this point in time S/4HANA has yet to be released, so 100% of customers in 2014 for SAP were running SD from ECC!!!!!

The following questions naturally arise.

  • Why does SAP need to create a new benchmark?
  • Is his because HANA does so poorly on the previous benchmark?
  • Is that how SAP does things?
  • It rigs benchmarks if its own products cannot perform well in them?

Conclusion

This article scores a 0 out of 10 for accuracy.

The article covers what purports to be a test, but does not test HANA’s transaction processing performance. Appleby can’t seem to put any effort into testing with every test being “quick and dirty” or otherwise lacking and incomplete.

Appleby then segues to intermingling more exaggerated claims about HANA in transaction processing. Then in the comments section admits that SD (a transaction processing module) probably does not run well on HANA! He then tops this off by accusing SAP of what we already knew, that SAP rigs benchmarks to benefit their products over competitors and then simply does not release benchmarks if the benchmark makes SAP’s products look bad.

What? What?

This has to be categorized as the most insane article we have ever critiqued. This is the brain that was being used to advise companies on HANA?

SAP’s Inaccurate Messaging on HANA as Communicated in SAP Videos

Fact-Checking SAP’s HANA Information

This video is filled with extensive falsehoods. We will address them in the sequence they are stated in this video.

SAP Video Accuracy Measurement

SAP's Statement
Accuracy
Brightwork Fact Check
Link to Analysis Article
HANA is a Platform
0%
HANA is not a platform, it is a database.How to Deflect You Were Wrong About HANA
HANA runs more "in-memory" than other databases.
10%
HANA uses a lot of memory, but the entire database is not loaded into memory.How to Understand the In-Memory Myth
S/4HANA Simplifies the Data Model
0%
HANA does not simplify the data model from ECC. There are significant questions as to the benefit of the S/4HANA data model over ECC.Does HANA Have a Simplified Data Model?
Databases that are not HANA are legacy.
0%
There is zero basis for SAP to call all databases that are not HANA legacy.SAP Calling All Non-HANA DBs Legacy.
Aggregates should be removed and replaced with real time recalculation.
0%
Aggregates are very valuable, and all RDBMS have them (including HANA) and they should not be removed or minimized in importance.Is Hasso Plattner Correct on Database Aggregates?
Reducing the number of tables reduces database complexity.
0%
Reducing the number of tables does not necessarily decrease the complexity of a database. The fewer tables in HANA are more complicated than the larger number of tables pre-HANA.Why Pressure SAP to Port S/4HANA to AnyDB?
HANA is 100% columnar tables.
0%
HANA does not run entirely with columnar tables. HANA has many row-oriented tables, as much as 1/3 of the database.Why Pressure SAP to Port S/4HANA to AnyDB?
S/4HANA eliminates reconciliation.
0%
S/4HANA does not eliminate reconciliation or reduce the time to perform reconciliation to any significant degree.Does HANA Have a Simplified Data Model and Faster Reconciliation?
HANA outperforms all other databases.
0%
Our research shows that not only can competing databases do more than HANA, but they are also a better fit for ERP systems.How to Understand the Mismatch Between HANA and S/4HANA and ECC.

The Problem: A Lack of Fact-Checking of HANA

There are two fundamental problems around HANA. The first is the exaggeration of HANA, which means that companies that purchased HANA end up getting far less than they were promised. The second is that the SAP consulting companies simply repeat whatever SAP says. This means that on virtually all accounts there is no independent entity that can contradict statements by SAP.

Being Part of the Solution: What to Do About HANA

We can provide feedback from multiple HANA accounts that provide realistic information around HANA — and this reduces the dependence on biased entities like SAP and all of the large SAP consulting firms that parrot what SAP says. We offer fact-checking services that are entirely research-based and that can stop inaccurate information dead in its tracks. SAP and the consulting firms rely on providing information without any fact-checking entity to contradict the information they provide. This is how companies end up paying for a database which is exorbitantly priced, exorbitantly expensive to implement and exorbitantly expensive to maintain. When SAP or their consulting firm are asked to explain these discrepancies, we have found that they further lie to the customer/client and often turn the issue around on the account, as we covered in the article How SAP Will Gaslight You When Their Software Does Not Work as Promised.

If you need independent advice and fact-checking that is outside of the SAP and SAP consulting system, reach out to us with the form below or with the messenger to the bottom right of the page.

The major problem with companies that bought HANA is that they made the investment without seeking any entity independent of SAP. SAP does not pay Gartner and Forrester the amount of money that they do so these entities can be independent as we covered in the article How Accurate Was The Forrester HANA TCO Study?

If you need independent advice and fact-checking that is outside of the SAP and SAP consulting system, reach out to us with the form below or with the messenger to the bottom right of the page.

Inaccurate Messaging on HANA as Communicated in SAP Consulting Firm Videos

For those interested in the accuracy level of information communicated by consulting firms on HANA, see our analysis of the following video by IBM. SAP consulting firms are unreliable sources of information about SAP and primarily serve to simply repeat what SAP says, without any concern for accuracy. The lying in this video is brazen and shows that as a matter of normal course, the consulting firms are happy to provide false information around SAP.

SAP Video Accuracy Measurement

SAP's Statement
Accuracy
Brightwork Fact Check
Link to Analysis Article
HANA runs more "in-memory" than other databases.
10%
HANA uses a lot of memory, but the entire database is not loaded into memory.How to Understand the In-Memory Myth
HANA is orders of magnitude faster than other databases.
0%
Our research shows that not only can competing databases do more than HANA, but they are also a better fit for ERP systems.How to Understand the Mismatch Between HANA and S/4HANA and ECC.
HANA runs faster because it does not use disks like other databases.
0%
Other databases also use SSDs in addition to disk.Why Did SAP Pivot the Explanation of HANA In Memory?
HANA holds "business data" and "UX data" and "mobile data" and "machine learning data" and "IoT data."
0%
HANA is not a unifying database. HANA is only a database that supports a particular application, it is not for supporting data lakes.
SRM and CRM are part of S/4HANA.
0%
SRM and CRM are not part of S/4HANA. They are separate and separately sold applications. SAP C/4HANA is not yet ready for sale. How Accurate Was Bluefin Solutions on C-4HANA?
Netweaver is critical as a platform and is related to HANA.
0%
Netweaver is not relevant for this discussion. Secondly Netweaver is not an efficient environment from which to develop.
HANA works with Business Objects
10%
It is very rare to even hear about HANA and Business Objects. There are few Buisness Objects implementations that use HANA.SAP Business Objects Rating
Leonardo is an important application on SAP accounts.
0%
Leonardo is dead, therefore its discussion here is both misleading and irrelevant.Our 2019 Observation: SAP Leonardo is Dead
IBM Watson is an important application on SAP accounts.
0%
Watson is dead, therefore its discussion here is both misleading and irrelevant.How IBM is Distracting from the Watson Failure to Sell More AI and Machine Learning
Digital Boardroom is an important application on SAP accounts.
0%
SAP Digital Boardroom is another SAP item that has never been implemented many places.

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Search Our Other John Appleby Fact Checking Content

References

https://blogs.sap.com/2014/11/18/who-says-hana-doesnt-work-fast-for-oltp/

TCO Book

 

TCO3

Enterprise Software TCO: Calculating and Using Total Cost of Ownership for Decision Making

Getting to the Detail of TCO

One aspect of making a software purchasing decision is to compare the Total Cost of Ownership, or TCO, of the applications under consideration: what will the software cost you over its lifespan? But most companies don’t understand what dollar amounts to include in the TCO analysis or where to source these figures, or, if using TCO studies produced by consulting and IT analyst firms, how the TCO amounts were calculated and how to compare TCO across applications.

The Mechanics of TCO

Not only will this book help you appreciate the mechanics of TCO, but you will also gain insight as to the importance of TCO and understand how to strip away the biases and outside influences to make a real TCO comparison between applications.
By reading this book you will:
  • Understand why you need to look at TCO and not just ROI when making your purchasing decision.
  • Discover how an application, which at first glance may seem inexpensive when compared to its competition, could end up being more costly in the long run.
  • Gain an in-depth understanding of the cost, categories to include in an accurate and complete TCO analysis.
  • Learn why ERP systems are not a significant investment, based on their TCO.
  • Find out how to recognize and avoid superficial, incomplete or incorrect TCO analyses that could negatively impact your software purchase decision.
  • Appreciate the importance and cost-effectiveness of a TCO audit.
  • Learn how SCM Focus can provide you with unbiased and well-researched TCO analyses to assist you in your software selection.
Chapters
  • Chapter 1:  Introduction
  • Chapter 2:  The Basics of TCO
  • Chapter 3:  The State of Enterprise TCO
  • Chapter 4:  ERP: The Multi-Billion Dollar TCO Analysis Failure
  • Chapter 5:  The TCO Method Used by Software Decisions
  • Chapter 6:  Using TCO for Better Decision Making