Executive Summary
- For years John Appleby provided information about HANA to the market.
- This is a study that measure’s John Appleby’s accuracy.
Introduction
HANA was introduced in 2011, although little actually happened with HANA in 2011. In 2012 and particularly into 2013 and forward, John Appleby of Bluefin Solutions published large amounts of inaccurate information about HANA. Appleby’s inaccuracies were so extensive, that this study was created in order to catalogue many (but certainly not all) of Appleby’s statements about HANA and to measure the typical accuracy level of Appleby’s statements about HANA.
The Method
This study is a document analysis of some of Appleby’s most popular and influential blog posts. That is one half of the study. The other half is the reality of HANA. Brightwork Research & Analysis has performed the most and the most extensive research of HANA and this website over 250 articles on HANA and S/4HANA (the ERP system mated to HANA). That is with the caveat of research outside of benchmarking. The entities with the benchmarking evidence around HANA are the database and hardware vendors. However, because of license and partnerships, these benchmarks cannot be published, and in the one case where a benchmark was published, it was heavily redacted and rigged in favour of HANA as we covered in the article The Problems with the Strange Lenovo HANA Benchmark.
Brightwork Research & Analysis is also one of the only information providing entities that cover SAP that is not financially tied to SAP (or to any other vendor) (for example, Forrester, Gartner, Nucleus Reseach and all of the US-based IT media outlets are financially connected to SAP). This lack of financial relationship allows us to publish our analysis of accuracy without concern for retaliation on the part of the SAP.
Understanding the Retaliatory Measures Against Independent Research
The standard retaliatory measure being for the vendor to threaten to remove or reduce funding unless the offending analyst is fired, as Oracle requested against DeWitt when he published a benchmark that caused Oracle displeasure as Oracle was not ranked at the very top of the benchmark. Oracle went further in the DeWitt case by threatening the University of Wisconsin with no longer hiring its graduates. The position of firms like SAP and Oracle is that those that publish research that is not consistent with their marketing materials should be removed from performing research. And in the case of Oracle, they were intent on bringing this pressure against a major university.
The Method Used
The method was to fact check statements made in Appleby’s articles and rate each of the statements for accuracy. This resulted in an average accuracy.
What Statements Are Measured
Appleby is very well connected to SAP and has received a steady stream of information from SAP since he developed a relationship with SAP’s upper reaches. In his article 10 HANA Predictions for 2015, Appleby made “predictions” but these were just things that development was working on at the time. We did not measure these statements for accuracy.
In some cases, Appleby made statements such as HANA worked with S/4HANA — which is true but would be like crediting someone for saying the “sun rises in the East.” Therefore, the study measured statements, where something more than elementary and universally agreed upon observations, were made.
The Evidence for the Percentage Rating
- What you see below in the Appleby Accuracy Checker Table. This table (which is searchable, see the search box at the top) is designed to provide a listing of the measured statements.
- The evidence that the statement’s accuracy percentage is found in the article link, which in each case links to a Brightwork Research & Analysis article where the statement is analyzed. Using a CTRL-F for the analysis article for the item you are looking for should bring you directly to the section you seek to find the supporting analysis.
- Because Appleby would not have made claims that were not cleared by SAP, and because most of the claims were also made by SAP, this study can also be used as a rough approximation of SAP’s accuracy on HANA. Our Study into SAP’s Accuracy covers that topic separately.
Appleby Accuracy Checker Table
Appleby's Statement | Accuracy % of the Comment | Link to Analysis Article |
---|---|---|
HANA runs on BW up to 100x faster. | 0% | What In Memory DB for BW |
SP06 of HANA had all the capabilities of a mature database. | 0% | What In Memory DB for BW |
HANA was the only database with full HA. | 50% | What In Memory DB for BW |
HANA simplifies the environment but installing multiple applications on a single HANA instance. | 0% | What In Memory DB for BW |
DB2 BLU does not accelerate all of the queries that HANA does. | 0% | What In Memory DB for BW |
There is a 1 TB limit on DB2 BLU | 0% | What In Memory DB for BW |
In Nov 2013 DB2 BLU did not support BW. | 0% | Appleby Beaten by Chris Eaton |
Maintenance is reduced with HANA | 0% | Appleby Beaten by Chris Eaton |
HANA only has two patches (of any kind) per year. | 0% | Appleby Beaten by Chris Eaton |
HANA had constant innovations. | 0% | Appleby Beaten by Chris Eaton |
In 2013 HANA had 24 months of stability behind it. | 0% | Appleby Beaten by Chris Eaton |
SAP did not purposefully withhold information from customers that would have shown DB2's certification. | 0% | Appleby Beaten by Chris Eaton |
Bluefin did DB2 substantial work. | 0% | Appleby Beaten by Chris Eaton |
HANA would never need separate row and column stores for transactional data and reporting data. | 0% | Appleby Beaten by Chris Eaton |
IBM BLU is not well suited to OLTP workloads. | 0% | Appleby Beaten by Chris Eaton |
HANA disrupted the database market. | 10% | Appleby Beaten by Chris Eaton |
Microsoft's "in memory" database is incremental compared to HANA's. | 0% | Appleby Beaten by Chris Eaton |
HANA is SAP's fastest growing product ever. | 0% | A Historical View of HANA |
The Model T was not quickly adopted. | 0% | A Historical View of HANA |
HANA was priced in a way to gain widespread adoption -- like the Model T. | 0% | A Historical View of HANA |
HANA was designed similarly to the Model T to reduce database maintenance. | 0% | A Historical View of HANA |
HANA, like the Model T, eliminates inefficient industries that surround databases. | 0% | A Historical View of HANA |
SAP has a club of customers that have seen 100,000 times performance improvement from implementing HANA. | 0% | A Historical View of HANA |
Oracle, IBM and Microsoft have formidable sales and marketing teams to push their databases (but by inference) SAP does not. | 0% | A Historical View of HANA |
HANA reduces the number of components required for development. | 0% | A Historical View of HANA |
TCO is only the license and hosting cost. | 0% | Appleby's TCO of HANA |
Because Appleby is not paid by SAP, he has no pro-SAP bias and no pro-HANA bias. | 0% | On HANA Replacing BW |
All duplicated data between ERP an the data warehouse is waste. | 0% | On HANA Replacing BW |
HANA is a data platform and application platform. | 0% | On HANA Replacing BW |
HANA does not provide a number of functions required for a complex enterprise data warehouse. | 100% | On HANA Replacing BW |
BW takes care of SAP security and authorizations, complex hierarchies, non-cumulative data for stock and many other capabilities. | 100% | On HANA Replacing BW |
SAP is still investing in BW. | 100% | On HANA Replacing BW |
HANA supports cold data. | 0% | On HANA Replacing BW |
BW is being implemented in the cloud. | 10% | On HANA Replacing BW |
HANA won't replace BW | 100% | On HANA Replacing BW |
HANA was doing things no other database could do. | 0% | HANA SPS08 |
Microsoft cannot get traction in the SAP market, which is why SAP does not support MySQL or PostgreSQL. | 0% | HANA SPS08 |
Microsoft cannot get traction in the SAP market, which is why SAP does not support MySQL or PostgreSQL. | 0% | HANA SPS08 |
Oracle produces FUD, but SAP only provides information. | 0% | HANA Versus DB2 |
SAP has to be sensitive to positioning HANA because of reseller agreements with IBM and Oracle | 0% | HANA Versus DB2 |
John Appleby is unbiased because he does not work for SAP. | 0% | HANA Versus DB2 |
HANA can solve problems that cannot be solved in DB2 or Oracle. | 0% | HANA Versus DB2 |
HANA is a platform not just a database. | 0% | HANA Versus DB2 |
A 34 question form can only function properly if supported by HANA. | 0% | HANA Versus DB2 |
Only SAP writes software that takes advantage of HANA's design. | 0% | HANA Versus DB2 |
HANA is enterprise ready (in 2014). | 10% | HANA Versus DB2 |
Oracle and IBM need to catch up to HANA. | 20% | HANA Versus DB2 |
HANA can be made reasonable in price. | 0% | HANA Versus DB2 |
SAP apps run far faster on HANA. | 0% | HANA Versus DB2 |
HANA running slower that competing databases is entirely due to poor code. | 0% | HANA Versus DB2 |
HANA is an application platform. | 0% | HANA FAQ |
HANA runs from 10 to 1000x faster than a regular database like Oracle. | 0% | HANA FAQ |
HANA is a reinvention of the database. | 0% | HANA FAQ |
HANA is based upon 30 years of technology improvements. | 0% | HANA FAQ |
HANA allows the build of applications not possible on traditional RDBMSs. | 0% | HANA FAQ |
HANA allows the renewal of existing applications like the SAP Business Suite. | 0% | HANA FAQ |
HANA simplifies SAP's applications. | 0% | HANA FAQ |
No other database vendor could have added a column store to their databases. | 0% | HANA FAQ |
HANA is not just a database | 0% | HANA FAQ |
HANA offers more out of the box than Oracle or DB2. | 0% | HANA FAQ |
HANA was built from the ground up. | 0% | HANA FAQ |
HANA was built from the p*Time and TREX acquisitions. | 100% | HANA FAQ |
HANA stores all data in memory. | 0% | HANA FAQ |
HANA stores all data in a columnar format. | 0% | HANA FAQ |
HANA can reduce the data footprint by 95% | 0% | HANA FAQ |
The real benefits to HANA come from when the logic from the applications are optimized and pushed down to the database level. | 0% | HANA FAQ |
The real benefits to HANA come from simplified applications. | 0% | HANA FAQ |
HANA supports all applications on the same instance of data at the same time. | 0% | HANA FAQ |
HANA perfectly serves the needs of all applications with one "system of record" instance. | 0% | HANA FAQ |
(adopted from McDermott) HANA is attached to everything SAP has | 0% | HANA FAQ |
HANA reduces TCO. | 0% | HANA FAQ |
HANA simplifies landscapes. | 0% | HANA FAQ |
HANA performance, advanced analytics (Predictive, Geospacial, Text analytics) and simplicity often mean a business process can be changed to be different compared to competitors. | 0% | HANA FAQ |
HANA provides risk mitigation | 0% | HANA FAQ |
A modern database should be a database and a platform. | 0% | HANA FAQ |
HANA is an entirely new database while databases from IBM and Oracle and Microsoft are "traditional" and therefore dated. | 0% | HANA FAQ |
Companies are looking to replace "traditional" databases. | 0% | HANA FAQ |
SAP tried to keep SAP licensing simple. | 0% | HANA FAQ |
Oracle requires 30 SKUs to equal HANA. | 0% | HANA FAQ |
HANA has more functionality than Oracle's entire software stack. | 0% | HANA FAQ |
HANA works out of the box for data warehouses. | 35% | HANA FAQ |
The fact that ByDesign was ported to HANA is evidence that this is a major factor for ByDesign. | 0% | HANA FAQ |
HANA was designed to replace DB2 and Oracle on SAP customers. | 100% | HANA FAQ |
It is feasible to scale HANA to very large sizes. customers. | 15% | HANA FAQ |
HANA stores data primarily in a column format with 99% of the tables for ERCP being column oriented. | 0% | HANA FAQ |
HANA is a good Big Data platform. | 0% | HANA FAQ |
HANA leads all database in HA. | 0% | HANA FAQ |
Appleby's explanation of his testing demonstrating that his test proved OLTP performance. | 0% | HANA Performance with OLTP |
HANA has simpler data structures. | 0% | HANA Performance with OLTP |
Simpler data structures lead to 5 to 10x faster performance than AnyDB. | 0% | HANA Performance with OLTP |
A test is sufficient to prove OLTP performance even if it was incorrectly setup. | 0% | HANA Performance with OLTP |
HANA is going to kill AnyDB | 0% | HANA Performance with OLTP |
SD does not accurately represent how customers use systems in 2014. | 0% | HANA Performance with OLTP |
The BW-EML benchmark should be discussed in conjunction with the SD benchmark. | 0% | HANA Performance with OLTP |
Endorsing Hasso's prediction that there would be large scale customer adoption of suite on HANA. | 0% | 2015 HANA Predictions |
HANA would be used for consolidated workloads. | 0% | 2015 HANA Predictions |
Transaction processing is 2x compared to competing DBs out of the box. | 0% | 2015 HANA Predictions |
HANA is the most comprehensive data platform. | 0% | 2015 HANA Predictions |
How customers use SD has changed making the old SD benchmark completely inapplicable. | 0% | BW-EML Benchmark |
The BW-EML benchmark should be considered a replacement for the previous SD benchmarks. | 0% | BW-EML Benchmark |
The BW-EML database should be discussed for HANA without consideration for other databases being compared to HANA. | 0% | BW-EML Benchmark |
Benchmarks become out of date as technology and business needs change, and this is why the BW-EML benchmark was created. | 0% | BW-EML Benchmark |
There is a clear move to OLATAP databases. | 0% | BW-EML Benchmark |
BW-EML is a cross vendor benchmark. | 0% | BW-EML Benchmark |
HANA is less expensive than Oracle | 0% | What Oracle Won't Tell You About HANA |
The inclusion of ASE means HANA is lower cost than Oracle. | 0% | What Oracle Won't Tell You About HANA |
Because SAP bundles various items with HANA, this means HANA is lower cost than Oracle. | 0% | What Oracle Won't Tell You About HANA |
SAP on Oracle has no future. | 0% | What Oracle Won't Tell You About HANA |
SAP will only be supported on Oracle until 2025 | 0% | What Oracle Won't Tell You About HANA |
HANA enables a new class of applications. | 0% | What Oracle Won't Tell You About HANA |
HANA SPS09 provided a ton of innovation. | 0% | What Oracle Won't Tell You About HANA |
In S/4HANA, SAP achieved a 70% data reduction. | 0% | What Oracle Won't Tell You About HANA |
Oracle locks you in to engineered systems. | 15% | What Oracle Won't Tell You About HANA |
SAP HANA is a cloud platform. | 0% | What Oracle Won't Tell You About HANA |
HANA is less expensive than Oracle. | 0% | 10 Customer Use Cases |
Server architecture would be able to run a typical large scale database in-memory. | 0% | 10 Customer Use Cases |
SAP is building a digital platform powered by HANA and the Cloud Platform. | 0% | 10 Customer Use Cases |
Conclusion
On average, for the statements measured, Appleby had a 6.43% accuracy to his statements around HANA.
Appleby had and continues to have a strong financial incentive to promote HANA. Individuals repeatedly say that just because they have a financial incentive to say something, does not mean that this financial bias impacts their statements and positions. We have been told that paid placements don’t impact media content. We have been told that consulting companies that staff for SAP, have no incentive to guide their clients to SAP solutions. And when we test that individual (or company) we find the opposite is true.
SAP customers continue to use sources of information with 100% SAP bias when making their SAP investment decisions, regardless of all the evidence of the outcomes of financial bias. This is why we consider SAP (and most other IT areas) to be entirely non-scientific.
Financial Disclosure
Financial Bias Disclosure
Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.
References
https://blogs.saphana.com/2013/11/13/comparing-sap-hana-to-ibm-db2-blu-and-oracle-12-in-memory-for-sap-bw/
https://news.ycombinator.com/item?id=15886333
That time Larry Ellison tried to have a professor fired for benchmarking Oracle from programming
https://en.wikipedia.org/wiki/David_DeWitt
“In essence, a DeWitt Clause forbids the publication of database benchmarks that the database vendor has not sanctioned. The original DeWitt Clause was established by Oracle at the behest of Larry Ellison. Ellison was displeased with a benchmark study done by David DeWitt in 1982, then an assistant professor, using his new Wisconsin Benchmark, which showed that Oracle’s system had poor performance.[6]” – Wikipedia
https://en.wikipedia.org/wiki/SAP_HANA
TCO Book
Enterprise Software TCO: Calculating and Using Total Cost of Ownership for Decision Making
Getting to the Detail of TCO
The Mechanics of TCO
- Understand why you need to look at TCO and not just ROI when making your purchasing decision.
- Discover how an application, which at first glance may seem inexpensive when compared to its competition, could end up being more costly in the long run.
- Gain an in-depth understanding of the cost, categories to include in an accurate and complete TCO analysis.
- Learn why ERP systems are not a significant investment, based on their TCO.
- Find out how to recognize and avoid superficial, incomplete or incorrect TCO analyses that could negatively impact your software purchase decision.
- Appreciate the importance and cost-effectiveness of a TCO audit.
- Learn how SCM Focus can provide you with unbiased and well-researched TCO analyses to assist you in your software selection.
- Chapter 1: Introduction
- Chapter 2: The Basics of TCO
- Chapter 3: The State of Enterprise TCO
- Chapter 4: ERP: The Multi-Billion Dollar TCO Analysis Failure
- Chapter 5: The TCO Method Used by Software Decisions
- Chapter 6: Using TCO for Better Decision Making