The Appleby Accuracy Checker: A Study into John Appleby’s Accuracy on HANA

Executive Summary

  • For years John Appleby provided information about HANA to the market.
  • This is a study that measure’s John Appleby’s accuracy.

Introduction

HANA was introduced in 2011, although little actually happened with HANA in 2011. In 2012 and particularly into 2013 and forward, John Appleby of Bluefin Solutions published large amounts of inaccurate information about HANA. Appleby’s inaccuracies were so extensive, that this study was created in order to catalogue many (but certainly not all) of Appleby’s statements about HANA and to measure the typical accuracy level of Appleby’s statements about HANA.

The Method

This study is a document analysis of some of Appleby’s most popular and influential blog posts. That is one half of the study. The other half is the reality of HANA. Brightwork Research & Analysis has performed the most and the most extensive research of HANA and this website over 250 articles on HANA and S/4HANA (the ERP system mated to HANA). That is with the caveat of research outside of benchmarking. The entities with the benchmarking evidence around HANA are the database and hardware vendors. However, because of license and partnerships, these benchmarks cannot be published, and in the one case where a benchmark was published, it was heavily redacted and rigged in favour of HANA as we covered in the article The Problems with the Strange Lenovo HANA Benchmark.

Brightwork Research & Analysis is also one of the only information providing entities that cover SAP that is not financially tied to SAP (or to any other vendor) (for example, Forrester, Gartner, Nucleus Reseach and all of the US-based IT media outlets are financially connected to SAP). This lack of financial relationship allows us to publish our analysis of accuracy without concern for retaliation on the part of the SAP.

Understanding the Retaliatory Measures Against Independent Research

The standard retaliatory measure being for the vendor to threaten to remove or reduce funding unless the offending analyst is fired, as Oracle requested against DeWitt when he published a benchmark that caused Oracle displeasure as Oracle was not ranked at the very top of the benchmark. Oracle went further in the DeWitt case by threatening the University of Wisconsin with no longer hiring its graduates. The position of firms like SAP and Oracle is that those that publish research that is not consistent with their marketing materials should be removed from performing research. And in the case of Oracle, they were intent on bringing this pressure against a major university.

The Method Used

The method was to fact check statements made in Appleby’s articles and rate each of the statements for accuracy. This resulted in an average accuracy.

What Statements Are Measured

Appleby is very well connected to SAP and has received a steady stream of information from SAP since he developed a relationship with SAP’s upper reaches. In his article 10 HANA Predictions for 2015, Appleby made “predictions” but these were just things that development was working on at the time. We did not measure these statements for accuracy.

In some cases, Appleby made statements such as HANA worked with S/4HANA — which is true but would be like crediting someone for saying the “sun rises in the East.” Therefore, the study measured statements, where something more than elementary and universally agreed upon observations, were made.

The Evidence for the Percentage Rating

  • What you see below in the Appleby Accuracy Checker Table. This table (which is searchable, see the search box at the top) is designed to provide a listing of the measured statements.
  • The evidence that the statement’s accuracy percentage is found in the article link, which in each case links to a Brightwork Research & Analysis article where the statement is analyzed. Using a CTRL-F for the analysis article for the item you are looking for should bring you directly to the section you seek to find the supporting analysis.
  • Because Appleby would not have made claims that were not cleared by SAP, and because most of the claims were also made by SAP, this study can also be used as a rough approximation of SAP’s accuracy on HANA. Our Study into SAP’s Accuracy covers that topic separately.

Appleby Accuracy Checker Table

Appleby's StatementAccuracy % of the CommentLink to Analysis Article
HANA runs on BW up to 100x faster.0%What In Memory
DB for BW
SP06 of HANA had all the capabilities of a mature database.0%What In Memory
DB for BW
HANA was the only database with full HA.50%What In Memory
DB for BW
HANA simplifies the environment but installing multiple applications on a single HANA instance. 0%What In Memory
DB for BW
DB2 BLU does not accelerate all of the queries that HANA does.0%What In Memory
DB for BW
There is a 1 TB limit on DB2 BLU0%What In Memory
DB for BW
In Nov 2013 DB2 BLU did not support BW.0%Appleby Beaten by Chris Eaton
Maintenance is reduced with HANA0%Appleby Beaten by Chris Eaton
HANA only has two patches (of any kind) per year.0%Appleby Beaten by Chris Eaton
HANA had constant innovations.0%Appleby Beaten by Chris Eaton
In 2013 HANA had 24 months of stability behind it.0%Appleby Beaten by Chris Eaton
SAP did not purposefully withhold information from customers that would have shown DB2's certification.0%Appleby Beaten by Chris Eaton
Bluefin did DB2 substantial work.0%Appleby Beaten by Chris Eaton
HANA would never need separate row and column stores for transactional data and reporting data.0%Appleby Beaten by Chris Eaton
IBM BLU is not well suited to OLTP workloads.0%Appleby Beaten by Chris Eaton
HANA disrupted the database market.10%Appleby Beaten by Chris Eaton
Microsoft's "in memory" database is incremental compared to HANA's.0%Appleby Beaten by Chris Eaton
HANA is SAP's fastest growing product ever.0%A Historical View of HANA
The Model T was not quickly adopted.0%A Historical View of HANA
HANA was priced in a way to gain widespread adoption -- like the Model T.0%A Historical View of HANA
HANA was designed similarly to the Model T to reduce database maintenance.0%A Historical View of HANA
HANA, like the Model T, eliminates inefficient industries that surround databases.0%A Historical View of HANA
SAP has a club of customers that have seen 100,000 times performance improvement from implementing HANA.0%A Historical View of HANA
Oracle, IBM and Microsoft have formidable sales and marketing teams to push their databases (but by inference) SAP does not.0%A Historical View of HANA
HANA reduces the number of components required for development.0%A Historical View of HANA
TCO is only the license and hosting cost.0%Appleby's TCO of HANA
Because Appleby is not paid by SAP, he has no pro-SAP bias and no pro-HANA bias.0%On HANA Replacing BW
All duplicated data between ERP an the data warehouse is waste.0%On HANA Replacing BW
HANA is a data platform and application platform.0%On HANA Replacing BW
HANA does not provide a number of functions required for a complex enterprise data warehouse.100%On HANA Replacing BW
BW takes care of SAP security and authorizations, complex hierarchies, non-cumulative data for stock and many other capabilities.100%On HANA Replacing BW
SAP is still investing in BW.100%On HANA Replacing BW
HANA supports cold data.0%On HANA Replacing BW
BW is being implemented in the cloud.10%On HANA Replacing BW
HANA won't replace BW100%On HANA Replacing BW
HANA was doing things no other database could do.0%HANA SPS08
Microsoft cannot get traction in the SAP market, which is why SAP does not support MySQL or PostgreSQL.0%HANA SPS08
Microsoft cannot get traction in the SAP market, which is why SAP does not support MySQL or PostgreSQL.0%HANA SPS08
Oracle produces FUD, but SAP only provides information.0%HANA Versus DB2
SAP has to be sensitive to positioning HANA because of reseller agreements with IBM and Oracle0%HANA Versus DB2
John Appleby is unbiased because he does not work for SAP.0%HANA Versus DB2
HANA can solve problems that cannot be solved in DB2 or Oracle.0%HANA Versus DB2
HANA is a platform not just a database.0%HANA Versus DB2
A 34 question form can only function properly if supported by HANA.0%HANA Versus DB2
Only SAP writes software that takes advantage of HANA's design.0%HANA Versus DB2
HANA is enterprise ready (in 2014).10%HANA Versus DB2
Oracle and IBM need to catch up to HANA.20%HANA Versus DB2
HANA can be made reasonable in price.0%HANA Versus DB2
SAP apps run far faster on HANA.0%HANA Versus DB2
HANA running slower that competing databases is entirely due to poor code.0%HANA Versus DB2
HANA is an application platform.0%HANA FAQ
HANA runs from 10 to 1000x faster than a regular database like Oracle.0%HANA FAQ
HANA is a reinvention of the database.0%HANA FAQ
HANA is based upon 30 years of technology improvements.0%HANA FAQ
HANA allows the build of applications not possible on traditional RDBMSs.0%HANA FAQ
HANA allows the renewal of existing applications like the SAP Business Suite.0%HANA FAQ
HANA simplifies SAP's applications.0%HANA FAQ
No other database vendor could have added a column store to their databases.0%HANA FAQ
HANA is not just a database0%HANA FAQ
HANA offers more out of the box than Oracle or DB2.0%HANA FAQ
HANA was built from the ground up.0%HANA FAQ
HANA was built from the p*Time and TREX acquisitions.100%HANA FAQ
HANA stores all data in memory.0%HANA FAQ
HANA stores all data in a columnar format.0%HANA FAQ
HANA can reduce the data footprint by 95%0%HANA FAQ
The real benefits to HANA come from when the logic from the applications are optimized and pushed down to the database level.0%HANA FAQ
The real benefits to HANA come from simplified applications.0%HANA FAQ
HANA supports all applications on the same instance of data at the same time.0%HANA FAQ
HANA perfectly serves the needs of all applications with one "system of record" instance.0%HANA FAQ
(adopted from McDermott) HANA is attached to everything SAP has0%HANA FAQ
HANA reduces TCO.0%HANA FAQ
HANA simplifies landscapes.0%HANA FAQ
HANA performance, advanced analytics (Predictive, Geospacial, Text analytics) and simplicity often mean a business process can be changed to be different compared to competitors.0%HANA FAQ
HANA provides risk mitigation0%HANA FAQ
A modern database should be a database and a platform.0%HANA FAQ
HANA is an entirely new database while databases from IBM and Oracle and Microsoft are "traditional" and therefore dated.0%HANA FAQ
Companies are looking to replace "traditional" databases.0%HANA FAQ
SAP tried to keep SAP licensing simple.0%HANA FAQ
Oracle requires 30 SKUs to equal HANA.0%HANA FAQ
HANA has more functionality than Oracle's entire software stack.0%HANA FAQ
HANA works out of the box for data warehouses.35%HANA FAQ
The fact that ByDesign was ported to HANA is evidence that this is a major factor for ByDesign.0%HANA FAQ
HANA was designed to replace DB2 and Oracle on SAP customers.100%HANA FAQ
It is feasible to scale HANA to very large sizes. customers.15%HANA FAQ
HANA stores data primarily in a column format with 99% of the tables for ERCP being column oriented.0%HANA FAQ
HANA is a good Big Data platform.0%HANA FAQ
HANA leads all database in HA.0%HANA FAQ
Appleby's explanation of his testing demonstrating that his test proved OLTP performance.0%HANA Performance with OLTP
HANA has simpler data structures.0%HANA Performance with OLTP
Simpler data structures lead to 5 to 10x faster performance than AnyDB.0%HANA Performance with OLTP
A test is sufficient to prove OLTP performance even if it was incorrectly setup.0%HANA Performance with OLTP
HANA is going to kill AnyDB0%HANA Performance with OLTP
SD does not accurately represent how customers use systems in 2014.0%HANA Performance with OLTP
The BW-EML benchmark should be discussed in conjunction with the SD benchmark.0%HANA Performance with OLTP
Endorsing Hasso's prediction that there would be large scale customer adoption of suite on HANA.0%2015 HANA Predictions
HANA would be used for consolidated workloads.0%2015 HANA Predictions
Transaction processing is 2x compared to competing DBs out of the box.0%2015 HANA Predictions
HANA is the most comprehensive data platform.0%2015 HANA Predictions
How customers use SD has changed making the old SD benchmark completely inapplicable.0%BW-EML Benchmark
The BW-EML benchmark should be considered a replacement for the previous SD benchmarks.0%BW-EML Benchmark
The BW-EML database should be discussed for HANA without consideration for other databases being compared to HANA.0%BW-EML Benchmark
Benchmarks become out of date as technology and business needs change, and this is why the BW-EML benchmark was created.0%BW-EML Benchmark
There is a clear move to OLATAP databases.0%BW-EML Benchmark
BW-EML is a cross vendor benchmark.0%BW-EML Benchmark
HANA is less expensive than Oracle0%What Oracle Won't Tell You About HANA
The inclusion of ASE means HANA is lower cost than Oracle.0%What Oracle Won't Tell You About HANA
Because SAP bundles various items with HANA, this means HANA is lower cost than Oracle.0%What Oracle Won't Tell You About HANA
SAP on Oracle has no future.0%What Oracle Won't Tell You About HANA
SAP will only be supported on Oracle until 20250%What Oracle Won't Tell You About HANA
HANA enables a new class of applications.0%What Oracle Won't Tell You About HANA
HANA SPS09 provided a ton of innovation.0%What Oracle Won't Tell You About HANA
In S/4HANA, SAP achieved a 70% data reduction.0%What Oracle Won't Tell You About HANA
Oracle locks you in to engineered systems.15%What Oracle Won't Tell You About HANA
SAP HANA is a cloud platform.0%What Oracle Won't Tell You About HANA
HANA is less expensive than Oracle.0%10 Customer Use Cases
Server architecture would be able to run a typical large scale database in-memory.0%10 Customer Use Cases
SAP is building a digital platform powered by HANA and the Cloud Platform.
0%10 Customer Use Cases

Conclusion

On average, for the statements measured, Appleby had a 6.43% accuracy to his statements around HANA.

Appleby had and continues to have a strong financial incentive to promote HANA. Individuals repeatedly say that just because they have a financial incentive to say something, does not mean that this financial bias impacts their statements and positions. We have been told that paid placements don’t impact media content. We have been told that consulting companies that staff for SAP, have no incentive to guide their clients to SAP solutions. And when we test that individual (or company) we find the opposite is true.

SAP customers continue to use sources of information with 100% SAP bias when making their SAP investment decisions, regardless of all the evidence of the outcomes of financial bias. This is why we consider SAP (and most other IT areas) to be entirely non-scientific.

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

References

https://blogs.saphana.com/2013/11/13/comparing-sap-hana-to-ibm-db2-blu-and-oracle-12-in-memory-for-sap-bw/

https://news.ycombinator.com/item?id=15886333

That time Larry Ellison tried to have a professor fired for benchmarking Oracle from programming

https://en.wikipedia.org/wiki/David_DeWitt

“In essence, a DeWitt Clause forbids the publication of database benchmarks that the database vendor has not sanctioned. The original DeWitt Clause was established by Oracle at the behest of Larry Ellison. Ellison was displeased with a benchmark study done by David DeWitt in 1982, then an assistant professor, using his new Wisconsin Benchmark, which showed that Oracle’s system had poor performance.[6]– Wikipedia

https://en.wikipedia.org/wiki/SAP_HANA

TCO Book

 

TCO3

Enterprise Software TCO: Calculating and Using Total Cost of Ownership for Decision Making

Getting to the Detail of TCO

One aspect of making a software purchasing decision is to compare the Total Cost of Ownership, or TCO, of the applications under consideration: what will the software cost you over its lifespan? But most companies don’t understand what dollar amounts to include in the TCO analysis or where to source these figures, or, if using TCO studies produced by consulting and IT analyst firms, how the TCO amounts were calculated and how to compare TCO across applications.

The Mechanics of TCO

Not only will this book help you appreciate the mechanics of TCO, but you will also gain insight as to the importance of TCO and understand how to strip away the biases and outside influences to make a real TCO comparison between applications.
By reading this book you will:
  • Understand why you need to look at TCO and not just ROI when making your purchasing decision.
  • Discover how an application, which at first glance may seem inexpensive when compared to its competition, could end up being more costly in the long run.
  • Gain an in-depth understanding of the cost, categories to include in an accurate and complete TCO analysis.
  • Learn why ERP systems are not a significant investment, based on their TCO.
  • Find out how to recognize and avoid superficial, incomplete or incorrect TCO analyses that could negatively impact your software purchase decision.
  • Appreciate the importance and cost-effectiveness of a TCO audit.
  • Learn how SCM Focus can provide you with unbiased and well-researched TCO analyses to assist you in your software selection.
Chapters
  • Chapter 1:  Introduction
  • Chapter 2:  The Basics of TCO
  • Chapter 3:  The State of Enterprise TCO
  • Chapter 4:  ERP: The Multi-Billion Dollar TCO Analysis Failure
  • Chapter 5:  The TCO Method Used by Software Decisions
  • Chapter 6:  Using TCO for Better Decision Making