The Appleby Accuracy Checker: A Study into John Appleby’s Accuracy on HANA

Executive Summary

• For years John Appleby provided information about HANA to the market.
• This is a study that measure’s John Appleby’s accuracy.

Introduction

HANA was introduced in 2011, although little actually happened with HANA in 2011. In 2012 and particularly into 2013 and forward, John Appleby of Bluefin Solutions published large amounts of inaccurate information about HANA. Appleby’s inaccuracies were so extensive that this study was created to catalogue many (but certainly not all) of Appleby’s statements about HANA and to measure the average accuracy level of Appleby’s comments about HANA.

The Method

This study is a document analysis of some of Appleby’s most popular and influential blog posts. That is one half of the study while the other half is the reality of HANA. Brightwork Research & Analysis has performed the most and the most extensive research of HANA and this website over 250 articles on HANA and S/4HANA (the ERP system mated to HANA). That is with the caveat of research outside of benchmarking. The entities with the benchmarking evidence around HANA are the database and hardware vendors. However, because of license and partnerships, these benchmarks cannot be published. In the one case where a benchmark was published, it was heavily redacted and rigged in favour of HANA, as we covered in the article The Problems with the Strange Lenovo HANA Benchmark.

Brightwork Research & Analysis is also one of the only information providing entities that cover SAP that is not financially tied to SAP (or to any other vendor) (for example, Forrester, Gartner, Nucleus Research, and all of the US-based IT media outlets are financially connected to SAP). This lack of financial relationship allows us to publish our analysis of accuracy without concern for retaliation on the part of the SAP.

Understanding the Retaliatory Measures Against Independent Research

The standard retaliatory measure being for the vendor to threaten to remove or reduce funding unless the offending analyst is fired, as Oracle requested against DeWitt when he published a benchmark that caused Oracle displeasure as Oracle was not ranked at the very top of the reference. Oracle went further in the DeWitt case by threatening the University of Wisconsin with no longer hiring its graduates. The position of firms like SAP and Oracle is that those that publish research that is not consistent with their marketing materials should be removed from performing research. And in the case of Oracle, they were intent on bringing this pressure against a major university.

The Method Used

The method was to fact check statements made in Appleby’s articles and rate each of the comments for accuracy. This resulted in an average accuracy.

What Statements Are Measured

Appleby is very well connected to SAP and has received a steady stream of information from SAP since he developed a relationship with SAP’s upper reaches. In his article 10 HANA Predictions for 2015, Appleby made “predictions,” but these were just things that development was working on at the time. We did not measure these statements for accuracy.

In some cases, Appleby made statements such as HANA worked with S/4HANA — which is true but would be like crediting someone for saying the “sun rises in the East.” Therefore, the study measured statements, where something more than elementary and universally agreed upon observations, were made.

The Evidence for the Percentage Rating

• What you see below in the Appleby Accuracy Checker Table. This table (which is searchable, see the search box at the top) is designed to provide a listing of the measured statements.
• The evidence that the statement’s accuracy percentage is found in the article link, which in each case links to a Brightwork Research & Analysis article where the statement is analyzed. Using a CTRL-F for the analysis article for the item you are looking for should bring you directly to the section you seek to find the supporting analysis.
• Because Appleby would not have made claims that were not cleared by SAP, and because most of the claims were also made by SAP, this study can also be used as a rough approximation of SAP’s accuracy on HANA. Our Study into SAP’s Accuracy covers that topic separately.

Appleby Accuracy Checker Table

Advice on Enjoying the Quiz

To see the full screen just select the lower right-hand corner and expand. Trust us, expanding makes the experience a whole lot more fun.

Conclusion

On average, for the statements measured, Appleby had a 6.43% accuracy to his statements around HANA.

Appleby had and continues to have a strong financial incentive to promote HANA. Individuals repeatedly say that just because they have a financial incentive to say something, does not mean that this financial bias impacts their statements and positions. We have been told that paid placements don’t impact media content. We have been told that consulting companies that staff for SAP has no incentive to guide their clients to SAP solutions. And when we test that individual (or company), we find the opposite is true.

SAP customers continue to use sources of information with 100% SAP bias when making their SAP investment decisions, regardless of all the evidence of the outcomes of financial bias. This is why we consider SAP (and most other IT areas) to be entirely non-scientific.

References

https://blogs.saphana.com/2013/11/13/comparing-sap-hana-to-ibm-db2-blu-and-oracle-12-in-memory-for-sap-bw/

https://news.ycombinator.com/item?id=15886333

That time Larry Ellison tried to have a professor fired for benchmarking Oracle from programming

https://en.wikipedia.org/wiki/David_DeWitt

“In essence, a DeWitt Clause forbids the publication of database benchmarks that the database vendor has not sanctioned. The original DeWitt Clause was established by Oracle at the behest of Larry Ellison. Ellison was displeased with a benchmark study done by David DeWitt in 1982, then an assistant professor, using his new Wisconsin Benchmark, which showed that Oracle’s system had poor performance.^[6]” – Wikipedia

https://en.wikipedia.org/wiki/SAP_HANA