The Appleby Accuracy Checker: A Study into John Appleby’s Accuracy on HANA

Executive Summary

  • For years John Appleby provided information about HANA to the market.
  • This is a study that measure’s John Appleby’s accuracy.

Introduction

HANA was introduced in 2011, although little actually happened with HANA in 2011. In 2012 and particularly into 2013 and forward, John Appleby of Bluefin Solutions published large amounts of inaccurate HANA information. Appleby’s inaccuracies were so extensive that this study was created to catalog many (but certainly not all) of Appleby’s statements about HANA and to measure the average accuracy level of Appleby’s comments about HANA.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Notice of Lack of Financial Bias: We have no financial ties to SAP or any other entity mentioned in this article.

  • This is published by a research entity, not some lowbrow entity that is part of the SAP ecosystem. 
  • Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. As you are reading this article, consider how rare this is. The vast majority of information on the Internet on SAP is provided by SAP, which is filled with false claims and sleazy consulting companies and SAP consultants who will tell any lie for personal benefit. Furthermore, SAP pays off all IT analysts -- who have the same concern for accuracy as SAP. Not one of these entities will disclose their pro-SAP financial bias to their readers. 

The Method

This study is a document analysis of some of Appleby’s most popular and influential blog posts. That is one half of the study, while the other half is the reality of HANA. Brightwork Research & Analysis has performed the most extensive research on HANA, and this website has over 250 articles on HANA and S/4HANA (the ERP system mated to HANA). That is with the caveat of research outside of benchmarking. The entities with the benchmarking evidence around HANA are the database and hardware vendors. However, because of licenses and partnerships, these benchmarks cannot be published. In the one case where a benchmark was published, it was heavily redacted and rigged in favor of HANA, as we covered in the article The Problems with the Strange Lenovo HANA Benchmark.

Brightwork Research & Analysis is also one of the only information providing entities that cover SAP that is not financially tied to SAP (or to any other vendor) (for example, Forrester, Gartner, Nucleus Research, and all of the US-based IT media outlets are financially connected to SAP). This lack of financial relationship allows us to publish our analysis of accuracy without concern for retaliation on the part of the SAP.

Understanding the Retaliatory Measures Against Independent Research

The standard retaliatory measure being for the vendor to threaten to remove or reduce funding unless the offending analyst is fired, as Oracle requested against DeWitt when he published a benchmark that caused Oracle displeasure as Oracle was not ranked at the very top of the reference. Oracle went further in the DeWitt case by threatening the University of Wisconsin with no longer hiring its graduates. The position of firms like SAP and Oracle is that those that publish research that is not consistent with their marketing materials should be removed from performing research. And in the case of Oracle, they were intent on bringing this pressure against a major university.

The Method Used

The method was to fact check statements made in Appleby’s articles and rate each of the comments for accuracy. This resulted in an average accuracy.

What Statements Are Measured

Appleby is very well connected to SAP and has received a steady stream of information from SAP since he developed a relationship with SAP’s upper reaches. In his article 10 HANA Predictions for 2015, Appleby made “predictions,” but these were just things that development was working on at the time. We did not measure these statements for accuracy.

In some cases, Appleby made statements such as HANA worked with S/4HANA — which is true but would be like crediting someone for saying the “sun rises in the East.” Therefore, the study measured statements where something more than elementary and universally agreed-upon observations were made.

The Evidence for the Percentage Rating

  • What you see below in the Appleby Accuracy Checker Table. This table (which is searchable, see the search box at the top) is designed to list the measured statements.
  • The evidence that the statement’s accuracy percentage is found in the article link, which in each case links to a Brightwork Research & Analysis article where the statement is analyzed. Using a CTRL-F for the analysis article for the item you are looking for should bring you directly to the section you seek to find the supporting analysis.
  • Because Appleby would not have made claims that were not cleared by SAP, and because most of the claims were also made by SAP, this study can also be used as a rough approximation of SAP’s accuracy on HANA. Our Study into SAP’s Accuracy covers that topic separately.

Appleby Accuracy Checker Table

Checking Appleby's accuracy.
Appleby's StatementAccuracy % of the CommentLink to Analysis Article
SAP is building a digital platform powered by HANA and the Cloud Platform. 0%10 Customer Use Cases
Server architecture would be able to run a typical large scale database in-memory.0%10 Customer Use Cases
HANA is less expensive than Oracle.0%10 Customer Use Cases
SAP HANA is a cloud platform.0%What Oracle Won't Tell You About HANA
Oracle locks you in to engineered systems.15%What Oracle Won't Tell You About HANA
In S/4HANA, SAP achieved a 70% data reduction.0%What Oracle Won't Tell You About HANA
HANA SPS09 provided a ton of innovation.0%What Oracle Won't Tell You About HANA
HANA enables a new class of applications.0%What Oracle Won't Tell You About HANA
SAP will only be supported on Oracle until 20250%What Oracle Won't Tell You About HANA
SAP on Oracle has no future.0%What Oracle Won't Tell You About HANA
Because SAP bundles various items with HANA, this means HANA is lower cost than Oracle.0%What Oracle Won't Tell You About HANA
The inclusion of ASE means HANA is lower cost than Oracle.0%What Oracle Won't Tell You About HANA
HANA is less expensive than Oracle0%What Oracle Won't Tell You About HANA
BW-EML is a cross vendor benchmark.0%BW-EML Benchmark
There is a clear move to OLATAP databases.0%BW-EML Benchmark
Benchmarks become out of date as technology and business needs change, and this is why the BW-EML benchmark was created.0%BW-EML Benchmark
The BW-EML database should be discussed for HANA without consideration for other databases being compared to HANA.0%BW-EML Benchmark
The BW-EML benchmark should be considered a replacement for the previous SD benchmarks.0%BW-EML Benchmark
How customers use SD has changed making the old SD benchmark completely inapplicable.0%BW-EML Benchmark
HANA is the most comprehensive data platform.0%2015 HANA Predictions
Transaction processing is 2x compared to competing DBs out of the box.0%2015 HANA Predictions
HANA would be used for consolidated workloads.0%2015 HANA Predictions
Endorsing Hasso's prediction that there would be large scale customer adoption of suite on HANA.0%2015 HANA Predictions
The BW-EML benchmark should be discussed in conjunction with the SD benchmark.0%HANA Performance with OLTP
SD does not accurately represent how customers use systems in 2014.0%HANA Performance with OLTP
HANA is going to kill AnyDB0%HANA Performance with OLTP
A test is sufficient to prove OLTP performance even if it was incorrectly setup.0%HANA Performance with OLTP
Simpler data structures lead to 5 to 10x faster performance than AnyDB.0%HANA Performance with OLTP
HANA has simpler data structures.0%HANA Performance with OLTP
Appleby's explanation of his testing demonstrating that his test proved OLTP performance.0%HANA Performance with OLTP
HANA leads all database in HA.0%HANA FAQ
HANA is a good Big Data platform.0%HANA FAQ
HANA stores data primarily in a column format with 99% of the tables for ERCP being column oriented.0%HANA FAQ
It is feasible to scale HANA to very large sizes. customers.15%HANA FAQ
HANA was designed to replace DB2 and Oracle on SAP customers.100%HANA FAQ
The fact that ByDesign was ported to HANA is evidence that this is a major factor for ByDesign.0%HANA FAQ
HANA works out of the box for data warehouses.35%HANA FAQ
HANA has more functionality than Oracle's entire software stack.0%HANA FAQ
Oracle requires 30 SKUs to equal HANA.0%HANA FAQ
SAP tried to keep SAP licensing simple.0%HANA FAQ
Companies are looking to replace "traditional" databases.0%HANA FAQ
HANA is an entirely new database while databases from IBM and Oracle and Microsoft are "traditional" and therefore dated.0%HANA FAQ
A modern database should be a database and a platform.0%HANA FAQ
HANA provides risk mitigation0%HANA FAQ
HANA performance, advanced analytics (Predictive, Geospacial, Text analytics) and simplicity often mean a business process can be changed to be different compared to competitors.0%HANA FAQ
HANA simplifies landscapes.0%HANA FAQ
HANA reduces TCO.0%HANA FAQ
(adopted from McDermott) HANA is attached to everything SAP has0%HANA FAQ
HANA perfectly serves the needs of all applications with one "system of record" instance.0%HANA FAQ
HANA supports all applications on the same instance of data at the same time.0%HANA FAQ
The real benefits to HANA come from simplified applications.0%HANA FAQ
The real benefits to HANA come from when the logic from the applications are optimized and pushed down to the database level.0%HANA FAQ
HANA can reduce the data footprint by 95%0%HANA FAQ
HANA stores all data in a columnar format.0%HANA FAQ
HANA stores all data in memory.0%HANA FAQ
HANA was built from the p*Time and TREX acquisitions.100%HANA FAQ
HANA was built from the ground up.0%HANA FAQ
HANA offers more out of the box than Oracle or DB2.0%HANA FAQ
HANA is not just a database0%HANA FAQ
No other database vendor could have added a column store to their databases.0%HANA FAQ
HANA simplifies SAP's applications.0%HANA FAQ
HANA allows the renewal of existing applications like the SAP Business Suite.0%HANA FAQ
HANA allows the build of applications not possible on traditional RDBMSs.0%HANA FAQ
HANA is based upon 30 years of technology improvements.0%HANA FAQ
HANA is a reinvention of the database.0%HANA FAQ
HANA runs from 10 to 1000x faster than a regular database like Oracle.0%HANA FAQ
HANA is an application platform.0%HANA FAQ
HANA running slower that competing databases is entirely due to poor code.0%HANA Versus DB2
SAP apps run far faster on HANA.0%HANA Versus DB2
HANA can be made reasonable in price.0%HANA Versus DB2
Oracle and IBM need to catch up to HANA.20%HANA Versus DB2
HANA is enterprise ready (in 2014).10%HANA Versus DB2
Only SAP writes software that takes advantage of HANA's design.0%HANA Versus DB2
A 34 question form can only function properly if supported by HANA.0%HANA Versus DB2
HANA is a platform not just a database.0%HANA Versus DB2
HANA can solve problems that cannot be solved in DB2 or Oracle.0%HANA Versus DB2
John Appleby is unbiased because he does not work for SAP.0%HANA Versus DB2
SAP has to be sensitive to positioning HANA because of reseller agreements with IBM and Oracle0%HANA Versus DB2
Oracle produces FUD, but SAP only provides information.0%HANA Versus DB2
Microsoft cannot get traction in the SAP market, which is why SAP does not support MySQL or PostgreSQL.0%HANA SPS08
Microsoft cannot get traction in the SAP market, which is why SAP does not support MySQL or PostgreSQL.0%HANA SPS08
HANA was doing things no other database could do.0%HANA SPS08
HANA won't replace BW100%On HANA Replacing BW
BW is being implemented in the cloud.10%On HANA Replacing BW
HANA supports cold data.0%On HANA Replacing BW
SAP is still investing in BW.100%On HANA Replacing BW
BW takes care of SAP security and authorizations, complex hierarchies, non-cumulative data for stock and many other capabilities.100%On HANA Replacing BW
HANA does not provide a number of functions required for a complex enterprise data warehouse.100%On HANA Replacing BW
HANA is a data platform and application platform.0%On HANA Replacing BW
All duplicated data between ERP an the data warehouse is waste.0%On HANA Replacing BW
Because Appleby is not paid by SAP, he has no pro-SAP bias and no pro-HANA bias.0%On HANA Replacing BW
TCO is only the license and hosting cost.0%Appleby's TCO of HANA
HANA reduces the number of components required for development.0%A Historical View of HANA
Oracle, IBM and Microsoft have formidable sales and marketing teams to push their databases (but by inference) SAP does not.0%A Historical View of HANA
SAP has a club of customers that have seen 100,000 times performance improvement from implementing HANA.0%A Historical View of HANA
HANA, like the Model T, eliminates inefficient industries that surround databases.0%A Historical View of HANA
HANA was designed similarly to the Model T to reduce database maintenance.0%A Historical View of HANA
HANA was priced in a way to gain widespread adoption -- like the Model T.0%A Historical View of HANA
The Model T was not quickly adopted.0%A Historical View of HANA
HANA is SAP's fastest growing product ever.0%A Historical View of HANA
Microsoft's "in memory" database is incremental compared to HANA's.0%Appleby Beaten by Chris Eaton
HANA disrupted the database market.10%Appleby Beaten by Chris Eaton
IBM BLU is not well suited to OLTP workloads.0%Appleby Beaten by Chris Eaton
HANA would never need separate row and column stores for transactional data and reporting data.0%Appleby Beaten by Chris Eaton
Bluefin did DB2 substantial work.0%Appleby Beaten by Chris Eaton
SAP did not purposefully withhold information from customers that would have shown DB2's certification.0%Appleby Beaten by Chris Eaton
In 2013 HANA had 24 months of stability behind it.0%Appleby Beaten by Chris Eaton
HANA had constant innovations.0%Appleby Beaten by Chris Eaton
HANA only has two patches (of any kind) per year.0%Appleby Beaten by Chris Eaton
Maintenance is reduced with HANA0%Appleby Beaten by Chris Eaton
In Nov 2013 DB2 BLU did not support BW.0%Appleby Beaten by Chris Eaton
There is a 1 TB limit on DB2 BLU0%What In Memory DB for BW
DB2 BLU does not accelerate all of the queries that HANA does.0%What In Memory DB for BW
HANA simplifies the environment but installing multiple applications on a single HANA instance. 0%What In Memory DB for BW
HANA was the only database with full HA.50%What In Memory DB for BW
SP06 of HANA had all the capabilities of a mature database.0%What In Memory DB for BW
HANA runs on BW up to 100x faster.0%What In Memory DB for BW

Conclusion

On average, for the statements measured, Appleby had a 6.43% accuracy to his statements around HANA.

Appleby had and continues to have a strong financial incentive to promote HANA. Individuals repeatedly say that just because they have a financial incentive to say something does not mean that this financial bias impacts their statements and positions. We have been told that paid placements don’t impact media content. We have been told that consulting companies that staff for SAP has no incentive to guide their clients to SAP solutions. And when we test that individual (or company), we find the opposite is true.

SAP customers continue to use information sources with 100% SAP bias when making their SAP investment decisions, regardless of all the evidence of the outcomes of financial bias. This is why we consider SAP (and most other IT areas) to be entirely non-scientific.