What This Article Covers
- Checking with Experienced Database Resources
- The Typical Coverage Available
We have been on one of the most prominent entities in researching and publishing on HANA. We have come to question virtually all of SAP’s proposals about HANA. Part of this has meant interpreting database benchmark studies, which has lead to investigating who does this type of testing. In this article, we will cover the topic of entities that verify the claims of database vendors.
Checking with Experienced Database Resources
Having conversations with multiple DB resources, that is people that have focused 100% of their career on databases for three decades, and the consensus is that there is not a single entity that verifies claims of database vendors, performs benchmarking, etc.. All benchmarking is performed by the vendors themselves. SAP has a single benchmark that they performed which is only one type of database processing. We analyzed this benchmark in the article What is the Actual Performance of HANA?
The Typical Coverage Available
Examples of entities that provide database coverage include DB-Engines, which tracks the popularity of databases.
Gartner which has a Magic Quadrant for databases but does not differentiate the database types in any way, as the following graphic indicates.
Gartner creates a fictitious category called ODMS operational database management systems as it is too lazy to analyze the different categories of databases.
It places Hadoop, which is a Big Data database in the same category as relational databases and in the same category with every other database type.
Gartner has no lab and does no testing and has very few people who even understand databases, much less touched a database as we covered in the article How Gartner Got HANA So Wrong.
Gartner places non-relational databases into a relational database magic quadrant and does not even differentiate the database in question from the vendor. Instead, they simply note the vendor on the Magic Quadrant; the database goes unmentioned.
The database category of software is filled with vendors making all manner of claims, but there is no entity which verifies any of these claims. This is a problem because it means that buyers in the database market have to perform their own testing.
This would mean gaining access to the databases in question and creating a laboratory environment including all the skill sets to do so. Very few companies do this.
Therefore, the ability to verify the claims made by the various database vendors is quite limited.
Financial Bias Disclosure
This article and no other article on the Brightwork website is paid for by a software vendor, including Oracle and SAP. Brightwork does offer competitive intelligence work to vendors as part of its business, but no published research or articles are written with any financial consideration. As part of Brightwork’s commitment to publishing independent, unbiased research, the company’s business model is driven by consulting services; no paid media placements are accepted.
The following is an interesting quote from Database Journal.
“One important concept to take away from this discussion is that there is no singular, all encompassing, definitive test that allows a vendor to claim their system is the best one out there, no ands, ifs or buts. For Oracle, Microsoft, IBM, or Sybase to claim they are the best overall, well, it’s simply not true. A particular system can be the best on a particular platform under certain conditions, but to say a particular system is the best overall is going to make that vendor suspect with respect to credibility.”