How Gartner Got HANA So Wrong

Executive Summary

  • Gartner made many false proposals about HANA, but without declaring how much money they take from SAP.
  • Forrester also provides pro-HANA information, in return for SAP’s money.

Introduction to Gartner Accuracy Verification

On January 11, 2013, ComputerWorld published the article Will Oracle’s Customer Base Migrate to SAP Business Suite. In this article, Gartner is quoted on the topic of HANA. In this article, you will find the accuracy of the Gartner article with the benefit of 4.5 years of hindsight.

Article Quotations

Biased Information Source Says What About HANA?

“SAP’s announcement that its Business Suite applications, which are deeply embedded in many global enterprises, will now be able to run on its in-memory database technology, HANA, leaves the IT community with a very interesting decision on its hands. SAP’s announcement that its Business Suite applications, which are deeply embedded in many global enterprises, will now be able to run on its in-memory database technology, HANA, leaves the IT community with a very interesting decision on its hands. SAP is the first vendor in the market to fully combine transactional applications with analytics in-memory and has two to five years, according to some analysts, to sell to customers before other vendors catch up.”

This is restated in the article from Readwrite.

Oracle and IBM are expected to match SAP in time. Microsoft has announced that it is working on the same technology for the next version of MS SQL. “I do believe that every other vendor is going to go in that direction, but it’s going to take them two to five years to do it, which gives SAP a huge head start,” Feinberg said.

This turned out to be false because ComputerWorld and Readwrite relied upon Gartner and Forrester. What Gartner did not know, and never investigated was how many areas HANA was behind all of the other database competitors. We cover one area, which is the fact that SAP had to add in row-oriented tables to run transaction processing as we cover in the article John Appleby on HANA Not Working Fast for OLTP.

This is perhaps because ComputerWorld also receives large amounts of money from SAP. At this point, Oracle 12c and IBM DB2 BLU have both exceeded HANA’s performance. This is covered in the article Which is Faster HANA or Oracle 12c?

Furthermore, there is an open question to how necessary it is to have RDBMSs with a column store — as this will always both increase the overhead of the database, and because applications tend to be focused on specific things, or specific types of processing. This topic is covered in our analysis in the article How Accurate with Bloor Research on Oracle In-Memory?

Notice that Gartner simply accepted the premise presented by SAP that their column orientation was the correct approach.

SAP Was First to Market?

“Computerworld UK spoke to Donald Feinberg, VP and distinguished analyst at Gartner, who explained the significance of the announcement for IT managers.

“Until now, this has been impossible to do. The structures that are needed in a disk-based database system are different for online analytical processing (OLAP) and online transactional processing (OLTP). However, what has changed now is that you have got a database in-memory where you can create OLAP structures virtually,” said Feinberg.

“Now SAP can set up a database to do my transactions and analytics with everything virtual. People have wanted to do both in the same database for years, but have not been able to do it because of the discrepancy between the two types of data.”

In the magazine ReadWrite further quoted Donald Fienberg in a separate article:

“It’s the only in-memory DBMS (database management system) that can do data warehousing and transactions in the same database. That’s where it’s unique.”

Natural Questions

A few questions come to the surface.

  • One being if that is actually a requirement driven by customers.
  • Secondly, even years later SAP customers do not perform analytics as described by Gartner.
  • This is because so few companies use HANA for their ERP system. This is covered in the article How SAP Controls Perceptions with Customer Numbers. The vast majority of HANA sales are for SAP BW. BW is a database warehouse. Therefore even with years of hindsight, the functionality that Gartner promoted does not have applicability. At least not yet. Therefore the impact of Feinberg’s statement has been very close to zero.

A second question is whether this is all that advantageous. Companies have been using data warehouses for many years and doing reporting off of them. It is preferable to perform some types of reporting in the application. For instance, ECC has many reports that are used. But more complex reports that draw results from many tables are most often pushed to the data warehouse. What SAP and Feinberg proposed is that the data warehouse and the ERP system should essentially unify. That might or might not be a good idea, but it is not where the IT industry is headed presently.

How HANA Lags

IBM had something very similar to HANA all the way back in 2008. Therefore, SAP was not the first. SAP was the first to heavily market this capability.

Since this article’s publication, HANA now lags other competitors because SAP’s design of HANA is inferior to the alternatives. This is demonstrated in the poor performance of HANA for anything but read performance on actual projects.

“I think this is going to be disruptive to other vendors, as I think it’s going to propel SAP from a database standpoint. It’s going to put them on the map – today they are number four in the market, and a small number four at that,” said Feinberg.

And SAP has not changed this position since this time.

HANA for Sales Forecasting?

For SAP customers, HANA-powered applications can speed up the sales process dramatically. For example, today when salespeople for a large manufacturer takes a large order from a customer, they may not be to say on the spot exactly when the order will be fulfilled. That information often comes hours later after the numbers are run separately through forecasting applications.

With HANA running SAP’s enterprise resource planning applications – called Business Suite – salespeople will be able to take the order and get forecasting information in seconds. “This changes the way they do business. It really does,” Feinberg said. “And that’s the kind of value proposition that HANA brings to the table because of the fact that it’s an in-memory database.” – Donald Fienberg

Considering the Process Outlined by Fienberg

This entire process as described by Fienberg is incorrect. Let us break it into individual parts.

  1. If a salesperson requires a data quote, that is part of order promising. Order promising is performed inside of either SAP SD (Now renamed to Order To Cash) or in GATP, which is the order promising module in the APO suite. But this does not require the creation of a forecast on the spot. The forecast is not generated in real time, and there is actually no reason for it to be. Forecasts actually should not be regenerated much more than monthly (in most cases) because it reduces forecast accuracy. This is covered in the article Monthly Versus Weekly Forecasting. A new order being generated (or in this case a quotation) does not change the forecast in any way. A sales order decrements the forecast. Forecasts are based upon previous sales orders, not based upon as sales order today. Therefore this explanation of how forecasting works is completely inaccurate.
  2. The information does not come hours later. Anyone who has worked with SAP in order management can attest that the order promise date comes back very quickly. It does this because it is simply checking on pre-calculated information.
  3. Fienberg states that with HANA the salesperson can get the information in seconds. Yet, they already get the information in seconds.
  4. Fienberg states that this changes the way the company does business, but it doesn’t because Donald Fienberg does not understand how the order promising system works in SAP.
  5. It does not the value proposition that HANA brings to the table. It has nothing to do with order promising as described by Fienberg.

SAP’s Adaptive Server?

“You now have SAP Adaptive Server Enterprise that runs underneath the applications, and they also have HANA. I suspect that you are going to see people moving from Oracle, IBM and Microsoft over to these. They just couldn’t do it before now,” he added.”

What does SAP’s Adaptive Server have to do with HANA? The question is about HANA, and Fienberg switches to SAP’s Adaptive Server which is row oriented database design. Is this article about standard RDMBSs? No, it is not.

Furthermore, SAP’s Adaptive Server, which is an old Sybase product is in a state of decline.

This shows the decline of the popularity of SAP Adaptive Server as tracked by DB Engines. 

Literally, what is Donald Fienberg talking about?

Fienberg’s Problematic Timeline

“Oracle today does not have a transactional, in-memory database. They have TimesTen, but none of the main enterprise application packages (SAP, Infor or Oracle’s e-business suite) run on this. I don’t believe SAP will ever certify an in-memory database for Oracle applications (when one is released) and so what you are asking is: will Oracle lose SAP customers that are running on Oracle databases? The answer is yes.”

Oracle 12c, which has transactional, in memory database (not the entire database as HANA, but parts of the database. Placing all of the database into memory is not something non-SAP vendors do because it does not make any sense. Therefore, Oracle had this capability 5 months after this article with Fienberg’s statement was made. Fienberg must have been in contact with Oracle and knew they were developing this. Why did Fienberg then propose that it would take HANA competitors “5 years to catch up?”

Now the Second Financially Biased Source is Used

“Duncan Jones, principal analyst at Forrester, agrees with Feinberg and believes that this announcement ‘sets SAP customers free from the clutches of the other database vendors’.

He said: “If it works, as SAP plans, it will be very damaging for Oracle. Oracle has some great products, but it does a lot of things to annoy its customers. There comes a point when you can only annoy your customer base so much, and if they are given an alternative that works, many of them will embrace it.”

Like Feinberg, Duncan Jones was also wrong. Forrester was paid a large amount of money by SAP to write a paper projecting a lower TCO for HANA. This exceedingly low-quality paper is covered in the article How Accurate Was Forrester’s TCO Study on SAP HANA?

Oracle’s Sales Tactics are Bad, But According to Forrester, SAP’s Aren’t?

“He said: “If it works, as SAP plans, it will be very damaging for Oracle. Oracle has some great products, but it does a lot of things to annoy its customers. There comes a point when you can only annoy your customer base so much, and if they are given an alternative that works, many of them will embrace it.

“Oracle overcharges, they are very aggressive in the sales process, they don’t necessarily negotiate, they have obsolete licensing rules, and they sell customers products they don’t necessarily need. If a CIO is desperate to save money, they might see SAP HANA as a way to do that.”

And all of these things apply to SAP as well. I constantly have customers reach out to me to

I constantly have customers and vendors reach out to me to describe in fine detail shady tactics used by SAP. I don’t focus on Oracle, so the same thing does not happen for Oracle. However, Forrester is as several hundred person company. It is simply infeasible to imagine that Forrester does not have similar insight into SAP’s underhanded business practices. Yet according to Forrester, the problematic business practices are with Oracle. Apparently, SAP gets a clean bill of health.

Gartner’s Feinberg Seems to Get Lost

“Gartner’s Feinberg generally agrees. He believes that for deeply embedded legacy applications, large enterprises will not consider the move – they will just wait for Oracle to come up with something similar. “Changing my DBMS platform underneath SAP is not that disruptive to the company, but changing my applications from Oracle to SAP, that’s expensive. I have to retrain everybody, I have to change all my data – it’s time consuming and disruptive.”

Right. That is true, but what is the relevance of this comment? The question of the article is whether companies will switch from Oracle to HANA.

“If I was a small company running on Oracle today and I grow and I need to replace my applications, that’s a risk for Oracle. If I have got a business case for making the transition from one set of applications to another, that will then mitigate the expense and the disruption,” he said.

This is another completely unrelated comment from Feinberg.

Gartner Distributes Their Bad Prediction Broadly

On January 10, 2013, ReadWrite published the article SAP’s HANA Deployment Leapfrogs Oracle, IBM and Microsoft.

Article Quotations

SAP has taken a big step ahead of rivals IBM, Microsoft and Oracle with the announcement on Thursday that its in-memory database called HANA is now ready to power the German software maker’s business applications. The pronouncement is sure to darken the mood of competitors, who one analyst says will need several years to match what SAP has accomplished.

What SAP has done is to provide one database that can perform both business analysis and transactions, something its rivals are able to provide only by using two databases, according to Gartner analyst Donald Feinberg. “It’s the only in-memory DBMS (database management system) that can do data warehousing and transactions in the same database. That’s where it’s unique.”

This was proposed repeatedly by Donald Feinberg at around this time. However, Donald’s forecasting was quite off, as the following graphic attests.

It is important to place the events in a timeline so that the prediction can be seen graphically. In this view, Fienberg’s prediction seems quite inaccurate. Furthermore, Fienberg greatly oversold the effect of the supposed advantage that SAP had with HANA. 

This statement in the IBM technical documentation for DB2 BLU is illustrative of what Brightwork has been saying is the primary benefit to what HANA does.

Columnar organization favors analytic queries that access a large number of values from a subset of the columns and make heavy use of aggregations and joins. Only the column data needed to process the query is loaded into memory from disk.

Taking SAP’s Word for It?

Donald Feinberg continues…

Those meetings are likely to go on for a while as Oracle and other vendors race to catch up.

Does ReadWrite actually check anything or do they simply allow themselves to be used as a propaganda apparatus by whomever they interview?

There are really so many problems with this statement. The first is that it is not actually established that what HANA does (which is an attempt to get both OLTP and OLTP from the same database) is all that needed. Other vendors have added the same capability, but the evidence is not in that this is all that necessary. But even if it were true, ReadWrite, taking Donald Fienberg’s/Gartner’s statements to mean that SAP was far ahead of other DB vendors on this capability, when they were not.

Gartner’s Conflicts of Interest

Donald Fienberg was both an analyst at Gartner while being a vendor relations manager with SAP, which is a conflict of interest.

Checking with Experienced Database Resources

Having conversations with multiple DB resources, that is people that have focused 100% of their career on databases for three decades, and the consensus is that there is not a single entity that verifies claims of database vendors, performs benchmarking, etc.. All benchmarking is performed by the vendors themselves. SAP has a single benchmark that they performed which is only one type of database processing. We analyzed this benchmark in the article What is the Actual Performance of HANA?

The Typical Coverage Available

Examples of entities that provide database coverage include DB-Engines, which tracks the popularity of databases.

Gartner which has a Magic Quadrant for databases but does not differentiate the database types in any way, as the following graphic indicates.

Gartner creates a fictitious category called ODMS operational database management systems as it is too lazy to analyze the different categories of databases.

It places Hadoop, which is a Big Data database in the same category as relational databases and in the same category with every other database type.

Gartner has no lab and does no testing and has very few people who even understand databases, much less touched a database as we covered in the article How Gartner Got HANA So Wrong.

Gartner places non-relational databases into a relational database magic quadrant and does not even differentiate the database in question from the vendor. Instead, they simply note the vendor on the Magic Quadrant; the database goes unmentioned.

Conclusion

Feinberg did not disclose his position as a relationship manager with SAP. His accuracy on when competitors would come up with something similar to HANA (which ended up being actually better) was off by years. His quotations at the end of the article are not related to the question in the article. He lacked the understanding of the order promising and the forecasting process in SAP to even explain how HANA would change it.

Overall, Fienberg and Gartner do not come across as independent entities. They come across as paid spokesman for SAP. In the articles where Fienberg is quoted, there is no mention that Gartner is not independent of SAP but receives probably somewhere around $100 million to $150 million from SAP annually. Secondly, Fienberg’s statements on HANA are undifferentiated from the SAP talking points. Therefore it looks like Feinberg was coached on what points to make by SAP, and that there was no processing of the information or attempt to validate whether it was true. These historical analyses are important because Gartner routinely face plants in their predictions, and they face plant the same way in each case, mispredicting due to the entities that pay them.

The database category of software is filled with vendors making all manner of claims, but there is no entity that verifies any of these claims. This is a problem because it means that buyers in the database market have to perform their own testing.

This would mean gaining access to the databases in question and creating a laboratory environment including all the skill sets to do so. Very few companies do this.

The Necessity of Fact Checking

We ask a question that anyone working in enterprise software should ask.

Should decisions be made based on sales information from 100% financially biased parties like consulting firms, IT analysts, and vendors to companies that do not specialize in fact-checking?

If the answer is “No,” then perhaps there should be a change to the present approach to IT decision making.

In a market where inaccurate information is commonplace, our conclusion from our research is that software project problems and failures correlate to a lack of fact checking of the claims made by vendors and consulting firms. If you are worried that you don’t have the real story from your current sources, we offer the solution.

Financial Disclosure

Financial Bias Disclosure

Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.

Search Our Other Who Was Right and Wrong on HANA Content

References

https://www.computerworlduk.com/it-vendors/will-oracles-customer-base-migrate-sap-business-suite-on-hana-3420138/

https://en.wikipedia.org/wiki/Oracle_Database

*https://readwrite.com/2013/01/10/saps-hana-deployment-leapfrogs-oracle-ibm-and-microsoft/

https://www.oracle.com/us/corporate/press/1967380%3Futm_source%3Drss%26utm_medium%3Drss%26utm_campaign%3Doracle-announces-general-availability-of-oracle-database-12c-the-first-database-designed-for-the

*https://readwrite.com/2013/01/10/saps-hana-deployment-leapfrogs-oracle-ibm-and-microsoft/

https://www.databasejournal.com/features/oracle/article.php/3462091/Database-Benchmarking.htm

The following is an interesting quote from Database Journal.

“One important concept to take away from this discussion is that there is no singular, all encompassing, definitive test that allows a vendor to claim their system is the best one out there, no ands, ifs or buts. For Oracle, Microsoft, IBM, or Sybase to claim they are the best overall, well, it’s simply not true. A particular system can be the best on a particular platform under certain conditions, but to say a particular system is the best overall is going to make that vendor suspect with respect to credibility.”

https://www.tpc.org/

https://www.quest.com/products/benchmark-factory/