Disregarding Gartner’s Deeper Technology Insights and Predictions

Executive Summary

• Gartner has an abysmal history of technology prediction.
• Part of the reason for this accuracy is not Gartner’s primary concern.

Video Introduction: How Financial Bias Impacts Gartner’s Predictions

Text Introduction (Skip if You Watched the Video)

In their reports, Gartner will sometimes make technology projections or try to describe the technology. At Brightwork Research & Analysis, we have analyzed these projections and have found quite a few of these statements to be unreliable. Gartner shows no evidence of being predictive; there is no reason to pay attention to their predictions. Furthermore, Gartner’s technology prediction opinions have been parodied in numerous articles on the Internet by experts in that particular category of technology. You will learn about how Gartner makes predictions and how they are designed to be used in marketing collateral by software vendors and consulting firms.

Our References for This Article

If you want to see our references for this article and other related Brightwork articles, see this link.

Lack of Financial Bias Notice: The vast majority of content available on the Internet about Gartner is marketing fiddle-faddle published by vendors who republish reports they paid Gartner to publish, or Magic Quadrants they paid Gartner to score well. The IT industry is petrified of Gartner and only publishes complementary information about them. The article below is very different.

• First, it is published by a research entity.
• Second, no one paid for this article to be written, and it is not pretending to inform you while being rigged to sell you software or consulting services as a vendor or consulting firm that shares their ranking in some Gartner report. Unlike nearly every other article you will find from Google on this topic, it has had no input from any company's marketing or sales department. 

Is Gartner Focused on Forecast Accuracy?

A number of the technology predictions I have read seem to have been designed to create a splash rather than stand the test of time. These are called “nuggets” and are designed specifically to be picked up and repeated by vendors and consulting firms. The nugget is always positive in nature, and Gartner designs the nugget to be repeatable, not to be accurate.

Gartner’s Error in Forecasting Windows Phone Market Share

One that sticks out as ludicrous was Gartner’s prediction that the Windows Phone would become the second most popular smartphone platform between 2012 and 2015 and account for 19.5% market share. 2015 is not yet upon us, but the Windows Phone has only a 3.2% market share.

Gartner is well known to take a large amount of money from Microsoft and routinely publishes pro Microsoft and anti-Linux.Android predictions. Microsoft has a tremendous amount of money to spend with Gartner. Open source projects have very little and at Gartner money talks.

Gartner’s SaaS Prediction

Another example of Gartner’s inaccuracy in long-term technology forecasting is taken from their prediction on SaaS.

Up until now, the unique nature of the software market has meant that buyers had very little negotiating power after the initial purchase of a software license,’ Gartner Vice President William Snyder said in a research note. ‘We expect those dynamics to change considerably over the next 5 to 10 years, giving CIOs and software procurement officers more bargaining power while potentially reducing software vendor profit margins.

Gartner also predicts that a fourth of all new business software will be delivered by software as a service by 2011.

Software costs have not significantly declined as predicted by Gartner. Gartner also overshot their second prediction that a fourth of all software would be delivered as SaaS by 2011. As of 2012, software provided by SaaS represents only four percent of total software sales. Granted, SaaS software is less expensive than on-premises software, so the number of seats served is no doubt higher than its sales percentage. But it isn’t easy to see how the delivery of SaaS as a percentage of all enterprise software would be anywhere close to the twenty-five percent value of software delivery estimated by Gartner.

Was this a bad analysis on the part of Gartner? Based on the enormous advantages that SaaS has in terms of cost and time to deploy and the efficiency of central administration by the vendor, I believe Gartner’s projections were reasonable at the time. However, what many people who predicted a faster growth failed to consider were issues ranging from security concerns to the lower customizability of SaaS solutions and the perception of integration issues that prevented SaaS from taking off as initially expected.

Other IT Analysts and Prediction

Prediction is not only a problem for Gartner but most other IT analysts as well.

There are several examples of this, and the largest analyst firms predicted positive things for the following:

1. i2 Technologies TradeMatrix (which was not a product)
2. The development of online marketplaces generally.
3. That MDM applications were going to be significant factors in software projects.
4. Predicting that “ERP was dead” (in the late 1990s) and that ERP companies would have to move into new products or be left out in the cold.
5. SAP Solution Manager (which wasted a tremendous amount of the company’s time and is probably the worst content management system ever broadly sold, although which has now been repurposed as is discussed in this post)
6. SAP’s acquisition of Business Objects (even at this late date) is hard to see one beneficial thing brought by this merger). Business Objects was not integrated into the SAP Suite — BTW, the SAP Business Objects Explorer, while having some useful capabilities, is so buggy that most companies have stopped using it. Business Objects’ customer service has also declined, while the price has gone up. Furthermore, Business Objects are no longer seen as an application that is innovating, a common problem for acquisitions.
7. SAP PLM (which was never an actual “product” as much as it was merely life-cycle functionality distributed in different applications.)
8. SAP Netweaver (never actually a product, but rather a marketing construct as can read in this post)*

This list of analyst misses was put together very quickly from memory. There are many more, and we will add to the list as time passes and more of their errors come to us.

The Similarity Between Analysts and Political Lobbying

There are several problems with software analyst firms. First, they have a bias towards the larger software firms. Industry insiders know this, but the payment from vendors to analysts is mentioned nowhere on their websites and is not discussed by them generally. Analyst firms solicit vendors in a similar way that lobbyists explain the benefits of political lobbying. Lobbyists cannot openly declare that their money buys changes to legislation (Although it does. In fact, the return on investment is estimated to 22,000% {if you win and the legislation goes the lobbyists’ way}, which is higher than any other competitive investment.) Therefore lobbyists state that the only thing their money buys is “access.” However, as with analyst firm, the people that pay to lobby know what they are doing and know what they are getting for their money. However, analyst firms receive money from both directions, both from the suppliers (vendors) and the consumers, that is, companies that buy enterprise software.

The Ridiculous Excuse Offered by Analyst Firms for Clear Conflicts of Interest

If pressed on the issue, Gartner representatives will say they have an “ombudsmen” that mediates disputes with vendors and prevents their influence. Therefore, they have no bias. However, that does not show in the positions that they take. Gartner does an excellent job of appearing unbiased, and they employ good writers. The effect of reading them is very similar to reading the Economist. At first blush, the Economist seems centrist and reasonable. However, upon many years of review, it is clear that the Economists represent the large monied interests that read the periodical and advertise in it. These interests are against the common interests, but the Economist cannot and say they represent wealthy bankers and opposed workers because they would lose credibility, and fewer people would read their magazine, as the bulk of their readership are essentially “want-to-bees.” (Interestingly, the Economist’s coverage of the enterprise software market is similarly overly focused on the biggest firms) Likewise, if analyst firms only openly stated that they sell their opinions to the highest bidder, their opinions would not be worth very much.

Why Analysts Firms Errors Are Also Due to Information Gaps

Therefore, the analysts’ firms have a clear bias towards vendors that pay them the most money (and the major monopoly vendors – SAP, Oracle IBM, can afford to pay the most, so get the rosiest coverage). The second part of the problem is that when I review articles written by analysts in software that I work in and configure, it’s clear that they have not used the software that they are writing about. One cannot expect an analyst to have implemented or used every application they cover. However, I also do not think they are even reading the software manuals (a very time-consuming process but also necessary). Instead, they are working off of vendor presentations and marketing material.

The Problem with Gartner’s High-Level Coverage of Technology

One industry colleague once confided in me that when they worked for a significant vendor, an analyst expected to see all of that vendor’s supply chain products by spending three days at the company headquarters and meeting with different people. Analysts seem to overestimate their ability to understand applications from these superficial materials and presentations that they rely upon. Many analysts seem to think they can understand an application without getting into the details, and of course, they also have deadlines to meet. The scarcity of real product knowledge came across very apparently in a recent review of SAP APO. I was reading from an analyst who had been “future sold” by SAP was describing the functionality to me that did not work. I know this because I both tested it, and there are multiple OSS notes open on the functionality described.

The Importance of a Low Embarrassment Threshold

I have since realized that to be a successful analyst. It is essential to have a high embarrassment threshold. This is because you are required to write pollyanish and uncritical articles for the vendors that contribute the most cash to your employer. However, as the articles will be around for a while, you will necessarily be wrong. You have to not feel back about the incorrect projection. You have the enthusiasm to keep making projections. Although, perhaps not, I never see anyone bring up inaccurate projections of software firms or hold them up for criticism.

Forecast Accuracy and Bias

Interestingly, in the field of supply chain forecasting, the topic of bias is frequently discussed in academic papers (although rarely acted upon to help reduce). The issue of forecast bias of software analysts is entirely unaddressed. However, software analyst firms have a clear bias towards both new functionalities and large vendors. Therefore they are not good sources of information as to the real likelihood of success using various applications. They do have value in synthesizing the marketing messages of many vendors into one place or one article. However, that is not what they are supposedly selling. They are allegedly selling critically thought out analysis of true vendor capabilities.

How Gartner Overestimates the Dynamic Nature of Various Software Categories

Using the software areas in which I have many years of implementing as a sample (supply chain), Gartner describes this software’s business environment as much more innovative and dynamic than it is in reality. This is interesting because Gartner developed a popular analytical product called the Hype Cycle, built on the premise that new technologies have an initial “Peak of Inflated Expectations Phase,” often followed by a “Trough of Disillusionment.” This is an acknowledgment by Gartner that it is much easier for a software vendor to develop new innovative technology than it is for that technology to be implemented.

But the strange thing about this is that while Gartner is famous for discussing hype cycles, it also promotes hype cycles!

How Gartner Promotes Hype Cycles

However, while the Hype Cycle points this out (and is used by buyers to help them time their investments in software or other technology), I do not see it acknowledged in other Gartner writings.

Gartner itself seems to drive hype cycles, which they, in turn, criticize in their Hype Cycle analytical product. This technique is used frequently by consulting companies to make their clients feel insecure about their level of progress. The more things are seen to change, the more demand for consulting and research services to keep up with that change.

Gartner and the ERP Hype Cycle

One of the most significant hype cycles in the history of enterprise software was ERP systems. Gartner coined the term ERP, but taking the term MRP and removing the “M” and replacing it with an “E,” which stood for the enterprise. The ideas promoted by Gartner about ERP systems back in the 1980s were the following:

1. ERP systems would be the last system that any company would ever need. It would replace every single legacy system (all systems that were not ERP at that time were called legacy by Gartner).
2. Companies that implemented ERP would require minimal customization because so much of the standard functionality would match the customer requirements.
3. ERP systems would transform business because they would vastly reduce waste and improve standardization by following best practices built into ERP software.
4. Companies that invested in ERP systems would enjoy a substantial ROI.
5. Companies that did not invest in ERP systems would be left behind by their competitors who invested in ERP systems.

The exciting feature of these promises is that, in hindsight, none of them came true. ERP did become the largest category of enterprise software. Still, they became much more expensive than anticipated, required extensive customization, and turned out to be a far better deal for software vendors and consulting companies than for the companies that implemented them.

Now, virtually no one looks at their ERP system as an enabler for competitive advantage. The ERP system is simply a large aggregation of what is often mediocre functionality that seems to rise in TCO as time passes.

Why Gartner  Promotes Hype Cycles

The question is, why? Why does Gartner promote hype cycles?

• Lack of Reality Basis: One reason is that Gartner never actually implements software. Few people that work at Gartner have any experience with implementing software. Gartner’s lack of authentic knowledge about the software is evident from reading its media output.
• Business Model: Gartner wants to appear leading edge. And an excellent way to look leading edge is to support the newest things. Of course, the problem with this is that the latest items are often not ready to be implemented.
• Paid by Software Vendors: Gartner will promote trends that come to them from software vendors. Gartner likes to say that they lead the trend, but they report on the trend. This is one of the problems of roughly 1/3 funded (of overall revenues) by software vendors. It means that they bring across those things that vendors want to be promoted.
• Audience: Gartner’s audience is at the top of companies. This is also not a realistic group of people, by in large. Many are attempting to look at the leading edge as possible themselves. If you run into an IT executive at conferences, they are not very likely to tell you the real story of what they have accomplished within their companies. They want to make a good impression, and most are quite cognizant of their position and appearance. Therefore, they often have resumes filled with the newest technologies, whether they have been implemented successfully or not.

Conclusion

The fact of the matter is that technology prediction is a tricky business. However, Gartner is not correct enough the time for it to be taken seriously. Gartner has not demonstrated any particular aptitude for technology prediction. On the other hand, if your client base does not notice Gartner’s prediction track record, and if the bold predictions make for good marketing copy, then I suppose there is no reason to stop.