- G2Crowd promised a new way to review software.
- However, G2Crowd has major disclosure issues that we cover.
Recently, during an interaction on LinkedIn where the head of CrowdReviews brought up the bias of Gartner in being a pay to play system. During that interaction, the topic of G2Crowd came up, which is another applications ranking entity that the head of CrowdReviews was also critical of. In my view at the time, G2Crowd was far more transparent that Gartner, and G2Crowd aggregated reviews from people that seemed to actually have application experience, unlike Gartner analysts. However, the discussion got me thinking about how transparent G2Crowd actually is, which brought up an investigation into G2Crowd and a test I performed on G2Crowd.
Inflated Application Scores
When investigating the G2Crowd site, the biggest thing that is not making sense for me is that G2Crowd’s scores for bad applications were just too high. That looks fishy to me. I know quite a few bad applications because I see them either fail at clients or simply add very little value. These happen to be applications that are most recommended by consulting companies because they maximize the profits for the consulting companies. SAP has a bunch of applications like this, and they are like a vacuum cleaner, hoovering up the budgets at IT departments with their long-term maintenance costs. But it is not only SAP. Tableau, which is an impressive application in many ways, does not implement anywhere close to its advertised level. Tableau has a lot to recommend it, but users will not be developing the self-generated reports that look anything like the beautiful reports that Tableau creates for demos or you can view on their website. And one of the problems with G2Crowd is that if most the people that write Tableau reviews are making money from their Tableau skills, you won’t get reality in the reviews. Do they know more about Tableau than almost anyone? Yes. Do they care about the benefits of Tableau?
Well, there is a big difference between what Tableau does for them (gives them a nice job) versus what it does for users.
Performing a Test
So I performed a test. I selected one of the worst applications I have ever tested. An application that adds incredibly little value, but which IT departments love spending money on. That is the SAP Business Warehouse or BW. So the first thing to test was whether SAP BW was rated low on G2Crowd. To my surprise, SAP BW was rated at 4 stars, with only one low review (2 stars) before I began this experiment.
Here is a tabulation of the reviews of SAP BW. There are four five-star reviews and five four-star reviews. How could such a weak application receive such high reviews? The one-star review is mine. Only one other review at G2Crowd is low.
Therefore the test I intended to perform was to see if G2Crowd would accept an accurate, but low review of SAP BW and see if G2Crowd would publish it. After submitting the review, it went into the review queue. And G2Crowd did publish my review around a day later.
Therefore G2Crowd passed the test of publishing the negative (but accurate) review. I was asked by another person on LinkedIn why I had even questioned why G2Crowd would not publish the review.
“Why should they censor? Their business model is expert review. As to your other question: Maybe there are other sides to BW that you do not look at? Everybody has a blind spot, I too (perhaps more than one in my case …)”
To me, this question is simply naive, and I responded with the following:
“That is simple, so that they don’t offend SAP, if SAP happens to be a funding entity. SAP funds all manner of entities such as Diginomica, Gartner to promote false information about SAP. But to Jeev Trika’s point, I have no idea if SAP even sponsors G2Crowd. As with Gartner, they do not explain who their funders are on their website, preferring to not have their readers know this information. The idea that BW is a good application, for anyone but people that make money from it, is simply untenable.
On repeated SAP projects I have found BW simply adds very little value. Reports take forever to develop, and it unsurprising because the BW Workbench is difficult to use and the overall design of the BW is extremely dated. I use a forecasting application to do exactly what the BW does for SAP DP, and I don’t have any of BW’s problems. This illustrates how dated BW is. So what you are proposing would not at all match with these observations, observations which I have collected over a decade. If BW cannot be called a bad application, then no application can be called a bad application.”
One might be tempted to say that G2Crowd has passed the test, and in a way they had. This sentiment would be echoed by G2Crowd’s press release of May 24, 2017.
“There is a paradigm shift occurring as B2B buyers increasingly behave like consumers, relying on authentic online reviews and social media to make purchase decisions and no other company is better suited to catalyze this market opportunity than G2 Crowd.”
I have no reason to believe that G2Crowd’s reviews are not authentic, and many have the person’s LinkedIn profile and show as verified. But that is not the end of the matter. Once again if bad applications can receive good ratings, then “authenticity” does not appear to be the magic elixir that it is presented to be. And secondly, a very large number of consultants in the particular applications are leaving reviews — and surprise surprise, those reviews are nearly always glowing. That alone tips the review average due to this bias. Obviously, the more consultants can pump up applications they work in, the more money they stand to make.
From the bigger picture, G2Crowd appears to have some type of bias. I say this because if poor quality applications get very good ratings, then the rating system is not effective. And the questions arise, why this application received not only good scores but in 4 cases a perfect score (5 stars). Again, this may be the bias of who decides to write reviews.
I found a similar overestimation in other poor quality applications on G2Crowd. I do not know the answer to that question, but I have a few ideas that I will discuss at the end of this article, and which leads to the next topic which is G2Crowd’s vendor funded business model.
G2Crowd’s Vendor Funded Business Model
The next topic that arose was G2Crowd’s funding. I noticed that G2Crowd has nowhere on their website where the list who their vendor funders are or how much they receive from each funding entity, or that is each software vendor. This should be a fairly obvious that this is something that G2Crowd should list. But G2Crowd does not seem to be interested in anyone knowing that they take money from vendors on their website. I would go as far as to state that G2Crowd is hiding this fact. So the question is why? And the answer is obvious.
Once again the person I was interacting with on LinkedIn stated.
“Re G2Crowd – while they are not required to publish investors there seems to be quite some info available, from them as well as from other sources.”
G2Crowd may not be required by law to declare their funders, but why is it considered acceptable for them not to be? Is this simply a svelt excusing of corruption? I could ask the question of why G2Crowd seems so little interested in being honest with its readers and would ask readers whether they would question the motivations of someone who manages the reviews of applications but then was not interested in disclosing that they receive income from the vendors who’s applications are rated.
The Obligation to Disclose
I should also point out that any company performing research is obligated to disclose its funding that may create a conflict of interest, and this has been well established in research for a great deal of time. Therefore, G2Crowd, like Gartner, cannot be considered an entity actually interested in research, as they do not follow the established rules of research.
I then looked up several of the links that were sent to me by the person I was corresponding with (and which I have copied in the reference section) and found the following quotations.
“The 2016 fiscal year saw more than 300 percent growth for the company as over 800 technology leaders, including Salesforce, Marketo and IBM have signed on to leverage G2 Crowd’s unique user generated content in their marketing programs.” – G2Crowd
So if we look at this, it is really intended to impress people, it is not intended to expose their funding for the users of G2Crowd. And of course, no specifics are mentioned. Three companies being mentioned without any other details is not real disclosure.
G2Crowd as a Marketing Extension of IBM
And the press release had another very interesting piece of information, something I don’t think the person to whom the quotation is attributed to knew the implication of what they were saying.
“G2 Crowd is an important partner in expanding online advocacy for IBM. The user review platform is giving us new ways to connect to our customers, hear their feedback about our products and services, and strengthen our culture of customer centricity.” – Kevin Eagan, Chief Digital Experience Officer, IBM
Kevin Eagan essentially gave up the game with that comment. IBM sees G2Crowd as an “advocate” for IBM. And advocate as a biased party. An advocate who is paid by a party is, of course, a super biased party.
Secondly, IBM is completely not a customer centric player. So this statement is rather ridiculous. IBM may want to become more customer-centric, but they are not now and they have not been in recent memory. But the part about seeing G2Crowd as a way to advocate for IBM applications is certainly true. But it also brings up a number of questions.
- Will IBM force feed reviews from its employees into G2Crowd in order to improve its ranking?
- Will G2Crowd effectively police this? Will they actually put resources to policing it?
- If G2Crowd does police this will it cause a problem with IBM’s funding?
These things are all quite relevant questions because IBM lies all the time. They lie at conferences. They lie on consulting engagements (I have worked with them and have seen them lie, and lie egregiously up front). They lie in their contracts (something covered in great detail in the excellent blog by I Cringley, one of the foremost experts on IBM. Especially at the higher levels in IBM it is virtually impossible to find honest people. In fact, they lied repeatedly to get to their positions. I have no doubt that if given the opportunity IBM will encourage its own consultants to write reviews on G2Crowd.
By G2Crowd hiding its vendor funding, it makes it more difficult for those outside the organization to catch the upward bias on IBM’s applications.
This non-disclosure is just becoming more and more obvious that these private entities just don’t disclose their income. Gartner offers their ombudsman, which I have analyzed in detail, and have found to be completely without merit. I have covered the ombudsman in these articles is The Gartner Ombudsman for Real? and Differentiating Between Gartner’s Ombudsman and Disclosure Requirements. But be it G2Crowd or Gartner or Forrester or other entities, the perspective is the same. The entity hides its funding sources, and if asked about funding, indicates that who is funding it is simply not important and would never impact what they write. Honestly, this is difficult to swallow. I have researched the effect of funding on media output and the conclusion is that funding does influence media output. I cover this in the article The History of Media Entities and Financial Bias.
If readers don’t expect more from rating entities, the rating entities are happy to continue hiding funding sources from their readers. It’s dishonest, but the people that own these entities generally don’t care. So unless they are pressured to change they won’t.
The Problem: Thinking that Gartner is Focused on What is True
Gartner is hired by companies who fundamentally don’t understand how Gartner functions. Gartner has virtually no first-hand experience in the technologies that they evaluate and get most of their information from speaking with executives at buyers or executives at vendors and consulting firms. Gartner is also not a research entity, they compare very poorly to real research entities once you dig into the details as we did in the article How Gartner’s Research Compares to Real Research Entities. Gartner serves to direct IT spending to the most expensive solutions as these are the companies that can afford to pay Gartner the most money. Gartner has enormously aggressive internal sales goals that place accuracy far below revenue growth in importance.
Being Part of the Solution: Fact-Checking Gartner
Gartner is quite expensive, and using them without independent verification ends up in a lot of money being wasted, both in terms of fees to Gartner and also in money spent on Gartner recommendations. We offer Gartner support services that make sure companies get a better value from Gartner, and keep companies from relying on advice where Gartner cannot substantiate the advice or where the advice is clearly in response to vendor pressure. This includes both covering how Gartner works, as well as reading and fact-checking the information they provide. Our clients don’t even let Gartner know that we are evaluating the information provided to them.
Financial Bias Disclosure
Neither this article nor any other article on the Brightwork website is paid for by a software vendor, including Oracle, SAP or their competitors. As part of our commitment to publishing independent, unbiased research; no paid media placements, commissions or incentives of any nature are allowed.
Search Our Other Research Bias Content
Gartner is the most influential IT analyst firm in the world. Their approval can make or break a vendor in an application category, or at the very least control their growth. Gartner has been behind most of the major IT trends for decades. However, many people read Gartner reports without understanding how Gartner works, how it comes to its information, its orientation, or even the details of the methods it uses for its analytical products. All of this and more is explained in this book.
Table of Contents
- Chapter 1: Introduction
- Chapter 2: An Overview of Gartner
- Chapter 3: How Gartner Makes Money
- Chapter 4: Comparing Gartner to the RAND Corporation, and Academic Research
- Chapter 5: The Magic Quadrant
- Chapter 6: Other Analytical Products Offered by Gartner
- Chapter 7: Gartner’s Future and Cloud Computing
- Chapter 8: Adjusting the Magic Quadrant
- Chapter 9: Is Gartner Worth the Investment?
- Chapter 10: Conclusion
- Appendix a: How to Use Independent Consultants for Software Selection
- Appendix b: What Does the History of Media Tell Us About This Topic
- Appendix c: Disclosure Statements and Code of Ethics