Explanation of the Brightwork Cross Peer Evaluation for Forecasting Vendors

Executive Summary

  • This article explains the Cross Peer Evaluation to vendors.
  • This gives vendors an alternative to Gartner.

Introduction

The IT analyst space has a major problem. While there are several major IT analysts, for software selection a single company, Gartner is dominant. Gartner has created a monopoly through more than 60 acquisitions and forces vendors to use them as shortlists are based very heavily on the Gartner MQ. This is the case even though the MQs are not auditable and are strongly rigged towards the largest vendors. Vendors constantly reach out to Brightwork and complain that they have no choice but to participate and fund Gartner. Vendors support this corrupt model because they feel they have to “play the game” and see no way out, as one vendor wrote about Gartner.

The opacity of Gartner is a big methodological problem. As long as the meta-ranking recipe is public, it’s fine. Consultants should be able to re-process the data of a given report according to a recipe of their own (assuming they are willing to re-enter manually the data).

Thus far, no IT analyst or other entity has come forward with a non-corrupt way of providing a software selection research product. The CPE is an attempt to do just this.

The Problem Keeping the Status Quo

Vendors should consider the negative implications of constantly having to function in a market where Gartner controls who is shortlisted. Many vendors do not realize how much they are supporting the current system. Vendors know that the MQs do little to help customers select the right application for their needs and that buyers are continually guided to inappropriate applications. They also know how easy it is for unscrupulous vendors to trick Gartner, having Gartner esteem trendy or hot topic technologies that are irrelevant or of little relevance to the task at hand in each software category. This forces vendors to follow these pathways of development that area dead-end or of marginal benefit, not because of any evidence that they are necessary or beneficial and therefore a good use of development effort, but because the marketing department of the least scrupulous vendor was able to convince Gartner that it was true.

Observing the Change in Vendor MQ Rankings Over Time

There is often a great emphasis on the latest Gartner MQ. However, there is very little focus on the MQs of previous years or comparing how a specific MQ changes over time. Every time I perform comparison across years of a different MQ, it leads to very interesting results.

Let us begin by reviewing the 2018 MQ for Supply Chain Planning and then move forward in time to the 2021 MQ.

The 2018 MQ for Supply Chain Planning

A few things come across in this MQ. One is how few vendors there are in the MQ. A major method problem with Gartner’s MQs is that they do not reflect the field of vendors in any category. In fact, they don’t even reflect all of the vendors from the previous years in most cases.

Why is this?

Well, Gartner must be able to use some stick against vendors to keep them paying Gartner. And a primary stick is to exclude non-paying vendors from the MQ. This is great for Gartner’s revenues but very bad for buyers, who Gartner tells to select from among vendors that are willing to pay Gartner’s toll.

However, getting in the MQ is not the goal of vendors. Vendors need the Leader designation or at least a high Visionary or Challenger ranking. This is because the MQ is frequently used to shortlist vendors for selection. Gartner will often keep disaffected vendors in the game by promoting the idea that they can do better next year’s MQ. Particularly if they invest more in “analyst services.” Gartner needs to keep doing this, or the more low-ranked vendors will drop out, and Gartner needs that revenue to meet its revenue targets, which are quite aggressive. Gartner will always have disaffected vendors, but they need to keep them around and keep them paying and have enough vendors to have not just a few vendors in any one MQ. This type of continual future selling is a common feature that Gartner uses. And Gartner uses it to impressive effect on both software buyers and vendors. It is a sales strategy they use, which is constantly overpromising what Gartner can do for vendors. This is covered in the article How IT Analyst Firms like Gartner and Forrester Overpomise Marketing Exposure to Vendors.

Before we get to a comparison, it is important to note that there is no reason for a buyer to restrict their search to even the 18 or so vendors in the MQ, much less just those in the Leader quadrant.

Here is why.

In just the forecasting category (which has fewer vendors than the overall SCP category), we have over 50 vendors that we think qualify as providing sufficient forecasting functionality levels. We do not want to have “holes” in our coverage. However, not all vendors will choose to participate. But any holes that do exist will be based upon the vendor choosing not to participate. Not because the vendor is a valid forecasting vendor, but they are excluded due to an unwillingness to contribute financially.

However, Gartner has holes in their coverage as part of their method. Again, Gartner must retain the ability to punish vendors with exclusion from the MQ to keep the money train from vendors continually coming. The only reason that SAP has such a truncated list is that Gartner is showing its clients those vendors that could meet Gartner’s price. What does this have to do with what is the right software for the software buyer to purchase?

Nothing is the answer. These two things have nothing to do with each other. This is why claims that Gartner is not “pay to play” are odd. First, why are so many vendors missing from every category? And second, why do vendors disappear from MQs from year to year? There are many questions that all have the same answer. Gartner is profit-maximizing its MQs.

Now let us move to the 2021 MQ because we can learn some very interesting things by comparing the MQ across years.

The 2021 MQ for Supply Chain Planning

Keeping the changes from 2018 and 2021 is greatly enhanced through the following table. 

Gartner SCP MQ Comparison 2018 vs 2021

NumberVendorQuadrant Ranking 2018Quadrant Ranking 2021Change Quadrant Or Out or In MQ from 2018 to 2021?
1Blue YonderLeaderLeader0
2KinaxisLeaderLeader0
3E2OpenVisionaryLeader1
4LogilityLeaderLeader0
5OMPLeaderLeader0
6SAPLeaderChallenger1
7OracleLeaderChallenger1
8Dassault Systemes Challenger1
9Demand SolutionsLeaderChallenger1
10Infor Challenger1
11GAINSystemsLeaderVisionary1
1209 Solutions Visionary1
13ArkievaNiche PlayerVisionary1
14AdexaChallengerVisionary1
15AnaplanChallengerNiche Player1
16ToolsGroupLeaderNiche Player1
17QAD DynaSysChallengerNiche Player1
18SlimstockNiche PlayerNiche Player0
19Blue RidgeChallenger1
20Relex SolutionsNiche Player1
21SynchronChallenger1
22QuintiqVisionary1
23FuturMasterChallenger1

For the last column, if there was a change with the vendor from the 2018 to 2021 MQ, this is designated with a “1.” “0” indicates no change.

Let us review the issues one by one.

Issue #1: Why Are Vendors Coming and Leaving the MQ?

Towards the middle of the table, you can see the vendors that appeared in the MQ in 2021 but weren’t there in the 2018 MQ. A vendor like o9 is valid as not being in 2018, as o9 was still getting started in 2018. However, Dassault Systemes was an established supply chain planning vendor in 2018.

Towards the bottom of the table, you can see vendors that appeared in the 2018 MQ but disappeared in the 2021 MQ. I checked to see if these vendors were consumed in an acquisition, particularly by any other vendors on this list, but I could not find any report of any of them being acquired. So they are still operating, but now Gartner is saying — in effect — they are not worthy of consideration.

Notice that none of them were ranked in the Leader quadrant in 2018.

I conclude that vendors that do not score very high in the MQ receive marginal benefit from being in the MQ.

  1. They often don’t get short-listed but still have to pay Gartner’s very high fees.
  2. It seems very likely that these vendors concluded paying Gartner that amount of money was not worth it, and they then dropped out.
  3. When they told Gartner this, Gartner would have put their persuasive salespeople on them and tried to sell them on how well they could do in next year’s MQ, but the vendors were not convinced.

There is never any explanation from Gartner as to why vendors come in or leave the MQ in each MQ. Gartner never mentions why a vendor comes in or leaves any MQ. The desire of Gartner is for software buyers not to think in historical terms, and only look and be concerned with the most recent year’s MQ.

Radio Silence

What is extremely curious is that vendors who are not part of the MQs and have no plans to be are still quiet in their criticism of Gartner. This topic is something I intend to explore in a future article. Recently, a well-known analyst came out in support of Gartner, even though I believe that their private opinion of Gartner is very different from their public opinion. How is Gartner able to maintain near radio silence on criticism, except on comments on LinkedIn or in private conversations where vendors FREQUENTLY share their critiques of Gartner — as long as they are never personally quoted.

Here is a recent message sent to me recently.

You are saying what everyone knows but are unwilling to say. I applaud you for your courage.

This is a bit of an overstatement. I believe the person who sent me this is using the term “everyone” to mean vendors. Software buyers certainly do not know this. A lack of knowledge of how Gartner works is exactly what allows Gartner to continue and be so successful.

The Problem with Excluding Vendors Unwilling to Pay Your Price?

It should go without saying that if your inclusion and exclusion is entirely based upon who is willing to pay your price, this is an enormous flaw in your method. If I had a car rating website and only rated cars that paid me and then left out cars that didn’t, it would be challenging to claim that my website was the right source for the buyers to review cars for consideration of purchase. If I then hid how much each car company paid me, and then charges large car companies more than smaller companies, this again undermines the website as an unbiased source of information on cars.

How do customers know that Gartner is not profit-maximizing its MQ? That is, Gartner is charging the maximum of vendors such that come of the walk away, and enough stay to make the MQ have enough participants — and that the MQ has nothing to do with who belongs in this MQ? More than an inkling, I am quite sure this is the truth. Why is this so obvious? Because there are so many vendors excluded from the MQ. Gartner cross markets its access. it tells the vendors they need to pay Gartner because they are so influential with buyers (which is true), and then tells software buyers they need to pay Gartner because they have such good information about vendors (that second part is partially true — that is the data access is real, but Gartner both cannot reflect it as they are trying to extract the most from vendors, and their method does not effectively process the information about the vendors. There is also a great deal of questioning on calls that I have with vendors whether Gartner analysts know enough to be ranking these vendors in the first place.)

Issue #2: The Instability of the Gartner Ranking

From 2018 to 2021, I counted 18 changes in the status of the vendors. That is the last column. This means that a vendor either disappeared from 2018 to 2021, appeared, or changed their quadrant ranking. What is the point of using a ranking (if we even assume that the MQ ranking has any validity) if a few years later, that vendor is no longer in the ranking, or not even in the MQ at all?

Issue #3: Illogical Ranking

Gartner’s model of ranking a vendor (without any publication of supporting data) based on two overall criteria (Completeness Vision + Ability to Execute) into the top and lower scoring vendors from a restricted vendor set based upon who was willing to pay them does not make any sense. Furthermore, the framework pushes up companies to have expansive suites. However, the problem with expansive functionality sets or expansive suites is that they tend to have more implementation problems and are naturally offered by larger vendors. However, buying from a large vendor is not a pathway to better outcomes. And even the “Ability to Execute” favors larger vendors in Gartner’s method. And this leads to the next issue.

Issue #4: The Lack of Transparency

Gartner’s method is entirely opaque. There is no way to audit Gartner’s MQs. There is both no disclosure of the payments to Gartner nor any disclosure about the math that supports the MQ. This is a constant issue with entities that cover software that claims to do research — they keep their conflicts and methods secret.

But there are several elementary checks on the validity of any rating agency/research firm etc.. This applies to car reviews or software reviews, or any comparison. (these are just two, there are more)

1. Is the method transparent?

There is no argument in favor of opaque research methods. In each case, when there is opacity, opacity is for hiding something the content producer does not want people to know.

2. Is there disclosure of relationships/payments/side benefits/etc.?

If payments or other considerations are being made to the entity, those payments are not disclosed, much less discussed — this is research corruption. No argument can be made against disclosure.

The method to be used must be published and transparent.

Issue #5: The Outcomes from Using a Gartner MQ

The outcome from shortlisting only the vendors in the MQ would lead to bad outcomes for the buyer. Just because Gartner rates a vendor as a Leader does not mean they are a good match for your requirements. Secondly, the larger and more expensive vendors tend to dominate the Leader rankings with Gartner — and this means that using Gartner leads to higher cost solutions.

The World Outside of Gartner

While Gartner is the most prominent, other recommendation entities like G2Crowd or Capterra have rigged reviews as was covered in the article G2Crowd’s Problem with Financial Disclosure and a model that hides the fact that these entities are paid by vendors — which again skews towards the largest vendors with the largest marketing and sales budgets. Both websites have no disclosure of how they function behind the scenes — pretending to offer unbiased reviews and follow no research rules. This falls into an established pattern where the method used and the scoring are non-auditable. These websites use the reviews as lead generation for software vendors but use opaque language to describe to the reader and hide how this works.

About the Brightwork CPE Survey

This survey is designed to obtain peer responses to other forecasting vendors and ask for vendors’ views on other vendors in the forecasting space. The survey has two parts. 

  1. The descriptive qualitative section.
  2. The ranking or a quantitative section. 

Part 1 of the CPE can be found here Part 1. Part 2 is here. This document serves as the explanatory text for Part 1 (the ranking spreadsheet) and Part 2 (the descriptive text section). It is recommended that you first read this document, complete the survey questions at the end, and then move on to the ranking spreadsheet. 

How The Information You Provide Will Be Presented in the Study

In the published CPE, every vendor that participated will be listed on an introductory page and some necessary information about the vendor.

An interactive portion of the study allows for filtration so software buyers may focus on or target interest areas. That portion will only be available through using the Brightwork website. Then there will be an entirely text portion that will be consumable online or exportable as a PDF. Every participating vendor receives access to the study. It is envisioned that vendors interact with the study with prospects by displaying the website to an overhead projector or through screen sharing when remote. This study will be more analytical than anything available to the prospect — at least that we have seen. 

Who is an Eligible Vendor?

This list of vendors included in the analysis is listed alphabetically in the Vendor List tab of Part 2, the spreadsheet link you have received.

Your peers, in turn, when they revisit this survey, may start appraising your solution. Not every vendor may choose to participate, but that does not stop every other vendor from offering their views on the eligible vendor list. 

Any vendor can apply to be included in the study. Who is included is based upon a combination of vendors and Brightwork, with Brightwork having the final vote. The only criteria Brightwork uses is whether a vendor has functionality that places them appropriately into the category of supply chain forecasting. For example, there are many forecasting applications for finance — however, a financial forecasting vendor would not be allowed in this category as the comparison would not make any sense. All vendors can be confident that they will only be compared against valid supply chain forecasting applications. 

The study is designed to reflect the views of vendors democratically. However, a vendor can be removed for things that violate the policies laid out in this document. That would include repeatedly and deliberately ignoring the instructions (one example of this would be a vendor ranking themselves in Part 2, in the ranking sheet). Something else that would qualify is repeatedly sharing the study with customers who have not purchased a license to the study, etc.  

How the CPE is Similar to an Open Source Project

  1. The CPE’s method is entirely transparent. This is distinct from all analytical products in the space.
  2. Brightwork does not add content to the vendor’s responses.
  3. The vendors get to recommend additions to the survey. However, other vendors have to agree in a sufficient number with the addition. 
  4. The vendors share the CPE with prospects. 
  5. Brightwork serves as the mediator and organizer of the CPE, more in the model of an open-source project.

How the CPE is Different Than an Open Source Project

  1. The contributions for changes as well as the content provided by vendors are owned by Brightwork, not the vendors. The reason for this is to prevent vendors from publishing their responses or their responses from others or otherwise undermining the comprehensive study. This is reinforced by an NDA that each vendor will need to sign which prohibits the publication of any part of the CPE. However, any vendor at any time can have their submission removed if they desire to exit the CPE. Any publication of details about the CPE on a vendor website, rather than a link to the original study, will result in that vendor being removed from the CPE.
  2. Vendors are able to propose adjustments to the CPE. Those proposals can be evaluated under a democratic framework where input or voting occurs by the participating vendors. But the final decision for a change resides with Brightwork as the owner and publisher.

More on Balance and Bias

And Brightwork does not provide the content of the study. Instead, Brightwork is the aggregator and organizer of information supplied by vendors. Brightwork does have views on forecasting (and those views are easily findable through reviewing past articles published at Brightwork or the Foresight Journal or in several books). However, those views are not part of this study. As soon as Brightwork weighs in on any functionality or approach, we include that as part of the study, the vendors on the other side of the issue will immediately call out Brightwork for either bias or that we do not understand the technology of the vendor. The method has been designed explicitly to stay away from those charges. 

For vendors reading this unfamiliar with Brightwork, Brightwork R&A does not take money from vendors to promote vendors and has been outspoken for several years, critiquing this model as anti-research. 

You would need to follow a few guidelines in your responses if you chose to participate.

Item #1: Formatting of Your Answers

We ask that you type your answers as plain text, devoid of any formatting except for line breaks. We need to be able to show a consistent format in the survey. 

Item #2: Comment Only On Other Products & Vendors

You should not turn a question that asks about a vendor into commentary about your company or product. Other vendors will do that. You have the opportunity to rank yourself after the question. 

This survey and the final study that reports its results are only designed to reflect your views of other vendors. 

For each question which asks you who is best, you can’t include your own company or product as a response.

Furthermore, you must restrict your rating as described in Item #3. 

Item #3: Select and Rate From Your Competitors

You must list your top 8 competitors.

And then only provide ratings to Part 2 that relate to those competitors.

The drop-down list you will see will show all vendors, but you must only select from the same eight listed as your competitors. We will check your results, and if you have selected a vendor outside of these eight that you have listed, we will send Part 2 of the survey back to you. 

Item #4: Agreement on Publishing

Vendors are not allowed to publish or otherwise sharing their answers submitted in this document to their prospects. This study is best-read and understood within the overall context of solutions. That is, the reader needs to be able to see the entirety of the responses. And the study will be filterable by multiple filters like by vendor and by question category. This will make reviewing the study very interactive and very efficient. This will be immediately apparent when you finally read the published study. 

The study will be priced affordably. Given the value it will provide to a prospect in decision support, the prospect can afford to sustain the effort put into this by Brightwork. If all vendors help make it sustainable, this can be a long-term tool for all participants. More of this is covered in the earlier section How the CPE is Different from Open Source. 

The Related Topic of the Eventual Pricing of the Survey

This study is designed to be affordable to companies. Roughly speaking, the idea is to charge more for access to study as more vendors are added. The overall future pricing has not been determined but it will be reasonable. However, for a number of months, the survey will be free for anyone to access as it is beta.  

The benefit to the vendors is that non-rigged research can be used in pursuits, and the study can be used as a counterpoint to Gartner. The price is per user, and to gain access, each vendor must also purchase a license. A small vendor may need one license for the survey, where a larger vendor would require more. This is not a participation charge, as there is no participation charge. This is an access charge. 

Document Versioning and Vendor Updates

You are filling out a specific version of this survey. This would be version 1 (see the name at the top of the screen). 

The survey will be updated on a rolling basis every four months, with a deadline determined when solidified. When the fourth month has passed, you will see your document copied over to the next version. Your document from the previous version will have a new version number. Still, all of the original information will be copied over, so all you will have to do is make changes to what you previously submitted. And you will have until the end of the four-month period to make any changes. 

  • We use track changes in Google Docs and Google Sheets from previous versions with you declaring if you have changed a section. In many cases, the same information will be carried forward, while only some of the responses will have been changed. If you don’t mark that a change was made, then we won’t reflect a change you made.  
  • With every new version, you begin again with the last version you already filled in. 

Your adjustments are then reflected in the survey two weeks after the rolling deadline.

Conclusion

This web page is designed to receive feedback. If enough vendors agree this is of value, then this survey can be created. The design is not complete or set in stone, and it may be adjusted depending upon the feedback received.

This survey and the overall approach have been explicitly designed to create a non-rigged research product that is 100% research valid.

If you have any questions, contact us at the chat bubble at the lower left-hand corner of this website.