- Consulting companies, vendors, IT analysts all seek to corrupt the software selection process.
- Find out our recommendations on how to manage the process the best way.
This is the most conventional chapter in the book. While there are few books on software selection, there are some articles that focus on how to do the basic parts of software selection such as asking for responses from the vendors, and creating a vendor selection matrix. The reason I call these tasks “basic” is that literally anyone can do them, and for the most part they are not what differentiates successful software selections from the unsuccessful. As I stated in the book’s introduction, the most important part of a software selection is not the elementary blocking and tackling, but the evaluation of the information that is brought into the software selection process. The almost total ignorance on this issue was the main motivation for writing this type of book in the first place. However, this book would be incomplete without a nod to these other undertakings, and in fact, I have found that there are some ways to improve even these basic activities.
The Different Roles of the Software Selection Team
Michael Burns, in his article Software Selection Done Right, describes the software selection team in the following way: “At our first meeting, we also explain the roles and responsibilities for the project. We typically identify roles for the sponsor, steering committee, project manager, project coordinator, business-process owners, subject-matter experts, and technical leads. The idea is to determine which tasks fall within the purview of each role and to name the people responsible. It is essential that the right people be assigned to the project. For example, the project manager must be very organized and subject-matter experts must be highly knowledgeable about their business processes. Also, they must have enough time for the project.” While I love the idea, I have never seen a software selection team so dutifully configured.
In fact, I cannot even recall very much input being solicited from the business-process owners, subject matter experts or the technical leads. Most executives seem to think they have all the requisite knowledge to make the decision as long as they have seen a demo and read the Gartner Magic Quadrant Report on the software to be selected. I don’t see how multiple areas of expertise could be leveraged (as proposed by a number of authors) into advice that is followed because I observe clients being blind-sided continually by requirements that cannot be met by the software they selected. Furthermore few people within the buyer have enough time to dedicate to the software selection. However, this intelligent orientation regarding team development and providing employees with sufficient time to participate and contribute to the software selection is shown in the quotation below from the book Modern ERP:
“ERP system implementations require a significant human resource cost. Smart companies dedicate their most valuable and knowledgeable employees to the ERP project for a significant period of time. Each team member will need to educate other members on their respective functional area. Commitment to the team will undoubtedly conflict with the employees’ normal job functions…. Creating the optimum team often requires backfilling key personnel to allow those chosen on the team to be fully dedicated to the implementation.”
Again, this also sounds great. But it just is not done in practice, or at least not frequently. Instead, the participants are expected to fit the software selection in with their regular duties, but they are given very little incentive to make software selection a priority. Those on software selection teams are generally juggling their software selection duties on top of their regular work. This is one basic reason that software selections are performed so poorly. When I arrive at a client, I can find no well-reasoned explanation that has been documented as to why the software was selected. If I am lucky I will be able to find a spreadsheet/matrix, which describes the different criteria that were selected. Companies simply do not make software selection a priority. Therefore, while Michael Burns is describing the right approach, I question how much time I should spend explaining the different roles if companies are unwilling to incorporate the input from such a large cross-section of the company. I found this comment from Michael Burns amusing:
“First, many of them know the business really well and can add a lot of value and input. Second, they are more likely to buy into the selection decision.”`
This is absolutely true, but it is also generally true that input from these individuals is not valued. Does this attitude lead to bad decision-making? Yes, it does.
However, I don’t think this will change by me writing a book on the topic.
We have entered a new era of elitism in the US where the executives know everything, add most of the value in the company, and the rest of the company is basically expected to “go with the flow.” How often do we hear about the insights and experience of the top managers of the company, that must receive top compensation or else they will fly away to another company. Compare this to how often we hear about the insight of technologists or applications users and how important they are to the company’s success. It’s a question worth considering because if we have a flawed model as to where the knowledge lies that is required to make good software selection decisions, poor decisions will continue to be made. On the other hand, I only have consulted with large companies, and large companies tend to be quite hierarchical and therefore exclusionary. Also, as an implementer, I have not seen as many software selections as others. I know the job is not being done well, because I deal with the fall out. The individuals who see the most software selections are those who work in a sales or marketing capacity for a software vendor. For this reason, I decided to ask a few colleagues their opinions of how well most software selections are performed. These individuals work in senior leadership roles in marketing in some vendors.
I received some interesting responses.
“I have seen so many variations, with perhaps these common themes:
Processes that are too ‘top down’ and IT-driven just as you describe, especially in ERP and commodity buys.
Others where the ‘users rule,’ also not so good because they tend to block innovative thinking.
The best processes tend to be driven by line-of-business middle managers, who often are low enough to understand the subtleties of what they need, but high enough to be able to think outside the box. So for instance, in our area, a supply chain director. I would also say that procurement (i.e., software selection) processes have actually gotten marginally better in terms of ‘due diligence’ since the ‘go-go’ days you and I remember. I can remember C-Level deals that weren’t vetted at all. In supply chain, that rarely happens, and in fact the RFI process can become rather too long and laborious, with perhaps adding little value for the extensive level of effort required.”
The Role of the Coordinator
In a software selection, the decision makers will be a company’s executive leadership, typically a combination of business and IT executives. Finance will be involved when it comes to discussions about budgets and money, but unless the application is finance-related (that is, software for their department), they are typically not that involved in the software selection process. However, while executives are the primary decision makers, they do not necessarily have to be the ones managing the process.
One weakness of many software selection initiatives is that the executives often try to manage all of the logistics of the process, when in fact it makes much more sense to assign the task to a manager-level individual in the company. As long as this person is not overwhelmed with other work (something to be considered when making the assignment), he or she provides a significant benefit in managing logistics and communication, in maintaining the folder of vendor documentation and the requirements list, and in tabulating the scores for the vendors. They may also have domain expertise if they are chosen from the area where the software is to be implemented.
These are the major roles of the software selection coordinator:
- Manage the Communication with all of the vendors: This includes everything from the initial introduction, to providing feedback on what the buyer would like to see demonstrated and how they would like the demonstration to be presented.
- Timing administration: Setting up the meetings and managing the calendar and logistics for web meetings and for vendor visits to the buyer’s site.
- Maintain the network folders for the implementation: These folders include correspondence from the vendors, their marketing literature, screen shots, IT analyst reports; anything that is pertinent to the software selection should be maintained in such a way that all the decision makers can easily find it and review it at their convenience.
- Development of the Vendor Grading Document: This means continually updating the document with contextual information as well as maintaining the scores.
- Serving as a mediator for all meetings: This ranges from managing introductions to walking the executives through the vendor grading document. Having a single person do this increases the likelihood that the things that need to get done will get done in a consistent manner. The executives themselves decide how much of a vote they would like the coordinator to have in the final software selection. As the coordinator should be from the area where the software is to be implemented, he or she should be able to offer good, detailed advice to the executives. Beyond domain expertise, the coordinator should have the following qualities:
- They should have the respect of the executives: Part of the coordinator’s role is to serve as a moderator, and this means managing the interaction of the executives. This is related particularly to the next topic.
- They should mediate selection bias: Software selection is frequently victim to selection bias. This means that each executive decision maker will seek to find the best software for the group or department that he or she represents, rather than the software that offers the best overall compromise for the company. This issue is most prevalent on ERP projects because ERP applications are such a diverse combination of functionalities. Applications that are more narrow in focus are less affected by this bias, but the problem arises again when buyers purchase software suites from one vendor rather than software from separate vendors for each category.
- They must be organized and good at documentation: This will be necessary to perform the coordination and maintain the documentation.
- They must be good communicators: The coordinator will need to communicate with many individuals and so must be able to do so in an efficient manner.
Implementing the Scientific Method in the Software Selection
I explained that the advice provided would focus on making the software selection process more scientific. Many people confuse science with sophisticated technology or scientific instruments. However, to be scientific is simply to follow principles related to how testing is performed. Following the scientific method does not require any scientific knowledge or even fancy equipment. This is explained in the quotation below from Wikipedia on the scientific method:
“Scientific inquiry is generally intended to be as objective as possible in order to reduce biased interpretations of results. Another basic expectation is to document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, giving them the opportunity to verify results by attempting to reproduce them. This practice, called full disclosure, also allows statistical measures of the reliability of these data to be established (when data is sampled or compared to chance).”
The analogy between software selection and science is, of course, not perfect. A company does not actually publish the results of its software selection documentation for other companies to evaluate and reproduce.
Unlike public entities, private companies don’t have any interest in helping other people understand what they are doing or in advancing knowledge. Instead, their objective is to “maximize shareholder value,” which essentially means to maximize the share price. In fact, even public companies do not publish the software selection results. However, portions of the software selection can be approached scientifically and the following sections will explain how.
The Importance of Documentation in the Software Selection Process
Gregor Mendel is an excellent example of the importance of documenting observations. He advanced the understanding of genetics through the rather low-tech approach of measuring the characteristics of successive generations of peas, which he grew in the garden of his monastery.
“By simply counting peas and keeping meticulous notes, Mendel established the principles of inheritance, coined the terms dominant and recessive, and was the first to use statistical methods to analyze and predict hereditary information. For eight years, Mendel cultivated thousands of pea plants and used a paintbrush to painstakingly transfer pollen from one plant to another to make his crosses (all the while still attending to his duties as a monk and a teacher).” — Wikipedia
His research paper is still available, at a link that is in the reference section of this book. His notes are quite meticulous, and this was a researcher who really knew how to document his results. With Mendel’s disciplined and scientific mind, he would have been excellent at software selection, or any area, which required a detailed and analytical mind. Science works because it follows a set of rules that allows for objective comparison of the variables to be observed. Documentation is a big part of the scientific approach because our memories are far from perfect, and recording allows for comparisons across observations that are separated in time. For example, not all the vendors that a buyer is looking at will present their demos on the same day. Information from vendors and from other sources will similarly be time lagged. Secondly, information will come to different people.
How to Use Consulting Advice on Software Selection
Consulting companies are major influencers for enterprise software purchasing decisions. Usually, one of the major consulting companies is permanently resident as an enterprise software customer. The best place to start in understanding the advice provided by large consulting companies is to analyze their institutional structure. This tends to apply to small consulting companies as well. Understanding how consulting companies make their money is also critical to understanding how they work. As they say in political thrillers and investigative journalism, “follow the money.” There are several lines of thought regarding predicting behavior. One theory says that the individual can determine the behavior of other individuals in an institution. History provides examples of this—that is, individuals who set their own agendas. However, the incentives of the institution are a more reliable guide to behavior. Individuals whose behavior diverges from the incentives of the institution tend to be short- lived in that institution.
Therefore, while consulting companies are made up of a large number of people, their policy can be determined by both observing their incentives and by observing their output. Output is a far more predictive measurement than an institution’s statements about itself because for the most part, an institution’s behavior diverges greatly (and I mean this quite generally) from its statements. For instance, the worst thing that one could do to understand how an oil company works is to go to its website and take the statements it makes there at face value.
The Institutional Structure of Consulting Firms
Consulting companies have the following features, which are important to interpreting their advice on software selection
- Consulting companies make their money based upon billing an hourly rate for their employees.
- Their employees cannot be capable in all—or even a small percentage—of the software in a category. Because enterprise software is complex, ordinarily a consultant will work in both a single software vendor’s application, as well as specialize in a single module within that brand. For instance, I work in SAP, which is the largest enterprise software vendor. I have worked in different modules, but I tend to get most of my work from a single module.
- Therefore, they tend to have a deep specialization in a relatively small number of applications. Unsurprisingly, they tend to specialize in the largest applications. Specializing in a large and popular application allows a consulting company to get the highest percentage of billing hours out of their resources.
- The people that make software recommendations to clients are called “partners.” Partners are the senior management of the consulting companies. There are different levels of partners, with the most senior partners driving the policy of the consulting company. In fact, some of the policies (e.g., the policies regarding which major vendors they should focus on in their consulting practices) were decided long ago, and these policies are not decided or re-determined in any way periodically.
- The partners at the major consulting companies are very motivated individuals with very high compensation who must meet yearly consulting services sales quotas. In many ways, being a partner is a cushy job and the only people a partner really answers to is other partners. However, their yearly services sales quota hangs over them. In order to meet these sales quotas, they must place their consultants and this means selling or placing the consultants that have experience in the applications from the major software vendors (although consulting companies do occasionally place independent contractors on projects, they prefer not to as their margin on an independent consultant is much less than their margin on a full time employee).
- Consulting companies are extremely hierarchical—I would say approaching that of a military organization. The resources below the partner level have no say in how the organization is managed. In fact, they don’t even have a say on the technical recommendations that they make if they happen to contradict with the position that the partner wants to send to the client. If a recommendation is to be provided by a consultant, which may affect sales of consulting services, the recommendation will be run past the partner first. The partner will then tell the consultant what his or her “professional opinion” on the matter will be. It should be relatively clear from the points listed above, but partners must be able to convince their clients to hire their consultants, who are trained in the applications from the major vendors.
Can Such Generalities be Applied to So Many Consulting Companies?
People not familiar with the consulting market may say,
“Wait a second. These consulting companies have many consultants at all different levels. Surely, when individuals work for them, a variety of viewpoints are available to consulting clients.”
This is a common misconception.
The consultants below the partner level have no influence on what recommendations are made. The partner discusses what will be recommended with their consultants before meeting with the client. And this not only applies to full-time employees.
Direct Experiences from Working with Many SAP Consulting Companies
When I have worked for consulting companies as an independent consultant (a so-called subcontractor) I was repeatedly pressured and told what viewpoints I should give the client. The advice I give must fit with the story that the partner wants to tell. However, they want to create the illusion that their opinion is my personal opinion. In fact, even the partners at the level I work with do not set policy. These policies, such as which software to recommend, are set far above the partners that actually work on projects, at the senior partner level.
The Bias of Vendor Association
In order to convince their clients to staff their resources, they must also convince their clients to implement the software of the major vendors, as that is the expertise that their consultants offer. Thus, consulting companies have a conflict of interest when making recommendations regarding software selection.
To prove this, let’s take a hypothetical example.
Let’s imagine a new partner joins IBM or Deloitte or any of the other major consulting companies (it does not matter which because they all operate in a similar fashion). Let’s further imagine that this partner is truly focused on selecting the best software on the market to meet his or her client’s requirements. An objective analysis would find that some of the best software available on the market to meet the client’s needs is not the software for which the partner has trained resources. If the partner were “honest” and put the client’s interests above his own, the partner would admit this fact to the client, increasing the likelihood that the client chooses a software vendor for which the partner has no resources that can be staffed. (At this point one might say that the partner could hire independent contractors to perform consulting, taking a lower margin on them. However, the contract market is well-developed for the major applications only, so for anything but the larger vendors, this is not an option.) As soon as the client chooses software, which the partner cannot staff, the partner loses out on both the services revenue and also their ability to control the implementation.
If the client were to select an application from a smaller vendor, in most cases the consultants will come from that vendor as well. A partner that cannot staff as many resources on projects will eventually fail to meet their quota, and will be asked to leave the consulting company. Therefore, even if a partner wanted to (and having worked with many of them I can say that most of them don’t care either way), they could not provide objective advice to their clients. This is the problem when any entity—not just consulting—receives a benefit if they advise a client to do one thing or another. This conflict of interest is rampant also in the financial industry, where recommendations are often made based upon the result that would maximize fees. Before moving on to the topic of how to interpret a consulting company’s advice, I will finish off with a short story about how the large consulting companies operate. I was at one time a manager at a large consulting company.
Consulting Firm Corruption
During software training, I met a person who was in a management position at a company that was implementing the software I worked with. At the beginning of the training session we went around the room and stated the company for whom we worked. Once this person found out the company I worked for, he approached me and asked if I could pass on information to someone in my company about our support in selecting a consulting company for their implementation. I want to emphasize that this person was very clear that they wanted my company’s help in selecting a consulting company, and not for actually performing the implementation. I passed on this information to the partner to whom I reported. Because this was a fresh opportunity, I was besieged by phone calls from partners from around the country about how to handle the situation. One partner was of the opinion that the appropriate approach was to accept the consulting selection project under the false pretense that we would help them find the best consulting company for their needs, but in actual fact, we would impress them so much with what we had to offer that the client would simply call off the selection process and choose us instead. To carry out this partner’s strategy, we would need to suppress information from the consulting companies that we were evaluating, but adjust the information so that it never seemed complete, and essentially stall the process so that the outcome would work to our advantage. It was not one rogue partner that recommended this approach, they all advised the same thing in one shape or form. The selection phase of a project (be it software selection or implementation partner selection) is much shorter than the implementation stage and uses far fewer resources. Therefore, no consulting company that performs software selection and also has the capability and resources to perform the software implementation for a company will be satisfied with just the software selection work. In fact, a core strategy of every consulting company I have come into contact with is to use initial projects to gain larger projects.
When I worked for KPMG I was told by a partner to snoop around for other work while I was there. Another partner used to repeat the phrase “penetrate and radiate” in meetings with large numbers of people and on conference calls (so it was not even a secret within the firm), so once into a client, you radiate through them by offering them more and more services. There is never even the pretense that services should be sold that the client actually needs—it is literally never even brought up in internal conversations. Consulting companies are generally unconcerned with whether these services are needed or appropriate because at the end of the day it all comes down to billing hours and the margin per hour.
Interpreting Information from Consulting Companies
One thing that I hope to establish is that all software selection information (and many other types of information) that comes from consulting companies is suspect. Depending on suspect information from consulting companies is one reason—and I will show many others—why quite often the software implemented by companies is inappropriate for their requirements. The software happened to be what the consultants that worked for the company were trained in and could bill for. Therefore, the software selected met the needs of the consulting company, but not the needs of the company making the actual software selection. The misinformation provided by consulting companies for software selection goes beyond simply prompting their clients to choose software that is not the best for their needs. Consulting companies were instrumental in completely overselling the benefits of every major new software technology, with ERP being an excellent example of this. In the book, The Real Story Behind ERP: Separating Fact from Fiction, I cover the rather surprising result of my research that ERP has failed to live up to not just one, but also nearly all of the promises that were used to sell ERP systems to clients. Consulting companies also misinform clients as to what they can expect in terms of the implementation effort, how well the software will work for them, and therefore their expected return on investment. When advising their clients, they follow a sales approach rather than a scientific approach. At times, the consulting companies I have worked for and worked with, as an independent consultant seem to be nothing more than sales arms for the largest software vendors (major consulting companies tend to have partnerships with only the largest software vendors).
A good example of this was a webinar that I was asked to attend by one of my clients; a consulting company presented the webinar. I had worked with the application that was the topic of the webinar. However, throughout the webinar, this consulting company consistently presented a viewpoint about the application, one that I had never experienced even though I had worked on multiple projects with this software. According to this company, the application was “easy to install,” “planners liked it,” and “it just worked.” Many of the statements they made were directly contradicted by my experience with this software on projects. However, when different companies that were logged into the webinar would ask a question, there was always a fast and easy answer for what could be done to mitigate the concern. The consultants who presented in the webinar were in full sales mode. They appeared to be willing to say anything in order to get the companies participating in the webinar interested in contacting them for consulting work. More detail on this experience is covered in the article, which I wrote on this topic and is available at the link.
Of course, new implementations are all positive for consulting companies, as they provide the consulting services—they receive benefits with no risk. However, the implementing company takes a risk on every implementation. Therefore, the consulting company will have a strong tendency to be more Pollyannaish on the potential of the implementation than is warranted based upon experience. As an independent consultant, I also interview to work as a consultant on implementation projects. From this experience, I can say that there is simply no doubt that potential clients prefer to hear positive stories versus realistic stories regarding experiences with an application. So the consultants and consulting firms that offer the rosiest future scenarios have the highest potential to get the most work. This is of course not an isolated problem for software and consulting. Among lawyers, it is well-known that those who paint a pretty picture to potential clients have a higher probability of getting more business.
Finding Entities That Lack Financial Bias
The only consulting companies that can be said to be without a financial conflict of interest are those that only perform the software selection and do not perform the implementation—if (and this is a big if) they are not resellers of the software. The same would be true of an independent consultant, although typically independent consultants are not hired to assist with software selection as this type of work tends to go to the large consulting companies that offer both software selection and implementation. However, while hiring an independent consultant or a consultant that only performs software selections addresses the financial bias, it does not control other types of bias. In my experience I have found many—if not most—of the consultants that work in my field have a strong bias toward the software that they work with. For example, at Brightwork Research & Analysis, I try to describe the reality of working with different software applications, and sometimes this means explaining the frustrating parts or poorly-designed parts of the application. There are several articles on thethis website about one particular application that had quite a few problems. I was actually contacted by a European independent consultant asking me not to do that, as it would decrease the demand for the application and by extension, his services. In other conversations, when I bring up some excellent functionality in a competing application and in my view, clearly superior functionality, the consultant will invariably say:
“Oh, well my application can do that. It may not do it the same way, but it does it.” I have observed that when the functionality is clearly inferior, the person defending the inferior product will use the term “different.”
I know of no other area of analysis where it would be accepted that because two items have something in common that they can be considered equal. For instance, both a bicycle and an airplane will get me from San Francisco to Los Angeles and the end result may be the same, but they are certainly not the same thing and they are not simply “different.” Therefore, if one is very careful, one can find entities in the market that do not have a financial bias. However, removing all bias is exceedingly difficult when contracting for consulting services. If someone who is advising you is to be taken seriously, at the very least they should not have a financial incentive based on your selection of a software application. Just achieving this modest goal would be a great improvement in the advice that you will receive over the current status quo.
Consistently documenting these disparate data points allows them to be analyzed in a controlled manner. It also allows others to analyze them in a comprehensive fashion both prior to the software selection decision as well as after the fact. There is a list of what is called cognitive biases (such as confirmation bias, which is the predisposition to search for data points that support pre-existing beliefs) and many of these cognitive biases are at least partially addressed through thorough documentation.
Tabulating the Scores for Each Vendor
Some software selections use a formalized scoring spreadsheet and some do not. Some companies simply write a short report for why the software was selected, but many do not. However, for the following reasons I would very much recommend putting together some type of scoring sheet:
- Software selection means comparing a large number of criteria. It is difficult to really see how the vendor products compare across so many criteria without some type of graphical representation.
- The scoring document clearly communicates to everyone what criteria are being used for the selection and how the software under consideration is weighted. I am surprised to see so many software selection spreadsheets that do not have the ability to weight the criteria. The executives do not have to agree on the weighting. Each can provide a weight, which can be adjusted based upon how much of a vote they have. However, the company needs to come to an agreement on the weight per criteria.
- At the final selection meeting, having a document helps ground the discussion and allows everyone to use a single frame of reference for the decision. Many of the executives will have missed some of the data that has been documented in the scoring spreadsheet. Reviewing the scoring document will help bring all the executives, as well as other software selection team members, to the same understanding.
- A software selection is an important decision for the company. For such a decision it’s important that the outcome is based upon as much objectivity as possible. I find that matrix helps increase objectivity.
- The scoring document becomes a long-term reference for the company. It is also useful for the stage that follows software selection: negotiation (covered in the following paragraphs).
- The software selection matrix is easy to do. I have placed a sample software selection matrix below.
A software selection matrix combines the scores for each vendor for a variety of criteria along with the criteria weight. This software selection matrix is simply one example, and of course it can be extended to have any number of criteria. The individuals who are part of the selection must agree as to the weight per criteria. As soon as that is done, the vendors are rate, and oftentimes sorted on the basis of their combined score. I have used a derivation of this matrix and I have always found it useful. Occasionally individuals who are pulling for a particular vendor will criticize the scoring document by saying something like,
“I already know what the scoring will be, so I don’t have to look at it.”
Typically these statements are an attempt to undermine a rational approach to the software selection and to minimize the inputs of others so that their preferred vendor can win. Emotions can run high during a software selection, which is why it’s important to gain agreement on both the criteria and the weight assigned to each criterion (which can be voted on, resulting in a blended weight) before the software is scored. The criteria and weights should be reviewed several times and then set in stone. If the weights are reviewed or requests to change the weights are received after the software vendor gets its rating, then what is most likely happening is that the data is being changed to fit a desired outcome. The methodology of the scoring for the software selection should be straightforward. The winning vendor should be the one with the highest combined weight output of all the criteria. Anything can be included as criteria—price, the perceived ease of working with the vendor—the criteria are subjective. They are whatever the software selection team chooses to value. There are competing ideas regarding the inclusion of vendors that the buyer cannot afford. On one hand, the buyer is wasting the vendor’s time and their own time. On the other hand the company learns more about what is available in the marketplace as well as understanding what they are trading off to get a vendor they can afford. There is no perfect answer to this question.
Making the Decision
There has been a good deal of research into group decision-making. One of the findings is that the more vocal members of a group easily influence other group members, even if the more vocal members do not necessarily have more domain expertise than the less vocal members. What is desirable is that each individual makes their own decision such that the decision is reached independently and is not unduly influenced by the personality of one or a few people. To accomplish this outcome, the RAND Corporation developed the Delphi Method, named after the city of Delphi in Greece, the location of the Oracles who were consulted by the Ancient Greeks and Romans, among many others. RAND’s research into the Delphi Method goes back to 1943 when they performed a number of studies on this topic. The original intent of the research was to obtain better group judgment, by performing research, for example, into how the effect of strong personality types on groups can be mitigated (the research isolated the participants from one another).
The Delphi Method eliminates group interaction as explained by RAND:
“RAND developed the Delphi method in the 1950s, originally to forecast the impact of technology on warfare. The method entails a group of experts who anonymously reply to questionnaires and subsequently receive feedback in the form of a statistical representation of the ‘group response,’ after which the process repeats itself. The goal is to reduce the range of responses and arrive at something closer to expert consensus. The Delphi Method has been widely adopted and is still in use today.” — RAND
The Delphi Method is often thought to relate directly to forecasting only, and in fact, is an approach to consensus-based forecasting. However, it applies to any decision-making situation. The research into consensus decision-making, much of which was rolled into the Delphi Method, is useful for software selection, as is shown in the following quote about the Delphi Method (the emphasis is mine):
“The Delphi method (/’dεlfai/ del-fy) is a structured communication technique, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The experts answer questionnaires in two or more rounds. After each round, a facilitator provides an anonymous summary of the experts’ forecasts from the previous round as well as the reasons they provided for their judgments. Thus, experts are encouraged to revise their earlier answers in light of the replies of other members of their panel. It is believed that during this process the range of the answers will decrease and the group will converge towards the ‘correct’ answer. Finally, the process is stopped after a predefined stop criterion (e.g., number of rounds, achievement of consensus, stability of results) and the mean or median scores of the final rounds determine the results. Delphi is based on the principle that forecasts (or decisions) from a structured group of individuals are more accurate than those from unstructured groups.” —Wikipedia
Interestingly, I have never heard of an approach to making the software selection process more scientific and less susceptible to the influence of one or a few individuals. I also very much doubt that executives at a company would agree to something like the Delphi Method, as getting together with other executives is the normal way that decisions are made in most companies. On the other hand, people that reviewed this book prior to its publication have told me that they did something similar in their software selection, even though they may not have related it to the Delphi Method. One quotation I have included below:
“I facilitated two software selections. As I recall our methodology was to form a team comprised of middle management from the user department and IT, and we reported to upper management. We ‘workshopped’ the vendor questionnaire. I facilitated the workshops in a participatory manner that honoured all input without (too much) judgment and arrived at a consensus on the questionnaire. The questionnaire was based on the current business process (with improvements that could be taken advantage of if new software was acquired) and on user requirements in addition to IT’s requirements. So, in this case executives did agree to both the consensus process (probably because they didn’t care how it got done, just that it got done) and to the decision that was made via the process. We continued the consensus all the way through to selection without much dispute, and the same team, more or less, worked on the implementation. Because there was firm agreement on the criteria and on the software selected, we knew exactly what we were getting into and the implementation was very smooth.”
However, as I have explained with numerous examples, the current approach to software selection results in poor outcomes. The current design means that vendor sales and marketing count for more than the actual software. Companies that follow the realistic approach as outlined in this book can greatly improve the outcomes of the software they select. Moving from Selection to Vendor Negotiations As I do not have experience in negotiating with software vendors, I am not qualified to advise on the topic of vendor negotiations. Furthermore, a different group than the team that performed the software selection typically handles this part of the process. But the software selection is an input to the negotiation process, and the thoroughness of the software selection process can put those that negotiate with the vendors in the best possible position. Here is how:
- Often the group that performs the software negotiation, which may be from the procurement or contracts department, will not be familiar with the software and will not have participated in the software selection process. Therefore, it’s important to have an easy-to-understand document that explains the conclusions of the software selection.
- The vendor negotiation group should keep in contact with the software selection coordinator. There are all types of reasons for this, but one is so that the negotiation group does not lose the context of the software selection. Secondly, there were statements made by the vendor during the software selection, and those statements should not “go away” when the contract is about to be signed. A frequent technique is to sell one thing, but then have the contract say something that is more modest. Some very experienced consultants recommend writing the important promises from the sales process into the contract. By keeping a good communication level between the software selection coordinator and the vendor negotiation group, the buyer can stay out of these traps. Finally, the coordinator, and possibly other members of the software selection team need to review the software license and consulting agreement before it is signed. This will be a tedious process because much of the documentation will be in legalese. Rather than having each person try to get through the contracts themselves, it can make more sense to go through the documents as a group and to have the company’s internal council there to translate what things mean in plain English.
- The software selection process will result in a list of vendors, which can be sorted from the first choice to the remaining choices. As with any purchase decision, the first choice may lose out to the second or third choice when negotiations determine that an agreeable arrangement cannot be arrived at with the first choice. A thoroughly-filled out software selection scoring matrix explains not only the sequence of the choices but also the degree of preference for one vendor over another. This gives very important information to the individual/group negotiating with the software vendor. A small differential between the top-rated vendors means that the negotiating group can look for opportunities for a lower price from another vendor to tip the balance in their favor. For instance, in the sample-scoring matrix shown previously, there is a very small difference between Vendor 1 and Vendor 2, and only a one-point difference between the estimated total cost of ownership (TCO). If Vendor 2, the top choice, were to take a hard position in the negotiation, Vendor 1 would seem to be an available option for this buyer.
- The software selection team should be careful to communicate the TCO aspect of the selection (which is a criterion in my software selection matrix), because without this context the negotiating team may be blind to these costs and make their negotiating decision based upon the initial purchase price only. This will require the software selection team to have made certain assumptions regarding the total cost of ownership. It’s not an easy task, as TCO is tricky and has many factors, which must be added to the calculation. I believe it is more realistic, given the resources that companies seem willing to allocate to software selection projects, to simply rate a vendor’s predicted TCO on a scale from one to ten. The TCO value can be either added to the scoring matrix or assigned a weight. However, a thorough software selection would result in an estimate. There is also a question of whether to exclude those vendors entirely when it is suspected that their price will be too high. However, vendors are cagey regarding the price of their software and it can depend upon many factors, including which other products produced by the vendor the buyer currently owns. Furthermore, the “real price” can drop different degrees per vendor per situation, depending upon how the negotiation proceeds. This, of course, greatly increases the complexity of the selection process and reduces the transparency and efficiency of the enterprise software market.
Most publications on software selection cover the tactical elements of software selection far better than the broader or interpretational elements. This book has taken the opposite approach, focusing on how to interpret information that is received from the multiple sources that influence software selection. I allocated much less space to the tactical elements of software selection. However, this chapter covered these tactical elements of how to manage the software selection process. The software selection coordinator role is quite important to the effectiveness of the overall software selection process. In this chapter, I laid out the characteristics that make a good coordinator. It’s relatively easy to begin a software selection process without a coordinator, however, this is misleading. As the software selection proceeds there is more coordination required, more items that must be followed up, and eventually the requirement for a mediator between all the members of the software selection team. The coordinator should have the most time allocated to the software selection project of any of the team members (and needs to have this reflected in the rest of their workload) and should be able to keep the team focused, as well as provide the team with the necessary context to both interpret information and explain to them accurately how the software selection has proceeded up until the current discussion. A coordinator can allocate the type of time to a software selection that such an initiative requires, and the type of time commitment that the executives often cannot make. Software selections can be greatly improved by increasing the use of the scientific method, which is a structured approach to the testing of any hypothesis. Science works because it follows a set of rules that allows for objective comparison of the variables to be observed. And a big part of this process is to document information that is uncovered during the software selection. This documentation helps increase the rationality of the overall process and can help to place on even footing information that is often gathered months apart. A software selection matrix, which is a condensed way of making these comparisons, is not uncommon, but documentation during the software selection process should not be limited simply to comparison spreadsheets of this type. Documents with detailed explanations of the rationale for the positives and negatives of the various compared applications should also be created and made available to the current software selection team members as well as archived for future software selections. Documentation, and reviewing this documentation, can have the positive effect of mitigating the natural tendency of one or a few individuals with strong personalities and strong opinions from imposing their selection on the rest of the team. For instance, while a person may have only one vote in terms of their official capacity on the selection team, through the force of their personality, and their ability to persuade, they may be able to co-opt other members to essentially give up their vote to these individuals.
The excessive influence of some individuals on group decision-making is why the RAND Corporation developed the Delphi Method. The Delphi Method makes the “voting” private, which tends to be less influenced by factors such as being in the same room with an influential individual. This is also why political voting is performed behind a curtain of some type. Once the software selection has been made, typically the negotiation of the contract is transitioned to another group. This is where the documentation that was developed during the software selection can be put to another good use by allowing the group that performs negotiation to understand why the vendor was selected. The group that negotiates must know not only the first choice but the second and even third choice because an agreeable arrangement may not be feasible between the buyer and the first choice. A thoroughly-filled out software selection scoring matrix explains not only the sequence of the choices but also the degree of preference for one vendor over another. However, someone who was part of the software selection team should keep in contact with the group that performs the negotiation. There are some with expertise in this area that believes that a person from the software selection team is in the best position to perform the negotiation with the software vendor, however, this is both often not feasible because of how companies are structured, but I question if this is the right approach because negotiation is really a different skill than software selection. If I take myself as an example, while I am qualified to perform any supply chain planning application software selection I do not negotiate contracts and would be at a distinct disadvantage performing the negotiation against a software vendor as they would certainly have a professional negotiator. I would think the best possible combination would be for the member of the software selection team to work with the individual with negotiating expertise in order to perform the negotiation together. Furthermore, the software selection team should be careful to communicate the TCO aspect of the selection (which is a criterion in my software selection matrix that I showed in this chapter), because without this context the negotiating team may be blind to these costs and make their negotiating decision based upon the initial purchase price only.
Search Our Other Using Software Advice Content
Software Selection Book
Enterprise Software Selection: How to Pinpoint the Perfect Software Solution Using Multiple Sources of Information
What the Book Covers
Essential reading for success in your next software selection and implementation.
Software selection is the most important task in a software implementation project, as it is your best (if not only) opportunity to make sure that the right software—the software that matches the business requirements—is being implemented. Choosing the software that is the best fit clears the way for a successful implementation, yet software selection is often fraught with issues and many companies do not end up with the best software for their needs. However, the process can be greatly simplified by addressing the information sources that influence software selection. This book can be used for any enterprise software selection, including ERP software selection.
This book is a how-to guide for improving the software selection process and is formulated around the idea that—much like purchasing decisions for consumer products—the end user and those with the domain expertise must be included. In addition to providing hints for refining the software selection process, this book delves into the often-overlooked topic of how consulting and IT analyst firms influence the purchasing decision, and gives the reader an insider’s understanding of the enterprise software market.
By reading this book you will:
- Learn how to apply a scientific approach to the software selection process.
- Interpret vendor-supplied information to your best advantage. This is generally left out of books on software selection. However, consulting companies and IT analysts like Gartner have very specific biases. Gartner is paid directly by software vendors — a fact they make every attempt not to disclose while consulting companies only recommend software for vendors that give them the consulting business. Consulting companies all have an enormous financial bias that prevents them from offering honest advice — and this is part of their business model.
- Understand what motivates a software vendor.
- Learn how the institutional structure and biases of consulting firms affect the advice they give you, and understand how to properly interpret information from consulting companies.
- Make vendor demos work to your benefit.
- Know the right questions to ask on topics such as integration with existing software, cloud versus on-premise vendors, and client references.
- Differentiate what is important to know about software for improved “implement-ability” versus what the vendor thinks is important for improved “sell-ability.”
- Better manage your software selection projects to ensure smoother implementations.
- Chapter 1: Introduction to Software Selection
- Chapter 2: Understanding the Enterprise Software Market
- Chapter 3: Software Sell-ability versus Implement-ability
- Chapter 4: How to Use Consulting Advice on Software Selection
- Chapter 5: How to Use the Reports of Analyst Firms Like Gartner
- Chapter 6: How to Use Information Provided by Vendors
- Chapter 7: How to Manage the Software Selection Process