Evaluate Options Dispassionately
Now is the time for solution demonstrations! This phase requires a considerable investment of time, attention, and expertise from the selection team and vendors, so I hope you’re starting it with excitement (and maybe some caffeine).
We’ll never answer every question we have about solutions, teams, track records, and expected performance – but we aim to answer enough of the critical questions to feel confident in our decision-making. So, focus demonstrations and surrounding discussions on the most important business goals and solution requirements.
This Phase at a Glance
In the Demos & Vendor Evaluation phase, our goal is to get a clear sense for how each option can – or can’t – meet the stated needs, and begin understanding compatibility between teams.
Here are facilitator activities at a glance:
Work with vendors to schedule and prep for demos
Work with stakeholders to create selection criteria/scorecard
Facilitate vendor demos
Rank options against each other and note further questions
Compile rankings and determine finalists
Here are selection team activities at a glance:
Participate in vendor discovery
Weigh in on scorecard
Participate in demos, noting questions and likes/dislikes
Score each option according to potential fit
Rank options
At this point in the project, our vendor count starts at about five and narrows to three or so. We use vendor demonstrations and discussions to further refine our understanding of which solutions seem best to meet our needs.
Key Activities
Let’s walk through each of the activities in this phase, with what success looks like and considerations to work through.
Work with vendors to schedule and prep for demos
Schedule a demo with each vendor and set clear expectations for what you want them to show. Often this includes demonstrating the use cases we provided in the RFI.
Be careful in asking vendors to use dummy data from you: they typically have demo environments pre-populated with data so that visuals look nice and workflows can be illustrated well. If you want to see specific data characteristics (in addition to the requested use cases), work with the vendor to balance tradeoffs of familiarity (to the selection team) and completeness (for the vendor team).
I tend to schedule 90-minute sessions and ask vendors to hold at least 30 minutes for questions. Discussion often happens during the demo, rather than in a dedicated Q&A session at the end, but I just don’t want vendors to fill the entire time with their planned content.
While we’re talking logistics, I aim for no more than two demos in a day, since these are very attention-intensive meetings. And resist scheduling them back-to-back.
Depending on how quickly demos proceed, it may help to schedule some debriefs as you go with project sponsors or the entire selection team. If all the demos occur in the same week, you can probably wait until the end to debrief.
Because I don’t want vendors to have the entire selection team’s email addresses, I create duplicate events, one with vendor participants and one with client participants. I do share some participant information with vendors, so they feel equipped: this may be teams and roles, or teams and first names (just know that LinkedIn stalking is going to occur, no matter how much info you share).
I also act as an intermediary between the vendor and selection team, passing along information or questions as needed. Again, all but one vendor team is going to be disappointed at the end of this process, thus the precautions in sharing the minimum information we can to get the outcome we’re after.
Vendors will spend considerable time preparing for the demo, and we want to support them through this. As the facilitator, expect to participate in one or two prep meetings, answering questions on what’s most helpful to prepare.
I typically advise vendors to skip or seriously condense the sales-y portion of their presentation to make the most of time with the selection team. If they insist on sharing extensive information on the company and/or product, offer to share a packet with the selection team before the demo and only hit the highlights in the time together. (Death by PowerPoint at the beginning of a session can really decrease the selection team’s engagement when we get to the more interactive part of the demo, so this admonition is truly trying to help the vendor make the best impression they can.)
Some vendors take the opportunity to do more detailed discovery at this point, so the facilitator may need to bring in a few SMEs from the selection team to get into details. I find that the facilitator can answer many of the vendor team’s questions, so I typically try to have an initial sync with just me first, to knock out the basics, and then focus a larger conversation on the details that are best addressed by experts.
Work with stakeholders to create selection criteria/scorecard
Create selection criteria and a scorecard to use in evaluating vendor demos and discussions. I like to have a simple document that the selection team can jot notes and questions in during demos, and then a very simple form for them to share reactions after the fact.
For instance, a scorecard may include three components:
Do you feel this solution will [insert the main business goal you’re seeking through the vendor selection effort]? Yes, Maybe, or No (with space for notes)
Would you be excited for us to implement this solution? Yes, Maybe, or No (with space for notes)
Do you have follow-up questions that are keeping you from feeling confident moving forward (with or without this solution)? If yes, please share. Yes or No (with space for questions in the case of yes)
Review the scorecard with the selection team before demos begin, to make sure it’s a helpful resource. Project sponsors may want team members to answer other specific questions, to aid in the group’s decision-making and alignment.
It can also help to prepare for demos with the selection team, discussing how we expect to engage with vendors. In some cases, it has worked well for the selection team to have specific areas (like a requirements category) that they are responsible for running through the requirements in real time and asking follow-up questions if they don’t see something addressed. It can also work well to be more organic around engagement, so no pressure to plan the selection team’s demo participation out too much.
Facilitate vendor demos
When the time comes for a vendor demo, make sure to take an active facilitation role. As facilitator of this process, you are likely one of the few people who knows everyone on the meeting. So, be punctual and act as a gracious host of the meeting while people are gathering. When everyone is assembled, give a quick overview of goals for the session, provide a brief introduction of both the vendor and selection teams, and preview that you’ll be keeping time and may jump in if we’re too far off-track on time and/or agenda. Then turn it over to the vendor team to lead the demo.
I like to record demos, since they can run together and we often encounter something like, “I really liked this in option x, but I don’t remember how option y handled it.” Having a recording to refer to is helpful in those moments.
The selection team will actively participate in each demo, jotting any notes that will help them score the option after the session, as well as follow-up questions.
I also like to coach selection team members to engage genuinely – but to avoid showing too many cards to vendors at the same time. Think of this like a real estate agent telling you not to “ooh” and “ah” too much in a house: we will want to negotiate with the final few vendors from the best position we can. Our negotiating position is negatively impacted if one vendor has heard, “this feature is better than all the others we’ve considered!”
As facilitator, be ready to intervene if the demo is going off course. This can come from the vendor side (going way over on time in one part of the agenda) or the selection team (asking about too many tangents). Be ready to step in and gently bring the demo back on track.
Vendors will often offer up more time, if the session is running long, but we’re seeking to keep things fair, including the amount of time spent together. Aim to use the scheduled time well, not rely on scheduling more.
It works well for selection team members to fill out their scorecard and note follow-up questions soon after each demo, while the information is fresh in their minds.
Rank options against each other and note further questions
After all demos are complete, ask the selection team to share three pieces of information with you:
Any options they’re unwilling to consider further (and why)
Rankings for the remaining options
What questions they want answered before feeling confident in moving forward with this round of narrowing
I prefer to have everyone’s answers in hand before sharing them with the group. (At least some selection team members will have talked among themselves, but we generally want to capture individual thoughts before coalescing as a group).
Compile rankings and determine finalists
In a meeting with the entire selection team, share overall rankings for each option and open up the floor to discuss surprises and follow-up questions. Sometimes the group rankings start off aligned – we all have the same top three – and other times, we see rankings all over the place. It’s important to discuss the why behind individual rankings, particularly if we see dramatic differences among the group.
Work with the selection team to align on the top three-ish options for further consideration. This may require follow-ups with vendors, particularly if the topics in demos varied (despite your best efforts as a facilitator!). If you don’t feel like demos were apples to apples in comparison, consider having brief, targeted follow-up conversations or asking vendors to record a quick walk-through on the missing pieces.
If, as the facilitator, you think an important piece of the analysis is being missed, speak up. I had an instance where a vendor spent the first 15 minutes (that felt like an hour) of the demo on a sales pitch, despite my coaching, and the selection team’s engagement when they got to the product demo was muted, since they were so annoyed that the vendor ignored our instructions. In the ranking discussion, I spoke up that I thought the product was good, in spite of the bad demo. But we’re seeking a vendor partner, and the selection team felt so unheard that they weren’t willing to proceed with the option.
After the selection team has aligned on the finalist vendors moving forward, share with each finalist that they’ve been shortlisted for due diligence. Be very clear with vendors which items we’re working through in due diligence (scoping, pricing, reference checks, etc.) and the timeline we’re aiming to hit.
Make sure to advocate for yourself on the reference check portion of due diligence. (We have warned vendors that we’re asking for reference checks, but this always seems to be a scramble.) Reinforce the characteristics you’re looking for in a reference (rather than the simple industry match that vendors often default to), likely including the specific product mix we’re considering and other organization profile or use case attributes.
After the finalist vendors have acknowledged next steps and confirmed they’re willing to work with you on them, share with the remaining vendors that they weren’t selected as a potential fit for this opportunity.
Again, vendors will likely ask why they didn’t move forward. Since they have invested significant time in responses and demos, it’s certainly worth spending a few minutes to debrief with them. Again, be very careful in giving critical feedback. Imagine it being displayed in a courtroom or on a billboard – and choose your words carefully.
I recommend sharing that all vendors were scored/ranked after demonstrations, and other options were deemed a better potential fit. Use your best judgment in sharing other option names – if in doubt, don’t share the other vendors’ names. (Vendors often know the other options they’re typically up against, and it’s reasonable for them to want to know which options they lost out to. It’s also reasonable for you to not want to share too many details with them. Navigate with care.)
Advanced Class
Watch out for analysis paralysis in this phase. It’s very easy to have “just one more question” that we need answered before we feel confident moving forward. We won’t turn over every stone, but we’re aiming to address the most important ones.
As facilitator, take the opportunity to ground each conversation with the selection team in what we’re ultimately after and how these steps are helping us get there. This sets you up to lead prioritization discussions: like, if we have 20 follow-up questions posed by the selection team, we may need to identify the five most important questions for this stage of our analysis.
Also remember that we want to make the selection team feel heard and understood throughout the consideration process, even if we don’t take each next action precisely as they’re envisioning.
Tools & Artifacts
Demos & Vendor Evaluation goes smoother when we leverage:
Specific demo agendas, providing clear expectations for vendor teams and making it likelier that the selection team gets the information and insights needed to assess options well
Demo scorecards and rankings, making it easier to compare notes across the selection team
Demo recordings, allowing us to refer back to specific sessions as needed
