Section on Bias (3rd party clean)
Section on skillset needed
Section on software needs.
You need to find a way to invite as many closed/won and closed/lost participants into interviews. Actually getting people into interviews is harder than it sounds. You’ll need to provide an incentive like a gift card, and for an average recruiting program you can expect between 10-20% of participants to accept your invite to join a call. You also need to think about a couple key tools. One is an email platform that will help you email multiple invitations to your list of contacts, a Linkedin membership (Sales Navigator) that will let you send messages to your recruits, and a scheduling tool like Calendly to keep track of all of the scheduling complexity that will ensue.
To conduct a good interview there are a number of things to keep in mind. First, you’ll want to create an interview guide and write down all of the questions you want to ask. The most important question will be why they did or didn’t choose your product, but you’ll also want to ask about the experience with your sales team, pricing, product, etc.
Make sure and keep the interview carefully focused on their experience. Interviews can easily slip into a space where people are offering you their opinion about your product, about your UI, about the market, etc. Almost all of the feedback is unreliable. Keep your questions tightly focused on “what happened” in their experience with your product. “Why didn’t you choose us”, “what did you like and dislike about your sales experience”, etc.
The analysis of the interviews is probably the heaviest lift in the whole Win/Loss process. Each interview is about 30 minutes of conversation all laid down on the page in transcript form, and you’ve got to move through each of the interviews individually to pull out the valuable parts, and then aggregate all of those valuable parts into high-level trends that can be understood by the various teams at your company (product, sales, executives, etc). A good rule of thumb is that if you conduct a 30 minute interview, you’ll need at least an additional 30-60 minutes of analysis time of the interview itself. So if you conduct 10 interviews, you should budget at least 10-15 hours of time to fully process them.
If you’re at the very beginning of thinking about a Win/Loss program, there are two different routes you can consider. You can build a program internally, using your own team and resources. Or, you can hire a 3rd party Win/Loss provider, and lean on their expertise to do most of the heavy lifting. Each approach has pros and cons, I’ll do my best to explain the most important aspects here.
If you’re building and running your own program you don’t need to get any budget approvals or go through a vendor selection process. All you need to do is get approval from your boss and team to start building the program. You can get off and running quickly, and that’s valuable. Additionally, the cost of a Win/Loss program can vary widely from vendor to vendor. Some of the more well-known vendors out there are, in our opinion, very overpriced. The Win/Loss service provider market is new, and it’s not an efficient market, so you have to proceed cautiously when it comes to spend. You’ll find some providers that charge double the price for half the quality of service. As the market matures these inefficiencies will begin to disappear, but today that’s where things stand.
As you’re trying to evaluate cost, a helpful lens to look through is the “cost of a single interview”. If you’re talking to a provider and they’re providing you with pricing information, break down the price to a “per interview” price, and do the math on the following inputs: time spent recruiting, time spent interviewing, time spent analyzing, access to their software (if they have their own software), interviewing skill-set (which the provider will have, and your team probably won’t). They might pitch you on additional services like surveys and such, but the nuts and bolts of cost really boils down to the interview itself. As you think about these inputs, keep in mind that doing these interviews is a serious time input, and the skill-set required to do them well is a real thing. Though if you do the interviews yourself, you’ll eventually get there once you’ve completed 40-50 interviews. And that the total time input for a single interview is 30 minutes for the interview, 30-60 minutes for the analysis, and about 15 minutes for the recruiting (see our article on recruiting if you haven't already). With that information you should be able to get to some helpful conclusions on cost.
Hiring a good Win/Loss provider on the low-end will cost you around $60k a year (that’s where we typically land), but other providers charge $120k per year and up, so it can get expensive. That said, running a robust Win/Loss program can become a full-time job, depending on how many interviews you want to conduct. And the outcome of an internal program can be limited since you're not inheriting the lessons and best practices from thousands of interviews and many clients. So if you’re planning on building a robust program, the cost of a 3rd party might equate to what you’d pay a full-time salaried employee. But again, the best way to think about cost is to break it down to the cost per interview, and do the math from there.
The amount of time it takes to build and run a Win/Loss program is substantial. It’s way more time than most people estimate. In addition to the obvious stuff (scheduling and conducting the interview, recruiting participants, analyzing the interviews), there are also plenty of hidden costs. On average, 25-30% of interviews don’t show up for their interview, so you’re sitting there for 20 minutes waiting for them to show up. You can be working on other things, but it tends to feel like broken time, and it’s difficult to be productive in those windows. You’ll have an executive who comes to you and says “what do people think about X new feature”. And suddenly you’re off on an unplanned 3 hour adventure scrubbing each interview for any morsels of information about that feature. You have rescheduling time, you need to get each participant their gift card as a thank you for participating. If you’re sending an exit survey after the call, you’ll need to get that sent, and collect and organize everyone’s answer to the survey, as well as combine the survey data with the Win/Loss data in a complimentary way. It’s a lot of time, and can easily turn into what feels like a full time job.
With a 3rd party, 90% of the time investment gets outsourced to the service provider, so it’s a major win if you’re trying to save time. A big part of the cost for a 3rd party provider is this time component. It’s just a lot of time and energy, there’s no way around it.
Interviewing participants well takes real skill. On average, it’ll take you at least 40-50 interviews to start really building out your skillset. When first started Kaptify we had a lot of clients who wanted to join our interviews because they were keen on seeing and hearing what people had to say. Seems innocent enough, right? We don’t let clients join interviews these days because on average, unskilled interviewers (which is basically everyone) have a hard time distinguishing what information is important from what information isn’t. It seems like an easy enough thing. But once you’re actually in a real conversation, you’re dealing with a brand new personality whose bringing all kinds of new dimensions to the conversation. If you don’t guide them correctly to the right questions, go deep enough in the right areas, and avoid the pitfalls, you can easily fill up 30 minutes of time with very little valuable data. You can also unintentionally pollute the data by simply asking for their opinion about things instead of keeping the conversation focused on their experience with your product.
The big difference between a 3rd party and doing it yourself is that as an internal analyst at you probably don’t have these skills, and it’ll take some time to get them, but you will eventually get them. Just plan on a learning curve of at least 3-6 months. Once you’re past the 40-50 interview mark you’ll start getting the hang of things.
When it comes to the big differences between a 3rd party provider and doing it yourself, ultimately this is the big difference. There are a small handful of Win/Loss companies that have built their own software to handle Win/Loss feedback interviews. The reason we’ve done that is because Win/Loss data-sets are really valuable, really big, and require a lot of massaging. If you think about a single interview, you’re talking about 5-10 pages of text on the page, and a 30 minute video. A single interview is a lot of data to review. And a Win/Loss program is trying to scale to hundreds and hundreds of interviews. You can start to see how keeping all of your interviews in Google Drive and Google Docs eventually breaks, and you stop being able to find the insights you need because the data is simply too scattered and unstructured. There are tools out there like Dovetail that can be helpful, and tools like that are usually better than just Google Docs, but they’re also severely limited given all of the nuanced needs a Win/Loss program has. When considering whether to build in house or use a 3rd party, if the cost of the 3rd party seems reasonable, this is really the thing that should swing your decision one way or the other.
Building a high-quality and robust Win/Loss program requires adequately covering all of these bases. If you want to drive high ROI, you need to figure out whether to outsource your program or do it internally, but either way, you need to kick the tires on all of the above issues to make sure you’re building a program that’s going to deliver the high-quality actionable insights you need to drive meaningful ROI.
