How to Get Structured Feedback from Employee Interviews
Ad hoc chats and “hallway conversations” might give you some great ideas. But structuring employee feedback with research plans and data analysis yields sharper insights that drive better results for your team.
Ben Jackson, March 6, 2018
“Talk to people.” In the startup world, the phrase has become a cliché: after all, who starts a business without meeting their customers? If you’re responsible for a product, qualitative research will help you avoid strategic mistakes. And if you’re responsible for a team, it can also give you deep insight into what drives your employees’ performance and the things holding them back.
But while you might already be spending hours in ad hoc conversations with customers and staff, there’s a catch: those informal chats aren’t the same as structured research. Even worse, they can actually lead you to the wrong conclusions by introducing biases or sampling errors into the data. Read on to find out why, and how you can avoid the traps to uncover real insights from listening to people.
Four Risks You Run by Relying on Informal Conversations
While informal chats might feel like they’re giving you insight (and even lead to some good decisions), they fall short over the long term for a few reasons:
1. Sampling Errors
By nature, informal chats only include data from participants who happen to end up speaking to leadership. There aren’t any controls in place to ensure the data is representative, and self-selection bias can skew results. For internal research, these dangers are very real: leadership may not speak with employees on the ground floor of the company, or feedback may only come from those they already know.
2. Framing Errors
Informal chats are often framed inconsistently. One interviewee might think they’re being asked about one specific aspect of the operation, while another might focus on a broad set of concerns. People vary in their willingness to give constructive feedback, too: most of us will tailor our answers to please the interviewer unless we’re asked to do otherwise.
3. Anonymity and the Observer Effect
More often than not, the person asking the questions in these conversations is also the one directly affected by the answers. This is not a recipe for candor. Just like no one wants to tell a founder their idea isn’t a game-changer, no employee wants to be the one who asks the CEO to work on their delivery at all-hands meetings.
4. Interpretation Errors
Even when data is collected with a representative sample, consistent framing, and guaranteed anonymity, different interpretations can muddle the results. With no record of the interview, building a shared set of agreed-upon facts becomes impossible.
Before You Start Interviewing
No one could summarize everything you need to know about research in one article. For a more comprehensive overview, I’ll point you to Erika Hall’s excellent book Just Enough Research. There is, however, a standard process that gets results.
Here’s what you’ll need to do if you want useful, accurate data:
Bring an Open Mind
Qualitative research is most useful when you use the data to construct a Grounded Theory. Building grounded theories works a bit like the opposite of the scientific method. Instead of starting with a hypothesis and then collecting data to validate it and forming a new one, grounded theory practitioners first collect data, then look for patterns and use the resulting insights to inform more data collection.
Because of this, the theories are “grounded” in real-world experience (hence the name).
Write a Research Plan
You can’t answer questions you haven’t written down. A research plan sharpens your thinking by forcing you to be specific about your goals, your audience, and your methodology. Your plan should outline the following:
- The questions you’re hoping to answer yourself
- The format for the interview (length, recording method, where it will happen)
- A pre-interview script explaining the purpose of the interviews and assuring the participants of anonymity if necessary
- A script with questions you’ll ask each participant
During Your Interviews
Your goal when interviewing is to make sure you’re getting useful data from people, and that you’re capturing it accurately so you can analyze it later.
Ask About Past Experiences
People are notoriously bad at self-reflection and predicting their own behavior, so stick to asking them for stories about the past. You’ll want to zero in on critical moments: “When did you first start thinking about…?”, or “What prompted you to…?” work much better than “What do you think about…?”
Pay Attention to Nonverbal Signals
Participants’ tone, facial expressions, body language, and rate of speaking will give you lots of extra information beyond what they’re saying. Pay attention for signs of discomfort or enthusiasm and use them to guide your conversation away from or deeper into specific subjects.
Be Prepared to Go Off-Script
At times, you’ll hear something surprising or unexpected during an interview that makes you want to dig further in. When this happens, don’t be afraid to deviate from the script! These tangents can yield some of the most useful insights, and you may find yourself updating the script as you uncover flawed assumptions and new questions.
Record and Transcribe Every Conversation
If at all possible, record audio, video, or both, since note-taking distracts from the conversation. Play it safe: charge your devices and make two copies of the recording.
For in-person interviews, use a podcast-quality microphone like the Blue Yeti, and make sure to adjust the settings to suit your environment for good sound quality. For remote interviews, get a good call recording app, ideally one like Zencastr that records both sides independently.
If you can’t record, take lots of notes, focusing on direct quotes that sound familiar, surprising, otherwise significant.
Transcription is time-consuming, difficult work, so if you can afford a transcription service ($1-3/minute), it’s worth the investment. If not, Descript or Trint will get you started with a decent automated transcription for a fraction of the cost, and both have editing interfaces that let you listen to audio while cleaning up.
Analyzing and Presenting Your Findings
Look for Trends in the Data
As soon as you start generating transcripts, it’s time to start finding common threads across interviews. For best results, use an online annotation app:
- Reframer is a decent free option.
- Dovetail has a great UX and is easily the best paid option for teams on a budget.
The process of labeling sections of text for analysis is called coding. The labels can be anything that comes to mind when reading sections of the text: some examples from real projects include “coffee”, “silos”, and “ad hoc”. You should expect to spend at least as much time coding as you spent on each interview.
Draw Insights from Recurring Themes
When you’re done coding, you’ll have a big pile of tags, and some idea of which ones show up most frequently. Start putting them into groups. Don’t think too hard about what to name each group—just concentrate on which things feel like they belong together.
In the process, you’ll probably start seeing some macro themes. Perhaps “coffee” helps bring people together who normally work in “silos”, a behavior that causes lots of “ad hoc” requests between teams that don’t communicate. If you’ve been careful about how you collect the data, the patterns will reveal themselves eventually. If you’re using Dovetail, you’ll want to convert some of these tags to insights so you can add further notes to them.
Report Your Findings
If you plan on doing anything with your newfound insights, you’re going to need to present them to leadership. By now, you should see some themes emerging in your insights. You might have a section on “Challenges”, and another on “Opportunities”.
Once you’ve identified and named some themes for your insights, you’re 90% of the way there! This is your an outline for a slide deck or written report:
- Each insight should come with a few quotes that illustrate it from different angles.
- Don’t overwhelm the reader with every single quote. Just pick the best ones.
What to Do Next
This is just a taste. There’s a lot more to read and learn before you’re ready to go out and start interviewing people:
- Read Just Enough Research, and make sure you’re conducting your research ethically and by the book.
- Steve Portigal’s Interviewing Users will help you make sure you’re getting the most accurate data possible from your subjects.
- If you’re willing to slog through a textbook and looking for a thorough introduction to the theory, Constructing Grounded Theory is worth the effort.