All qualitative data doesn’t need to end up in a formal written report to make it useful. In fact, very often the opposite is true.

Sometimes I conduct focus groups or a set of interviews for an organization whose decision makers want to use the findings right away. They need the data to develop policy or improve programs, products or systems that need fixing sooner rather than later. These clients know their industry, understand the value of hearing directly from their constituents or staff, and have some understanding of the issues they want to address. They don’t need a lengthy traditional report with background, literature search, methodology, and conclusions.

What they need instead is someone to collect the data in a rigorous and thorough manner and organize the findings in a format they can easily access. I like these kind of “hit-the-ground-running” projects because they are so utilization focused. The questions are generally pragmatic, intentional, and likely to generate useful information. And, rather than wasting away in a thick report that sits for months on decision-makers’ bed stands, findings are generally put to use within a month or two after they are generated.

So in place of a report, I organize the data I have collected for the client in a multiple page Excel spreadsheet. On the spreadsheet, each response is entered on a separate line, coded by category and organized by question. Each response is also accompanied by face codes—gender, race/ethnicity, age, longevity in the profession/work place, and title/position—that may be helpful for further data sorts by subgroup.

But the hand-off doesn’t end there. Because qualitative results require a bit more skill to interpret and apply than quantitative data, I like to spend time talking with the client about how to get the most out of the data and use it in an objective, thoughtful way.

Over the years I’ve developed guidelines that I encourage clients to consider as they sift through the spreadsheets. Here are eight of the ones I never fail to mention:

1. Be careful not to over-interpret the data. Don’t read more into the data than can actually be substantiated by the findings. It’s a natural tendency to “fill in the blanks” with data from our own mental files. Instead, be curious about options other than the ones you’re most familiar with.
2. Resist the urge to quantify qualitative data. Because many of us are more comfortable making decisions based on numbers, we have a tendency to count up the number of times a particular comment was made. Qualitative data can “point” us to several areas that need addressing, but cannot quantify the extent to which any of those areas is an issue in the larger population. It’s not that similar frequently occurring comments don’t indicate an area to which we should pay particular attention. It’s just that open-ended responses collected from a small number of participants may not be significant. On the other hand, an issue brought up by two people can be bigger than one brought up by seven people, if the two people have an in-depth understanding of the issue and speak for several others not in the room. When it’s important to quantify qualitative findings, follow up with a survey distributed to the bigger population from which focus group or interview participants were drawn.
3. Use your whole brain to interpret the findings. Interpretation is tricky work. We need to capitalize on the logical, analytical, objective left brain while at the same time involving the intuitive, thoughtful and subjective right brain. If we only needed the logical left brain, we could have a computer do the analysis for us. But that’s just not the case. Interpretation of qualitative data depends on a combined creative-rational process that integrates the best of both sides of our brain. So let your left brain consider the frequency of similar comments, the best categories for organizing the information and if certain comments make sense in light of other comments. At the same time, allow your right brain the freedom to integrate information from outside sources, draw on previous experience with similar topics and detect emerging patterns and themes.
4. Don’t get hung up on any one response. Qualitative research is not investigative reporting. We’re not looking for one or two key clues that lead to a complete understanding of the issues. Instead, try to capitalize on the collective wisdom of the group of interviewees or focus group participants.
5. Look for convergence of data from different sources. This occurs when multiple respondents provide a similar answer to any one question, when respondents across diverse roles or age categories say the same things, or when responses to different questions all point to the same conclusion. These points of convergence signify major themes in the data. This is where the power lies in qualitative data. Pay particular attention to what these convergences tell you about the population you seek to serve.
6. Pay attention to saturation, but not too much. When no new information is revealed about a topic with each subsequent focus group or interview it generally means respondents have reached a point of saturation. In other words, there isn’t necessarily anything else new to say about the topic. Saturation, like convergence, adds power to the finding. But be careful not to fall into the counting trap. Just because the same point is made 38 times (33 times more than it needs to be made) doesn’t mean it is any more relevant to the issue than a point that hasn’t reached saturation.
7. Resist the tendency to identify the individual behind any one response. Particularly in small organizations and rural communities, it can be easy to identify some individuals by the tenor of their response or certain words they always use. Set aside what you know, or think you know, about who said what so you can look at that comment as objectively as you do all of the other responses.
8. Contextualize the data. Consider qualitative findings within the context of other available information. This may include other data sets (survey results, etc.), past history in addressing the problem (successful or otherwise), constituent demographics, organizational politics, budgetary realities and internal capacity for addressing the issue. What the data means or how it gets used can change dramatically depending on any of these contextual variables.

Nothing excites me more than hearing about how individuals and agencies use qualitative data to make changes in their organizations, programs and products. Based on your own experiences, do you have other suggestions to add to the list?

Leave a comment below and let me know if you’d like me to expand on any of these in a future blog.

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

When we want to know more than the boxes and circles on a survey can reveal, we turn to the open-ended interview. Thoughtfully designed and skillfully delivered, interviews generate an infinite possibility of responses—more than we ever thought to ask.

By its nature, the interview is a fluid process. But, not unlike surveys, interviews require pre-designed questions to steer the questioning in a consistent way. This ensures that the short time we have with the interviewee is used efficiently and we use the same line of questioning with each interviewee.

The pre-designed interview guide is used for both individual interviews and focus groups. It is generally one of two types:

• A checklist of possible topics
• A fixed set of open-ended questions

Checklist Approach

An experienced interviewer may use a “checklist” rather than a prescribed set of questions to conduct the interview. By a checklist, I mean a list of topics to be explored under one broad category, not a list of yes-or-no items. For example, a study exploring the needs of seniors to “age in place” in a new public housing project might include the following checklist:

Housing features:

• Meals
• Transportation
• Housekeeping
• Safety
• Socializing
• Medical care

Using a broad open-ended question, interviewees might first be asked to describe the housing features they feel would be necessary to meet their needs. A good group of interviewees would probably cover many of the topics on the list with little or no prompting from the interviewer.

The experienced interviewer would use the checklist as a flexible tool to move the interview through the topics, not necessarily in the order they are written, but in response to where the interviewee (s) takes the discussion (within given topic areas).  At the same time, the checklist helps the interviewer keep an eye on out for topics that are not mentioned by the interviewee. And finally, the interviewer can weave those topics not covered into the interview at the appropriate  break in conversation.

As you might imagine, it takes a good deal of preparation and skill to conduct an interview by checklist. The interviewer must be thoroughly familiar with the items on the checklist and the intent of the study. S/he must also be adept at spontaneously developing relevant questions, keeping track of what has been covered and what has not, engaging each respondent appropriately, making sure the interview flows smoothly, and keeping the entire interview focused on study goals — and doing all of it simultaneously!

The main benefits of the checklist approach include:

• The dialogue can evolve naturally, rather than according to a prescribed set of pre-formed questions.
• The interview can be customized to the interviewee’s experiences and conversational style.
• Topics that prove especially rich can be probed more deeply than others.
• Those topics no one thought to ask about can be allowed to emerge.

Fixed Question Set

Instead of a checklist, most individual interviews and focus groups in applied research settings are conducted using a fixed set of fully structured questions developed in advance of the interview. These questions (usually between five and ten per interview) are written as open-ended complete sentences and are carefully worded to elicit a broad range of responses. Accompanying each question is a set of prompts designed to steer the conversation into areas of particular interest, especially those not raised by the interviewee.

Here’s an example of a fully structured question on the same topic (aging in place) used in the above checklist example:

Other senior adults have told us they want to grow old in an environment where they feel safe and can easily get to the store and bus stop. They want to be independent as long as they can and want to have important services close by. What would the ideal “aging in place” environment look like for you?

Prompts: What about crime? Noise levels on the street? Public transportation needs? Proximity to your doctor’s office, post office, library, etc.? Walkability?

In contrast to the checklist question above, this question focuses on just one aspect—the individual’s exterior environment. Other questions might be designed to address the interior environment, for example, or medical support needs (e.g., medication reminders). With the fixed question approach, each question in the set is fully developed and can stand on its own.

Questions in fixed question sets are written in an engaging conversational tone and sequenced in a way that follows a natural progressive line of thinking. All questions are asked the same way (usually read out loud) and in the same order. Prompts are used if necessary to raise topics that do not naturally arise during the open-ended interview.

Benefits of the fully developed question set include:

• All respondents answer the same question, increasing the comparability of responses across interviews or focus groups.
• Decision-makers can see exactly what will and won’t be asked, giving them an opportunity to provide input on questions and increasing the likelihood they will use the results.
• A less skilled interviewer or lay person can conduct the interview.
• It’s easy to make sure that all intended topics are covered.
• Analysis can be easily organized to follow the sequence of questioning.

Which Interview Guide is Best?

Two factors are important in answering that question: the anticipated level of interviewee engagement and, secondly, the skill level of the interviewer. Many interviewees have a lot of knowledge but it’s often tacit and takes a bit of probing and prompting to reveal. Others who may be considered experts in the topic (doctors, parents, program directors, etc.) can easily articulate their experiences.

Regarding interviewers, veterans are adept at handling a variety of question types and engaging reticent interviewees with minimal support; less experienced or lay interviewers benefit from a detailed guide with written out prompts and probes. So, in general, a checklist works best in situations when the interviewer is highly skilled or interviewee (s) are openly knowledgeable and articulate; a  fixed question guide works best in situations in which the interviewer may be less experienced or when respondents require lots of probes and prompts to reveal what they know.

A Third Option: The Hybrid Approach

If I were conducting my own research, it would be great to use a checklist approach—I could make adjustments in the interview guide as findings were revealed and not have to consult with anyone. But generally, that’s not the case. I’m usually hired by a client group to conduct a study in which the study agenda is theirs, not mine.

To meet the multiple needs of the most client groups, I’ve found that using a hybrid approach (combining checklist and fixed-question approach) is the most effective. I start by constructing a detailed question guide with probes (fixed question set approach), which I share with the client group before conducting any interviews in order to gather their input. I also let the client know immediately after conducting the first interview or focus group if a question doesn’t seem to be working as intended so we can eliminate or change it.

When I’m actually in the interview/focus group, I ask questions in some variation of the original, but almost never word-for-word (checklist approach). I like the flexibility to adjust my questioning to the tenor of the interviewee(s) and allow them some latitude in how the topics roll out. It makes for a more naturalistic approach that seems to increase the comfort level of interviewee (s) resulting in a deeper conversation. That way I also have the opportunity to expand on certain topics we didn’t know would be so rich or explore areas raised by interviewees that neither the client or I thought to include.

But regardless of the study type or topic, the best question guide is the one that not only includes the right questions but enables the interviewer to be successful in asking those questions with a live audience.  Any approach that produces high quality, usable data is the right approach.

Wishing you great success in designing and executing all future interview guides!

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

Those of you who regularly follow my blog know that I have a thing for listening. I believe good listening is fundamental to good qualitative work.

The January-April 2012 edition of The International Journal of Listening includes research on the validation of the Ford, Wolvin, Chung Listening Competence Scale (Mickelson, W.T and Welch, S.A., pp. 29-39). The research was inconclusive but the authors said that there was a  “well-established theory and logic behind the dimensions of the Listening Competence Scale.”

That’s good enough for me. I found the Scale valuable in assessing my own listening skills (or lack thereof) and in identifying specific areas that I need to sharpen. I include it here in its near entirety. I left out one of the six dimensions—Therapeutic Listening—because I didn’t think it applied to qualitative work. The other five dimensions are highly applicable though.

The tool is simple and requires just a few minutes to take. Give it a try and see how you fare. Each item is scored on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree) with a midpoint of 3 (neutral).

Discriminative Listening

I can easily identify somone’s feelings when s/he is speaking to me.
I recognize when someone is not telling the truth.
I can correctly interpret someone’s facial expression.
I can recognize when someone is withholding information from me.

Critical Listening

I express opinions that differ from what others express.
I critically evaluate the content of information that is presented to me.
I carefully assess information as it is being shared with me.
I give feedback to others to let them know what I think of their message.

Comprehensive Listening

I ask follow-up questions until I fully understand someone.
I correctly recall information a few minutes after I hear it.
I ask for additional information or explanation as needed.
I pay close attention to make sure I understand what is being communicated.

Appreciative Listening

I appreciate hearing another’s point of view.
I enjoy listening to others.
I listen with an open mind to what others have to say.
I appreciate what others have to say.

Attending Behaviors

I maintain eye contact with someone while s/he is speaking.
I give someone my complete attention when s/he is speaking.
I maintain an attentive posture while someone is speaking.
I respond nonverbally to let someone know I am listening.

What other dimensions or items would you add?

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

If you’re a doctoral candidate struggling to define your qualitative study, a faculty member who needs help implementing the qualitative component of a newly awarded grant, or a large public agency interested in using qualitative methods to design programs or policies, then you must meet Dr. Asher Beckwitt!

Dr. Beckwitt is a qualitative research consultant and the CEO of Asher Consulting, LLC.   She created the company in 2005 to empower students, scholars, and professionals to conduct excellent qualitative research. She helps researchers at any stage in the process, from conceptualizing the topic, to using qualitative data analysis software, to writing up the findings. She provides live and recorded online webinars on a variety of qualitative topics as well as a number of customized services. She’s truly a researcher’s research consultant.

I had the pleasure of interviewing Dr. Beckwitt recently to see what I could learn from her. Here are excerpts from our conversation:

Eliot: I’m curious about how you developed an expertise for working specifically with students and faculty. How did you get started?

Beckwitt: About 11 years ago when I was pursing my Master’s degree in Sociology, I was hired to create a campus wide qualitative research consultation service at George Mason University in Fairfax, Virginia. At that time, I worked across disciplines with faculty members as well as students. I worked with researchers from conflict analysis resolution, nursing, even bioterrorism. Anybody who was working on a qualitative project of any kind could come and see me. I would instruct them in what methods to use and on the questionnaire design or development of their project. I would also instruct them about data analysis and show them how to use NVivo.

Eliot: I’m often surprised about the number of inquiries I receive from doctoral students looking for help with their qualitative research. They don’t know how many focus groups to conduct or who exactly to include in their study. Some tell me they don’t have a faculty advisor that really understands, or appreciates, qualitative research. Is George Mason unique in providing this type of service?

Beckwitt: Yes, George Mason truly is unique.  There are only a handful of schools in this country that offer that kind of assistance.  Oftentimes doctoral candidates receive very little training in qualitative methods. Some doctoral programs don’t offer more than just a half of a class—if you get even that much—in qualitative methods.  Other doctoral programs have a full class, but you’re bouncing from one method to the next.  You might spend two weeks on grounded theory, two weeks on phenomenology, two weeks on case studies or whatnot, but you don’t ever truly get a fully in-depth understanding of what qualitative methods are, let alone how to actually implement them.  Others receive no training at all in qualitative methods, but know that they want to use those methods for their dissertation.

Eliot: And now you’re providing that same service outside the walls of George Mason?

Beckwitt: Yes, I went independent in 2005 and created Asher Consulting, LLC. In the past two years, Dr. Jennifer Carter joined the team. She’s a licensed psychologist and does most of our interviews, focus groups, and participant observation.  We have over 11 years of experience.  What we offer even goes further than what I was able to offer at George Mason.  We provide researchers with the tools and support they need. That could be very, very early on with developing their proposal.  Or it could be later on, with data analysis, coding, writing, NVivo training, or editing. We help them at any stage of the research process.  What I try to do is instruct them in the appropriate use of the [qualitative] methods to use to address their specific research questions and the gaps within the literature. Out of that process should flow the research questions for their study and the methods they need to address those questions.

Eliot: So talk more about how you assist students and others with choosing and using qualitative methods. I know you offer several online methods courses and workshops.

Beckwitt: Yes, we have two sides to the company: one is our education side and the other one is our services side—and these two actually work together. On the education side, we have online live and recorded webinars on qualitative methods. We have a webinar on introduction to qualitative methods that distinguishes between quantitative and qualitative methods. We have one on writing a qualitative dissertation proposal, one on phenomenology, and another on qualitative content analysis. In that webinar we talk about trustworthiness, credibility, validity, and how they differ from quantitative methods.  We also have another workshop on case study methods and one on grounded theory.  Finally, we offer an applied approach to the NVivo software.  Our approach to NVivo is different than what you might find elsewhere.

Eliot: Tell me more about your different approach to NVivo.

Beckwitt: We don’t just show people how to click through the program and utilize the tool. We teach them how to implement multiple methodologies within that tool. In other words, we teach researchers how to code their data using their chosen methodological approach in addition to showing them the basic functions of the program such as how to set up a project, how to query, how to code within NVivo, how to do memos and things like that. We offer one-on-one customized sessions and/or group sessions for teams of researchers. These sessions are offered online or in person.

Eliot: Can you walk me through a typical student project? I know that each situation is customized, but what could a student expect if they decided to work with you?

Beckwitt: It would depend upon what they’re actually working on and where they are in the process.  For example, one of the things that we might do is assist them with their proposal.  If they’re still in the early stages of figuring out their proposal, we might assist them with understanding the process of constructing a literature review.  We might even assist them with narrowing their dissertation topic. Sometimes folks come up with very broad topics or three projects in one. Then we can help them brainstorm and think about research questions. Next, we help them to identify the appropriate methodology to address these overall research questions. If they’ve already submitted the proposal and have gotten comments from their committee, we help them with addressing those comments. Once we get past the proposal phase, depending on what they’re doing, we oftentimes will do the transcription of the focus groups, interviews, or whatever data they’re collecting. Then it comes down to the data analysis portion. By this point hopefully they will have taken some of our workshops in terms of understanding the methods. In many cases, folks want to learn NVivo.

Eliot: So this would be one of those times when you provide customized instruction in using NVivo?

Beckwitt: Yes. In this case, we would conduct two or three one-on-one web conference sessions to train them on how to use the Nvivo software to analyze their data. We would help them set up their project, and show them how to start and think about coding. Then they would go off and do all the coding themselves. When they come back, we would go through another session, possibly two, to help them combine the codes to think about larger categories.  We won’t tell them what to put where, but we will facilitate their understanding of the methods and ways of looking at the data. Then we will oftentimes do some kind of editing services, either help them perfect their APA style or with the organization and style of their dissertation presentation.

Eliot: So how do most of the doctoral candidates you work with pay for your services?  Are they able to use student loan money? Are they on research grants that can help pay for your services?

Beckwitt: We try to make it reasonable. That was originally why I started this company: for doctoral candidates. Some folks have grants; some folks have loans or things like that. Others work full-time or part-time and are able to fund things that way.  We offer very, very generous payment plans. As long as folks are reasonable, they can negotiate the terms of their payment plan with me.  I’ve literally had people come in and pay minimal amounts for a year-and-a-half to pay it off. I’m very flexible as far as assisting folks.  We do also accept credit card services through our website as well.

Eliot: I know that you’ve now expanded your services to include non-profits and government agencies, nationally and internationally.

Beckwitt: Yes. With non-academic clients, we’ve done things like run an entire study.  They’ll literally contract us out to do the entire study, or pieces of a study. We’ve structured questionnaires or focus group guidelines and then sometimes actually gone out and run interviews and focus groups, or conducted participant observations.  We provide instruction regarding the methodology that they’re using to conduct their study and write up their results. Also some faculty members have hired us to run the study for them.  We have done all of the data analysis, all of the data coding, come up with the themes depending upon the method, and then done the entire write-up and editing of the report.  We do the full gamut—all the transcription, everything across the board.

Eliot: What I love about your service is that it’s a “one stop shop.”  I mean, you pull it all together for them.  You have the expertise in all of the relevant areas to be able to bring that coherency and consistency to the whole process, which I think is wonderful.

Beckwitt: That was actually my dream when I created this company. My goal was to always design it so that it was easy for students, faculty, companies or non-profits — anybody, really — to be able to access either one or all of our services so that people can have that continuity if they would like it, or they can just drop in and get their transcripts done, or get their coding done, or whatever it is that they need.

Eliot: If you could distill it down to the most valuable thing about your services, what would that be?  What’s the real value of working with you?

Beckwitt: My goal is for people to produce good quality qualitative research. My goal in working with students is to make sure that they have a very clear understanding of the methods that they’re using and why they are using those methods so that during defense, or with any subsequent publication, they can back up what they’re doing, why they’re doing it, and have the rigor behind it. My goal in working with professionals is to make sure that they are provided with a very good quality product that they can either publish in a peer review publication or can use to implement the proper policies, programs, or whatever it is that they are working towards.  I strive to make sure that their piece of work is solid.

Eliot: You really help folks build capacity for understanding qualitative methods, don’t you? There’s this enduring value in what you provide them.

Beckwitt: Yes, it goes beyond [just doing the research for them]. I mean, I’m more of a methodologist and I enjoy doing and teaching the whole qualitative research process, but I want to make sure that people are coming away with something good and substantial.  I’ve seen some qualitative projects where I haven’t truly understood what methods those people are using, and so my goal is to make sure that people have a good quality project and that they can back things up appropriately.

Eliot: I like the fact that you not only help people create a really high quality product, but you help them have the confidence in that product, as well.

Beckwitt: Exactly, exactly. My goal is really—because I’m an educator—to educate folks, not only in terms of okay, here are the results, but also here’s why the results came out the way they did.  I want people to understand and be able to say, “I used “X” “Y” and “Z” method, here’s the rigor behind that process, and here’s what you can do with that.  Qualitative results are not generalizable, but here’s a very rich textual description as to what is going on with this situation and we can take this and learn from it.”  Basically, my goal is for folks to feel confident that they can use their qualitative research to create programs, implement policies, or have a very strong dissertation.

Eliot: I think that’s especially important in the research world in which we live where qualitative methods aren’t always as valued as quantitative methods.

Beckwitt: Right, absolutely.  I think that’s even more of a reason I emphasize the rigor within qualitative methods. People automatically think that qualitative methods are just fluff or they don’t necessarily give it as much weight as quantitative methods.  I think that there’s a lot of lack of understanding as to what qualitative methods actually are, and the different nuances between grounded theory and phenomenology and the different thoughts, ideas, and philosophies behind those different methodologies.  I do work a lot with quantitative researchers and help them to understand the differences, too, because quantitative and qualitative analyses are like comparing apples to oranges. Some quantitative researchers assume that the same steps will go into a qualitative project, but it’s completely different.  They’re completely different methodologies.

Eliot: If a doctoral student, faculty member, or organization wanted to enlist your services, how might they go about contracting with you?

Beckwitt: Our services really depend upon the project and what folks actually need.  If people were just interested in webinars, those have a set fee of $100 for a professional or $80 for a student. As far as one-on-one sessions with NVivo, coding, data collection, editing, or any of those kind of services, those fees are all based on a project basis, so that’s how they’re assessed.  We do provide free estimates for all of our clients. We are based in Maryland, but we work with clients nationally and internationally via web conferencing. The best way to contact us is via email.  My email is asher@asherconsult.com.  Or folks can find more information through our website at www.qualitativeresearch.org, or by phone at (703)577-3689.

Eliot: Thank you Asher. It’s been really great to learn about the much needed services and trainings you provide. I look forward to taking some of your online courses myself very soon!

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

Most parents understand the difference between surface-level and deep  listening all too well. Listening for details about what exactly their teenager will be doing, and with whom, are just as important as hearing their teen’s promise to be home no later than midnight. When they can take the time to do it, parents know that deep or advanced listening can have real payoffs. The same is true for qualitative researchers.

Most of the listening we do as we go about our everyday lives does not require a high level of skill. We interpret the words and gestures of coworkers and family members quickly and automatically with little conscious awareness on our part. Most of the time this surface-level listening is all we need to interact effectively.

If, however, we are conducting focus groups or interviews with individuals we have never met and from whom we want to obtain meaningful data, we need more advanced listening skills. Advanced listening is different from ordinary listening in that it involves in-depth processing of the messages we receive. It’s a cognitively demanding activity that requires a mindful, systematic scrutiny of the message and its source.

So what does it take to engage in, or learn to engage in, advanced listening?

Dr. Brant R. Burleson of Purdue University has identified several factors associated with the deep listening process in his recent article in The International Journal of Listening (vol. 25). In the article, he points out several situations in which deep listening is essential, such as uncertainty about how to interpret a message or confusion about the speaker’s motivation for saying what they’ve said. The advanced listening factors Dr. Burleson has identified are equally applicable to qualitative data collection. Here are five of those factors along with my tips for implementing each one in a qualitative setting:

1. Motivation. Motivation deepens listening.  For example, we are more likely to be motivated listeners when we strongly desire accurate information or a full understanding of the matter from the speaker.

Tip: Be committed to collecting the most accurate data and seek to truly understand what your interviewees are trying to convey.

2. Thinking. Some people have a high need for cognition—the tendency to engage in and enjoy thinking. Others who are less patient prefer  “cognitive closure”—the need to make judgments or decisions quickly. Those with a tendency toward cognition, or at least a low need for cognitive closure, are more likely to process information deeply when listening.

Tip: Don’t come to the interview or focus group with preconceived notions or be in a hurry to find the answers. Instead, bring your curiosity and “need to know more”.

3. Focus. Distractions of any kind have been shown to undermine the ability to process messages deeply.

Tip: Conduct interviews and focus groups in a quiet, pleasant environment with good lighting and a comfortable temperature. Be sure to also turn off mentally distracting thoughts that compete for your attention (which is sometimes easier said than done!).

4. Preparation. The more pre-existing knowledge we have about a topic or individual, the easier it is to deeply process new information.

Tip: Learn as much as you can about the study topic and the general characteristics of the individuals you will be interviewing before walking into the focus group or interview.

5. Person-centeredness. Acknowledging and legitimizing the feelings and perspectives of the respondent within the broader context of the questioning increases the complexity of our thinking and therefore the depth of our listening.

Tip: Consider the feelings and context of the person from whom you are collecting data to achieve a better understanding and deeper meaning of that individual’s words.

Short of confidence intervals and p values, your ears are your best tools for collecting accurate and meaningful qualitative data. Maximize their effectiveness by embracing a determination to collect accurate data, taking your time to think through responses, reducing distractions during questioning, knowing your topic and subjects, and considering the feelings of the person in front of you.

What are your experiences with advanced listening?  What would you add to the list?

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

“Scheduling” has to be one of the most common and least loved of all activities in today’s society. Whether it’s setting a business meeting with ten managers or figuring out who will pick your son up from soccer practice, arranging for multiple people to be in the right place at the right time can be a daily headache.

Organizing an in-person focus group is no different.  In fact, there may be times when it is nearly impossible to bring participants together in person.  But there is another option: telephone focus groups. The same principles used in face-to-face groups, combined with conference call technology, can be just the solution.

As a general rule of thumb, focus groups are best conducted in person. But when that’s not possible, telephone groups are a great alternative for the following situations:

• Geography: Telephone focus groups are ideal when participants are dispersed over a wide geographic area.
• Convenience: People with busy schedules can attend a telephone group at any time of day from the convenience of their offices or homes.
• Inclusion: Telephone groups make it possible for those who are disabled or lack transportation to participate in the conversation.
• Skilled facilitator: A highly skilled focus group facilitator can be enlisted from virtually anywhere when one is not available locally.
• Limited budget: Telephone focus groups are more economical than face-to face groups since costs such as meeting rooms, food, baby-sitter, travel, etc. are not incurred.

For the most part, telephone groups adhere to the same principles as in-person groups. They are open-ended discussions with small groups of people led by a skilled facilitator. The question guide is prepared with as much care and the same rules of confidentiality apply. The group is recorded and proceedings transcribed and analyzed just as traditional focus groups. But there are a few differences based on the different communication medium.

Smaller, Shorter

Telephone groups are generally smaller in number and shorter in length than face-to-face groups: three to five participants instead of five to seven, and 40 to 50 minutes instead of 60 to 90 minutes. The reasons: 1) it’s hard to maintain participants’ attention on the phone for longer than an hour, and 2) it can be difficult to manage a large group when visual cues are not available.

More Structure

Telephone focus groups require more structure than face-to-face groups. It’s hard to tell when someone wants to make a comment or is completely befuddled when you can’t see facial expressions and body language. And some people, like myself, have a harder time understanding what they hear without visual cues.

To help participants track the discussion, I send out the question guide in advance so they can “see” the questions as we move through the discussion. It also gives them an opportunity to think about their responses in advance, allowing for more complex questioning and in-depth discussions.

More Facilitation

As a focus group facilitator, I like to call on participants I have not heard from—a little harder to do by phone, but not impossible. Before starting the group I draw a little grid with the names of the participants across the top and the question numbers along the left hand column. I ask participants to say their name before they speak. As each participant responds, I put a check mark in the appropriate box for each question to keep track of their participation. I also write a few key words in the box to help me pick up the thread later on when they respond to another question.

The Right Technology

Although the conferencing technology available these days can have lots of bells and whistles, telephone focus groups don’t require anything fancy. All you need is audio-conferencing. You can use FreeConferenceCall.com but, if long-distance fees are a barrier to participation, use a system that provides you with a toll-free number. Every conference call system gives you the option of recording the call. For a small fee you will receive a CD of the recording, or some companies make the conference call recording accessible by phone or computer free of charge.

Ground Rules

The ground rules for telephone focus groups are similar to in-person groups: one person talks at a time, everyone respects the rights of others to speak, there are no right or wrong answers, and all cell phones are turned off during the discussion. In addition, if I hear any background noise, I ask people to mute their phones when they are not speaking. I also request that participants not answer emails, drive their cars, or clip their dog’s toenails while participating in the group. (I once had a woman ask me if it would be all right to participate in the focus group while she took her husband out for his birthday. For her husband’s sake I said, “No!!”)

Added Advantages

Lastly, telephone focus groups offer a slight edge over in-person groups in a handful of ways:

• Increased anonymity: The absence of visual contact creates psychological comfort for some people who are shy or have a difficult time speaking up in an in-person group.
• Informality: Participants like the fact they don’t need to “dress-up” for telephone groups.
• Better attendance: The no-show rate is much lower for telephone groups because there are generally fewer barriers.
• More focus: Due to decreased visual stimuli, there are less disruptions, sidebar conversations, and repetition in the discussion.
• Cost-effectiveness: Telephone focus groups are more economical than face-to face groups since costs such as meeting rooms, food, babysitter, travel, etc. are not incurred.

Think again

So the next time focus groups seem out-of the question because participants are far apart or too busy to attend, think again. The answer may be only a phone call away!

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

In my last blog, I described the power of stories to help organizations and programs measure outcomes.  To assist the AmeriCorps Student Assistance Program in the Forest Grove school district round out their evaluation plan, I developed a story methodology to capture a missing voice in the program’s evaluation process: the students they serve.

In that post, I promised to share a story I wrote using the methodology and story framework I developed. So here’s the story. It’s about Carlos (not his real name), a 4^th grade student who struggles with math and sitting still. The story is based on true facts and written from the perspective of the AmeriCorps member who worked with Carlos last school year. Each of the 20 AmeriCorps members serving the Forest Grove school district this upcoming year will be composing three stories like this using the methodology I developed.

Background

Carlos is a 4^th grade student who joined our after-school club (which focuses on homework and enrichment projects) at the beginning of the year.  From the very first after-school meeting, I noticed that Carlos had an extremely difficult time holding his body still for even a few seconds.  He was often disruptive during instruction time, and more interested in talking to neighbors than completing either his homework or projects.

Carlos’s teachers shared with me that he had been somewhat difficult to have in class ever since coming in as a 1^st grader.  Their suspicions ranged from a learning disability, to difficulties understanding instruction (he is a native Spanish speaker), to the possibility of diabetes.  I asked his teacher if I could spend my last open half hour of each day with him.  My strategies included letting him know when I notice him on task and talking about what to do when he isn’t, listening when he needs to talk about stuff going on at home, getting to know him as a person, and generally helping him understand that he is a good person and fully capable (with effort) of being a good student.

Student Story

When I interviewed Carlos he said, “I kind of like school more now that I can help the other kids.” He was referring to a time during one of our most recent small reading groups when he was so on top of things that he was helping the other two students in our group find their place when they were lost and offering to “lead corrections.”  He knows he is doing better in school because he received an A- in math on his last report card. “I never got any A’s before in anything,” he told me.

One thing that really made a difference, a breakthrough for Carlos actually, was learning to manipulate the multiplication table. Although he is still not functioning at grade level, he is so proud of his ability to use the multiplication table to complete his homework and study for quizzes. He has developed so much confidence that he asked his teacher if he could retake an earlier test to bring his grade up to a solid A for the next report card. Carlos doesn’t know exactly why he’s doing so much better except that he wants to keep getting better so he can continue to help other students.

When he grows up he want to be a professional baseball player and knows he will need math to figure out baseball statistics for himself and his teammates. He said: “And if they don’t know how to do it, if they didn’t get good at math like me, I’ll help them to figure out the batting averages and strike outs and all that stuff.”

Comments/Reflections

The road thus far has definitely not been smooth for Carlos.  He was suspended for three days at the beginning of November for fighting, and I have had to ask him to take time-outs numerous times during my group enrichment period. But Carlos can be a completely different student when working one-on-one with an adult.

Over the last few weeks, I have noticed a greater effort on his part to participate in class, monitor his own behavior, and get his work completed.  I think this is due to a concerted effort by his two classroom teachers, his ESL teacher, the two instructional assistants that work with him, and me to show him that we all care about him enough to hold him accountable for trying to do his best.

Because Carlos feels so good about himself when he is helping others, we have all created opportunities for him to do that in small but meaningful ways. We have asked him to help collect student papers at the end of class, orient new students to the after-school club, and take responsibility for letting students who miss math class know about the homework assignment they have due. Keeping Carlos engaged in this way seems to have the dual benefit of directing his energy in a more positive way while at the same time increasing his confidence in his ability to do well in school.

. . . . . . . . . . . . .

Stories like Carlos’ play a crucial role in program evaluations because they provide a window into the impact the AmeriCorps program is making on the lives of its students.  Combined with the quantitative data collected by tracking student and AmeriCorps members’ activity and outputs, these stories complete the picture. And when it comes time to report to funders, make program improvements, or seek additional funding, having the complete big picture is a priceless asset.

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

Stories open up our world—they make us laugh, they make us cry, and sometimes they even give us new insights about others and ourselves. Stories are part of every culture and help us make sense of the world.

As I wrote in a previous post, stories can also help programs and organizations make sense of their worlds. When collected with rigor, information gathered through stories can be used to influence policies, develop programs, and measure effectiveness. Sometimes stories are the best way to gather this information.

I recently turned theory into practice with a method I developed for collecting evaluation stories from elementary and middle school students. My goal was to help the AmeriCorps Partnership for Student Achievement program in the Forest Grove school district find a practical evaluation method for measuring student outcomes.

Best Outcomes Hardest to Capture

For the past several years, Forest Grove has received funding to place 20 AmeriCorps members each year in district schools as tutors and mentors in the Partnerships for Student Achievement (PSA) program. Each AmeriCorps member works one-on-one with several students throughout the school year in an effort to bring them up to grade level requirements in reading, math and English.

As the evaluator hired to help Forest Grove structure a meaningful PSA evaluation for the 2011-12 school year, I started by reviewing the existing evaluation plan. I found components for tracking numbers and types of students, for monitoring and assessing AmeriCorps member activity and for collecting pre/post assessment data on student achievement in math, reading and English from teachers. I was impressed with the comprehensiveness of the PSA evaluation plan. But I also realized that a process for eliciting direct feedback from program recipients–the students–was missing.

It’s not that Forest Grove officials didn’t think it was important to include student feedback in the evaluation. To the contrary: active student reflection with teachers and members was a stated program goal and genuinely encouraged. But the people at Forest Grove found themselves in the same dilemma I often witness among nonprofits: their best program outcomes were the hardest to measure.

A Qualitative Approach

After listening to what Forest Grove officials had to say about program goals and the realities of implementing a student data collection effort on the ground, it was clear that the situation called for a qualitative approach. But it needed to be a practical approach that could be implemented without additional funding. It also needed to be one that would harvest rich student outcome data in an objective and reliable manner without intrusion or additional requirements on teacher time.

Knowing these realities (and the dedication of PSA staff and their funders to a quality evaluation), I suggested collecting stories from students through semi-structured interviews conducted by AmeriCorps members. Not only would the stories engage students in reflecting on their experience, they would enlist AmeriCorps members in learning about evaluation (a skill they are encouraged to build). The stories would also complement the numeric data collected with quantitative strategies and enhance the meaning of that data. The program director loved the idea!

Beyond the evaluation benefits, the director envisioned the stories building student self esteem in the telling of them and pride in academic and personal achievements. She also felt the stories would be useful for program promotion and attracting additional funding. It prompted to her to start calling them “Great Student Stories.”

Story Making

To help AmeriCorps members compose a story that would address key evaluation questions, I designed an interview guide to structure the dialogue with the student. Throughout the upcoming school year, AmeriCorps members will be collecting stories from three of the many students they mentor. The students they select must be those with whom they: (1) have worked extensively throughout the year, (2) have seen considerable progress, and (3) feel have a reasonable ability to articulate their experience.

When writing the story up, the AmeriCorps member will weave in relevant information from direct experiences s/he has had with the student as well as pertinent information garnered informally from conversations with teachers and parents throughout the year. This creates a broader context in which to situate the student story. Ultimately, the member crafts a one-page story containing  background information, the student’s story, and their own reflections on the student’s process.

At the end of the year, the PSA advisory committee selects the most compelling stories to incorporate in the evaluation report. When funders read the report they will learn not only how many students were served and by what types of services, they will get a front row seat to the lived experiences of those students. Some of the stories may make them laugh, some may even make them cry. What’s certain though is that every story will put a human face on the program they’re funding.

Carlos’ Story

Stay tuned for my next blog where you’ll meet Carlos, a 4^th grade Forest Grove student who struggles with math and sitting still.  I wrote his story (based on true events) as an example of the evaluation stories that AmeriCorps members in Forest Grove will be composing this upcoming school year.

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

From small nonprofits to large corporations, virtually every organization collects both qualitative—what people say and feel about things—data and quantitative—costs, productivity, surveys—data to set policy, measure progress, and effect change.  Unfortunately, many CEO’s, program designers, and policy makers put more stock in the quantitative data they gather than the qualitative data. To them, hard numbers are the only true indicators of where things really stand and they wonder how interviews and focus groups could ever produce findings with the same level of trustworthiness, objectivity, and usefulness.

Actually, when designed and executed with as much rigor, qualitative data is just as defensible as quantitative data. It is simply that the methods for making it so are very different. Because qualitative inquiry can look as natural as any informal conversation between inquisitive and interesting people, an outside observer may not understand the underpinnings that structure the conversation and give it credibility. Instead of using a “p value” to convey trustworthiness, qualitative approaches incorporate strategies like saturation, member checks, and information rich sampling to ensure sound and credible findings.

Granted, there are many opportunities to inadvertently interject our biases, opinions, and habits into qualitative processes—the questions we ask, the way we ask the questions, how we select study participants, the way we analyze the data, etc.—but the trick is to convey our awareness of those potential pitfalls and explicitly describe the steps we’ve taken to proactively address them. In this essential aspect, qualitative research is no different than quantitative research.

Listed below are five basic measures that will help you avoid common pitfalls and increase the trustworthiness of your qualitative data. Use these to describe how you designed the study, collected the data, and generated your conclusions, and your findings are much more likely be taken seriously.

1. Define clear participant criteria. Develop a series of inclusion and exclusion criteria for selecting study participants and stick to it throughout the recruitment process. Your efforts will pay off in the identification of “information rich” participants who will share information essential to unraveling the complex and unique issues you’re studying.

If you set clear criteria, you can’t be accused of selecting participants just because they are the easiest to access or will say whatever you want them to say to receive the incentive you’re paying them.

2. Develop a set of thoughtful, targeted, unbiased questions. To paraphrase philosopher Solomon Ibn Gabirol, a question well designed is a question half answered. Although open-ended questions seem easier to develop than survey questions, don’t be fooled. Take time to thoughtfully develop the question guide in collaboration with key stakeholders and end users of the information. This will ensure that you are: aligning questions with project goals, wording questions in a way they are most likely to be honestly answered, responding directly to your study questions, and producing data that will get used.

If you design an airtight set of questions, you can’t be accused of asking biased questions or just having an informal conversation—especially if you asked for input from stakeholders and end users.

3. Collect data in multiples. Conduct enough focus groups or interviews to include a diversity of perspectives and ensure a prevalence of information rich subjects in your study. Ideally, you want to keep interviewing until you’re not hearing anything new (reaching saturation). As a general rule, plan to conduct at least three focus groups or no less than ten individual interviews on any one topic using the same set of questions.

If you conduct multiple interviews with individuals who meet your inclusion/exclusion criteria, you will be more certain that you haven’t missed any major themes and can’t be accused of delivering the limited perspective of only a few interviewees.

4. Use a systematic process for analyzing qualitative data. After all your hard work designing the study, and your respondents’ contributions of time and effort to answer your questions, the next step is to systematically analyze your data as if each piece were equally valuable. We all have far too many implicit biases of which we are unaware to simply read through the transcripts a few times and write a report from our unformed impressions. Whether you cut-and-paste, use an Excel spreadsheet (see my recent blog), or use sophisticated qualitative software, you MUST systematically and objectively sift the data until the shiny nuggets emerge. It’s tedious and time consuming, but well worth the confidence you’ll achieve in your data.

If you follow a systematic and reproducible data analysis process, anyone using the same process should be able to closely replicate your results. That makes it a scientific, verifiable process akin to quantitative analysis methods.

5. Conduct ‘Member checks.’ Member-checking refers to a process of sharing your findings with those from whom you have collected data. Sharing a list of bulleted findings—major themes and sub-themes—with the people you interviewed gives you an opportunity to, not only validate your findings, but also elicit further clarification of your results.

This step validates your distillation and interpretation of the data. It verifies that you were really listening when respondents spoke and that your main points are the same as theirs.

There are several more measures you can take to increase the trustworthiness of your qualitative data, but adhering to these five will put you on solid ground.

I’ll be adding to this list from time to time, so stay tuned for more trustworthy tips in future blogs!  My next addition to this list will address measures specific to data validity.

Until then, please let me know if there is any way I can support you in designing more trustworthiness into your qualitative data.

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.

Bill Clinton, Rudy Giuliani, and Alan Greenspan are some of the highest paid public speakers in the United States.  Toastmaster clubs groom thousands of aspiring thespians each year.  And virtually every college requires at least one speech class to earn a diploma.  But, when was the last time you heard of someone receiving an award or earning a degree for good listening? Granted, good speaking skills are critical for success in all aspects of life—but they’re only half of the communication equation.

Because I listen for a living, I get many opportunities to hone my skill. But, until recently, most of what I knew I learned through trial and error. Then I discovered the International Listening Association (ILA), an agency whose mission is to “promote the study, development, and teaching of listening and the practice of effective listening skills and techniques.” It opened a whole new level of learning for me.

The ILA publishes the International Journal of Listening, and their website contains a variety of resources from listening assessments to bibliographies, books, and interesting listening facts.  For example, did you know that the average person talks at a rate of about 125 – 175 words per minute, while we listen at a rate of up to 450 words per minute?

But one of my favorite ILA resources, a welcome gift I received for joining the organization, is a gem of a book called Right Listening by Mark Brady, PhD.  Right Listening is chock full of insights and recommendations on becoming a better listener.

I discovered ten that I find particularly applicable to and useful for conducting individual and group interviews.

1. Promote an atmosphere of trust. There are many ways to establish trust, but perhaps the most effective is to be genuinely trustworthy. Many people intuitively sense an authentic personality and rarely betray that trust.

2. Shut up and listen. When we interrupt, the unwitting message we send to the speaker is: “What I have to say is more important than what you have to say.” By learning to hold our tongue and become genuinely curious about what others are saying, we greatly improve our listening skills.

3. Give up control. Many people feel that giving up control in a conversation is a sign of weakness. But the biblical injunction to “Seek first to understand, then to be understood” is excellent advice for those intent on developing good listening skills.

4. Cultivate “Beginner’s Ear.” “Beginner’s ear” is a way of paying attention to the present moment with openness and curiosity—hearing it for the first time even though we think we’ve heard it before. Neuroscientists have shown that the more we practice this technique, the better our brain gets at it.

5. Double check for meaning. It is difficult not to overlay our own biases onto what we hear. One way to counter the process is to regularly double check for accuracy by paraphrasing what we hear and the meaning we make of it. Skilled listeners endeavor to reflect back a speaker’s truth and deeper reality, not simply a version of their own.

6. Listen for differences. When we listen to others, often what we listen for are the things we understand or agree with. A skillful listener deliberately seeks out and pays attention to the way others are different.

7. Ask specific clarifying questions. Author and researcher Larry Barker said, “Words have no meaning; people have meaning.” When we engage in dialogue, we frequently speak thoughts off the top of our heads. First thoughts are like first drafts—they require a good editing to clarify meaning. Asking clarifying questions can help a speaker bring their subject into clear focus.

8. Monitor for inconsistencies. Voice, tone, and body language can contradict spoken words. Skillful listeners learn to recognize inconsistencies and get to the bottom of them in a compassionate way that does not provoke defensiveness.

9. Be mindful of age, race, and gender bias. Over 100 documented cognitive biases can color everything we see, hear, and think. Skillful listeners examine how they listen to various age groups and different races or genders, then work to correct any discrepancies accordingly.

10. Cultivate patience. Skillful listeners possess a ready willingness to suspend self-expression while they focus on others without a pressing need for them to be succinct, speedy, or clear in what they have to say.

Dr. Brady’s book contains 42 more suggestions for skillful listening. I encourage you to check it out as well as the ILA website. Who knows, maybe you’ll be the first to win a Great Listener award!

I’d also love to hear any of your suggestions or techniques for developing skill in this critical area.  I’m listening!

Thank you for reading my blog. Just enter your email address below to become a subscriber. It’s free and you’ll receive a notice every time I post.