Using Qualitative Data

Written on April 24, 2012 at 2:06 pm, by Susan Eliot

All qualitative data doesn’t need to end up in a formal written report to make it useful. In fact, very often the opposite is true.

Sometimes I conduct focus groups or a set of interviews for an organization whose decision makers want to use the findings right away. They need the data to develop policy or improve programs, products or systems that need fixing sooner rather than later. These clients know their industry, understand the value of hearing directly from their constituents or staff, and have some understanding of the issues they want to address. They don’t need a lengthy traditional report with background, literature search, methodology, and conclusions.

What they need instead is someone to collect the data in a rigorous and thorough manner and organize the findings in a format they can easily access. I like these kind of “hit-the-ground-running” projects because they are so utilization focused. The questions are generally pragmatic, intentional, and likely to generate useful information. And, rather than wasting away in a thick report that sits for months on decision-makers’ bed stands, findings are generally put to use within a month or two after they are generated.

So in place of a report, I organize the data I have collected for the client in a multiple page Excel spreadsheet. On the spreadsheet, each response is entered on a separate line, coded by category and organized by question. Each response is also accompanied by face codes—gender, race/ethnicity, age, longevity in the profession/work place, and title/position—that may be helpful for further data sorts by subgroup.

But the hand-off doesn’t end there. Because qualitative results require a bit more skill to interpret and apply than quantitative data, I like to spend time talking with the client about how to get the most out of the data and use it in an objective, thoughtful way.

Read more

3 Comments

Category Qualitative Analysis, Qualitative vs. Quantitative | Tags:

The Interview Guide

Written on March 27, 2012 at 12:57 pm, by Susan Eliot

When we want to know more than the boxes and circles on a survey can reveal, we turn to the open-ended interview. Thoughtfully designed and skillfully delivered, interviews generate an infinite possibility of responses—more than we ever thought to ask.

By its nature, the interview is a fluid process. But, not unlike surveys, interviews require pre-designed questions to steer the questioning in a consistent way. This ensures that the short time we have with the interviewee is used efficiently and we use the same line of questioning with each interviewee.

The pre-designed interview guide is used for both individual interviews and focus groups. It is generally one of two types:

  • A checklist of possible topics
  • A fixed set of open-ended questions

Checklist Approach

Read more

2 Comments

Category Interviews, Qualitative approach | Tags: ,

Listening Competence

Written on March 15, 2012 at 10:16 pm, by Susan Eliot

Those of you who regularly follow my blog know that I have a thing for listening. I believe good listening is fundamental to good qualitative work.

The January-April 2012 edition of The International Journal of Listening includes research on the validation of the Ford, Wolvin, Chung Listening Competence Scale (Mickelson, W.T and Welch, S.A., pp. 29-39). The research was inconclusive but the authors said that there was a  “well-established theory and logic behind the dimensions of the Listening Competence Scale.”

That’s good enough for me. I found the Scale valuable in assessing my own listening skills (or lack thereof) and in identifying specific areas that I need to sharpen. I include it here in its near entirety. I left out one of the six dimensions—Therapeutic Listening—because I didn’t think it applied to qualitative work. The other five dimensions are highly applicable though.

The tool is simple and requires just a few minutes to take. Give it a try and see how you fare. Each item is scored on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree) with a midpoint of 3 (neutral).

Discriminative Listening

Read more

1 Comment

Category Listening | Tags:

Qualitative Research Trainer, Guide, Consultant

Written on January 23, 2012 at 10:30 pm, by Susan Eliot

If you’re a doctoral candidate struggling to define your qualitative study, a faculty member who needs help implementing the qualitative component of a newly awarded grant, or a large public agency interested in using qualitative methods to design programs or policies, then you must meet Dr. Asher Beckwitt!

Dr. Beckwitt is a qualitative research consultant and the CEO of Asher Consulting, LLC.   She created the company in 2005 to empower students, scholars, and professionals to conduct excellent qualitative research. She helps researchers at any stage in the process, from conceptualizing the topic, to using qualitative data analysis software, to writing up the findings. She provides live and recorded online webinars on a variety of qualitative topics as well as a number of customized services. She’s truly a researcher’s research consultant.

I had the pleasure of interviewing Dr. Beckwitt recently to see what I could learn from her. Here are excerpts from our conversation:

Eliot: I’m curious about how you developed an expertise for working specifically with students and faculty. How did you get started?

Read more

No Comments

Category Qualitative approach, Qualitative research, Qualitative research proposals | Tags: ,,

Five Factors Associated With Deep Listening

Written on November 22, 2011 at 1:34 pm, by Susan Eliot

Most parents understand the difference between surface-level and deep  listening all too well. Listening for details about what exactly their teenager will be doing, and with whom, are just as important as hearing their teen’s promise to be home no later than midnight. When they can take the time to do it, parents know that deep or advanced listening can have real payoffs. The same is true for qualitative researchers.

Most of the listening we do as we go about our everyday lives does not require a high level of skill. We interpret the words and gestures of coworkers and family members quickly and automatically with little conscious awareness on our part. Most of the time this surface-level listening is all we need to interact effectively.

If, however, we are conducting focus groups or interviews with individuals we have never met and from whom we want to obtain meaningful data, we need more advanced listening skills. Advanced listening is different from ordinary listening in that it involves in-depth processing of the messages we receive. It’s a cognitively demanding activity that requires a mindful, systematic scrutiny of the message and its source.

So what does it take to engage in, or learn to engage in, advanced listening?

Read more

No Comments

Category Focus Groups, Interviews, Listening, Uncategorized | Tags:

Telephone Focus Groups

Written on October 25, 2011 at 6:44 pm, by Susan Eliot

“Scheduling” has to be one of the most common and least loved of all activities in today’s society. Whether it’s setting a business meeting with ten managers or figuring out who will pick your son up from soccer practice, arranging for multiple people to be in the right place at the right time can be a daily headache.

Organizing an in-person focus group is no different.  In fact, there may be times when it is nearly impossible to bring participants together in person.  But there is another option: telephone focus groups. The same principles used in face-to-face groups, combined with conference call technology, can be just the solution.

As a general rule of thumb, focus groups are best conducted in person. But when that’s not possible, telephone groups are a great alternative for the following situations:

  • Geography: Telephone focus groups are ideal when participants are dispersed over a wide geographic area.
  • Convenience: People with busy schedules can attend a telephone group at any time of day from the convenience of their offices or homes.
  • Inclusion: Telephone groups make it possible for those who are disabled or lack transportation to participate in the conversation.
  • Skilled facilitator: A highly skilled focus group facilitator can be enlisted from virtually anywhere when one is not available locally.
  • Limited budget: Telephone focus groups are more economical than face-to face groups since costs such as meeting rooms, food, baby-sitter, travel, etc. are not incurred.

Read more

1 Comment

Category Focus group guidelines, Focus Groups | Tags: ,,

Great Student Stories: Part Two

Written on September 23, 2011 at 12:01 pm, by Susan Eliot

In my last blog, I described the power of stories to help organizations and programs measure outcomes.  To assist the AmeriCorps Student Assistance Program in the Forest Grove school district round out their evaluation plan, I developed a story methodology to capture a missing voice in the program’s evaluation process: the students they serve.

In that post, I promised to share a story I wrote using the methodology and story framework I developed. So here’s the story. It’s about Carlos (not his real name), a 4th grade student who struggles with math and sitting still. The story is based on true facts and written from the perspective of the AmeriCorps member who worked with Carlos last school year. Each of the 20 AmeriCorps members serving the Forest Grove school district this upcoming year will be composing three stories like this using the methodology I developed.

Background

Carlos is a 4th grade student who joined our after-school club (which focuses on homework and enrichment projects) at the beginning of the year.  From the very first after-school meeting, I noticed that Carlos had an extremely difficult time holding his body still for even a few seconds.  He was often disruptive during instruction time, and more interested in talking to neighbors than completing either his homework or projects.

Read more

No Comments

Category Nonprofit programs, Program evaluation, Stories, Uncategorized | Tags: ,,

Great Student Stories

Written on September 13, 2011 at 12:40 pm, by Susan Eliot

Stories open up our world—they make us laugh, they make us cry, and sometimes they even give us new insights about others and ourselves. Stories are part of every culture and help us make sense of the world.

As I wrote in a previous post, stories can also help programs and organizations make sense of their worlds. When collected with rigor, information gathered through stories can be used to influence policies, develop programs, and measure effectiveness. Sometimes stories are the best way to gather this information.

I recently turned theory into practice with a method I developed for collecting evaluation stories from elementary and middle school students. My goal was to help the AmeriCorps Partnership for Student Achievement program in the Forest Grove school district find a practical evaluation method for measuring student outcomes.

Best Outcomes Hardest to Capture

Read more

No Comments

Category Qualitative approach, Stories, Uncategorized | Tags:

Making Your Qualitative Data Trustworthy

Written on August 18, 2011 at 9:39 am, by Susan Eliot

From small nonprofits to large corporations, virtually every organization collects both qualitative—what people say and feel about things—data and quantitative—costs, productivity, surveys—data to set policy, measure progress, and effect change.  Unfortunately, many CEO’s, program designers, and policy makers put more stock in the quantitative data they gather than the qualitative data. To them, hard numbers are the only true indicators of where things really stand and they wonder how interviews and focus groups could ever produce findings with the same level of trustworthiness, objectivity, and usefulness.

Actually, when designed and executed with as much rigor, qualitative data is just as defensible as quantitative data. It is simply that the methods for making it so are very different. Because qualitative inquiry can look as natural as any informal conversation between inquisitive and interesting people, an outside observer may not understand the underpinnings that structure the conversation and give it credibility. Instead of using a “p value” to convey trustworthiness, qualitative approaches incorporate strategies like saturation, member checks, and information rich sampling to ensure sound and credible findings.

Granted, there are many opportunities to inadvertently interject our biases, opinions, and habits into qualitative processes—the questions we ask, the way we ask the questions, how we select study participants, the way we analyze the data, etc.—but the trick is to convey our awareness of those potential pitfalls and explicitly describe the steps we’ve taken to proactively address them. In this essential aspect, qualitative research is no different than quantitative research.

Listed below are five basic measures that will help you avoid common pitfalls and increase the trustworthiness of your qualitative data. Use these to describe how you designed the study, collected the data, and generated your conclusions, and your findings are much more likely be taken seriously.

Read more

No Comments

Category Data trustworthiness, Focus Groups, Interviews, Qualitative approach, Qualitative Sampling | Tags: ,,

10 Suggestions for Skillful Listening

Written on August 3, 2011 at 8:43 pm, by Susan Eliot

Bill Clinton, Rudy Giuliani, and Alan Greenspan are some of the highest paid public speakers in the United States.  Toastmaster clubs groom thousands of aspiring thespians each year.  And virtually every college requires at least one speech class to earn a diploma.  But, when was the last time you heard of someone receiving an award or earning a degree for good listening? Granted, good speaking skills are critical for success in all aspects of life—but they’re only half of the communication equation.

Because I listen for a living, I get many opportunities to hone my skill. But, until recently, most of what I knew I learned through trial and error. Then I discovered the International Listening Association (ILA), an agency whose mission is to “promote the study, development, and teaching of listening and the practice of effective listening skills and techniques.” It opened a whole new level of learning for me.

The ILA publishes the International Journal of Listening, and their website contains a variety of resources from listening assessments to bibliographies, books, and interesting listening facts.  For example, did you know that the average person talks at a rate of about 125 – 175 words per minute, while we listen at a rate of up to 450 words per minute?

But one of my favorite ILA resources, a welcome gift I received for joining the organization, is a gem of a book called Right Listening by Mark Brady, PhD.  Right Listening is chock full of insights and recommendations on becoming a better listener.

Read more

4 Comments

Category Interviews, Listening, Oral Histories, Stories | Tags: ,,,

« Older Entries