Speaker 1: Hey friends, Katherine here from Research Rockstar. Thanks for joining me here today. I'd like to talk today about qualitative specifically, and I'm a big fan of both qualitative and quantitative data. There's no secret about that, but one of the things about qualitative data that we often come into as a bit of an issue is that we put a lot of time and effort into planning the qualitative research, whether it's focus groups or in-depth interviews or ethnographies or online qual. There are so many different options these days, but we have a tendency to put a lot of thought and effort into planning and recruiting and actually collecting the data as it were. Unfortunately, I often see cases where not as much time has been spent on how we're actually going to plan the written report, because most qualitative research projects do end with some sort of written report. It may be a short-form report or a long-form report, but unfortunately we don't always take as much time as we could to really plan that fantastic report, and there's a good reason for it. It's really hard. In fact, there are a lot of reasons why writing a great qualitative research report is hard. So let's talk about seven reasons, seven reasons why writing qualitative reports is hard, especially when we're trying to write great reports. So there are some issues here that are familiar. For example, we know that qualitative data is very complex, so not only are we trying to report unstructured data, the nature of qualitative data, but we've got really complex data these days. Very often, qualitative data these days can include text, audio, images, or video, again depending on what the methodology is, so it can be really, really complex. We also know that qualitative data is very messy. There's a lot of messiness. The raw data that we collect from qualitative research methods is often in multiple formats and different file types, and so just literally bringing together all of this data so that I can get my arms around it can be really difficult. Also, with qualitative data, we have to deal with inconsistencies. Not every person answers the same exact question. Not every person interprets every question the same way, and even in observational methods, if we're doing ethnographic research or other observational methods, there's also inconsistencies there, too. Just by how different people behave in different situations we may be observing them in. In online qualitative techniques, we also have challenges because people comply with instructions differently. For example, if you've tried any of the mobile ethnography or self-ethnography applications like D-Scout or Indemo, I love these applications, but when you're giving people tasks to do using those types of smartphone-based apps, some people comply with the instructions better than others. It's just reality, so there's a lot of inconsistencies. Another reason why reporting qualitative data is so hard is the sheer volume. Again, whether you're doing focus groups or interviews or ethnographies or online techniques, the sheer volume of data that you're collecting is overwhelming. You know, when people do survey research, it's a lot easier, right? So when I'm doing survey research, I'm pretty much automatically creating charts and graphs that help me to visualize and explore my data. Not true with Qual. I have sheer volumes of data that I have to deal with. Another issue with qualitative reporting is that it is context-sensitive. When we are writing up our qualitative research reports, we have to put a lot of thought and effort into understanding context. You know, the times that the research was being conducted, the places, were there seasonality issues, was there other context that creates an important lens by which we should be interpreting the data from either the overall Qual study or phases of the study? Researcher bias is also an issue. Good qualitative researchers know that we have to be very vigilant about not imposing our own biases into the research. Unfortunately, this can happen because, well, frankly, a lot of people who do qualitative research, for example, have graduate degrees and they have specific cultural backgrounds. Just the fact that they have a certain educational background can create bias in their observations and as well in the writing of the results. So we have to be really careful to keep our own cultural experiences and context and personal experiences out of it. Does that mean that we don't use our knowledge to help interpret the data? No, not at all. We just have to make sure we're doing it in an objective way. And finally, the seventh reason why qualitative reporting is so hard is because we know we're dealing with imperfect and incomplete data. So we have to be really vigilant about this because we know when we are either interviewing people or talking to groups or observing people that there can be a lot of things going on with the participants. Most specifically, what I find is there's often a big disconnect between what people say and what people tell me versus what they actually think or do or feel. And that doesn't mean that it's bad data, it just means I have to be really vigilant about understanding when I'm being told something that is not the full story because, again, I know I'm going to have imperfect or incomplete data. So there are obviously methods that we use as either moderators or interviewers to try to mitigate that risk, but we have to understand when we're getting ready to write that report that we have to be really vigilant about how we're interpreting the data because, again, we may need to, say, point out for our readers that while people are, say, sharing a particular point of view or self-reporting a specific attitude, we know that there are situations where that may not be the whole story. So it's just important to have as a caveat in our reporting. So again, there are a lot of different reasons why qualitative reporting is so difficult. So what's the answer? So you've just done six focus groups or 30 in-depth interviews or in a case study that I just did with a couple of our instructors here at Research Rockstar, we had 10 college students participate in an online qual study about their studying habits. So we did a study about studying and in all of these cases we need a way to approach the report writing process. And again, I feel like there's a lot of great articles and books about research methods for qualitative research. There's a lot less information out there about what's the best approach to writing qualitative research reports. And again, it really is very difficult. Now when you start thinking about different options for writing qualitative research reports, a lot of it's going to come down to your analysis. You know, how am I going to analyze and synthesize this unstructured, unwieldy data to get the most important value out of it? And you will hear different qualitative researchers use different approaches. You'll hear about comparative analysis. That's the simplest form of analysis. For example, if you are doing focus group research in, say, three cities, Boston, Chicago, and Dallas, simply when you do that comparison of the results by city, that would be a simple form of comparative analysis. So that's very common. Then you'll hear other qualitative researchers refer to framework analysis or grounded theory, which ultimately have to do with different ways of extracting value from the data using different types of coding approaches. There are also researchers, including myself, who like to do visual exploration. Now visual exploration of quantitative data is something that a lot of people have written about, and if you go to Google Scholar and you search for visual exploration of quantitative data, you'll find tons and tons of articles. Not so true on the qualitative data side, but on the qualitative data side there are methods that we use related to using diagrams and frameworks to visually display your qualitative research results. Now by the way, there are other types of analysis that you can get into, and you'll definitely see, especially with more academic researchers or people doing large-scale health-related qualitative studies. For example, you'll hear of researchers using things like semiotics, narrative analysis, and then just various forms of techniques to get to counting, say, the themes or words or responses in a large-scale qualitative study. What I'd like to do is just share with you one thing that I like to do that you can use as an example of visual exploration of qualitative data. So one of the things I like to do is I like to use diagrams, and I'm going to put a link to this diagram for the people who are looking at this audio only, or listening to this audio only, but this is a diagram that we use actually in our class on writing qualitative research reports, and it can be modified for many different purposes. But in this case what I'm trying to do is, let's imagine I'm doing some qualitative analysis and I'm trying to understand the customer experience with a specific product. I can use a simple diagram to help me to categorize my results. So I might take a look through, for example, my focus group results or my ethnography results and put them into categories based on customer experience. So I'll use the example of the college student study that I just referred to. In this case we're trying to understand their experience with studying. So the customer experience, if you will, is about studying. Why is it also a customer experience? Well, this could be something where the students are customers of either the college or the students are customers of textbook or online textbook resources. So they're college students but they're also customers. And in this case what I want to do is use a very simple way of categorizing their studying behaviors into where, who, what, when, why, and how. And I can do this both on an individual basis, I can diagram each student in terms of what their behaviors were in these six categories, or I could do it by groups of students. For example, if I find that the students who attend four-year colleges generally were pretty consistent and different than people who were attending two-year colleges, then maybe I do one diagram for my four-year respondents and another diagram for my two-year respondents. Or again, I might do it by geography or I might do it by gender, right? So I might do multiple diagrams to do some of these comparisons. But in the case of studying, the where would be where they studied. For example, in this case we found that a lot of our college students, again in the small case study project, study on their bed. They don't go to a table, they don't go to a desk, they're in their dorm room, they're on their bed when they study. Well, who are they studying with? So there's some information there about people who study solo versus the types of people who study with others and study groups and with friends. What are they doing while they're studying? What else is going on? Well, when people study on their bed it turns out that they have different ways of liking to set up all their supplies, all their study supplies. So what they're doing to make the studying happen is actually also really interesting. Of course, for when? When are they studying? What day of week? What time of day? Are there any patterns there? Why are they studying? Are they studying to get good grades? Are they studying because they are trying to impress a professor? Are they studying because they just want to make sure that they get a good enough grade that they could pass? So what's the motivation there? Is there something there that I can extract? And anything about how? How are they studying? Are they studying using a book, an online resource? Are they studying by making flashcards? If they're using flashcards, are they using physical flashcards or are they using electronic flashcards? So again, simply by categorizing their behaviors into where, who, what, when, why, and how, it can help me to understand where those recurring themes and patterns are. And by categorizing it that specifically, it also gives me a meaningful way to do some comparative analysis. Again, perhaps by gender or location, college type, or maybe major. Simply by creating a diagram, it gives me an opportunity to create a structured approach to the analysis. And this does help me to write my research report. If I really have some forms of structured analysis like this for really synthesizing this unwieldy qualitative data, it's going to help me to make sure that I'm doing a very thorough, objective, unbiased reporting of the qualitative data. Without a structured approach, or without a thoughtful approach, even without a visual approach, even, you know, even if you don't use a diagram, but having some sort of structured approach does provide a way of writing a great qualitative research report. In contrast, if you have a qualitative researcher who's writing their report basically by recall of what they saw, there's too much risk that it's going to be incomplete and frankly not very thoroughly analyzed. And unfortunately, I do see that. I see a lot of top-line memos specifically, or short qualitative research reports that are written based on researchers' recall and their personal notes. And I think that's really something that creates a lot of risk in terms of your ability to create a great qualitative research report. So clearly there are other steps to creating great qualitative research reports, but I hope you found this specific example to be useful. Again, if nothing else, give it a try. Try creating some diagrams. And if where, who, what, when, why, and how aren't the most relevant categories for you, what are the categories? There may be other ways that you can diagram it out so that you again have that consistent, thorough way of interpreting the results of the unstructured data. For those of you who are interested in writing qualitative research reports, I want to share that we do have our next session coming up, excuse me, starting March 6th, Writing Qualitative Research Reports. Our wonderful instructor Corey Mann is going to be teaching that course that meets once a week for four consecutive weeks. That's four consecutive 90-minute sessions. You can learn more about that at training.researchrockstar.com. And again, that course starts on March 6th. If you have any questions or comments, please do leave them here. And again, I will also post a PDF to the diagram. Thanks, everyone.