Human Oversight in AI-Assisted Qualitative Research (Full Transcript)

[00:00:00] Speaker 1: The topic is about AI-assisted qualitative research. I am very passionate about this topic. Why? Because AI is here to stay and we as qualitative researchers, we have to find the best way of using it and not only using it, systematically using it and being able to communicate how you use it. So one of the quality assurance of qualitative study is about dependability. So dependability is about being able to document any actions and decisions that you take and being able to communicate what exactly you did. When you communicate what happened, people believe what you found because qualitative study is more about subjective. You and your participants, you and your data. If you don't show your work, people are not going to believe what you found. I'm going to focus on three main tasks. It's about Mechanica, Analytica, and Interpretive tasks. At the end of the day, you'll be able to classify your research tasks. So think about Mechanica as any kind of task that is related to preparing or arranging information. Think about your research. After collecting data, you have your audio. What did you do? You transcribed. Maybe you didn't do it yourself. You used AI to help you. That is Mechanica. You're looking for literature. What did you do? You went, maybe used AI to help you to select relevant literature for your study. It's Mechanica. You want to compile all the interview transcripts. Maybe you want to transfer all of them into an Excel spreadsheet. You can use AI to do that. It's Mechanica. Any activity that is more about organizing, retrieving, and formatting information is more about Mechanica. Another thing is that Mechanica, most of the time, you don't need any critical thinking. You don't have to analyze things before you complete the task. If you are manually transcribing it, you don't have to think through. You just listen and then write what the participant is saying. So it's more about preparing and arranging information. What if you give this task to AI? What are you going to do? You are going to verify. After AI has transcribed your audio for you, you have to listen to the audio because AI can make mistakes, right? Listen to audio and make the necessary correction. It's called verification. Verifying and making sure that everything is correct before you use. That is your role. Normally, most of the time, we give Mechanica activities or tasks to AI to do because it doesn't involve a lot of thinking. So think about what you are doing, right? Is this Mechanica? Can you give it to AI and you feel comfortable concerning that? What are you going to do? You have to verify because every output is your responsibility to verify, making sure that everything is right. So think about your activity. Which of them is Mechanica? Just note that. We will come back. Just note if you have any activity that you listed and it's Mechanica, just note that activity. So let's go to the next one. So the next one is Analytica. So think about this one. I think this one will be very easy for you to think through. When I think about Analytica, I think about data analysis, right? What happened during data analysis? You have your transcript. You have your research question. Participant has given you information. You have to extract information that are significant to help you to address your research question, right? In analyzing something, you have to break the information down. Break it down. It's just like you are an investigator. You are looking for evidence, right? You pick this one. You pick that one. You pick that one. Like analyzing your data, you pick all the information that what is said that can help you to address your research question. You are analyzing. You pick what is relevant. You bring them together. You group them. You compare. You identify patterns. That is all about analysis. Think about this situation. In terms of analysis, will you give AI the chance to analyze your data for you? Oh, AI can do that. AI can analyze your transcript. But think about it as suggestions. AI is suggesting things to you. They are suggesting calls to you. They are suggesting paths. They are suggesting relationship between things. And you have to evaluate what AI has given you. You have to examine and making sure that everything is right. Is this call directly come from the data or the system made it up? If this theme comes from the data, can we trace the theme back to the data? That is your work. If that's your work, then you have to do something. You have to know the skill in analyzing data. Because if you don't know the skill in analyzing data, you know what is going to happen? You accept what AI gives you. I've uploaded all my transcripts and I've gotten themes addressing my research question. I present that information. I didn't check. Why? Because maybe you are not knowledgeable about data analysis. You don't know anything about systematic analysis. So it's very important as an evaluator, you have to develop a skill in analyzing data before you give your transcript to AI to analyze. If you don't do that, you know what will happen? AI can make mistakes. You reported mistakes. And you are not doing justice to your participants. Participants give you rich information. So before you become evaluator, you have to be skilled in the subject matter that you are evaluating. If you don't do that, you're going to accept everything. Then you will not be needed anymore. AI can do everything for you. Then you have no control. You are losing your agency. And this is what is happening. If you don't take care and be a critical evaluator of AI output, you will not be needed anymore. And AI will take over the analysis. And that's why you always have to think about developing the skill, although AI can do this, so that you can even ask the right question. For AI giving you themes, you can ask AI, where did you get this theme from? What are the evidence from the data to support this theme? AI has seen this pathway. Where is the evidence and data? Can you show me? You ask critical questions because you are knowledgeable about a subject matter. If you are not knowledgeable about a subject matter, AI will take over your job. And that's what I always preach. So if you want to know about data analysis, my book will be helpful for you. Let me quickly put the presentation in a chart, if you haven't gotten it, and then you can download. If you go to the latter part, you'll see my book there, Step-by-Step Guide to Qualitative Coding, so that you'll be able to learn more about that. So let's move. So think about it. You have identified information that is significant from the data. AI has suggested to you two codes. One is after-hours charting, based on this information. Another one is theme isolation, based on this information. You have to evaluate. Is this one based on the data? Yes. But this one too. Maybe it's based on the data because it talks about isolation, but is this one addressing the research question? No. I have to refine that. Change it a little bit to address the research question, representing information that has been extracted. Oh, there's another theme that the AI didn't come up with. AI missed that. I brought that information. So you see how you are critically analyzing AI information because you know what the code stands for. You know how to evaluate a code, right? So the list that you provided or the list that you have, which one is analytical tasks indicates that. So let's go to interpretive, making sense, right? You're explaining what something means. This one, you see, I don't know whether you have seen this pattern. We're from Mechanica. AI can do a lot of Mechanica job. If analytical, AI can do a lot, but you have to supervise. Interpretive, AI can do, but you have to be very careful because sometimes AI can make things up when it comes to interpretation. It will take into consideration some factors that may not be supported by the data. So interpretive, if you want AI to do that for you, you have to be very careful, critically analyze or review the interpretation. And if you want to review the interpretation, do you have the skill of knowing what is an interpretation? What is interpretation? The meaning is a conclusion. It's a claim. It's a speculation. All these things, you have to think about it. What is a speculation? What is a claim? What is a conclusion? What is a critical review? So in case you want AI to make interpretation, then you have to review and decide whether you accept, reject, or modify the interpretation. You can do it the other way around. You can interpret your findings and give it to AI to critically review or provide you feedback. You can do that too, right? The most important thing is that you have evaluated AI out to making sure that information is right. When it comes to interpretation, it's good to do it yourself. If you want to use AI, make sure that you have the skill in evaluating the outputs. AI is becoming very good. You know, at the beginning, it started with Mechanica, maybe two years ago, and then we moved to Analytica. Mechanica aspect, when we were at that stage, AI was finding difficult to analyze data, but now it's very good. And now it's getting closer to interpretation, right? But we have to be very careful because it can misinterpret participant information. So you have to be skilled in analyzing that information. So at this moment, try and do it yourself and you can get ideas. You can see, you can view AI as like a critical aspect, having conversation. Oh, I'm thinking about interpreting this way. What do you think about that? You can get ideas, but the final decision should be you. You have to decide. Thank you.