When AI-Led Interviews Fit Qualitative Research (Full Transcript)

[00:00:00] Speaker 1: Today, I came across a study that was done by Anthropic. They did a study to find out how professionals use AI and their perspective about the use of AI and how it's imparting their work. They focused on 1,250 professionals. What really caught my eyes is how they collected the data, the strategy that they used to collect the data. They used AI to help them to collect rich information from participants. And how did they do that? I'm going to get into that. But the most important thing is that it's becoming so common for researchers to use AI to help them to collect rich information from participants. I'll put a link of this article in the description so that you can get access to it. So as you can see, they focused on 1,250 professionals. And then this is the breakdown. You can see that the general workforce were 1,000, and also scientists were 125, and the creative were 125. So one thing I really like about this information is that they were so transparent in terms of how they designed the study using AI, how they collected the data using AI, and how they also analyzed the data. And this is what I preach all the time. You have to be very transparent when you use AI to help you to do your study. When you do that now, people are going to learn the best practice. So let's go through and see how we can learn from this study. So as you can see here, 1,250 interviews was conducted, and it was done by AI. So it was AI-led interviews. And this is the brief information about their findings, but we are not going to focus on the findings now. When you get the time, just go through. We just want to focus on the methodology so that we learn from how AI was used to collect data from participants. So let's go to the methodology. So as you can see here, they wanted to explore how they integrate AI into their professional practice, and also want to find out how they feel about its role in their future. So this is the purpose of the study. And they used a tool called Anthropic Interviewer. It was in three stages. The first stage is the planning stage, where the software was used to develop interviewed rubric that will help the system to interview participants. It's just like semi-structured interview, class participant question. And based on the response, the system asks further questions to help to collect rich information from participants. And this one is a form of text. So the system provides text, and then participants respond to the text. The system also helped them to conduct an interview. So it was, as I said, AI-led interviews. So the system interviewed participants and got all the information that it needs. And then the system also did the analysis, analyzed the transcript, and then be able to help them to address the purpose of the study. Another interesting thing that they did was they made the data available for you to get access. So if you want to see the raw data, then you can go to, let me show you where to find that information. That would be very helpful if you want to see the raw data. We are publicly releasing our interview data from this initial test. So when you click on this link, it will take you to the source and all the data set. So one interesting thing that it did is that when they recruit third participants and they were about to interview them, or they were about to use the system to interview them, they asked them, do you want your information to be public? Some of them indicated that they want their information to be public. So these are the participants that they want their information to be public. So you can see here, when you click on it, you can see what the AI system asks and then what are their responses. So what is the lesson here? I think it's very important for you to let participants know what you're going to use the data for, how you're going to make the data available if you want to make it in public. Make them be part of the decision making process in terms of what you're going to use their data for. I think it's very important. So transparency is not only telling the consumers of your research about how you analyze and collect data, but transparency is also making it clear to your participants how you're going to use the data. So what are the lessons that we have learned here before I go into my main topic, which is conducting AI-led interview? So let me go back. So I was going to put the link, this link in the description, so that if you want to download or get access to the raw data for further exploration, you can do that. So I can provide that information. So what are the lessons here? So the lesson here is that it's possible for you to use AI to help you to collect rich information from participants. It's very possible. And as you can see here, they were, within a few months, they were able to collect information from over 250. So you see that within a short period of time, they were able to collect rich information from participants. And the question that you always want to ask yourself is that, can you use AI to help you to collect data in terms of interview from participants? Yes, you can use it, but there are some factors that you have to take into consideration when you are planning to use AI to help you to collect information from participants or to conduct interview on your behalf. So there are four things that you have to think about when you want to use AI to help you to collect rich information from participants. First one is a sample size. If you plan to focus on a large number of participants and you don't have the time and resources to interview all of them, then one option is to use AI to do the interviews on your behalf. If you have a very good tool, it will be able to help you to do semi-structured interview. Ask questions, they respond, and then based on the response, the system asks further questions. And you can use it in the form of verbal conversation or in the form of text. So as you can see here, for this research, you see that they are focusing on a large number of participants and maybe they don't have time and resources to interview them one-on-one using human to do that. So this is where AI can play a very good role. So your sample size can also help you to determine a strategy that you want to collect data from participants. The second factor is the purpose of the study. What is the goal of your study? So if your goal is more of interpretation, you are interpreting participant information. If you are doing interpretative phenomenological study, it's good to do one-on-one interview by humans. You have to collect the data by interviewing participants, because in terms of interpretation, there are a lot of factors that should be considered as you are making sense of participant information, as you are interviewing participants, as you are collecting rich information from them. They provide you some information, you have to process it, making sure that you understand. You have to take into consideration the participant background, the situation that they find themselves in, in order to make sense and then ask further questions. It's going to be useful for you to do the interview yourself instead of using AI to do that interview, because you are not going to get rich information when you are doing a study in which the purpose of the study is more about interpretation. If the purpose of the study is more descriptive in nature, one example is that if you want to just explore consumer's view about a specific service or a product. If you are exploring software and you want users' experience and views, and you just want a descriptive information about their experience, their perception, their views, their opinions. If that's the main focus, similar to what a traffic team did, then using AI to collect data on your behalf will be a very good thing, right? Because it's just describing, it's not making interpretation, it's more about describing, giving participants an opportunity to describe the experience, their views, and opinions about something, then AI-led interview will be the best for you. Another factor that you have to take into consideration is access, right? So when it comes to access, how are you going to access participants? Where are they? Where are they located? Are they close to you that you can go and see them one-on-one to interview them? Or can you use Zoom to interview them? Do they have tight schedule where it's very difficult to schedule with them and then interview them? So all these things should be considered when you are deciding to use AI to help to collect data. If they have tight schedule and they are not available, not close to you, they are spread across multiple locations, then you may consider using AI-led interviews, right? But if they are close to you and the number of participants is not a lot, and you can easily contact them, schedule, and interview them, then doing a one-on-one interview by yourself would be the best. So the access is very important. Another thing about access is that, what is their perception about using AI to collect data from them? Do they feel comfortable using AI? Getting an idea about what they think about AI will help because if most of the participants that you are focusing on have negative perception about AI, they don't want to share the ideas and opinions using AI, then AI-led interview will not be the best because you end up excluding about half or majority of your participants. So it's very important as a researcher to consider accessibility in terms of their views concerning the use of AI and do they have access to the technology that you can use to collect data from them. All these things should be considered if you want to use AI because we don't want to exclude even the small number of your potential participants. So the last factor that we want to look at is sensitivity. If the topic that you are focusing on is sensitive, then you have to be careful using AI-led interview. One example is that if a participant has passed through a specific trauma like domestic violence, you don't want to use AI to interview them because sometimes if they are not treated and you are interviewing them, your questions may re-traumatize them and you have to be there to know their responses and if there's a need for you to pause or stop the interview, you can quickly do that. AI may not be able to observe all these non-verbal language that are needed for you to make informed decisions immediately. If it's a sensitive topic, then it's very important for you to do it yourself instead of using AI to help to collect the data. But there's an exception here, right? So one thing that you have to think about is that sometimes the topic is sensitive and participants want to remain anonymous. They don't feel comfortable talking to someone. They feel comfortable talking to AI. They feel comfortable sharing their ideas with AI, instead of a person. So in this case, AI-led interview will be the best. One example is that maybe mental health stigma, right? And because of stigma around mental health, maybe the participant that you want to focus on want to remain anonymous. This is where AI-led interviews will be a very good option for that situation. So these are the four things that you have to think about when you want to use AI to help you to collect data from participants. So let's recap. Think about SPAS. Sample size, T stands for the purpose of the study, A stands for accessibility, and S stands for sensibility. So these are the things that you have to think about to help you to decide. And you can also look at the best practices here from Anthropic Study to learn more about the best way of using AI to help you to collect data from participants in the form of interviews. If you have any questions concerning this, you can put it in the comments section. I'll be happy to address them for you. And also, I want to hear from you. Will you one day use AI to help you to interview participants? And what are your perceptions about using AI to collect data? I hope this presentation was helpful. Let me hear from you, and thank you so much for your time.