Intercoder Reliability in NVivo: Setup, Run, Interpret (Full Transcript)

[00:00:00] Speaker 1: In this presentation, I'm going to talk about what Intercoder Reliability measures and also how to interpret the output after running the Intercoder Reliability here in Envivo. So let's go to Envivo, and I'm going to show you how you can do the Intercoder Reliability. So the first step is to open Envivo, right? So I'm opening my Envivo. And then sometimes it will prompt you to confirm the coder, right? For me, I can type my name and then my initial, right? Normally it prompts you when you have two coders using the same computer and the same software, right? So that when anybody opens, you indicate who you are so that we know who coded what, right? So I click on OK. Okay, so when I click on OK, this is what you're going to see. And I'm going to use a project sample here, right? This is about environmental change down east. This is a sample project that I'm going to use for this demonstration. So you can just click here, right? I already opened, so I will click this part, right? But if you haven't opened it before, you can always click on that and then open it. So when you open, this is what you're going to see. So when you go to File, under File, you go to Area and Township, and then you click on Interviews, and this is the interviews that they did, right? And then one thing that we have to talk about is that imagine that your coder or your co-researcher has already coded his or her transcript and have sent the project to you. What you can do is that you can click on Import and click on Project and then browse and look for the project that the person sent to you and then you can upload it and then you can merge it into this project, right? So this is done when the person has already coded the data and then you have coded, but they are in different projects. So you can bring the person's one and import it and then you can merge it into this one. That's the first option. The second option is you can code yours and send your project to the person and the person will code his or hers in your project so that everything will be in one place and then you can open. That's another second option. If you want to know who are the users of this project, you can go to File and you go to Project Properties and you go to Users and you see all the users, right? So these are the users of the project, right, in case you want to find out. And you see when I open the project it prompts me to indicate who is the user, right? So how do you indicate that, especially if the person is using the same project on your same computer, right? And you want the system to prompt the person whenever they use their computer, right? So what you can do is that you go to File and you go to Options and you see here Prompt for User on Launch. You have to check this part so that all the time, whenever somebody is opening this project, it will prompt the person to type the name and the initial so that you know who use the project apart from you. And also make sure that there's a consistency. So if it's PA all the time, you have to type PA all the time so that the system will know that you are the person who is analysed, not a different person, right? So when you check that, you just click on Apply and then click on OK. So now we are assuming that you and your co-researcher have coded the data using a shared codebook, right? Now let's start with the intercoder reliability, how to do that. You go to Explore and you see the query here, right? And you go to Coding Comparison. You click on that and this is where you indicate the users, right? So if you have two users, you just indicate the first user here and the second user here. If you are more than two, you can decide that, OK, let me group them. Let me bring two users here and two users here. Or maybe you are only three users. Two users can be here and the third user can be here. You can group any way that you want. You can start with only two users. You go to Select and you choose the first user and you click on OK and go to Select here and you choose maybe the second user. Let me choose this person, right? And then you can go ahead. Or sometimes you can group all the users into two. Choose this one and add it to the first user, right? And then you can, if another user is here, you can just add it to the second one. You can always do that. If you change your mind, you can close it and go back again to Queries and Comparison and then you choose new users. Let me choose the first user and then let me choose the second user. Now you see the place has all codes, right? You don't have to use all the codes from the codebook to run the reliability test, right? What you can do, you can select a few of the codes, right? So how do you do? You can click here and go to Selected Codes. Click on Select and then click on Codes and you can see all the codes here. What I'm interested in is about Economy. All the codes under Economy, I checked that. I just want to see how they apply these codes. One, two, three, four codes, right? I click on OK. You can choose more than four. You just choose the one that you think will give you good results for you to determine what are the next steps, right? So DEA is also giving you the scope. What kind of document or what specific transcripts do you want to use, right? Do you want to use all the transcripts? If you want to use all the transcripts, you leave this one alone. If you want to use some of the transcripts, you select that. You go to Select and then you click on Files. Files here. This one is a little bit different because you have to press the plus sign and then under that you see Interviews. You click on that. So just technically you are looking for all your transcripts so that you'll be able to select which one you are interested in. So let me choose only one transcript for now. You can choose more than one, but for this example, so that you understand it well, let me choose one. I click on OK and then you make sure that you choose the display kappa coefficient, right? And also display percentage agreements. These are the two that are very important for you to really make a good decision, right? And then are you comparing tests or you're comparing the area? Normally, I choose the test coding. This shows that you are comparing how they apply the codes to specific tests in the data, right? So when you are done, you click on Run. Before you click on Run, you can also save the criteria in case you want to run it several times. You can click on Save, but when you click on Save, it will ask you for the name so that when you go here, you can find it, right? So the name can be, I will just say R1, Dakota Reliability 1. And then I click on Run and then the system will give me this table, right? Sometimes you may not see what is out here. So what I always do is to right click here and go to Undock so that it will pull the table so that you see everything. Now I see the results here, right? Okay, so sometimes the results may be confusing, but it's a good option. The first option is to use this sheet that I've created here to help you to interpret or you can download it and then upload it on ChatGPT and ask ChatGPT to help you to understand this output from in vivo, right? And then ChatGPT will help you to understand. So let's start with the kappa coefficient. So you see that here is 1. Let's just forget about the rest first. Let's focus on kappa. What does 1 mean? I think 1 means that there's a potential higher reliability because if it's in between 0.6 and 1, then it's a very acceptable coefficient, right? So acceptable reliability that we can see here. It's potential because we haven't looked at the other numbers yet, right? Now let's pull this one a little bit that this is the code is called Agriculture, right? And it's from Thomas Fall and then we want to see how this code is applied on Thomas' document. And it looks like, looking at only the kappa score here, it looks like there's consistency in terms of how they apply in the area of the transcript across the two coders. And you see the agreement here is 100%, right? So it's also a potential and indication that the area that the first coder apply is the same as the area that the second coder apply. That's what it's indicating. But when you look at A and B, it makes you think about, okay, why is it 0 here? It's 0 means that it's an indication that they technically didn't apply. It shows here that even if they apply the code Agriculture, they didn't apply at the same place or the same significant information. That's what the 0 means. So that it brings into question kappa 1 and 100% agreement, right? It's giving you an idea that it means that they made similar decision in terms of not applying this Agriculture in the same place. And then the next one, which is 100%, not A and not B. So this means that they did not apply this one in a particular area, right? That's why you say not A and not B. They did not even apply this code to any of the areas of any part of the data. So these three numbers in conclusion give us an idea that there's an agreement between them in terms of not applying this code to a specific significant information. When you take two of the coders, right? Maybe the third coder, maybe apply Agriculture to a specific place in Thomas' document. But when we take these two coders, they did not, right? It can be that the decision is reliable, right? But we are looking into agreement in terms of applying the codes, not agreement in terms of not applying the codes, right? There's a little question mark here. That's what I want to say, right? It's a little complicated. But let's look at the second one, right? So you see that kappa is 0.7664, right? So this means that there's a potential reliability. And the agreement is 97.58, which is very good. The area that they applied, that shared application, is 4.26, right? And the area that they did not apply is 39 points. So for this one, it shows that they agreed to apply this code, which is Fishing and Aquaculture, right? But they apply only 4%. So I would say that this is a positive one when it comes to application, because they have something that they shared, right? They shared 4%, right? That's a good one. For this one, I wish what they share should be more than what they don't share, right? But I think it's not all that bad. At least they share something. It's similar to this one too. They also share 60%, but they don't share 78. That's okay. At least they have something that they shared. This one too. So I can say that based on the results, there is consistency in terms of application of the codes to a specific significant information. But the area that they shared is not a lot, right? I wish the area that they shared, I'm talking about how they apply that code to a specific area or quotation specific information that participants said. I wish it has been more, right? So what do you do next with this one? I think what I would do is I will have a discussion with my co-researcher about the area that we apply these codes and why we did not have much in common in terms of the area. As I said, if statistics is not your area, I think the interpretation might be a little bit challenging, but there's always a solution. The solution is using this one, which is cheat sheet that you can refer to. And it also has a sample of how to even write about the results in terms of the CARPA results. And if you are working on a dissertation, right? This is the template that you can also use in terms of writing the report. You can export this one. If you want to, you can right click on it in the spots list, right? And then you click on save to export. And then if you want, you can go to CHATGPT and then you can attach that results for CHATGPT to help you. So let's try to see whether CHATGPT can help us. I run intercoder reliability tests in in vivo and I've attached the Excel spreadsheet about the intercoder reliability results. Can you go through and help me to understand the findings? And then I click on enter. Let's see. Okay. So you see that agriculture one is, it looks like this one is perfect agreement. As I said, you have to be very careful here. It's a perfect agreement, but when you look at the other numbers, where we're going to go through a little bit, you see. So looking at the two columns, which is the Kappa and an agreement, it looks like, yes, the first one is perfect. The second one is substantial. The next one is the almost perfect. And they're almost, so they are all good, but you also have to look at the other numbers too, to help you to conclude, right? So you see here, Kappa is 100, agreement is 100. This means that both coded this concept exactly the same, right? There are zero disagreement, neither coded miss any coding. Okay. This is the ideal results. Okay. So you look, it looks like the system only focus on the two columns. So I'm going to ask, can you help me to interpret the A and B and not A and not B columns? So you see here, A and B, this column represent the percentage of the document where both coders apply the code to exactly the same test, right? So in other words, both coders agree that this session should receive this code, right? So let's see. So let's say coder A coded 120 units, coder B coded 115 units. They both coded the same 100 units. So it's going to be 100 divided by a thousand. That will be 10%. So let's see. Okay. So let's look at this one, how the system interpreted this job and cost of living, right? So you can see here that A and B is 16 points. Okay. Let's see the interpretation. So A and B is equal to 16.84. This means approximately 16.84 of the transcripts were coded by both coders as job and cost of living. Okay. This represents a strong form of agreement because both coders independently identify the same passage. Okay. Perfect. So if it's around 10 and above in terms of the percentage that is shared, it looks like it's very good. So going back to here, it looks like the 16 is very good. 6.4 is moderate and 4 is not a lot, right? But if you have 10 and above percentage, it's good. So it has also explained that not A and not B, right? Things that they don't share in common. So let's see here. So if not A and not B is equal to 93.32%, it means that about 93% of the transcript was judged by both coders as not discussing fishing or aquaculture. This is still agreement. Okay. So it looks like here is why percentage agreement can be misleading. And then they provide an example. So I told you about how it can be misleading. So you have to be very careful interpreting this one. So I see how I was able to use Chachibi to help me to understand parts of the data. You can continue to have a discussion with it, interaction with it and see so that you can understand the findings. Let me know your thoughts. I hope I explained it well to you. I hope my explanation helps. And if you are not clear about it, you can also let me know. But the end of the day, if you are not comfortable using the intercoder reliability, you can use the collaborative coding strategy, which is more consistent with the philosophical paradigm that inform qualitative study, because we don't really focus too much on numbers. We just focus on words and explaining and having, exploring the issues using non-numeric data. I hope this one was helpful for you. Let me know whether you have any questions and also thank you for your time.