How to Run and Interpret Intercoder Reliability in NVivo (Full Transcript)

[00:00:00] Speaker 1: I receive a lot of questions about intercoder reliability and how to use in vivo to run intercoder reliability. In this presentation, I'm going to talk about what intercoder reliability measures and also how to interpret the output after running the intercoder reliability here in vivo. And I'm going to talk about the common mistakes that researchers made when it comes to intercoder reliability. And also, the best process is to improve coding reliability. So these are the things I'm going to talk about. If you have any questions, you can put it in the comment section there. I will be happy to address them. And also, don't forget to subscribe to this channel. OK. So one thing that you have to note is that intercoder reliability really is all about exploring the consistency in terms of how two or more coders apply a specific code or a group of codes, right? So this means that the first thing that you have to make sure that you have is you should have a codebook. In order to adequately run intercoder reliability, it's very important to have the same codebook. And what is the codebook? So codebook contains all the code and the definitions of the code and sometimes examples of what segment should be applied to a specific code and also the criteria in terms of what not to be coded and what should be coded. So one thing that you have to do is that before you run intercoder reliability, you should have a transcript, a selected transcript. It can be one transcript or it can be one or more transcript. And you should have two or more coders coding the same transcript. And then you should have a codebook used by all the coders. After I coded a transcript, then you can go ahead and run intercoder reliability in in vivo, right? Looking for kappa and agreement. And I'm going to talk about kappa, which measures the degree of consistency among the coders in terms of how they apply the code to a specific segment of the data, right? There should be a selected transcript that you have to give to two or more coders. And there should be an agreed codebook that should be used in coding. And after that, then you can run the analysis to measure the intercoder reliability. In order to have a meaningful result is very important to have a shared codebook, right? So this means that all the coders should have access to one codebook and they should be aware of the definitions of the code and examples and also the exclusion and including criteria in terms of which part of the data needs to be selected, right? So the coherence kappa is the most important statistics in the report. When you look at the results, one thing that you first look at is the coherence kappa. If the coherence kappa is between 0.6 and one, then it's acceptable. There's consistency among the coders in terms of application of the codes. So you want to see numbers that's from 0.6 or maybe sometimes 0.7 to one, right? When it's 0.5 and lower, then this means that there's less consistency, right? So that's how you can look at it. But we don't only look at the kappa number. We also look at the percentage of agreement. I'm going to talk a little bit about that and when we go to in vivo, you understand it more. The fact that you have a high percentage doesn't necessarily mean that there's high reliability, right? You always have to first look at the kappa before you also look at the agreement, right? So it's very important in terms of interpreting the results. These are the columns that you will see in a vivo report. And let me show you how it looks like. Before I run it. So this is how it looks like. So you have the kappa column. You have agreement column A and B column, not A, not B column and disagreement column and then not A and not B column and B and not A column. The most important areas that you really have to understand is the first three. The kappa, that's what you have to read first, right? What's the kappa score? Is it 0.7? Is it 0.8? Is it 0.3? So that will also give you an indication of a potential reliability, right? You also look at the overall agreement, right? In terms of the application of a specific code, how much do they agree in terms of how they apply the code in a specific area of the data, right? And then in terms of the A and B, how much agreement do they have between the coders in terms of applying a specific code, right? So you can see that kappa is 1, which shows an indication of potential higher reliability. Agreement is also high. Agree. It's very high. It's 100%, right? But when you look at A and B, what do they share in terms of area that they coded is 0. This means that they don't have a shared area in the documents that was coded, right? And then here, not A, not B is also 100%. This means that both coders with A and B did not apply the code in a specific area, right? So this result might be misleading. If you look at all the kappa, you think, oh, they have an agreement. Technically, they agreed on the area that should not be coded, right? So I think as you can see here, it shows that nothing was coded, right? So it's a perfect agreement because nothing was coded. Nothing was coded because A and B, they don't have any shared applied area in terms of their document where they coded. So it didn't mean that it did not apply the code to a place that is common among the two coders, right? So when you look at it, it's an indication that although there's a perfect agreement, nothing was coded, right? So what is the essence of calculating this kappa and calculating the intercoder reliability? It's to see whether the application of the code is consistent across coders, right? So if, let's say, kappa is low and there's low reliability, what are you going to do, right? What you can do is you and your coder can go through the code book and then review all the code and how the code is applied. And after making some changes to the code, then you can go ahead and apply the code book again to a set of transcripts. And then you can run the kappa again and compare. And then you also have a discussion about the results. And if there's anything that you have to do, you can refine it again and go through the process until you get an acceptable reliability score, right? So it's very important for you to do that because imagine you run a reliability and then it's low and you just report it and don't do anything about it. People will question your result, right? Will your audience trust what you found if the kappa is low, right? And because the question people are going to ask you is that what did you do about the low kappa? And it's very important for you to provide action that you took to improve the code book and also maybe apply the codes to significant information that you see from the data. But there's an alternative. Based on best practices, I personally prefer using collaborative coding strategy. And I think that when it comes to qualitative analysis, you focus on where's not numbers, right? To ensure quality. So what is collaborative coding strategy? So the beginning, you have to build a code book with the second coder, making sure that you all agree about the kinds of code that you have to use. And in what situation do you have to apply the codes? And you have the definition of the codes and also inclusion and exclusion criteria in terms of the significant information that you apply the codes to. So after agreeing on the code book, then the next step is to select one or two or few of the transcripts and then code the transcript individually using the code book, right? So you're going to use the same code book, code it individually. Sometimes in initial stage, you just use one transcript, code it and then you meet and have a discussion. In terms of how you apply the codes and then is there any similarities and differences between the codes that you have applied? So based on discussion, you can improve the code book, right? And then you can go ahead and use a code book to do the coding. You can try to apply the code book to different sets of transcript and then come back again and discuss. Or after you have making changes to the code book based on the initial coding, you can now share the transcripts. You say, let's say if the transcripts are 20, you can, one person will do 10 of the transcript and another person do another 10 of the transcript and then you put everything together, right? So that's what collaborative coding is all about. This one, you don't use numbers to measure the reliability, right? This one is more about having a discussion, comparing how you apply the codes and then trying to resolve the differences between how code was applied, right? So that's the alternative way of doing the coding. So let's go to Enviro and I'm going to show you how you can do the intercoder reliability. So the first step is to open Enviro, right? So I'm opening my Enviro and then sometimes it will prompt you to confirm the coder, right? For me, I can type my name and then my initial, right? Normally it prompts you when you have two coders using the same computer and the same software, right? So that when anybody opens, you indicate who you are so that we know who coded what, right? So I click on OK. Okay, so when I click on OK, this is what you're going to see and I'm going to use a project sample here, right? This is about environmental change down east. This is a sample project that I'm going to use for this demonstration. So you can just click here, right? I already opened, so I will click this part, right? But if you haven't opened it before, you can always click on that and then open it. So when you open, this is what you're going to see. So when you go to file, under file, you go to area and township and then you click on interviews and this is the interviews that they did, right? And then one thing that we have to talk about is that imagine that your coder or your co-researcher has already coded his or her transcript and have sent the project to you. What you can do is that you can click on import and click on project and then browse and look for the project that the person sent to you and then you can upload it and then you can merge it into this project, right? So this is done when the person has already coded the data and then you have coded but they are in different projects. So you can bring the person's one and import it and then you can merge it into this one. That's the first option. The second option is you can code yours and send your project to the person and the person will code his or hers in your project so that everything will be in one place and then you can open. That's another second option. If you want to know who are the users of this project you can go to file and you go to project properties and you go to users and you see all the users, right? So these are the users of the project, right? In case you want to find out. And you see when I open the project it prompts me to indicate who is a user, right? So how do you indicate that? Especially if the person is using the same project on your same computer, right? And you want a system to prompt the person whenever they use their computer, right? So what you can do is that you go to file and go to options and you see here prompt for user on launch. You have to check this part so that all the time whenever somebody is opening this project it will prompt the person to type the name and the initial so that you know who use the project apart from you. And also make sure that there's a consistency. So if it's PA all the time you have to type PA all the time so that the system will know that you are the person who is analysed not a different person, right? So when you check that you just click on apply and then click on okay. So now we are assuming that you and your co-researcher have coded the data using a shared codebook, right? Now let's start with the intercoder reliability. How to do that? You go to explore and you see the query here, right? And you go to coding comparison. You click on that and this is where you indicate the users, right? So if you have two users you just indicate the first user here and the second user here. If you are more than two you can decide that okay let me group them. Let me bring two users here and two users here or maybe you are only three users. Two users can be here and the third user can be here. You can group any way that you want. You can start with only two users. You go to select and you choose the first user and you click on okay and go to select here and you choose maybe the second user. Let me choose this person, right? And then you can go ahead or sometimes you can group all the users into two. Choose this one and add it to the first user, right? And then you can if another user is here you can just add it to the second one. You can always do that. If you change your mind you can close it and go back again to queries and comparison and then you choose new users. Let me choose the first user and then let me choose the second user. Now you see the place has all codes, right? You don't have to use all the codes from the code book to run the reliability test, right? What you can do you can select few of the codes, right? So how do you do? You can click here and go to selected codes. Click on select and then click on codes and you can see all the codes here. What I'm interested in is about economy. All the codes and the economy. I checked that. I just want to see how they apply these codes. One, two, three, four codes, right? I click on okay. You can choose more than four. You just choose the one that you think will give you good results for you to determine what the next steps, right? So DEA is also giving you the scope. What kind of document or what specific transcripts do you want to use, right? Do you want to use all the transcripts? If you want to use all the transcripts you leave this one alone. If you want to use some of the transcripts you select that. You go to select and then you click on files. Files here. This one is a little bit different because you have to press the plus sign and then under that you see interviews. You click on that. So you just technically you are looking for all your transcripts so that you'll be able to select which one you are interested in. So let me choose only one transcript for now. You can choose more than one but for this example so that you understand it. Well, let me choose one. I click on okay and then you make sure that you choose the display kappa coefficient, right? And also display percentage agreements. These are the two that are very important for you to really make a good decision, right? And then are you comparing tests or you're comparing the area? Normally I choose the test coding. This shows that you are comparing how they apply the codes to specific tests in the data, right? So when you are done you click on run. Before you click on run you can also save the criteria in case you want to run it several times. You can click on save but when you click on save it will ask you for the name so that when you go here you can find it, right? So the name can be I will just say R1 Dakota Reliability 1 and then I click on run and then the system will give me this table, right? Sometimes you may not see what is out here so what I always do is to right click here and go to undock so that it will pull the table so that you see everything. Now I see the results here, right? Okay, so sometimes the results may be confusing but it's a good option. The first option is to use this sheet that I've created here to help you to interpret or you can download it and then upload it on ChatGPT and ask ChatGPT to help you to understand this output from in vivo, right? And then ChatGPT will help you to understand. So let's start with the kappa coefficient. So you see that here is one. Let's just forget about the rest first. Let's focus on kappa. What does one mean? I think one means that there's a potential higher reliability because if it's in between 0.6 and 1 then it's very acceptable coefficient, right? So acceptable reliability that we can see here. It's potential because we haven't looked at the other numbers yet, right? Now let's pull this one a little bit. That this is the code is called Agriculture, right? And it's from Thomas file and then we want to see how this code is applied on Thomas document. And it looked like looking at only the kappa score here it looks like there's consistency in terms of how they apply in the area of the transcript across the two coders. And you see the agreement here is 100%, right? So it's also a potential and indication that the area that the first coder apply is the same as the area that the second coder apply. That's what it's indicating. But when you look at A and B it makes you think about, okay, why is it zero here? It's zero means that it's an indication that they technically didn't apply. It shows here that even if they apply the code Agriculture, they didn't apply at the same place or the same significant information. That's what the zero means. So that it brings into question kappa one and 100% agreement, right? It's giving you an idea that it means that they made similar decision in terms of not applying this Agriculture in the same place. And then the next one, which is 100%, not A and not B, so this means that they did not apply this one in a particular area, right? That's why you say not A and not B. They did not even apply this code to any of the areas of any part of the data. So these three numbers in conclusion give us an idea that there's an agreement between them in terms of not applying this code to a specific significant information. When you take two of the coders, right? Maybe the third coder maybe apply Agriculture to a specific place in Thomas' document. But when we take these two coders, they did not, right? It can be that the decision is reliable, right? But we are looking into agreement in terms of applying the codes, not agreement in terms of not applying the codes, right? There's a little question mark here. That's what I want to say, right? It's a little complicated. But let's look at the second one, right? So you see that Kappa is 0.7664, right? So this means that there's a potential reliability and the agreement is 97.58, which is very good. The area that they apply that a shared application is 4.26, right? And the area that they did not apply is 39 points. So for this one, it shows that they agreed to apply this code, which is Fishing and Aquaculture, right? But they apply only 4%. So I would say that this is a positive one when it comes to application because they have something that they shared, right? They shared 4%, right? That's a good one. For this one, I wish what they share should be more than what they don't share, right? But I think it's not all that bad. At least they share something. It's similar to this one too. They also share 60%, but they don't share 78. That's okay. At least they have something that they shared. This one too. So I can say that based on the results, there is consistency in terms of application of the codes to a specific significant information. But the area that is shared is not a lot, right? I wish the area that is shared, I'm talking about how they apply that code to a specific area or quotation, specific information that participants said, I wish it has been more, right? So what do you do next with this one? I think what I would do is I will have a discussion with my co-researcher about the area that we applied these codes and why we did not have much in common in terms of the area. As I said, if statistics is not your area, I think the interpretation might be a little bit challenging, but there's always a solution. The solution is using this one, which is cheat sheet that you can refer to. And it also have a sample of how to even write about the results in terms of the CARPA results. And if you are working on a dissertation, right? This is the template that you can also use in terms of writing the report. You can export this one if you want to. You can right click on it and export list, right? And then you click on save to export. And then if you want, you can go to ChachiPT and then you can attach that results for ChachiPT to help you. So let's try to see whether ChachiPT can help us. I run intercoder reliability tests in in vivo and I've attached the SL spreadsheet about the intercoder reliability results. Can you go through and help me to understand the findings? And then I click on enter. Let's see. Okay, so you see that agriculture one is, it looks like this one is perfect agreement. As I said, you have to be very careful here. It's a perfect agreement. But when you look at the other numbers where we're going to go through a little bit, you'll see. So looking at the two columns, which is the CAPA and an agreement, looks like, yes, the first one is perfect. The second one is substantial. And this one is the almost perfect. And they're almost, so they are all good. But you also have to look at the other numbers too to help you to conclude, right? So you see here, CAPA is 100. Agreement is 100. This means that both coded this concept exactly the same. Right. There are zero disagreement. Neither code miss any coding. Okay. This is the ideal results. Okay. So you look, it looks like the system only focus on the two columns. So I'm going to ask, can you help me to interpret the A and B and not A and not B columns? So you see here, A and B. This column represent the percentage of the document where both coders apply the code to exactly the same test. Right. So in other words, both coders agree that this session should receive this code. Right. So let's see. So let's say coder A coded 120 units. Coder B coded 115 units. They both coded the same 100 units. So it's going to be 100 divided by a thousand. That will be 10%. So let's see. Okay. So let's look at this one, how the system interpreted this one. Job and cost of living. Right. So you can see here that A and B is 16 points. Okay. Let's see the interpretation. So A and B is equal to 16.84. This means approximately 16.84 of the transcripts were coded by both coders as job and cost of living. Okay. This represents a strong form of agreement because both coders independently identify the same passage. Okay. Perfect. So if it's around 10 and above in terms of the percentage, that is shared, it looks like it's very good. So going back to the, to here, it looks like the 16 is very good. My 6.4 is moderate and 4 is not a lot. Right. But if you have 10 and above percentage, it's good. So it has also explained that not A and not B. Right. Things that they don't share in common. So let's see here. So if not A and not B is equal to 93.32%, it means that about 93% of the transcript was judged by both coded as not discussing fishing or aquaculture. This is still agreement. Okay. So it looks like here is why percentage agreement can be misleading. And then they provide an example. So I told you about how it can be misleading. So you have to be very careful interpreting this one. So I see how I was able to use Chachibi to help me to understand part of the data. You can continue to have a discussion with it, interaction with it, and see so that you can understand the findings. Let me know your thoughts. I hope I explained it well to you. I hope my explanation helps. And if you are not clear about it, you can also let me know. But the end of the day, if you're not comfortable using the intercoder reliability, you can use the collaborative coding strategy, which is more consistent with the philosophical paradigm that inform qualitative study, because we don't really focus too much on numbers. We just focus on words and explaining and exploring the issues using non-numeric data. I hope this one was helpful for you. Let me know whether you have any questions and also thank you for your time.