Speaker 1: Hello, this is Professor Russell James from Texas Tech University and this is your super quick guide to how to read academic research even if you're not an expert. First tip, don't freak out. You're going to run into some things that are very complicated, something you haven't seen before. Don't freak out. Just keep going through the article. Why? Because you don't need to eat the whole cow. You can get important concepts out of a research article without fully understanding every detail. You can get the gist of what's going on without being an expert in every field that you read research in. So, what's the approach? Well, how do you eat a cake with rocks in it? Answer, don't try to eat the rocks. In the same way, you can read research articles and if you run into something you don't understand, skip over it and get to the next part that you do understand. There's still a lot of good stuff that you can understand even without being an expert in every single field. When you look at an article, there's three questions you want to ask. First, do I care about the research topic? Second, do I believe the findings? And three, so what? Each of the different sections of the article gives you an opportunity to answer different questions. You look at the title and the abstract, that says, answers the question, do I care? You look at the tables, what did they really find? You look at the methods, do I believe what's in the table? You look at the discussion section, so what? And then the literature view, what did we already know about this topic before we started? So let's take a look at an article. This is a brilliant one because I wrote it. Title and abstract, do I care? Read the title, do I care about that? Read the abstract. Am I interested in learning more? If the answer is no, then toss it, go on to the next one. If the answer is yes, then what do we do? Well, let's take a look at the tables. What did they actually find out? And we can look at whether there's an interesting relationship there or not. After that, I look at the methods. Methods lets me know, should I believe what's in the table? So suppose I do believe what's in the table, then we look at the discussion in order to find out, well, so what? So I believe what's in the table, you did it right, what's important about what you found here? And ultimately, you can also get around to the very beginning of the article, which has a literature view that tells you, what did we already know? So the real question is, should you believe the findings? The reality is, research is messy. Research often disagrees, and we want to be able to distinguish strong results from weak ones. How do you do that? Bad news. Knowing whether you should believe the findings usually requires some statistics. So I'm going to suggest to you there are three core statistics concepts you must know in order to be able to decide what is good research and what's weak research. That's association versus causation, correlation as compared to multiple regression, and significance as compared to magnitude. Let's take a look at the first one. What is association? Straightforward. A and B tend to occur together more frequently than one would expect by random chance. How do you explain an association? Well, guess what? There's lots of different ways to explain an association. One, it may be just random chance. Stuff happens. We got a weird sample, and that's what happened. Two, it could be that A causes B, or A causes B sometimes. But guess what? It's just as likely that B causes A, or B causes A sometimes. And besides all of that, maybe neither A causes B nor B causes A. It may be that something else causes both A and B to happen. So let's take an example. Suppose sleeping in your shoes is associated with waking up with a headache. We did a research study, and we found that this was true. Why? What are some explanations? Well, one, it could be random chance. Two, it could be that sleeping in shoes causes headaches. Three, it could be that the very early stages of a forthcoming headache causes sleeping in shoes. Finally, it could be going to bed drunk causes both results. So remember, association versus causation. Statistics can show only association. Statistics can never show causation. We have to infer causation from experimental design or theory combined with statistical association. Statistics is really good at telling us whether something might have happened by random chance, but it's not quite as good at telling us whether A caused B, B caused A, or sometimes something else causes A and B. What's the difference between simple correlation and multiple regression? Simple correlation says A and B tend to occur together more frequently than one would expect by random chance. Multiple regression says that is true even when comparing those who are otherwise similar in certain ways. So for example, we take a look at correlation, and we say higher education and charitable giving tend to occur together more frequently than one would expect by chance. If we then add that that is still true when comparing those with otherwise similar income and wealth, that's multiple regression, and that gets us to the next step, which means we can eliminate some causes that might have been in number four, something that causes both A and B. Multiple regression allows us to exclude specific items, but guess what? We can't exclude items that we don't know about or we didn't measure. Let's take an example. A famous article from Nature in 1999 says kids' night lights may cause myopia or be a precipitating factor. So here's a quote from the article. Although it doesn't establish a causal link, the statistical strength of association of nighttime light exposure and childhood myopia does suggest that the absence of a daily period of darkness during early childhood is a potential precipitating factor in the development of myopia. So are kids' night lights later causing them myopia in later life? Four explanations for this association. One, random chance, statistics can eliminate that. Two, A causes B, that is the night light is causing the kids to later develop myopia. Three, B causes A, not likely because here we measured the myopia initially. It wasn't there and it didn't come until years later. Or perhaps four, something else causes both A and B, and guess what? A rebuttal argument says exactly it's number four. We should have looked at parents' myopia causing both the night lights and the child's myopia. The rebuttal article says we find that myopic parents are more likely to employ nighttime lighting aids for their children. Moreover, there's an association between myopia in parents and their children. How could we have fixed that previous study? They say the study should have controlled for parental myopia. How do you control for it? Multiple regression and you put that in as a variable. Now let's talk about significance versus magnitude. What is significance? Well, basically, statistics test a small population to predict the whole population, a small sample to predict the whole population. Significance shows how likely our results might have been due to an unusual random sample rather than the actual difference in the population. So it's like taking a spoonful out of a pot of soup. We want to just test that spoonful and say something about the whole pot of soup. Most papers are going to report some measure of statistical significance, a chance that the association was due to a weird random sample and wasn't actually something that we see in the population. Most commonly, we're going to see a p-value or a confidence interval. What's a p-value? Well, think of it this way. How likely is it to randomly draw these five fruits from a truckload that had just as many apples as oranges? Well, the answer to that is going to give you your p-value. Most commonly, we look at 5% for a p-value and what that means, p less than .05 means there's less than a 5% chance that the result was caused by an unusual random sample where there was no actual population difference. So here's an example from a research article. Was there a significant gender difference between these two groups? We don't see an asterisk on there and in this table, that means there was no statistical difference. Well, what does it mean if there's no statistical difference? Well, guess what? It means that the sample difference could have easily occurred even if the two population groups were the same. But it does not mean that the two population groups do not differ, only that we can't tell. So don't listen to somebody talking about what does it mean that this result is statistically insignificant. It only means we can't tell. No asterisk or no significance in a table means we can't confidently tell what the effect of this item is regardless of the number and the size and the sign of that number. Sometimes, although much more rarely, you will see not a p-value used but a 95% confidence interval. And what this tells us, if you kept taking random samples again and again 95% of the time, the true actual population value would appear inside the confidence interval associated with each sample. Here's an example of what one looks like. The red line gives us our prediction and the dashed line tells us how confident we are of that prediction. And you can see at either end, we're becoming less confident than in the middle. There's also a problem of multiple comparisons. Now I've already asked you, we look at a p-value and say how likely is it to randomly draw these five fruits from a truckload with as many apples as oranges and you might have a guess for that. But how would your answer change if I got you to draw 20 times to find this group? If I took a group of five, didn't like that, put it back. Took another group of five, didn't like it, put it back. Took another group of five, didn't like it, put it back. And finally, on my 20th try, I got this group and I liked it the best and I kept it and I said, okay, what's the p-value of this? Well, guess what? Whatever p-value you get from that draw is completely irrelevant because you kept drawing again and again. Well, this comes up a lot because if you think about it this way, if all of our numbers are completely random, about one out of every 20 is going to have a p-value less than .05. So if you see something that says, hey, we tested 100 items, found five of them to be significant at p less than .05, well, guess what? You get that exact same result if you were looking at pure random noise. Now let's look at significance versus magnitude. It is possible to be highly confident of a very small effect. Now you can get this sort of thing published, but practically it may be completely useless. The things that you can tell from magnitude, sometimes if the technique is very complex, you may not be able to tell exactly what the numbers mean. Numbers resulting from complex statistical techniques may not be directly interpretable in terms of real-world magnitude. So what do we do with them? Let's say you've got a table like this that says the impact of children on the probability of being an exclusively secular giver is negative 0.089, but there's no way to interpret what that number means. It's not directly translatable to a percentage. So what do you do? Well, even with these complex techniques, we can easily compare sign and magnitude relative to other variables. So I can go back to that same table and I can say, well, it looks like race and education factors are three to four times as large as the impact of having one more child. We can also say that more children have an opposite relationship compared with more education on our outcome. This works almost all the time. The only time it doesn't work is for odds ratios. Odds ratios are the weird duck. Usually you can compare sign and size, but guess what? Odds ratios are always positive. So what do you do? Odds ratios are the odds of an event occurring in one group over the odds of it occurring in another group. So if you see an odds ratio, you can just subtract one and you'll get to the same point because anything less than one is considered negative. Anything above one is considered positive. Anything equal to one is considered to be a zero. So here's an example that shows you an odds ratio of 0.92 in this particular study is equivalent to a beta value of negative 0.08. And so the negative values in a normal approach reflect something less than one in an odds ratio. How do you find academic research articles? Well Google Scholar is the most common way, but guess what? Google Scholar gives you everything. Everything including working papers, gray literature, industry literature. Maybe you don't want everything. Maybe you put in a topic and you get way too much stuff back. You can go to Web of Science if you have access through a university library and then you're going to get only the highly ranked academic journal articles will come back from that search technique. Here's a little bit about how to read academic research even if you're not an expert.