Turn Limitations Into Strengths in Peer Review (Full Transcript)

[00:00:00] Speaker 1: Aha the dreaded limitations section I'm so glad I got asked this question and one of this week's live sessions. How do you deal with the limitation section? I see so many papers that don't even deal with it. What put myself out there List of all the weaknesses of my paper. I don't want to do that so I see some actually skip over it and others make it sound like they messed up or almost like are confessing to a priest or something like oops and kind of burying their head in shame but the limitations if done right is quietly one of the strongest and most important sections of your paper this is almost your opportunity to get out in front of critiques from reviewers kind of imagine putting on your i don't know chain mail breastplate whatever that's going to armor you up that when they come to sling arrows at you it's like just going to bounce off of you because it's like ha Aha, but I already thought of that and think about this instead. So what I wanna do very briefly in this video is show you common mistakes that I see made in limitation sections and how you can do it better so that you've got the right mental model, whatever paper you're working on now and into the future for doing a winning limitation section that's gonna help you sail through the peer review process. So what's the problem with limitations? It's that, look, any paper, just be aware, any paper is gonna have limitations. No matter what paper, it's kind of Stuckler's iron law. No matter what paper you do, it's gonna have a weakness. And remember, your reviewers are seeing your paper with an eye to spotting weakness. They're intentionally with a critical eye trying to find what's weak about this paper. How can I make it better? How can I improve it? If they're doing a good job as a reviewer, bad job as a reviewer is they're just gonna be poking holes. But either way, they're gonna be looking for a critique. Good reviewer is gonna help lift you up and help make the paper better. It doesn't always work that way. Another conversation on broken peer review. So take a moment, and sometimes we even recommend using AI to help you spot these. Take a moment to think about what are your paper's weaknesses and get out in front of that. Ask colleagues for opinions as well. What's weak about my paper? If you were a reviewer, what would you spot? We actually do this inside our mentorship communities and have an internal round of peer review and have colleagues comment on and spot those weaknesses. And you actually want to acknowledge it. Error on the side of transparency here. Bring it to the forefront. Don't just, if there's an elephant in the room challenge of your paper, don't try to hide it and be like, I hope those reviewers aren't going to spot that. Get in front of it. And the right way to get in front of this is by reframing it as either outside the scope of your current project, so that you say everything's tidy and contained here and future research would be great to deal with this limitation or you try to say that it's a field level limitation. To say it's not just a problem with my papers, our entire field needs to address this problem. Let me give you an example. So suppose you're doing, let me use a quant example and then I'll use an example from a late review. So, let's say you're doing a quant paper and you use cross-sectional data. So you just have a snapshot, right? Cross-sectional is just one moment in time. Instead of following people over time, that can create a reverse causality problem to where you can't tell chicken and egg kind of problem. You can't tell which comes first, your independent variable or your outcome variable. Okay. I digress. I don't want to get into methodological discussion, but one is maybe you want to say, well, as we use cross-sectional data, we could not establish temporality, that which came first, and reflecting the need for longitudinal data sets, or reflecting the need for future research to collect and track individuals over time. And then you might follow up that limitation and close the narrative arc to say, though, that nevertheless, something that was strong, or how you dealt with this limitation in the best possible way. That's kind of your ideal formula for a limitation. Instead, right, a weak way of formulating this, what you don't want to do is just leave it as, you know, we were limited by cross-sectional data, which couldn't establish reverse causality. Right? That doesn't close the narrative arc, and it leaves it hanging that there's something wrong with your paper. And it leaves the reviewer to think, well, okay, well, this is just weak cross-sectional data. You need to complete the narrative arc in the limitation. As another example, in a systematic review, maybe you want to say you didn't do a meta-analysis, which is a way of quantifying the papers. You don't want to just say what I sometimes even see, it's like, well, we did not perform a meta-analysis in this paper, which is a limitation. That makes the reader think, well, you just didn't do your job properly, right? Instead, you want to go through and say, no, due to wide heterogeneity in measuring outcomes or exposures or something, it was not possible to perform a comparative analysis quantitatively, or it's not possible to perform a meta-analysis. Future studies are needed to better standardize measurement approaches to better facilitate qualitative quantitative comparisons. So what are you doing there? You're again point taking this and pointing it to a field limitation saying future research needs to address it saying it's almost like imagine the reviewers going to charge it you like a bull and you're like oh here's the uh here's that red uh red flag and I'm just going to make you sidestep and go over there the direction I want to reframe this limitation and have you go. So I hope you're beginning to see, right, if you just leave it, if you leave, imagine if you left that section out entirely, that bull's just going to charge at you anyway. Take this opportunity, this gift of an opportunity, to set out the limitations and reframe that discussion so it's going to be favorable to what you did, and so you can't get caught with a reviewer, ah, gotcha, you didn't think about this obvious thing, you're an idiot, kind of moment. You don't want that. And I think sometimes people shy away from it because they know their limitations and they're nervous about it and they just kind of want to hide it and hope nobody spots it but that really is the wrong way to go. The last kind of reframe that I mentioned briefly is to say it's outside the scope of the research. Sometimes they'll push back on this and that's okay but it's still worth doing if you don't want to get dragged into doing it. So if you haven't done something that they think that might be important then this is a a great way to go. So sometimes I see people, maybe there's a subgroup analysis, you want to disaggregate, I don't know, by gender or something. And you could just point to say that, you know, in this study, come up with a reason, due to perhaps small numbers, we were not able to perform a subgroup analysis, or you might just want to say, you know, right, if you just could put it as a point for future research and limitations that our future research is needed to better disaggregate by gender as maybe women could exhibit different behaviors or responses for some reason. And there it slips a bit if you're going to go outside the scope you might not even want to acknowledge this limitation but start you might want to move that to your suggestions for future research section of your discussion. By the way if you want our whole guide walkthrough for the discussion check out this video up here where we go through how to write the discussion, and once you see it, you can't unsee it. There's actually a formula for writing the discussion, and when you have these mental models and these formulas, it just saves you a ton of time because it takes the guesswork out. I wish I would have this when I was first starting out because I didn't have the formula and I was just winging it on my papers every time, and some of the professors I work with, once they get these formulas, they're like, wow, I had 15 papers on the shelf and I was able to plow through them so much faster because I'm working with an actual system for publishing rather than just every time doing kind of muddling through, kind of groping for stones to try to get across the river and figure it out. Final little bonus tip. Sometimes it's helpful to leave one thing out. I know this sounds counterintuitive, but remember your reviewers are going through and trying to spot something, something wrong. So imagine it's like, you could just be like, oh, here's like a pothole right there. And you actually leave it so you want them to step in that pothole specific rather than start grasping at minutia that's gonna be annoying to fix. And so, for example, one tactic we do on systematic reviews, I don't know if we'll continue doing this in the future, is we often in the first pass don't do a quality assessment. And the reason is, is we know the reviewers are gonna say, do a quality assessment. And we want them to bring that up. Well, we don't wanna do the quality assessment in the first instance because it adds an extra month, doesn't fundamentally change the story of the paper usually, and often the reviewers, you could do the quality assessment and then the reviewer wants you to use a different quality assessment tool. By the way, quality assessment is just a rigorous way of assessing how strong methodologically is the research that you've included in your lit review. It's just an example of something though, this pothole that we intentionally left there. So when we leave that out, usually we leave it out because it's big enough that a reviewer will spot it, but not too big to where they think they're not going to be able to do this and it's going to be a deal breaker for the paper and cause them to lose confidence. So it's just one of those tactics of that can be an intentional kind of limitation that you leave out. But again, use some of these advanced tactics sparingly, master the basics, and then start getting into this more advanced publication strategy that we employ, especially inside our mentorship communities. I mean, what I've got on this channel is fantastic, but we can do a whole lot more working more directly together in a one-to-one mutually supportive way. I hope you got value from this video. Check out that full discussion training and I will see you in the next video, guys, with our best publishing tips.