How AI Can Stress-Test Papers Before Journal Submission (Full Transcript)

[00:00:00] Speaker 1: Professor Stuckler here. Listen, you need to be using AI to peer review your paper, before you submit it, and even potentially at earlier stages of development to know that it's publishable in the first place. I don't think I would have said this if we were having this conversation just a year ago, I wouldn't have said that. But the field has changed. Even I personally, I've published over 400 peer reviewed papers, did it without AI. And in just this last year, everything's changed. I now use AI, have it fully integrated into my entire workflow that I use personally, and that I use with the researchers who I work with. So why peer review? Well, internal analyses have been done of the peer review process by journals, and they found 53% of reviewers were using AI as part of the peer review process. I mean, it's not hard to see what's going on. We're asking peer reviewers to do free work. So they're dropping your paper into an AI model, asking the AI to do the heavy lifting and come up with the points that it needs to make, and maybe then they're trying to humanize it on the back end. Putting aside all the ethical quandaries with this, there's just a reality that you're going to have to deal with. And so what that means now is you need to know how to be in front of this, and you need to anticipate what the AI may say about your paper. In some cases, it may say some nonsensical things. You need to be aware of that. We've actually gone through an appeal recently with a journal when a reviewer seems to maybe dump the wrong paper into AI and came up with a completely huge 10-page peer review clearly bearing the mark of AI, and we managed to get it thrown out. But that was cumbersome and laborious and really happened because I think the reviewer had the wrong paper loaded up to the wrong journal. But regardless, I am hearing these horror stories of 13, 20 peer reviews coming in that are very clearly not human generated. So here's what we're going to do. I'm going to show you how you need to do this yourself, and you need to do it in the right way so that your manuscript doesn't get out, you've got to have the right privacy settings on your AI. And then I want to show you how you can use AI as your devil's advocate, as your fiercest critic along the way to help you strengthen your own research. And that is part of our AI workflows that we use. I want to bring you AI workflows you can be proud of, that you can share with others, that you don't have to hide in a dark corner. I'm going to humanize my AI and hope I don't get caught. No, I want you to be proud that you're the leading generation using AI, that your supervisor is even going to want to learn from you how you used our AI workflows in the process. And by the end, you'll have this for one of the workflows that we recommend, which is on simulating peer review. So let's go. You can do this with any LLM, I just happen to recommend cloud. One thing before we start, I want you to go into your settings. So click on your user, go into your settings, go into privacy, and go into your data. So here you can manage all your stuff on your data. And what I want you to do help AI improve our models, I want you to turn that off. If you don't already, I just don't want it saving your original research and your manuscript to use it for trading coding sessions, because then anthropic somebody can be querying about your topic and get an answer to a query from your paper when it's not even published. Just don't do that with your research. I also recommend you consider for research making projects, but I don't want to do that right now. Because right now we want to simulate as best we can what's going to happen if some peer review out there reviews your paper. So the way you're going to do that, what I recommend best, is going into incognito. Because if you use it here, it has memory. It already knows stuff about your paper. You actually need to strip all the information you've already given the AI about your research, and kind of blind it to best anticipate what's going to be coming from the peer reviewers. So click on this little funny looking ghost guy, use incognito. Any of the LLMs should have this functionality. Now we're incognito, I'm going to load up a manuscript of something that's already been published, just for the sake of example. And you can see here, chats aren't saved, either memory or user-trained models. Great. This is an extra layer of protection for this. So imagine the name of the game here is not doing the peer review for your own knowledge, which is also valuable. I'll show you how to do that as you're building up your paper. But I'm intentionally not choosing the most powerful model. So we're going on Opus. Again, I'm trying to imagine what my peer reviewer is going to have, as best I can. And say, conduct a peer review, even naive. I'm going to assume my peer reviewer is dumb and doesn't know how to prompt properly. Conduct a peer review for what journal? Let's say the American Journal of Public Health of this paper. I'm not even going to say what type of paper, I'm only going to say for the specific journal. And then I'm going to let it run. And then we sit and wait for a second. Because it's going to take a moment. Whenever you do this, you feel some kind of excitement. Like, oh, I'm getting a peer review. I wonder what it's going to say. It's incognito. You should have that sense. I mean, your paper should feel a little bit like your baby. Like, it's your darling. And it's like, oh, I'm showing my baby to the world. What if everybody hates it? And it's my beautiful baby. Okay. Because, you know, I've been an academic for a long time. So some of the nerdiness does slip through. I do think this is the most fun generation. Because you can use all these tools now that are, they make research a lot more fun, but dangerous. You'll see that our tool's intentionally building guardrails. So intentionally for that reason. So you don't get yourself into trouble. We used to have a running joke on some of the data analysis software that my generation trained on. And they would say, what do they call a grad student with Stata? Dangerous. Wonderful academic humor. If you're dangerous just with a data analysis tool, what you have at your fingertips with AI now is tremendous. Okay. So it's checking the author requirements, which is good. It's running structured peer review. We don't know what mental model it's building off of to do this peer review. And that's great. That's intentional. Here we go. I've calibrated it to their headings. Actually pretty good review. It says when you do a peer review of the paper, you want to say what the paper did, showing that you know it, and then the comments itself. Okay. I think I might've picked the wrong journal, but it found fit. Fit is the number one reason something gets rejected. This is good. We actually have an entire workflow on helping you pick the right journal because fit is so important. You can have the best paper, but if it's not a good fit for the journal, it's going to get rejected. But this is helpful. Comment with the largest consequence. Yeah. The AJPH is clearly not a good fit. That's not where it got published by the way. Public hate case is asserted rather than made. Okay. So yeah, it's pushing back on the fit. That's a fair comment. It would probably get rejected from AJPH for that reason. Comparative claims outrun what vote counting can do. This is good. So this is a substantive comment. So when you get these comments, you need to first do a sense check. Does this make sense? Sometimes the AI will not know your field norms and it will just kind of blindly pass on things that may or may not make sense for what you're trying to do. And that's an error. There's a nuance error that the AI doesn't always catch, but it will find internal coherence. And if it says you're kind of overstating what you can do, like your synthesis method is going too far, then these are comments that you want to know. Now you may not be able to address all of these. You may not need to. So the best way to defend against this is my general recommendation in the paper is to go to the discussion section and put them as a potential limitation so that you say, Hey, reviewer, I'm aware of this. Or in this case, you're saying, Hey, AI reviewer, I'm aware of this. And here's how I've dealt with it. So if your AI then goes back and lands on it, it's already like, Oh, but you already dealt with it in this way. That was the best way possible. It's almost like a peer reviewer is going to charge at you and you're holding out the red carpet and saying, Aha, well, I'm just going to have you glide that way past me. That's what you want to do here. You want to go and assess what's going on here. Quality assessment tool. We always know this is going to come up. And we, it's a tactic we use on a systematic review. We keep it out because the AI could recommend a different tool for multiple reasons. Then you've got to redo it anyway. It's such a pain. So we actually tactically at this moment in time, again, our workflows to as the field evolves, we evolve to, but right now we actually don't do it in the first pass for multiple reasons. But this is a comment I would expect. And the gap as state is wider than the gap the paper actually fills. So again, there's a little bit overstating, no review is comprehensively evaluated. But you know, this kind of thing, there's some nuance here. So you just need to make a decision what on these you want to get, but what's critical about this. So because I was involved in this paper, I know it's not going to get caught completely off. And this is an older version of it, not the final one that went out, just using it for the sake of example, but AI is going to help you avoid embarrassing mistakes. It's going to avoid rookie mistakes by doing this test. So it behooves you to do this. And all our researchers now go through both an internal human review, and this internal peer review process. Now, if you wanted to do this in a way as you are building up the paper, so it's also caught these we also have a full reference checker workflow that that we use is one of our skills you can upload to Claude, I'm not going to go into that I've got other videos that cover skills for you and how you can use them. And it's doing a desk reject scan. So this is all this all very helpful here. Oh, dear. Oh, dear. Okay, part of the reasons output is good. I put on incognito, and it's still invoked a skill is still invoked a skill. Okay, guys, I'm very sorry, I feel horrible. I made a skill that is the one that we use in our paper, not the submission checker is the peer review. And I wanted to limit it to a few major comments. This is brutal 13 page stuff. But I actually wanted the 13 page stuff, we're gonna have to run this again. So even the incognito browser didn't inviolate that, okay, I thought this feedback was looking pretty good. And wasn't looking like a terrible AI peer review, I often get this actually ran it using our model. Okay, that was not an intentional side of hand, I should have tested this beforehand, we're gonna run this again on a new incognito window. And I'm going to intentionally say don't use any skills. Let's see how it comes out. Now. Don't use any skills and do a peer review of this paper for American Journal of Public Health, what I may need to do is just go into chat GPT and run this for you guys. Okay, it's trying to do it. Oh, no, wait, any of the okay, it's doing that wrong. Don't use any of our Claude connected skills, because it will automatically invoke a skill where it thinks a skill is appropriate. But right, that's the problem. You have to be very careful. Because if you're you already you kidded up your AI, it's not going to give you the feedback that's going to reflect the feedback that you're going to get from reviewers. I had a feeling of that feedback was just too, too good, even with the incognito browser. So okay, so it's doing without the external tools. Let's see what it comes up with doing it this way. So this is going to run in the background. Again, round two guys, what you'll have seen on the other one was actually our review skill that's going to encourage all the researchers to come to a workshop that we offer we have a series of private workshops. Because when you get AI feedback, you need a real human feedback from professors who review and publish for a living to look at it and help you separate the wheat from the chaff. Well, the beginner can get a whole lot of feedback and not know what's really important and what's not. That's really hard to do when you're just starting out as a researcher. So let's let this run for a second. Okay, this is looking better already, just while that's running. So what I've done to make the peer review process internally more effective, and this is important for you as you go and instead of having a million different comments to action, I've set up a review to highlight just on the most important comments, the major ones to cluster them and condense them and get rid of any of those rookie mistakes that can lead to a desk reject because we're really optimizing on our submission to get to a revision for us revisions win because we have an over 90% success rate. Once we get to revision, the vast majority of papers are killed at the desk reject stage. And that's why tactically we have a few things like even if we've overstayed a little bit the gap, we're just trying to inch past line and avoid desk reject. That's just some of the tactics and craftsmanship. Those are more advanced tactics. And some of those more advanced tactics, I really do work better and working together in a more direct capacity inside our mentorship communities or as part of our guarantee programs where we work with you all the way from the beginning to the end and stick with you guarantee we're going to stick with you until your paper gets published. So we can't guarantee which journal or when no one can watch out for anybody who promises that that's a little bit iffy, you might want to see another video I've done on paper mills where they do do that. It's hair raising stuff is pretty awful. So you can see it's emulating some of the workflows. I've just tried to take away its peer review muscles that I've given it before that made that last peer review. I just had a feeling going through it. It was too good. But let's have a look here. Okay, we've got the report. So this is good. Major revision, AGPH fit. So a lot of these things are coming out. Let's see what it says here. I'm going to try to open this here on the side. Let's see if I can get this to be the full screen. Okay, here we go. And this is the kind of thing I'm talking about where you get this very long review. Look at this long review, this huge review. So how many pages is this? I don't even know. It's a lot of pages. Let me open this up in Word and let's check because this is the kind of stuff that I see coming out and can drive people crazy. So 3000 word, they get a review, eight pages, that's even longer than the article itself. So common, and especially when I see these bolds in the peer review. I mean, you can see why people are doing this. All right, this is this is the output that you want. But this is not the output I want you to work on yourself. So let's see what we jump out at here. Numerical inconsistencies in the results. These are things to check and tweak. Synthesis method doesn't do this extract tabulate effect. So this this Okay, I'm glad we spotted this. This is the kind of comment that is absurd for this paper. Absolutely absurd. It's trying to tip you towards a meta analysis. And so this is not what you want for this kind of systematic review without meta analysis is where it starts to deviate from field norms. So here, you'd need to protect against that by going into the discussion and explaining what you did. Because an AI reviewer would get exactly this and misunderstand it and would get it wrong. And the naive editor, I always assume the reviewers and editors are dumb. That's the best assumption might miss this. They might not get that they may not know that this is the kind of thing that I would really want to fend against this out of certainty of evidence grade. This just makes no sense for this paper. It's again, blanketly blindly applying field norms here. Not right, not right for this. And like I said, this paper later got published in a very good journal. So fortunately, we didn't have these but I think we also did maybe shore up our discussion section against this. Although we didn't get these specific ones. I don't think we ended up with AI reviewers, thankfully so because this is nonsensical. And this is the stuff that people in peer review are just banging their head against the wall. And you can see why I'm glad we went through this example because you can see if it was building off some context or some knowledge, it's going to mess up the peer review output you're going to get to be able to work on. Let me zoom in so you guys can see this a little bit better. Again, just highlighting the stuff that it was saying here. So again, this here is again, misapplying some of the field norms that can be done here. And so this is why you need some real human feedback on it. And if I get a comment like this, I'm like report an interrater agreement statistic. I mean, people don't really do this in these kinds of systematic reviews. It's as an AI comment. And this is stuff that when you go through peer review makes you want to bang your head against the wall. So what does this mean that you need to go through the methods and explain things a little bit further. And in this one, this earlier version probably needs to clarify how the multiple reviewers engaged, but this is clearly off. And guys, this is again, why we're going through this example, because you need to know that this mud can get thrown at you. And this is literally mud because it's nothing that you want to stick quality appraisal. This is this is fair enough. Again, I'm not saying you have to take all of these on, but you don't want to use it like publication bias, you can't do without doing a meta analysis. And this paper is not a meta analysis. So this would be a nightmare to get this kind of review. And I know people are getting these kinds of reviews, just a complete nightmare, you hope that the editors will have good sense and help you with this. But your best defense, rather than rely on them, is again, shoring up the things that you can shore up, so that this clearly becomes nonsensical. So in this case would be citing the guideline that you follow, citing the way that you handled the screening process in more detail, because clearly that wasn't clear enough, adding things to the discussion about how you use or didn't use your quality appraisal tool here, adding things about why you didn't do meta analysis. And this is all very valuable information for the peer review itself. If you want to do the peer review as part of an actual workflow process to actually improve the research, I would recommend using our more custom tools and of course recommend getting human feedback. That was a that was a lot of fun and deeply distressing at the same time for the state of peer review. And the horror that looks like the worst night of Halloween came to you in that kind of AI drivel peer review that people out there are getting. And remember, over 53% of peer reviews are now being conducted using AI. It's a structural problem. I don't think it's going to go away, because the volume of research has increased with AI enhanced workflows, but the number of peer reviews has stayed the same. There are just not enough peer reviewers to keep pace with all this research and this volunteer labor system is really at a breaking point. If you want a video that goes deeper into that, check out this one here. But guys, if you are interested in installing AI workflows into research you can be proud of, click the link below. Let's get on a call, see if we're a good fit to work together in one of our research mentorship communities. And in the meantime, you're not going to want to miss this connector that I've got for you right here that you can use with Claude to give a vision into over 250 million papers. I'll see you guys in the next video.