Using AI in Research Without Sacrificing Rigor (Full Transcript)

[00:00:00] Speaker 1: One year ago I did not recommend using AI for research, especially not for beginners. My official recommendation was to first master the fundamentals, learn the basics, and AI was like strapping a turbocharger onto a car for someone who didn't know how to drive and could literally drive beginning researchers off a cliff. The failure modes were well known. I articulated it actually in a video that's right up here that you can go remember. Fast forward one year, everything's changed. I now officially recommend using AI in the right way, in a way you can be proud of, not a sketchy way that you have to hide in a corner, it's like oh I hope they don't find out about me, but AI workflows that you can stand behind, you can be proud of, share with your supervisor, and he or she will probably want to learn them from you because they're that good. So let me run through the assumptions and what evidence was first leading me to vividly not and actively not recommend AI to early stage researchers and what changed and what that means for you. It's a good test against your own AI workflows to make sure you're not falling into the some of the older failure modes that are still applying, that are still there, but with a new generation of AI tools, connectors, and skills, as we'll talk about, you can avoid these old pitfalls and traps. So the AI failures that were the basis of my stance against AI was the following. First, AI would often act as a sick event. It would cheerlead you into a dead-end topic that was not publishable. It would cheerlead you as you drive off a cliff and it really could suck you down a rabbit hole saying, oh, would you like me to do this? Would you like me to do that? And a novice researcher not being able to see what good looked like, not knowing good for bad, could just get pulled down into the rabbit hole and not even realize it, and then months later only surfaced to realize their project was unviable and unworkable. Second was that AI would hallucinate. It would just generate fake references and there have been a lot of studies now in the papers that came out over the past few years and found a massive rise in just clearly fake references in journals that got through our systems and still got published with completely made-up references. Third problem, AI just not being reproducible. A hallmark of scientists, like two core features, is reproducibility and transparency. That's what sets us apart from just a random person on the street with an opinion. We set up our scientific process so that anybody can take the steps that you could take, follow the same steps, get the same results. Well, that still is a problem as I'll come back to, but there have been big leaps forward in how this reproducibility can be handled and the transparency that that's involved and goes along with it. The other problem that beginner researchers I would see do is they would get AI to deliver swaths of seemingly polished text that once you would poke at, it was almost like pulling a thread that would unravel an entire sweater or like the house of cards would completely come crumbling. So researchers were using lots of jargon and stuff that they might not have even fully understood. Polished text that a busy supervisor might pass along but only later realized was AI and wasn't internally coherent or made sense. So it was polished text that wouldn't stand up to scrutiny, often voluminous. I had a researcher with a 78 page lit review that was doing none of the things that literature review should do. And the last is that the AI would violate field norms. It would just apply kind of blanket. Here's the prism of guidance on systematic reviews, but not really calibrate to the specificity of a project. Just this kind of blunt instrument application or even deviation from field norms. So these are significant failure modes that could get a beginner into trouble. All right, why have I changed my mind? What's different now? Well, the first is ability to create effective. I was one of the early users of creating custom GPDs and chat GPT, never was happy with them, but the ability to create customized guardrails. So now the fully ethical model of AI use can be built into your research workflow. And I've done that personally with a series of Claude skills. I've got a topic validator skill that's going to bring you through our process of finding a winning topic using our duplication test, our impact test, our feasibility test, tests that you always need to do. You may have been doing them and not even been aware that you were doing them. But for a beginner researcher who may or may not have an excellent supervisor behind them, this takes them through a guided process with the guardrails. So they don't go off a deep end and fall into the sycophantic trap of AI saying, Hey, you've got a great topic that you were just duplicating with somebody else's done. So the first thing is guardrails can now be built in with customized skills that are really quite effective. And I've got the just download zip file. I've got another video showing you how to set all that up. It's down here below. So you can test that out and see what those guardrails look like for yourself. We've got them throughout the year, all tasks you can do in research. We've got the right workflow for you. Second hallucination. Well, now you can give AI's vision into real literature with a connector. I've made a connector for you. I was not happy with any of the paid connectors. So I made one for you. I'm actually paying for it myself, but that is my gift to you. Use it. You're going to love it. It will literally change your life as a researcher. I I'm jealous. I wish I could go back 20 years time before I'd published 400 papers and had this tool would save me a ton of time. I love Google Scholar. I still love Google Scholar and it has its place, but this is now on equal footing for me. Check out this connector. So gifts insight gives Claude or whatever LLM you're using vision into over 250 million real academic papers, not just open access ones. It takes you straight to the DOI so you can verify it. So the way you solve that hallucinations by verifying, and now verification is built in through those connectors. Reproducibility. Well, before, right, you would try to get an LLM maybe to make you a figure or analyze your data, and you couldn't retrace the steps. But now there are steps like you're this one with Claude in Excel, Claude can live inside your Excel. So you can actually follow the exact steps it took to analyze and see what was done to your data, check its assumptions and where it got to. And with explainable AI, it's not perfect. Explainable AI is kind of a model trying to explain what the AI did. So a bit black box, the reproducibility is better. I'm not going to give this fully a green light like I can on the previous two, more of a yellow light reproducibility is still a problem. And you can see this also, as I've discussed about how AI can and can't be used in systematic reviews and why I actually still don't advocate it for a lot of systematic review workflows, just because of field guidelines, see that here, reproducibility and full transparency is still not the same as what you'd have with a fully human process. But the ability of AI to sit inside the tools that do the analysis, so you can revise and use the code and report on that code that was done is a huge step forward towards reproducibility. And I think where we're going now is people who do use AI, I always say err on the side of disclosure, disclose it, use AI proudly, use AI in a way you can be proud of, or if you're not, there's something wrong with your AI workflow. And we're going to be disclosing the prompt that we actually use in the journal, what prompt we did for what and what it did to the data. That's the direction I think things are going. And finally, in terms of field norms, you can codify that with skills. And skills are again, like giving these customized GPT instructions to your LM, but they've just gotten a whole lot better. And they're easier to build. Now, they're basically like standard operating procedures, but they ensure that your field norms are being applied better. Is this still perfect? No, I'm still going to give that a yellow light. And the same thing I'm going to give to the previous one, which I just skipped over the Polish text, the AI will still give you to Polish text, and there's still signatures of AI text out there. So still, I don't think that should be a main workflow, I still think you need to master the fundamentals and have a writing system. So you can actually improve. It's kind of like if you have a robot arm, you want to gain muscle, and the robot arms lifting the weight for you, well, you lift with weight, but you didn't really get the muscle, you just the robot did it for you, you got the result, but is it really the result that you want? So the large pause text, I still want you as a beginner to learn what good looks like. So I definitely think there's some that are in between two extremes, ostrich, head in the sand, don't use AI, grumpy curmudgeon-like behavior, which Professor Stuckler is guilty of sometimes, but on the other side, AI is going to solve everything and get you in a Q1 journal, and you just pop it in this AI tool I made, and it's beautiful. So this nuanced way of using AI, I think is passing critical things that lead me to say, with the safeguards in place, the guardrails, through the connectors, through the skills, through AI's living inside, say, your excels, so you can be reproducible and transparent. I now have found it actually to enhance the workflows, the learning process, and the steps beginners can take without compromising you actually learning and acquiring the skill. So what do you think? Do you disagree with me? Are you not recommending AI? What guardrails are you putting in place for yourself? I personally find that, you know, it's easier to use AI safely if you already know what good looks like. It still is a turbocharger, but I find now AI multiplies what you can produce if you know how to do it, and by locking in the right workflow into the AI instrument itself, for me, has been a game changer, and it's like the economist famously say, when the facts change, I change my mind too. I'll continue to curate and bring you the best AI tools, and when I'm not happy with them, I will make new ones for you, like with the skills and the literature connector that I've got for you below. They're free. They're lovely. There's nothing else out there like them. I highly recommend it, and look, guys, I'll see you in the next video. Disagree with me? Let me know in the comments. I read and personally reply to all of them.