AI Tools Strengthen Research Checks and Reviews (Full Transcript)

[00:00:00] Speaker 1: Today we're going to do something that I hope you find to be a bit of fun. We're going to compare me, Professor David Stuckler, with AI. Me, AI, Professor David Stuckler. In the background, I've quietly worked on building AI tools that are freely available for you that will massively up your AI research workflows and ensure you're using AI in an ethical way that is calibrated and connected to a real coherent research system so that you avoid a lot of the AI failure modes like going down rabbit holes, AI sycophantically cheerleading a dead-end topic, and others that you can check out in this video I made here. So what I'm going to do is I'm going to show you the feedback that AI Professor Stuckler gives compared with what I would spot quite rapidly and quite quickly within a few seconds myself. And you can see the differences and the power of these tools. I hope you like them. I've got links for you below where you can use them yourselves. Just briefly, I work within Claude. I do recommend switching over to Claude. It's rapidly overtaken ChetGPT and Comparative Head-to-Head Test. I'm not sponsored by them, but that is where a lot of our research infrastructure tools now live. All you need is a Claude account. You can use some of the tools for free, but there's a limited number of them. I do recommend investing in the $20 package just to get you up to that next level. Another thing to say is I'm going to be using base models that are accessible to you and not the most powerful ones just to provide a more fair comparison of what the AI setup is doing and something that's accessible to you without open access, without having to spend buckets of money for it. Okay, I'm going to cover checking a topic, so making sure a research topic is sound and publishable. I'm going to check a research proposal. This is using our GIFT method. I'll tell you more about that in a second. I'm going to check an extraction sheet for a systematic review, which is a step of doing a lit review, where you've got to extract material from the literature and synthesize it. And we'll look at the performance of me versus AI me and all of these. So let's dive straight in. And at the end, I'll show you how you can incorporate these tools yourself. I think it's easier to learn by doing, so I'll show you how to use them and then how to set them up. So first, let me pick out here a topic that I've prepared for you. That's something that you might want to check yourself called check this systematic review topic and a kind of working title. Now, what I've done here is this is a skill in Claude. You can invoke your skills by hitting dash. I've got a range of different skills available. I just picked the topic Balligator skill. To deepen your understanding, skills are like standard operating procedures, kind of a set of instructions that you want to execute over and over and over. You might find in your research, there's standard kinds of functions and research things that you do over and over and over. You can codify these in an SOP, just like a business says, if my assistant was going to a, Hey, assistant, I want you every Monday morning, this time to do this and follow these instructions. That's an SOP. I've created these SOPs with our research systems so that you can deploy them on your topic. Now, looking at this topic, I break topics down into Pico models, which breaks it into its atomic elements. And I make sure they're each solid. And Pico models are population, intervention, comparison, outcome. It's one of many tools available. It's one I find very helpful to make sure your topic is well-defined. And what I can see here is you've got some vague features. So we haven't really defined. We kind of got a setting, but not a clear population. Like who in higher education, university students? Are we sometimes people that refer to high school here? Sometimes just university, undergrad, graduate, vaguely defined. Generative AI. I think they're looking at the impact. This is something submitted, by the way, by a real researcher that we covered in our workshops. This here, yes. But impact of AI. I think it's like connecting this impact of AI on academic transformation and multilingual writing development. I think behind the scenes here, I'd look at this and say, is this a second language learner of English or another language? What's going on? Are you looking at writing skills? Is it the impact of AI in teaching or feedback or something else in writing skills? So I'd want to clarify a lot of stuff. Let's see how the AI did. So TRACK figured out it's systematic review. It differentiates because that process to validate and test your topic is going to go in different tracks. If it's a systematic review or some other kind of paper, qualitative, quantitative, experimental study in a lab. And so it's saying here, the tests we need to invoke are feasibility. Do you have enough studies for a review? And duplication tests to make sure that you're not just duplicating something that's already been done. So this is good so far. It finds that in the process of finding your topic, you're in the topic neighborhood. But you haven't really honed in. You need to do some more foundation work. So it's actually AI saying, back to the drawing board. We need to do some things right first. But it's saying that this is definitely an important area. So you're in a good space. Now, the issues that matter. It came up with the titles carrying three constructs. This is what I found as well. Just looking at this at a glance. It was modeling some things. Nice. It doesn't really work as a PICO. So you need to clarify that before going too far. Very good. AI Professor Stuckler gets a plus point here. This is nice. Back to the worksheet to tighten this. Needs inputs. It's saying, hey, I need your nearest neighbor reviews. So this is where you want to go find the paper that's closest to yours to make sure you're not duplicating it. And you can establish the value you're going to add. I can't tell you how many researchers doing research don't actually have this. And this is core, fundamental. You need this. And it's giving you a warning. Duplication risk is high. Because it's moving fast. There's lots of systematic reviews. So you need to make sure you've overcome this. This is nice. This is nice. And just it's saying a narrow geographic gap. No one's done this because of some small derivative thing. Like no one's done this in the Philippines, right? Might not be enough here to survive peer reviews. So it's asking for a nearest neighbor papers. Great. So I went further and said, OK, well, help me find those papers using our literature search tool. This is a free tool I'm going to share with you as well. The skill is also free. But you can use on your own topics today. And loads on the connectors. Running the first pass. OK. So it searched using our tool. Real literature. So not hallucinating anything. Let's see what it came up with. Surface systematic review. Did multiple passes. This taps into the power of Cloud2Think. But also 250 million research papers at its disposal. It's finding some things that are busy. OK. So what I would have done here myself is I would have pulled up, probably before having this tool, Google Scholar. And I'd look for systematic reviews that are close. And find the ones that neighbor it. After I locked in the topic. And, but this is nice. Exploring the application of this and EFL. It's not writing specifically. So this gen, oh, this is a good one. Generative AI and written feedback study. So it looks like it's focusing on feedback. This is great. This is going to help you scope your field very quickly. And lock in your topic. And for each of these, these are reference points. You need to make sure you can add value to. So tell you what. AIMeet did a pretty good job here. This has done about, well, 75% of what I would have done myself. And if we were having a conversation right now. And you came to me with this topic. This is pretty close. Kind of scarily close to what I would be doing. And it's nice. It chucked out a whole bunch of noise from your search. This is something other tools won't get for you. It's because Cloud is built into the search. So nice. What this means for the duplication test is no one synthesized this exact claim. It's going to be hard to sustain against these guys. You know, just knowing that something's a dead end. Like you want to know that. Before you invest a whole bunch of time and effort into this. This is in some ways better than my feedback. Because it's reduced the data that you need right away. Where if we were working directly together. I'd be like, okay, let's go tweak the Pico. And go search for it right now. Already taking some steps here for you. Okay. Let's go through another use case. Let's look at a research proposal. So to anonymize it, I've buzzed out the name. And let's take a look. So what I've done here is this invoked our proposal checker. This works great for grants and proposals. It builds on, not the prospero check, the proposal check. Builds on our gift method. You can see some videos here about our approach to grants and research proposals. One over 10 million in competitive research funding. So I know a thing or two about common pitfalls in constructing grant proposals. And here's what it came up with. Again, I'm blurring out some stuff to keep it anonymous. And it's back to the drawing board on the gap. Okay. It says the instinct is sound. But the researcher here in the gap. Actually, it says the researcher contradicted themselves saying there's a gap on something. Then citing papers that actually filled or had that gap. So this researcher hadn't done the duplication test run and recommended that they run it. Very nice. Forces it to close the gap. And the gap is the first thing that I'd be sifting for. I want you guys in proposals to have one big gap. One big gap is better than that you can fill. Is better than a bunch of small gaps. And here you need your idea to come into contact with the gap. This is nice. It just listed. I see this too so often for students. It lists a laundry list of methods that you may not even be able to do. That doesn't even sync up to your gap. This is a common research proposal failure mode. So this is nice. Okay. I probably would have looked at this very quickly and said back to the drawing board. Your gap isn't clear. Let's move on to the next thing. Let's clarify the gap and check the literature first. This went a bit further than I would have gone already now. But it's been more precise and forensic and detailed in some of the issues than even I would have been. Professor Stuckler is getting scarily smart here. But it's kind of clawed in hands with Professor Stuckler. I'd say about 30-40% of this is coming from being able to search the literature effectively. 30-40% is from our SOP that we put in the skill. I'll show you in a second. And 30% is also clawed coming in. And again, I used a base model. So I didn't push very hard on the model using the most powerful fastest ones. Feasibility, another one. This is a big thing on research proposals. If they think you can't actually do the project, they're going to kill it. I have my own proposal skilled for similar reasons. And here we are. It's showing the timelines impossible. You guys need to know this stuff. We're going too far. Like, imagine you even get it approved. I see people get their proposal approved. And it's completely impossible. And the researcher comes to me and says, well, my department approved it. I'm like, well, they approved something that's impossible. Maybe they did a cursory job. They didn't check it properly. But I don't care what they approve or say. If you've got something that's constructed, that's impossible. That's the lock. But you're going to keep banging your head against the wall. This is really nice feedback. And it's going to push you to, again, I blurred out some details here. But it's going to push you to come and get real to our mentorship communities and get real human feedback. So that is in there. It's going to encourage you to go to either us or somebody else to get real human feedback at the right stage in the process. All right. I hope you find this helpful. You can skip forward in the video to where I show you how to load this up yourself. Let's do one other, which is a data extraction sheet. And you'll notice a common theme is that AI can do machine tasks really well. But the real human judgment, it's a little bit worse at. So you don't want to outsource your judgment to AI. And you don't want to blindly use AI. So you just pop these questions into AI. Without these tools, you're flying blind. You probably haven't synced up the full academic literature that it can tap into. Maybe you have. That's good if you have. But it may not be working from a research system that's coherent for the type of project you want to do. So it means it's optimizing in the moment, not necessarily taking the full view. And it doesn't know the full research methodology and the field norms and standards that you're going to have to follow. OK. Let's go to this extraction sheet. So it's an Excel extraction sheet. And again, what this does is, so can you help on an extraction sheet? It asks what stage of data extraction you are. And we're kind of at the early stages. It asks for you to attach the file and study counts. Because when you're extracting data here, this is a systematic review. You've already locked in the final set of papers. And now you're moving to analyzing them. You're kind of in the second phase, second half of the systematic review. And it's saying, hey, you've got to go back to extraction. There's stuff missing. So you're missing universal columns. So I would have picked this up right away. You're just missing data. We can't analyze data. So we've got to go extract some more stuff. And it's going to link you to the right training on that. This skill, by the way, is firewalled. Because it's just not going to work without access to our courses. So bear that in mind. But the topic skill is readily available to you below. The outcome? Yeah. So we have an atomic rule. Kind of one cell, one data point. And it kind of, wow, this is more than I would have done. 413 characters per cell. That's a messy extraction. You're trying to turn an extraction. You're trying to turn the literature in a systematic review into data. And this is not going to work. 65 of the 95. This is quantitatively dissecting this extraction sheet in a way that, obviously, that's not something I could have produced just off the cuff. Or I wouldn't have aspired to do. The 63 of 95 contain results, language, or statistics. So it's finding lots of misalignment in the sheet. And this misalignment is important to fix. Because you can't interpret or synthesize your data effectively if you've got all this messiness. Your data aren't cleaned up. And so I see people getting stuck writing. It's like, well, you can't write until you got this step locked in. And you're blaming your writing. But it's actually a previous step that wasn't right here. And it's caught a whole bunch of infelicities along the way. Oh, this is a big one. This one I would have caught right away. Study design doesn't contain actual study designs. Nice. Absolutely right. Yep. Genuine design labels. Oh, man. Again, this AI is... Duh. Wow. I mean, this is good, guys. Genuinely. This is... As somebody who's done this for two decades, this is very, very good. You won't get this output just from, like, I throw my extraction sheet in. Try it yourself. Throw it into ChetGBT. See what you get. Researchers routinely come to me with huge messes. And I hate it. Because I've got to spend all this time cleaning up stuff that could have been avoided. You waste so much time cleaning up your AI mess than just doing it right in the first place. This is really good. What you really want to know and might have skipped forward to here is, how do you do this yourself? So you want to go into Customize. We've got Skills. The skill that I've made available to you below is a zip file for Topic Validator. All you're going to do in Skills is go Add, upload the skill, and drag and drop that file there. And when you look at it more closely, it is going to give you... It is a very detailed set of instructions of what it's going to do. This is made to sync up with our connector. This is 100% free. And let me show you how to add it. It's going to Connectors. And this is what's going to enable you to see the literature. And I want you to load up this one here. FastTrack Literature Open. So all you're going to do to show you the process is, you're going to add a custom connector this time. And you're going to do FastTrack. Title it whatever you want. Literature Open. And drop in this URL here. I've got it for you below. And this is what's going to give Claude eyes into seeing the literature. So the literature comes in. Claude can engage with the literature based on the skill to get exactly what you need to complete the steps of our research system to go from start to finish in a coherent way without getting deviating off track or going down rabbit holes. It's incredibly powerful. It's what makes AI Professor Circler very good. And you guys can have complete access to it here by incorporating these two steps. I'd love to hear how you use it. What you find. Quality of the feedback. Any errors or hiccups or snags that you see. But if you haven't given Claude vision into the literature using our free tool, do it right away. Because it's going to solve big problems with AI hallucinating references. It's going to make it so much easier to sync things into Zotero. I've got a full video for you here on that if you want to see more detail on how to do it and how to use it to search the literature. Guys, I look forward to seeing you in the next video. I'm going to continue to curate and now even develop AI tools that I can bring to you in the spirit of open access. Bye for now.