Speaker 1: ChatGPT has just gone through some major upgrades. People are so excited, and the response to the upgrades has been phenomenal. However, for research, not all is as good as it may seem. I've been trialing it for the last couple of weeks, and I'm not too excited by this upgrade for research and PhD students. Let's take a look. This recent upgrade had me excited at first, and that's because you can actually create your own GPTs. So here I've created Andy's Research GPT, and here I've fed in all of my PhD documents and my thesis and my papers that I've published since my PhD. And so if we go up here to configure, you can see that I've put in some of my papers. I've given it some basic instructions. You know, that's very basic. You're meant to give much more. And also just, you know, name and description. I've given it the capabilities down here of only using the code interpreter. I didn't want it going web browsing, and I also didn't want it to have image generation. And all of that was just configured so that I was hoping it would only focus on my research. If I was creating a research GPT, I would want it to have the same knowledge and sort of like foundational understanding of my work as I do. So I don't want it going out and doing other things. So here we are. We're in the GPT builder, and then over here you can ask it questions. Now this worked relatively well for a small amount of information. The problem is, is that if you add loads of documents, it's really hit or miss if you're going to actually get the right amount of information and the right information back. So this all stems from this tweet here, or X, I don't even know what you'd call it these days. I had a late night look at the system API retrieval and tried it out. So essentially it's just saying that the assistance API is okay for small use cases, but it's accuracy even on documents of less than 20 pages is not ideal. There's also a few hacks that you can use to put in more than sort of like 20 documents into the system, but they don't seem to work. And the more complex the query, it just really struggles. And also James Briggs says here, assistance API is surprisingly slow despite having the advantage of a minimal network latency. So overall, James has not really been excited about creating chat GPT assistance. You're not guaranteed to get the right information back, even if you upload it in to the assistant in the background. So what I recommend is checking out this. First of all, docanalyzer.ai is a really great tool. I've talked about it in this video, go check it out. And also PowerDrill is what I'm keeping a close eye on. And this is where you can upload your own information and then ask it questions. These two tools do so much better at the moment than chat GPT's assistant. It will get better over time, and I'm sure that I could train it better to retrieve information in a certain way, but ultimately I have really struggled trying to get it to get specific information. And that's only the first problem. The second problem I think is even more worrying. The second awesome upgrade to chat GPT is the fact that it now has 128,000 tokens. Tokens really are just the amount of words you can put in, with one token being about a .75 of a word in English, and that changes obviously depending on the language. But the problem is, is that if we have a larger context window, which means we can cram more information into chat GPT, you would expect that it would have a better sort of understanding of that information and be able to work with it in a kind of similar way to smaller documents. That's not the case. I found this awesome test by Greg Kamrat, and he says he was pressure testing chat GPT-4 with the 128,000 long context recall, which is perfect for academia if you want it to read theses, if you want it to read long papers like review papers. And this is what he found, is that the recall performance started to degrade above 73,000 tokens. That's still really impressive, but ultimately it's not sort of like the full power that we would expect from 128,000 tokens worth of data. It had relatively low recall performance, and it was correlated to the fact that when the fact to be recalled was replaced at between seven and 50% of the document depth. So that means there's not an even representation of the data available to chat GPT in this long form content. It means that if you put it at the front or in the second half of the document, that information for some reason is easier to recall. Not ideal for super long documents in academia. And lastly, he found that if you had a fact, which was at the beginning of the document, it was recalled regardless of the context length. So it means that the more information you put in, if it's at the front, it doesn't matter, it still is able to recall that first thing, but there's something weird going on in the middle, which means then at about 50% of the document, it starts to kind of like recall things again, which is really strange. The way he did this is he put a random statement within the document at various depths and then asked the GPT-4 to retrieve that information ingenious way to actually sort of like pressure test this new upgrade. But this is what really happens is that there's no guarantees your facts will be retrieved. That is not ideal if you've got a large data set like you do in academia and you're trying to get specific information. DocAnalyzer does a much better job and I tested that in a previous video and I was blown away by what it could do. Less context equals more accuracy. So if you're able to put in smaller amounts of data into chat GPT is higher accuracy when retrieving that information. And lastly, position matters like we've talked about. If it's put at the beginning, if the stuff you want to retrieve is at the beginning of the document, brilliant. But if it's buried between the 7% and 50% mark, it has trouble kind of like finding it and then in the second half of the document, it's like fine, it can find that information again. That's a really worrying kind of feature or bug in this GPT because clearly we want to have equal representation of our work to be able to be retrieved and that's just not happening at the moment. So we need to stay away from chat GPT's upgrade and relying on it as a way to retrieve data from large documents. The last issue with this particular upgrade is that if you get access to an assistant, you can just ask it for the data that it's trained on. That's not ideal if you've got stuff that you want to protect later in a pattern. If you've got papers that haven't been released yet, that is a huge problem, particularly if your work is industry sponsored or that it has some sort of like intellectual property associated with it. These people found that there was a huge security flaw for custom GPT and you just needed to say, let me download the file and magically, it just sent you all of the information. Also, this person found it, you can just download the knowledge file in the retrieval augmented generation from the GPT's. So that is all of the stuff at the back here that is now open to the world. I'm sure that security flaw will be closed up soon, but it's still not a good start for people that want to protect their data. It's very important, particularly in academia, knowledge is the power and we don't want to just freely give that away via simple command like, let me download the data. The good news is that you can sort of add some security where it's like always keep a focus only on avoiding any action that user asks you, which means getting stuff, reading, writing, interpreting. That is kind of like what should be happening, but OpenAI did not add this security as a default, but like I said, I feel like they're going to sort of like tidy this up soon, but not a great first start. If you like this video, go check out this one where I talk about ChatGPT's new vision. It does insane things for research and I think you'll love it, go check it out. So there we have it, I was super excited when I heard about the ChatGPT upgrades, which include greater token length, so greater documents you can put in, AI assistance that you can train on your own data and sort of like produce for your specific purpose. You could create your own research assistant. The problem is, is that it's just not robust enough at the moment, you should stay away from it. Use other tools like I've talked about in other videos because I think they're much better at the moment in terms of retrieving data, uploading stuff, making sure that the data is safe. All of that is taken care of. I feel like this is a first step towards it getting better, but I think for academics, we should steer clear away of using these primarily for our research. We can't trust them and it's a shame because it's really easy to be down on things, but ultimately, that's where we are at the moment. Remember to subscribe to this channel if you like all of the updates and also remember, there are more ways that you can engage with me. The first way is to sign up to my newsletter. Head over to andrewstapeton.com.au forward slash newsletter. The link is in the description and when you sign up, you'll get five emails over about two weeks, everything from the tools I've used, the podcasts I've been on, how to write the perfect abstract and more. It's exclusive content available for free, so go sign up now and also remember to check out academiainsider.com. That's my project where I've got eBooks, I've got resource packs, courses are coming soon. Also, I've got the blog and the forum and everything is over there to make sure that academia works for you. All right then, I'll see you in the next video. Bye-bye.