Speaker 1: Hello world, it's Siraj, and I'm going to show you how I read research papers and give you some additional tips on how you can consume them more efficiently. Reading research papers is an art. Whether the topic is machine learning or cryptography, distributed consensus or networking, in order to truly have an educated opinion on a particular topic in computer science, you've got to get yourself acquainted with current research in that subfield. It's easy to agree with a claim if it's got enough hype behind it, but being critical and balanced in your assessment is a skill that can be learned. PhD students are taught how to do this in grad school, but you too can learn how to do this. It just takes patience and practice. And coffee. Lots of coffee. Every single week, I read between 10 to 20 research papers in order to keep up with the field, and I've gotten better at it over time. And I don't have any graduate degrees. I'm just a guy who really loves this stuff, and I teach myself everything using our new collective university, the internet. One of my favorite resources to find papers on machine learning is the machine learning subreddit. People post papers they find interesting every day. And they've also got this cool weekly what-are-you-reading thread where people post the papers that interest them the most currently. Additionally, there is this web app called archivesanity.com created by Andrej Karpathy, which basically goes through Archive and finds the papers that are most relevant. You can filter them by what interests you, by which ones are most popular, or by the ones that are most cited lately. Google and DeepMind respectively publish their work on their websites for easy access. And there are, of course, journals like Nature that you can find some top papers in easily. The pace of research is accelerating in machine learning because of a few reasons, not including Schmidhuber. In academia and in the public sphere, the democratization of data, computing power, education, and algorithms is all steadily happening over the internet. Because of this, more people are able to make their own insights into this field. In the industry, the big tech companies profit more when their own teams discover new machine learning methods. So there's this race to create faster, more intelligent algorithms. All that is to say that there are a lot of papers you could be reading right now. So how are you supposed to know what to read? What I've found is that every week there are maybe two or three papers that are getting the most attention in machine learning. And the tools I've mentioned help me find them and read them. But most of my reading is a result of me having a goal. That goal could be to learn more about activation functions or perhaps probabilistic models that use attention mechanisms. Once I've got that goal, it makes it much easier to create a reading strategy that points towards that goal. Being a good, math-heavy, machine learning paper reader is not a goal to aspire to. Your stamina is more of a function of human motivation, which is a function of the goals you're trying to accomplish. I've found that I can crush through and understand the most difficult papers much more when I have a real reason to do so. So let's take the landmark paper by a friend of mine, Ian Goodfellow, on generative adversarial networks as an example. There is a lot in this paper. He synthesizes some ideas here that made Yann LeCun say that this concept was the coolest idea in deep learning in the last 10 years. The way I read papers is by performing a three-pass approach. On the first pass, I'll just skim through the paper to get a gist of it. Meaning, I'll first read the title. If the title sounds interesting and relevant, generative adversarial networks, yo, let's go. I'll read the abstract. The abstract acts as a short, standalone summary of the work of the paper that people can use as an overview. If the abstract is compelling, an adversarial process between two neural networks that resembles a game, alright, this is lit. Then, I'll skim through the rest of the paper. By that I mean, I'll carefully read the introduction, then read the section and subsection headings, but ignore everything else. Mainly, ignore the math. I never read the math on the first pass. I'll read the conclusion at the end, and maybe glance over the references, mentally ticking off the ones I've already read, if there are any. I just assume the math is correct on the first pass. My goal for this first pass is to just be able to understand the aims of the author. What are the paper's main contributions here? What problems does it attempt to solve? Is this a paper I'm actually interested in reading more of? Once I've done the first pass, I'll go back to see what other people are saying about this paper, and compare my initial observations to theirs. Basically, the aim of this first pass is to ensure that it's worth my time to continue analyzing this paper. Life's short, and there are too many things to read. If it does pique my interest, then I'll reread it a second time. On the second pass, I'll read it again, this time more critically, and I'll also take notes as I go. I'll actually read all the English text, and I'll try to get a high-level understanding of the math that's happening in the paper. So, it's a minimax game that looks to optimize a Nash equilibrium. Okay, I kind of get that. Eventually, the generator network creates fake samples that are indistinguishable from the real thing. So, the discriminator is powerless. Cool. I'll read the figure descriptions, any plots and graphs that are available, and try to understand the algorithm at a high level. A lot of times, the author will break down an equation by factoring it out. I avoid trying to analyze this on the second pass. I see that it's using a loss function called the Kullback-Leibler Divergence. Never heard of that one, but I do get the concept of minimizing a loss function. When I read the experiments, I'll try to evaluate the results. Are they repeatable? Are the findings well-supported by evidence? Once I've done that, hopefully, there is some associated code with the repository available on GitHub. I'll download the code and start reading it myself. I'll try to compile and run the code locally to replicate the results as well. Usually, comments in the code help further my understanding. I'll also look for any additional resources on the web that help further explain the text. Articles, summaries, tutorials. Usually, a popular paper will have a breakdown that someone else has done online. That will help drive the key points home for me. After this second pass, I'll have a Jupyter notebook full of notes and associated helper images since I teach this stuff on YouTube. Teaching is really the best way to fully understand any topic. When it comes to the third pass, it's all about the math. My focus on the third pass is to really understand every detail of the math. I might just use a pen and paper and break down the equations in the paper myself. I'll use Wikipedia to help me understand any of the more formal math concepts fully, like the KL divergence. And if I'm feeling really ambitious, I'll try to replicate the paper programmatically using the hyperparameter settings and equations that it describes. After all of this, I'll feel confident enough to discuss it with other people. Reading papers is not easy and nobody can read long manipulations of complicated equations fast. The key is to never give up. Turn your frustrations into fuel to get better. You will understand this paper. You will master this subject. You will become awesome at this. It gets easier every time as you build your Merkledag of knowledge. See what I did there? If you don't get a math concept, guess what? Khan Academy will teach you anything you need to know for free. And lastly, do not hesitate to ask for help. There are study groups and communities online that are centered around the latest research in machine learning that you can post your questions to. Don't be afraid to reach out to researchers as well. You're actually doing them a favor by having them explain to you in terms you understand. All scientists need more experience translating complex topics. I've got lots of great links for you in the description, and I hope you found this video useful. If you want to learn more about machine learning, AI, and blockchain technology, hit the subscribe button. And for now, I've got to reread the Capsule Network paper, so thanks for watching.