Speaker 1: Hi everyone, today we're looking at the paper Nearest Neighbor Machine Translation by authors from Stanford University and Facebook AI Research. The paper introduces k-nearest neighbor machine translation, i.e. neural machine translation augmented by a k-nearest neighbor search model, which is capable of querying the training data set of the machine translation system dynamically during the generation process, accessing some knowledge from this training data and using the closest examples from this training data set to infer something about what is the most likely next word, and to use that information to improve the base neural machine translation model and to improve the translation performance. This approach to integrating KNN to neural machine translation leads to improvements in a wide range of scenarios, for example, it's effective for high-resource translation on the German-to-English WMT task, authors get 1.5 blue improvement, which is state-of-the-art, and also, particularly KNN-MT seems to be very effective when doing domain adaptation of a general NMT model to a custom domain, such as medicine, seems to be very effective for that, because you could use a domain-specific data store, and then you can get improvements of up to, well, 9.2 blue on average over zero-shot transfer, and also KNN-MT is effective for multilingual translation, where you could have a language-specific data store to use to augment this model, train on multiple languages, which seems to be effective for a wide range of languages. Finally, an interesting nice benefit of KNN-MT is that it is easily interpretable, and it can be, without much effort, plugged in into a wide range of scenarios. So going into a little bit of a more detail, how does this work? As you may be aware, the standard neural machine translation, machine translation scenario is to learn a model which tries to predict the next word, yi, given the, this is the next word in the translation, given the source sentence in the source language, x, and the translation context up to now. This is an autoregressive tech generation task, and the new KNN search model is integrated via linear interpolation, combined with the standard machine translation model, and it's computing the same probability distribution. The way the KNN model works is that it is, given that you have a training dataset of examples in French and English in this case, for each possible translation context for modeling the probability of the translation up to now, for example, I have, I had, I enjoy, when you're trying to predict the next word, which is been here, up to when you, for each context up to this point, you are going to be computing a representation for all of those contexts. Basically, this is computed by passing the source language and the translation context up to the current point, and by computing a vector representation for this context, and then you're going to be storing that representation into a KNN database. And then what you can be doing, what you're going to be doing with that those representations during inference is given that you have your source sentence, and then you have generated some context, you're going to compute a representation for the current context, and then you're going to search for the closest contexts in your database, and look for what are the next words for the, for the context, the next words could be been, summer, my, and so on. And then you're going to use the distance from the representation computed during inference to the representations from the data store to get some score, basically measuring how close are those contexts and how close are how likely are the next words, given the current context, for example, it could be that bean has a score of four, summer has a score of 100, my has a score of one, and then you're going to convert these distances to some sort of a probability distribution over your vocabulary. And then my is going to get a probability of 0.4, and being 0.6, because it occurs more frequently in the context. The size of the K of the number of contexts that you will look at is going to be a hyperparameter. The authors experiment with a wide range of hyperparameters for that from one to 128. It seems that there's a trade off, I think 32 or so was found to be a pretty good one, but this is a hyperparameter to be tuned. And then you can just combine, you get this probability distribution, you combine with your original neuromachine translation model, and you get an updated distribution, which is taking into account these k-NNN scores and seems to lead to improvements in performance, as I said in the beginning, over a wide range of scenarios and benchmarks, multilingual translation, domain adaptation for domains like medical, law, IT, Korean, subtitles. On average, you're getting always quite a nice improvement for this domain specific translation. And by the way, you're going to have your domain specific database as well. And so it seems to be consistently leading to improvements, no matter how big the size of your data store is pretty much. And also for all number of neighbors, yeah, it seems that 16 seems to be probably the best, but it also depends on some other hyperparameters that you have to set. And in overall, this is a pretty nice paper with nice results and easy application. You could pretty much take your existing translation model, and as long as you can compute those representations, you could leverage this strategy to get some improvements. And one interesting direction that I'm particularly interested in is to see if this can also work for monolingual tasks like summarization or question answering, question generation. You could, in theory, apply the same strategy for those as well. So it's interesting whether this can be effective. Thanks for watching and I'll talk to you in the next video.