Exploring Whisper: OpenAI's Multilingual Speech Model (Full Transcript)

Speaker 1: What is Whisper? Whisper is a speech-to-text model released by the team at OpenAI. It works on multiple languages, which is very cool. It uses attention setups in a typical Transformer-esque fashion to actually take what they call a log-mell spectrogram, which is a representation of how frequencies in the audio are changing over time. We then have a couple of convolution layers, positional encoding, as is very common in these Transformer setups, and then a number of encoder layers fed into a number of decoder layers, with the additional output, which is a marker to say it's the start of the tokens, and it's saying, look, this is English. There is some language detection in here, which is quite cool. Please transcribe the following, and then the output becomes the transcription. Transcription by ESO. Translation by —