Stanford's Neural Network Transcribes Songs to MIDI and Sheet Music
Discover how Stanford researchers' open-source neural network transcribes songs into MIDI files and sheet music, excelling with pop songs.
File
AI turns songs into sheet music and MIDI
Added on 09/06/2024
Speakers
add Add new speaker

Speaker 1: Apparently, it's not only the artists and programmers who are going to get automated in the nearest future, but also musicians, because these three guys, Stanford researchers, to whom I'm incredibly grateful, they dropped this incredible paper, a neural network, completely free and open source, that can transcribe any song into MIDI files or even sheet music. I played for a little bit, and it does incredible stuff, gets most of the stuff right. It's been trained on mostly pop songs, so it does better with them. For those interested, I'm going to leave all the links in the description. Let's give it a try, just to demonstrate how it works. So first, I'm going to need to run the program here. It's called the SheetSage, and if I want to use YouTube as a source, I'm just going to need to copy the link. Let's try Miss You. So copy link address, paste it here as an argument to this program, and it starts fetching the audio, retrieving audio. You can also pass an MP3 file, of course. It starts detecting beats after that, and you can run it on a potato, on a calculator, anything. There's also a much more demanding version of it, that requires an NVIDIA GPU with 10 plus gigs of VRAM, I think, or 12 plus. Yeah, but it's also supposed to work better if you're getting unsatisfactory performance with that one. So it is done transcribing, and let's see what it got us. So this is a new folder, and by default, it outputs the PDF. It's supposed to be the, what, Miss You, I think. Yeah, harmonies, and MIDI file, too. I'm going to play the MIDI file, let's see how close it is to the original. You know, it doesn't sound close. Which one I transcribed, I don't remember. Which one is that? Miss You, yeah. This one, it didn't get, it did get the harmonies, though, the melody, not so much. Let's try something else. Something I'm actually familiar with. This one, this one's nice. I'll have to do that one. In general, the ones I tried, it did a fantastic job on that. Sometimes it refuses to run, especially if a song has multiple key changes, tempo changes, the, what, 4 over 4, I don't remember what it's called, key signature, I think. Key signature, and all of that modulation. It has some trouble getting that correctly, but don't consider it a final product, consider it a stepping stone, and maybe even a foundation to what's yet to come, because this is incredible. It's maybe 20 to 30% better and more accurate than the previous models. And the fact that you can just run it like that, okay, which song is this? Oh, Stay Together for the Kids. Let's listen to the MIDI file it produced. Whoa. Wow. You can sing along to that, because the harmonies are all right. At least for Monét. Whoa, this piece is not perfect. Oh no, actually. I lost it, okay. Well, let's try something else now, because this one got partially right, but I want to see whether it can get something even better. All the small things, it's incredible. Let's try this one. You can try a bunch of others, various genres, not just pop songs, but I personally did not give it a try yet, curious as to how that's going to turn out. Let me know if you have any trouble installing that. For now, you need Ubuntu or Debian or other Linux distro for that. Maybe you can run it with Docker, not that sure. Let me know how it goes. All this stuff, I'm overly excited, more than I should be. Just imagine what's yet to come. Transcribing, almost done, that fast, it would take a musician much longer than that. The one who doesn't have a perfect pitch at least. All the small things, let's give it a listen. That's the intro, I think. Where is it? Da-da-da.

ai AI Insights
Summary

Generate a brief summary highlighting the main points of the transcript.

Generate
Title

Generate a concise and relevant title for the transcript based on the main themes and content discussed.

Generate
Keywords

Identify and highlight the key words or phrases most relevant to the content of the transcript.

Generate
Enter your query
Sentiments

Analyze the emotional tone of the transcript to determine whether the sentiment is positive, negative, or neutral.

Generate
Quizzes

Create interactive quizzes based on the content of the transcript to test comprehension or engage users.

Generate
{{ secondsToHumanTime(time) }}
Back
Forward
{{ Math.round(speed * 100) / 100 }}x
{{ secondsToHumanTime(duration) }}
close
New speaker
Add speaker
close
Edit speaker
Save changes
close
Share Transcript