Speaker 1: Guys, this is one of those moments in the AI tech space where you see something made possible by AI that is just magic. Today we're going to be taking a look at some research and a demo that you can try for free by MetaAI. Yes, that's right, Facebook. That is seamless, real-time communication between languages. So what's going to happen here is it's going to take my English-speaking voice and convert it into another language in essentially real-time. And this is a huge moment where seemingly magical AI technology starts to remove and break down language barriers. The whole time you're watching today's video, I want you to think about a little pair of headphones, kind of like this, that you would wear in another country, you'd speak your native language, and a speaker somewhere would output your voice translated in that language. And the device could take in someone else's speech in their native language and translate it to yours and play it in your ears. Because that is absolutely what we are seeing unfold in real-time here. So here we are, guys, Seamless Communication AI Research by Meta. You actually can download these models, I want to make that very clear right off the bat. Check it out, right on GitHub. So while you absolutely can download, install, and use these models for the seamless communication, it is not currently available for non-commercial use, so you can't actually build any products with it. However, it is fully open for research purposes, and you can redistribute it for research. So that's sort of where we are on all of this. I'm sure that they probably will open-source a lot of this stuff at some point in the future. If you want to learn more, they do have license information at the bottom here. But yeah, this is a lot better than fully closed-source stuff. It's still fairly open, and it's a good sign, and with Facebook's actually good track record of releasing open-source software, I think we can hopefully expect to see this fully open in the near future. They have a little introduction, so let's take a look at that.

Speaker 2: At Meta, we're working with the AI community to help remove language barriers and encourage open, authentic communication. We're excited to introduce a suite of new models. Seamless M4Tv2, an improved version of our foundational model. Seamless Expressive, which preserves expression across languages. Seamless Streaming, which translates speech and text in just under 2 seconds of latency. And finally, Seamless, our unified model that combines capabilities of all three. Our improved model, Seamless M4Tv2, serves as the foundation for our new Seamless Expressive and Seamless Streaming models. Next in our family of models, we have Seamless Expressive, which preserves the intricacies of speech, such as pauses, speech rate, and emphasis on certain words, as well as vocal style and emotional tone. We believe it's imperative our translations not only accurately convey the words we speak, but also capture the subtleties of human expression. Please keep the volume down. We just put the baby to sleep. Por favor, mantén el volumen bajo. Acabamos de dormir al bebé. We're also thrilled to share Seamless Streaming. With just under 2 seconds of latency, it's the first massively multilingual model able to translate speech and text in near real-time. Consider being in a social situation where language spoken is unfamiliar to you. Then imagine being able to not only follow the conversation with minimal delay, but also seamlessly translate what you want to say in that language. We can now build towards that very idea. We believe this is another step forward in the journey towards a more connected world, and we eagerly await the innovative ways in which the AI community will build upon this work.

Speaker 1: Well, okay then. Not only is it going to capture the expressive parts of my voice, but apparently it's going to capture the overall tone, of course the cloning of my very voice, and all with a near real-time latency, under 2 seconds of delay. That's definitely enough to be usable in the real world. Like I said, they have a free demo. Let's check it out. Seamless Expressive is an AI model that aims to maintain expressive speech style elements in the translation. So we've also got the pitch of your voice as well as the volume, the tone, whether it be excited, sad, or you're whispering. Obviously speech style, so how fast I'm speaking, and if I'm pausing. They've got a few more examples down here.

Speaker 3: So glad you are here. I am so happy to see you.

Speaker 1: Dang, that is good, man. Wow. That does sound like her voice. The cloning probably isn't as perfect as we'd hope. Two seconds of latency, guys. Wow, that is so good. It's even more mind-blowing to hear it back in English. Okay, I am so excited to try that. All of you multilingual folks are going to have to help me out in the comments by telling me which demos work better and which ones don't sound as good. Really need your help on that.

Speaker 4: Wow, man, it's so good.

Speaker 1: Now whispering. That's so good, dude. This is so usable. I am so excited for the language barriers to just be demolished. Everyone can communicate, and it just makes sense. All right, so I will obviously speak in English. We will translate first into Spanish. Right now they only have these languages in this demo. However, I believe in the actual code, there's a lot more languages to choose from. So we'll translate to Spanish. Again, all you Spanish-speaking folks out here are going to have to help me out in the comments. Wow, we're loading up the camera today. All right, for some reason they insist on having the camera in this as well. So I've got my webcam. The two mats can look at each other. All right, this is a bit strange, but let's give it a shot. Say, try excited. It wants me to say this specific text, but I'm not going to do that. Subscribe to the MadvidPro AI YouTube channel. Okay, this was the expressive Spanish translation.

Speaker 5: Suscribe to the MadvidPro AI YouTube channel.

Speaker 1: Dude, that is mind-blowing. Did you hear the way it said MadvidPro? Oh, I love it. Subscribe to the MadvidPro AI YouTube channel.

Speaker 5: Suscribe to the MadvidPro AI YouTube channel. Yeah, okay.

Speaker 1: It doesn't exactly sound like my voice, but it's so darn good. And it's definitely very expressive. You can hear it. Even though I don't speak Spanish, I obviously know what it's supposed to be saying, and I can hear the expression in there. Suscribe to the MadvidPro AI YouTube channel. This is the non-expressive translation, by the way, for the context.

Speaker 6: Suscribe to the MadvidPro AI YouTube channel.

Speaker 1: I mean, that is just straight robot. We don't want that, right? I can't get over that. It is so dang cool. Oh, share your Spanish translation with friends and family. That's cool. That's why they have the video here. Subscribe to the MadvidPro AI YouTube channel.

Speaker 5: Suscribe to the MadvidPro AI YouTube channel. All right.

Speaker 1: I see now. I see. That's why the webcam was there. I suggest you guys share some of these clips around when you try it for yourself. I'd love to see how different people react. You can get back to me on my Discord server and post different people's opinions on this. I want to know how good the translations are. So, for now, we're going to stick to English and Spanish. We will move on to the German and French, though. Let's try fast-talking instead. Can you please buy me some ice cream? I'm very hungry for ice cream. Please purchase me some ice cream right now. Or I'm going to be really sad and I'm going to cry all over the floor. I'm really, really wanting ice cream. Please get it. I'm very hungry for ice cream. Please purchase me some ice cream right now. Or I'm going to be really sad and I'm going to cry all over the floor. I'm really, really wanting ice cream, please get it. I think this is going to be a little tough.

Speaker 7: muy triste. Voy a llorar por todo el suelo. Realmente, realmente quiero helado, por favor.

Speaker 1: How good was that? Let me know, please. Did it screw any of the words up? Is it sound like he's speaking very fast? It certainly sounds like it to me. Do we even want to listen to the non-expressive? Yeah, you can even see the non-expressive is 11 seconds, where the original clip was 9, and the expressive translation is 10. Can you please buy me some ice cream? I'm very hungry for ice cream. Please purchase me some ice cream right now, or I'm going to be really sad, and I'm going to cry all over the floor. I'm really, really wanting ice cream. Please get it. Okay. You did a pretty good job translating. Oh, I love stuff like this. Okay, I really got to try this whispering. I'm going to try using some words that I know in Spanish, kind of give me a better idea. Dog, dog, dog, dog, cat, cat, house, house. Okay, this was pretty quiet. Dog, dog, dog, dog. I don't like whispering, man. Dude. Oh my gosh. The whispering actually works so well. As much as I hate listening to whispering, I have to try that again. Now that I've locked you in my basement, I can force you to endlessly watch MadFitPro AI content until you can't watch it anymore. Okay, I wasn't trying to be creepy, guys. Why is it so much more creepy in Spanish? I can't believe how good the whispering works. I

Speaker 8: didn't think it would blow my mind so much, but it totally does. Okay, I was a little bit closer

Speaker 9: to the microphone for this one. Let me know if the translation is good. A little bit more

Speaker 1: raspiness to that one, but to be fair, my original whisper voice was pretty raspy as well. The non-expressive translation is just useless. Get that out of here. All right, I want to try sad. I'm also going to try some emotions that aren't listed in here. I want to see how well it does with those. Oh my God, you unsubscribed from MadFitPro AI? What's wrong with you? I'm

Speaker 10: crying right now. This is the saddest thing ever. Wow. Oh my gosh. He definitely sounds very sad. He's a little bit more robotic, I think,

Speaker 1: than the other ones. He does sound like he's on the verge of tears. All right, let's try an emotion that's not even listed in their demo. Let's try anger. Are you serious right now? You didn't get me a single lemon for Christmas? You know that's my favorite fruit. What is wrong with you? I'm really trying to push this expressive model to its limits. This one's going to be tough for the model. That works better than the sad, I think. You guys that speak Spanish will have to let me know, but to me, at least that sounds better than the sad one. All right, now, finally, I want to try singing just to see what that's like. Cats and dogs, it's raining cats and dogs. Please buy me some sort of

Speaker 11: a rain hat because the dogs are falling on my face. Gatos y perros, está lloviendo. Gatos y perros, por favor, cómprame algún tipo de sombrero de lluvia porque los perros me caen a la cara.

Speaker 1: Yeah, it's kind of like soft spoken like singing is, but not really singing. Better than I thought though, to be honest. Definitely serviceable. All right, let's move on to French. Hey, I'm speaking French now. Matt VidPro can actually speak French fully. I've always known how to speak it. I know the lips don't look like they're moving with French, but yeah, I kind of like these little videos that it makes. Hey, I'm speaking French now. Matt VidPro can actually speak French fully. I've always known how to speak it. I know the lips don't look like they're moving with French, but yeah. Hey, I'm speaking French now. Matt VidPro can actually speak French fully. I've always known how to speak it. I know the lips don't look like they're moving with French, but yeah. That sounded like my voice. That voice cloning was nuts. You guys have to remember it's cloning my voice, doing the expression, and translating the text accurately into another language, all in about two seconds. Just straight magic. Let's do something a little more emotional. I

Speaker 10: can't believe it. AI has taken over the world. Everyone's gone. My friends, my family, AI has eaten them all. I can't believe it. AI has taken over the world. Everyone's gone. My friends, my family, AI has eaten them all. Correct me if I'm wrong, but the French is

Speaker 1: actually better than the Spanish. That sounds a lot like my voice. It's not perfect, but, man, it's close. It's impressively close. Let's try whispering again. Hey, everybody,

Speaker 8: I have to whisper to you, but I'm coming back at you with another AI video. Sorry, the AI clones are upstairs trying to rumble through my stuff, so I gotta be quiet. Didn't make it to the end. Hey, everybody, I have to whisper to you, but I'm coming back at you with another AI video. Sorry, the AI clones are upstairs trying to rumble through my stuff,

Speaker 12: so I gotta be quiet. Hey, everybody, I have to whisper to you, but I'm coming back at you with another AI video. Sorry, the AI clones are upstairs trying to rumble through my stuff.

Speaker 1: So when it gets a little bit too long, it just cuts my voice off, or rather cuts the video and just continues the voice. I'm still impressed though. Dang it, it's really still very good with French. It might actually be better at French than Spanish. You guys, again, will have to let me know. Let's try German. Hey, everybody, welcome back to another Madvid Pro AI video. I'm actually German. I've been German this whole time, and I've been speaking German this whole time. Hey, everybody, welcome back to another Madvid Pro AI video. I'm actually German. I've been German this whole time, and I've been speaking German this whole time. That sounds like me. That definitely sounds like me. I mean, the quality obviously isn't great in terms of audio, but it absolutely sounds like me speaking German. Oh man, this is so dang exciting. Let's try anger in German. What? Are you kidding me? You don't like my festive decorations for Christmas? Well, you're just a disgrace to the Madvid Pro channel. We believe in festivities. By the way, no hate to you if you actually don't like my Christmas decorations. I just like to get festive for the holidays. What? Are you kidding me? You don't like my festive decorations for Christmas? Well, you're just a disgrace to the Madvid Pro channel. We

Speaker 9: believe in festivities. Okay, I think that German anger one was pushing the model to its limits.

Speaker 1: It was good and absolutely usable, don't get me wrong, but a little bit more on the robotic side, you could tell there's been some processing there. No, please, I didn't mean it. Don't unsubscribe. You can hate my decorations, just don't unsubscribe to the Madvid Pro channel. No, please, I didn't mean it. Don't unsubscribe. You can hate my decorations, just don't unsubscribe to the Madvid Pro channel. No, please. I didn't mean it.

Speaker 10: Don't unsubscribe. You can hate my decorations,

Speaker 1: Just don't unsubscribe to the mad vid bro channel had an actual fail in the translation It just did english to english So this is clearly a little bit of a problem here that they have to fix Seems to work most of the time this doesn't look like it would be occurring that often But yeah, it kind of just put like a slight german accent on my voice All right. Now I want to try spanish to english I don't speak spanish, but i'm going to try to read some spanish words and see how they translate in english

Speaker 4: Pero pero pero gato gato gato casa casa casa. Yeah, that's right. Those are the three spanish words

Speaker 1: I know don't shame me for it. But but but cat cat cat house house house

Speaker 4: Okay. Well, I thought I said dog but apparently I said but but but but cat cat cat house house house

Speaker 1: It definitely sounds like my voice. Let's do some phonetic chat gpt Translation and see how close I can get. Hola a todos soy adicto a la tecnologia genial I screwed that up. Hi a totos. I'm adicto a la tecnologia genial I don't think i'm speaking good spanish there Sorry guys, let's just try to actually read the spanish. Hola a todos soy adicto a la tecnologia Genial I was supposed to say genial. Hi everyone. I'm addicted to geno technology Here we go. Okay. I just have to read the spanish that worked a lot better Dios mio este tipo no puede hablar espanol para salvar su vida My god, this guy can't speak spanish to save his life Yes, dude Yes, oh my gosh, I don't know why that gets me so excited that sounds like me that sounds like me this gives us a better idea of of uh The translation because we're translating to english my god

Speaker 3: This guy can't speak spanish to save his life. What a good translation, too

Speaker 1: All right. I want to try german apologies in advance Da bist du ja enlich ich bin so froh dich zu sehen By the way, what I was reading there was what was on the meta demo there you are

Speaker 3: Enlich, i'm so glad to see you. All right, we got this man

Speaker 4: Okay, I couldn't even get to the end man

Speaker 1: This guy's jumping through hoops. He doesn't have all I was actually getting pretty good Apparently my german speech is good enough for this thing to pick it up I think that my sentence was a little too long and that definitely sounded like me though I like trying english sayings in other languages deser

Speaker 3: Carolist der heat this guy's hunt. Whoa

Speaker 1: Okay, so the saying something's the beanie the bee's knees. It just says this guy's hot. Yeah, screw it Let's try some french, too. Mon dieu Mon dieu It's technology. That's changing our world I'm, really impressed by its ability to pick up my horrible different languages Like I don't speak these languages at all. It's probably the thickest american accent you've ever heard lasheen

Speaker 3: They met vid pro me and dormir the met vid pro channel my fate my sleeper

Speaker 1: I don't know i'm not getting close to this one So that's the demo that they have of course They do have these other models too But those aren't demoed on the site if you want to use the other models You'll have to download them on github. I mean talk about a mind blow guys. This one truly blew me away It's one of those rare times that I have. Uh, well lately not so rare But it's one of those ai products that just truly blasts me away. That was so fun I really recommend you try this yourself again Give me feedback if you speak any of those languages share your best results in the discord server Thanks so much for watching and i'll see you in the next one. Goodbye