AssemblyAI Bets on Scalable Conversational Voice AI (Full Transcript)

[00:00:00] Speaker 1: How often do you call a business and run into a voice AI? I would say it's like maybe two out of ten phone calls I make, I can tell I'm talking to an AI. When you sense you're talking to a machine on the phone, you just want to get out of that experience immediately. The voice right now is a tell for, are you talking to something that's intelligent or not? If it's intelligent, like cool, I'll talk to it. If it's not, then I want to talk to a human.

[00:00:28] Speaker 2: Okay, we are back. I'm here with Dylan Fox, who is the founder and CEO of Assembly AI, a voice AI company whose speech-to-text API is still under apps that you may know, like Granola, HeyGen, ClickUp, and which says that it handles more than 120 million conversations a week. Yeah, and we're here to talk about voice agents and text-to-speech, speech-to-text. So super excited. Welcome, Dylan.

[00:00:52] Speaker 1: Yeah, thanks for having me on. Excited to be on here.

[00:00:55] Speaker 2: Yeah, so tell us about your new text-to-speech model, TTS1.

[00:01:03] Speaker 1: Yeah, yeah, for sure. So we started Assembly really focusing on speech-to-text and transcription. There's voice being implemented in really every piece of software and hardware right now. A lot of that is ambient voice agents that need to be able to listen really well, tell how many people are in a room, who's saying what. So we've been really focused on speech-to-text, speaker identification in real time and on real time. And then today, we just shipped our first text-to-speech model that's purpose-built for conversational voice agents. So it's trained on a ton of proprietary conversational data to make the voices sound very conversational. And then it's deployed on our inference infrastructure in the cloud that's handling over 100 million conversations a week already to run at huge scale, super fast, super low cost for companies and developers building voice agents. So we're super excited to have it out live for people to play with.

[00:02:10] Speaker 2: Yeah, awesome. So to share some details, it's at $15 per 1 million characters. Lowest price frontier text-to-speech model for voice agents. I'm curious, how are you guys thinking about voice agent adoption? I assume you guys are very bullish on this concept of that being the natural interface. But yeah, what was the theory behind the model?

[00:02:37] Speaker 1: Yeah, so it's actually $15 per million characters. So it's super low cost for the frontier quality that it's at. Actually, no other TTS model beats us on the combination of quality, cost, and latency. So it's really important for us to be optimal across those dimensions. What I think about is... I don't know what the experience is like for you. How often do you call a business and run into a voice AI? It's been increasing a lot, which has been cool to see. But it's still pretty infrequent. It's like maybe 2 out of 10 phone calls I make, I can tell I'm talking to an AI. It's still like a human there. So we see conversational voice agents both for consumer, but also B2B applications. The deployment of those are going to increase exponentially over the next year or two years. And in order to make that happen, the technology has to get better. It has to get more scalable. And so for us, we've been really focusing on the listening part of a voice agent with our speech-to-text models that are getting really good at focusing on the right speaker who's talking so your AI knows who to pay attention to. Our listening model is working really well in environments with heavy background noise. But on the text-to-speech side, the models, they need to sound more natural, more conversational, less like a produced audiobook or podcast. And then they need to be able to run at a larger scale at lower cost, given how widely they're going to be deployed for all these different use cases. And so for us, we're really bullish on voice agents in B2B and in consumer scenarios. I have small kids. I'm already starting to see toys come out that they can talk to. A lot of our customers are building those. And those are really cool experiences. It's like hardware devices on the consumer side. So we're really bullish on both of those things. A lot of infrastructure has to be built to make this kind of reality realized. And that's what our company is squarely focused on, is building out all this voice infrastructure on the models side, on the imprint side, to help power all these applications that we see coming to market over the next couple of years.

[00:05:18] Speaker 2: Yeah, totally. So it feels like there's a lot of things to solve for when designing these voice agents, because you want it to sound, I guess, conversational and natural. But I guess you also want the voice agents to be able to get to the root of whatever the user, the questions that they actually have, or if it's customer service, stuff like that. So do you guys think about that as well? And how do you adapt your model for that?

[00:05:46] Speaker 1: Yeah, it's such a great point. For voice applications, what you want the AI to do is really so different depending on the industry and the use case. So in consumer, people are having... We have customers that are building consumer voice AI agents on our platform. And users will have 30-45-minute conversations with these AI, sometimes even longer. And that's very different than you're calling into a plumber and you want to book an appointment or check on the status of something. And it's more of a quick transactional B2B scenario, where, especially on the business side, they want that experience to be more on the rails. They're very risk-averse in that type of putting AI in front of their customers. And so when we're building these models, what we do is... Because we're fully focused on third-party applications that are building on our platform. We really have a good sense of... And I think this is one of the things that we do best in our company, is we really have a deep understanding of all these different use cases and what the experience needs to be optimized for in those different use cases, both on latency and capability and even the economics of these different use cases. So we really design versions of models with these use cases in mind, because we get so much feedback from the over a million developers on our platform and almost 10,000 customers on what they want these things to do. So we almost co-design these different models that we're building with our customers in mind.

[00:07:31] Speaker 2: Right. So do you find that users actually care about the natural... How much does the voice sound like a real human? What gets them? Or do they feel tricked, maybe, if it sounds too natural, like it feels like that's an intricate balance? What generally makes somebody not hang up on a call versus hang up on the call?

[00:07:55] Speaker 1: It's so funny. People have such different preferences for this stuff. Even at our company, we were deciding like, Oh, what voices do we want to showcase on the website? And everyone has a different opinion. And it's why we have over 4,000 voices that are available on our API that you can pick from. And we also support instant voice cloning as well. So if you don't like any of those with as little as 10 seconds of someone's voice, you can instantly clone their voice and then have the AI speak in that voice. And it captures that voice really well. Obviously, if you have the consent to do that. And so what someone in our company said, and I agree with this, it's like the person building the AI, the voice for their AI, in a lot of ways, that is the user experience. And so every company has a different idea of what they want that user experience to be like. And so if you're building an app, and I'm building an app, I might want my voice to sound funny and sarcastic because that fits my brand. And you might have a different point of view on what's on brand for you. But also, that's on the customer side. On the brand side or brand company side, on the user side, if you're calling into a business, you want it to sound local. And so it has to... If you're in a town in the UK, you want it to have the same regional accent as where you're living. Now, a lot of that is back to your question. Because humans are conditioned right now to... When you sense you're talking to a machine on the phone, you just want to get out of that experience immediately because we don't think it's worth our time. Because we've all called into IVR trees and they're terrible. And we're just like, get me out of this matrix. I want to just...

[00:09:53] Speaker 2: Yeah, don't dare to talk to someone.

[00:09:55] Speaker 1: But it's funny. I build this stuff now, right? And so when I now call businesses and I can tell I'm talking to a voice AI, not an IVR system, but a voice AI, I care less about actually the voice. I'm more just like, oh, I know I'm talking to a voice AI. So therefore, I know this thing is worth my time because it will be able to complete the task for me. So I do think that there's going to be some cultural change where as AI is rolled out, more mainstream... Voice AI is rolled out more widely in society. People are going to trust it more and trust that it's worth their time. So they might... It's more like the voice right now is a tell for, are you talking to something that's intelligent or not? And if it's intelligent, like, cool, I'll talk to it. If it's not, then I want to talk to a human. So there's like, I'm not wasting my time on something that's not intelligent.

[00:11:00] Speaker 2: Right, right, right. Yeah, that's an interesting viewpoint. So I'm also curious, you guys had focused on speech to text sort of before, and now you're focused on text to speech. So what are some of the gains that come from owning sort of the entire voice loop?

[00:11:18] Speaker 1: Yeah, totally. So we handle over 2 billion API calls to our platform every month now, well over 100 million conversations a week for over a million developers, thousands and thousands of customers. So we've had to build a ton of proprietary inference software and a runtime to serve our speech to text models at this scale with high reliability, at low latency, at low cost for all of those customers and users on our platform that have been using our transcription and speech to text APIs. That's been hugely beneficial as we've now launched our text to speech because it's served on the same inference stack. And so what that means is it's a frontier quality model that can be super low cost because of all the inference work we've done. If you're using our API, you don't have to worry about rate limits or compute. There's uncapped rate limits on the API. So you can open up as many sessions as you want, use the API at whatever scale you want to. You don't have to get in touch with us to negotiate rate limits. All that is because of the inference work that we've spent years on for speech to text. And now we're able to benefit from that and put that behind our text to speech model. And also, other voice models that we're shipping in the future. We have a live speech to speech duplex model that we're shipping early December. It's going to run on the same voice inference stack that we've been spending years building out. And for us, it's to make it just effortless for companies to build and innovate with voice. We don't want them to have to think about GPUs or compute or rate limits. We just want them to have an idea and be able to go build it and put it into the world. We want to get out of their way.

[00:13:24] Speaker 2: Right. Yeah, yeah, yeah. Yeah. Well, I certainly have begun using voice agents an incredible amount in just the last month or two. So I think it's very much where things are going. Well, thanks so much, Dylan. It was great chatting with you about Assembly AI and voice agents.

[00:13:41] Speaker 1: Yeah, thanks for having me on. It was fun to be here.

[00:13:44] Speaker 2: Awesome. And we'll be right back with MTS.