Build a LiveKit Voice Agent With AssemblyAI Services (Full Transcript)

[00:00:00] Speaker 1: Hi, today we're going to build a voice agent with Assembly AI and LifeKit.

[00:00:04] Speaker 2: Heads up, this isn't Harnoor, it's his voice clone made right here in this video.

[00:00:09] Speaker 1: First of all, why today? Today you should appreciate how easy it has become. Look at this chart, which I don't even want to look at, how difficult it was back in the day to build a voice agent. You have to figure out phone provider, you have to figure out audio plumbing, you have to figure out like delays in audio, which you all have to debug by yourself. But now it's as easy as thinking about three things. Number one, ear of a voice agent, which is speech to text. Number two, brain of a voice agent, which is LLM, chat GPT clone. Number three, voice of a voice agent, that is text to speech. And these three steps are seamless today with Assembly AI's speech to text model, LLM gateway and text to speech model. Let's get started. And all these three can be plugged into any open source voice agent provider. We're going to choose LifeKit today for an example, and plug in all these three with one prompt and build it right away. So let's go to Claude. No one opens web, so I'm going to start with Claude directly. I'm going to say, build me a voice agent with Assembly AI's TTS and STT on LifeKit. That's all you need in case your agent struggles. Let's say you're using some random agent, like random coding agent. You could actually go to our example repo that I'm going to provide to you today. It's called Assembly AI's LifeKit reference voice agent on our Assembly AI solutions page. And this repo actually breaks down building a voice agent while installing our plugin on LifeKit. So the steps are pretty straightforward if you want to look at the code as well. You basically install, first of all, clone this repo. You install LifeKit CLI, and then you tell LifeKit, hey, use our Assembly AI's API key. And LifeKit figures out all three, our brain, voice, and ears. Basically, it figures out and builds your voice agent as seamless as that. But our Claude is smart enough. We're going to let it build right away. And do not forget, you need Assembly AI's API key. You can go to Assembly AI's dashboard, and then go to manage keys, and then get an API key and provide it to your agent. Now, Claude will create an important file called agent.py, which you can find right in agent folder right here. Right now, my Claude is telling me, give me Assembly AI's API key. So I'm going to tell it that, hey, I have my API key in zshrc. It's a file path in Mac that I've already provided. So use from there directly. And look at what Claude just did. It figured out our speech-to-text model, LLM gateway, as well as text-to-speech. It's using a default OpenAI model right now as default. You can change it later. And now you can actually open Visual Studio Code. No one looks at code. I'm going to still show you for a second. Basically, your code will be in agent.py. Here is some system prompt. I'm telling it that you are Sunset Wi-Fi. I asked in Claude to use it for Wi-Fi. Basic instruction is that you need to be strong and not negotiate my Wi-Fi plan with me. And number two thing that is very important for voice agent is that keyword. So, for example, in my keywords, I provided that you are Sunset Wi-Fi, just a random Wi-Fi provider name, some other Wi-Fi provider names, and my name, Harnoor. So when I say my name, it actually figures out well. So you can define brain. And in medical word, you can define some words as well. I have headache or I'm taking minoxidil. You can give all the spellings you need right here in the key term so that your voice agent figures out well. And here right now, this is the model it chose, GPD 5.6, LLM gateway. This is from Assembly Eye. It figured out well. This is the greeting. You can customize it. This is the instruction that when customer says goodbye, you need to end. And let's just run it. So I'm going to run this voice agent.

[00:04:10] Speaker 2: Thanks for calling Sunset Wi-Fi. This is Jordan on the loyalty team. How can I help you today?

[00:04:16] Speaker 1: I'm Harnoor Singh, and today I'm here to negotiate my Wi-Fi plan. Look at the spellings.

[00:04:22] Speaker 2: Absolutely, Harnoor. What would you like to negotiate about your Sunset Fiber 500 plan?

[00:04:28] Speaker 1: I'm paying, I think I'm overpaying like, you know, $111 per month. I don't think I should be paying that much. I think in San Francisco, there are companies which can give me way better rate like $80 per month.

[00:04:40] Speaker 2: I really hear you, Harnoor, and I'm not able to change the price. But here's what that $111 gets you. True symmetrical 500 megabit fiber, no caps or throttling, and whole home mesh included.

[00:04:54] Speaker 1: Maybe do you think it's a weather problem, maybe in Chicago that I have bad Wi-Fi today or in San Francisco?

[00:05:01] Speaker 2: San Francisco is currently foggy with light wind and Sunset Fiber.

[00:05:06] Speaker 1: So right now I asked it for weather for a reason, because number two thing I want to teach you is tool calling. And how does that work? If you look at the code, of course, you don't need to look at the code. I have actually created a function call called get weather as well. So what's really happening is in the process of voice agent, that as you speak, it's converted to text and text sent to LM. Now in that process, particularly what happens is this text in LLM, when it detects that you're talking about weather. So let's say it detects that, oh, there's a bad weather in Chicago, bad weather in, let's say, San Francisco. I talk about I have a bad Wi-Fi, maybe because of weather. LLM, because of its smartness, the moment it figures out that I should check for weather, it can call a function called get weather and give you the details. And that is the smartness you need. And that's why you should choose a smarter model based on your particular need. If you think you need to do some like some like complicated inference, figuring out whether figuring out other complications, you can use a smarter model model from our LLM gateway and have a tool calling like right here and use an API service to figure out which weather provider I should use. Right now, I only hardcoded weather for San Francisco, New York, Chicago. You can use an API service for weather provider and that'll give you live weather data. And that's where tool calling comes in. And now you have a voice agent with speech to text, text to speech with the smartness of figuring out whether when to blame weather for bad Wi-Fi. And now last step would be to connect to your phone. If you want to make your voice agent do calls for you, then LifeKit has that service. It gives you one free phone number with around 100 minutes that you can use to actually call on your behalf as well. So that's how you can build your own voice agent with Assembly AI and LifeKit. But that's not it. If you notice the voice of the sunset Wi-Fi provider, that was a clone voice in southern accent. And for that, I'm going to show you our text to speech model that actually build that voice. You can go to our dashboard, go to text to speech, go to voices. Over there, you'll find 4,000 plus voices or you can clone a voice. You can see 4,000 voices in different accents, male, female, accent, American, Australian, Indian, tons of accents in six languages. And you can click on create a clone voice. You can clone your voice in just as much as 20 seconds of audio. Next, I'm going to call it Harnoor test October 26. Confirm. I have consent. We always ask for consent. If you want to clone someone else's voice, hit that submit button. And then right now, the voice will show up right here as new voice that I cloned right now. I can hit the play button. Thanks for calling Alliance Appliance Repair. I have an opening this Thursday at 2.15 p.m. Would that work for you?

[00:08:13] Speaker 2: Or should I look at Friday morning instead?

[00:08:15] Speaker 1: Wow, it's so close. Let me try another voice. Hello, I am Harnoor. Hello, I am Harnoor. Wow, it got it right live, right in front of your eyes. So we can't wait to see how you clone your voice with our text to speech model on Assembly Eyes Playground. Go check out and thank you.