LiveKit Adds Automatic Context for Better Voice Transcripts (Full Transcript)

[00:00:00] Speaker 1: I'm excited to share with you Agent Context Carryover, which is live today on LiveKit for Universal 3.5 Pro Realtime. If you've ever built a voice agent, you've probably wondered if you could boost the accuracy of transcription by providing the model with context on the question that your voice agent is asking. With Universal 3.5 Pro Realtime, we released the Agent Context Parameter, which you can use to guide the model in terms of the context of the conversation. But now on LiveKit, we feed in that context automatically, so there's no extra plumbing that you have to do on your end to get that context fed into the model. Instead, you enable one parameter, Agent Context Carryover, and we do all of that extra plumbing for you, so you get a boost in accuracy without having to set up key terms or sending over the agent's last message. We handle all of that for you with our LiveKit plugin. Today, I'll build a simple demonstration app with the LiveKit Quick Start, and we'll implement Agent Context Carryover to see if it helps with transcription. So, for starters, I'm here on the LiveKit docs, where they offer great advice on using coding agents with LiveKit through either the Cloud MCP or the LiveKit CLI. I have both of these installed, but the link to this doc is in the description if you need help getting set up with this, if you're using cloud code like I am. We also have this example starter kit from the LiveKit team, which you can use to quickly spin up a Python example project for your LiveKit agent. So, I'll go ahead and copy this command into my terminal, and let's build. All right, so now we have our example project set up. I'm going to go ahead and open Cloud Code, and I'll run this command to install the Assembly AI plugin for LiveKit. All right, so now that we have the plugin, I'm just going to go ahead and tell Cloud Code to use Assembly AI for this agent, and use Agent Context Carryover. All right, and right here, we can see we're using Assembly AI for speech-to-text. The Universal 3.5 Pro model and Agent Context Carryover is set to true. While Cloud spins away at it, I'm going to go ahead and add my API key into my .env.local file. Now that that's done, let's check back on Cloud. All right, so we've already added our API key.

[00:02:59] Speaker 2: All that's left next is to run our agents to see if it works. Hello, can you see this?

[00:03:15] Speaker 3: Yes, I can. How can I help you today?

[00:03:18] Speaker 1: All right, so the agent works. Let's go ahead and set it up now to be like a restaurant delivery takeout type of system. Cloud should be able to do that just fine. All right, so Cloud ran for eight minutes, but now it's finally set up. We have a phone ordering flow for a fictional restaurant. I'm already liking the name. Let's see if Universal 3.5 Pro can transcribe it.

[00:04:11] Speaker 2: So I'm going to go ahead and rerun this dev command, and give a refresh here, and start a session.

[00:04:29] Speaker 3: Welcome to Lopericord Fusion. Would you like to place an order, or may I read through our menu for you?

[00:04:36] Speaker 1: Yeah, what do you have for appetizers?

[00:04:42] Speaker 3: We have escargots for $14.50, for $13, for $11.75, and for $12.25. Do any of those sound good to you?

[00:04:57] Speaker 1: Yeah, let's do escargots à la bourguignonne.

[00:05:05] Speaker 3: Great choice. Would you like to add anything else to your order?

[00:05:09] Speaker 1: Yeah, do you have any drinks?

[00:05:13] Speaker 3: We have horchata for $5, and yuzu kombucha for $6. Which one would you like?

[00:05:18] Speaker 1: Let's do yuzu kombucha.

[00:05:22] Speaker 3: One yuzu kombucha, coming up. Will there be anything else for you today?

[00:05:27] Speaker 1: Yes, do you have any dessert?

[00:05:31] Speaker 3: We have mille-feuille for $9.50, kouign-amann for $8.75, and pistachio semifreddo for $9. Do any of those interest you?

[00:05:40] Speaker 1: Yeah, let's do kouign-amann.

[00:05:44] Speaker 3: Excellent choice. Is there anything else you would like to add, or does that complete your order?

[00:05:48] Speaker 2: That's all.

[00:05:52] Speaker 3: You have one escargots à la bourguignonne, one yuzu kombucha, and one kouign-amann for a subtotal of $29.25. I will transfer you to the checkout post now to handle your delivery and payment. I can help you finalize your order. Please provide your full delivery address.

[00:06:11] Speaker 1: 1214 Madison Avenue, New York, 10012.

[00:06:22] Speaker 3: Thank you. How would you like to pay for your order? How would you like to pay for your order?

[00:06:27] Speaker 1: Can I use my credit card, please?

[00:06:31] Speaker 3: Your order includes escargots à la bourguignonne, yuzu kombucha, and kouign-amann for a subtotal of $29.25. We will deliver to 1214 Madison Avenue, New York 10012, and you will be paying via credit card. Is everything correct?

[00:06:47] Speaker 1: Yes.

[00:06:50] Speaker 3: Thank you for your order. We will have it delivered to you shortly.

[00:06:55] Speaker 1: As you can see, even though I botched every single menu item on there, Universal 3.5 Pro Realtime was able to transcribe everything perfectly, and that's all thanks to Agent Context Carryover, which you can enable now on your LiveKit agents available today. With Agent Context Carryover, you get gains in accuracy, especially on proper nouns and key terms, without having to do any plumbing for key terms prompting or adding context from the agent into your LiveKit configuration. It's better accuracy without the extra steps, and so if you're using LiveKit, why not go ahead and give Agent Context Carryover a try today? We are currently the only speech-to-text provider on the market that does this drag-and-drop context carryover. We're so excited to hear feedback from you on how Agent Context Carryover performs, and all the links that I discussed in this video are in the description for you to build your own projects with Assembly AI, LiveKit, and Agent Context Carryover. That was all I had for you today. Thanks for watching, and I'll see you again. Bye.