How AssemblyAI Built a Voice Agent in Under 15 Minutes (Full Transcript)

[00:00:00] Speaker 1: In this video, you'll get a backseat into how I built this voice agent with our new voice agent API powered by Cloud Code. It was a truly magical experience of putting in one prompt and getting a voice agent out on the other end that was ready in production. I'll show you my tips and tricks so that you'll have everything you need to build great voice AI agents on top of Assembly AI. Let's dive right in. This week at Assembly AI is extra exciting because we're releasing our voice agent API.

[00:00:30] Speaker 2: That's a big launch. What was the one moment during the build that made you realize it actually worked?

[00:00:35] Speaker 1: I mean for me it was when I was able to go from zero to a fully working voice agent just by using our Assembly AI MCP server and also just putting in one prompt into Cloud Code.

[00:00:49] Speaker 2: So the whole prototype was built in 20 minutes with a single prompt. Just tweaking it afterward. What was the first bug that showed up and made you realize how fast the loop was?

[00:00:59] Speaker 1: I mean for me it was it was a bug related to Railway which is where I'm hosting the app but I don't want this to show up in my LinkedIn post because it's kind of irrelevant to the voice agent API.

[00:01:11] Speaker 2: You said it's about letting you focus on your expertise. What's the one thing you're most excited to build now that the hard part is gone?

[00:01:19] Speaker 1: Wow I mean I have a ton of ideas for voice agents that I'm sitting on but the one that I'm most excited is actually this voice agent that I'm using right now.

[00:01:27] Speaker 2: If you were to say this to a friend in one sentence what would you tell them about why they should try this now?

[00:01:33] Speaker 1: They should try this now because it's never been easier to build a voice agent on top of Assembly AI's voice AI platform. So the first thing I'd recommend is that you install our Assembly AI MCP server. The full command to install our MCP server is in our docs in the link in the description. Once you have that installed our MCP server will automatically pull our latest docs into your Cloud Code session. There's no more needing to search our documentation for updated information. All of that gets automatically injected into your Cloud Code instance. Alright so kicking off our demo project I tell Cloud Code to build a voice agent that interviews me and asks me questions and to help me write better LinkedIn posts. So that's the big idea of this voice agent that I'm trying to build. As you can see I copy and paste some docs links just to make sure that it gets it just right and I wanted to use the browser integration page as well. So from there I just ask it to build a very simple demo that showcases the API and I'm planning to host this voice agent on Railway and so I mentioned that from the very start. So Cloud Code starts churning away at the code building some front-end and back-end code as we can see here. I've split it up so that we don't wait too long but as you can see in about 10 minutes or so we have a GitHub repository up and running with our voice agent demo. I go ahead and add it into Railway which is where I've chosen to host my app. Now I'm just looking up what command I need to start running the app. So I set the root directory to the front-end as well as the back-end and I insert my Assembly AI API key. So I go on to our dashboard, sign in with Google. So once I have my API key I insert it onto Railway and save and now we're ready to deploy our app. From there I need to add another instance which will then host our front-end and that's the URL that we'll connect to to see our app and to play with it. So this one has a root directory of front-end. As you can tell I've used Railway a couple of times and so I'm kind of familiar with the process but what I've learned is that Cloud Code is actually really good at using the Railway CLI to do a lot of this for you. And the great thing about our voice agent API is that it's framework agnostic. It's a simple API that works with practically any coding language and any platform and so if you want to host it on Vercel, Railway, if you want to self-host it on your own server, anything works because it's just so simple and so easy to deploy. So here now I'm setting the back-end URL with the Railway service but eventually I figure out that actually Cloud Code can just help do it for me and after a little more wrangling and finagling of Railway, in any second now I should be live with my voice agent app. All right, so that's our app and as we hit start interview we bump into yet another error of fail to fetch. So this was because I think on the front-end we set the wrong back-end URL to call for a token. So for context, if you're deploying the voice agent on the browser you can make a request to your back-end for a token that you can use to authenticate your session on the front-end and then you can authenticate your session on the front-end and that way you're able to start the voice agent on the client side without having to expose your API key. All right, great, so here we go. We have our very first voice agent in right just a little under 15 minutes and yeah it was really exciting to get it up and running so quickly but we're going to work on some improvements right away now that we see that it's working. The first thing I'm going to want to do is to style it and kind of make it look like LinkedIn a little more just to kind of nail that design and really get people into the mood of like writing something for their social media. So I go ahead and tell Qualcode it worked great, can you, you know, make some design improvements, styling changes and we kind of want to make it look like LinkedIn. So we'll see fairly quickly that Qualcode is able to design my voice agent to look like LinkedIn and lean into their brand and palette and so after we push to git and refresh we should slowly see that yep it looks like LinkedIn. It's got those colors and a nice off-white background and maybe we'll just start another interview to see how it works. Yep, it's working great and so on to the next thing. So one of our features of our voice agent API is that we support tool calling right out of the box. So the tool that I choose to add to this voice agent is Exa's search which will basically allow the voice agent to kind of search up sources to back up certain claims that we make in our post which will make it look a little more informed and backed by research. So I go ahead and get my Exa API key and insert it onto the railway while Qualcode chips away at implementing Exa on the backend. So I can see here it's made lots of progress. We should be getting a voice agent with Exa in China. All right so to test out tool calling I just started a new session and you know stated a stat just to see if the voice agent is smart enough to go and look it up using Exa and sure enough it finds this report by MIT on Medium which is great. Lastly, we're going to add some different voices to the voice agent. So Assembly AI supports a wide range of voices with more coming soon and I want to let the user pick the voice that they choose for their session. So what we'll do is we'll just add a drop down with all of the choices for voices with some sound defaults and in no time you should see that live as well. If you do end up trying our voice agent API I'd love to know what your favorite voice is. Our team put a lot of work into picking great voices so let me know in the comments below if there's any voice that stood out to you. All right so Claude has gone ahead and made the change and as you can see here's our full list of voices that are available. I just hit start interview with a new voice and there we go. There we have a fully fleshed out voice AI agent in you know very little time.

[00:09:20] Speaker 2: Okay the draft is on screen now. Want to tweak the angle or zoom in on a different moment?

[00:09:24] Speaker 1: No this looks great thank you so much.

[00:09:26] Speaker 2: You're welcome. Enjoy launching the API.

[00:09:30] Speaker 1: So on that note it concludes our voice agent build along today. I hope you enjoy what we were able to put together and let me know if you do end up trying our voice agent API. I'm really excited to know what you think and if you have any questions don't be afraid to reach out to our team in the comments below or on our website. Our team is always happy to help. So until the next one I'll see you bye.