How to Build Fast, Clean Dictation Workflows (Full Transcript)

Learn how Sync transcription, warming, key terms, and LLM cleanup enable sub-second dictation experiences.
Download Transcript (DOCX)
Speakers
add Add new speaker

[00:00:00] Speaker 1: Thank you, everyone, for joining us today. We are here with Andrew, who runs our technical account management team at Assembly. And we're super excited about all things dictation, building dictation features into your apps, building dictation products. And Andrew has lots of exciting updates for you on how we have seen customers building this and on some new developments to our API and how you can use it to build dictation into your products. So thanks again for joining us today. This is recorded, and we will send it out afterwards along with everything that Andrew has built so that you have a place to jump off of. And yeah, I'll turn it over to Andrew. Please feel free to use the Q&A, and we'll leave some good time for questions and answers at the end. And otherwise, I'll turn it over to Andrew, who can kick us off.

[00:00:58] Speaker 2: Yeah, perfect. Thank you. I appreciate that. Again, my name's Andrew. So I work as a forward deployed engineer here at Assembly. Today, we're just going to touch base on dictation, how to build that with Assembly AI, and some things that come in the future as well. So first off, what exactly is the agenda? So we're going to go over, like, what is the Sync API? This is what we're building dictation around. What we're going to go through and build some dictation. Code's pretty much done. We're not going to do anything too live here for the sake of time, but we will step through it. And then we'll review any of the changes and start going into questions. So why Sync? What is a Sync API? Why does it exist for us? Essentially, historically, we have offered two forms of transcription, our async and our streaming endpoints. This would work. Async is kind of your pre-recorded. I have an audio file. I just need to transcribe it after the fact. Streaming is your real time, whether it's captions, voice agents, sound days, so on and so forth. So really, with 3.5, the model became fast enough to start offering something in between. This became super important for if you want to set up an IVR system or you're just trying to get a voicemail or something like really small transcribed very, very quickly with not a lot of overhead. And that's where Sync comes into play, because essentially, if you look at how these requests happen across the API, if you're going to do async, traditional approach would be you're going to send us a post request with your audio file. We return back like an upload URL. You could skip this if you have a pre-signed URL or something like that. You're going to send that to the API to then start transcribing. You're going to pull for this transcription over and over again or wait for webhook. You could do that as well until you eventually get back to transcript. This does happen fairly quickly, but you do have to stand that up when your side is just not going to be as quick as a single post request sync on smaller files, which in these cases, dictation obviously would rule it out. Streaming we'll get into next has been considered from time to time when we talk to people about what they're trying to do with dictation. But the issue you have there is the overhead of maintaining the WebSocket, because there you're going to have to do a typical lifecycle that you really don't have to when you have something like sync, especially with the speed that you get the files back in. And obviously with sync, all you have to do is send the post request and you get back the transcript itself. So why would we do sync over streaming? Again, we can look at the WebSocket. Streaming is really great if you have like a one-time session where you can connect only once. You're controlling how that happens. So if you're some type of note taker or agent, you know when this event is happening. You can get everything set up and it's going to be more or less negligible. But if you're doing this on a typical dictation style setup, most dictation bursts are probably around the realm of like 10 to 12 seconds, I would venture to say. So at which point you're setting up a WebSocket over and over again, that does take some time, even though it's small, maybe a couple hundred milliseconds. But when you start adding that into a dictation process, that becomes overhead you don't want to have and latency that you don't want to introduce. So this is what really forces us to land on the sync endpoint. So what does this actually look like? We're going to play around in a fake version of Slack today. And I have a simple JavaScript setup around here that we can step through, kind of chapter by chapter. So the first thing we're going to do is look at what happens if you just send a sync request. So we're using our SDK, assembly AI SDK that we bring in up here. We're using the 3.5 pro model for sync. And all we're going to do is essentially make the call client saying transcribe, pass in the audio. And we do have this option here, key terms prompt. We'll get to that in a moment. But for now, we can just kind of skip ahead and just say we're passing audio and we're transcribing it. So what that would look like here once we actually do that, actually, we will fix this because I did forget to clear this, but we would assume we have no vocabulary saved. We would go ahead and transcribe. Hello, my name is Andrew Galleon, man. This is very clearly wrong. You'll see that our server request time is what we report back from assembly to server to transcribe. Anything else over that is overhead that we have from not like warming connection and other things that happen in transit. So, you know, in this case, we're seeing we're well over like 200 milliseconds, which slows us down. We'll get to that in a moment. There's ways to speed that up. But how do you make good dictation? One of the things you have to do before we even get the cleanup steps is get basic information, right? So in this case, we want to say, hey, my name is not spelled that way. We'll save that. We'll try again. Hello, my name is Andrew Galleon, man. And now we have the correct dictation. This is being achieved through key terms prompting. So with key terms prompt, we're just giving an explicit list of items that we want them all recognized. And that is what is being passed in here in vocabulary. So you can see how that's set up on just kind of this one step. We have a transcript. So hello, world, successful. Great. But now we want to go ahead and speed this up because this is not. Where we want to be, this is a cash response, but that first one, as we saw, was pretty slow. So we want to speed that up and we can do that by using. We have a warm endpoint essentially to keep alive. We can ignore a lot of the imports that are going on here just so everyone's following, though. What is happening is we're just pulling in everything from chapter one, which is right here. All this remains. That's what the citation is now. And all we're going to add to it is the ability to call this warm endpoint. So let's pay attention to this call. It's client that sync that warm and we're going to say which model we're using, which again was specified earlier. So we're just going to continue to pass this on as we go chapter by chapter. And then we're going to call this. This is what's really important while we are recording. So we don't have to pay attention to all the US stuff. But when we hit that record button, we're going to call this this warm function and we're going to start with a result. But we're going to go ahead and call that so we can go ahead and take care of a lot of that, like all the a lot of the networking stuff, the tolls that we have to pay when we make a request. So let us actually update that first. So restart and I'll go ahead. Hello, my name is Andrew Galleon, ma'am. And now you'll see the time here becomes much closer in time and we get a faster response, especially as we continue to transcribe. My name's Andrew and I work at Assembly. And you see we're no longer in that realm of like always like 200 milliseconds plus apart. So this is one way of getting your latency a bit lower so we can have faster transcription. So after warming, one of the things we're going to run into now is cleanup. So what do we do with a case of these are just quick, easy blurts. But what if I'm doing something like we've all seen in dictation where we want to we are we don't write how we speak. And so I may say something like, you know, I think we should meet in five minutes on Tuesday. No, actually, I would prefer on Tuesday. And I'm going to see the full transcription come back. Everything looks fine as far as latency. However, now if I was imagine this was like if I'm responding with something way more complicated, this becomes a huge thing to go and edit when really what I want to say is like, let's meet five minutes on on Tuesday or I repeat it myself. But you get the point. We can call us Wednesday if we wanted to. We got to go clean all that up. So the way we would want to do that is a cleanup step. The way you would do this today with Assembly AI is that you are going to go ahead and call our LLM gateway. So LLM gateway is we actually offer a whole suite of models that LLMs behind like the OpenAI standard. So it should be very easy to work with. And we are going to use we're actually self hosting the Quinn for be fast, which is extremely fast, extremely efficient. And we're going to pass in essentially this style prompt. So what we are asking for here is a cleanup prompt where we're stating essentially that it's not conversational. It's not an agent. It's just keeping the intent of the message. And we're going to do this by also giving it a few shot example. So we pass in the system prompt, which is what's going to go ahead and start get everything kicked off for us. All this is kind of some prompting techniques that you have probably seen over the years. Not much has changed here. It works really well in this case. So we'll do some safeguards of what makes like what we don't want to change. And then we show some examples of what we do want to change so that the model knows exactly how it should operate around this. And then that cleanup is what is going to get posted into our app here. So in this case, we will go ahead and bring that in. And again, all we're doing is continue to pass us through. So we still have the warm feature. We still have all the previous sync. All we're doing now is doing this this cleanup and passing back that result.

[00:10:51] Speaker 1: So we have.

[00:10:52] Speaker 2: Yep, yep. Hey, let's meet in five, actually no, it's 10 minutes on Tuesday. And you see, let's meet on Tuesday and well, I said in 10 minutes, but for 10 minutes, you'll see here that we have now the full pipeline kind of operating. And the step here now is cleanup. Again, if we look at the documentation, which we will have, there's a whole slew of models that you could be looking for. And the particular one that works really, really well here due to the price, it's how fast it is and the short utterances we're working on is this three point five four be fast. So this is what we're using. And just that cleanup that that call to that model to have everything run in this case with five hundred and twenty milliseconds, giving us a whole time from we stop transcription or sorry, we stop recording, we send the file and we get everything back into our app and just under a second. This works really great. This obviously is going up in time as we have longer dictations. So something to think about. So if something's rambling, you're doing cleanup like that's where you'll start to see this latency creep in. But on your typical everyday quick burst, you'll see it goes really well. Let's meet in five minutes. Yeah, sounds great. No worries on my end. You see all of these come back well under a second. So really, the only time you're going to start seeing kind of us getting closer to that second mark is when you're answering or monologuing something longer. It's going to take a lot, a lot of time to process any other any one of these junctions here, essentially. So a quick review of this, you're able to do this essentially one request on a simple post request. And what we're doing here is passing in raw audio. But if you had like a prerecorded, if you had a voicemail or something, you already had the file. You could also do that in this case. And we're passing in vocabulary as a way to alter and get the entity right before we do anything else. Which is what we were passing in essentially, it looks like this would be an example. The connection and warming, why this is extremely vital in this particular case, is we want to pay this like DNS, TCP and TLS handshake front. We don't want to be having to do this on like a cold request. And then you see, for example, that first instance where it was like 400, 500 milliseconds or something of the like. So the trick is we do this while we are recording. While we're recording this happen, we go ahead, we send the warm requests that pays all that debt for us so that once we go ahead and upload, we're just getting inference right away, which brings down our turnaround time, which is great. And then lastly, the model I would suggest that you start with is QUIN 3.54b. It's extremely fast. It's extremely cost efficient when you look at like the leading models in the Frontier labs. But you can, if you want, experiment with different ones like Cloud Haiku, you know, if you wanted to go that route, definitely has the speed to get in the ballpark. You also have from QUIN like the Gemma you could use as well for something that could be considered a little bit more intelligent. But all of these changes and tradeoffs will come with increased latency. So you'd want to be careful in those decisions and how you're doing that to make sure that you're that you're optimizing for the right for the right situation that you might be developing in. And then a quick review of like the prop design, what's working really well here, how you could apply this to different ones or different scenarios. Essentially, we just let the model know that it's operating as a function within a pipeline, that it's not an assistant. We specify in the prompt what it should revolve, such as like, no, actually, sorry, I mean, scratch that, that kind of thing. And we make it very clear that we are to preserve intent. So all this is in that paragraph in that prompt. Essentially, you can specify this however you would like. So if you want to do something in the medical field with like a predetermined report, like SOAP is a popular one, you could definitely do all the same specifications around that. And then providing a few shot example is always helpful. Not only do you want to provide what it should fix, so this is the behavior that we are after, but we should provide some guardrails into what it should not touch and what it should not fix so that the model has a very clear understanding of what to do in both cases. An example of this latency, kind of like in this architecture of the budget that you have, a lot of this currently will be just like in the cleanup step, the transport we will see, sorry, from the server, if you're just looking at assembly AI, like inference, RP50 on very small files is somewhere around like 140 milliseconds. So this is higher. This is assuming like a 12 second or higher file. Your transport is probably probably in that ballpark, depending on where you're at in the world or like how far you are from our server. This could change, but generally you might see something like 100 milliseconds once your war is maybe a bit less and then you have your cleanup time, which is typically well under like 600 milliseconds, give or take. But the whole goal typically that I have seen work really, really well in the field is if we can get something like on the screen in less than a second or thereabouts, it works really well. And a lot of users, I think, probably understand, too, that if they go rambling on for a minute and a half and there's a lot of cleanup, this may go maybe longer. But it's really, really important when I'm going really quickly, one sentence utterance, like I'm in Slack saying, can you check this? Let's do that. Yeah, sounds good. We want to have that quick, like we saw earlier, three, four hundred, five hundred millisecond turn on sign. So typically when you look at server requests, how do you how do you solve this short eclipse? Do transcribe faster, removing any silence, things like that. Typically anything you do to get that a bit smaller is helpful on the client side. You want to warm the connection, stay in region, be nearby the cleanup. If you can use smaller, faster models, shorter prompts, you will get a faster response compared to if you're going to use a obviously like a much bigger model. And then essentially if we can do all of this above, well, there's no reason why we're not seeing well below one second or most most dictation results. And coming soon, actually, what we want to do is instead of having you go glue all this together with Elm Gateway, figuring out the right prompt, if you're just looking for dictation kind of like out of the box, I need this fluency is gone. All starts that kind of thing. We are going to be rolling out dictation API very shortly. So this is an example of what this would look like in the same SDK format that we typically would have. You call something like client dictation, describe you pass in the audio that you want. Same vocabulary. And then you would be able to use Elm instruction. So in this case, we're just doing a simple one, for example. And what you would get back is actually the verbatim text of what was transcribed. So you don't lose that. And then you're going to get the Elm response, which will give you the corrected result. So you can see exactly how your prompts performing, how the API is performing and adjust it as necessary, along with the other kind of metadata that you typically see, for example, save time and request time. And then just some of the knobs again, there's not we're trying to do this very simple, so it's not like a really complicated. We'll keep things like you turn from if you don't ship an Elm instruction, we have a default prompt that will do just like basic dictation cleanup for you. So, again, this is out of the box, two, three lines of code and like you're up and running on the Elm instruction side, you could then pass in your own your own prompt. So in this case, very, very small, but would be a simple example of what that might look like. And you can join this waitlist today at assembly.ai.com products dictation API. So that would be. Over here, actually. So you this would be the page you're looking for and you'll be able to get on the waitlist. Oh, and one other thing, I did not show this in a demo, but because this runs on our 3.5 pro model, which is the same that we have in real time and async, for example, we were able to do this in 19 different languages and does extremely well. It's not just English examples today. We're here in English, but that's something that you can easily tweak on your end and you can, again, see what the dictation API will look like, give you the ability to put in your own instructions and get the desired output. So with that. We can get back. Open it up for some questions.

[00:20:04] Speaker 1: Thanks, Andrew, that was great. We have a couple of questions rolling in, so to start back to our decision in the initial conversation that you talked about the build, can you talk a little bit more about why you decided to sync rather than async and specifically how that decision plays into load? The question here is, is the usage of the sync API going to increase the latency and block other transcription attempts?

[00:20:37] Speaker 2: Yeah, so they're going to want to revisit the whole load and transcription attempts as far as it's like at the very surface level, why we would pick sync over async is purely it's the overhead of what you're maintaining and then the speed of it itself. So, for example, if I am going to record, if I were like we're doing in this app as we're reporting everything I'm saying or saving that like in a raw PCM file, if I'm going to send that to the async endpoint, that is going to have a floor that we cannot improve, meaning that like it's probably going to come back in. Something like 10 seconds, five seconds, which is no one's going to wait around for five, 10 seconds to get dictation back. So if you're trying to do dictation that is like it shows up on my screen right away, you're going to need something that's the latency is just lightning fast, which kind of leaves you with either real time or sync. So the same thing is if you go to that stream, your real time approach, you're dealing with WebSockets, you have the overhead of potential latency being introduced by establishing a WebSocket and getting the session started, which if I'm only speaking for one, maybe two seconds is a huge portion of that latency that doesn't quite pay off, which is where sync comes to play. As far as could you repeat maybe that load piece just to make sure I'm addressing

[00:22:08] Speaker 1: that. Is that going to increase the latency and block other transcription attempts?

[00:22:15] Speaker 2: I guess I'm I'm not clear on what is meant there, if I make an assumption that it is. You have a concurrency on your account or there's some like 429 limit, I would say no. I mean, like obviously you're trying to find the like where that breaks. We would work with you. Like there's like we have enterprise customers, for example, like using this workflow. So there's there's nothing within reason that would would block that or cause latency. No.

[00:22:48] Speaker 1: OK, next up, we have a question on punctuation when dictating punctuation. We have a specific mode that disables formatting and interprets literal speech. How would you suggest handling punctuation dictation? And then the question here specifically is, do we have plans to introduce a parameter for dealing with punctuation?

[00:23:10] Speaker 2: Yeah, we are going to build this out with the dictation mode. This is something that I've come across quite a lot. What I've had good luck with is I don't have this on the demo, but if you go to. Let me see here.

[00:23:29] Speaker 1: If you go to our playground. You'll see all the different parameters you can play around with. You can test this yourself.

[00:23:38] Speaker 2: It is conversation context. I believe it was. I can look this up and we can share it afterwards. It's one of these. It's not key terms. Yeah, it's conversation context. I'm like 98 percent sure you can actually pass in by our playground. You can actually pass in by our playground. 98 percent sure you can actually pass in by our conversation turns here and it gives like the model context of how it should answer in that regard. And in this particular field, we've had good luck with saying essentially, hey, you should return a spoken punctuation verbatim things of this nature and the model will return. So if I say like, my name is Andrew and I like to run, it would give me and may give me the punctuation mark currently of comma. But it would then tell me the word like comma, typically speaking, at which point you can strip or something of that nature, which is like admittedly what we want to make easier as we roll out dictation with that with the dictation API. So, yeah, we're going to be adding modes that are like verbatim, for example, that will be able to handle these particular use cases. So that's actually in the works. So if you need it today, it's possible through here, but if you can wait for the release, it'll probably be done for you.

[00:25:00] Speaker 1: Yeah, and we'll share the link to the dictation API wait list in the follow up to this as well so that you can get notified as soon as that's live. Cool. OK, one here, Andrew, on templates and formatting like soap notes, how could you specify the template that you want output to come to be returned in or how would that change the implementation so that the dictation is aware of that formatting?

[00:25:28] Speaker 2: Yeah, exactly. That is something where we come back to here. You would want to have a prompt that's like specifying that you want the soap note return, how it should look. You could even say, like, I don't have it here for the sake of complexity, but I was I was doing this earlier and I've done this in other cases where you could say, like, return to me JSON and then you could have like you can have steps that validates this JSON or errors out. And then we would essentially format the response into that template every time. So if I know that I always want a soap note in this particular context, I would have a prompt that is specifying what that look like. I'd probably do a few few shot examples of what that might be. And I may even go as far as saying, like, I want it in a JSON response. And then I would have like type checks on my side to make sure, like, OK, are we actually adhering to this? At which point we can just like display that directly into whatever our text boxes. But I was actually debating having that in here or not. But for the sake of clarity, if I'd be simpler to keep it, keep it here. But yeah, typically you want to be a prompt. And then you a few shot examples are usually helpful, but you may not need that sometimes just a prompt with a good JSON check will do the trick.

[00:26:46] Speaker 1: Well, I think that the dictation waitlist page also has some examples for prompts, too, so we can be sure to share those.

[00:26:52] Speaker 2: Yeah, we I think I went off of it. Let me pull it back up. I lost it. So, yeah, it is on there. You can play with it. You can actually play with it on the page.

[00:27:01] Speaker 1: That's good call. Sweet. OK, we have a couple of questions around Sync's language performance. How do we have benchmarks for how it performs across different languages? And then a two parter there, the follow up for LM Gateway models that we recommend for translation, which balance speed and accuracy.

[00:27:25] Speaker 2: Yeah, as far as languages go, we do have benchmarks. I don't have them in front of me. We could share that out like after the call. We do. I mean, anecdotally do really well. I so I speak Spanish and English. I have friends that speak other languages. We've all played around with that. And what's really cool about this model, right, not just the API, but 3.5 Pro and that whole family of models is that it is able to it's a completely unified model that can handle any of the languages that it offers at the word level. So if in one particular moment I am just sitting there talking in English, it'll do English. If I do Spanish, it'll do Spanish. So an example of like if you want to see this in in real time. If we come here, you do real time. Yeah. So if we're here, I can literally start talking. Hola, como estan ustedes? Today we're speaking about our new API. I'm super excited to share that with you. Hasta luego. And so you can see that, I mean, it just you could it just goes back and forth. This is true of sync as well. You can you can even pin languages, too, if you want to be guided more towards one language. So we do have benchmarks on this, but it does it does remarkably well, which we can show that after the fact. And then sorry, was the second part of that question.

[00:28:52] Speaker 1: How would you choose a model in LM Gateway based on language performance?

[00:29:02] Speaker 2: Yeah, I think it just depends on the problem I'm trying to solve, like in dictation, if I want to be super fast, I'm probably still going to try with Quinn. And some of the issues you have with like translation might be, do we get like I've done this with some streaming applications and it's worked really well. Like Gemini Flash, it's for 12th Quinn. But the problem you'll get is like the context. So and sometimes in certain languages, like in Spanish, the captain's armband in soccer is literally called a bracelet. And so if you're not careful, like maybe you get this weird transcription where it's like, oh, yeah, he's got the bracelet. It's like, what does that even mean? Most of these models actually do really, really well if you give it clues and hints. And so this is where we can come back to this conversation context by passing in a conversation context like here, you can see we have past examples. So by being able to say, like, hey, here's the question that was asked or here with the previous thing that was said, here's what we're trying to transcribe now. It does really, really well on these like weird, nuanced things. So in the dictation setting, which is what we're talking about, I would probably still try with Quinn, see how that goes. If I start having issues, maybe maybe I go to like the 30 30 V model like Gemma, which will the latency does go up, but maybe that's something I can live with to get better translation. But that's the direction I would probably start with and then go from

[00:30:33] Speaker 1: there. Yeah, I've been hearing from customers that Gemma is a great larger model, too. And I just wanted to add, we are also we're working on some benchmarks to help with those types of decisions for all the models that are on LM Gateway. So that doesn't answer the question in this moment, but more to come there on which models we recommend for what in the very near future. OK, I have a couple more questions here. Would you recommend using the SDK to get started or how would you recommend getting started if you're building this for the first time?

[00:31:13] Speaker 2: Yeah, maybe personal preference, the SDK is what's cool about SDKs, like when SDKs can make things easy, I would recommend the SDK. I probably would say that. Yeah, I probably recommend this. Reason why I don't mean to flip around, but like a lot of us are using coding agents these days. The cool thing with SDKs is the agent is able to scan through the SDK, understand the product really, really well. You're not relying on it like pulling old docs or anything of that nature. So I typically will find that if you're developing in that way or at least just trying to get like your initial hello world, like boilerplate version of this stood up and you're using an agent, it is just so much faster and so much easier because it's just able to get through that much more quickly. So, yeah, I favor the SDKs primarily for that reason right now.

[00:32:06] Speaker 1: Sweet. OK, back to LM Gateway briefly, we have a question here. Can you only do cleanup with LM Gateway or is it possible to use external providers or local LLMs to do cleanup?

[00:32:19] Speaker 2: Yeah, that's a great question. So LM Gateway, so this is all just stitched together in the code, right? Like the dictation itself, we will roll. We're going to roll this as like a single API. But if you're looking at this, like building dictation with sync STT plus LM Gateway, you are having to make two API calls in this instance. It's also LM Gateway is using the OpenAI schema, so you can plug and play that to different providers as needed. So, yes, you can go outside of LM Gateway with us. You can use a different provider. You can go direct to a provider or you can use if you have a way to set this up locally, if you stand up on your side, if you're hosting your own model in a VPC or something that you want to use, you can 100 percent do that. The idea is that you're just prompting it correctly. You're giving it the text and you're getting it back. You're doing so quickly. So, yeah, this what we showed today is actually not the dictation app. It's sync plus LM Gateway. Dictation will simply be making this a bit easier to do. And you're more like out of the box situation in the future.

[00:33:24] Speaker 1: Yeah, and with the sync, to Andrew's point, like you can plug in any LM provider, you can do that with a locally hosted model. What I will say is we're partial to recommending the Quen 3.5 for B because we also optimize that model internally for use cases like this specifically. So that is a great option. And you can you can we've seen customers use all kinds of options, too. Cool. OK, another one around limitations. Is there a time limit or context limit for the sync API?

[00:34:05] Speaker 2: Two minutes, 120 seconds. No, an 80 millisecond floor, I believe. And then there's no context as far as like we'd understand with the LM. But, yeah, you would with Quen. I think we have that. I wonder if we have that here. Roll over. Yeah, so you have the max context here. So you have to go, you just find the model you're using and be there. So you'd have that on the LM gateway side. But yeah, sync STT is 80 millisecond floor, 120 second top end. So.

[00:34:41] Speaker 1: Well. Wonderful, I think that captures all of our questions in our Q&A, so unless if anyone wants to drop some last ones in as we're finishing up, but Andrew, do you want to tell folks where they can go to get an API key if they don't have one already?

[00:34:59] Speaker 2: Yeah, I mean, just go to it's pretty much everywhere. I find it in the docs with sign in or sign up. But yeah, if you just go to assembly.com, you'll be able to go ahead and I'm logged in, so I have that. But yeah, you'll be able to just jump directly and sign up for free down here as well. So.

[00:35:19] Speaker 1: Wonderful, and we'll include in our follow up email, too, if you use the link in the follow up email to log back in or sign up, we will add 20 extra dollars in API credits to your account within the next 24 hours. So that'll happen automatically and you should see some extra credits in there to start building. Awesome, OK, great question, we just saw a question come through on that, so you'll get that in your follow up for extra credits. Fantastic, thanks, everyone, for joining us today, this is great and thank you again, Andrew, for all the time you spent prepping this. We will send out a recording and all the info in the follow up.

[00:36:03] Speaker 2: Thanks. Thanks, everyone.

ai AI Insights
Arow Summary
Andrew from AssemblyAI explains how to build fast dictation workflows using the Sync API, which returns a transcript in a single POST request and is designed for short audio bursts. Compared with asynchronous transcription, Sync avoids polling and multi-step job handling; compared with streaming, it avoids repeated WebSocket setup overhead for short dictation sessions. The recommended workflow uses the Universal-3.5 Pro model, key terms prompting for names and domain-specific vocabulary, and connection warming initiated while the user records to reduce DNS, TCP, and TLS setup latency. For cleanup of disfluencies, repetitions, and self-corrections, the transcript can be sent to AssemblyAI's LLM Gateway—using a fast model such as Qwen 3.5 4B Fast—with a constrained prompt and few-shot examples that preserve intent. Typical short utterances can complete in under a second. A forthcoming Dictation API will combine transcription and cleanup into a simpler endpoint while returning both verbatim text and cleaned output. It will support custom LLM instructions, vocabulary, 19 languages, and planned modes for verbatim or punctuation-sensitive dictation. The session also covers formatting structured outputs such as SOAP notes, multilingual performance, model tradeoffs, external or local LLM cleanup options, and Sync API limits of 80 ms minimum and 120 seconds maximum audio duration.
Arow Title
Building Low-Latency Dictation with AssemblyAI Sync API
Arow Keywords
AssemblyAI Remove
Sync API Remove
dictation Remove
speech-to-text Remove
Universal-3.5 Pro Remove
low latency Remove
connection warming Remove
key terms prompting Remove
LLM Gateway Remove
Qwen 3.5 4B Fast Remove
transcript cleanup Remove
SOAP notes Remove
multilingual transcription Remove
punctuation dictation Remove
Dictation API Remove
Arow Key Takeaways
  • Use the Sync API for short dictation bursts when a single-request, low-latency workflow is preferable to async jobs or repeated WebSocket sessions.
  • Start warming the Sync connection when recording begins to reduce cold-request networking overhead before audio is submitted.
  • Use key terms prompting to improve recognition of names, product terms, and specialized vocabulary.
  • Add an LLM cleanup pass for self-corrections, repetitions, and disfluencies; preserve user intent with explicit instructions and few-shot examples.
  • Qwen 3.5 4B Fast is recommended as a fast, cost-efficient starting model for short dictation cleanup, while larger models may improve quality at the cost of latency.
  • For structured outputs such as SOAP notes, prompt the LLM to return a specified template or JSON and validate the response in the application.
  • The forthcoming Dictation API will package transcription and cleanup together, provide verbatim and corrected text, support custom instructions, and simplify deployment.
  • Sync supports 19 languages through the Universal-3.5 Pro family and can handle code-switching; language and model benchmarks are planned or available on request.
  • External providers and locally hosted LLMs can replace LLM Gateway for cleanup if desired; the key requirement is fast, well-prompted text transformation.
  • Sync accepts audio from 80 ms up to 120 seconds; LLM context limits depend on the cleanup model selected.
Arow Sentiments
Positive: The webinar is enthusiastic and practical, emphasizing fast performance, clear implementation guidance, upcoming product capabilities, and support for developers building dictation products.
Arow Enter your query
{{ secondsToHumanTime(time) }}
Back
Forward
{{ Math.round(speed * 100) / 100 }}x
{{ secondsToHumanTime(duration) }}
close
New speaker
Add speaker
close
Edit speaker
Save changes
close
Share Transcript