AssemblyAI Launches a Voice-Focused LLM Benchmark (Full Transcript)

[00:00:00] Speaker 1: Hi. Models that are great for coding aren't necessarily great for voice outputs. Let me give you an example. Let's say you just got done with your one hour meeting. Now you want to take out some tasks and then you want to maybe change it to British spellings or maybe you want to just summarize. How do you figure out which model is best for that voice output? Today we have an answer. For that we are today introducing our voice standard benchmark. Get started. Let's go to our assembly AI dashboard and then LLM gateway and here you can compare 50 LLMs and use our 50 LLMs directly. So let's say I select a transcript. Here you could upload a transcript. If needed, I have uploaded one already. Let's pick this one. This is about our Universal 3.6 Pro launch and this transcript. I'm going to click continue and then I could choose on one side Quen 3.5 4b fast and on the other hand Gemini 2.5 flashlight and then let's say summarize the transcript and you notice right here Quen 3.5 4b is extremely fast but Gemini 2.5 flashlight is very precise. You can ask a question of your type. Let's say give me tasks for me from that audio clip and then you see again that Quen is extremely fast but the output maybe you want is better from Gemini. It gave you seven tasks. Now let me tell you why our benchmark is important because think about it. Typical benchmarks measure coding, mathematics or reasoning but our benchmark is solely on voice outputs and something you really need out of your voice agents or speech to text and all other use cases. Every task we actually measured the performance, the cost as well as the speed. So for example if you have like you know information instruction let's go to one of this task and let's say the length of your audio was two minutes or 30 seconds and in this case the best performance comes out to be from GPD 5.6, lowest cost from Nemotron and fastest again what I showed you Quen 3.5 4b but things change when you increase the length of audio to an hour then bigger models perform way better. For example GPD 6.1 sole performed way better and the lowest cost was still Nemotron but fastest speed came out of from Gemini 3.5 flash and similarly you can check the results for summarization, text correction, dictation cleanup and all your use cases but our transcriptions are not yours. You might have different accent, different English needs, your vocabulary might be different, your needs might be different so we recommend using your audio to test on our benchmark because our benchmark is open source and you can analyze you can analyze directly on our LLM gateway or directly through our GitHub repo which is open source as well and most importantly why LLM gateway. Think about 50 models that we give access to you on our portal directly because of which you could actually get results faster. Think about let's say you have an output you would on a typical scenario you will transfer that data to your actual server then route to a particular LLM. Why don't you save that cost and direct to LLM on our endpoint through our LLM gateway directly so you save time and latency and get access to 50 powerful models that are best for voice outputs directly on assembly AI's developer platform as well as through our APIs. So we can't wait to see what results you get with your own voice data on our LLM gateway. Let us know what models are best for your use case. Thank you.