Voice Benchmark Compares LLM Quality, Speed and Cost (Full Transcript)

[00:00:00] Speaker 1: Which LLM should you choose to run your transcripts on? We have leaderboards for coding, math, and general trivia, but none of them really test for what an LLM should do in a voice product, like a meeting transcriber or dictation app. That's why we built Voice Standard Bench. We took every model on LLM Gateway and gave them the same transcripts, from a 30-second voicemail to an hour-long meeting, and we tested it on various different jobs, like information extraction, summarization, and general text cleanup. What we then did was had an independent LLM score every answer, and we put quality, speed, and cost side-by-side. My favorite finding was that GPT-6 Luna, with reasoning turned off, nearly matches the best open models at every length for about a cent per hour of audio. That means if you're running one of these frontier models on every call, you might be overspending on your LLMs. But don't just take our word for it. The Bench is open source, so you can run it on your own audio, and even switch models on LLM Gateway by just changing one line. The link is down in the description, so good luck with your tests, and let us know what you find. Bye.