How to Evaluate AI-Generated Clinical Notes Reliably (Full Transcript)

[00:00:00] Speaker 1: For us, a lot of it is user feedback that we'll get like, Hey, this thing was spelt incorrectly or done this way, or I had to edit this wasn't picked up. So that's typically a lot of like the user feedback we get on specifically the transcript part. Then from like the final output part, we use a measure of like F1 scores, rogue, birds, LLM as a judge, workflows, essentially every note that we generate, we will compare it to the sign note that the provider submits and writes back to their electronic health record system. And we'll do essentially evals for F1, but also LLM as a judge. So every note that comes in, we'll process nightly and come up with like a score based off our grading criteria for this is a good medical notes versus not. So we'll find things because just because the provider didn't edit it doesn't mean it's a perfect note also. So that's why you need this like other signal to counteract in that in those cases.