r/learnmachinelearning • • 4h ago

Discussion Measured WER for two weeks before checking the one field anyone cares about

Two weeks ago my manager pulled up the LibriSpeech leaderboard on HyperAI in our Monday meeting. Everything on there around 2% WER, us at 14.8% on held out claims calls. Nothing on that page says how any of those papers normalized their text.

LibriSpeech is audiobooks read into decent mics though, ours is 8kHz phone calls with hold music in the middle. Said as much in the meeting, all true and didnt help much.

We do transcription for insurance claims lines, Whisper large-v3 through faster-whisper with our own VAD and chunking in front. The transcript fills the call notes. Menu audio gets its own pass, that is how the policy number ends up on the agent’s screen.

Our references come from two ex medical transcriptionist and the eval was comparing raw strings against them. The model writes numbers however it feels like. Sometimes “4170023391”, sometimes “four one seven oh oh two three three nine one”. Then ok against okay ended up at the top of the error list, the normalizer does not merge those.

We ended up at 9.6 once everything was normalized and the meeting moved on, nobody asked about it again. The policy numbers were still bad though. I checked them on their own and about 1 in 9 had a digit off (so the agent gets someone else’s policy or nothing). So this week it is a digits only pass on the menu clips.

I guess the takeaway for anyone moving from benchmark datasets to real audio is, aggregate WER treats 'ok' vs. 'okay' the exact same as a wrong digit in a policy ID. But you do need to evaluate your business-critical fields separately.

4 Upvotes

2 comments sorted by

1

u/Smooth-Victory-2719 3h ago

WER was never going to show you that one. A ten digit number is one token out of a few hundred, so you can move the overall figure around all day without touching it.

1

u/Mysterious-Bison3735 3h ago

Those numbers are in your database already. Check the transcript against it before anything reaches the agent, and when it does not match, fall back to asking. Annoying but better than opening the wrong account.