Semantic and Functional Evaluation of Automated Transcripts

This direction is split into other directions: semantic error weighting. Its last version stays readable and citable here.

Version 1 of 146 papers60 conceptsfrozen 2026-10-04

For decades, the gold standard for evaluating Automatic Speech Recognition (ASR) has been Word Error Rate (WER). It is a simple, distance-based metric: count the insertions, deletions, and substitutions required to turn a hypothesis into a reference transcript, then divide by the total number of words.

However, WER is a "blunt metric." It treats every word with equal weight, regardless of its semantic importance. In a clinical setting, mishearing "the" as "a" is a triviality; mishearing "no" as "go" is a catastrophic failure. As speech models move from simple transcription to complex reasoning and downstream understanding, researchers are shifting toward evaluation frameworks that prioritize meaning over mere character matching.

The Failure of Uniform Weighting

The core problem with WER is its lack of semantic differentiation. As noted by Andrea Vanzo and colleagues in the development of Conformer-1, standard metrics fail to distinguish between minor filler words and critical information like proper nouns or technical terms.

This discrepancy creates a "usability threshold." Cosmin Munteanu and Gerald Penn demonstrated that transcripts only become beneficial to users once the WER drops below 25%; above that, the cognitive load of deciphering errors can actually make a transcript less useful than having no text at all Measuring the acceptable word error rate of machine-generated webcast transcripts.

The impact is even more pronounced in specialized domains:

Mapping Errors to Human Cognition

To bridge the gap between WER and actual utility, researchers are developing metrics that incorporate linguistic and cognitive hierarchies.

One approach is to use semantic embeddings rather than literal word matching. Jimmy Tobin and Katrin Tomanek showed that BERTScore—which measures the similarity of word vectors—aligns much more closely with human judgments of speech quality than WER, particularly for disordered speech where phonetic accuracy may be low but intelligibility remains high Assessing ASR Model Quality on Disordered Speech using BERTScore.

In specialized fields, these metrics are being "grounded" in domain knowledge. Joel Shor and Ehud Rivlin introduced Clinical BERTScore (CBERTScore), which uses medical knowledge graphs to assign higher penalties to errors involving clinically significant terms Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings.

Taking this further, Chao-Han Huck Yang and Shinji Watanabe proposed SpeechIQ (SIQ). This framework maps model performance to Bloom’s Taxonomy—Remember, Understand, and Apply—to provide a holistic "Intelligence Quotient" for voice-understanding models SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models. This shift acknowledges that the goal of modern ASR is often not just to transcribe, but to act on information.

Error Propagation and Downstream Utility

The push for semantic evaluation is also driven by how ASR errors propagate through AI pipelines. In Spoken Language Understanding (SLU), the transcript is merely an intermediate step toward determining intent.

Dmitriy Serdyuk and Christian Fuegen argue that the traditional objective of minimizing WER often conflicts with the needs of NLU, where specific "slot" words are critical and filler words are irrelevant Towards End-to-end Spoken Language Understanding. Similarly, Ori Shapira and colleagues developed the ENDOW framework to stress-test how specific error types—like misheard names versus verbs—impact downstream tasks like summarization and QA Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks.

Identifying Hallucinations

As ASR systems increasingly rely on Large Language Model (LLM) backbones, they are becoming prone to "hallucinations"—generating fluent but factually incorrect text that doesn't match the audio. Traditional WER cannot distinguish between a phonetic mishearing and a semantic hallucination.

To address this, Alkis Koudounas and Moreno La Quatra introduced SHALLOW, a benchmark that decomposes errors into lexical, phonetic, and semantic dimensions Hallucination Benchmark for Speech Foundation Models. This allows researchers to identify "catastrophic semantic reversals"—such as the addition or omission of a negation—which might only change a single word (keeping WER low) but completely flip the meaning of the transcript.

By moving beyond the "all words are equal" paradigm, the field is evolving toward evaluation that reflects the practical, human-centric utility of speech technology.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_and_functional_evaluation_of_au_32e26316,
  title = {Semantic and Functional Evaluation of Automated Transcripts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-and-functional-evaluation-of-automated-transcripts/rdr_32e26316d84c40c3bd00f78e5bba85e1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_and_functional_evaluation_of_au_32e26316_v1,
  title = {Semantic and Functional Evaluation of Automated Transcripts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-and-functional-evaluation-of-automated-transcripts/rdr_32e26316d84c40c3bd00f78e5bba85e1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}