Abstract
Word Error Rate (WER) has been the predominant metric used to evaluate the performance of automatic speech recognition (ASR) systems. However, WER is sometimes not a good indicator for downstream Natural Language Understanding (NLU) tasks, such as intent recognition, slot filling, and semantic parsing in task-oriented dialog systems. This is because WER takes into consideration only literal correctness instead of semantic correctness, the latter of which is typically more important for these downstream tasks. In this study, we propose a novel Semantic Distance (SemDist) measure as an alternative evaluation metric for ASR systems to address this issue. We define SemDist as the distance between a reference and hypothesis pair in a sentence-level embedding space. To represent the reference and hypothesis as a sentence embedding, we exploit RoBERTa, a state-of-the-art pre-trained deep contextualized language model based on the transformer architecture. We demonstrate the effectiveness of our proposed metric on various downstream tasks, including intent recognition, semantic parsing, and named entity recognition.
Introduction
For decades, Word Error Rate (WER) has served as the standard metric for evaluating Automatic Speech Recognition (ASR) systems. WER is calculated based on the edit distance (insertions, deletions, and substitutions) between a reference transcript and the ASR hypothesis. However, a significant limitation of WER is its "meaning-blind" nature. It treats all word errors with equal weight, regardless of their impact on the semantic content of the utterance.
As illustrated in Figure 1, two different hypotheses can have the same WER but vastly different semantic implications. For the reference "This is a cat," the hypothesis "This is the cat" (substitution of 'a' with 'the') and "This is a cap" (substitution of 'cat' with 'cap') both result in a 25% WER. However, the former preserves the core meaning, while the latter changes the subject entirely. This discrepancy makes WER a poor predictor of performance for downstream Spoken Language Understanding (SLU) tasks like intent recognition or slot filling.
Methodology: Semantic Distance (SemDist)
To address the limitations of WER, we propose Semantic Distance (SemDist). This metric evaluates ASR performance in a sentence-level embedding space rather than at the literal word level.
The SemDist calculation involves three primary steps:
- Sentence Embedding: We utilize RoBERTa, a pre-trained transformer-based language model, to encode both the reference text and the ASR hypothesis.
- Pooling: A single 768-dimensional vector is generated for each sentence by taking the mean of the output vectors from the RoBERTa model.
- Distance Calculation: The semantic gap is measured using cosine distance: $SemDist = 1 - \text{cosine\_similarity}(e_{ref}, e_{hyp})$.
The resulting score ranges from 0 (semantically identical) to 1 (completely different). Because RoBERTa is trained on vast corpora, it implicitly learns that certain words (like nouns and verbs) carry more semantic weight than others (like articles), allowing SemDist to naturally prioritize meaningful errors.
Experimental Setup
The metric was validated using several large-scale datasets, including 1.7 million hours of de-identified English video data and 50,000 hours of voice assistant commands for ASR training. Evaluation sets included open-domain dictation (23k utterances) and assistant-domain tasks (15k utterances). For NLU-specific validation, a subset of 10,000 utterances was used to evaluate intent recognition and semantic parsing performance.
Results and Discussion
Our experiments demonstrate that SemDist is a superior indicator of downstream NLU performance compared to WER.
As shown in Figure 6, we compared different sets of ASR hypotheses. Sets A, B, and C were constructed to have the exact same WER (7.44%). Despite the identical WER, their performance on a Named Entity Recognition (NER) task varied significantly. Set C (BetterSem), which had the lowest SemDist (0.0028), achieved an NER-F1 score of 0.846. In contrast, Set B (WorseSem), with a higher SemDist (0.0044), saw its F1 score drop to 0.590. This confirms that SemDist effectively captures the "quality" of errors that WER misses.
Furthermore, in assistant-domain tasks, reducing SemDist while maintaining a constant WER improved Intent Accuracy from 94.28% to 96.22%. Qualitative analysis also showed that SemDist correctly identifies that "he is so cute" is semantically closer to "she is so cute" than "heat is so cute," despite identical WERs.
Conclusion
Semantic Distance provides a more nuanced and NLU-aligned evaluation of ASR systems. By leveraging deep contextualized embeddings, it distinguishes between trivial and critical transcription errors. While computationally more expensive than WER, it serves as a vital tool for model selection in production NLU systems. Future work will explore using SemDist as a direct loss function during ASR model training to optimize systems for semantic accuracy.
Summary updated