TL;DR
This paper introduces novel methods to evaluate Automatic Speech Recognition (ASR) errors beyond the standard Word Error Rate (WER) by quantifying the severity of mistakes. They demonstrate that these severity measures, based on sentiment analysis and text embeddings, correlate better with human judgment and can be used to improve ASR systems by incorporating them into the training process.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems have become ubiquitous, but their performance is typically measured by Word Error Rate (WER). WER simply counts substitutions, deletions, and insertions, treating all errors equally. However, human perception of error severity varies greatly; a minor misspelling is far less problematic than a word that completely changes the meaning of a sentence. This is particularly critical in high-stakes domains like healthcare, where ASR transcriptions are used for diagnosis and patient care. The paper addresses the need for ASR evaluation metrics that better reflect human judgment of error severity and can be used to improve ASR system development.
Methods
The authors propose two primary methods for quantifying ASR error severity:
- Sentiment Analysis: They leverage pre-trained sentiment analysis models (NLTK, FLAIR, TextBlob) to measure the difference in sentiment between the ground-truth transcription and the ASR output. The absolute difference in sentiment scores serves as a severity score.
- Text Embeddings: They utilize sentence-level text embeddings from pre-trained models (BERT, MiniLM, MPNET, DisRob) to represent the semantic meaning of utterances. The cosine distance between the embeddings of the ground-truth and ASR output is used as a measure of semantic difference, with a higher distance indicating greater severity.
For evaluating ASR systems, they propose three metrics: Mean Absolute Error of Sentiment Difference (MAE-DS), Mean Squared Error of Sentiment Difference (MSE-DS), and the mean cosine distance across all utterances.
To improve ASR systems, they incorporate severity into the training process by modifying the Connectionist Temporal Classification (CTC) loss function. The cosine distance between embeddings is used as a weight (w) to scale the CTC loss, effectively penalizing errors that are semantically dissimilar more heavily.
Datasets
- Simulated Patient-Doctor Conversations: A dataset of 272 audio files with transcripts was used for evaluating the proposed severity measures. This dataset was split into over 39,600 non-silent utterances.
- Human-Labeled Severity Scores: 150 pairs of transcripts were rated by three medical school students on a scale of 0 (very minor error) to 2 (severe error).
- LJ Speech Dataset: This dataset of 13,100 short audio clips from a single speaker was used for training and evaluating the ASR system modified with the severity-aware loss function.
Key Insights and Ideas
- WER is Insufficient: The paper strongly demonstrates that WER does not always correlate well with human judgment of error severity, as illustrated by the "I love you" vs. "I loathe you" vs. "I luv you" example.
- Text Embeddings Excel: Text embeddings, particularly those from models like MiniLM and DisRob, show a significantly better correlation with human severity ratings than WER. This suggests that capturing semantic meaning is crucial for accurate severity assessment.
- Sentiment Analysis Captures Nuances: While less correlated than embeddings, sentiment analysis can capture certain types of errors that WER misses, especially those involving disfluencies or subtle shifts in tone.
- Severity Can Improve ASR: Incorporating semantic severity into the ASR training loss function leads to substantial improvements in both WER and the proposed severity metrics, indicating that ASR systems can be trained to produce less semantically damaging errors.
Key Results
- Correlation with Human Ratings: Text embedding-based severity scores achieved correlations ranging from 0.53 to 0.59 with human ratings, while WER had a correlation of 0.43. Sentiment analysis scores had lower correlations (0.29 to 0.34).
- ASR System Evaluation: When evaluating 5 ASR systems using the proposed metrics, Whisper and Vosk generally performed best, with improvements in severity metrics often not perfectly aligning with WER improvements.
- CTC-by-Cosine Loss: Training an ASR system with the CTC-by-Cosine loss function resulted in a significant reduction in both WER (from 0.268 to 0.219 on validation data) and average cosine distance (from 0.249 to 0.201) compared to a baseline CTC loss.
Limitations and Future Work
- Limited Data and Raters: The study's conclusions were based on a limited number of raters and data. Future work should involve more medical professionals, more audio data, and a wider variety of contexts to solidify empirical results.
- Single Architecture: Experiment 3 (using severity in training) was conducted on a single ASR architecture. Future work should explore the impact of severity-aware training on various architectures, seeds, and audio data.
- Future ASR Development: The authors suggest exploring different architectures, data, and methods for using semantics in ASR training. They also highlight the potential for using severity in more sophisticated ways beyond just a weighted loss function.
Summary updated