TL;DR
This paper proposes a novel metric for evaluating Automatic Speech Recognition (ASR) output that better reflects human perception than traditional metrics like Word Error Rate (WER) or Character Error Rate (CER). By training models on human-annotated scores, they demonstrate that features like substitution errors and consecutive errors are crucial for predicting human judgment, achieving correlations close to inter-human agreement.
What Problem Does This Paper Address?
Evaluating the quality of ASR output is crucial for developing better systems. Traditional metrics like WER and CER, while objective and useful for comparing algorithms, don't always align with how humans perceive the intelligibility and usefulness of the recognized text. For instance, a transcript with a low CER might still be difficult for a human to read or understand due to unnatural phrasing or specific types of errors. This is particularly challenging in languages like Japanese, where word boundaries can be ambiguous and orthographic units are not always clear-cut. The authors aim to bridge this gap by developing a metric that captures human perceptual quality.
Methods
The core of this paper is the development of a human-perception-based evaluation metric. The authors collected human-annotated scores for ASR output in two phases: before listening to the audio and after listening. They then explored various features derived from the ASR output, including:
- Baseline metrics: Character Error Rate (CER) and Syllable Error Rate (SyER), broken down into substitution (Sub), insertion (Ins), and deletion (Del) rates.
- Keyword error rates: Focusing on errors within identified important keywords.
- Consecutive errors: Measuring the length of consecutive character or syllable errors.
- Positional errors: Indicating whether the first or last word of a message was misrecognized.
Two machine learning approaches were used to predict human-perceived scores: linear regression and random forest. The random forest model, being a non-linear approach, was expected to capture more complex relationships between features and human judgment. They used a leave-one-out cross-validation strategy for evaluation.
Datasets
The paper does not explicitly name a publicly available dataset. Instead, they created their own dataset of ASR output samples generated by IBM's Attila recognizer using a general-purpose acoustic model and an open-domain Japanese language model with a vocabulary of approximately 500K words. The samples were selected based on character error rate (CER) to ensure a distribution across different quality levels (less than 20% CER, 20-35% CER, and 35-50% CER). The average length of the samples was 14.1 characters. A total of 240 evaluated samples were obtained from 5 participants.
Key Insights and Ideas
- Traditional metrics fall short: CER and WER, while useful, have a limited correlation with human perception, especially for after-listening scores. This suggests humans can compensate for certain errors when they have context.
- Consecutive and substitution errors matter most: The study found that the number of substitution errors and the length of consecutive errors were the most important features for predicting human perception of ASR quality.
- Context is key for human evaluation: Human annotators' scores improved significantly after listening to the audio, indicating that the audio context is vital for accurate quality assessment.
- Japanese language nuances: The difficulty in defining word units in Japanese makes syllable-based metrics and character-based analysis particularly relevant.
- Model-based metrics can rival human agreement: The proposed model-based metrics achieved correlations with human scores that were close to the agreement between different human annotators, suggesting their effectiveness.
Key Results
The paper reports significant improvements in correlation with human perception compared to traditional metrics:
- CER correlation: 0.456 (before listening), 0.368 (after listening).
- Best model-based metric (Random Forest, RF5): Achieved a Spearman's correlation of 0.715 (before listening) and 0.622 (after listening).
- Human-to-human agreement: The average correlation between human annotators was 0.765 (before listening) and 0.645 (after listening).
This shows that their best model-based metric (RF5) comes very close to the inter-human agreement, demonstrating its ability to predict human perception effectively. The most influential features for prediction were identified as syllable substitution rate, maximum consecutive character error length, and character substitution rate.
Limitations and Future Work
- Language specificity: The authors acknowledge that the finding that CER is less correlated with human perception might be specific to Japanese and suggest further investigation in other languages.
- Dataset size: While they used leave-one-out cross-validation, the overall number of samples (240) and annotators (5) is relatively small.
- Feature set: While they explored several features, there might be other linguistic or acoustic features that could further improve the prediction model.
Future work could involve exploring these features in other languages, expanding the dataset, and investigating more sophisticated modeling techniques. The authors also plan to further study how specific error types influence human perception.
Summary updated