TL;DR
This paper introduces a novel evaluation metric for Automatic Speech Recognition (ASR) within the context of Speech Translation (SLT). By leveraging word embeddings, their proposed metric, WER-E, better captures the impact of ASR errors on downstream translation quality than the traditional Word Error Rate (WER). This leads to improved correlation with SLT performance and the ability to select better ASR hypotheses for translation.
What Problem Does This Paper Address?
Evaluating the quality of Automatic Speech Recognition (ASR) output is crucial, especially when that output is used as input for other tasks like Speech Translation (SLT). The standard metric for ASR is Word Error Rate (WER), which counts substitutions, insertions, and deletions. However, WER treats all substitutions equally, regardless of whether the substituted word is semantically or morphologically very similar to the correct word. This is problematic for SLT because a substitution like "cat" for "cats" might have a much smaller negative impact on translation quality than "cat" for "dog." Existing research has shown that sub-optimal WER scores don't always translate to poor SLT performance, highlighting the need for an ASR evaluation metric that better reflects its impact on the entire SLT pipeline.
Methods
The core contribution of this paper is an extension of the traditional WER metric, termed WER-E (WER with embeddings). The key idea is to modify the substitution cost within the WER calculation. Instead of a binary cost (0 for a match, 1 for a substitution), WER-E uses the cosine distance between word embeddings of the hypothesized word and the reference word. This means that morphologically similar words (e.g., "souveraine" and "souveraines") will have a smaller substitution cost, reflecting their lesser impact on translation.
The authors explore two variants:
- WER-E: The alignment path is kept the same as standard WER, but substitution costs are replaced by cosine distances.
- WER-S (WER soft): The substitution costs are replaced by cosine distances, and the alignment path is recomputed to find the optimal path with these new costs.
They also introduce a "WER soft" (WER-S) variant that recalculates the optimal alignment path using the embedding-based substitution costs, aiming to find the absolute lowest error score.
Datasets
The experiments were conducted on a French-English speech translation corpus. This corpus initially contained 2643 French utterances and was extended with an additional 4050 sentences from a news domain. For ASR output, N-best lists (N=1000) were generated for each utterance. The MT system was trained on data from IWSLT 2012 evaluation, including Europarl, Ted, and News-Commentary corpora.
Key Insights and Ideas
- Word embeddings can bridge the gap between ASR and SLT: By using word embeddings, the authors demonstrate that ASR evaluation can become more sensitive to the actual impact on downstream translation tasks.
- Morphological similarity matters: The proposed WER-E metric correctly penalizes morphologically similar substitutions less, aligning better with human intuition and the reality of translation.
- Beyond simple word matching: The paper shows that a more nuanced evaluation of ASR errors can lead to better overall SLT performance, even if the raw ASR WER score doesn't improve.
- Oracle experiments reveal potential: The "oracle" experiments, where the best hypothesis from the N-best list is selected based on different metrics, show that WER-E and WER-S can identify hypotheses that lead to better translations than those selected by standard WER.
Key Results
- Improved Correlation: The proposed WER-E and WER-S metrics showed a better correlation with SLT performance metrics (TER, BLEU, METEOR) compared to the standard WER. For instance, on the test set, WER-E and WER-S had a correlation of -0.799 with METEOR, while WER had -0.753.
- Oracle Performance: When selecting the best hypothesis from the ASR N-best list using WER-E or WER-S, the resulting SLT performance (measured by TER, BLEU, and METEOR) was consistently better than when selecting based on standard WER. For example, on the dev set, using WER-S for oracle selection improved TER by 0.18 points and BLEU by 0.12 points compared to WER.
- Preliminary ASR Tuning: A preliminary experiment tuning the ASR language model weight using WER-E showed small gains in TER and BLEU for the SLT system, suggesting potential for direct ASR optimization.
Limitations and Future Work
- Limited ASR Tuning: The authors acknowledge that their preliminary ASR tuning experiment was limited to optimizing only the language model weight. They suggest that optimizing a larger set of ASR meta-parameters using their new metrics could yield more significant improvements.
- Translation Evaluation Metrics: The paper notes that translation evaluation metrics themselves are imperfect, which can affect the perceived gains from ASR optimization.
- Future Work: The authors propose further research into optimizing ASR systems using their novel metrics and exploring their applicability to other downstream tasks. They also make their code and corpora publicly available for reproducible research.
Summary updated