Better Evaluation of ASR in Speech Translation Context Using Word Embeddings

Ngoc-Tien Le, Christophe Servan, François Portet, Laurent Besacier2016 · DOI

TL;DR

This paper introduces a novel evaluation metric for Automatic Speech Recognition (ASR) within the context of Speech Translation (SLT). By leveraging word embeddings, their proposed metric, WER-E, better captures the impact of ASR errors on downstream translation quality than the traditional Word Error Rate (WER). This leads to improved correlation with SLT performance and the ability to select better ASR hypotheses for translation.

What Problem Does This Paper Address?

Evaluating the quality of Automatic Speech Recognition (ASR) output is crucial, especially when that output is used as input for other tasks like Speech Translation (SLT). The standard metric for ASR is Word Error Rate (WER), which counts substitutions, insertions, and deletions. However, WER treats all substitutions equally, regardless of whether the substituted word is semantically or morphologically very similar to the correct word. This is problematic for SLT because a substitution like "cat" for "cats" might have a much smaller negative impact on translation quality than "cat" for "dog." Existing research has shown that sub-optimal WER scores don't always translate to poor SLT performance, highlighting the need for an ASR evaluation metric that better reflects its impact on the entire SLT pipeline.

Methods

The core contribution of this paper is an extension of the traditional WER metric, termed WER-E (WER with embeddings). The key idea is to modify the substitution cost within the WER calculation. Instead of a binary cost (0 for a match, 1 for a substitution), WER-E uses the cosine distance between word embeddings of the hypothesized word and the reference word. This means that morphologically similar words (e.g., "souveraine" and "souveraines") will have a smaller substitution cost, reflecting their lesser impact on translation.

The authors explore two variants:

  1. WER-E: The alignment path is kept the same as standard WER, but substitution costs are replaced by cosine distances.
  2. WER-S (WER soft): The substitution costs are replaced by cosine distances, and the alignment path is recomputed to find the optimal path with these new costs.

They also introduce a "WER soft" (WER-S) variant that recalculates the optimal alignment path using the embedding-based substitution costs, aiming to find the absolute lowest error score.

Datasets

The experiments were conducted on a French-English speech translation corpus. This corpus initially contained 2643 French utterances and was extended with an additional 4050 sentences from a news domain. For ASR output, N-best lists (N=1000) were generated for each utterance. The MT system was trained on data from IWSLT 2012 evaluation, including Europarl, Ted, and News-Commentary corpora.

Key Insights and Ideas

  • Word embeddings can bridge the gap between ASR and SLT: By using word embeddings, the authors demonstrate that ASR evaluation can become more sensitive to the actual impact on downstream translation tasks.
  • Morphological similarity matters: The proposed WER-E metric correctly penalizes morphologically similar substitutions less, aligning better with human intuition and the reality of translation.
  • Beyond simple word matching: The paper shows that a more nuanced evaluation of ASR errors can lead to better overall SLT performance, even if the raw ASR WER score doesn't improve.
  • Oracle experiments reveal potential: The "oracle" experiments, where the best hypothesis from the N-best list is selected based on different metrics, show that WER-E and WER-S can identify hypotheses that lead to better translations than those selected by standard WER.

Key Results

  • Improved Correlation: The proposed WER-E and WER-S metrics showed a better correlation with SLT performance metrics (TER, BLEU, METEOR) compared to the standard WER. For instance, on the test set, WER-E and WER-S had a correlation of -0.799 with METEOR, while WER had -0.753.
  • Oracle Performance: When selecting the best hypothesis from the ASR N-best list using WER-E or WER-S, the resulting SLT performance (measured by TER, BLEU, and METEOR) was consistently better than when selecting based on standard WER. For example, on the dev set, using WER-S for oracle selection improved TER by 0.18 points and BLEU by 0.12 points compared to WER.
  • Preliminary ASR Tuning: A preliminary experiment tuning the ASR language model weight using WER-E showed small gains in TER and BLEU for the SLT system, suggesting potential for direct ASR optimization.

Limitations and Future Work

  • Limited ASR Tuning: The authors acknowledge that their preliminary ASR tuning experiment was limited to optimizing only the language model weight. They suggest that optimizing a larger set of ASR meta-parameters using their new metrics could yield more significant improvements.
  • Translation Evaluation Metrics: The paper notes that translation evaluation metrics themselves are imperfect, which can affect the perceived gains from ASR optimization.
  • Future Work: The authors propose further research into optimizing ASR systems using their novel metrics and exploring their applicability to other downstream tasks. They also make their code and corpora publicly available for reproducible research.

Summary updated

Abstract

This paper investigates the evaluation of ASR in spoken language translation context. More precisely, we propose a simple extension of WER metric in order to penalize differently substitution errors according to their context using word embeddings. For instance, the proposed metric should catch near matches (mainly morphological variants) and penalize less this kind of error which has a more limited impact on translation performance. Our experiments show that the correlation of the new proposed metric with SLT performance is better than the one of WER. Oracle experiments are also conducted and show the ability of our metric to find better hypotheses (to be translated) in the ASR N-best. Finally, a preliminary experiment where ASR tuning is based on our new metric shows encouraging results. For reproductible experiments, the code allowing to call our modified WER and the corpora used are made available to the research community.
BibTeX
@misc{le2016better,
  author = {Ngoc-Tien Le and Christophe Servan and François Portet and Laurent Besacier},
  title = {{Better Evaluation of ASR in Speech Translation Context Using Word Embeddings}},
  year = {2016},
  doi = {10.21437/interspeech.2016-464},
}
Figure