MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation

Dominik Macháček, Ondřej Bojar, Raj DabreACL 2023

TL;DR

This paper investigates whether standard offline machine translation (MT) evaluation metrics can reliably assess the quality of simultaneous speech translation (SST). The authors found that metrics like COMET, BERTSCORE, and chrF2 show strong correlations with human judgments of SST, suggesting they can serve as effective proxies for costly human evaluations.

What Problem Does This Paper Address?

Evaluating simultaneous speech translation (SST) is challenging. Unlike traditional machine translation (MT) where the entire source text is available, SST involves real-time translation of spoken language. This means translators have limited time to process input and produce output, and listeners have a different cognitive load than when reading text. While offline MT metrics like BLEU, chrF2, BERTSCORE, and COMET are widely used for text-to-text MT, their effectiveness for SST, especially when compared to human ratings of real-time translation, has been unclear. This paper aims to bridge this gap by systematically analyzing the correlation between these established MT metrics and human evaluations of SST.

Methods

The researchers leveraged the English-to-German Simultaneous Speech Translation task from IWSLT 2022. They collected human ratings using a "Continuous Rating" (CR) method, where evaluators continuously score the translation quality on a scale of 1 to 4 as they experience it in a simulated real-time event. This CR method is analogous to Direct Assessment in text-based MT.

They then calculated correlations between these human ratings and four popular offline MT metrics:

  • BLEU: A classic n-gram overlap metric.
  • chrF2: A character n-gram overlap metric.
  • BERTSCORE: A metric based on contextual embedding similarity from BERT.
  • COMET: A neural metric trained to predict human judgments.

The study also explored different reference types (translation vs. interpreting) and alignment methods (single sequence vs. multi-word error rate segmenter) to see how they impact metric performance.

Datasets

The primary dataset used for evaluation was the English-to-German Simultaneous Speech Translation task data from IWSLT 2022. This dataset consists of two subsets:

  • Common subset: TED talks from native speakers.
  • Non-Native subset: Mock business presentations from European high school students.

The human evaluation data was collected as part of this task, comprising ratings for five competing SST systems across three latency regimes, plus a human interpreter's output.

Key Insights and Ideas

  • Offline metrics are surprisingly good for SST: The most significant finding is that established offline MT metrics can indeed correlate well with human judgments of simultaneous speech translation. This is a crucial insight for the field, as it suggests a path towards more automated and scalable evaluation.
  • COMET shines: COMET emerged as the top-performing metric, showing the strongest correlation with human ratings. This aligns with its design goal of mimicking human evaluation.
  • Translation reference is better than interpreting: When evaluating SST, using a human translation as the reference yielded significantly higher correlations with MT metrics compared to using a human interpreter's output. This suggests that current metrics are better suited for evaluating translations that aim to be faithful to the source text, rather than those that prioritize brevity and fluency like simultaneous interpreting.
  • Test set size matters: While correlations were generally strong, they were even higher when considering larger test sets (like the entire test set) rather than individual short documents. This is consistent with how BLEU is traditionally used.
  • Human evaluators are reliable: The study also reinforced previous findings that bilingual evaluators using the CR method are reliable for SST assessment.

Key Results

  • Strong correlations observed: The study found strong correlations (coefficients generally above 0.6) between human ratings (CR) and the MT metrics BLEU, chrF2, BERTSCORE, and COMET at the document level for both the Common and Non-Native subsets.
  • COMET leads: COMET showed the highest correlation, followed by BERTSCORE and chrF2. BLEU also correlated well, especially on larger test sets.
  • Translation reference preferred: Metrics correlated significantly higher with CR when using a translation reference compared to an interpreting reference.
  • Single sequence alignment for neural metrics: For neural metrics like COMET and BERTSCORE, using a single sequence alignment method (concatenating all sentences) performed better than more complex alignment methods when dealing with unaligned references.

Limitations and Future Work

  • Limited scope: The analysis was restricted to one language pair (English-German) and a specific set of SST systems trained on standard parallel texts. The authors acknowledge that future systems with different approaches (e.g., those incorporating summarization, document context, or mimicking interpretation more closely) might show different correlations.
  • Single human interpreter: The study used only one human interpreter's output as a reference. A more in-depth study with multiple human interpretations would be beneficial to further assess the translation vs. interpreting reference question.
  • Future work: The authors suggest that future research should explore these metrics on a wider range of language pairs, system architectures, and evaluation setups. They also highlight the need for further investigation into how different alignment strategies impact metric performance, especially for non-neural metrics.

Summary updated

BibTeX
@inproceedings{machacek2023mt,
  author = {Dominik Macháček and Ondřej Bojar and Raj Dabre},
  title = {{MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure