SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window

Vikas Raunak, Tom Kocmi, Matt PostNAACL 2024 · DOI

TL;DR

This paper introduces SLIDE, a novel reference-free evaluation metric for machine translation that leverages document context. By sliding a window across documents and feeding these chunks to existing quality estimation models, SLIDE significantly improves accuracy over sentence-level baselines, sometimes even matching reference-based metrics. This offers a more robust way to evaluate machine translation quality without needing human-written reference translations.

What Problem Does This Paper Address?

Evaluating machine translation (MT) quality is crucial, but current standard metrics often operate at the sentence level. This is problematic because many linguistic phenomena, like pronoun resolution or lexical choice, are inherently context-dependent and cannot be fully disambiguated from a single sentence alone. While reference-based metrics (which use human-written translations as a gold standard) can capture this context, they are expensive and time-consuming to create. Reference-free Quality Estimation (QE) metrics, which only use the source and system output, are more scalable but typically lag behind reference-based metrics in accuracy due to this lack of context. This paper aims to bridge this gap by exploring how to effectively use document-level source context to improve reference-free QE metrics.

Methods

The core innovation is SLIDE (SLIding Document Evaluator). It works by taking an existing, off-the-shelf quality estimation model (specifically, COMET-QE in their experiments) and applying it to chunks of sentences from a document, rather than just individual sentences.

Here's how it works:

  1. Sliding Window: A window of a specified size (w sentences) slides across a document.
  2. Chunking: Each window creates a "chunk" of sentences (source and system output).
  3. Contextualized Evaluation: These chunks are fed into the unmodified QE model.
  4. Accumulation: Scores from all chunks within a document are accumulated (averaged) to produce a document-level score.
  5. Strides: The window can advance by a specified number of sentences (s) at each step, allowing for both overlapping and non-overlapping chunks.

The authors also explore variations, including how to handle "partial windows" (documents shorter than the window size or where the window doesn't perfectly divide the document length) and how to weight chunks based on their sentence length.

Datasets

The experiments were conducted using data from the WMT22 metric evaluation task. This task provided human rankings of system outputs for various language pairs, specifically using MQM (Multi-dimensional Quality Metrics) and DA+SQM (Direct Assessment + Quality Score) annotations. The paper does not introduce new datasets but leverages existing benchmarks for evaluation.

Key Insights and Ideas

  • Context is King (Even for Reference-Free): The most significant insight is that providing document-level source context to reference-free QE models can dramatically improve their performance, often closing the gap with reference-based metrics. This suggests that source context can indeed substitute for a human reference in disambiguating ambiguities.
  • Simple Extension, Big Gains: SLIDE is a straightforward modification that requires no changes to the underlying QE model's architecture. This makes it a highly practical and "free" improvement if document boundary annotations are available.
  • QE22 Benefits Most: The COMET-QE model with cross-attention (QE22) showed substantial improvements with SLIDE, suggesting its architecture is better equipped to leverage fine-grained source context compared to older models like QE20.
  • Window Size Matters: Performance generally increases with more context, but there's a point of diminishing returns where too much context can lead to a slight decrease in accuracy.
  • Partial Documents Can Be Handled: While initially ignored, experiments show that incorporating partial documents (with appropriate weighting) can still yield improvements, though it can sometimes lead to a drop in accuracy compared to the "full windows only" approach.

Key Results

  • Pairwise System Accuracy: SLIDE significantly outperforms its sentence-level baseline. For the QE22 model, adding context (e.g., window size 8, stride 8) improved pairwise system accuracy by up to 3.3 points over the no-context baseline on the WMT22 MQM task.
  • Competitive with Reference-Based: In some configurations, SLIDE (e.g., window 6, stride 6) achieved pairwise system accuracy comparable to or even exceeding the reference-based COMET22 model (0.843 vs. 0.839 for MQM).
  • General Improvement: Even the "worst" SLIDE variants showed improvement over the sentence-level baseline.
  • Handling Partial Documents: While incorporating partial documents without weighting led to a drop in accuracy, weighting them based on sentence length helped mitigate this issue and still showed improvements over the context-less baseline.

Limitations and Future Work

  • Document Boundary Requirement: The primary limitation is the need for document boundary annotations. SLIDE cannot be applied in scenarios where this information is unavailable.
  • Token Length Limits: The underlying models (like InfoXLM) have maximum token length limits, which can lead to chunk truncation. While the paper explores a token-based windowing approach to address this, it did not yield consistent gains in their experiments.
  • Future Work: The authors suggest further evaluation with document-context-aware MT engines, speculating that SLIDE could help discriminate their outputs more effectively. They also mention exploring more sophisticated ways to handle partial documents and token length constraints.

Summary updated

BibTeX
@inproceedings{raunak2024slide,
  author = {Vikas Raunak and Tom Kocmi and Matt Post},
  title = {{SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.18},
}
Figure