TL;DR
This paper introduces RUSE, a novel metric for evaluating machine translation (MT) quality that leverages universal sentence embeddings. By moving beyond traditional N-gram based methods, RUSE captures richer semantic information, achieving state-of-the-art performance on WMT evaluation tasks and demonstrating the power of pre-trained embeddings for MT evaluation.
What Problem Does This Paper Address?
Evaluating the quality of machine translation (MT) is crucial for developing and deploying better MT systems. Current automatic MT evaluation metrics, like BLEU, primarily rely on comparing N-grams (sequences of words or characters) between a machine-generated translation (hypothesis) and a human-written reference. While effective to some extent, this approach has limitations: it can only capture local lexical overlap and struggles to grasp the deeper semantic meaning of sentences. This means a translation might be semantically correct but receive a low score if it uses different wording than the reference. The paper aims to address this by developing a metric that can better understand the meaning of sentences, thus providing a more accurate assessment of translation quality.
Methods
The core of RUSE is a regression model that predicts an MT quality score based on sentence embeddings. Here's a breakdown of their approach:
- Universal Sentence Embeddings: Instead of training embeddings from scratch (which is data-intensive and difficult for MT evaluation), RUSE utilizes pre-trained "universal" sentence embeddings. These embeddings are trained on massive datasets for tasks like natural language inference or general text understanding, allowing them to capture rich semantic information. The paper specifically uses three types:
- InferSent: A supervised model trained on Natural Language Inference datasets, making it sensitive to meaning differences.
- Quick-Thought: An unsupervised model trained on consecutive sentences, learning to distinguish contextually relevant sentences.
- Universal Sentence Encoder (USE): A model trained on diverse web data for various downstream tasks, demonstrating strong performance across document classification and semantic similarity.
- Feature Engineering: For a given MT hypothesis and its reference translation, their respective sentence embeddings (vectors) are extracted. To capture the relationship between these embeddings, they combine them using three methods:
- Concatenation: Simply joining the two embedding vectors.
- Element-wise product: Multiplying corresponding elements of the two vectors.
-
Absolute element-wise difference: Calculating the absolute difference between corresponding elements. These combined features (resulting in a 4d-dimensional vector) are then fed into a regression model.
-
Regression Model: They experiment with two regression models:
- Multi-layer Perceptron (MLP): A neural network that learns to map the combined features to a quality score.
- Support Vector Regression (SVR): A machine learning model that finds a regression function. The MLP model, particularly when using a combination of all three sentence embeddings, ultimately achieved better results and was chosen for submission.
Datasets
The experiments were conducted using datasets from the Workshops on Statistical Machine Translation (WMT) shared tasks. Specifically, they used:
- WMT15, WMT16, and WMT17 datasets: These provided segment-level human evaluation data (Direct Assessment scores) for various to-English language pairs.
- WMT16 and WMT17 datasets: These also provided system-level evaluation data.
The paper notes that the training data was split into training and development sets (9:1 ratio), and the final model was trained on a combined dataset from WMT15, WMT16, and WMT17 for submission to WMT18.
Key Insights and Ideas
- Sentence Embeddings are Powerful for MT Evaluation: The most significant insight is that universal sentence embeddings, trained on large-scale data for other NLP tasks, can effectively capture semantic nuances crucial for MT evaluation, outperforming traditional N-gram based metrics.
- Moving Beyond Lexical Overlap: RUSE demonstrates that focusing on semantic similarity rather than just word overlap leads to more accurate MT quality assessment.
- Pre-trained Embeddings are Key: The ability to leverage pre-trained embeddings significantly reduces the need for large, MT-specific human evaluation datasets for training, making the approach more practical.
- MLP Outperforms SVR for this Task: While both regression models were tested, the MLP architecture proved more effective and efficient for RUSE.
- Quick-Thought Embeddings Show Promise: Even using only Quick-Thought embeddings (an unsupervised method) led to state-of-the-art results, suggesting its potential for languages where supervised data is scarce.
Key Results
RUSE achieved state-of-the-art performance in both segment-level and system-level MT evaluation tasks on the WMT16 and WMT17 datasets for to-English language pairs.
- Segment-Level: RUSE (using MLP with a combination of InferSent, Quick-Thought, and USE embeddings) consistently achieved the highest Pearson correlation coefficients with human judgments across multiple language pairs, outperforming existing metrics like SentBLEU, chrF++, MEANT 2.0, and Blend. For example, in WMT17, RUSE (MLP) achieved an average correlation of 0.685, significantly higher than many baselines.
- System-Level: Similarly, RUSE (MLP) demonstrated superior performance on system-level evaluations, with average correlations reaching 0.974 on WMT16 and 0.975 on WMT17, surpassing established metrics like BLEU, BEER, and ReVal.
Limitations and Future Work
- Language Pair Specificity: The current experiments focus on to-English language pairs. While Quick-Thought's unsupervised nature suggests potential for other languages, further investigation is needed.
- Reliance on Pre-trained Embeddings: The performance of RUSE is dependent on the quality and availability of universal sentence embeddings.
- Future Work: The authors suggest exploring RUSE for other language pairs and investigating the effectiveness of different sentence embedding models and regression architectures. They also highlight the potential for RUSE to be effective in MTE for languages other than to-English, given the unsupervised nature of some of the embeddings used.
Summary updated