Introduction
Quality Estimation (QE) is a critical task in machine translation (MT) that involves predicting the quality of a translated text without access to a human-authored reference. Traditionally, QE models have relied solely on textual features. However, text alone is often insufficient to resolve semantic ambiguities. For instance, the English word "shorts" can function as a noun (a garment) or an adjective (indicating length). In an e-commerce context, a machine translation system might incorrectly translate "Bermuda Shorts" using a French adjective for "short" (court) instead of the appropriate noun.
This paper introduces Multimodal QE, an approach that incorporates product images into the evaluation process. By providing visual grounding, the model can "see" the object being described, allowing it to identify translation errors that a text-only model would likely miss.
Methodology
The researchers investigated two primary architectural frameworks for integrating visual information into the QE pipeline.
Feature-based Approach (QuEst++)
The traditional QuEst++ framework utilizes 15 hand-crafted textual features. To incorporate visual data, the authors extracted 4,096-dimensional image vectors using a pre-trained ResNet-style Convolutional Neural Network (CNN). To prevent these high-dimensional vectors from overwhelming the textual features, Principal Component Analysis (PCA) was employed to compress the visual data into a lower-dimensional space (1–15 dimensions).
Neural-based Architectures
The study also explored modifications to Bi-directional Recurrent Neural Networks (BiRNN) and BERT-augmented BiRNNs. As shown in the architecture diagram below, the authors experimented with three distinct integration points for visual features:
- Embed: Fusing visual features immediately following the word embedding layer.
- Annot: Fusing features after the RNN has encoded the source and target sentences.
- Last: Fusing visual information just before the final output layer.
Furthermore, various fusion strategies were tested. The authors found that element-wise multiplication (mult) generally outperformed simple concatenation (conc), suggesting that multiplication allows visual context to more effectively "gate" or modulate the textual representations.
Experimental Setup
The experiments were conducted using data from the WMT’18 Shared Task 4, derived from the Amazon Reviews Dataset (Sports & Outdoors category). The dataset consists of English product titles and descriptions paired with French machine translations and corresponding product images. Quality was measured using Multidimensional Quality Metrics (MQM) scores based on human error annotations. The authors also processed this data to create a new sentence-level version of the dataset for more granular testing.
Results and Discussion
The inclusion of visual information yielded significant improvements across different model types:
- Sentence-Level Performance: The multimodal BiRNN (
+Vis-embed-mult2) achieved a Pearson’s $r$ correlation of 0.535, a notable increase over the monomodal baseline of 0.504. - Document-Level Performance: The feature-based QuEst++ model improved from 0.503 to 0.536 when utilizing just two principal components of the visual features.
- State-of-the-Art: The multimodal BERT-BiRNN variant reached a correlation of 0.602, setting a new benchmark for the task.
A key finding was that visual grounding is most effective when integrated early (at the embed stage). This allows the model to process text through a visually-informed lens from the start. While BERT-based models showed smaller relative gains compared to BiRNNs, the addition of vision still provided a competitive edge, suggesting that even high-capacity language models benefit from external visual context.
Conclusion and Future Work
This research demonstrates that multimodal information is a powerful asset for Machine Translation Quality Estimation, particularly in domains like e-commerce where visual context is readily available and semantically rich. Future work remains to address document-level BERT performance and the scalability of these models to more abstract domains where images may be less functional. Nevertheless, the success of visual grounding in resolving ambiguity marks a significant step forward for "blind" translation evaluation.
Summary updated