Adaptive Quality Estimation for Machine Translation

Marco Turchi, Antonios Anastasopoulos, José G. C. de Souza, Matteo NegriACL 2014 · OpenReview

Abstract

The automatic estimation of machine translation (MT) output quality is a hard task in which the selection of the appropriate algorithm and the most predictive features over reasonably sized training sets plays a crucial role. When moving from controlled lab evaluations to real-life scenarios the task becomes even harder. For current MT quality estimation (QE) systems, additional complexity comes from the difficulty to model user and domain changes. Indeed, the instability of the systems with respect to data coming from different distributions calls for adaptive solutions that react to new operating conditions. To tackle this issue we propose an online framework for adaptive QE that targets reactivity and robustness to user and domain changes. Contrastive experiments in different testing conditions involving user and domain changes demonstrate the effectiveness of our approach.

Introduction

Quality Estimation (QE) serves as a critical component in machine translation workflows, aiming to predict the quality of translated text without access to human reference translations. While laboratory evaluations have shown promise, real-world deployment reveals significant "brittleness" in static models. This instability stems from two primary factors: subjectivity and domain shift. Subjectivity refers to the varying standards of quality among different human translators, while domain shift occurs when a model trained on one type of text (e.g., legal) is applied to another (e.g., technical manuals). Traditional "batch" models, trained once on fixed datasets, cannot adapt to these dynamic conditions. This paper proposes an online learning framework designed to evolve through continuous interaction with professional translators.

Methodology

The core of the proposed approach is an online learning paradigm that replaces the static Train-then-Deploy workflow with a continuous feedback loop. The system architecture involves several integrated components:

  1. Feature Extraction: For every source-target pair, 17 standard features are extracted using an adapted version of the QuEst tool.
  2. Prediction: An online regressor predicts the Human-mediated Translation Edit Rate (HTER).
  3. Feedback and Update: After a human translator post-edits the sentence, the system calculates the actual HTER using TERCpp. The model then immediately updates its weights based on the prediction error.
Sequence diagram of the adaptive QE framework showing the interaction between the Quality Estimation System and the Online Learning Algorithm.

Two online algorithms were evaluated: OnlineSVR, which offers stability by maintaining support vectors, and Passive Aggressive (PA), which provides high reactivity by updating weights only when errors exceed a specific threshold.

Experimental Setup

The framework was evaluated using two distinct datasets:

  • WMT12 English-Spanish: A homogeneous benchmark dataset representing a controlled environment.
  • CAT Data (English-Italian): Collected via the MateCat tool, this dataset includes Legal and IT domains post-edited by eight different professionals. This setup allowed for testing scenarios with significant user style variations and domain shifts.

Results and Discussion

Experimental results demonstrate that adaptive models are highly effective in handling data instability. On homogeneous data (WMT12), the online approach maintained parity with batch models, ensuring no loss of accuracy in stable environments. However, the true strength of the adaptive framework emerged in real-world scenarios involving user and domain changes.

Performance comparison (MAE) of Batch, Adaptive, and Empty models across Legal and IT domains under various user change scenarios.

Key findings include:

  • Reactivity to Radical Changes: In cases of extreme user change (e.g., switching from a conservative to a radical editor), "empty" models—starting with no prior knowledge—often outperformed batch models trained on mismatched data.
  • Algorithm Performance: The PA algorithm proved superior in highly volatile conditions due to its reactivity, while OnlineSVR excelled in more consistent environments.
  • Error Reduction: In the Legal domain, adaptive models achieved up to a 10-point reduction in Mean Absolute Error (MAE) compared to static baselines when facing significant translator style shifts.

Conclusion

This research highlights the necessity of adaptive QE systems in professional translation environments. By integrating online learning directly into Computer-Assisted Translation (CAT) tools, QE models can profile individual translator behavior and adapt to new domains in real-time. Future work will focus on addressing feedback latency and exploring online feature selection to further refine model responsiveness.

Summary updated

Abstract

The automatic estimation of machine translation (MT) output quality is a hard task in which the selection of the appropriate algorithm and the most predictive features over reasonably sized training sets plays a crucial role. When moving from controlled lab evaluations to real-life scenarios the task becomes even harder. For current MT quality estimation (QE) systems, additional complexity comes from the difficulty to model user and domain changes. Indeed, the instability of the systems with respect to data coming from different distributions calls for adaptive solutions that react to new operating conditions. To tackle this issue we propose an online framework for adaptive QE that targets reactivity and robustness to user and domain changes. Contrastive experiments in different testing conditions involving user and domain changes demonstrate the effectiveness of our approach.
BibTeX
@inproceedings{turchi2014adaptive,
  author = {Marco Turchi and Antonios Anastasopoulos and José G. C. de Souza and Matteo Negri},
  title = {{Adaptive Quality Estimation for Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2014},
  url = {https://openreview.net/forum?id=HJE5soxO-H},
}
Figure