Can Neural Machine Translation be Improved with User Feedback?

Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, Stefan Riezler

Abstract

We present the first real-world application of methods for improving neural machine translation (NMT) with human reinforcement, based on explicit and implicit user feedback collected on the eBay e-commerce platform. Previous work has been confined to simulation experiments, whereas in this paper we work with real logged feedback for offline bandit learning of NMT parameters. We conduct a thorough analysis of the available explicit user judgments—five-star ratings of translation quality—and show that they are not reliable enough to yield significant improvements in bandit learning. In contrast, we successfully utilize implicit task-based feedback collected in a cross-lingual search task to improve task-specific and machine translation quality metrics.

Introduction

Standard Neural Machine Translation (NMT) models typically rely on massive parallel corpora—pairs of sentences manually translated by experts. While effective, this data is expensive and slow to acquire. Large-scale e-commerce platforms like eBay generate a vast amount of user feedback daily, which could theoretically serve as a "gold mine" for model improvement. However, leveraging this data presents significant challenges: feedback is often "bandit" (only available for the shown translation), learning must happen offline to avoid poor user experiences, and non-expert feedback is notoriously noisy.

Methods

The study explores two primary feedback mechanisms for English-to-Spanish translations:

  1. Explicit Feedback: Users provide 1-5 star ratings on translated product titles.
  2. Implicit Feedback: Logged interactions from cross-lingual searches. A user click on a translated product title following a search query serves as a proxy for translation adequacy.
The eBay interface for collecting explicit user feedback via star ratings.

The researchers applied several Offline Bandit Learning techniques, including Bandit-to-Supervised (MLE), Minimum Risk Training (MRT), and counterfactual objectives like Deterministic Propensity Matching (DPM). A key innovation was the Word-based Reward (W-MIX), which calculates rewards at the word level by matching translated title terms to the user's original search query.

The Star Rating Fallacy

A critical finding of the research is the unreliability of explicit feedback. An expert re-evaluation of 1,000 titles revealed a low inter-annotator agreement ($\kappa = 0.12$) and a lack of correlation between user stars and expert quality judgments. Users frequently rated the product itself or its price rather than the linguistic quality of the translation.

Comparison of user ratings vs. expert judgments, illustrating the lack of correlation and high noise in explicit feedback.

Results and Discussion

In contrast to explicit ratings, implicit feedback proved highly effective. By grounding the reward in a specific task (search), the model received a functional signal of translation success. Using the W-MIX objective, the BLEU score improved from 28.38 to 34.52, a substantial 6-point gain achieved without new professional translations. Furthermore, "Query Recall"—the accuracy of translating search terms—rose from 45.96% to 68.12%.

These results suggest that while direct human ratings are often too noisy for training, implicit task-based signals provide a robust path for continuous NMT improvement in production environments. Future work will explore applying these techniques to longer-form text and addressing reward sparsity.

Summary updated

Abstract

We present the first real-world application of methods for improving neural machine translation (NMT) with human reinforcement, based on explicit and implicit user feedback collected on the eBay e-commerce platform. Previous work has been confined to simulation experiments, whereas in this paper we work with real logged feedback for offline bandit learning of NMT parameters. We conduct a thorough analysis of the available explicit user judgments---five-star ratings of translation quality---and show that they are not reliable enough to yield significant improvements in bandit learning. In contrast, we successfully utilize implicit task-based feedback collected in a cross-lingual search task to improve task-specific and machine translation quality metrics.
BibTeX
@inproceedings{kreutzer2018can,
  author = {Julia Kreutzer and Shahram Khadivi and Evgeny Matusov and Stefan Riezler},
  title = {{Can Neural Machine Translation be Improved with User Feedback?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=H1VrcXZ_-r},
}
Figure