Abstract
We present the first real-world application of methods for improving neural machine translation (NMT) with human reinforcement, based on explicit and implicit user feedback collected on the eBay e-commerce platform. Previous work has been confined to simulation experiments, whereas in this paper we work with real logged feedback for offline bandit learning of NMT parameters. We conduct a thorough analysis of the available explicit user judgments—five-star ratings of translation quality—and show that they are not reliable enough to yield significant improvements in bandit learning. In contrast, we successfully utilize implicit task-based feedback collected in a cross-lingual search task to improve task-specific and machine translation quality metrics.
Introduction
Standard Neural Machine Translation (NMT) models are typically trained on massive parallel corpora—sentence pairs perfectly translated by humans. However, in production environments like e-commerce platforms, we have access to a different, more abundant signal: bandit feedback. This is a form of reinforcement where we only observe the reward for the specific translation shown to the user, without knowing the potential reward for alternative translations. This paper explores the transition from simulated bandit learning to real-world application using logged data from eBay.
The Noise in Explicit Feedback
One might assume that explicit user feedback, such as five-star ratings, would be the most direct way to improve a model. However, the authors found these judgments to be highly unreliable. Users often rate based on factors unrelated to translation quality, such as product availability or image clarity.
As shown in the data analysis, there is a significant discrepancy between user ratings and expert judgments. In some cases, a perfectly accurate translation receives a one-star rating from a user, likely due to dissatisfaction with the product itself. This lack of inter-annotator agreement makes explicit feedback a poor signal for reinforcement learning in NMT.
Leveraging Implicit Feedback and Task-Based Rewards
In contrast to noisy star ratings, implicit feedback—such as user clicks in a cross-lingual search task—provides a "gold mine" of information. The authors introduced a W-MIX (Word-level Rewards) objective. Instead of a single reward for an entire title, the model is rewarded based on how well the translation preserves the specific keywords from the user's original search query.
By focusing on Query Recall—the frequency with which search terms appear in the translated product title—the model learns to prioritize the information most critical to the user's intent. If a user searches for "candado" (lock) and clicks on a title containing "cerradura" (bolt/lock), the system recognizes a successful match regardless of literal string overlap.
Methodology
The study compared several counterfactual learning methods to handle the "logged" nature of the data:
- Bandit-to-Supervised Conversion: Filtering logs for high-reward examples to use in standard fine-tuning.
- Minimum Risk Training (MRT): Optimizing the model to maximize expected rewards.
- Counterfactual Learning (DPM & DC): Using Deterministic Propensity Matching and Doubly Controlled estimation to correct for the bias of the original logging policy, supported by a CNN-based reward estimator.
Key Results
The results demonstrate a clear advantage for implicit feedback over explicit ratings:
- Translation Quality: Using the W-MIX objective on implicit feedback, the model's performance jumped from 28.38 BLEU to 34.52 BLEU, a gain of over 6 points.
- Search Success: Task-specific metrics saw even greater gains, with Query Recall improving from 45.96% to 68.12%.
- Explicit Failure: Models trained on 5-star ratings failed to show significant improvements in BLEU or Translation Edit Rate (TER).
Conclusion
This research highlights that while explicit user feedback is often too noisy for direct optimization, implicit task-based feedback is a powerful resource for refining NMT models in production. By aligning translation rewards with user intent (search queries), models can significantly improve both linguistic quality and task-specific utility. Future work will focus on real-time model updates and addressing the biases inherent in reward estimation models.
Summary updated