Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation

Carolin Lawrence, Artem Sokolov, Stefan Riezler

Abstract

The goal of counterfactual learning for statistical machine translation (SMT) is to optimize a target SMT system from logged data that consist of user feedback to translations that were predicted by another, historic SMT system. A challenge arises by the fact that risk-averse commercial SMT systems deterministically log the most probable translation. The lack of sufficient exploration of the SMT output space seemingly contradicts the theoretical requirements for counterfactual learning. We show that counterfactual learning from deterministic bandit logs is possible nevertheless by smoothing out deterministic components in learning. This can be achieved by additive and multiplicative control variates that avoid degenerate behavior in empirical risk minimization. Our simulation experiments show improvements of up to 2 BLEU points by counterfactual learning from deterministic bandit feedback.

Introduction

In real-world applications of Statistical Machine Translation (SMT), improving a model often relies on historical logs. Ideally, these logs would be generated through "stochastic" processes where the system explores various translation options to gather diverse feedback. However, most commercial systems are risk-averse and log only their single best guess. This "deterministic logging" creates a significant hurdle for standard counterfactual learning techniques like Inverse Propensity Scoring (IPS), which typically require non-zero probabilities for all possible actions to avoid degenerate behavior.

The counterfactual learning framework: training a new system from logs generated by a historic logging system.

The Challenge of Deterministic Logs

In a standard bandit feedback loop, a system makes a prediction, receives a reward from a user, and updates itself. In the counterfactual setting, we aim to train a new system using logs from an old one. When the old system is deterministic, the propensity (probability) of the logged action is always 1, and the propensity of all other actions is 0. Mathematically, this lack of exploration makes it difficult to estimate how a different model would have performed, often leading the learner to over-fit to specific training samples rather than generalizing to new ones.

Proposed Methods: Smoothing via Control Variates

To overcome the limitations of deterministic logs, the authors propose using control variates to stabilize and "smooth" the learning objective. These techniques force the optimizer to consider the broader output space, compensating for the lack of explicit exploration in the logs.

Gradient formulations for the proposed reweighted (DPM+R) and doubly controlled (DC) learning objectives.
  1. Reweighted Deterministic Propensity Matching (DPM+R): This method employs a multiplicative control variate, effectively performing weighted importance sampling. It reduces estimator variance by reweighting model probabilities across the entire training set.
  2. Doubly Controlled Empirical Risk Minimization ($\hat{c}$DC): This more sophisticated approach uses an additive control variate based on a "doubly robust" estimator. A separate regression model (e.g., a Random Forest) is trained to predict rewards (BLEU scores) for any translation. This allows the learner to "peek" at the potential rewards of unlogged actions. The parameter $\hat{c}$ is optimized to minimize the estimator's variance.

Experimental Setup

The methods were evaluated on two domain adaptation tasks using simulated bandit feedback:

  • TED DE-EN: Adapting a general-domain German-to-English system to the domain of TED Talks.
  • News FR-EN: Adapting a French-to-English system to News Commentary.

To create the "ground truth" for evaluation, the researchers used existing parallel corpora. They generated translations using an old system and calculated sentence-level BLEU scores against reference translations to serve as the reward signal.

Key Results and Insights

The experiments demonstrated that counterfactual learning is indeed possible from deterministic logs:

  • Significant Performance Gains: The $\hat{c}$DC method achieved an improvement of 2.02 BLEU points over the baseline on the TED DE-EN task.
  • Closing the Stochastic Gap: On the TED dataset, the best deterministic method performed nearly as well as a system trained on stochastic logs (logs that included explicit exploration).
  • Implicit Exploration: A key insight is that while the output for a single sentence is deterministic, the diversity of inputs across a large corpus provides "implicit exploration." The model effectively sees a wide range of actions over time because the same words and phrases appear in many different contexts.

Conclusion

This work challenges the prevailing belief that counterfactual learning requires stochastic logging. By applying mathematical "smoothing" through control variates, deterministic logs—which are a goldmine of data in commercial settings—can be successfully leveraged for model improvement. Future research will need to address the transition from simulated BLEU rewards to noisier, real-world user feedback.

Summary updated

Abstract

The goal of counterfactual learning for statistical machine translation (SMT) is to optimize a target SMT system from logged data that consist of user feedback to translations that were predicted by another, historic SMT system. A challenge arises by the fact that risk-averse commercial SMT systems deterministically log the most probable translation. The lack of sufficient exploration of the SMT output space seemingly contradicts the theoretical requirements for counterfactual learning. We show that counterfactual learning from deterministic bandit logs is possible nevertheless by smoothing out deterministic components in learning. This can be achieved by additive and multiplicative control variates that avoid degenerate behavior in empirical risk minimization. Our simulation experiments show improvements of up to 2 BLEU points by counterfactual learning from deterministic bandit feedback.
BibTeX
@inproceedings{lawrence2017counterfactual,
  author = {Carolin Lawrence and Artem Sokolov and Stefan Riezler},
  title = {{Counterfactual Learning from Bandit Feedback under Deterministic Logging : A Case Study in Statistical Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=BJ4EwMzO-S},
}
Figure