Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translation

Carolin Lawrence, Artem Sokolov, Stefan Riezler

Abstract

The goal of counterfactual learning for statistical machine translation (SMT) is to optimize a target SMT system from logged data that consist of user feedback to translations that were predicted by another, historic SMT system. A challenge arises by the fact that risk-averse commercial SMT systems deterministically log the most probable translation. The lack of sufficient exploration of the SMT output space seemingly contradicts the theoretical requirements for counterfactual learning. We show that counterfactual learning from deterministic bandit logs is possible nevertheless by smoothing out deterministic components in learning. This can be achieved by additive and multiplicative control variates that avoid degenerate behavior in empirical risk minimization. Our simulation experiments show improvements of up to 2 BLEU points by counterfactual learning from deterministic bandit feedback.

What Problem Does This Paper Address?

In machine learning, "counterfactual learning" (or off-policy learning) involves training a new model (the target policy) using data collected by a previous model (the logging policy). This is highly valuable for production systems like Google Translate or Bing Microsoft Translator, where developers want to learn from millions of user interactions without deploying experimental, potentially low-quality models to live users.

The counterfactual learning workflow: a logging system generates predictions, user rewards are recorded in a log, and a new system is trained offline using that log.

Standard counterfactual learning relies on Inverse Propensity Scoring (IPS), which requires the logging system to be stochastic—meaning it must "explore" by occasionally choosing less-likely translations. However, commercial SMT systems are typically deterministic, always showing the most probable translation to maintain quality. Theoretically, this lack of exploration makes counterfactual learning difficult because there is no data on alternative paths the system could have taken. This paper explores how to bridge the gap between this theoretical limitation and the practical reality of deterministic logs.

Methods

The authors propose moving beyond simple Empirical Risk Minimization (ERM) by introducing "smoothing" techniques derived from variance reduction literature. These techniques are designed to handle the deterministic nature of the logging policy by applying control variates to the gradient estimators.

Gradient estimators for the proposed smoothing techniques: Deterministic Propensity Matching (DPM), Multiplicative Control Variates (DPM+R), and the Doubly Controlled (DC) estimator.
  • Deterministic Propensity Matching (DPM): A baseline objective that attempts to match the deterministic outputs of the logging system.
  • Multiplicative Control Variates (DPM+R): This uses reweighting to prevent the model from "cheating." Without this, a model might minimize risk simply by assigning zero probability to training samples where it observed a low reward. Reweighting forces the model to balance probabilities across the logged data.
  • Additive Control Variates (Doubly Controlled / $\hat{c}$DC): This incorporates a "reward estimator"—a separate regression model trained to predict the quality (e.g., BLEU score) of any given translation. This allows the learner to estimate potential rewards for translations that were not logged, effectively smoothing the deterministic spikes in the data.
  • Optimal Scaling ($\hat{c}$): The authors derive an optimal scalar to balance logged rewards against estimated rewards, minimizing overall variance during training.

Datasets and Experimental Setup

The study utilizes two standard SMT domain-adaptation tasks to evaluate these methods:

  1. TED DE-EN: German-to-English translation using transcribed TED talks (122k training sentences).
  2. News FR-EN: French-to-English translation using News Commentary data (30k training sentences).

As this is a simulation study, "user feedback" is generated by calculating sentence-level BLEU scores against hidden reference translations. This setup allows for a direct comparison against an "Oracle" system that has access to full information.

Key Insights and Results

  • Implicit Exploration: The authors argue that even if a system is deterministic at the sentence level, the variety of inputs (different sentences containing similar phrases in different contexts) provides a form of "implicit exploration" over the underlying phrase-table.
  • Preventing Degeneracy: Simple learning on deterministic logs often leads to "degenerate behavior," where the model ignores training samples to lower perceived risk. Control variates act as a regularizer, forcing the model to consider the entire output space.
  • BLEU Gains: The best-performing method ($\hat{c}$DC) improved the baseline out-of-domain system by 2.02 BLEU points on the TED task and 1.13 BLEU points on the News task. On the TED data, this method closed the gap between the baseline and the full-information Oracle system by approximately 50%.
  • Viability of Deterministic Logs: Surprisingly, learning from deterministic logs performed nearly as well as learning from stochastic logs, suggesting that the risk of logging deterministically in production may be lower than previously assumed for SMT.

Limitations and Future Work

While promising, the study relies on simulated BLEU feedback. Real-world user feedback (such as clicks) is significantly noisier and sparser, which may require even more robust reward estimators. Additionally, while these techniques are theoretically model-agnostic, these experiments were conducted on linear SMT models; applying them to modern Neural Machine Translation (NMT) architectures is a critical next step. Finally, the multiplicative control variates (DPM+R) require large mini-batches for stability, which could pose challenges in memory-constrained deep learning environments.

Summary updated

Abstract

The goal of counterfactual learning for statistical machine translation (SMT) is to optimize a target SMT system from logged data that consist of user feedback to translations that were predicted by another, historic SMT system. A challenge arises by the fact that risk-averse commercial SMT systems deterministically log the most probable translation. The lack of sufficient exploration of the SMT output space seemingly contradicts the theoretical requirements for counterfactual learning. We show that counterfactual learning from deterministic bandit logs is possible nevertheless by smoothing out deterministic components in learning. This can be achieved by additive and multiplicative control variates that avoid degenerate behavior in empirical risk minimization. Our simulation experiments show improvements of up to 2 BLEU points by counterfactual learning from deterministic bandit feedback.
BibTeX
@inproceedings{lawrence2017counterfactual,
  author = {Carolin Lawrence and Artem Sokolov and Stefan Riezler},
  title = {{Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=BJ4EwMzO-S},
}
Figure