Introduction
In many industrial and scientific workflows, machine learning models are deployed in a "predict-then-optimize" pipeline. For instance, a healthcare model might predict patient recovery rates to decide on a treatment, or a news platform might predict click-through rates (CTR) to rank articles. Traditionally, these models are trained to minimize prediction error, such as Mean Squared Error (MSE). However, high accuracy does not always translate to high-quality decisions.
This paper addresses the fundamental gap between prediction and optimization in settings characterized by "black-box" reward functions and "bandit feedback"—where we only observe the outcome of the action taken, not the counterfactuals. The authors propose Empirical Soft Regret (ESR), a novel loss function designed to minimize the actual reward lost from suboptimal decisions, proving it to be asymptotically optimal.
The Empirical Soft Regret (ESR) Framework
The core challenge in optimizing for regret directly is that the regret function is typically non-differentiable, involving discrete choices (e.g., choosing Action A over Action B). ESR overcomes this by employing a differentiable surrogate.
Sigmoid Approximation and the Parameter $k$
ESR replaces the sharp "step function" of true regret with a smooth sigmoid function. This allows the model to be trained using standard gradient-based optimization techniques. A critical component of this approach is the parameter $k$, which controls the "softness" of the approximation. As $k$ increases, the sigmoid curve becomes steeper, more closely mimicking the true regret function.
Handling Missing Counterfactuals
Because real-world data often lacks counterfactual outcomes (we don't know what would have happened if we chose a different action), ESR utilizes a nearest-neighbor approach. For each data point, the algorithm identifies a similar context in the historical data where a different action was taken. The difference in rewards between these paired points serves as an estimate for the potential regret, allowing the model to learn from "what might have been."
Experimental Evaluation
The researchers benchmarked ESR against several state-of-the-art "metalearners" (S, T, R, and DR learners) across two diverse domains: healthcare and digital recommendations.
Healthcare: IHDP Dataset
Using the Infant Health and Development Program (IHDP) dataset—a semi-synthetic benchmark where true counterfactuals are known—ESR demonstrated a significant reduction in regret. While standard learners like the T-learner achieved a regret range of [0.70, 0.97], ESR achieved a much lower range of [0.35, 0.43].
News Recommendation: Yahoo! R6A
On the Yahoo! News Recommender dataset, which contains over 45 million visits, ESR-trained neural networks consistently outperformed MSE-based and other benchmark methods. In a 10-day trial, ESR achieved a higher mean CTR every single day, proving its efficacy in large-scale, real-world environments.
Key Insights: The Distraction Hypothesis
A major theoretical contribution of the paper is the "Distraction Hypothesis." The authors argue that standard MSE training can be "distracted" by trying to fit the baseline reward (the "level") of different contexts. If a model has limited capacity, it may prioritize fitting these large baseline variations while ignoring the smaller differences between actions (the "slope") that actually determine the best decision. By focusing directly on regret, ESR forces the model to prioritize the features that distinguish one action from another, leading to better decision-making even when the model cannot perfectly predict the absolute reward.
Conclusion and Future Directions
ESR provides a theoretically grounded and practically robust framework for decision-focused learning. It achieves asymptotic optimality, meaning that with sufficient data, the model's decisions become as good as possible within its class. Future work aims to extend this framework to multi-action and continuous action spaces, as well as exploring its potential for fine-tuning Large Language Models (LLMs) using human preference data (RLHF).
Summary updated