Distributionally Robust Policy Gradient for Offline Contextual Bandits

Zhouhao Yang, Yihong Guo, Pan Xu, Anqi Liu, Animashree AnandkumarAISTATS 2023 · OpenReview

Abstract

Learning an optimal policy from offline data is notoriously challenging, as it requires evaluating the learning policy using data pre-collected from a static logging policy. We study the policy optimization problem in offline contextual bandits using policy gradient methods. We employ a distributionally robust policy gradient method, DROPO, to account for the distributional shift between the static logging policy and the learning policy. Our approach conservatively estimates the conditional reward distribution and updates the policy accordingly. We show that our algorithm converges to a stationary point with a rate of $O(1/T)$, where $T$ is the number of time steps. We conduct experiments on real-world datasets under various scenarios of logging policies to compare our proposed algorithm with baseline methods. We also propose a variant, DROPO-exp, to further improve performance when a limited amount of online interaction is allowed. Our results demonstrate the effectiveness and robustness of the proposed algorithms, especially under heavily biased offline data.

Introduction

In contextual bandits—the engine behind recommendation systems and clinical trials—an agent learns to pick the best action based on a given context. While "online" learning (learning by doing) is standard, it is often too risky or expensive in critical domains. For instance, one cannot explore random medical treatments on patients or risk crashing a robot to gather data.

This necessitates offline learning, where we train a policy using a static "logging" dataset. The core hurdle here is distributional shift: the logging policy that collected the data likely behaves very differently from the optimal policy we are trying to learn. Standard solutions like Inverse Propensity Scoring (IPS) suffer from massive variance, while Direct Methods (DM) often suffer from high bias. DROPO aims to find a robust, computationally efficient middle ground.

Methodology

The DROPO Framework

DROPO (Distributionally Robust Offline Policy Optimization) addresses distributional shift through two main innovations:

  1. Covariate Shift Perspective: Instead of assuming the entire environment might change, DROPO assumes the relationship between (context, action) and the reward remains stable, but the frequency of those context-action pairs differs between the logging data and the target policy.
  2. Robust Reward Estimation: Instead of a simple point estimate for rewards, DROPO solves a min-max optimization problem. It seeks a reward distribution that performs well even under the "worst-case" data-generating distribution consistent with the logging data.
  3. Closed-Form Solution: Crucially, the authors derive a closed-form analytical solution for this robust reward estimator (assuming a Gaussian prior). This avoids the heavy computational cost usually associated with distributionally robust optimization, allowing it to be plugged directly into standard policy gradient updates.

DROPO-exp

The authors also introduce DROPO-exp, a hybrid variant that allows for a small amount of online interaction (e.g., 7-14% of the total data) to "refresh" the model. This significantly reduces the bias inherited from the offline dataset.

Experimental Results

The researchers evaluated their approach by converting standard classification datasets (Optdigits, MNIST, CIFAR-10, and Adult) into contextual bandit problems. To simulate the offline challenge, they generated logging datasets with varying levels of bias using Tweak-1 and Dirichlet distributions.

Regret comparison on Optdigits and MNIST datasets.

As shown in the figures above, DROPO (the red line) significantly outperformed baselines such as IPS, DM, and Doubly Robust (DR) in scenarios with large shifts. While other methods like SDRPL (purple) were overly conservative, DROPO maintained a lower and more stable regret across samples.

Regret comparison on the Adult dataset.

Key Insights

  • Density Ratio as a Regularizer: In traditional IPS, the ratio of the target policy to the logging policy ($π/β$) can explode if an action is rare, leading to instability. In DROPO’s robust formulation, the inverse ratio ($β/π$) appears in the variance calculation. If the logging data doesn't support an action, the model becomes more uncertain and defaults to a safe prior.
  • Theoretical Convergence: The algorithm is proven to converge to a stationary point at a rate of $O(1/T)$, with a statistical error term that shrinks as the dataset size $N$ increases ($O(1/\sqrt{N})$).

Conclusion

DROPO provides a stable and effective framework for offline policy optimization by reframing distributional shift as a covariate shift problem. Its ability to scale to large datasets via a closed-form solution makes it a practical choice for real-world applications. Future work may explore more sophisticated exploration strategies for the DROPO-exp variant, such as UCB or Thompson Sampling, to further enhance online data efficiency.

Summary updated

Abstract

Learning an optimal policy from offline data is notoriously challenging, which requires the evaluation of the learning policy using data pre-collected from a static logging policy. We study the policy optimization problem in offline contextual bandits using policy gradient methods. We employ a distributionally robust policy gradient method, DROPO, to account for the distributional shift between the static logging policy and the learning policy in policy gradient. Our approach conservatively estimates the conditional reward distributional and updates the policy accordingly. We show that our algorithm converges to a stationary point with rate $O(1/T)$, where $T$ is the number of time steps. We conduct experiments on real-world datasets under various scenarios of logging policies to compare our proposed algorithm with baseline methods in offline contextual bandits. We also propose a variant of our algorithm, DROPO-exp, to further improve the performance when a limited amount of online interaction is allowed. Our results demonstrate the effectiveness and robustness of the proposed algorithms, especially under heavily biased offline data.
BibTeX
@inproceedings{yang2023distributionally,
  author = {Zhouhao Yang and Yihong Guo and Pan Xu and Anqi Liu and Animashree Anandkumar},
  title = {{Distributionally Robust Policy Gradient for Offline Contextual Bandits}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=RHaKWT8jrM},
}
Figure