Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approach

Yunzhe Zhou, Zhengling Qi, Chengchun Shi, Lexin Li

Abstract

In this article, we propose a novel pessimism-based Bayesian learning method for optimal dynamic treatment regimes in the offline setting. When the coverage condition does not hold, which is common for offline data, the existing solutions would produce sub-optimal policies. The pessimism principle addresses this issue by discouraging recommendation of actions that are less explored conditioning on the state. However, nearly all pessimism-based methods rely on a key hyper-parameter that quantifies the degree of pessimism, and the performance of the methods can be highly sensitive to the choice of this parameter. We propose to integrate the pessimism principle with Thompson sampling and Bayesian machine learning for optimizing the degree of pessimism. We derive a credible set whose boundary uniformly lower bounds the optimal Q-function, and thus we do not require additional tuning of the degree of pessimism. We develop a general Bayesian learning method that works with a range of models, from Bayesian linear basis model to Bayesian neural network model. We develop the computational algorithm based on variational inference, which is highly efficient and scalable. We establish the theoretical guarantees of the proposed method, and show empirically that it outperforms the existing state-of-the-art solutions through both simulations and a real data example.

Introduction

Learning Dynamic Treatment Regimes (DTRs) from offline historical data is a critical task in precision medicine. However, offline datasets often suffer from a lack of "coverage," where certain patient states and treatment combinations are poorly represented. Standard reinforcement learning algorithms can be over-optimistic in these scenarios, mistakenly recommending treatments based on statistical noise. To address this, researchers use the "pessimism principle" to penalize actions with high uncertainty. A significant drawback of existing pessimistic methods is their reliance on a manually tuned hyperparameter ($c$) that dictates the degree of skepticism. If $c$ is poorly chosen, the resulting policy can be either dangerously reckless or overly conservative.

Methodology

We present Pessimism-based Bayesian Learning (PBL), a framework that automates the calibration of pessimism using Bayesian uncertainty. PBL employs Bayesian Machine Learning models—specifically Bayesian Linear Basis Models (BLBM) and Bayesian Neural Networks (BNN)—to estimate the Q-function.

Instead of relying on point estimates, PBL derives uniform credible sets for the Q-function. By optimizing the lower boundary of these sets, the algorithm naturally adopts a conservative stance in regions of the state-action space where data is sparse. This approach effectively asks, "What is the statistically plausible worst-case scenario for this treatment?" and selects the action with the best worst-case outcome. To maintain computational efficiency and scalability, we utilize Variational Inference (VI) and Monte Carlo sampling to approximate the posterior distributions.

Empirical Results

Our evaluation across synthetic simulations and real-world data demonstrates that PBL provides a robust alternative to fixed-parameter methods. A key finding is the stability of PBL across varying noise levels ($\sigma$). As shown in Figure 1, while the performance of methods with a fixed pessimism parameter $c$ is highly sensitive to noise, PBL (represented by stars) consistently maintains low regret.

Regret performance of PBL compared to fixed pessimism parameters $c$ across varying noise levels ($\sigma$).

Furthermore, PBL demonstrates superior convergence and lower regret as sample sizes increase, particularly in low-coverage regimes ($\epsilon=0.5$). Figure 4 illustrates this performance across both linear (BLBM) and non-linear (BNN) model architectures, where PBL (dashed orange line) consistently outperforms non-pessimistic baselines.

Regret vs. sample size for BLBM and BNN models across different coverage levels ($\epsilon$).

In a clinical application using the MIMIC-III sepsis dataset, PBL outperformed standard Q-learning and Conservative Q-Learning (CQL) in terms of average reward. Notably, PBL's treatment recommendations for IV fluids and vasopressors aligned more closely with expert physician behavior, avoiding the extreme dosages often suggested by non-pessimistic models.

Conclusion

PBL offers a principled, automated approach to offline DTR learning by leveraging Bayesian uncertainty to set the degree of pessimism. This eliminates the need for sensitive hyperparameter tuning while providing strong theoretical guarantees and empirical performance. Future research will focus on enhancing the computational efficiency of the sampling process for high-dimensional neural network architectures.

Summary updated

Abstract

In this article, we propose a novel pessimism-based Bayesian learning method for optimal dynamic treatment regimes in the offline setting. When the coverage condition does not hold, which is common for offline data, the existing solutions would produce sub-optimal policies. The pessimism principle addresses this issue by discouraging recommendation of actions that are less explored conditioning on the state. However, nearly all pessimism-based methods rely on a key hyper-parameter that quantifies the degree of pessimism, and the performance of the methods can be highly sensitive to the choice of this parameter. We propose to integrate the pessimism principle with Thompson sampling and Bayesian machine learning for optimizing the degree of pessimism. We derive a credible set whose boundary uniformly lower bounds the optimal Q-function, and thus we do not require additional tuning of the degree of pessimism. We develop a general Bayesian learning method that works with a range of models, from Bayesian linear basis model to Bayesian neural network model. We develop the computational algorithm based on variational inference, which is highly efficient and scalable. We establish the theoretical guarantees of the proposed method, and show empirically that it outperforms the existing state-of-the-art solutions through both simulations and a real data example.
BibTeX
@inproceedings{zhou2023optimizing,
  author = {Yunzhe Zhou and Zhengling Qi and Chengchun Shi and Lexin Li},
  title = {{Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approach}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=dRE0BmdvJc},
}
Figure