Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approach

Yunzhe Zhou, Zhengling Qi, Chengchun Shi, Lexin Li

Abstract

In this article, we propose a novel pessimism-based Bayesian learning method for optimal dynamic treatment regimes in the offline setting. When the coverage condition does not hold, which is common for offline data, the existing solutions would produce sub-optimal policies. The pessimism principle addresses this issue by discouraging recommendation of actions that are less explored conditioning on the state. However, nearly all pessimism-based methods rely on a key hyper-parameter that quantifies the degree of pessimism, and the performance of the methods can be highly sensitive to the choice of this parameter. We propose to integrate the pessimism principle with Thompson sampling and Bayesian machine learning for optimizing the degree of pessimism. We derive a credible set whose boundary uniformly lower bounds the optimal Q-function, and thus we do not require additional tuning of the degree of pessimism. We develop a general Bayesian learning method that works with a range of models, from Bayesian linear basis model to Bayesian neural network model. We develop the computational algorithm based on variational inference, which is highly efficient and scalable. We establish the theoretical guarantees of the proposed method, and show empirically that it outperforms the existing state-of-the-art solutions through both simulations and a real data example.

Introduction

In precision medicine, Dynamic Treatment Regimes (DTRs) provide sequences of decision rules to tailor treatments to a patient’s evolving health state. While online trials are ideal for learning these rules, healthcare often necessitates the use of offline data—historical records—due to safety and cost considerations.

A primary challenge in offline learning is the lack of data coverage. If certain treatments are rarely administered under specific conditions, the data provides insufficient information about their effectiveness. Standard algorithms may erroneously recommend these unexplored actions due to high variance in their estimates. Existing "pessimistic" solutions attempt to mitigate this by penalizing actions with limited data. However, these methods typically depend on a "degree of pessimism" hyperparameter ($c$) that is difficult to tune. Excessive pessimism can lead to missing effective treatments, while insufficient pessimism may result in recommending risky, unproven interventions.

Methodology

We introduce Pessimism-based Bayesian Learning (PBL), a framework that integrates the pessimism principle with Bayesian machine learning to automate hyperparameter tuning.

  1. Bayesian Machine Learning (BML): We employ models such as Bayesian Linear Basis Models (BLBM) and Bayesian Neural Networks (BNN) to estimate the Q-function, representing the expected reward for a given action.
  2. The Pessimism Principle via Credible Sets: Rather than manual penalties, we derive a uniform credible set for the Q-function. The policy then selects the action maximizing the lower bound of this set. This approach ensures pessimism is applied only where the model exhibits genuine uncertainty.
  3. Thompson Sampling Integration: We adapt the principles of Thompson sampling to quantify pointwise uncertainty in the offline setting.
  4. Variational Inference: To ensure scalability for complex models like neural networks, we utilize variational inference for efficient computation.

Experimental Evaluation

The PBL framework was evaluated using synthetic simulations and real-world data from the MIMIC-III v1.4 dataset. Simulations included both linear and nonlinear signals to test performance under varying levels of data coverage ($\epsilon$).

Regret comparison between PBL and fixed pessimism parameters ($c$) across different coverage levels ($\epsilon$) and noise levels ($\sigma$).

As shown in Figure 2, PBL (represented by dashed lines) demonstrates significant robustness compared to fixed-parameter methods. In scenarios with poor data coverage (high $\epsilon$), PBL consistently achieves lower regret. Unlike methods sensitive to the choice of $c$, PBL's data-driven approach to pessimism allows it to remain competitive even when data coverage is high, whereas fixed $c$ values can lead to sub-optimal performance if improperly tuned.

Regret vs. sample size for BLBM and BNN models under various coverage conditions ($\epsilon$) for linear signals.

Figure 4 illustrates the scalability of PBL across different sample sizes and model architectures. The method effectively reduces regret as sample size increases, maintaining superior performance across both linear (BLBM) and non-linear (BNN) modeling assumptions. In scenarios where data coverage was poor, PBL significantly reduced regret compared to standard Q-learning and the PEVI baseline.

In the MIMIC-III application, PBL achieved the highest average reward among compared methods, including Conservative Q-Learning. The resulting policy aligned with clinical intuition—exhibiting conservatism with vasopressors while suggesting optimized dosages for intravenous fluids.

Discussion and Conclusion

PBL addresses a critical gap in offline DTR optimization by automating the tuning of pessimism. By leveraging Bayesian credible sets, the degree of pessimism becomes data-driven, enhancing stability across diverse scenarios. While the framework is model-agnostic and scalable, it is computationally intensive due to the Monte Carlo sampling required for lower bound estimation. Future research will focus on accounting for model misspecification and improving the efficiency of the optimal action search within credible sets.

Summary updated

Abstract

In this article, we propose a novel pessimism-based Bayesian learning method for optimal dynamic treatment regimes in the offline setting. When the coverage condition does not hold, which is common for offline data, the existing solutions would produce sub-optimal policies. The pessimism principle addresses this issue by discouraging recommendation of actions that are less explored conditioning on the state. However, nearly all pessimism-based methods rely on a key hyper-parameter that quantifies the degree of pessimism, and the performance of the methods can be highly sensitive to the choice of this parameter. We propose to integrate the pessimism principle with Thompson sampling and Bayesian machine learning for optimizing the degree of pessimism. We derive a credible set whose boundary uniformly lower bounds the optimal Q-function, and thus we do not require additional tuning of the degree of pessimism. We develop a general Bayesian learning method that works with a range of models, from Bayesian linear basis model to Bayesian neural network model. We develop the computational algorithm based on variational inference, which is highly efficient and scalable. We establish the theoretical guarantees of the proposed method, and show empirically that it outperforms the existing state-of-the-art solutions through both simulations and a real data example.
BibTeX
@inproceedings{zhou2023optimizing,
  author = {Yunzhe Zhou and Zhengling Qi and Chengchun Shi and Lexin Li},
  title = {{Optimizing Pessimism in Dynamic Treatment Regimes: A Bayesian Learning Approach}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=dRE0BmdvJc},
}
Figure