Quantile-Optimal Policy Learning under Unmeasured Confounding

Zhongren Chen, Siyu Chen, Zhengling Qi, Xiaohong Chen, Zhuoran Yang

Abstract

We study quantile-optimal policy learning where the goal is to find a policy whose reward distribution has the largest $\alpha$-quantile for some $\alpha \in (0, 1)$. We focus on the offline setting whose generating process involves unobserved confounders. Such a problem suffers from three main challenges: (i) nonlinearity of the quantile objective as a functional of the reward distribution, (ii) unobserved confounding issue, and (iii) insufficient coverage of the offline dataset. To address these challenges, we propose a suite of causal-assisted policy learning methods that provably enjoy strong theoretical guarantees under mild conditions. In particular, to address (i) and (ii), using causal inference tools such as instrumental variables and negative controls, we propose to estimate the quantile objectives by solving nonlinear functional integral equations. Then we adopt a minimax estimation approach with nonparametric models to solve these integral equations, and propose to construct conservative policy estimates that address (iii). The final policy is the one that maximizes these pessimistic estimates. In addition, we propose a novel regularized policy learning method that is more amenable to computation. Finally, we prove that the policies learned by these methods are $\tilde{\mathscr{O}}(n^{-1/2})$ quantile-optimal under a mild coverage assumption on the offline dataset. To the best of our knowledge, we propose the first sample-efficient policy learning algorithms for estimating the quantile-optimal policy when there exist unmeasured confounding.

Introduction

In high-stakes domains such as healthcare and finance, decision-makers often prioritize risk mitigation and fairness over average performance. While traditional offline reinforcement learning (RL) focuses on maximizing expected rewards, quantile-optimal policy learning aims to optimize specific points in the reward distribution, such as the median or the 10th percentile. This ensures a "safety floor" or a guaranteed level of performance for a specific portion of the population.

However, learning such policies from historical data is fraught with challenges. Unmeasured confounders—hidden variables that influence both historical actions and outcomes—introduce significant bias. Furthermore, offline datasets often lack sufficient coverage of the state-action space, making it difficult to predict the performance of new policies.

Problem Formulation and Challenges

The objective is to find a policy $\pi$ that maximizes the $\alpha$-quantile of the reward distribution $Q_\alpha(\pi)$. This problem presents three primary hurdles:

  1. Nonlinearity: Quantiles are nonlinear functionals of the reward distribution, unlike the mean, which is linear. This complicates statistical estimation.
  2. Unmeasured Confounding: Hidden factors (e.g., patient motivation in a clinical trial) can bias the relationship between actions and rewards.
  3. Partial Coverage: Offline data is fixed. If a policy selects actions rarely seen in the training data, performance estimates become unreliable.

Methodology: Causal-Assisted Policy Learning

To overcome these obstacles, Chen et al. (2025) introduce a framework that integrates causal inference with pessimistic estimation.

Causal Identification

The authors utilize auxiliary variables to bypass unmeasured confounding. Specifically, they employ Instrumental Variables (IV) and Negative Controls (NC). These tools allow for the formulation of nonlinear functional integral equations that uniquely identify the structural quantile function.

DAGs representing the observational and interventional processes for the Instrumental Variable (IV) setup.

In the IV setup, an external variable $Z$ influences the action $A$ but only affects the outcome $Y$ through $A$. In the NC setup, additional variables help control for the influence of the unobserved confounder $U$.

DAG of the interventional process for the Negative Control (NC) approach.

Minimax Estimation and Pessimism

The identified integral equations are solved using a minimax approach. By framing the estimation as a game between an estimator and an adversary, and applying Fenchel duality, the method maintains unbiasedness.

To address the coverage issue, the authors incorporate the pessimism principle. Instead of a "greedy" maximization of the estimated quantile, the algorithm selects the policy that performs best under a worst-case scenario within a confidence set. This ensures that the learner does not over-rely on regions of the data with high uncertainty.

Theoretical and Empirical Results

The paper establishes that the proposed algorithms are sample-efficient. The regret—the difference between the learned policy's performance and the true optimal—shrinks at a rate of $\tilde{O}(n^{-1/2})$. This matches the optimal rates typically seen in simpler mean-maximization problems.

Empirical evaluations using synthetic contextual bandit simulations demonstrate the robustness of the pessimistic approach. In scenarios with poor data coverage, greedy baselines often overfit to noisy, under-sampled regions, leading to high regret. In contrast, the pessimistic algorithms consistently identify safer, more effective policies.

Conclusion

This work provides the first sample-efficient framework for quantile-optimal policy learning under unmeasured confounding. By bridging causal inference and pessimistic offline RL, it offers a robust toolkit for risk-sensitive decision-making in real-world applications. Future research may focus on extending these theories to discrete reward distributions and reducing the computational overhead of the minimax optimization.

Summary updated

Abstract

We study quantile-optimal policy learning where the goal is to find a policy whose reward distribution has the largest $\alpha$-quantile for some $\alpha \in (0, 1)$. We focus on the offline setting whose generating process involves unobserved confounders. Such a problem suffers from three main challenges: (i) nonlinearity of the quantile objective as a functional of the reward distribution, (ii) unobserved confounding issue, and (iii) insufficient coverage of the offline dataset. To address these challenges, we propose a suite of causal-assisted policy learning methods that provably enjoy strong theoretical guarantees under mild conditions. In particular, to address (i) and (ii), using causal inference tools such as instrumental variables and negative controls, we propose to estimate the quantile objectives by solving nonlinear functional integral equations. Then we adopt a minimax estimation approach with nonparametric models to solve these integral equations, and propose to construct conservative policy estimates that address (iii). The final policy is the one that maximizes these pessimistic estimates. In addition, we propose a novel regularized policy learning method that is more amenable to computation. Finally, we prove that the policies learned by these methods are $\tilde{\mathscr{O}}(n^{-1/2})$ quantile-optimal under a mild coverage assumption on the offline dataset. Here, $\tilde{\mathscr{O}}(\cdot)$ omits poly-logarithmic factors. To the best of our knowledge, we propose the first sample-efficient policy learning algorithms for estimating the quantile-optimal policy when there exist unmeasured confounding.
BibTeX
@misc{chen2025quantile,
  author = {Zhongren Chen and Siyu Chen and Zhengling Qi and Xiaohong Chen and Zhuoran Yang},
  title = {{Quantile-Optimal Policy Learning under Unmeasured Confounding}},
  year = {2025},
  eprint = {2506.07140},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.07140},
}
Figure