Blessing from Experts: Super Reinforcement Learning in Confounded Environments

Jiayi Wang, Zhengling Qi, Chengchun Shi

Abstract

We introduce super reinforcement learning in the batch setting, which takes the observed action as input for enhanced policy learning. In the presence of unmeasured confounders, the recommendations from human experts recorded in the observed data allow us to recover certain unobserved information. Including this information in the policy search, the proposed super reinforcement learning will yield a super-policy that is guaranteed to outperform both the standard optimal policy and the behavior one (e.g., the expert's recommendation). Furthermore, to address the issue of unmeasured confounding in finding super-policies, a number of non-parametric identification results are established. Finally, we develop two super-policy learning algorithms and derive their corresponding finite-sample regret guarantees.

Introduction

In offline Reinforcement Learning (RL), the goal is to derive optimal decision-making policies from historical datasets. A significant challenge in this domain is unmeasured confounding, where variables influencing both actions and rewards are absent from the logs. Traditionally, such hidden variables are viewed as a "curse" that biases learned policies. This paper proposes a paradigm shift: treating the actions of past agents (e.g., human experts) as a "blessing" that reveals hidden information.

The Super RL Paradigm

The core innovation is the super-policy, denoted as $\nu(a|s, a')$. Unlike a standard policy $\pi(a|s)$ that relies solely on the observed state $s$, a super-policy conditions on the behavior agent's action $a'$. Since $a'$ was chosen based on both observed and unobserved features, it serves as a valuable proxy for missing data.

Causal diagram illustrating the relationship between unobserved confounders (U), states (S), behavior actions (A'), and the resulting super-policy (ν*) vs. standard policy (π*).

By incorporating the behavior action, the AI can effectively "see" through the expert's eyes, capturing unstructured knowledge like a clinician's intuition or a trader's reaction to unlogged signals. This leads to the "Super-Optimality Lemma," which proves that the optimal super-policy is guaranteed to be at least as good as, and often strictly better than, the best policy that ignores the behavior agent's input.

Identification via Proximal Causal Inference

To identify the value of super-policies without direct access to hidden confounders, the authors leverage proximal causal inference. This involves using "bridge functions" ($q$-bridge and $v$-bridge) and "proxy" variables. These proxies are related to the hidden confounders but satisfy specific independence conditions relative to rewards and actions. This framework allows for the non-parametric identification of counterfactual outcomes from purely observational data.

The paper introduces two primary algorithms:

  1. Algorithm 1: Designed for confounded contextual bandits (single-step).
  2. Algorithm 2: Tailored for sequential decision-making in multi-step Partially Observable Markov Decision Processes (POMDPs) using a fitted-Q-iteration approach.

Experimental Results

The researchers validated Super RL across synthetic and real-world clinical datasets.

Synthetic Simulations

In environments with varying degrees of confounding ($\epsilon$), the super-policy consistently met or exceeded the performance of standard optimal policies.

Table comparing policy values for the behavior policy (πb), standard optimal policy (π*), and super-policy (ν*) across different confounding levels.

As shown in the table, when confounding is present ($\epsilon = 0$ or $1$), the super-policy $\mathcal{V}(\nu^*)$ achieves a value of 1.0, significantly outperforming the standard optimal policy $\mathcal{V}(\pi^*)$ which remains at 0.4. In continuous settings, the Super policy's regret was as low as 0.06 compared to 0.40 for a state-only policy.

Real-World Clinical Data

  • ICU Admission: Using a dataset of 13,011 patients from UK hospitals, the Super-policy achieved a value of 88.70, outperforming state-only (88.18) and state+proxy (88.10) baselines.
  • MIMIC-III Sepsis: In this task, the Super-policy performed similarly to standard policies, suggesting that for this specific dataset, behavior actions might not have contained significant information beyond the recorded variables.

Conclusion

Super Reinforcement Learning demonstrates that unmeasured confounding can be mitigated by leveraging the implicit knowledge embedded in expert actions. By transforming a behavior signal into a feature, researchers can develop policies that are mathematically guaranteed to be superior to standard offline RL approaches. Future work will focus on reducing computational complexity in sequential settings and exploring alternative identification frameworks like instrumental variables.

Summary updated

Abstract

We introduce super reinforcement learning in the batch setting, which takes the observed action as input for enhanced policy learning. In the presence of unmeasured confounders, the recommendations from human experts recorded in the observed data allow us to recover certain unobserved information. Including this information in the policy search, the proposed super reinforcement learning will yield a super-policy that is guaranteed to outperform both the standard optimal policy and the behavior one (e.g., the expert's recommendation). Furthermore, to address the issue of unmeasured confounding in finding super-policies, a number of non-parametric identification results are established. Finally, we develop two super-policy learning algorithms and derive their corresponding finite-sample regret guarantees.
BibTeX
@inproceedings{wang2023blessing,
  author = {Jiayi Wang and Zhengling Qi and Chengchun Shi},
  title = {{Blessing from Experts: Super Reinforcement Learning in Confounded Environments}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=47C06k5D2cn},
}
Figure