TL;DR
In offline reinforcement learning, unmeasured confounders (hidden variables influencing both decisions and outcomes) often lead to severely suboptimal policies. This paper introduces Super RL, a framework that treats the human expert's recommended action not just as historical data, but as an informative proxy for hidden states. By taking the human recommendation at decision time and selectively overriding it, the resulting "super-policy" is mathematically guaranteed to outperform both the human expert and standard RL policies.
What Problem Does This Paper Address?
Offline RL seeks to learn decision-making policies from pre-collected batch data. In high-stakes settings like healthcare, learning algorithms frequently face unmeasured confounding: human experts (e.g., ICU clinicians) base treatment decisions on unrecorded observations, like bedside intuition or subtle visual cues.
Standard offline RL attempts to find a policy $\pi^*(a \mid s)$ based solely on recorded states $S$, ignoring the unobserved confounders $U$. This leads to biased evaluations and policies that can perform worse than the original human behavior policy $\pi^b$. The authors ask: Can we use the human expert’s action itself as a signal to partially reveal the unobserved confounder, and learn a policy that conditions on—and can override—that recommendation?
Methods
The authors formalize Super RL across contextual bandits and sequential decision-making (partially observable MDPs):
- Super-Policy formulation: Instead of mapping $S \to \mathcal{A}$, a super-policy searches over $\nu(a \mid s, z, a')$, which takes observed features $s$, an action proxy $z$, and the human expert's recommendation $a'$ at decision time, deciding whether to follow or override it.
- Proximal Causal Inference: Because $U$ is unobserved, counterfactual values $E[R(a) \mid S, A=a']$ are non-parametrically unidentifiable without additional structure. The authors adopt proximal causal inference, introducing action proxies $Z$ and outcome/reward proxies $W$.
- Bridge Functions: They establish identification using linear integral equations that solve for Q-bridge functions $q(W, S, a)$. They extend identification from single-stage bandits to:
- Stationary policies in confounded environments;
- Full history-dependent policies;
- Practical $k$-step memory policies;
- Tabular POMDPs via matrix inversion.
- Learning & Regret: They design practical algorithms using minimax conditional moment estimation and empirical risk minimization (Fitted-Q Iteration style), proving polynomial $\mathcal{O}(n^{-1/2})$ finite-sample regret bounds characterized by VC dimensions and RKHS local Rademacher complexities.
Datasets
The authors evaluate Super RL across synthetic and real-world benchmarks:
- Synthetic Bandits and POMDPs: Discrete and continuous synthetic environments parameterized by confounding strength $\epsilon$.
- SUPPORT (Right Heart Catheterization / RHC): An observational medical dataset of 5,735 critically ill ICU patients with 71 clinical covariates, framed as a contextual bandit to optimize 30-day survival. Proxies include blood gas measurements ($pafi1, paco21$) and physiological markers ($ph1, hema1$).
- MIMIC-III: Electronic health records used to manage sepsis treatments (vasopressor and IV fluid dosages) across an episodic POMDP ($T=2$), targeting the SOFA score as reward.
Key Insights and Ideas
- Expert recommendations as a blessing, not just a baseline: Traditional RL views observational actions solely as data generated by an imperfect behavior policy. Super RL views the expert's decision as a noisy measurement device for the latent state $U$.
- Guaranteed Super-Optimality: The paper proves $V(\nu^*) \ge \max\{V(\pi^*), V(\pi^b)\}$. The super-policy is theoretically guaranteed to be no worse than either the unassisted optimal policy $\pi^*$ or the expert behavior policy $\pi^b$.
- Greater confounding yields greater relative gains: When the behavior policy strongly depends on hidden confounders, the expert's recommendation leaks more information about $U$, expanding the performance gap between the super-policy and standard policies.
Key Results
- Synthetic Contextual Bandits: In continuous state simulations with strong confounding ($\epsilon = 0.9$), Super RL achieves a regret of 0.065, compared to 0.11 for state+proxy policies (SZONLY) and 0.40 for state-only policies (SONLY).
- Sequential Simulation: Super RL reduces cumulative regret to 2.2 (std: 0.49), versus 5.3 (SZONLY) and 5.4 (SONLY).
- RHC Healthcare Dataset: Super RL achieves an estimated 30-day survival value of 0.69 (std: 0.011), outperforming both standard covariate-adjusted policies at 0.55.
- MIMIC-III Sepsis Study: Super RL achieves an average policy return of -1.75, substantially improving over SZONLY (-2.81) and SONLY (-2.83).
Limitations and Future Work
- Human-in-the-Loop Requirement at Inference: To deploy a super-policy, a human expert must still provide a tentative recommendation at each decision point, which the model evaluates and potentially overrides.
- Strict Proxy Assumptions: Identification requires valid action and reward proxies satisfying conditional independence and completeness (invertibility) conditions, which can be challenging to select and verify in practice.
- Theoretical Regret Scope: Regret bounds in the multi-step sequential setting explicitly rely on a memoryless confounding assumption ($U_t \perp \text{past} \mid S_t$). Extending finite-sample guarantees to general non-Markovian confounding across long horizons remains open.
Summary updated