Abstract
Deep reinforcement learning (DRL) typically requires the collection of interventional data, which can be expensive or unethical in real-world domains like autonomous driving and medicine. Offline reinforcement learning offers a solution by utilizing existing observational datasets. However, these datasets are often confounded by unobserved variables that influence both actions and outcomes, leading agents to learn biased, suboptimal policies. This paper proposes two "plug-in" deconfounding methods—reweighting and resampling—based on causal inference techniques. These methods adjust the influence of samples in the offline dataset to ensure unbiased learning. Compatible with standard model-free DRL algorithms such as Soft Actor-Critic (SAC) and Deep Q-Learning (DQN), these methods are validated through theoretical proofs and experiments on modified OpenAI Gym tasks.
Introduction: The Challenge of Confounded Data
Offline Reinforcement Learning (RL) allows agents to learn from pre-collected "observational" datasets, such as logs of human drivers or medical records, without direct environmental interaction. While powerful, this paradigm faces a significant hurdle: observational data is often confounded.
Consider a human driver who slows down because they hear a traffic report on the radio. If the car's sensors (the data the RL agent sees) do not record the radio report, the agent might incorrectly learn that "slowing down for no reason" is a high-reward policy. These unobserved variables—the radio report or weather conditions—are confounders ($w$) that affect both the action ($a$) and the resulting state/reward ($rs'$). Standard offline RL algorithms fail in these scenarios because they cannot distinguish between a true causal relationship and a coincidental association driven by a hidden variable.
Deconfounding Methods
The authors propose transforming the biased offline loss function into an unbiased online equivalent using two primary techniques:
- Reweighting Method: Each sample in the offline dataset is assigned an importance weight $d_1(\tau)$ calculated as a ratio of conditional probability densities. By weighting the RL algorithm's loss function, the agent effectively ignores misleading non-causal associations.
- Resampling Method: Instead of weighting the loss, the probability of a sample being drawn from the dataset is adjusted. Over-represented samples due to confounding are sampled less frequently, creating a "pseudo-interventional" dataset.
Technical Implementation
To calculate these weights, the authors employ Least-Squares Conditional Density Estimation (LSCDE). They also introduce a "jittering" technique—adding controlled noise during density estimation—to handle datasets containing both continuous (e.g., velocity) and discrete (e.g., gear shifts) variables.
A key insight is the utilization of the frontdoor criterion. By identifying an intermediate state $m$ (such as the actual physical force applied) that sits between the intended action $a$ and the outcome $rs'$, the influence of the hidden confounder $w$ can be mathematically blocked. Because these methods are decoupled from the learning step, they act as "plug-ins" for any model-free DRL algorithm, including SAC, DQN, and Conservative Q-Learning (CQL).
Experimental Results
Since standard RL benchmarks lack hidden confounders, the authors developed four new tasks based on the OpenAI Gym Pendulum environment:
- EmotionalPendulum: Actions are influenced by hidden "emotions" that also trigger environmental rewards.
- WindyPendulum: Hidden wind forces affect operator choices, rewards, and physical movement.
- Variants (*): Versions where confounders are partially observed to test conditioning capabilities.
Using the d3rlpy library, the researchers found that algorithms equipped with deconfounding (e.g., CQL_RW or CQL_RS) consistently and significantly outperformed standard counterparts. As the influence of the hidden confounder increased, the performance gap widened. Notably, simple Behavior Cloning (BC) failed across the board, proving that imitating a human is ineffective if the human possesses information the agent cannot see.
Conclusion and Future Work
This research provides a robust framework for applying deep RL to biased real-world data by separating deconfounding from policy learning. While the current method requires prior knowledge of the causal graph, future work will explore causal discovery—algorithms that can automatically infer the relationships between variables from data. This would further reduce the human expertise required to deploy safe and effective RL agents in complex, confounded environments.
Summary updated