Causal Deep Reinforcement Learning Using Observational Data

Wenxuan Zhu, Chao Yu, Qiang Zhang

Abstract

Deep reinforcement learning (DRL) typically requires the collection of interventional data, which can be expensive or unethical in real-world domains like autonomous driving and medicine. Offline reinforcement learning offers a solution by utilizing existing observational datasets. However, these datasets are often confounded by unobserved variables that influence both actions and outcomes, leading agents to learn biased, suboptimal policies. This paper proposes two "plug-in" deconfounding methods—reweighting and resampling—based on causal inference techniques. These methods adjust the influence of samples in the offline dataset to ensure unbiased learning. Compatible with standard model-free DRL algorithms such as Soft Actor-Critic (SAC) and Deep Q-Learning (DQN), these methods are validated through theoretical proofs and experiments on modified OpenAI Gym tasks.

Introduction: The Challenge of Confounded Data

Offline Reinforcement Learning (RL) allows agents to learn from pre-collected "observational" datasets, such as logs of human drivers or medical records, without direct environmental interaction. While powerful, this paradigm faces a significant hurdle: observational data is often confounded.

The transition from confounded offline observational data to an online environment.

Consider a human driver who slows down because they hear a traffic report on the radio. If the car's sensors (the data the RL agent sees) do not record the radio report, the agent might incorrectly learn that "slowing down for no reason" is a high-reward policy. These unobserved variables—the radio report or weather conditions—are confounders ($w$) that affect both the action ($a$) and the resulting state/reward ($rs'$). Standard offline RL algorithms fail in these scenarios because they cannot distinguish between a true causal relationship and a coincidental association driven by a hidden variable.

Deconfounding Methods

The authors propose transforming the biased offline loss function into an unbiased online equivalent using two primary techniques:

  1. Reweighting Method: Each sample in the offline dataset is assigned an importance weight $d_1(\tau)$ calculated as a ratio of conditional probability densities. By weighting the RL algorithm's loss function, the agent effectively ignores misleading non-causal associations.
  2. Resampling Method: Instead of weighting the loss, the probability of a sample being drawn from the dataset is adjusted. Over-represented samples due to confounding are sampled less frequently, creating a "pseudo-interventional" dataset.

Technical Implementation

To calculate these weights, the authors employ Least-Squares Conditional Density Estimation (LSCDE). They also introduce a "jittering" technique—adding controlled noise during density estimation—to handle datasets containing both continuous (e.g., velocity) and discrete (e.g., gear shifts) variables.

Causal graph illustrating the relationships between state (s), confounder (w), action (a), intermediate state (m), and outcome (rs').

A key insight is the utilization of the frontdoor criterion. By identifying an intermediate state $m$ (such as the actual physical force applied) that sits between the intended action $a$ and the outcome $rs'$, the influence of the hidden confounder $w$ can be mathematically blocked. Because these methods are decoupled from the learning step, they act as "plug-ins" for any model-free DRL algorithm, including SAC, DQN, and Conservative Q-Learning (CQL).

Experimental Results

Since standard RL benchmarks lack hidden confounders, the authors developed four new tasks based on the OpenAI Gym Pendulum environment:

  • EmotionalPendulum: Actions are influenced by hidden "emotions" that also trigger environmental rewards.
  • WindyPendulum: Hidden wind forces affect operator choices, rewards, and physical movement.
  • Variants (*): Versions where confounders are partially observed to test conditioning capabilities.

Using the d3rlpy library, the researchers found that algorithms equipped with deconfounding (e.g., CQL_RW or CQL_RS) consistently and significantly outperformed standard counterparts. As the influence of the hidden confounder increased, the performance gap widened. Notably, simple Behavior Cloning (BC) failed across the board, proving that imitating a human is ineffective if the human possesses information the agent cannot see.

Conclusion and Future Work

This research provides a robust framework for applying deep RL to biased real-world data by separating deconfounding from policy learning. While the current method requires prior knowledge of the causal graph, future work will explore causal discovery—algorithms that can automatically infer the relationships between variables from data. This would further reduce the human expertise required to deploy safe and effective RL agents in complex, confounded environments.

Summary updated

Abstract

Deep reinforcement learning (DRL) requires the collection of interventional data, which is sometimes expensive and even unethical in the real world, such as in the autonomous driving and the medical field. Offline reinforcement learning promises to alleviate this issue by exploiting the vast amount of observational data available in the real world. However, observational data may mislead the learning agent to undesirable outcomes if the behavior policy that generates the data depends on unobserved random variables (i.e., confounders). In this paper, we propose two deconfounding methods in DRL to address this problem. The methods first calculate the importance degree of different samples based on the causal inference technique, and then adjust the impact of different samples on the loss function by reweighting or resampling the offline dataset to ensure its unbiasedness. These deconfounding methods can be flexibly combined with existing model-free DRL algorithms such as soft actor-critic and deep Q-learning, provided that a weak condition can be satisfied by the loss functions of these algorithms. We prove the effectiveness of our deconfounding methods and validate them experimentally.
BibTeX
@inproceedings{zhu2023causal,
  author = {Wenxuan Zhu and Chao Yu and Qiang Zhang},
  title = {{Causal Deep Reinforcement Learning Using Observational Data}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=ivvYr9iP0m},
}
Figure