Abstract
Deep reinforcement learning (DRL) requires the collection of interventional data, which is sometimes expensive and even unethical in the real world, such as in autonomous driving and the medical field. Offline reinforcement learning promises to alleviate this issue by exploiting the vast amount of observational data available in the real world. However, observational data may mislead the learning agent to undesirable outcomes if the behavior policy that generates the data depends on unobserved random variables (i.e., confounders). In this paper, we propose two deconfounding methods in DRL to address this problem. The methods first calculate the importance degree of different samples based on the causal inference technique, and then adjust the impact of different samples on the loss function by reweighting or resampling the offline dataset to ensure its unbiasedness. These deconfounding methods can be flexibly combined with existing model-free DRL algorithms such as soft actor-critic and deep Q-learning, provided that a weak condition can be satisfied by the loss functions of these algorithms. We prove the effectiveness of our deconfounding methods and validate them experimentally.
Introduction: The Challenge of Hidden Confounders
Reinforcement learning typically thrives on interventional data—data gathered by an agent actively experimenting in an environment. However, in high-stakes fields like medicine or autonomous driving, such experimentation is often impractical or unethical. Offline RL attempts to solve this by learning from pre-existing observational data (e.g., historical logs).
A major hurdle in offline RL is the presence of "hidden confounders." These are variables that influence both the actions taken in the historical data and the resulting outcomes, but are not recorded in the dataset. For instance, a human driver might adjust their speed based on the weather ($w$), but if the car's sensors only record position ($s$), an offline agent might incorrectly attribute the driver's success solely to the position, leading to biased value estimations and poor real-world performance.
Methodology: Reweighting and Resampling
The core contribution of this work is the introduction of two "plug-and-play" deconfounding methods that can be integrated into standard model-free DRL algorithms like SAC, DQN, or CQL. These methods rely on the use of causal graphs and do-calculus to transform biased expectations into unbiased ones.
- Reweighting Method: This approach modifies the loss function of the neural network. It calculates an importance sampling ratio using causal do-calculus to "correct" the weight of each sample. This ensures the loss function reflects true environment dynamics rather than the biased observational ones.
- Resampling Method: Instead of modifying the loss function, this method adjusts the dataset itself. It changes the probability of a sample being drawn during training based on its "importance degree," ensuring the training batch is representative of an unbiased interventional distribution.
To implement these, the authors utilize Least-Squares Conditional Density Estimation (LSCDE) to estimate the necessary probability distributions from the offline data.
Experimental Evaluation
To test these methods, the authors developed new benchmark tasks based on the OpenAI Gym Pendulum environment, introducing artificial confounders like "emotions" or "wind" that affect both torque choices and rewards.
Key findings include:
- Significant Performance Gains: In the
EmotionalPendulumtask, standard CQL often resulted in negative rewards. Integrating the Reweighting (RW) or Resampling (RS) methods pushed these rewards into positive territory, demonstrating successful deconfounding. - Robustness and Speed: The proposed methods consistently outperformed original algorithms across various noise levels and confounding strengths. Furthermore, agents using these methods reached higher reward plateaus significantly faster in tasks like
WindyPendulum.
Conclusion and Future Directions
This research demonstrates that causal inference is a powerful tool for bridging the gap between biased observational data and effective real-world deployment. By decoupling the deconfounding process from the specific RL algorithm, these methods offer a flexible solution for large-scale continuous deep RL problems.
However, the current framework assumes the causal graph is known a priori. Future work will focus on Causal Discovery—algorithms that can automatically infer the causal structure from data—and improving density estimation in high-dimensional spaces, such as raw image data.
Summary updated