Abstract
Reinforcement learning (RL) agents have been deployed in complex environments where interactions are costly, and learning is usually slow. One prominent task in these settings is to reuse interactions performed by other agents to accelerate the learning process. Causal inference provides a family of methods to infer the effects of actions from a combination of data and qualitative assumptions about the underlying environment. Despite its success of transferring invariant knowledge across domains in the empirical sciences, causal inference has not been fully realized in the context of transfer learning in interactive domains. In this paper, we use causal inference as a basis to support a principled and more robust transfer of knowledge in RL settings. In particular, we tackle the problem of transferring knowledge across bandit agents in settings where causal effects cannot be identified by do-calculus [Pearl, 2000] and standard learning techniques. Our new identification strategy combines two steps -- first, deriving bounds over the arm’s distribution based on structural knowledge; second, incorporating these bounds in a dynamic allocation procedure so as to guide the search towards more promising actions. We formally prove that our strategy dominates previously known algorithms and achieves orders of magnitude faster convergence rates than these algorithms. Finally, we perform simulations and empirically demonstrate that our strategy is consistently more efficient than the current (non-causal) state-of-the-art methods.
Introduction
A central challenge in reinforcement learning is the efficient reuse of data collected by other agents to accelerate learning in new tasks. However, in many real-world scenarios, unobserved confounders (UCs)—hidden variables that influence both actions and rewards—can severely bias the learning process. Standard transfer learning techniques often fail in these settings because they cannot distinguish between correlation and causation. This leads to "negative transfer," where an agent performs worse by incorporating source data than it would by learning from scratch.
Methodology
To address the challenge of unobserved confounders, we propose a framework that integrates causal inference with Multi-Armed Bandits (MAB). Our approach consists of two main components:
- Causal Bounding: When causal effects are non-identifiable (i.e., cannot be uniquely determined from data), we leverage Structural Causal Models (SCMs) and do-calculus to derive mathematical upper and lower bounds on the expected rewards. These bounds provide a theoretically sound range for the true causal effect of each action.
- Causal-Enhanced Algorithms: We introduce two algorithms that utilize these bounds to guide exploration:
- B-kl-UCB (Bounded kl-UCB): This algorithm uses causal bounds to prune suboptimal arms and truncate upper confidence bounds, focusing exploration on the most promising actions.
- B-TS (Bounded Thompson Sampling): This variant employs rejection sampling to ensure that reward estimates remain within the theoretically derived causal bounds.
Experimental Setup
We evaluate our approach using three canonical transfer tasks where expected rewards are non-identifiable due to unobserved confounders. These tasks involve transferring knowledge between agents with different action spaces or unmeasured contexts.
- Task 1: Transfer from a contextual bandit with unmeasured context to a standard MAB.
- Tasks 2 & 3: Transfer between agents with differing actuators (action spaces).
Results
Our simulations demonstrate that causal-enhanced algorithms significantly outperform both standard MAB solvers and naive transfer methods. In Task 2, for instance, B-kl-UCB and B-TS achieved near-zero cumulative regret, effectively identifying the optimal arm almost immediately.
Key findings include:
- Efficiency: Causal bounds allow for orders of magnitude faster convergence by significantly reducing the search space. In Task 2, causal methods achieved zero regret, while standard kl-UCB and Thompson Sampling reached regrets of approximately 25 and 10, respectively.
- Safety: By acknowledging non-identifiability and using bounds, our methods avoid the catastrophic negative transfer seen in naive approaches (e.g., kl-UCB⁻ and TS⁻), which are misled by spurious correlations and suffer from extreme regret.
- Robustness: In scenarios with weak causal information (e.g., Task 3), the performance of our algorithms gracefully matches that of standard non-transfer methods, ensuring no performance penalty for using the causal framework.
Conclusion
This work provides a principled foundation for causal transfer learning in interactive domains. By leveraging structural knowledge to derive causal bounds, we enable agents to learn safely and efficiently even in the presence of unobserved confounders. Future research will focus on extending these methods to continuous action spaces and full Markov Decision Processes.
Summary updated