Variable-Agnostic Causal Exploration for Reinforcement Learning

Minh Hoang Nguyen, Hung Le, Svetha Venkatesh

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in complex domains, yet it remains challenged by environments characterized by sparse rewards and "Noisy-TV" distractions—unpredictable but task-irrelevant stimuli. Traditional exploration strategies, such as curiosity-based intrinsic motivation, often fail in these scenarios as they may incentivize agents to focus on irrelevant noise. Furthermore, existing causal RL methods typically require pre-defined environment variables, a requirement that is often impractical in real-world applications where agents receive raw, unstructured data.

We present Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL), a framework designed to overcome these limitations by autonomously discovering the causal structure of an environment without prior knowledge of its variables.

The VACERL Framework

VACERL operates through a three-phase iterative process that filters environmental noise and identifies the causal chain leading to a goal.

Overview of the VACERL framework, including environment interaction (a), the resulting causal tree (b), and the three-phase architectural pipeline (c).
  1. Crucial Step Detection: The framework utilizes a Transformer model to analyze successful trajectories stored in a buffer. By training the Transformer to predict the final goal-reaching step and examining its attention scores, VACERL identifies "Crucial Observation-Action Steps" (SCOAS). This attention mechanism serves as a critical filter, isolating task-relevant actions from environmental noise.
  2. Causal Structure Discovery: Identified crucial steps are used to learn a Structural Causal Model (SCM). This phase constructs a directed graph where nodes represent crucial steps and edges denote causal dependencies (e.g., "picking up a key" must precede "opening a door").
  3. Agent Training with Causal Information: The discovered causal graph is transformed into a hierarchical Causal Tree. This tree informs the agent's training through causal intrinsic rewards—incentivizing the traversal of the causal chain—and by providing structured subgoals for hierarchical reinforcement learning (HRL).

Experimental Evaluation

The authors benchmarked VACERL against standard RL algorithms (e.g., PPO) and curiosity-based methods (e.g., RND) across several environments:

  • Minigrid (MG-1 to MG-4): Testing complex causal chains involving keys and doors.
  • Noisy-TV Environments: Modified Minigrid tasks featuring distracting, unpredictable noise.
  • Robotics (FetchReach/FetchPickAndPlace): Evaluating performance in continuous state spaces.

Results and Discussion

VACERL demonstrated significant performance gains, particularly in sparse-reward and high-noise settings. In the MG-4 task, VACERL was the only method to achieve a non-zero success rate (0.29), while all baselines failed. In "Noisy-TV" scenarios, VACERL maintained high performance by effectively "muting" irrelevant stimuli through its causal filtering process.

A learned causal graph illustrating the interpretable dependencies discovered by VACERL, such as the necessity of picking up a key before opening a door to reach the goal.

Furthermore, the framework produces interpretable causal structures. As shown in the learned graph, the system correctly identifies the logical sequence of operations required for task completion. In robotics tasks, the use of causal subgoals significantly accelerated convergence compared to standard hierarchical methods.

Conclusion

VACERL introduces a robust, variable-agnostic approach to causal discovery in RL. By leveraging attention mechanisms to filter noise and SCMs to map task dependencies, it enables efficient learning in complex, distracting environments. Future work will focus on addressing the "cold start" problem and automating hyperparameter tuning to further enhance the framework's generalizability.

Summary updated

BibTeX
@inproceedings{nguyen2024variable,
  author = {Minh Hoang Nguyen and Hung Le and Svetha Venkatesh},
  title = {{Variable-Agnostic Causal Exploration for Reinforcement Learning}},
  booktitle = {ECML/PKDD},
  year = {2024},
  url = {https://openreview.net/forum?id=LSAz1I6HHi},
}
Figure