Goal Discovery with Causal Capacity for Efficient Reinforcement Learning

Yan Yu, Yaodong Yang, Zhengbo Lu, Chengdong Ma, Wengang Zhou, Houqiang LiarXiv 2025 · DOI · arXiv

Abstract

Reinforcement Learning (RL) agents often struggle in environments with sparse rewards, where meaningful feedback is rare. This paper introduces Goal Discovery with Causal Capacity (GDCC), a novel framework designed to autonomously identify "critical points" in an environment to serve as subgoals. By leveraging the concept of Causal Capacity—a metric quantifying an agent's influence over its future trajectory—GDCC identifies high-control states, such as crossroads, to guide exploration. Experimental results on MuJoCo Maze and 3D Habitat navigation benchmarks show that GDCC improves success rates by over 25% compared to existing exploration baselines, while requiring minimal computational overhead.

Introduction

In many real-world RL applications, agents face a "sparse reward desert." Without intermediate milestones, an agent may spend excessive time wandering aimlessly before encountering a reward signal. While intrinsic motivation and curiosity-driven exploration have been proposed, they often lack purposeful direction. The ideal solution is to decompose complex tasks into manageable subgoals. However, automatically discovering these subgoals without prior environment knowledge or expert demonstrations remains a significant challenge.

Methodology: Causal Capacity and GDCC

The core innovation of this work is the Causal Capacity metric. Intuitively, a state with high causal capacity is one where the agent's actions significantly diverge the future outcome.

Illustration of causal capacity in different scenarios and its distribution in a maze environment.

As shown in the figure above, a "crossroad" represents a high-capacity state because the agent's choice determines which path is taken. Conversely, a "one-way street" has low capacity because the agent's actions have little impact on the final destination.

The GDCC Framework

The GDCC framework operates in three distinct stages:

  1. Measuring Causal Capacity: Using Granger entropy, the authors quantify how much an action affects future states. To handle continuous spaces, a Monte Carlo-based approach is employed using data collected from a simple random policy ($\pi_{ran}$).
  2. Clustering and Critical Point Identification: Agglomerative clustering groups state transitions. States that lead to many distinct clusters of future outcomes are flagged as high-capacity "critical points."
  3. Subgoal Prediction: A self-supervised subgoal predictor is trained to map current states to relevant high-capacity subgoals. This model effectively partitions the environment into regions governed by specific milestones.
The GDCC framework architecture, showing the pretraining phase for subgoal discovery and the subsequent training phase.

Experimental Evaluation

The researchers evaluated GDCC on two major benchmarks:

  • MuJoCo Maze: Physics-based navigation in medium and large 2D mazes.
  • Habitat: Challenging 3D indoor navigation using real-world home scans. The "Applewold" map is particularly difficult, requiring navigation across three floors and stairs.

Key Results

  • Performance Gains: GDCC combined with standard RL algorithms (e.g., TD3) achieved at least a 25% higher success rate on average than baselines in complex environments.
  • Comparison with Baselines: GDCC outperformed popular methods like Random Network Distillation (RND) and Causal Action Influence (CAI). In 3D Habitat maps, where RND often fails to reach distant goals, GDCC maintained robust performance.
  • Efficiency: The pre-training phase for subgoal discovery is highly efficient, accounting for less than 3% of the total training time.

Conclusion

GDCC provides a mathematically grounded approach to subgoal discovery by treating control as a compass for exploration. By identifying states where actions have the most causal influence, the framework enables agents to navigate complex, sparse-reward environments more effectively. Future work includes extending this approach to robotic manipulation tasks, where critical points could represent physical interactions with objects.

Summary updated

Abstract

Causal inference is crucial for humans to explore the world, which can be modeled to enable an agent to efficiently explore the environment in reinforcement learning. Existing research indicates that establishing the causality between action and state transition will enhance an agent to reason how a policy affects its future trajectory, thereby promoting directed exploration. However, it is challenging to measure the causality due to its intractability in the vast state-action space of complex scenarios. In this paper, we propose a novel Goal Discovery with Causal Capacity (GDCC) framework for efficient environment exploration. Specifically, we first derive a measurement of causality in state space, \emph{i.e.,} causal capacity, which represents the highest influence of an agent's behavior on future trajectories. After that, we present a Monte Carlo based method to identify critical points in discrete state space and further optimize this method for continuous high-dimensional environments. Those critical points are used to uncover where the agent makes important decisions in the environment, which are then regarded as our subgoals to guide the agent to make exploration more purposefully and efficiently. Empirical results from multi-objective tasks demonstrate that states with high causal capacity align with our expected subgoals, and our GDCC achieves significant success rate improvements compared to baselines.
BibTeX
@misc{yu2025goal,
  author = {Yan Yu and Yaodong Yang and Zhengbo Lu and Chengdong Ma and Wengang Zhou and Houqiang Li},
  title = {{Goal Discovery with Causal Capacity for Efficient Reinforcement Learning}},
  year = {2025},
  eprint = {2508.09624},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.09624},
}
Figure