Abstract
Reinforcement Learning (RL) agents often struggle in environments with sparse rewards, where meaningful feedback is rare. This paper introduces Goal Discovery with Causal Capacity (GDCC), a novel framework designed to autonomously identify "critical points" in an environment to serve as subgoals. By leveraging the concept of Causal Capacity—a metric quantifying an agent's influence over its future trajectory—GDCC identifies high-control states, such as crossroads, to guide exploration. Experimental results on MuJoCo Maze and 3D Habitat navigation benchmarks show that GDCC improves success rates by over 25% compared to existing exploration baselines, while requiring minimal computational overhead.
Introduction
In many real-world RL applications, agents face a "sparse reward desert." Without intermediate milestones, an agent may spend excessive time wandering aimlessly before encountering a reward signal. While intrinsic motivation and curiosity-driven exploration have been proposed, they often lack purposeful direction. The ideal solution is to decompose complex tasks into manageable subgoals. However, automatically discovering these subgoals without prior environment knowledge or expert demonstrations remains a significant challenge.
Methodology: Causal Capacity and GDCC
The core innovation of this work is the Causal Capacity metric. Intuitively, a state with high causal capacity is one where the agent's actions significantly diverge the future outcome.
As shown in the figure above, a "crossroad" represents a high-capacity state because the agent's choice determines which path is taken. Conversely, a "one-way street" has low capacity because the agent's actions have little impact on the final destination.
The GDCC Framework
The GDCC framework operates in three distinct stages:
- Measuring Causal Capacity: Using Granger entropy, the authors quantify how much an action affects future states. To handle continuous spaces, a Monte Carlo-based approach is employed using data collected from a simple random policy ($\pi_{ran}$).
- Clustering and Critical Point Identification: Agglomerative clustering groups state transitions. States that lead to many distinct clusters of future outcomes are flagged as high-capacity "critical points."
- Subgoal Prediction: A self-supervised subgoal predictor is trained to map current states to relevant high-capacity subgoals. This model effectively partitions the environment into regions governed by specific milestones.
Experimental Evaluation
The researchers evaluated GDCC on two major benchmarks:
- MuJoCo Maze: Physics-based navigation in medium and large 2D mazes.
- Habitat: Challenging 3D indoor navigation using real-world home scans. The "Applewold" map is particularly difficult, requiring navigation across three floors and stairs.
Key Results
- Performance Gains: GDCC combined with standard RL algorithms (e.g., TD3) achieved at least a 25% higher success rate on average than baselines in complex environments.
- Comparison with Baselines: GDCC outperformed popular methods like Random Network Distillation (RND) and Causal Action Influence (CAI). In 3D Habitat maps, where RND often fails to reach distant goals, GDCC maintained robust performance.
- Efficiency: The pre-training phase for subgoal discovery is highly efficient, accounting for less than 3% of the total training time.
Conclusion
GDCC provides a mathematically grounded approach to subgoal discovery by treating control as a compass for exploration. By identifying states where actions have the most causal influence, the framework enables agents to navigate complex, sparse-reward environments more effectively. Future work includes extending this approach to robotic manipulation tasks, where critical points could represent physical interactions with objects.
Summary updated