Introduction
In continuous control reinforcement learning (RL), such as training a bipedal robot to walk, agents often struggle with deceptive reward landscapes. A common failure mode is convergence to a "local optimum"—for instance, a robot might learn to fall forward to gain a small distance reward rather than discovering the complex, globally optimal behavior of stable walking. Traditional exploration methods, which typically rely on adding random noise to parameters or maximizing the entropy of a single Gaussian policy, often fail to cover the action space effectively.
The Attraction-Repulsion Actor-Critic (ARAC) Framework
To address these challenges, we introduce Attraction-Repulsion Actor-Critic (ARAC), a population-based RL algorithm designed to explicitly manage diversity in the policy space. ARAC prevents premature convergence by pushing agents away from poor past behaviors and pulling them toward successful ones.
Core Components
- Normalizing Flows (NFs): Unlike standard methods that use simple Gaussian distributions, ARAC employs radial flows. This allows agents to represent complex, multimodal distributions, enabling them to consider multiple distinct types of actions simultaneously.
- Attraction-Repulsion (AR) Objective: The algorithm maintains an archive of past policies. Using a Kullback-Leibler (KL) divergence loss, it "attracts" the current agent to high-performing policies (proactive strategy) or "repulses" it from low-performing ones (reactive strategy).
- Diversity-Preserving Archive: A "niching" strategy ensures the archive remains diverse. Policies are clustered by fitness using k-means, and old policies are replaced within these clusters to maintain a broad set of "ancestors."
Key Insights: Policy Space and the "Gaussian Trap"
A central tenet of ARAC is that exploring in the behavior (policy) space is more effective than perturbing neural network weights (parameter space), as distinct weights can often result in redundant behaviors.
Furthermore, the use of Normalizing Flows is critical. In standard Gaussian policies, repulsion often leads to a simple divergence of means or an explosion in variance, which can be detrimental. Normalizing Flows provide the flexibility for an agent to "deform" its behavior around a repulsive obstacle without losing coherence.
Experimental Results
ARAC was evaluated on the MuJoCo locomotion suite and a custom SparseHumanoid-v2 benchmark, demonstrating superior performance and sample efficiency:
- Humanoid Locomotion: On the high-dimensional Humanoid (rllab) task, ARAC reached a 4,000-reward milestone in under 2 million steps, twice as fast as a standard SAC agent.
- Deceptive Rewards: In the HumanoidStandup environment, ARAC achieved a maximum average return of 175,000, significantly outperforming SAC (149,000) and other population-based methods like CERL (117,000).
- Sparse Rewards: In SparseHumanoid-v2, where rewards are only given for maintaining a specific height, ARAC was the only method to achieve non-zero performance, while all baselines failed to learn.
Conclusion
ARAC demonstrates that explicit diversity management in the policy space, enabled by Normalizing Flows and an Attraction-Repulsion mechanism, significantly enhances exploration and performance in complex continuous control tasks. Future work will explore integrating multi-objective optimization to further balance performance and diversity.
Summary updated