Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning

Thang Doan, Bogdan Mazoure, Audrey Durand, Joelle Pineau, R. Devon HjelmarXiv 2019 · OpenReview

Introduction

In continuous control reinforcement learning (RL), such as training a bipedal robot to walk, agents often struggle with deceptive reward landscapes. A common failure mode is convergence to a "local optimum"—for instance, a robot might learn to fall forward to gain a small distance reward rather than discovering the complex, globally optimal behavior of stable walking. Traditional exploration methods, which typically rely on adding random noise to parameters or maximizing the entropy of a single Gaussian policy, often fail to cover the action space effectively.

The Attraction-Repulsion Actor-Critic (ARAC) Framework

To address these challenges, we introduce Attraction-Repulsion Actor-Critic (ARAC), a population-based RL algorithm designed to explicitly manage diversity in the policy space. ARAC prevents premature convergence by pushing agents away from poor past behaviors and pulling them toward successful ones.

Core Components

  1. Normalizing Flows (NFs): Unlike standard methods that use simple Gaussian distributions, ARAC employs radial flows. This allows agents to represent complex, multimodal distributions, enabling them to consider multiple distinct types of actions simultaneously.
  2. Attraction-Repulsion (AR) Objective: The algorithm maintains an archive of past policies. Using a Kullback-Leibler (KL) divergence loss, it "attracts" the current agent to high-performing policies (proactive strategy) or "repulses" it from low-performing ones (reactive strategy).
  3. Diversity-Preserving Archive: A "niching" strategy ensures the archive remains diverse. Policies are clustered by fitness using k-means, and old policies are replaced within these clusters to maintain a broad set of "ancestors."
Conceptual overview of the Attraction-Repulsion mechanism (a) and the overall ARAC algorithmic loop (b).

Key Insights: Policy Space and the "Gaussian Trap"

A central tenet of ARAC is that exploring in the behavior (policy) space is more effective than perturbing neural network weights (parameter space), as distinct weights can often result in redundant behaviors.

Furthermore, the use of Normalizing Flows is critical. In standard Gaussian policies, repulsion often leads to a simple divergence of means or an explosion in variance, which can be detrimental. Normalizing Flows provide the flexibility for an agent to "deform" its behavior around a repulsive obstacle without losing coherence.

t-SNE visualizations of action distributions across different states, demonstrating how ARAC (lambda=1) maintains distinct modes compared to independent agents (lambda=0).

Experimental Results

ARAC was evaluated on the MuJoCo locomotion suite and a custom SparseHumanoid-v2 benchmark, demonstrating superior performance and sample efficiency:

  • Humanoid Locomotion: On the high-dimensional Humanoid (rllab) task, ARAC reached a 4,000-reward milestone in under 2 million steps, twice as fast as a standard SAC agent.
  • Deceptive Rewards: In the HumanoidStandup environment, ARAC achieved a maximum average return of 175,000, significantly outperforming SAC (149,000) and other population-based methods like CERL (117,000).
  • Sparse Rewards: In SparseHumanoid-v2, where rewards are only given for maintaining a specific height, ARAC was the only method to achieve non-zero performance, while all baselines failed to learn.

Conclusion

ARAC demonstrates that explicit diversity management in the policy space, enabled by Normalizing Flows and an Attraction-Repulsion mechanism, significantly enhances exploration and performance in complex continuous control tasks. Future work will explore integrating multi-objective optimization to further balance performance and diversity.

Summary updated

Abstract

Continuous control tasks in reinforcement learning are important because they provide an important framework for learning in high-dimensional state spaces with deceptive rewards, where the agent can easily become trapped into suboptimal solutions. One way to avoid local optima is to use a population of agents to ensure coverage of the policy space, yet learning a population with the "best" coverage is still an open problem. In this work, we present a novel approach to population-based RL in continuous control that leverages properties of normalizing flows to perform attractive and repulsive operations between current members of the population and previously observed policies. Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work, including Soft-Actor Critic (SAC).
BibTeX
@misc{doan2019attraction,
  author = {Thang Doan and Bogdan Mazoure and Audrey Durand and Joelle Pineau and R. Devon Hjelm},
  title = {{Attraction-Repulsion Actor-Critic for Continuous Control Reinforcement Learning}},
  year = {2019},
  eprint = {1909.07543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.07543},
}
Figure