Abstract
Traditional robotic controllers often focus on finding a single optimal solution for a task, making them fragile to environmental changes or obstacles. This paper introduces Generative Adversarial Policy Networks (GPNs), a framework for creating a continuous, generative "behavioural repertoire." By modeling the manifold of successful policies, GPNs allow robots to sample an infinite variety of strategies for a given goal. This approach provides a compact, memory-efficient representation that enhances robustness, enabling robots to bypass unexpected obstacles by switching between diverse movement patterns.
Introduction
In robotics, the ability to solve a task in multiple ways is a key component of resilience. While standard Reinforcement Learning (RL) typically converges to a single optimal policy, real-world environments often present unforeseen challenges that render a specific solution ineffective. Existing repertoire-based methods, such as Quality-Diversity (QD) algorithms, generate libraries of diverse solutions but are often constrained by memory limits and fixed discrete examples.
This work proposes GPNs to learn a continuous mapping from a latent space to a policy space. By leveraging the generative capabilities of GANs, the system can produce an unbounded number of successful behaviors for any given target, facilitating rapid adaptation in dynamic environments.
Methodology
The GPN framework follows a two-stage developmental process. First, a bootstrapping phase uses a QD search algorithm to collect a dataset of approximately 15,000 diverse throwing policies in a simulated environment using a Baxter robot.
Second, the GPN architecture is trained on this dataset. The generator takes a noise vector and a target goal to produce a policy, while a discriminator learns to distinguish these from the original QD dataset. To ensure goal-directed behavior, a goal regularization term is added to the loss function, requiring the generator to reconstruct the target coordinates from the resulting policy. Policies are represented as 15-dimensional vectors defining joint states at the point of release, which are then interpolated into trajectories.
Experiments and Results
The model was evaluated using a Baxter robot in the Gazebo simulator and validated on a physical platform. The primary task involved throwing a ball to various target points on a 5x5 grid.
Diversity and Robustness
GPNs demonstrated a significantly higher degree of behavioral diversity compared to traditional library-based approaches. When faced with random environmental occlusions, the GPN-equipped robot could iteratively sample its internal generative model to find collision-free paths. Quantitative analysis showed that GPN outperformed baselines like Kernel Density Estimation (KDE) and Bayesian Optimization in finding successful paths in complex, obstacle-rich environments.
Real-World Transfer
The system successfully enabled a physical Baxter robot to hit targets even when its primary throwing path was blocked by a wall. By sampling alternative behaviors from the GPN, the robot could find a trajectory that cleared the obstacle while still reaching the intended goal.
Conclusion
GPNs offer a promising path toward more versatile robotic systems by treating policy spaces as continuous manifolds. By moving away from single-solution paradigms toward generative repertoires, robots can maintain high success rates even in the presence of unforeseen physical constraints. Future work will focus on extending this to closed-loop control and online learning to further enhance adaptability in real-time interactions.
Summary updated