Abstract
Teaching legged robots to navigate complex, uneven terrain using high-dimensional visual input remains a significant challenge in robotics. While Evolution Strategies (ES) like Augmented Random Search (ARS) offer robust optimization for non-differentiable robot dynamics, they scale poorly with the high-dimensional parameter spaces required for visual processing. This paper presents PI-ARS (Predictive Information Augmented Random Search), a framework that decouples representation learning from policy optimization. By utilizing a gradient-based encoder trained via Predictive Information (PI) and a gradient-free policy optimized via ARS, the system achieves superior performance in both simulated and real-world visual-locomotion tasks, including dynamic environments with moving platforms.
Introduction
Visual-locomotion—the ability of a legged robot to navigate using vision—requires the integration of complex perception and robust motor control. Traditionally, researchers face a dilemma: use high-capacity models (like deep CNNs) that are difficult to optimize with gradient-free methods, or use simpler sensors that may fail in complex environments.
Evolution Strategies, specifically Augmented Random Search (ARS), are highly effective for robot locomotion because they are invariant to non-differentiable contact forces and provide excellent exploration. However, as the number of parameters in a neural network increases to accommodate visual data, the search space for ARS becomes prohibitively large. PI-ARS addresses this by separating the "learning to see" phase from the "learning to move" phase, allowing each to use the most efficient optimization strategy.
Methodology
The PI-ARS architecture consists of two primary components designed to work in tandem:
1. The Gradient-Based Encoder
To process high-dimensional depth images, a Convolutional Neural Network (CNN) encoder is employed. Rather than relying on sparse task rewards—which provide little signal for visual features—the encoder is trained using a self-supervised objective called Predictive Information (PI). Using an InfoNCE contrastive loss, the encoder learns to compress visual input into a 128-dimensional latent vector by predicting future rewards and future visual observations. Because this objective is differentiable, the encoder can be trained efficiently using standard backpropagation.
2. The Gradient-Free Policy
The compressed representation from the encoder is fed into a relatively small policy network. Because the input dimensionality is reduced and the policy network is compact, ARS can effectively optimize the control parameters. This allows the robot to leverage the robust exploration capabilities of ES without the computational burden of high-dimensional parameter perturbations. The system alternates between updating the vision module (PI) and the movement module (ARS), resulting in a controller that is both perceptive and agile.
Experimental Setup
The researchers evaluated PI-ARS using the Unitree Laikago quadruped platform. Testing was conducted in the PyBullet simulation environment and validated through real-world deployments.
Simulation Tasks
- Uneven Stepping Stones: Navigating randomly placed stones with varying heights and gaps.
- Quincuncial Piles: Crossing a grid of small, elevated contact points.
- Moving Platforms: A dynamic task requiring the robot to time its steps onto platforms moving horizontally and vertically.
- Indoor Navigation: Navigating through scanned indoor environments filled with random obstacles.
Real-World Validation
The learned policies were transferred to a physical Laikago robot equipped with Intel RealSense depth cameras. The robot was tasked with crossing a physical stepping stone course and navigating a cluttered office environment.
Results and Discussion
Performance and Efficiency
In simulation, PI-ARS demonstrated a significant advantage over standard ARS. On the stepping stone task, PI-ARS achieved a 48% higher mean return after 2 million episodes compared to the baseline. Notably, standard gradient-based Reinforcement Learning methods, such as Soft Actor-Critic (SAC), failed to make significant progress on these tasks, highlighting the necessity of the exploration provided by ARS.
Sim-to-Real Robustness
The most significant result was the framework's success in real-world transfer. PI-ARS achieved a 100% success rate (10 out of 10 trials) on the physical stepping stone course. This represents a substantial improvement over previous state-of-the-art ARS methods, which achieved only a 40% success rate on the same setup. The predictive nature of the representation also allowed the robot to succeed in dynamic tasks, such as the moving platforms, by anticipating future environmental states.
Limitations
Despite its success, PI-ARS faced challenges with complex obstacle geometries. In real-world navigation, the robot occasionally collided with intricate objects like chair arms, likely because the simulation training utilized simplified box-shaped obstacles. Furthermore, while the system performed well indoors, its efficacy in chaotic outdoor environments (e.g., tall grass or mud) remains to be tested.
Conclusion
PI-ARS demonstrates that decoupling representation learning from policy optimization is a powerful strategy for visual-locomotion. By using Predictive Information to handle the high-dimensional "vision" problem and ARS to handle the "control" problem, the framework enables quadruped robots to master tasks that were previously unreachable for evolution-based methods. Future work will focus on expanding the diversity of training environments to further bridge the sim-to-real gap and improve obstacle avoidance in complex settings.
Summary updated