PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations

Kuang-Huei Lee, Ofir Nachum, Tingnan Zhang, Sergio Guadarrama, Jie Tan, Wenhao Yu

TL;DR

PI-ARS combines the robust exploration of Evolution Strategies (ARS) with the efficiency of gradient-based representation learning (Predictive Information). By decoupling "learning to see" from "learning to move," this approach allows quadruped robots to master complex visual-locomotion tasks—like crossing moving platforms—achieving a 100% success rate on real-world stepping stones where previous methods struggled.

What Problem Does This Paper Address?

Teaching a legged robot to navigate uneven terrain using only vision (visual-locomotion) is a massive challenge. You have two conflicting requirements:

  1. High-capacity models: To process complex visual data (like depth images), you need deep Convolutional Neural Networks (CNNs) with thousands or millions of parameters.
  2. Robust optimization: Robot dynamics are non-linear and often non-differentiable (due to contact forces), making gradient-free "Evolution Strategy" (ES) algorithms like Augmented Random Search (ARS) very attractive.

The catch? ES algorithms scale poorly. As the number of parameters grows, the "search space" becomes too vast for random perturbations to find a good solution efficiently. Consequently, researchers often have to choose between using simple sensors with ES or complex vision with standard Reinforcement Learning (which often struggles with the "brittleness" of robot dynamics).

Methods

The authors introduce PI-ARS (Predictive Information Augmented Random Search), which splits the learning process into two specialized modules:

  • The Encoder (Gradient-Based): This module uses a CNN to compress high-dimensional depth images into a concise 128-dimensional vector. Instead of using the robot's reward to train this (which is "sparse" and slow), it uses a self-supervised objective called Predictive Information (PI). It learns by trying to predict future rewards and future visual observations (using an InfoNCE contrastive loss). Because this part is differentiable, it can be trained efficiently using standard backpropagation.
  • The Policy (Gradient-Free): This is a much smaller neural network that takes the compressed representation and decides which actions the robot should take. Because this network is small, ARS can optimize it effectively, leveraging its strengths in parameter-space exploration and parallelism without getting bogged down by the high dimensionality of raw video.

By alternating between updating the "vision" (PI) and the "movement" (ARS), the system learns a robust controller that understands its environment.

Datasets and Environments

The researchers evaluated PI-ARS using the Unitree Laikago quadruped robot platform in both simulation (PyBullet) and the real world.

Simulation Tasks: * Uneven Stepping Stones: Randomly placed stones with varying heights and gaps. * Quincuncial Piles: A 2D grid of small, elevated contact points. * Moving Platforms: Stepping stones that move horizontally and vertically, requiring the robot to time its steps. * Indoor Navigation: Navigating a scanned indoor environment cluttered with random obstacles.

Real-World Validation: The learned policies were deployed on a physical Laikago robot equipped with Intel RealSense depth cameras to cross a physical stepping stone course and navigate a cluttered room.

Key Insights and Ideas

  • Decoupling is the "Aha!" Moment: The core insight is that "learning to see" doesn't require a reward signal; it just requires an understanding of how the world changes over time. By offloading the heavy lifting of vision to a self-supervised gradient-based learner, the "blind" evolution strategy only has to solve the motor control problem.
  • Predictive Information as a Proxy for Task-Relevance: By forcing the encoder to predict the future, the robot naturally learns to ignore "visual noise" and focus on geometric features (like the edge of a stone) that are critical for survival.
  • Superiority over SAC: Interestingly, the authors found that standard gradient-based RL (like Soft Actor-Critic) failed to make any progress on these tasks, suggesting that the exploration capabilities of ARS are indeed vital for complex locomotion.

Key Results

  • Massive Performance Gains: In simulation, PI-ARS outperformed the standard ARS baseline by a wide margin. On the stepping stone task, it achieved a 48% higher mean return after 2 million episodes.
  • Real-World Robustness: The most striking result was the sim-to-real transfer. PI-ARS achieved a 100% success rate (10 out of 10 trials) on the real-world stepping stone course. For comparison, the previous state-of-the-art (which used a different ARS approach) only managed a 40% success rate on the same setup.
  • Dynamic Environments: PI-ARS was able to solve the "Moving Platforms" task, demonstrating that the predictive nature of the representation allows the robot to anticipate where a platform will be by the time its foot lands.

Limitations and Future Work

  • Obstacle Geometry: During real-world navigation, the robot occasionally collided with complex obstacle parts (like a chair arm). This happened because the simulation training only used simple box-shaped obstacles. Future work would involve training on more diverse 3D shapes.
  • Outdoor Terrains: The current study focused on indoor and controlled environments. Testing PI-ARS in more chaotic outdoor settings (mud, tall grass, etc.) remains an open challenge.
  • Sim-to-Real Gap: While the transfer was successful, it still required significant "domain randomization" (adding noise to the simulation). Developing representations that are even more invariant to the differences between sim and reality is a continuing goal.

Summary updated

Abstract

Evolution Strategy (ES) algorithms have shown promising results in training complex robotic control policies due to their massive parallelism capability, simple implementation, effective parameter-space exploration, and fast training time. However, a key limitation of ES is its scalability to large capacity models, including modern neural network architectures. In this work, we develop Predictive Information Augmented Random Search (PI-ARS) to mitigate this limitation by leveraging recent advancements in representation learning to reduce the parameter search space for ES. Namely, PI-ARS combines a gradient-based representation learning technique, Predictive Information (PI), with a gradient-free ES algorithm, Augmented Random Search (ARS), to train policies that can process complex robot sensory inputs and handle highly nonlinear robot dynamics. We evaluate PI-ARS on a set of challenging visual-locomotion tasks where a quadruped robot needs to walk on uneven stepping stones, quincuncial piles, and moving platforms, as well as to complete an indoor navigation task. Across all tasks, PI-ARS demonstrates significantly better learning efficiency and performance compared to the ARS baseline. We further validate our algorithm by demonstrating that the learned policies can successfully transfer to a real quadruped robot, for example, achieving a 100% success rate on the real-world stepping stone environment, dramatically improving prior results achieving 40% success.
BibTeX
@inproceedings{lee2022pi,
  author = {Kuang-Huei Lee and Ofir Nachum and Tingnan Zhang and Sergio Guadarrama and Jie Tan and Wenhao Yu},
  title = {{PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=rZso9mFDmbB},
}

Loading related content...

Figure