LIPM-Guided Reinforcement Learning for Stable and Perceptive Locomotion in Bipedal Robots

H.-t. Su, H. L. Luo, Shunpeng Yang, Kaiwen Jiang, Wei Zhang, Hua Chen2025 · DOI

TL;DR

This paper introduces a novel reinforcement learning approach for bipedal robots to achieve stable and perceptive locomotion in challenging outdoor environments. By integrating insights from the Linear Inverted Pendulum Model (LIPM) into a new reward function and employing a vision-enhanced teacher-student learning framework, the robot can adapt to uneven terrain and disturbances while prioritizing stability.

What Problem Does This Paper Address?

Bipedal robots, despite their human-like form, face significant challenges in achieving stable and robust locomotion, especially in unstructured outdoor environments. Unlike their quadrupedal counterparts, bipedal robots have inherently underactuated dynamics and fewer contact points, making them more susceptible to external disturbances and complex terrain. Existing reinforcement learning (RL) methods have shown success in structured settings, but deploying them in the wild, where sensor noise, ground slippage, and unpredictable terrain are common, remains an underexplored area. This paper aims to bridge this gap by enabling bipedal robots to navigate diverse outdoor terrains with stability and perception, ensuring a consistent viewpoint for their onboard cameras.

Methods

The core of this work lies in a novel reward design and a vision-enhanced learning framework.

  • LIPM-Guided Reward Design: The authors leverage the Linear Inverted Pendulum Model (LIPM) to provide theoretical guidance for dynamic balance. This is translated into a "Stable Reward" component that regulates the center of mass (CoM) height and torso orientation, crucial for maintaining a stable viewpoint.
  • Reward Fusion Module (RFM): To adaptively balance the competing objectives of velocity tracking and stability, an RFM is employed. This mechanism prioritizes stability when necessary, ensuring the robot doesn't compromise balance for speed.
  • Decoupled Velocity Tracking Reward: To further refine velocity tracking, it's decoupled into direction and magnitude components. This prevents stability prioritization from degrading velocity tracking performance, especially in challenging scenarios.
  • Vision-CTS Framework: The learning framework is based on a Concurrent Teacher-Student (CTS) paradigm, enhanced with exteroceptive visual inputs from onboard cameras. This "Vision-CTS" framework allows student agents to learn from privileged information provided by teacher agents, while also incorporating visual data for terrain awareness.
  • Double-Critic Architecture: To improve training efficiency and robustness, a double-critic architecture is used. One critic evaluates locomotion objectives, while the other focuses on stability objectives, allowing for separate and more effective learning.

Datasets

The paper does not introduce new datasets. Instead, it utilizes:

  • Simulated Environments: Training and evaluation are conducted in simulation (Isaac Lab) with a terrain curriculum similar to existing work [7]. The simulation includes diverse terrains like stairs, slopes, discrete obstacles, and rough terrain.
  • Real-world Experiments: The trained policy is deployed on a point-foot bipedal robot and tested in various real-world outdoor environments, including grass fields, slopes, stairs, and gravel paths.

Key Insights and Ideas

  • LIPM as a Foundation for Perceptive Locomotion: The paper demonstrates that theoretical models like LIPM can provide valuable insights for designing rewards that directly contribute to perceptive capabilities (e.g., stable viewpoint for cameras).
  • Prioritizing Stability is Crucial for Bipeds: The RFM approach highlights that for bipedal robots, stability must be a primary concern, even if it means temporarily sacrificing velocity tracking. This is a key differentiator from methods that might solely focus on command following.
  • Decoupling Velocity Tracking Enhances Robustness: Separating direction and magnitude for velocity tracking proves effective in maintaining performance even when stability is prioritized, a common pitfall in simpler reward formulations.
  • Vision-CTS for Enhanced Terrain Awareness: Integrating visual perception into the teacher-student framework significantly improves the robot's ability to adapt to and navigate complex, unstructured terrains.

Key Results

  • Superior Terrain Adaptability: The proposed method achieved the highest success rates across various challenging terrains in simulation, outperforming ablations that removed the stable reward or stable critic.
  • Improved Disturbance Rejection: The robot demonstrated remarkable robustness in push recovery experiments, maintaining stability under both moderate and extreme external disturbances.
  • Effective Velocity Tracking: The decoupled velocity tracking reward formulation mitigated performance degradation when stability was prioritized, as shown by improved absolute velocity tracking error compared to a standard l2-norm reward.
  • Real-world Success: The robot successfully traversed diverse outdoor environments, including slippery slopes and mixed terrain, showcasing its adaptability and robust control in real-world scenarios.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the LIPM provides a simplified yet powerful framework, its assumptions can lead to mismatches in real-world scenarios with complex dynamics. The current approach focuses on point-foot robots, and extending it to other bipedal designs might require further adaptation.
  • Future Work: The authors plan to explore more expressive dynamic models like the Spring-Loaded Inverted Pendulum (SLIP) and Variable-Height Inverted Pendulum (VHIP) to capture a broader range of dynamics. This could lead to more dynamic and agile behaviors, enabling tasks like parkour under a unified perceptive-control framework. They also aim to further integrate model-based approaches with RL for enhanced dynamic principles.

Summary updated

Abstract

Achieving stable and robust perceptive locomotion for bipedal robots in unstructured outdoor environments remains a critical challenge due to complex terrain geometry and susceptibility to external disturbances. In this work, we propose a novel reward design inspired by the Linear Inverted Pendulum Model (LIPM) to enable perceptive and stable locomotion in the wild. The LIPM provides theoretical guidance for dynamic balance by regulating the center of mass (CoM) height and the torso orientation. These are key factors for terrainaware locomotion, as they help ensure a stable viewpoint for the robot's camera. Building on this insight, we design a reward function that promotes balance and dynamic stability while encouraging accurate CoM trajectory tracking. To adaptively trade off between velocity tracking and stability, we leverage the Reward Fusion Module (RFM) approach that prioritizes stability when needed. A double-critic architecture is adopted to separately evaluate stability and locomotion objectives, improving training efficiency and robustness. We validate our approach through extensive experiments on a bipedal robot in both simulation and real-world outdoor environments. The results demonstrate superior terrain adaptability, disturbance rejection, and consistent performance across a wide range of speeds and perceptual conditions.
BibTeX
@misc{su2025lipm,
  author = {H.-T. Su and H. L. Luo and Shunpeng Yang and Kaiwen Jiang and Wei Zhang and Hua Chen},
  title = {{LIPM-Guided Reinforcement Learning for Stable and Perceptive Locomotion in Bipedal Robots}},
  year = {2025},
  doi = {10.1109/humanoids65713.2025.11203201},
}
Figure