Residual Physics Learning and System Identification for Sim-to-real Transfer of Policies on Buoyancy Assisted Legged Robots

Nitish Sontakke, Hosik Chae, Sangjoon Lee, Tianle Huang, Dennis W. Hong, Sehoon Ha

Introduction

The Buoyancy Assisted Lightweight Legged Unit (BALLU) represents a radical departure from traditional robotics. Composed of helium-filled balloons and slender, cable-driven legs, BALLU is inherently safe for human interaction due to its low mass and "soft" construction. However, this same lightweight nature makes it exceptionally difficult to control. Traditional physics simulators often fail to capture the nuances of its movement, leading to a significant "sim-to-real gap." This paper introduces a novel two-stage pipeline—combining Actuator System Identification (SysID) and a reinforcement learning-based residual physics model called EnvMimic—to achieve stable and accurate sim-to-real transfer for this unique class of robots.

The Challenge of Buoyancy-Assisted Dynamics

The primary obstacles to accurate simulation for BALLU are two-fold: complex actuation and sensitive aerodynamics. The robot's legs are driven by a combination of cables and servos, a setup that introduces non-linear friction and cable slack that are notoriously difficult to model using standard mathematical equations.

Furthermore, because BALLU is so light, it is highly susceptible to minute air currents and complex drag forces. These "residual physics" dominate the robot's dynamics, yet they are often omitted or oversimplified in "vanilla" simulators like PyBullet. Consequently, policies trained in standard simulation environments frequently fail when deployed on physical hardware, as they cannot account for the unpredictable forces acting on the balloons.

Methodology

To address these challenges, the researchers developed a two-stage approach to "upgrade" the simulator's fidelity before training control policies.

1. Actuator System Identification (SysID)

Instead of relying on idealized linear models for the motors, the team collected real-world hardware data to understand the relationship between motor commands and joint angles. They employed the L-BFGS-B optimization algorithm to fit non-linear polynomial curves to this data. This process was essential for capturing the specific mechanical "quirks" of the cable-driven system, including the inherent asymmetry between the left and right legs caused by hardware imperfections.

2. EnvMimic: Residual Physics Learning

The core innovation of this work is EnvMimic. While previous attempts at bridging the sim-to-real gap often used Supervised Learning (SL) to predict the discrepancy between simulated and real-world states, SL models often fail to generalize when the robot encounters a state slightly outside its training data.

EnvMimic instead utilizes Deep Reinforcement Learning (Proximal Policy Optimization, or PPO) to train an "external force policy." This policy learns to apply subtle, invisible forces to the simulated robot's balloons to ensure its trajectory matches real-world ground-truth data recorded via a Vicon motion capture system. By treating the "reality gap" as an external force to be learned via imitation, the simulator becomes a high-fidelity reflection of the real world. This "Inside-Out Imitation" approach trains the environment's forces to match the motion, rather than training the robot's internal controller to match a motion.

Experimental Results

The effectiveness of the EnvMimic pipeline was demonstrated through forward walking and turning tasks on the physical BALLU hardware. The researchers compared their method against several baselines, including standard Domain Randomization (DR).

  • Forward Walking: The policy trained with EnvMimic achieved a travel distance of 1.12 meters on hardware. In contrast, baselines using only SysID or standard DR struggled significantly, often veering off-course and traveling only 0.27m to 0.56m.
  • Turning Accuracy: In turning tasks, the EnvMimic-trained model reduced orientation error (the difference between intended and actual yaw) to a mere 3.50°. Other models exhibited errors as high as 38°, highlighting the importance of capturing the robot's sensitive aerodynamics.
  • Trajectory Fidelity: Qualitatively, the augmented simulation's trajectories closely mirrored the "wobbly" and sensitive nature of the real BALLU, a characteristic that vanilla simulators completely missed.

Discussion and Insights

A critical finding of this research is that RL-based residual physics models generalize significantly better than those based on SL. Because the RL agent explores the state space during its training process, it develops a more robust understanding of the dynamics, preventing the "drift" commonly seen in SL models. Additionally, the SysID revealed that capturing mechanical asymmetry is crucial; without accounting for the differences between the left and right legs, the robot is unable to maintain a straight path.

Limitations and Future Work

While EnvMimic represents a significant step forward, some limitations remain. Currently, a new residual physics model must be trained for each specific task (e.g., one for walking, one for turning). Future research aims to develop a more generalized physics model. Furthermore, the current approach does not account for the gradual loss of buoyancy as helium leaks from the balloons over time. Implementing "lifelong learning" to update the physics model dynamically could address these temporal changes. Despite these challenges, the combination of non-linear SysID and RL-based residual force modeling provides a powerful framework for the sim-to-real transfer of highly sensitive, lightweight robotic systems.

Summary updated

Abstract

The light and soft characteristics of Buoyancy Assisted Lightweight Legged Unit (BALLU) robots have a great potential to provide intrinsically safe interactions in environments involving humans, unlike many heavy and rigid robots. However, their unique and sensitive dynamics impose challenges to obtaining robust control policies in the real world. In this work, we demonstrate robust sim-to-real transfer of control policies on the BALLU robots via system identification and our novel residual physics learning method, Environment Mimic (EnvMimic). First, we model the nonlinear dynamics of the actuators by collecting hardware data and optimizing the simulation parameters. Rather than relying on standard supervised learning formulations, we utilize deep reinforcement learning to train an external force policy to match real-world trajectories, which enables us to model residual physics with greater fidelity. We analyze the improved simulation fidelity by comparing the simulation trajectories against the real-world ones. We finally demonstrate that the improved simulator allows us to learn better walking and turning policies that can be successfully deployed on the hardware of BALLU.
BibTeX
@inproceedings{sontakke2023residual,
  author = {Nitish Sontakke and Hosik Chae and Sangjoon Lee and Tianle Huang and Dennis W. Hong and Sehoon Ha},
  title = {{Residual Physics Learning and System Identification for Sim-to-real Transfer of Policies on Buoyancy Assisted Legged Robots}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342062},
}
Figure