Introduction
The Buoyancy Assisted Lightweight Legged Unit (BALLU) represents a radical departure from traditional robotics. Composed of helium-filled balloons and slender, cable-driven legs, BALLU is inherently safe for human interaction due to its low mass and "soft" construction. However, this same lightweight nature makes it exceptionally difficult to control. Traditional physics simulators often fail to capture the nuances of its movement, leading to a significant "sim-to-real gap." This paper introduces a novel two-stage pipeline—combining Actuator System Identification (SysID) and a reinforcement learning-based residual physics model called EnvMimic—to achieve stable and accurate sim-to-real transfer for this unique class of robots.
The Challenge of Buoyancy-Assisted Dynamics
The primary obstacles to accurate simulation for BALLU are two-fold: complex actuation and sensitive aerodynamics. The robot's legs are driven by a combination of cables and servos, a setup that introduces non-linear friction and cable slack that are notoriously difficult to model using standard mathematical equations.
Furthermore, because BALLU is so light, it is highly susceptible to minute air currents and complex drag forces. These "residual physics" dominate the robot's dynamics, yet they are often omitted or oversimplified in "vanilla" simulators like PyBullet. Consequently, policies trained in standard simulation environments frequently fail when deployed on physical hardware, as they cannot account for the unpredictable forces acting on the balloons.
Methodology
To address these challenges, the researchers developed a two-stage approach to "upgrade" the simulator's fidelity before training control policies.
1. Actuator System Identification (SysID)
Instead of relying on idealized linear models for the motors, the team collected real-world hardware data to understand the relationship between motor commands and joint angles. They employed the L-BFGS-B optimization algorithm to fit non-linear polynomial curves to this data. This process was essential for capturing the specific mechanical "quirks" of the cable-driven system, including the inherent asymmetry between the left and right legs caused by hardware imperfections.
2. EnvMimic: Residual Physics Learning
The core innovation of this work is EnvMimic. While previous attempts at bridging the sim-to-real gap often used Supervised Learning (SL) to predict the discrepancy between simulated and real-world states, SL models often fail to generalize when the robot encounters a state slightly outside its training data.
EnvMimic instead utilizes Deep Reinforcement Learning (Proximal Policy Optimization, or PPO) to train an "external force policy." This policy learns to apply subtle, invisible forces to the simulated robot's balloons to ensure its trajectory matches real-world ground-truth data recorded via a Vicon motion capture system. By treating the "reality gap" as an external force to be learned via imitation, the simulator becomes a high-fidelity reflection of the real world. This "Inside-Out Imitation" approach trains the environment's forces to match the motion, rather than training the robot's internal controller to match a motion.
Experimental Results
The effectiveness of the EnvMimic pipeline was demonstrated through forward walking and turning tasks on the physical BALLU hardware. The researchers compared their method against several baselines, including standard Domain Randomization (DR).
- Forward Walking: The policy trained with EnvMimic achieved a travel distance of 1.12 meters on hardware. In contrast, baselines using only SysID or standard DR struggled significantly, often veering off-course and traveling only 0.27m to 0.56m.
- Turning Accuracy: In turning tasks, the EnvMimic-trained model reduced orientation error (the difference between intended and actual yaw) to a mere 3.50°. Other models exhibited errors as high as 38°, highlighting the importance of capturing the robot's sensitive aerodynamics.
- Trajectory Fidelity: Qualitatively, the augmented simulation's trajectories closely mirrored the "wobbly" and sensitive nature of the real BALLU, a characteristic that vanilla simulators completely missed.
Discussion and Insights
A critical finding of this research is that RL-based residual physics models generalize significantly better than those based on SL. Because the RL agent explores the state space during its training process, it develops a more robust understanding of the dynamics, preventing the "drift" commonly seen in SL models. Additionally, the SysID revealed that capturing mechanical asymmetry is crucial; without accounting for the differences between the left and right legs, the robot is unable to maintain a straight path.
Limitations and Future Work
While EnvMimic represents a significant step forward, some limitations remain. Currently, a new residual physics model must be trained for each specific task (e.g., one for walking, one for turning). Future research aims to develop a more generalized physics model. Furthermore, the current approach does not account for the gradual loss of buoyancy as helium leaks from the balloons over time. Implementing "lifelong learning" to update the physics model dynamically could address these temporal changes. Despite these challenges, the combination of non-linear SysID and RL-based residual force modeling provides a powerful framework for the sim-to-real transfer of highly sensitive, lightweight robotic systems.
Summary updated