Using Simulation to Improve Sample-Efficiency of Bayesian Optimization for Bipedal Robots

Akshara Rai, Rika Antonova, Franziska Meier, Christopher G. Atkeson

Abstract

Learning for control can acquire controllers for novel robotic tasks, paving the path for autonomous agents. Such controllers can be expert-designed policies, which typically require tuning of parameters for each task scenario. In this context, Bayesian optimization (BO) has emerged as a promising approach for automatically tuning controllers. However, when performing BO on hardware for high-dimensional policies, sample-efficiency can be an issue. Here, we develop an approach that utilizes simulation to map the original parameter space into a domain-informed space. During BO, similarity between controllers is now calculated in this transformed space. Experiments on the ATRIAS robot hardware and another bipedal robot simulation show that our approach succeeds at sample-efficiently learning controllers for multiple robots. Another question arises: What if the simulation significantly differs from hardware? To answer this, we create increasingly approximate simulators and study the effect of increasing simulation-hardware mismatch on the performance of Bayesian optimization. We also compare our approach to other approaches from literature, and find it to be more reliable, especially in cases of high mismatch. Our experiments show that our approach succeeds across different controller types, bipedal robot models and simulator fidelity levels, making it applicable to a wide range of bipedal locomotion problems.

Introduction

Learning controllers for bipedal robots is a high-stakes challenge. Unlike digital agents in a video game, physical robots like the 64kg ATRIAS are expensive, fragile, and time-consuming to reset after a fall. While Deep Reinforcement Learning (DRL) has shown success in simulation, it often requires millions of samples—an impossible feat for physical hardware.

Bayesian Optimization (BO) is a more data-efficient alternative, but its performance typically degrades as the number of controller parameters (dimensionality) increases. Furthermore, while simulators are fast, they are rarely perfect. This "sim-to-real" mismatch often results in controllers that perform beautifully in a simulator but cause the real robot to collapse immediately. This paper presents a framework to bridge this gap by leveraging simulation to reshape how BO "thinks" about controller parameters.

Methods

The core innovation is the use of Informed Kernels. In standard BO, a "kernel" measures the similarity between two sets of controller parameters. Usually, this is a simple distance metric (like Squared Exponential), which assumes that if two sets of numbers are close, the resulting robot behavior will be similar. In bipedal locomotion, this is often false; a tiny change in a feedback gain can be the difference between a steady stride and a faceplant.

Overview of the proposed framework: simulation data is used to build a feature transform for Bayesian Optimization on the physical robot.

The authors propose two ways to build kernels informed by simulation:

  1. Determinants of Gait (DoG) Transform: A feature transform based on physiological human walking heuristics (e.g., swing leg retraction, center-of-mass height). Controllers are scored in simulation, and BO measures similarity based on these scores rather than raw parameters.
  2. Neural Network (NN) Transform: A more general approach where a neural network is trained on massive simulation data (0.5 million points) to predict "trajectory summaries" (energy use, torso angle, etc.) from controller parameters. BO then operates in this learned "behavior space."

Crucially, the framework introduces Kernel Adjustment for Mismatch. If the robot fails on hardware despite a high simulation score, the system uses a Gaussian Process to model the mismatch and adjusts the kernel to steer the optimization toward regions where the simulation and hardware are more likely to agree.

Experimental Setup

The study utilizes a mix of hardware and simulated environments to validate the approach:

  • Hardware: The ATRIAS robot, a parallel bipedal platform.
  • Simulation: A 7-link serial biped model, used to test the generalizability of the NN-based transform.
  • Simulators of Varying Fidelity: To test robustness, the authors created "degraded" simulators for ATRIAS, including one with simplified gear dynamics and a 2D-only version that ignored the robot's stabilizing boom.
The ATRIAS bipedal robot platform used for real-world hardware experiments.

Key Results

  • Extreme Efficiency: On the 5D ATRIAS controller, BO with the DoG kernel found walking gaits in 100% of runs within just 3 trials. Standard BO succeeded only 60% of the time after 10 trials.
  • Scaling to High Dimensions: In a 9D controller task where standard BO failed completely, the informed kernel approach found successful gaits in every run within 5 trials.
  • Robustness to "Bad" Simulators: Even when using the most inaccurate simulator (no boom, simplified gears), the adjusted DoG kernel outperformed competitive baselines like Intelligent Trial and Error (IT&E), proving it can handle severe modeling errors.

Discussion and Conclusion

The primary insight of this work is that similarity should be measured by what the robot does (behavior), not the raw values in the controller. By mapping parameters to a behavior-based space, the optimization landscape becomes much smoother and easier to search. Instead of trying to build a "perfect" simulator, the authors use the simulator to bias the search; the simulator doesn't need to be perfectly accurate, it just needs to be correlated with reality.

Future work may address the pre-computation cost of the NN-based transform and expand the mismatch modeling from one dimension to multiple dimensions. Overall, the experiments show that this approach succeeds across different controller types, bipedal models, and simulator fidelity levels, making it applicable to a wide range of bipedal locomotion problems.

Summary updated

Abstract

Learning for control can acquire controllers for novel robotic tasks, paving the path for autonomous agents. Such controllers can be expert-designed policies, which typically require tuning of parameters for each task scenario. In this context, Bayesian optimization (BO) has emerged as a promising approach for automatically tuning controllers. However, when performing BO on hardware for high-dimensional policies, sample-efficiency can be an issue. Here, we develop an approach that utilizes simulation to map the original parameter space into a domain-informed space. During BO, similarity between controllers is now calculated in this transformed space. Experiments on the ATRIAS robot hardware and another bipedal robot simulation show that our approach succeeds at sample-efficiently learning controllers for multiple robots. Another question arises: What if the simulation significantly differs from hardware? To answer this, we create increasingly approximate simulators and study the effect of increasing simulation-hardware mismatch on the performance of Bayesian optimization. We also compare our approach to other approaches from literature, and find it to be more reliable, especially in cases of high mismatch. Our experiments show that our approach succeeds across different controller types, bipedal robot models and simulator fidelity levels, making it applicable to a wide range of bipedal locomotion problems.
BibTeX
@article{rai2019using,
  author = {Akshara Rai and Rika Antonova and Franziska Meier and Christopher G. Atkeson},
  title = {{Using Simulation to Improve Sample-Efficiency of Bayesian Optimization for Bipedal Robots}},
  journal = {J. Mach. Learn. Res.},
  year = {2019},
  url = {https://openreview.net/forum?id=1ujEJAAhTc},
}
Figure