Using Simulation to Improve Sample-Efficiency of Bayesian Optimization for Bipedal Robots

Akshara Rai, Rika Antonova, Franziska Meier, Christopher G. Atkeson

TL;DR

Tuning high-dimensional locomotion controllers directly on bipedal hardware is risky and sample-constrained, while pure simulation optimization fails under the sim-to-real gap. This paper introduces a two-stage approach that uses offline simulations to learn domain-informed kernel transforms (via physiological gait determinants or trajectory-reconstructing neural networks) for Bayesian optimization (BO). By mapping controller parameters into a behavioral feature space and dynamically tracking hardware mismatch, the method learns robust walking controllers on the ATRIAS robot in under 10 hardware trials.

What Problem Does This Paper Address?

Dynamic bipedal walking is fragile: slight parameter changes can lead to catastrophic falls that damage the robot. Optimization objectives in robotics are typically non-convex, non-differentiable, and noisy. Deep reinforcement learning requires millions of samples, while standard black-box methods like Bayesian optimization (BO) scale poorly to higher dimensions ($>5\text{D}$) when evaluating directly on hardware.

Conversely, policies optimized purely in simulation frequently fail on real robots due to unmodeled dynamics (the "reality gap"). Existing multi-fidelity BO methods often inject simulation costs directly into the GP prior mean, which can dangerously mislead the optimizer when the simulation-to-reality mismatch is high.

Methods

The authors introduce a hybrid, sample-efficient BO framework that embeds simulation knowledge directly into the GP covariance (kernel) function rather than relying solely on cost priors:

  1. Simulation-Informed Feature Transforms ($\phi(x)$): Controller parameters $x$ are evaluated in simulation to extract behavioral summaries, warping the parameter space so that controllers with similar behaviors are clustered close together:
  2. Determinants of Gait ($\phi_{\text{DoG}}$): A hand-crafted 1D transform inspired by biomechanics (foot clearance, CoM vertical displacement, trunk pitch stability, and forward speed).
  3. Neural Network Kernel ($\phi_{\text{trajNN}}$): A 3-layer fully connected network trained offline on simulated trajectories to reconstruct low-dimensional summary features (walking time, CoM endpoints, energy consumption, torso angle) without requiring expert biomechanical metrics.
  4. Informed Kernel ($k_\phi$): A squared-exponential kernel computed over feature space $\phi(x)$: $$k_\phi(x_i, x_j) = \sigma_k^2 \exp\left(-\frac{1}{2}(\phi(x_i)-\phi(x_j))^T \text{diag}(\ell)^{-2}(\phi(x_i)-\phi(x_j))\right)$$ This introduces non-stationarity into the original parameter space while remaining stationary in $\phi$.

  5. Mismatch Compensation ($k_{\phi_{\text{adj}}}$): A secondary GP models the residual discrepancy between simulated and observed hardware behavior $d_x = \phi^{\text{sim}}(x) - \phi^{\text{hw}}(x)$. The kernel vector is augmented with this predicted mismatch $\bar{g}(x)$, penalizing simulated regions that fail on physical hardware.

Datasets

Rather than using static benchmark datasets, the authors generated synthetic datasets using physical simulations:

  • ATRIAS Simulators: 20,000 to 200,000 parameter samples generated via quasi-random Sobol grids across varying fidelity levels (full dynamics, simplified gear dynamics, and 2D planar models without boom dynamics).
  • Planar 7-link Biped Simulator: 100,000 simulated rollout summaries used to train the $\phi_{\text{trajNN}}$ representation.
  • Hardware Platform: Real-world trials conducted on the physical ATRIAS 2.5D bipedal robot attached to a boom.

Key Insights and Ideas

  • Kernel Warping Beats Cost Priors Under Mismatch: Placing simulation costs into the GP prior heavily biases BO toward solutions that look optimal in simulation. If that simulation is inaccurate, the optimizer struggles to escape. Warping the kernel space groups similar behaviors without biasing the objective value, allowing BO to quickly rule out large swathes of failing controllers.
  • Cost-Agnostic Behavioral Representations: Because $\phi_{\text{trajNN}}$ and $\phi_{\text{DoG}}$ summarize kinematic and dynamic trajectories rather than task rewards, the kernels remain valid if target speeds, penalty weights, or cost formulations change—eliminating the need to re-run simulations.
  • Closing the Reality Gap in Feature Space: Modeling sim-to-real mismatch directly in the behavioral transform space allows the GP to differentiate controllers that look identical in simulation but diverge on hardware.

Key Results

  • Hardware Experiments (ATRIAS):
  • 5D Reactive Controller: BO with $\phi_{\text{DoG}}$ found stable walking in 100% of runs within 3 trials, whereas standard BO with a squared exponential (SE) kernel required up to 20 trials to achieve 80% success.
  • 9D Controller: BO with $\phi_{\text{DoG}}$ succeeded in 3/3 runs within 5 trials, while baseline SE found zero walking solutions in 10 trials.
  • 9D Controller with Learned $\phi_{\text{trajNN}}$: Found successful controllers in 5/5 runs within 6 trials (vs. 2/5 runs in 10 trials for baseline SE).
  • Cost Agnosticism (7-link Biped, 16D Controller):
  • On non-smooth costs with severe fall penalties, $\phi_{\text{DoG}}$ and $\phi_{\text{trajNN}}$ achieved ~70% walking success within 100 trials, whereas baseline SE succeeded in only 1 out of 50 runs (2%).
  • Robustness to Severe Sim-to-Real Mismatch (50D VNMC Controller):
  • When tested on unmodeled boom dynamics and simplified gear mechanics, the adjusted kernel ($k_{\text{DoG}_{\text{adj}}}$) achieved 100% walking within 35 trials. Competitive prior-based algorithms (such as Intelligent Trial and Error / IT&E) succeeded in only 77% of runs due to map misalignments.

Limitations and Future Work

  • Offline Simulation Burden: Training $\phi_{\text{trajNN}}$ or generating dense behavioral grids requires substantial simulation compute upfront (e.g., 100k–200k simulation runs), even if it saves expensive hardware evaluations later.
  • Mismatch Adjustment for Learned Kernels: The online mismatch model was demonstrated primarily on 1D $\phi_{\text{DoG}}$; extending multi-output GP mismatch compensation to high-dimensional $\phi_{\text{trajNN}}$ outputs was identified as future work.
  • Theoretical Non-stationarity: While the formulation connects intuitively to multi-fidelity optimization, deeper theoretical analysis of non-stationarity in the transformed feature space remains open.

Summary updated

Abstract

Learning for control can acquire controllers for novel robotic tasks, paving the path for autonomous agents. Such controllers can be expert-designed policies, which typically require tuning of parameters for each task scenario. In this context, Bayesian optimization (BO) has emerged as a promising approach for automatically tuning controllers. However, sample-efficiency can still be an issue for high-dimensional policies on hardware. Here, we develop an approach that utilizes simulation to learn structured feature transforms that map the original parameter space into a domain-informed space. During BO, similarity between controllers is now calculated in this transformed space. Experiments on the ATRIAS robot hardware and simulation show that our approach succeeds at sample-efficiently learning controllers for multiple robots. Another question arises: What if the simulation significantly differs from hardware? To answer this, we create increasingly approximate simulators and study the effect of increasing simulation-hardware mismatch on the performance of Bayesian optimization. We also compare our approach to other approaches from literature, and find it to be more reliable, especially in cases of high mismatch. Our experiments show that our approach succeeds across different controller types, bipedal robot models and simulator fidelity levels, making it applicable to a wide range of bipedal locomotion problems.
BibTeX
@article{rai2019using,
  author = {Akshara Rai and Rika Antonova and Franziska Meier and Christopher G. Atkeson},
  title = {{Using Simulation to Improve Sample-Efficiency of Bayesian Optimization for Bipedal Robots}},
  journal = {J. Mach. Learn. Res.},
  year = {2019},
  url = {https://openreview.net/forum?id=1ujEJAAhTc},
}

Loading related content...

Figure