TL;DR
Tuning high-dimensional locomotion controllers directly on bipedal hardware is risky and sample-constrained, while pure simulation optimization fails under the sim-to-real gap. This paper introduces a two-stage approach that uses offline simulations to learn domain-informed kernel transforms (via physiological gait determinants or trajectory-reconstructing neural networks) for Bayesian optimization (BO). By mapping controller parameters into a behavioral feature space and dynamically tracking hardware mismatch, the method learns robust walking controllers on the ATRIAS robot in under 10 hardware trials.
What Problem Does This Paper Address?
Dynamic bipedal walking is fragile: slight parameter changes can lead to catastrophic falls that damage the robot. Optimization objectives in robotics are typically non-convex, non-differentiable, and noisy. Deep reinforcement learning requires millions of samples, while standard black-box methods like Bayesian optimization (BO) scale poorly to higher dimensions ($>5\text{D}$) when evaluating directly on hardware.
Conversely, policies optimized purely in simulation frequently fail on real robots due to unmodeled dynamics (the "reality gap"). Existing multi-fidelity BO methods often inject simulation costs directly into the GP prior mean, which can dangerously mislead the optimizer when the simulation-to-reality mismatch is high.
Methods
The authors introduce a hybrid, sample-efficient BO framework that embeds simulation knowledge directly into the GP covariance (kernel) function rather than relying solely on cost priors:
- Simulation-Informed Feature Transforms ($\phi(x)$): Controller parameters $x$ are evaluated in simulation to extract behavioral summaries, warping the parameter space so that controllers with similar behaviors are clustered close together:
- Determinants of Gait ($\phi_{\text{DoG}}$): A hand-crafted 1D transform inspired by biomechanics (foot clearance, CoM vertical displacement, trunk pitch stability, and forward speed).
- Neural Network Kernel ($\phi_{\text{trajNN}}$): A 3-layer fully connected network trained offline on simulated trajectories to reconstruct low-dimensional summary features (walking time, CoM endpoints, energy consumption, torso angle) without requiring expert biomechanical metrics.
-
Informed Kernel ($k_\phi$): A squared-exponential kernel computed over feature space $\phi(x)$: $$k_\phi(x_i, x_j) = \sigma_k^2 \exp\left(-\frac{1}{2}(\phi(x_i)-\phi(x_j))^T \text{diag}(\ell)^{-2}(\phi(x_i)-\phi(x_j))\right)$$ This introduces non-stationarity into the original parameter space while remaining stationary in $\phi$.
-
Mismatch Compensation ($k_{\phi_{\text{adj}}}$): A secondary GP models the residual discrepancy between simulated and observed hardware behavior $d_x = \phi^{\text{sim}}(x) - \phi^{\text{hw}}(x)$. The kernel vector is augmented with this predicted mismatch $\bar{g}(x)$, penalizing simulated regions that fail on physical hardware.
Datasets
Rather than using static benchmark datasets, the authors generated synthetic datasets using physical simulations:
- ATRIAS Simulators: 20,000 to 200,000 parameter samples generated via quasi-random Sobol grids across varying fidelity levels (full dynamics, simplified gear dynamics, and 2D planar models without boom dynamics).
- Planar 7-link Biped Simulator: 100,000 simulated rollout summaries used to train the $\phi_{\text{trajNN}}$ representation.
- Hardware Platform: Real-world trials conducted on the physical ATRIAS 2.5D bipedal robot attached to a boom.
Key Insights and Ideas
- Kernel Warping Beats Cost Priors Under Mismatch: Placing simulation costs into the GP prior heavily biases BO toward solutions that look optimal in simulation. If that simulation is inaccurate, the optimizer struggles to escape. Warping the kernel space groups similar behaviors without biasing the objective value, allowing BO to quickly rule out large swathes of failing controllers.
- Cost-Agnostic Behavioral Representations: Because $\phi_{\text{trajNN}}$ and $\phi_{\text{DoG}}$ summarize kinematic and dynamic trajectories rather than task rewards, the kernels remain valid if target speeds, penalty weights, or cost formulations change—eliminating the need to re-run simulations.
- Closing the Reality Gap in Feature Space: Modeling sim-to-real mismatch directly in the behavioral transform space allows the GP to differentiate controllers that look identical in simulation but diverge on hardware.
Key Results
- Hardware Experiments (ATRIAS):
- 5D Reactive Controller: BO with $\phi_{\text{DoG}}$ found stable walking in 100% of runs within 3 trials, whereas standard BO with a squared exponential (SE) kernel required up to 20 trials to achieve 80% success.
- 9D Controller: BO with $\phi_{\text{DoG}}$ succeeded in 3/3 runs within 5 trials, while baseline SE found zero walking solutions in 10 trials.
- 9D Controller with Learned $\phi_{\text{trajNN}}$: Found successful controllers in 5/5 runs within 6 trials (vs. 2/5 runs in 10 trials for baseline SE).
- Cost Agnosticism (7-link Biped, 16D Controller):
- On non-smooth costs with severe fall penalties, $\phi_{\text{DoG}}$ and $\phi_{\text{trajNN}}$ achieved ~70% walking success within 100 trials, whereas baseline SE succeeded in only 1 out of 50 runs (2%).
- Robustness to Severe Sim-to-Real Mismatch (50D VNMC Controller):
- When tested on unmodeled boom dynamics and simplified gear mechanics, the adjusted kernel ($k_{\text{DoG}_{\text{adj}}}$) achieved 100% walking within 35 trials. Competitive prior-based algorithms (such as Intelligent Trial and Error / IT&E) succeeded in only 77% of runs due to map misalignments.
Limitations and Future Work
- Offline Simulation Burden: Training $\phi_{\text{trajNN}}$ or generating dense behavioral grids requires substantial simulation compute upfront (e.g., 100k–200k simulation runs), even if it saves expensive hardware evaluations later.
- Mismatch Adjustment for Learned Kernels: The online mismatch model was demonstrated primarily on 1D $\phi_{\text{DoG}}$; extending multi-output GP mismatch compensation to high-dimensional $\phi_{\text{trajNN}}$ outputs was identified as future work.
- Theoretical Non-stationarity: While the formulation connects intuitively to multi-fidelity optimization, deeper theoretical analysis of non-stationarity in the transformed feature space remains open.
Summary updated