Task Feasibility Maximization Using Model-Free Policy Search and Model-Based Whole-Body Control

Ryan Lober, Olivier Sigaud, Vincent PadoisFrontiers Robotics AI 2020 · DOI · OpenReview

Abstract

Humanoid robots often struggle with complex tasks due to the gap between high-level motion planning and low-level physical constraints. This paper presents a hybrid control framework that integrates model-based Whole-Body Control (WBC) with model-free Bayesian Optimization to maximize task feasibility. By parameterizing trajectories and using simulation-to-reality bootstrapping, the system enables an iCub robot to autonomously learn a stable sit-to-stand transition in fewer than 30 trials.

Introduction

Humanoid robots are "highly redundant" systems, possessing many more joints than are strictly necessary for a single task. Controlling them usually involves Whole-Body Control (WBC), where a mathematical model calculates the exact torques needed to achieve multiple simultaneous goals. However, a significant gap remains in feasibility: high-level motion plans are often designed in a vacuum, ignoring the specific limits of the low-level controller or the messy realities of the physical world, such as friction or modeling errors. When a robot executes these "perfect" plans, it often becomes unstable. Traditionally, engineers fix this through tedious hand-tuning. This paper proposes a method for the robot to learn to fix its own infeasible plans.

Methods

The authors propose a "proof-of-concept optimization loop" that bridges the gap between planning and execution using a multi-layered approach:

  1. Model-Based WBC: At the lowest level (1 kHz), a momentum-based hierarchical controller handles the physics, ensuring the robot respects joint limits and maintains ground contact.
  2. Task Abstraction: To make learning practical, the motion is parameterized using a single Center of Mass (CoM) waypoint. This reduces a high-dimensional problem to just a few variables.
  3. Bayesian Optimization (BO): This model-free component uses a Gaussian Process to model the relationship between the CoM waypoint and a "feasibility cost," allowing for sample-efficient learning on real hardware.
The proposed hybrid control framework architecture, integrating high-level task planning with low-level whole-body control and a model-free optimization loop.

The system evaluates trials based on a cost function that considers tracking accuracy, goal attainment, and energy consumption. To minimize risk to the physical hardware, the optimizer is first trained in a Gazebo simulation environment before being transferred to the physical iCub humanoid robot.

Key Insights and Results

A primary finding of the study was the discovery of a "dip" strategy. To stand up successfully, the optimizer found that the robot needed to move its CoM downward initially. This counter-intuitive move increased ground reaction forces and shifted weight to the feet early, a behavior a human designer might not have included in a standard straight-line plan.

Time-lapse of the iCub humanoid robot executing a feasibility-optimized standing motion over 8 seconds.

Key results include:

  • From Failure to Success: The initial "straight-line" trajectory caused the iCub to fall every time, whereas the optimized trajectory achieved stability.
  • Efficiency: In simulation, the optimization converged within 24 rollouts. On the real robot, the "bootstrapped" trials showed a significant decrease in energy consumption, particularly in the torso pitch joint.
  • Practicality: By abstracting the motion, the researchers solved a complex stability problem with just 10–30 real-world trials.

Discussion and Future Work

While effective, the framework currently requires a human expert to decide which task parameters (like the CoM waypoint) need optimization. Future work will focus on automatically identifying these bottleneck variables. Additionally, while one waypoint worked for a sit-to-stand motion, more complex behaviors like parkour would likely require more waypoints, potentially challenging the sample efficiency of Bayesian Optimization. The authors also noted that the feasibility cost is somewhat binary—the robot either stands or falls—making smooth convergence difficult as the line between a wobble and a fall is very thin.

Summary updated

Abstract

Producing feasible motions for highly redundant robots, such as humanoids, is a complicated and high-dimensional problem. Model-based whole-body control of such robots can generate complex dynamic behaviors through the simultaneous execution of multiple tasks. Unfortunately, tasks are generally planned without close consideration for the underlying controller being used, or the other tasks being executed, and are often infeasible when executed on the robot. Consequently, there is no guarantee that the motion will be accomplished. In this work, we develop a proof-of-concept optimization loop which automatically improves task feasibility using model-free policy search in conjunction with model-based whole-body control. This combination allows problems to be solved, which would be otherwise intractable using simply one or the other. Through experiments on both the simulated and real iCub humanoid robot, we show that by optimizing task feasibility, initially infeasible complex dynamic motions can be realized—specifically, a sit-to-stand transition. These experiments can be viewed in the accompanying Video S1.
BibTeX
@article{lober2020task,
  author = {Ryan Lober and Olivier Sigaud and Vincent Padois},
  title = {{Task Feasibility Maximization Using Model-Free Policy Search and Model-Based Whole-Body Control}},
  journal = {Frontiers Robotics AI},
  year = {2020},
  doi = {10.3389/frobt.2020.00061},
}
Figure