Learned graphical models for probabilistic planning provide a new class of movement primitives

Elmar A. Rückert, Gerhard Neumann, Marc Toussaint, Wolfgang Maass

Introduction

In the fields of robotics and motor control, movement has traditionally been viewed as a sequence of pre-defined trajectories. However, biological systems exhibit a level of flexibility and efficiency that fixed scripts cannot match. This paper, published in Frontiers in Computational Neuroscience, introduces Planning Movement Primitives (PMPs)—a framework where movement is generated dynamically by an internal planner that optimizes for goals and costs rather than following a rigid path.

The Challenge of Redundancy and Rigidity

Movement in both biological and robotic systems is inherently redundant; there are infinite ways to achieve a single goal, such as reaching for an object. Traditional Movement Primitives (MPs), like Dynamic Movement Primitives (DMPs), typically encode a specific "reference trajectory." While effective in controlled environments, these methods suffer from two primary limitations:

  1. Rigidity: They struggle to adapt when targets or starting positions change without complex, manual adjustments.
  2. Inefficiency under Noise: They attempt to correct every minor deviation from the path. In contrast, biological systems follow the "minimum intervention principle," only correcting errors that jeopardize the final objective.

Methodology: Movement as Probabilistic Inference

The core innovation of this research is the conceptualization of movement as a probabilistic inference problem. Instead of a differential equation defining a path, a PMP is structured as a probabilistic graphical model.

Probabilistic graphical model of a Planning Movement Primitive (PMP) showing the relationship between states (x), controls (u), and the intrinsic cost function.

The system utilizes the Approximate Inference Control (AICO) algorithm to determine the most likely sequence of actions given a set of goals and effort constraints. Key components include:

  • Learning the Cost Function: Rather than learning a trajectory, the system uses Reinforcement Learning (specifically CMA-ES) to optimize the parameters of an intrinsic cost function. This allows the robot to understand why a movement is performed (e.g., passing through a specific point) rather than just how it looks.
  • Model-Based Learning: The robot learns its own dynamics through experience using Locally Weighted Regression (LWR), enabling it to predict the outcomes of its actions within the planner.

Experimental Evaluation

The authors tested PMPs against traditional DMPs in two simulated scenarios: a 1D via-point task and a complex 4-link humanoid balancing task. In the latter, a 70kg humanoid must recover its balance after being pushed by an external force.

Key Results

  • Accelerated Learning: In the humanoid task, PMPs converged on successful policies an order of magnitude faster than DMPs.
  • Superior Performance: PMPs achieved significantly lower costs. In the balancing task, PMPs reached a cost of ~307, compared to ~568 for the best DMP.
  • Zero-Shot Adaptation: PMPs demonstrated remarkable generalization. Because they are planners, they can adapt to new target positions or starting configurations immediately by re-running the inference process without additional training.
  • Robustness to Noise: By adhering to the minimum intervention principle, PMPs remained stable under control noise that caused DMP-based systems to fail.

Discussion and Future Directions

While PMPs offer significant advantages in flexibility and learning speed, they come with a higher computational cost. Generating a trajectory takes nearly ten times longer than DMP-based methods, though still under one second. Additionally, as local optimizers, they may struggle with highly complex paths like mazes without a suitable initial guess.

Future research aims to explore imitation learning, where PMPs could infer the underlying "intent" or cost function from human demonstrations. The integration of multiple PMPs to handle multi-stage tasks also remains a promising avenue for developing more sophisticated robotic behaviors.

Summary updated

BibTeX
@article{ruckert2012learned,
  author = {Elmar A. Rückert and Gerhard Neumann and Marc Toussaint and Wolfgang Maass},
  title = {{Learned graphical models for probabilistic planning provide a new class of movement primitives}},
  journal = {Frontiers Comput. Neurosci.},
  year = {2012},
  doi = {10.3389/fncom.2012.00097},
}
Figure