Neural Dynamic Policies for End-to-End Sensorimotor Learning

Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta, Deepak Pathak

Abstract

Neural Dynamic Policies (NDPs) represent a novel class of robot control policies that embed differentiable dynamical systems directly into deep neural network layers. By reparameterizing the action space to output high-level trajectory parameters—such as goals and forcing function weights—rather than raw motor commands at every timestep, NDPs enable robots to learn complex, physically plausible motions with significantly higher efficiency and inherent smoothness. We demonstrate that NDPs outperform standard reinforcement learning baselines across a variety of tasks, including digit writing, robotic manipulation, and the Meta-World benchmark.

Introduction

In standard deep reinforcement learning (RL), a robot's policy typically outputs "raw" actions, such as joint torques or velocities, at every single timestep. While flexible, this approach often leads to computationally expensive training, jittery movements, and significant challenges in "long-horizon" tasks where a single error early in a sequence can lead to failure.

Classical robotics addressed these issues using Dynamic Movement Primitives (DMPs)—mathematical frameworks that describe motion as a spring-damper system pulling toward a goal. However, traditional DMPs are difficult to tune manually and do not easily integrate with the end-to-end visual learning capabilities of modern deep learning. Neural Dynamic Policies bridge this gap by making the dynamical system itself a differentiable component of the neural network.

Methods: Architecture and Integration

The core innovation of NDPs is the reparameterization of the network's output. Instead of predicting a single action $a_t$ for state $s_t$, the network predicts the parameters of a trajectory.

The Neural Dynamic Policy architecture. The network outputs parameters (g, w) for a dynamical system, which is then integrated to produce a sequence of actions.
  1. The Architecture: The final layer of the policy network $(\Phi)$ outputs the parameters of a DMP: a goal state $(g)$ and a set of weights $(w)$ for a forcing function that shapes the path.
  2. Differentiable Dynamical Layer: The system treats the dynamical equations as a layer within the network. By using Euler integration to "unroll" the trajectory and deriving gradients with respect to the DMP parameters, the entire pipeline becomes end-to-end differentiable. This allows the robot to learn the "shape" of a motion directly from raw pixels or state inputs.
  3. Multi-Action Critic: To stabilize reinforcement learning (using PPO), the authors implemented a "multi-action critic." Since one network decision results in a sequence of $k$ actions (a rollout), this critic predicts value estimates for each step in that rollout, improving training stability.

Experimental Evaluation

The researchers evaluated NDPs across several challenging simulated environments, including 2D digit writing, MuJoCo-based robotic manipulation (Throwing and Picking), and the Meta-World (MT50) benchmark, which consists of 50 distinct motor tasks.

Comparison of vector fields for a digit-writing task. NDPs (right) produce a structured, globally convergent field compared to the localized, jittery field of a vanilla policy (left).

Key Results

  • Imitation Learning: In digit-writing tasks, NDPs were significantly more accurate and produced smoother paths than vanilla CNNs. As shown in the vector field analysis, NDPs create a global structure that pulls the agent toward the desired trajectory, whereas vanilla policies often produce localized, inconsistent commands.
  • Reinforcement Learning Efficiency: In tasks like Soccer and Faucet Opening, NDPs reached high success rates much faster than standard PPO baselines. In the Soccer task specifically, NDPs achieved nearly double the learning efficiency of the baseline.
  • Built-in Smoothness: Because NDPs are mathematically constrained by underlying differential equations, they are guaranteed to produce smooth trajectories. This is inherently safer for physical hardware and more effective for tasks involving momentum, such as throwing objects into a bin.

Discussion and Future Work

NDPs provide a natural form of temporal abstraction, grouping sequences of actions into meaningful units without the overhead of traditional hierarchical RL. By operating in a lower-dimensional parameter space, the network does not need to learn the physics of "smoothness" from scratch.

However, challenges remain. If the rollout length $(k)$ is too large, the reward signal can become sparse, making it difficult for the agent to credit specific parts of a trajectory for success. Future work may explore learning global parameters (like damping coefficients) dynamically or integrating other dynamical structures, such as Riemannian Motion Policies (RMPs), into the NDP framework.

Summary updated

BibTeX
@inproceedings{bahl2020neural,
  author = {Shikhar Bahl and Mustafa Mukadam and Abhinav Gupta and Deepak Pathak},
  title = {{Neural Dynamic Policies for End-to-End Sensorimotor Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=OqmJ7Og6rr5},
}
Figure