Physical Derivatives: Computing policy gradients by physical forward-propagation

Arash Mehrjou, Ashkan Soleymani, Stefan Bauer, Bernhard SchölkopfarXiv 2022 · OpenReview

Abstract

This paper introduces Physical Derivatives, a novel approach for policy optimization that bridges the gap between model-free and model-based reinforcement learning. By directly estimating the sensitivity of a physical system's trajectories to policy perturbations—rather than learning a full transition model—the method enables sample-efficient, task-agnostic control without the bias of analytical models. We demonstrate the effectiveness of this approach on a custom robotic finger platform, achieving high predictive accuracy and successful zero-shot planning.

Introduction

In reinforcement learning (RL), practitioners typically face a stark trade-off. Model-free methods are robust to complex dynamics but require a massive number of interactions with the environment, which is often impractical for real-world robotics. Conversely, model-based methods are more sample-efficient but suffer from "model bias"; if the simulator or mathematical model deviates even slightly from reality, the learned policy may fail when deployed on hardware.

This research identifies a middle ground: we do not necessarily need a full model of a robot's dynamics to improve its behavior. Instead, we only require the sensitivity of the robot's path to changes in its controller. This sensitivity—the mapping of how a small tweak in motor parameters affects the resulting trajectory—is what the authors term the "Physical Derivative."

Methodology

The core innovation is computing policy gradients through physical forward-propagation. Instead of backpropagating through a neural network model, the researchers use the physical world itself as the "computer."

The Learning Pipeline

  1. Perturbation: A "nominal" policy (baseline controller) is subjected to small, random perturbations in its parameters ($\theta$).
  2. Observation: These perturbed policies are executed on the real robot, and the resulting trajectories are recorded.
  3. Finite Differences: By comparing perturbed trajectories to the nominal one, the system estimates directional derivatives, mapping policy changes ($\delta\theta$) to state changes ($\delta\mathbf{x}$) across time.
  4. Probabilistic Learning: To handle real-world noise, Gaussian Processes (GPs) learn a non-linear mapping from policy changes to trajectory changes, allowing the system to generalize to unseen perturbations.
Visualization of policy parameter space sampling and voxelization

Denoising Techniques

To manage the inherent "messiness" of hardware, the authors introduce two critical refinements:

  • Correlation-based delay estimation: This addresses "temporal noise" or timing jitters in motor execution.
  • Voxelization: The state space is divided into small 3D boxes (voxels) to average out spatial noise caused by mechanical imperfections, as illustrated in the parameter space partitioning above.

Experimental Setup

The study utilizes a custom-built "Finger Robot" platform rather than standard simulations. This hardware consists of a three-joint articulated arm powered by brushless DC motors. The researchers collected hundreds of trajectories using various controller types, including linear open-loop, non-linear sinusoidal, and PD (Proportional-Derivative) feedback controllers.

The Finger Robot platform executing various trajectories

Key Insights

  • The World as its Own Model: By perturbing actual hardware, Physical Derivatives inherently capture complex effects like friction, cable tension, and electrical lag that are notoriously difficult to simulate accurately.
  • Task-Agnosticism: Because these derivatives describe the system's sensitivity rather than a specific reward, they are reusable. Once learned, they can optimize for any reward function (e.g., speed, safety, or accuracy) without additional physical experiments.
  • Low-Dimensional Manifolds: Physical trajectories typically live on low-dimensional manifolds, meaning a relatively small number of perturbations can provide sufficient information to predict behavior across a wide range of policy changes.

Results

The Gaussian Process models achieved high predictive accuracy, with GP scores reaching up to 0.98 for PD controllers. Furthermore, predicted derivative vectors showed strong alignment with ground-truth measurements, frequently exceeding cosine similarity scores of 0.95.

In a "zero-shot planning" task, the robot was required to reach a specific intermediate state it had not been explicitly trained for. Using the learned derivatives, the system calculated new policy parameters that allowed the robot to successfully adjust its trajectory and pass through the target point on the very first attempt.

Limitations and Future Work

While promising, the method currently relies on finite differences, which may become computationally expensive as the number of policy parameters increases (e.g., in deep neural networks). Additionally, while the assumption that trajectories change "slowly" holds for stable robotic tasks, the method may struggle with chaotic systems where tiny perturbations lead to radically different outcomes. Future work will explore multi-task learning to exploit temporal relationships between derivatives at successive time steps.

Summary updated

Abstract

Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.
BibTeX
@misc{mehrjou2022physical,
  author = {Arash Mehrjou and Ashkan Soleymani and Stefan Bauer and Bernhard Schölkopf},
  title = {{Physical Derivatives: Computing policy gradients by physical forward-propagation}},
  year = {2022},
  eprint = {2201.05830},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.05830},
}
Figure