Abstract
This paper introduces Physical Derivatives, a novel approach for policy optimization that bridges the gap between model-free and model-based reinforcement learning. By directly estimating the sensitivity of a physical system's trajectories to policy perturbations—rather than learning a full transition model—the method enables sample-efficient, task-agnostic control without the bias of analytical models. We demonstrate the effectiveness of this approach on a custom robotic finger platform, achieving high predictive accuracy and successful zero-shot planning.
Introduction
In reinforcement learning (RL), practitioners typically face a stark trade-off. Model-free methods are robust to complex dynamics but require a massive number of interactions with the environment, which is often impractical for real-world robotics. Conversely, model-based methods are more sample-efficient but suffer from "model bias"; if the simulator or mathematical model deviates even slightly from reality, the learned policy may fail when deployed on hardware.
This research identifies a middle ground: we do not necessarily need a full model of a robot's dynamics to improve its behavior. Instead, we only require the sensitivity of the robot's path to changes in its controller. This sensitivity—the mapping of how a small tweak in motor parameters affects the resulting trajectory—is what the authors term the "Physical Derivative."
Methodology
The core innovation is computing policy gradients through physical forward-propagation. Instead of backpropagating through a neural network model, the researchers use the physical world itself as the "computer."
The Learning Pipeline
- Perturbation: A "nominal" policy (baseline controller) is subjected to small, random perturbations in its parameters ($\theta$).
- Observation: These perturbed policies are executed on the real robot, and the resulting trajectories are recorded.
- Finite Differences: By comparing perturbed trajectories to the nominal one, the system estimates directional derivatives, mapping policy changes ($\delta\theta$) to state changes ($\delta\mathbf{x}$) across time.
- Probabilistic Learning: To handle real-world noise, Gaussian Processes (GPs) learn a non-linear mapping from policy changes to trajectory changes, allowing the system to generalize to unseen perturbations.
Denoising Techniques
To manage the inherent "messiness" of hardware, the authors introduce two critical refinements:
- Correlation-based delay estimation: This addresses "temporal noise" or timing jitters in motor execution.
- Voxelization: The state space is divided into small 3D boxes (voxels) to average out spatial noise caused by mechanical imperfections, as illustrated in the parameter space partitioning above.
Experimental Setup
The study utilizes a custom-built "Finger Robot" platform rather than standard simulations. This hardware consists of a three-joint articulated arm powered by brushless DC motors. The researchers collected hundreds of trajectories using various controller types, including linear open-loop, non-linear sinusoidal, and PD (Proportional-Derivative) feedback controllers.
Key Insights
- The World as its Own Model: By perturbing actual hardware, Physical Derivatives inherently capture complex effects like friction, cable tension, and electrical lag that are notoriously difficult to simulate accurately.
- Task-Agnosticism: Because these derivatives describe the system's sensitivity rather than a specific reward, they are reusable. Once learned, they can optimize for any reward function (e.g., speed, safety, or accuracy) without additional physical experiments.
- Low-Dimensional Manifolds: Physical trajectories typically live on low-dimensional manifolds, meaning a relatively small number of perturbations can provide sufficient information to predict behavior across a wide range of policy changes.
Results
The Gaussian Process models achieved high predictive accuracy, with GP scores reaching up to 0.98 for PD controllers. Furthermore, predicted derivative vectors showed strong alignment with ground-truth measurements, frequently exceeding cosine similarity scores of 0.95.
In a "zero-shot planning" task, the robot was required to reach a specific intermediate state it had not been explicitly trained for. Using the learned derivatives, the system calculated new policy parameters that allowed the robot to successfully adjust its trajectory and pass through the target point on the very first attempt.
Limitations and Future Work
While promising, the method currently relies on finite differences, which may become computationally expensive as the number of policy parameters increases (e.g., in deep neural networks). Additionally, while the assumption that trajectories change "slowly" holds for stable robotic tasks, the method may struggle with chaotic systems where tiny perturbations lead to radically different outcomes. Future work will explore multi-task learning to exploit temporal relationships between derivatives at successive time steps.
Summary updated