TL;DR
This paper introduces a "derivative-free" framework for model-based reinforcement learning that eliminates the need for noisy velocity and acceleration sensors. By representing the system state as a history of raw position measurements and using a novel physically-inspired kernel for Gaussian Process Regression, the authors achieve superior control performance and data efficiency on real robotic hardware.
What Problem Does This Paper Address?
In classical mechanics, the state of a system is defined by positions, velocities, and accelerations. To use Model-Based Reinforcement Learning (MBRL), an agent needs an accurate model of how these states evolve. However, many real-world robots only have position sensors (like encoders).
Engineers typically "solve" this by numerically differentiating positions to estimate velocity and acceleration. This process is notoriously problematic: it introduces significant noise and time delays, which can cripple a learning algorithm's accuracy. While Gaussian Process Regression (GPR) is a powerful tool for learning these models, standard GPR assumes inputs are noise-free—an assumption that fails the moment you feed it a calculated derivative.
Methods
The authors propose two major shifts from the standard approach:
- Derivative-Free State Representation: Instead of a state like $[pos, vel, acc]$, they define the state as a window of past position measurements: $x_k = [q_k, q_{k-1}, \dots, q_{k-k_p}]$. This preserves the raw information content without the distortions of numerical differentiation.
- Physically Inspired Derivative-Free (PIDF) Kernels: Usually, physical equations of motion are functions of velocity and acceleration. The authors developed a set of guidelines to "translate" these physical equations into a GPR kernel. They replace derivative terms with polynomial kernels of the position history.
By combining this PIDF kernel with a standard non-parametric kernel (like RBF), they create Semiparametric Derivative-Free (SPDF) models. These models use physical "hints" to generalize quickly while remaining flexible enough to learn unmodeled effects like friction or gear backlash.
Datasets
The framework was validated on two custom-built physical platforms:
- Ball-and-Beam (BB): A system where a motor tilts a beam to control a rolling ball. Data consisted of 3 minutes of operation at 30Hz (5400 samples).
- Furuta Pendulum (FP): A more complex rotational inverted pendulum held by a robotic arm. The training set included 100 seconds of experience (15,000 samples), with a separate 20,000-sample set for testing generalization.
Key Insights and Ideas
- Raw Data is King: Numerical derivatives are essentially lossy filters. By feeding the GPR the raw history, the model can "learn" its own internal representation of dynamics and delays that are more accurate than human-engineered filters.
- Physics as a Structural Prior: You don't need to know the exact parameters (mass, friction) of a system to benefit from physics. Just knowing the structure of the equations (e.g., that gravity involves a sine term of the angle) and embedding that into the kernel significantly boosts data efficiency.
- Handling Latency: Because the state includes a history of positions, the model can implicitly identify and compensate for system delays (like the lag between a control command and the actual movement of a 3D-printed gripper).
Key Results
- Superior Accuracy: On the Ball-and-Beam system, the derivative-free approach achieved a Root Mean Squared Error (RMSE) approximately 20% lower than the best models using traditional causal filters (like Kalman filters).
- Data Efficiency: In the Furuta Pendulum tests, the physically-inspired kernel (PI) outperformed purely data-driven models (NP) after only 2 seconds of training data.
- Real-World Control: The authors used the learned models to perform complex tasks, such as a swing-up maneuver for the Furuta Pendulum, using an iLQG trajectory optimizer. The models were accurate enough that the trajectories could be executed in "open-loop" (without active feedback) and still remain close to the target.
Limitations and Future Work
- Open-Loop Limitations: While the open-loop success demonstrates model accuracy, the authors note that an open-loop sequence cannot stabilize an unstable equilibrium (like the top of a pendulum swing) indefinitely.
- Data Sparsity: The models lost accuracy near unstable points where training data was naturally harder to collect. Future work could involve active learning strategies to specifically target these "knowledge gaps."
- Computational Scaling: GPR can be computationally heavy as datasets grow. While the authors used stochastic gradient descent to manage 15,000 samples, scaling to even larger systems with many degrees of freedom remains a challenge.
Summary updated