Model-Based Reinforcement Learning for Physical Systems Without Velocity and Acceleration Measurements

Alberto Dalla Libera, Diego Romeres, Devesh K. Jha, Bill Yerazunis, Daniel NikovskiarXiv 2020 · arXiv

TL;DR

This paper introduces a "derivative-free" framework for model-based reinforcement learning that eliminates the need for noisy velocity and acceleration sensors. By representing the system state as a history of raw position measurements and using a novel physically-inspired kernel for Gaussian Process Regression, the authors achieve superior control performance and data efficiency on real robotic hardware.

What Problem Does This Paper Address?

In classical mechanics, the state of a system is defined by positions, velocities, and accelerations. To use Model-Based Reinforcement Learning (MBRL), an agent needs an accurate model of how these states evolve. However, many real-world robots only have position sensors (like encoders).

Engineers typically "solve" this by numerically differentiating positions to estimate velocity and acceleration. This process is notoriously problematic: it introduces significant noise and time delays, which can cripple a learning algorithm's accuracy. While Gaussian Process Regression (GPR) is a powerful tool for learning these models, standard GPR assumes inputs are noise-free—an assumption that fails the moment you feed it a calculated derivative.

Methods

The authors propose two major shifts from the standard approach:

  1. Derivative-Free State Representation: Instead of a state like $[pos, vel, acc]$, they define the state as a window of past position measurements: $x_k = [q_k, q_{k-1}, \dots, q_{k-k_p}]$. This preserves the raw information content without the distortions of numerical differentiation.
  2. Physically Inspired Derivative-Free (PIDF) Kernels: Usually, physical equations of motion are functions of velocity and acceleration. The authors developed a set of guidelines to "translate" these physical equations into a GPR kernel. They replace derivative terms with polynomial kernels of the position history.

By combining this PIDF kernel with a standard non-parametric kernel (like RBF), they create Semiparametric Derivative-Free (SPDF) models. These models use physical "hints" to generalize quickly while remaining flexible enough to learn unmodeled effects like friction or gear backlash.

Datasets

The framework was validated on two custom-built physical platforms:

  • Ball-and-Beam (BB): A system where a motor tilts a beam to control a rolling ball. Data consisted of 3 minutes of operation at 30Hz (5400 samples).
  • Furuta Pendulum (FP): A more complex rotational inverted pendulum held by a robotic arm. The training set included 100 seconds of experience (15,000 samples), with a separate 20,000-sample set for testing generalization.
Fig. 1: In-house built Ball-and-Beam experimental setup.

Key Insights and Ideas

  • Raw Data is King: Numerical derivatives are essentially lossy filters. By feeding the GPR the raw history, the model can "learn" its own internal representation of dynamics and delays that are more accurate than human-engineered filters.
  • Physics as a Structural Prior: You don't need to know the exact parameters (mass, friction) of a system to benefit from physics. Just knowing the structure of the equations (e.g., that gravity involves a sine term of the angle) and embedding that into the kernel significantly boosts data efficiency.
  • Handling Latency: Because the state includes a history of positions, the model can implicitly identify and compensate for system delays (like the lag between a control command and the actual movement of a 3D-printed gripper).

Key Results

  • Superior Accuracy: On the Ball-and-Beam system, the derivative-free approach achieved a Root Mean Squared Error (RMSE) approximately 20% lower than the best models using traditional causal filters (like Kalman filters).
  • Data Efficiency: In the Furuta Pendulum tests, the physically-inspired kernel (PI) outperformed purely data-driven models (NP) after only 2 seconds of training data.
  • Real-World Control: The authors used the learned models to perform complex tasks, such as a swing-up maneuver for the Furuta Pendulum, using an iLQG trajectory optimizer. The models were accurate enough that the trajectories could be executed in "open-loop" (without active feedback) and still remain close to the target.
Fig. 2: Comparison of the prediction errors obtained in the test set with physically inspired estimators, together with a detailed plot of the evolution of ˙p computed by means of numerical differentiation and a Kalman filter.
Fig. 8: Evolution of RMSEk with its 99% confidence intervals.

Limitations and Future Work

  • Open-Loop Limitations: While the open-loop success demonstrates model accuracy, the authors note that an open-loop sequence cannot stabilize an unstable equilibrium (like the top of a pendulum swing) indefinitely.
  • Data Sparsity: The models lost accuracy near unstable points where training data was naturally harder to collect. Future work could involve active learning strategies to specifically target these "knowledge gaps."
  • Computational Scaling: GPR can be computationally heavy as datasets grow. While the authors used stochastic gradient descent to manage 15,000 samples, scaling to even larger systems with many degrees of freedom remains a challenge.

Summary updated

Abstract

In this paper, we propose a derivative-free model learning framework for Reinforcement Learning (RL) algorithms based on Gaussian Process Regression (GPR). In many mechanical systems, only positions can be measured by the sensing instruments. Then, instead of representing the system state as suggested by the physics with a collection of positions, velocities, and accelerations, we define the state as the set of past position measurements. However, the equation of motions derived by physical first principles cannot be directly applied in this framework, being functions of velocities and accelerations. For this reason, we introduce a novel derivative-free physically-inspired kernel, which can be easily combined with nonparametric derivative-free Gaussian Process models. Tests performed on two real platforms show that the considered state definition combined with the proposed model improves estimation performance and data-efficiency w.r.t. traditional models based on GPR. Finally, we validate the proposed framework by solving two RL control problems for two real robotic systems.
BibTeX
@misc{libera2020model,
  author = {Alberto Dalla Libera and Diego Romeres and Devesh K. Jha and Bill Yerazunis and Daniel Nikovski},
  title = {{Model-Based Reinforcement Learning for Physical Systems Without Velocity and Acceleration Measurements}},
  year = {2020},
  eprint = {2002.10621},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.10621},
}
Figure