DOOMED: Direct Online Optimization of Modeling Errors in Dynamics

Nathan D. Ratliff, Franziska Meier, Daniel Kappler, Stefan Schaal

Abstract

It has long been hoped that model-based control will improve tracking performance while maintaining or increasing compliance. This hope hinges on having or being able to estimate an accurate inverse dynamics model. As a result, substantial effort has gone into modeling and estimating dynamics (error) models. Most recent research has focused on learning the true inverse dynamics using data points mapping observed accelerations to the torques used to generate them. Unfortunately, if the initial tracking error is bad, such learning processes may train substantially off-distribution to predict well on actual observed acceleration rather then the desired accelerations. This work takes a different approach. We define a class of gradient-based online learning algorithms we term Direct Online Optimization for Modeling Errors in Dynamics (DOOMED) that directly minimize an objective measuring the divergence between actual and desired accelerations. Our objective is defined in terms of the true system's unknown dynamics and is therefore impossible to evaluate. However, we show that its gradient is measurable online from system data. We develop a novel adaptive control approach based on running online learning to directly correct (inverse) dynamics errors in real time using the data stream from the robot to accurately achieve desired accelerations during execution.

Introduction

In the field of robotics, model-based control is often considered the "holy grail" for achieving high-performance tracking. By utilizing an inverse dynamics model—a mathematical function that maps desired accelerations to the necessary torques—robots can achieve precise movement while remaining compliant and safe for human interaction.

Traditionally, these models are refined using "indirect loss" functions. Researchers record the torques applied and the resulting accelerations, then train a model to map one to the other. However, this approach faces significant hurdles in real-world environments. For instance, phenomena like stiction (static friction) can cause a robot to apply torque without moving, leading an indirect learner to incorrectly conclude that torque results in zero acceleration. Furthermore, if a robot's initial tracking is poor, it collects data on what it actually did rather than what it intended to do, leading to models that are trained on off-distribution data.

The DOOMED Framework

To address these limitations, we propose Direct Online Optimization for Modeling Errors in Dynamics (DOOMED). Rather than predicting torques, DOOMED directly minimizes a loss function based on the discrepancy between the desired acceleration ($\ddot{q}_d$) and the actual observed acceleration ($\ddot{q}_a$).

The core technical insight of DOOMED is the derivation of the gradient for this direct loss. Typically, calculating the gradient of acceleration error would require precise knowledge of the robot's complex and unknown mass matrix ($M$). However, we demonstrate that when deriving the gradient of the direct loss, these unknown terms cancel out or vanish. This leaves a gradient that can be measured directly from system data:

$$\nabla_w l(w) = -J_f^T (\ddot{q}_d - \ddot{q}_a)$$

Where $J_f$ represents the Jacobian of the model's offset function. This allows the robot to update its model in real-time without requiring a perfect prior physical model.

Optimization Strategies

To ensure the algorithm's efficacy in noisy, real-world conditions, DOOMED incorporates several techniques from modern machine learning:

  • Momentum: Used to smooth out noisy sensor data and prevent unwanted oscillations during adaptation.
  • Variance-based Scaling: Similar to optimizers like Adam, this adjusts the learning step size based on the noise level of the acceleration signal.
  • Adaptive Forgetting Factors: This mechanism balances the robot's ability to adapt quickly to new external forces with the need for long-term model stability.

Theoretical Bridge: PID as Online Learning

A key theoretical contribution of this work is the unification of classical control theory and online learning. We prove that the standard Proportional-Integral-Derivative (PID) controller is mathematically equivalent to running vanilla online gradient descent on a specific combination of position, velocity, and acceleration error terms. This insight suggests that many classical controllers can be viewed as specialized instances of online gradient-based learners.

Experimental Results

We evaluated DOOMED across three increasingly complex environments:

  1. 2D Simulation: A controlled environment with severe model inaccuracies and sinusoidal friction patterns.
  2. Baxter Simulation: A 7-DOF robot arm simulation where we introduced heavy, position-dependent nonlinear friction and torque biases. DOOMED successfully compensated for these errors even when the underlying model lacked joint damping coefficients.
  3. Physical KUKA Lightweight Arm: Using the "Apollo" platform, we tested Cartesian space control. The system achieved end-effector positioning accuracies of approximately $10^{-5}$ meters within seconds of starting the adaptation process.

In physical demonstrations, the robot showed remarkable robustness to external perturbations. When the arm was physically shoved, the online adaptation treated the external force as a modeling error and adjusted its torque output in real-time to maintain its trajectory, effectively "learning" the external force and compensating for it.

Conclusion and Future Work

DOOMED provides a robust framework for real-time dynamics correction by focusing on direct acceleration outcomes rather than indirect torque predictions. While this work primarily utilized simple torque offset models, the framework is extensible. Future research will focus on integrating more complex function approximators, such as neural networks or Gaussian processes, to model complex friction across the robot's entire workspace. Additionally, we are interested in the interplay between "fast" adaptive control for immediate corrections and "slow" learning control for building comprehensive, long-term dynamics memories.

Summary updated

Abstract

It has long been hoped that model-based control will improve tracking performance while maintaining or increasing compliance. This hope hinges on having or being able to estimate an accurate inverse dynamics model. As a result, substantial effort has gone into modeling and estimating dynamics (error) models. Most recent research has focused on learning the true inverse dynamics using data points mapping observed accelerations to the torques used to generate them. Unfortunately, if the initial tracking error is bad, such learning processes may train substantially off-distribution to predict well on actual observed acceleration rather then the desired accelerations. This work takes a different approach. We define a class of gradient-based online learning algorithms we term Direct Online Optimization for Modeling Errors in Dynamics (DOOMED) that directly minimize an objective measuring the divergence between actual and desired accelerations. Our objective is defined in terms of the true system's unknown dynamics and is therefore impossible to evaluate. However, we show that its gradient is measurable online from system data. We develop a novel adaptive control approach based on running online learning to directly correct (inverse) dynamics errors in real time using the data stream from the robot to accurately achieve desired accelerations during execution.
BibTeX
@inproceedings{ratliff2016doomed,
  author = {Nathan D. Ratliff and Franziska Meier and Daniel Kappler and Stefan Schaal},
  title = {{DOOMED: Direct Online Optimization of Modeling Errors in Dynamics}},
  booktitle = {Big Data},
  year = {2016},
  url = {https://openreview.net/forum?id=bce0zuS8k2},
}
Figure