DOOMED: Direct Online Optimization of Modeling Errors in Dynamics

Nathan D. Ratliff, Franziska Meier, Daniel Kappler, Stefan Schaal

TL;DR

DOOMED introduces a novel adaptive control framework that corrects robot dynamics models in real-time by directly minimizing the error between desired and actual accelerations. By proving that the gradient of this "direct loss" is measurable even when the true system dynamics are unknown, the authors bridge the gap between classical PID control and modern online gradient-based learning.

What Problem Does This Paper Address?

In robotics, model-based control is the "holy grail" for achieving high-performance tracking while keeping the robot compliant (safe and soft to the touch). This relies on an inverse dynamics model—a mathematical function that tells the robot exactly how much torque to apply to achieve a specific acceleration.

Traditionally, researchers use "indirect loss" to learn these models: they record the torques applied and the accelerations produced, then train a model to map one to the other. However, this fails in common real-world scenarios like stiction (static friction). If a robot is stuck, it might apply torque and see zero acceleration. An indirect learner would mistakenly learn that "torque equals zero acceleration," which doesn't help it break free. Furthermore, these models often train on "off-distribution" data—learning from what the robot actually did rather than what it intended to do.

Methods

The authors propose DOOMED (Direct Online Optimization for Modeling Errors in Dynamics). Instead of predicting torques, it directly minimizes a loss function based on the difference between the desired acceleration ($\ddot{q}_d$) and the actual observed acceleration ($\ddot{q}_a$).

The technical "magic" lies in the gradient calculation. Usually, calculating the gradient of acceleration error requires knowing the robot's true, complex mass matrix ($M$). The authors show that when you derive the gradient of the direct loss, the unknown terms (like the true mass matrix) actually cancel out or vanish, leaving a gradient that can be measured directly from system data: $$\nabla_w l(w) = -J_f^T (\ddot{q}_d - \ddot{q}_a)$$ Where $J_f$ is the Jacobian of the model's offset function.

To make this work in the messy real world, they incorporate several "tricks" from the machine learning literature:

  • Momentum: To prevent oscillations and smooth out noisy sensor data.
  • Variance-based Scaling: Adjusting the learning step size based on the noise level of the acceleration signal (similar to modern optimizers like Adam or Adagrad).
  • Adaptive Forgetting Factors: Tuning how quickly the model "forgets" old data to balance fast adaptation with long-term stability.

Datasets

The paper evaluates the algorithm across three increasingly difficult setups:

  1. 2D Simulation: A toy problem with severe model inaccuracies and sinusoidal friction.
  2. Baxter Simulation: A simulated 7-DOF robot arm where the authors added heavy, position-dependent nonlinear friction and torque biases.
  3. Real-world KUKA Lightweight Arm: A physical 7-DOF manipulator (the "Apollo" platform) used to test Cartesian space control and optimized grasping behaviors.

Key Insights and Ideas

  • The "Aha!" Moment on Gradients: The most significant insight is that you don't need to know the true physics of the robot to calculate the direction in which to improve the model. The acceleration error itself is the gradient (when pushed through the model's Jacobian).
  • PID as Machine Learning: The authors provide a fascinating theoretical bridge: they prove that classical PID control is mathematically equivalent to running vanilla online gradient descent on a specific combination of position, velocity, and acceleration error terms.
  • Direct vs. Indirect: By focusing on the result (acceleration) rather than the input (torque), the robot naturally learns to overcome non-invertible effects like stiction that baffle traditional learners.

Key Results

  • High Precision: On the real KUKA arm, the system achieved end-effector positioning accuracies of approximately $10^{-5}$ meters within seconds of adaptation.
  • Robustness: In video demonstrations, the robot was repeatedly shoved and perturbed. Because it adapts its model online, it was able to return to its target smoothly, effectively "learning" the external forces acting on it in real-time.
  • Handling Bias: In the Baxter simulation, DOOMED successfully zeroed out acceleration errors even when the internal model was missing joint damping coefficients and faced massive nonlinear torque biases.

Limitations and Future Work

  • Simple Offset Models: In this paper, the authors primarily used a simple constant torque offset ($f_{offset} = w$). While effective, this is a "memory-less" approach.
  • Function Approximators: The authors suggest that future work should involve more complex function approximators (like neural networks or Gaussian processes) to model complex friction patterns that vary across the robot's entire workspace.
  • Interaction with Learning Control: There is an open question about how this "fast" adaptive control (which forgets quickly) should interact with "slow" learning control (which builds a long-term memory of the robot's dynamics).

Summary updated

Abstract

It has long been hoped that model-based control will improve tracking performance while maintaining or increasing compliance. This hope hinges on having or being able to estimate an accurate inverse dynamics model. As a result, substantial effort has gone into modeling and estimating dynamics (error) models. Most recent research has focused on learning the true inverse dynamics using data points mapping observed accelerations to the torques used to generate them. Unfortunately, if the initial tracking error is bad, such learning processes may train substantially off-distribution to predict well on actual observed acceleration rather then the desired accelerations. This work takes a different approach. We define a class of gradient-based online learning algorithms we term Direct Online Optimization for Modeling Errors in Dynamics (DOOMED) that directly minimize an objective measuring the divergence between actual and desired accelerations. Our objective is defined in terms of the true system's unknown dynamics and is therefore impossible to evaluate. However, we show that its gradient is measurable online from system data. We develop a novel adaptive control approach based on running online learning to directly correct (inverse) dynamics errors in real time using the data stream from the robot to accurately achieve desired accelerations during execution.
BibTeX
@inproceedings{ratliff2016doomed,
  author = {Nathan D. Ratliff and Franziska Meier and Daniel Kappler and Stefan Schaal},
  title = {{DOOMED: Direct Online Optimization of Modeling Errors in Dynamics}},
  booktitle = {Big Data},
  year = {2016},
  url = {https://openreview.net/forum?id=bce0zuS8k2},
}

Loading related content...

Figure