A new data source for inverse dynamics learning

Daniel Kappler, Franziska Meier, Nathan D. Ratliff, Stefan Schaal

TL;DR

This paper introduces a novel way to train robot dynamics by treating feedback control signals as a direct source of training data. By combining traditional "indirect" data (what the robot actually did) with this new "direct" data (what we wanted the robot to do), robots can learn accurate inverse dynamics models faster and more consistently, even when using safe, low-gain "compliant" controllers.

What Problem Does This Paper Address?

For a robot to move gracefully and safely around humans, it needs an accurate inverse dynamics model. This model tells the robot exactly how much torque to apply to its joints to achieve a specific acceleration.

The "classic" way to get this model is through physics equations (Rigid Body Dynamics), but real-world factors like friction, cable tension, and varying payloads make these equations imperfect. To fix this, researchers use machine learning to learn an "error model."

However, traditional learning is indirect: it looks at the torque applied and the actual acceleration achieved. If the robot's controller is "compliant" (low-gain/soft), it might not move much at all due to friction, meaning the learning algorithm never gets the data it needs to improve. This creates a catch-22: you need a good model to use low gains, but you need high gains (stiff, potentially unsafe movement) to collect the data to build the model.

Methods

The authors propose a Joint Learning approach that uses two distinct data sources:

  1. Indirect Data: The standard approach. It maps the state $(q, \dot{q})$ and actual acceleration $(\ddot{q}_a)$ to the total torque applied.
  2. Direct Data: The innovation. The authors show that the feedback control signal—the "correction" the robot applies when it misses its mark—can be mathematically transformed into a training signal at the desired acceleration $(\ddot{q}_d)$.

By combining these, the robot learns from both where it ended up and where it was trying to go.

The Workflow: * Task-Specific Learning: Instead of trying to learn a "global" model for everything, the robot learns a model specifically for one task (e.g., a pick-and-place motion). * Iterative Updates: The robot attempts the task, collects data from both sources, updates a feedforward neural network offline, and tries again. * Function Approximator: They use a simple feedforward neural network (fully connected layers with ReLU-like activations) to model the torque errors.

Datasets

The researchers evaluated their approach in two environments:

  • 2D Simulation: A synthetic system designed with significant mass discrepancies (simulating a "wrong" physics model), friction, and stiction. They tested this under four different sensor noise levels.
  • Real-World Robot: A 7-degree-of-freedom (DOF) KUKA Lightweight Arm. The model had to handle a 21-dimensional input space (positions, velocities, and accelerations for all joints) in a hard real-time loop (1kHz).

Key Insights and Ideas

  • Feedback as a Teacher: The core "aha" moment is realizing that feedback control isn't just a way to stay on track—it's an online measurement of the model's error. If the feedback term is high, the model is wrong; that high signal is exactly what the learner needs to see.
  • Bridging the Gap: Indirect learning often fails because the data is "off-trajectory." By including the "direct" data source, the algorithm gets a clear signal of what the torque should have been at the desired state, effectively pulling the model toward the target trajectory.
  • Compliance-First Learning: This method allows robots to start with low-gain, "soft" controllers. Because the joint data source is so efficient, the robot doesn't need to be stiff and dangerous to learn how to be accurate.

Key Results

  • Faster Convergence: In simulations, the "Joint" approach converged to low error significantly faster than using indirect or direct data alone.
  • Consistency: While direct learning alone was sensitive to sensor noise and indirect learning was slow to overcome friction, the joint approach remained stable across various noise and friction levels.
  • Real-World Success: On the KUKA arm, the system significantly improved tracking for a "pick and place" task involving a heavy drill after just a single iteration.
  • Low-Gain Performance: The authors demonstrated that even with gains an order of magnitude lower than traditional settings, the joint learning approach could successfully identify the dynamics model where traditional methods stalled.

Limitations and Future Work

  • Payload Ambiguity: The current model assumes the relationship between state and torque is constant for a task. If a robot picks up an object of unknown weight mid-task, the model might get confused because the same movement now requires different torques.
  • Data Selection: The authors suggest that future work could involve "active learning"—deciding which of the two data sources (direct or indirect) is more reliable at any given moment to further speed up training.
  • Task Switching: Since the models are task-specific, the system needs a way to detect which "task model" to load when the robot switches from, say, wiping a table to opening a door.

Summary updated

Abstract

Modern robotics is gravitating toward increasingly collaborative human robot interaction. Tools such as acceleration policies can naturally support the realization of reactive, adaptive, and compliant robots. These tools require us to model the system dynamics accurately - a difficult task. The fundamental problem remains that simulation and reality diverge-we do not know how to accurately change a robot's state. Thus, recent research on improving inverse dynamics models has been focused on making use of machine learning techniques. Traditional learning techniques train on the actual realized accelerations, instead of the policy's desired accelerations, which is an indirect data source. Here we show how an additional training signal - measured at the desired accelerations - can be derived from a feedback control signal. This effectively creates a second data source for learning inverse dynamics models. Furthermore, we show how both the traditional and this new data source, can be used to train task-specific models of the inverse dynamics, when used independently or combined. We analyze the use of both data sources in simulation and demonstrate its effectiveness on a real-world robotic platform. We show that our system incrementally improves the learned inverse dynamics model, and when using both data sources combined converges more consistently and faster.
BibTeX
@inproceedings{kappler2017new,
  author = {Daniel Kappler and Franziska Meier and Nathan D. Ratliff and Stefan Schaal},
  title = {{A new data source for inverse dynamics learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=h8IKE98DXj},
}

Loading related content...

Figure