RoboMorph: In-Context Meta-Learning for Robot Dynamics Modeling

Manuel Bianchi Bazzi, Asad Ali Shahid, Christopher Agia, John Irvin Alora, Marco Forgione, Dario Piga, Francesco Braghin, Marco Pavone, Loris Roveda

Abstract

Accurate system identification is a cornerstone of effective robotic control, yet traditional methods often struggle with the non-linearities and shifting physical parameters inherent in real-world hardware. This paper introduces RoboMorph, a Transformer-based framework that applies in-context meta-learning to robot dynamics. By treating joint torques and positions as sequential tokens, RoboMorph learns to predict future motion across a wide range of physical configurations without requiring explicit physical equations. Experimental results using the Franka Emika Panda arm demonstrate that the model can achieve high accuracy (Fit Index $\ge 90$) and adapt to complex new tasks through efficient fine-tuning, outperforming models trained from scratch.

Introduction

In robotics, "system identification"—the process of determining the mathematical model of a robot’s dynamics—is notoriously difficult. Robots are highly non-linear systems, and their behavior changes based on payload, joint friction, or mechanical wear. Traditional methods typically require explicit physical modeling or extensive retraining for every new scenario, making them brittle in dynamic environments.

While Transformers have revolutionized Natural Language Processing (NLP) by learning complex patterns in text, their application to low-level robot dynamics has historically been limited. RoboMorph addresses this gap by asking: Can a Transformer learn the "language" of robot motion well enough to adapt to a new robot's specific physical quirks just by observing a few seconds of movement?

Methods

The researchers developed a meta-dynamical model using an encoder-decoder Transformer architecture designed to handle the continuous nature of robotic signals.

Architecture and In-Context Learning

Unlike standard NLP Transformers that use discrete word embeddings, RoboMorph utilizes linear layers to process continuous, real-valued signals, specifically joint torques and positions. The model operates via in-context learning: it is provided with a "context" consisting of a short sequence of initial torque signals and resulting positions (representing 20% of a trajectory). It then uses this context to predict the "query"—the output for the remaining 80% of the trajectory.

Meta-Learning Strategy

To ensure generalization, the model is trained on thousands of simulated robots, each with randomized physical properties. This meta-learning approach teaches the model the underlying relationship between force and motion (the physics) rather than simply memorizing a single path for a single robot. By training across a diverse distribution of mass, center of mass, and stiffness parameters, the model learns to identify a specific robot's dynamics "in-context" during the inference phase.

Experimental Setup

The authors leveraged NVIDIA’s Isaac Gym for massively parallel simulations. While Isaac Gym is typically used for Reinforcement Learning (RL), it provided the scale necessary to generate a robust dataset for dynamics modeling.

Datasets and Domain Randomization

The training involved over 3,000 unique robot configurations for the Franka Emika Panda arm. The researchers varied link masses, centers of mass, and joint stiffness to create a broad "morphology" space. Two primary task families were used:

  1. Synthetic Inputs: Multi-sinusoidal and "chirp" torque signals used to excite the robot's dynamics across various frequencies.
  2. Operational Space Control (OSC): Structured tasks, such as drawing circles or spirals in 3D space, to test the model's performance on functional movements.

Key Results and Insights

Accuracy and Fit

The researchers measured performance using a "Fit Index" (FI) and $R^2$. For in-distribution tasks (tasks similar to the training data), the model achieved excellent results, often maintaining an FI $\ge 90$ across most joint positions and Cartesian coordinates. This suggests the Transformer successfully captured the mapping from torque to motion.

Zero-Shot vs. Fine-Tuning

A critical finding was the efficiency of adaptation. While the "Zero-Shot" model (no prior exposure to a specific task) performed well on basic movements, it struggled with complex 3D spirals, particularly in the Z-coordinate. However, the pre-trained meta-model could be adapted to these difficult tasks in just 2.5 hours of fine-tuning. In contrast, a model trained from scratch required 9 hours to reach similar performance levels, and even then, the fine-tuned RoboMorph model achieved higher overall accuracy (FI in the 80s and 90s).

Frequency and Horizon Sensitivity

The model demonstrated a unique "slightly out-of-distribution" capability, showing higher effectiveness when applied to higher-frequency signals than those it was trained on. Additionally, the model performed best when the test horizon matched the training horizon, indicating that it learns to recognize whole trajectory patterns rather than just instantaneous state transitions.

Discussion and Limitations

Despite its success, RoboMorph faces challenges with vertical dynamics. The model consistently showed lower accuracy in the Z-axis during complex tasks, likely due to the intrinsic complexity of gravity compensation. Furthermore, while the model generalizes well within a family of control actions, moving from random torques to structured operational tasks still requires fine-tuning.

Conclusion

RoboMorph demonstrates that robot dynamics can be effectively treated as a sequence-to-sequence problem. By leveraging the pattern-recognition power of Transformers and the scale of parallel simulation, the framework allows robots to identify their own dynamics in real-time. Future work aims to expand this to transfer learning across different robot morphologies, potentially creating a "foundation model" for robot dynamics that works across various arm types and configurations.

Summary updated

BibTeX
@inproceedings{bazzi2024robomorph,
  author = {Manuel Bianchi Bazzi and Asad Ali Shahid and Christopher Agia and John Irvin Alora and Marco Forgione and Dario Piga and Francesco Braghin and Marco Pavone and Loris Roveda},
  title = {{RoboMorph: In-Context Meta-Learning for Robot Dynamics Modeling}},
  booktitle = {ICINCO},
  year = {2024},
  doi = {10.5220/0012945500003822},
}
Figure