Arnold: a generalist muscle transformer policy

Alberto Silvio Chiappa, Boshi An, Merkourios Simos, Chengkun Li, Alexander MathisarXiv 2025 · DOI · arXiv

Introduction

Controlling musculoskeletal models, which involve hundreds of non-linear muscles and complex tendon arrangements, presents a significant challenge for traditional robotics and AI. Most existing agents are "specialists," trained for specific tasks like reaching or walking. Researchers at EPFL have introduced Arnold, a generalist transformer policy designed to master multiple tasks across diverse human embodiments, including the hand, arm, leg, and elbow. Arnold often exceeds the performance of the human-designed experts it was initially trained to imitate.

Problem Statement

In biomechanics, controllers are typically built for specific problems—a "walking policy" or a "pen-rotating policy." This specialist approach suffers from inflexibility, as policies cannot easily adapt to different body parts or new tasks. Furthermore, it lacks biological accuracy; the human nervous system utilizes shared representations to control various limbs. The primary technical hurdle is that different tasks possess varying observation and action spaces (e.g., a hand has 39 muscles while a leg has 80), making universal motor control difficult for standard neural networks.

Methods

Arnold treats motor control as a translation problem, utilizing a Transformer architecture (Encoder-Decoder) similar to Large Language Models.

Sensorimotor Vocabulary

The core innovation is a compositional "sensorimotor vocabulary." Instead of raw numerical inputs, the network receives descriptors (e.g., ["right", "soleus", "muscle", "length"]) converted into embeddings. This allows the model to generalize the concept of "length" or "velocity" across different body parts.

Architecture of Arnold, highlighting the sensorimotor vocabulary and transformer-based policy.

Training Pipeline

The training process follows a three-step pipeline:

  1. On-Policy Behavior Cloning (OBC): Arnold learns 14 tasks in parallel by imitating experts. Unlike standard BC, OBC involves the student interacting with the environment while the expert provides corrective actions for the student's current state.
  2. Reinforcement Learning (RL) Fine-tuning: The model is specialized for individual tasks using Proximal Policy Optimization (PPO) to enhance performance.
  3. Self-distillation: These specialized "super-expert" versions are distilled back into a single generalist model.
Overview of tasks and the three-step training pipeline including OBC, RL fine-tuning, and self-distillation.

Datasets and Tasks

Arnold was evaluated using MyoSuite, a collection of musculoskeletal environments powered by the MuJoCo physics engine. The study utilized models including MyoElbow, MyoHand, MyoArm, and MyoLeg across 14 tasks such as Finger Reach, Baoding Balls, Pen/Die Reorient, Object Relocation, and Walk to Point. Initial training data was derived from winning policies of previous NeurIPS MyoChallenge competitions.

Key Insights

  • Semantic Embeddings: Using a compositional vocabulary significantly outperformed task-specific embeddings, facilitating faster learning through shared sensor meanings.
  • Data Efficiency: Pre-training allows Arnold to learn novel tasks much faster than training from scratch, indicating the acquisition of a foundational understanding of muscle dynamics.
  • Motor Synergies: While Arnold utilizes low-dimensional patterns within specific tasks, these patterns do not transfer well between tasks. This suggests that while the "brain" (policy) is a generalist, execution remains highly specialized.

Results

Arnold achieved an average of 105.13% of expert performance across all 14 tasks. While standard RL (PPO) often failed to solve complex manipulation and locomotion tasks from scratch, Arnold's OBC-based approach successfully mastered them. The entire training process required approximately 55 million iterations, demonstrating high efficiency for such complex musculoskeletal control.

Limitations and Future Work

Currently, Arnold does not support zero-shot learning and requires expert demonstrations to adapt to entirely new tasks. It also struggles with autonomous exploration for complex behaviors like walking without initial guidance. Future research may involve integrating natural language instructions and comparing Arnold's internal activations with biological data from the spinal cord and motor cortex.

Summary updated

Abstract

Controlling high-dimensional and nonlinear musculoskeletal models of the human body is a foundational scientific challenge. Recent machine learning breakthroughs have heralded in-silico policies that master individual skills like reaching, object manipulation and locomotion in musculoskeletal systems with many degrees of freedom. However, these agents are merely "specialists", achieving high performance for a single skill. In this work, we develop Arnold, a transformer-based musculoskeletal control policy that masters multiple tasks and embodiments. Arnold combines behavior cloning and reinforcement learning to address 14 challenging control tasks spanning dexterous object manipulation, reaching, and locomotion, matching or exceeding the performance of single-task specialist policies. A key innovation is Arnold's sensorimotor vocabulary, a compositional representation of the semantics of heterogeneous sensory modalities, objectives, and actuators. Arnold leverages this vocabulary via a transformer architecture to deal with the variable observation and action spaces across tasks. This framework supports efficient multi-task, multi-embodiment learning and facilitates rapid adaptation to novel tasks, while encouraging universal motor strategies such as action and kinematic smoothness. Finally, causal probing of the motor output reveals that low-dimensional muscle synergies remain largely task-specific and that variance-based analyses systematically underestimate functional control dimensionality, consistent with biological observations on the limited transferability of such synergies. Code and data are available here: https://github.com/amathislab/arnold
BibTeX
@misc{chiappa2025arnold,
  author = {Alberto Silvio Chiappa and Boshi An and Merkourios Simos and Chengkun Li and Alexander Mathis},
  title = {{Arnold: a generalist muscle transformer policy}},
  year = {2025},
  eprint = {2508.18066},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.18066},
}
Figure