Introduction
Controlling musculoskeletal models, which involve hundreds of non-linear muscles and complex tendon arrangements, presents a significant challenge for traditional robotics and AI. Most existing agents are "specialists," trained for specific tasks like reaching or walking. Researchers at EPFL have introduced Arnold, a generalist transformer policy designed to master multiple tasks across diverse human embodiments, including the hand, arm, leg, and elbow. Arnold often exceeds the performance of the human-designed experts it was initially trained to imitate.
Problem Statement
In biomechanics, controllers are typically built for specific problems—a "walking policy" or a "pen-rotating policy." This specialist approach suffers from inflexibility, as policies cannot easily adapt to different body parts or new tasks. Furthermore, it lacks biological accuracy; the human nervous system utilizes shared representations to control various limbs. The primary technical hurdle is that different tasks possess varying observation and action spaces (e.g., a hand has 39 muscles while a leg has 80), making universal motor control difficult for standard neural networks.
Methods
Arnold treats motor control as a translation problem, utilizing a Transformer architecture (Encoder-Decoder) similar to Large Language Models.
Sensorimotor Vocabulary
The core innovation is a compositional "sensorimotor vocabulary." Instead of raw numerical inputs, the network receives descriptors (e.g., ["right", "soleus", "muscle", "length"]) converted into embeddings. This allows the model to generalize the concept of "length" or "velocity" across different body parts.
Training Pipeline
The training process follows a three-step pipeline:
- On-Policy Behavior Cloning (OBC): Arnold learns 14 tasks in parallel by imitating experts. Unlike standard BC, OBC involves the student interacting with the environment while the expert provides corrective actions for the student's current state.
- Reinforcement Learning (RL) Fine-tuning: The model is specialized for individual tasks using Proximal Policy Optimization (PPO) to enhance performance.
- Self-distillation: These specialized "super-expert" versions are distilled back into a single generalist model.
Datasets and Tasks
Arnold was evaluated using MyoSuite, a collection of musculoskeletal environments powered by the MuJoCo physics engine. The study utilized models including MyoElbow, MyoHand, MyoArm, and MyoLeg across 14 tasks such as Finger Reach, Baoding Balls, Pen/Die Reorient, Object Relocation, and Walk to Point. Initial training data was derived from winning policies of previous NeurIPS MyoChallenge competitions.
Key Insights
- Semantic Embeddings: Using a compositional vocabulary significantly outperformed task-specific embeddings, facilitating faster learning through shared sensor meanings.
- Data Efficiency: Pre-training allows Arnold to learn novel tasks much faster than training from scratch, indicating the acquisition of a foundational understanding of muscle dynamics.
- Motor Synergies: While Arnold utilizes low-dimensional patterns within specific tasks, these patterns do not transfer well between tasks. This suggests that while the "brain" (policy) is a generalist, execution remains highly specialized.
Results
Arnold achieved an average of 105.13% of expert performance across all 14 tasks. While standard RL (PPO) often failed to solve complex manipulation and locomotion tasks from scratch, Arnold's OBC-based approach successfully mastered them. The entire training process required approximately 55 million iterations, demonstrating high efficiency for such complex musculoskeletal control.
Limitations and Future Work
Currently, Arnold does not support zero-shot learning and requires expert demonstrations to adapt to entirely new tasks. It also struggles with autonomous exploration for complex behaviors like walking without initial guidance. Future research may involve integrating natural language instructions and comparing Arnold's internal activations with biological data from the spinal cord and motor cortex.
Summary updated