TL;DR
Library of Hands (LoH) is a training-free method that fixes robot failures caused by changes in hardware or camera perspective without retraining the underlying world model. By using the frozen world model itself as a sensor to "read" the actual actions being performed, LoH identifies and remembers new command-to-action mappings, doubling success rates in dynamic environments.
What Problem Does This Paper Address?
Modern robot planning often relies on world models—neural networks that predict how a scene will change given a specific action. However, these models are brittle. If you move the camera 90 degrees or swap a motor controller for one with a different gain, the world model’s understanding of physics remains correct, but the "meaning" of a command changes. It’s like trying to play a video game where someone has rotated your joystick; you know what you want to happen, but your inputs produce the wrong results.
Current solutions usually involve retraining the model or fine-tuning it on new data, which is computationally expensive and often causes the model to "forget" previous setups. The authors argue that we shouldn't fix the world model—which still understands physics perfectly—but rather fix the "hand": the interface between the planner's commands and the robot's actual movements.
Methods
LoH is a modular addition to a frozen world model that operates in a four-stage loop:
- Sense: After the robot executes a block of commands, LoH looks at the resulting change in the environment. It "queries" the frozen world model to find the action that best explains that change. Essentially, it uses the model in reverse to perceive what the robot actually did.
- Explain: It compares the "read" action to the original command to estimate a mapping (a "hand"). This mapping can account for rotations, scaling (gain), mirrors, offsets, and even temporal delays.
- Remember: LoH maintains a library of these identified "hands." Using Bayesian model comparison, it decides whether the current behavior matches a known hand in its library or represents a completely new setup.
- Act: When planning the next move, the robot's planner (e.g., Cross-Entropy Method) sends its commands through the identified "hand" mapping before they reach the motors, ensuring the intended physical action is the one actually performed.
Datasets
The authors evaluated LoH across several simulated environments and world models:
- Simulated Tasks: PushT (pushing a T-block), Reacher (moving an arm to a target), and Two-Room (navigation).
- World Models: LeWM (LeWorldModel), DINO-WM, and JEPA-WM.
- Real-Data Benchmarks: RoboCasa, a simulated 3D kitchen environment using models pretrained on the DROID dataset (a large-scale real-robot manipulation dataset).
Key Insights and Ideas
- The Model is the Sensor: One of the most elegant ideas in the paper is that a high-quality world model contains enough information to act as its own proprioceptive sensor. You don't need extra hardware to know the camera moved; the mismatch between prediction and reality tells you everything you need to know.
- Recalling vs. Re-learning: In real-world deployments (like a lab with three different camera mounts), setups recur. By storing hands in a library, LoH can switch back to a known configuration in just one or two steps, whereas "no-memory" baselines have to re-identify the mapping from scratch every time.
- Performance Scales with Model Quality: The "gain" or benefit of using LoH is directly correlated with how well the world model can read its own actions ($R^2$). As world models get better, the effectiveness of LoH-style mapping increases.
Key Results
- Doubled Success: In closed-loop tests where the "hand" changed between episodes, LoH increased success rates from 28.6% to 62.3% compared to a frozen planner.
- Closing the Gap: LoH closed approximately 70% of the performance gap between a failing frozen model and an "oracle" planner that was explicitly told the correct hand mapping.
- Zero-Shot Adaptation: On models pretrained on real robot data (RoboCasa), LoH successfully corrected for camera yaw changes that otherwise caused the frozen planners to fail almost entirely.
- Low Overhead: When the environment didn't change, LoH only cost a negligible 0.6 percentage points in success, proving it is safe to leave "always on."
Limitations and Future Work
- Changes of the World: LoH assumes the hand changes, not the physics. If the friction of the floor changes or the object's center of gravity shifts, LoH can struggle, as these are changes to the world model's domain, not the action interface.
- Detection Latency: There is a "blind" first block in every episode where the robot uses the previous hand before it has enough data to sense a change. This results in a small but measurable performance hit immediately after a switch.
- Linear Assumption: The current "library" uses linear maps. While this covers rotations, gains, and offsets, it cannot yet model highly non-linear "hands," such as complex motor dead-zones or state-dependent control laws.
The authors suggest that future work could involve modules that discover new kinds of hand mappings online, moving beyond the five fixed classes used in this study.
Summary updated