Abstract
World Foundation Models (WFMs) have demonstrated remarkable capabilities in generating realistic video content, yet they often lack the physical precision and action-conditioning necessary for robotic manipulation. We introduce AdaPower, a framework that transforms general-purpose WFMs into action-conditioned "specialist" world models. By employing Temporal-Spatial Test-Time Training (TS-TTT) and Memory Persistence (MP), AdaPower enables high-fidelity dynamics prediction in novel environments. When integrated into a Model Predictive Control (MPC) framework, AdaPower empowers existing Vision-Language-Action (VLA) models, increasing success rates on unseen tasks by over 41% without requiring additional policy training.
Introduction
The emergence of World Foundation Models (WFMs) like Sora and Cosmos has revolutionized video generation. However, for robotics, "visual realism" is insufficient; models must be physically accurate and responsive to specific robot actions. Current WFMs primarily focus on text-to-video generation, whereas robots require action-to-video capabilities to predict the outcomes of their movements. Furthermore, standard models often struggle with generalization to new environments and long-horizon consistency.
AdaPower addresses these challenges by adapting a pre-trained WFM (specifically Cosmos-Predict-2B) into a high-fidelity "mental simulator." This Specialist World Model (SWM) is then used within a Model Predictive Control (MPC) loop to evaluate and select the best action sequences proposed by a pre-trained VLA policy, such as CogACT or $\pi_0$.
Methodology
AdaPower introduces several architectural innovations to bridge the gap between general video generation and precise robotic simulation.
Action-Conditioned Adaptation
The framework replaces the standard text encoder with a multi-layer perceptron (MLP) action encoder. This allows the model to process robot action sequences as tokens, enabling precise action-to-video prediction rather than relying on vague text descriptions.
Temporal-Spatial Test-Time Training (TS-TTT)
To handle distribution shifts in new environments, AdaPower performs self-supervised updates during inference. It leverages spatial and temporal redundancies in video data—such as static backgrounds and local pixel similarities—to reconstruct the current scene accurately. This "test-time training" allows the model to adapt to unfamiliar lighting or objects on the fly.
Memory Persistence (MP)
To maintain object consistency over long sequences, AdaPower utilizes a DINOv2 encoder to extract object-centric features from historical frames. These features are integrated via cross-attention, preventing the model from "forgetting" object states or locations during long-horizon tasks.
Results
The framework was evaluated on the LIBERO benchmark and in real-world settings, demonstrating significant improvements over base VLA models.
- Success Rate Boost: On LIBERO-LONG unseen tasks, the base CogACT model's success rate improved from near-zero to 41.5% when empowered by AdaPower.
- Efficiency: AdaPower is highly efficient, adding only approximately 150M parameters (<10% of the base model size), making it a lightweight alternative to full policy fine-tuning.
- Real-World Transfer: In physical experiments with a Franka Research 3 arm, AdaPower increased zero-shot success rates by over 30% on tasks involving complex manipulations like pouring and stacking bowls.
Conclusion
AdaPower demonstrates that WFMs can serve as powerful dynamics engines for robotic planning rather than just data generators. By enabling on-the-fly adaptation and long-term consistency, it provides a scalable path for improving robotic manipulation in diverse, unseen environments. Future work will focus on reducing inference latency and extending the framework to more complex physical dynamics, such as deformable objects.
Summary updated