TL;DR
Just Action Models (JAM) demonstrate that the "secret sauce" of high-performing world-action models isn't the act of generating future images, but rather the supervision provided by future observations. By replacing expensive visual generation with a simple "foresight loss" that predicts future latent representations, JAM achieves state-of-the-art robot control while being up to 73% more computationally efficient during training.
What Problem Does This Paper Address?
In the quest for better robot "brains," two main camps have emerged: Vision-Language-Action (VLA) models, which adapt internet-scale knowledge from text and images to predict actions, and World-Action Models (WAMs), which learn by predicting how the world changes (generating future video frames).
While WAMs often show superior control performance, they come with a heavy "generation tax"—producing high-resolution images or videos during training is computationally expensive and slow. This paper investigates whether the generation itself is necessary for good performance, or if the model just needs to learn from the information contained in future observations.
Methods
The authors introduce Just Action Model (JAM), a framework that decouples the choice of model backbone from the training objective.
- Foresight Loss: Instead of a complex visual generator, JAM uses a small, training-only "prediction head." This head takes current visual features and tries to predict the features of future observations.
- Future Anchors: The targets for this prediction are provided by a "future anchor"—a frozen, pretrained model (like a VAE or a vision backbone) that processes recorded future frames.
- Discardable Machinery: Crucially, the prediction head and future anchor are used only during training. At deployment (inference), they are discarded. The final robot policy is just a standard visual backbone and an action expert, making it as fast and lightweight as a VLA.
- Backbone Agnostic: Because it doesn't require a generative path, JAM can be applied to almost any pretrained visual model, including generative DiTs, self-supervised encoders (like DINOv2), or vision-language models (like Qwen).
Datasets
The researchers evaluated JAM across several rigorous robot manipulation benchmarks:
- LIBERO: A suite of 40 tasks focused on lifelong learning and knowledge transfer.
- LIBERO-Plus: A more challenging version of LIBERO that introduces significant perturbations in camera angles, lighting, and background to test robustness.
- RoboTwin 2.0: A benchmark for complex bimanual (two-armed) manipulation tasks in simulated environments.
Key Insights and Ideas
- Generation is Optional, Foresight is Key: You don't need to "see" the future (generate pixels) to learn from it. Predicting future latent representations provides a training signal that is just as effective, if not more so, for learning control policies.
- Massive Efficiency Gains: By avoiding pixel-level generation and caching future targets, JAM drastically reduces the computational cost of training. It cuts estimated online training FLOPs by up to 73% compared to models like FastWAM.
- Universal Applicability: The "foresight recipe" is a universal booster. It improved performance across all tested model families, whether they started as image editors, video generators, or chatty vision-language models.
- The "Future-Offset" Advantage: The study found that predicting what happens later in a trajectory (future-offset targets) is significantly more beneficial for control than just trying to represent the current state more accurately.
Key Results
- State-of-the-Art Performance: On the challenging LIBERO-Plus benchmark, JAM (using an Ovis-U1 backbone) achieved a 79.95% success rate, a massive jump over its generative counterpart, ImageWAM, which scored 65.27%.
- RoboTwin Success: JAM reached a 91.4% success rate on RoboTwin 2.0, matching or slightly exceeding the performance of more complex world models.
- Training Speedup: JAM reduces estimated online training FLOPs per sample by 73.0% relative to FastWAM and 36.8% relative to ImageWAM. Actual GPU time measurements on a GH200 showed similar reductions (67.3% and 37.8%, respectively).
Limitations and Future Work
- Target Sensitivity: The authors noted that the "best" target representation for the foresight loss (e.g., VAE latents vs. backbone features) depends on the specific architecture being used. There isn't a one-size-fits-all target yet.
- Data Dependency: While JAM removes the need for generation, it still requires high-quality recorded trajectories that include future observations for the training phase.
- Future Directions: The authors suggest exploring more complex foresight targets or "adaptive anchors" that might evolve during training to provide even better supervision.
In summary, JAM offers a "best of both worlds" approach: the high performance of world models with the efficiency and simplicity of vision-language-action models.
Summary updated