Vision-Language-Action or World-Action? Just Action Models

Yonghao Yu, Lang Huang, Runyi Li, Toshihiko YamasakiarXiv 2026 · arXiv

TL;DR

Just Action Models (JAM) demonstrate that the "secret sauce" of high-performing world-action models isn't the act of generating future images, but rather the supervision provided by future observations. By replacing expensive visual generation with a simple "foresight loss" that predicts future latent representations, JAM achieves state-of-the-art robot control while being up to 73% more computationally efficient during training.

What Problem Does This Paper Address?

In the quest for better robot "brains," two main camps have emerged: Vision-Language-Action (VLA) models, which adapt internet-scale knowledge from text and images to predict actions, and World-Action Models (WAMs), which learn by predicting how the world changes (generating future video frames).

While WAMs often show superior control performance, they come with a heavy "generation tax"—producing high-resolution images or videos during training is computationally expensive and slow. This paper investigates whether the generation itself is necessary for good performance, or if the model just needs to learn from the information contained in future observations.

Figure 1: The three choices that separate VLAs, WAMs, and JAM: backbone initialization, the objective on recorded future observations, and the modules kept at inference. (a) The VLA reference uses no future objective, only the action loss Lact. (b) WAMs add visual generation loss LG. The dashed loop shows generated future observations entering the action expert in some WAMs. (c) JAM uses foresight loss LF across pretraining families, discards its supervision branch, and retains the corresponding action path.

Methods

The authors introduce Just Action Model (JAM), a framework that decouples the choice of model backbone from the training objective.

  1. Foresight Loss: Instead of a complex visual generator, JAM uses a small, training-only "prediction head." This head takes current visual features and tries to predict the features of future observations.
  2. Future Anchors: The targets for this prediction are provided by a "future anchor"—a frozen, pretrained model (like a VAE or a vision backbone) that processes recorded future frames.
  3. Discardable Machinery: Crucially, the prediction head and future anchor are used only during training. At deployment (inference), they are discarded. The final robot policy is just a standard visual backbone and an action expert, making it as fast and lightweight as a VLA.
  4. Backbone Agnostic: Because it doesn't require a generative path, JAM can be applied to almost any pretrained visual model, including generative DiTs, self-supervised encoders (like DINOv2), or vision-language models (like Qwen).
Figure 2: JAM training (left) and deployment (right), with the notation of Section 3.2. The input and text encoders stand for architecture-specific interfaces. Recorded futures f1:T enter only the future anchor branch inside the dashed box; deployment discards this branch and the prediction head, leaving the backbone and action expert.

Datasets

The researchers evaluated JAM across several rigorous robot manipulation benchmarks:

  • LIBERO: A suite of 40 tasks focused on lifelong learning and knowledge transfer.
  • LIBERO-Plus: A more challenging version of LIBERO that introduces significant perturbations in camera angles, lighting, and background to test robustness.
  • RoboTwin 2.0: A benchmark for complex bimanual (two-armed) manipulation tasks in simulated environments.

Key Insights and Ideas

  • Generation is Optional, Foresight is Key: You don't need to "see" the future (generate pixels) to learn from it. Predicting future latent representations provides a training signal that is just as effective, if not more so, for learning control policies.
  • Massive Efficiency Gains: By avoiding pixel-level generation and caching future targets, JAM drastically reduces the computational cost of training. It cuts estimated online training FLOPs by up to 73% compared to models like FastWAM.
  • Universal Applicability: The "foresight recipe" is a universal booster. It improved performance across all tested model families, whether they started as image editors, video generators, or chatty vision-language models.
  • The "Future-Offset" Advantage: The study found that predicting what happens later in a trajectory (future-offset targets) is significantly more beneficial for control than just trying to represent the current state more accurately.

Key Results

  • State-of-the-Art Performance: On the challenging LIBERO-Plus benchmark, JAM (using an Ovis-U1 backbone) achieved a 79.95% success rate, a massive jump over its generative counterpart, ImageWAM, which scored 65.27%.
  • RoboTwin Success: JAM reached a 91.4% success rate on RoboTwin 2.0, matching or slightly exceeding the performance of more complex world models.
  • Training Speedup: JAM reduces estimated online training FLOPs per sample by 73.0% relative to FastWAM and 36.8% relative to ImageWAM. Actual GPU time measurements on a GH200 showed similar reductions (67.3% and 37.8%, respectively).

Limitations and Future Work

  • Target Sensitivity: The authors noted that the "best" target representation for the foresight loss (e.g., VAE latents vs. backbone features) depends on the specific architecture being used. There isn't a one-size-fits-all target yet.
  • Data Dependency: While JAM removes the need for generation, it still requires high-quality recorded trajectories that include future observations for the training phase.
  • Future Directions: The authors suggest exploring more complex foresight targets or "adaptive anchors" that might evolve during training to provide even better supervision.

In summary, JAM offers a "best of both worlds" approach: the high performance of world models with the efficiency and simplicity of vision-language-action models.

Summary updated

Abstract

The vision-language-action (VLA) and world-action model (WAM) communities debate whether control policies should inherit pretrained knowledge from vision-language models or from visual generators. The debate, however, bundles three separable choices: backbone initialization, the objective applied to future observations, and whether the machinery for that future loss is kept at inference. A gap between a VLA and a WAM therefore cannot simply be attributed to any one of them. We show that decoupling these choices yields a common framework for both families, which we term Just Action Model (JAM): a pretrained visual backbone with an action expert, trained with an action loss and a foresight loss. The foresight loss trains current-observation features to predict frozen encodings of the recorded future through a prediction head discarded after training, so no visual generation is involved in either training or inference, and the backbone can be chosen freely. We first compare objectives within fixed policy architectures, then evaluate the foresight recipe across backbone families. The resulting JAM policies outperform their matched FastWAM and ImageWAM counterparts on LIBERO and LIBERO-Plus, and achieve similar or slightly higher success rates on RoboTwin. Compared with FastWAM, JAM improves the success rates from 97.0%/55.3%/91.2% to 98.2%/64.5%/91.4%, respectively. With cached foresight targets, JAM reduces estimated online training FLOPs per sample per update by 73% relative to FastWAM. Our results show that future supervision improves control without visual generation or additional inference machinery, and strong foresight performance is attainable with either generator-initialized or vision-language backbones. Project Page: https://y0uroy.github.io/JAM
BibTeX
@misc{yu2026vision,
  author = {Yonghao Yu and Lang Huang and Runyi Li and Toshihiko Yamasaki},
  title = {{Vision-Language-Action or World-Action? Just Action Models}},
  year = {2026},
  eprint = {2610.37219},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37219},
}
Figure