ALPS: Visual State Encoders for World Models

Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang XiearXiv 2026 · arXiv

TL;DR

ALPS (A Language-aligned, Pixel-renderable State encoder) is a new visual foundation model designed specifically for world models. By unifying physical-state estimation, language alignment, and pixel reconstruction into a single set of patch tokens, ALPS enables robots to follow instructions, plan future actions, and render "imagined" futures for human inspection—all using a single frozen representation.

What Problem Does This Paper Address?

Modern world models turn visual inputs into predictions and actions. To be truly effective, their visual "backbone" needs to do three things at once:

  1. State Estimation: Understand the physical location and orientation of objects for planning.
  2. Language Alignment: Connect visual scenes to natural language instructions (e.g., "pick up the black bowl").
  3. Pixel Reconstruction: Retain enough detail to render predictions back into images, allowing humans to verify what the model is "thinking."

Existing encoders usually specialize. CLIP is great at language but poor at spatial detail; DINOv2 excels at spatial features but lacks a native language interface; MAE is king of reconstruction but struggles with high-level semantics. Combining these often leads to a "tug-of-war" where improving one capability degrades another. The authors of ALPS set out to build a "jack-of-all-trades" that doesn't sacrifice mastery in any single domain.

Methods

The researchers developed a two-stage pretraining curriculum for a ViT-B/16 architecture:

  • Stage 1: Multitask Pretraining from Scratch. Instead of starting with a pretrained model, they trained ALPS from zero on image-text pairs using four simultaneous objectives:
  • Contrastive Alignment: Global image-text matching.
  • Captioning: Generative language modeling to deepen semantic understanding.
  • Reconstruction: A lightweight decoder forces tokens to preserve fine-grained pixel details.
  • Dense Correspondence: A local spatial objective that supervises patch relationships across different views.
  • Stage 2: Cross-view Self-distillation. To sharpen the model's physical-state awareness, they used a "student-teacher" setup. A trainable student predicts the tokens of a frozen Stage 1 teacher across different crops, scales, and augmentations.
Figure 1. One frozen encoder for state, language, and pixels. (a) Conceptual illustration of the three interfaces of ALPS. (b) Every encoder of Tab. 1 on LIBERO state-probe error, Flickr30k image-to-text R@1, and the ℓ1 error of a decoder trained on its tokens, bars oriented so that longer is better; n/a: no text tower released (not zero). No compared encoder is better than ALPS on all three.

Crucially, they found that a linear mean-pooled language interface (Stage 1) remains stable during Stage 2 refinement, whereas more complex pooling methods used in models like CLIP or SigLIP tend to "drift" and lose language performance when fine-tuned for spatial tasks.

Figure 2. Two-stage visual state pretraining. Stage 1 trains an encoder E from scratch with four objectives on shared patch tokens. Stage 2 initializes a student from E and keeps E frozen as the teacher. Given student tokens from an augmented crop A and query tokens encoding position and relative scale, a predictor regresses standardized teacher tokens from a clean crop B. Only patches of B whose centers lie inside A contribute to the loss. After training, the predictor is discarded and the refined student serves as ALPS.

Datasets

  • Pretraining: Stage 1 used DataComp-1B (web captions + synthetic captions). Stage 2 used a curated collection of 5 million images from public image-editing datasets (e.g., SEED-Data-Edit, UltraEdit) to provide high-resolution, object-centric scenes.
  • Evaluation: The model was tested on a massive suite of benchmarks including LIBERO (robot manipulation), Flickr30k/COCO (retrieval), ImageNet (classification), and several world-model planning tasks like PushT, Wall, and Granular.

Key Insights and Ideas

  • The Unified Token: ALPS proves that a single set of patch tokens can support diverse downstream "heads"—state probes, text towers, and pixel decoders—without needing separate encoders for each.
  • Stability of Linear Readouts: A key technical finding is that linear projection commutes with average pooling. This mathematical property helps ALPS preserve its language alignment during spatial refinement, a feat that non-linear models like SigLIP 2 struggle to achieve.
  • Visual Verifiability: Because the tokens are "pixel-renderable," world models using ALPS can generate visual rollouts of their plans. This makes the model's internal reasoning transparent to human operators.

Key Results

ALPS was compared against 15 leading public encoders and achieved impressive results:

  • State Estimation: It achieved the lowest aggregate state-probe error among all ViT-B models, matching the performance of the 3.5x larger V-JEPA 2-L.
  • Language: It maintained strong alignment, reaching 93.2 Recall@1 on Flickr30k, within 2 points of top-tier specialized image-text models.
  • Robotics: In downstream robot control (LIBERO), frozen ALPS tokens reached a mean policy success rate of 85.5%, outperforming specialized spatial encoders like DINOv3 and C-RADIOv3.
  • Planning: It matched or exceeded leading encoders in world-model planning across diverse simulated environments.

Limitations and Future Work

While ALPS is a significant step forward, the authors note a few areas for growth:

  • Scale: The current work focuses on the ViT-B scale; scaling up to ViT-L or ViT-H remains to be explored.
  • Video Data: ALPS is pretrained entirely on still images. Incorporating video and interaction data into the Stage 2 distillation could further enhance its temporal understanding.
  • Unified Planning: The current evaluation tests control and planning in separate settings. The ultimate goal is a single, language-guided, pixel-verifiable planner that integrates all three capabilities in real-time.

Summary updated

Abstract

World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.
BibTeX
@misc{liu2026alps,
  author = {Yanqing Liu and Letian Zhang and Chengzhi Liu and Zijun Wang and Jieru Mei and Hongliang Fei and Yuyin Zhou and Cihang Xie},
  title = {{ALPS: Visual State Encoders for World Models}},
  year = {2026},
  eprint = {2610.30120},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.30120},
}
Figure