Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Models

Shuanghao Bai, Jing Lyu, Wanqi Zhou, Zhe Li, Dakai Wang, Lei Xing, Xiaoguang Zhao, Pengwei Wang, Zhongyuan Wang, Cheng Chi, Badong Chen, Shanghang Zhang

TL;DR

LaRA-VLA introduces a unified Vision-Language-Action framework that internalizes multimodal Chain-of-Thought (CoT) reasoning into continuous latent representations. By eliminating the need to generate explicit text or visual tokens at inference time, it slashes inference latency by up to 90% while achieving state-of-the-art success rates on complex, long-horizon robotic manipulation tasks.

What Problem Does This Paper Address?

Chain-of-Thought (CoT) reasoning has significantly boosted the ability of Vision-Language-Action (VLA) models to handle complex, multi-stage robotic tasks. However, existing CoT methods suffer from two major flaws:

  1. High Computational Overhead: Generating verbose textual reasoning steps autoregressively during inference inflates token sequences, balloons KV-cache usage, and drives control frequencies down to unacceptable levels (sometimes below 1 Hz), making real-time control impossible.
  2. Representational Mismatch: Most CoT methods rely on discrete tokens (language words or tokenized visual goals). Robotics, however, operates in continuous perception and control spaces. Forcing continuous physical interactions through discrete token bottlenecks creates a fundamental mismatch.

LaRA-VLA addresses these challenges by shifting the entire reasoning process from explicit, discrete token generation to implicit, continuous latent space dynamics.

Methods

LaRA-VLA utilizes Qwen3-VL as its backbone vision-language model and incorporates a 16-layer Diffusion Transformer as a dedicated action expert. To transition the model from explicit token generation to continuous "thinking," the authors propose a progressive three-stage curriculum training paradigm:

Figure 2. Overview of LaRA-VLA. Training proceeds in three stages: (i) explicit CoT fine-tuning with aligned visual prediction latents and inverse-dynamics supervision for actions; (ii) a curriculum-based transition from explicit CoT to compact text latents, gradually reducing the number of text tokens while increasing reliance on latent reasoning, where the latent representations are also implicitly supervised by visual and action signals; and (iii) adaptation of latent-conditioned VLM features to an action expert for efficient action generation without explicit CoT at inference time.
  • Stage I: Explicit CoT Fine-Tuning: The model is trained on fully annotated datasets using teacher forcing to generate explicit textual CoT tokens (subtasks, bounding boxes, motion descriptions) alongside predicting future visual states via a dedicated <img_next> token. Actions are supervised using an autoregressive discrete action loss.
  • Stage II: Curriculum-based Latent Replacement: Discrete textual CoT tokens are progressively masked out and replaced with learnable continuous <thinking> tokens. The model gradually learns to map its internal reasoning directly into these continuous latents, guided implicitly by future visual prediction objectives.
  • Stage III: Action Generation via Flow Matching: The explicit action token prediction is removed. The model is optimized using a flow matching objective, where the continuous action expert generates smooth, real-time control trajectories conditioned entirely on the learned multi-modal latent context.

To prevent representation collapse during latent learning, the target visual features are stabilized using an Exponential Moving Average (EMA) version of the online visual encoder.

Datasets

The authors developed an automated, anchor-first annotation pipeline using Qwen3-VL, GroundingDINO, and SAM3 to generate structured CoT data (subtasks, target object bounding boxes, and directional motion descriptors) without human intervention. Using this pipeline, they introduced two new benchmark datasets:

  • LIBERO-LaRA: Built on the simulated LIBERO manipulation benchmark.
  • Bridge-LaRA: Built on the SimplerEnv simulation environment for evaluating real-to-sim generalization.

They also applied this pipeline to a real-world robotic manipulation dataset collected on an Agilex Cobot Magic wheeled platform.

Key Insights and Ideas

  • Reasoning Doesn't Need Language at Inference: The core realization is that CoT is effective because it enforces structured intermediate computation, not because it is expressed in human language. Internalizing these structures into continuous latents preserves the computational benefits without the text-generation bottleneck.
  • Visual Latents as Multimodal Anchors: Instead of treating visual and textual reasoning separately, LaRA-VLA uses future-oriented visual goal latents to implicitly regularize and guide the textual reasoning latents, ensuring the "thoughts" remain grounded in physical reality.
  • Attention Constraints Matter: A specialized attention mechanism ensures that future image tokens and action tokens attend causally to the latent text representations, preventing information leakage while maximizing the utility of the compact latent "thought steps."

Key Results

  • Massive Latency Reduction: LaRA-VLA achieves an inference latency of just 135 ms per rollout on an NVIDIA A100 GPU. Compared to explicit CoT methods like ECoT-7B (4434 ms) and ThinkAct-7B (7513 ms), this represents a 90%+ reduction in computational overhead, making it highly suitable for real-time control.
  • Simulation Performance: On the LIBERO benchmark, LaRA-VLA achieved a state-of-the-art average success rate of 97.9% (outperforming OpenVLA's 76.5%). On SimplerEnv-WidowX, it achieved the highest average success rate of 68.8%.
  • Real-World Robustness: In physical robot experiments across four long-horizon tasks, LaRA-VLA outperformed strong baselines like ACT and ECoT, demonstrating superior resistance to error accumulation over extended task horizons. It also maintained high success rates under heavy visual perturbations (Gaussian noise and blur) due to its stable, semantically clustered latent space.

Limitations and Future Work

  • Expressiveness Bottleneck: To mitigate the risk of latent collapse, the current implementation restricts latent reasoning to a single token per step. This constraint may limit the model's capacity for highly intricate, multi-step reasoning.
  • Training Efficiency: While inference is incredibly fast, the curriculum learning strategy increases the number of tokens processed during training stages, leading to higher upfront training costs. Optimization of the training pipeline remains an open direction for future work.

Summary updated

Abstract

Vision-Language-Action (VLA) models benefit from chain-of-thought (CoT) reasoning, but existing approaches incur high inference overhead and rely on discrete reasoning representations that mismatch continuous perception and control. We propose Latent Reasoning VLA (LaRA-VLA), a unified VLA framework that internalizes multi-modal CoT reasoning into continuous latent representations for embodied action. LaRA-VLA performs unified reasoning and prediction in latent space, eliminating explicit CoT generation at inference time and enabling efficient, action-oriented control. To realize latent embodied reasoning, we introduce a curriculum-based training paradigm that progressively transitions from explicit textual and visual CoT supervision to latent reasoning, and finally adapts latent reasoning dynamics to condition action generation. We construct two structured CoT datasets and evaluate LaRA-VLA on both simulation benchmarks and long-horizon real-robot manipulation tasks. Experimental results show that LaRA-VLA consistently outperforms state-of-the-art VLA methods while reducing inference latency by up to 90\% compared to explicit CoT-based approaches, demonstrating latent reasoning as an effective and efficient paradigm for real-time embodied control. Project Page: https://loveju1y.github.io/Latent-Reasoning-VLA/
BibTeX
@inproceedings{bai2026latent,
  author = {Shuanghao Bai and Jing Lyu and Wanqi Zhou and Zhe Li and Dakai Wang and Lei Xing and Xiaoguang Zhao and Pengwei Wang and Zhongyuan Wang and Cheng Chi and Badong Chen and Shanghang Zhang},
  title = {{Latent Reasoning VLA: Latent Thinking and Prediction for Vision-Language-Action Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=P64X2q1n1H},
}
Figure