Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space

Chao Chen, Zhixin Ma, Yongqi Li, Yupeng Hu, Yinwei Wei, Wenjie Li, Liqiang Nie

Abstract

Multimodal reasoning aims to enhance the capabilities of Multimodal Large Language Models (MLLMs) by incorporating intermediate reasoning steps before reaching the final answer. It has evolved from text-only reasoning to the integration of visual information, enabling the thought process to be conveyed through both images and text. Despite its effectiveness, current multimodal reasoning methods depend on explicit reasoning steps that require labor-intensive vision-text annotations and inherently introduce significant inference latency. To address these issues, we introduce multimodal latent reasoning with the advantages of multimodal representation, reduced annotation, and inference efficiency. To facilitate it, we propose Interleaved Vision-Text Latent Reasoning (IVT-LR), which injects both visual and textual information in the reasoning process within the latent space. Specifically, IVT-LR represents each reasoning step by combining two implicit parts: latent text (the hidden states from the previous step) and latent vision (a set of selected image embeddings). We further introduce a progressive multi-stage training strategy to enable MLLMs to perform the above multimodal latent reasoning steps. Experiments on M3CoT and ScienceQA demonstrate that our IVT-LR method achieves an average performance increase of 5.45% in accuracy, while simultaneously achieving a speed increase of over 5 times compared to existing approaches.

Introduction

To solve complex visual tasks, current MLLMs typically utilize a "Chain-of-Thought" (CoT) approach, generating a series of explicit, human-readable sentences and image patches before arriving at a final answer. While effective, this approach suffers from high inference latency due to the autoregressive nature of token generation and a heavy annotation burden for training. The authors of this paper propose that a model should not have to "speak its thoughts out loud" to think effectively. Instead, it should process multimodal information internally—"in the dark"—to save time and avoid the constraints of natural language.

Interleaved Vision-Text Latent Reasoning (IVT-LR)

The core contribution is the Interleaved Vision-Text Latent Reasoning (IVT-LR) framework. Instead of predicting discrete words at each reasoning step, the model produces a "latent" representation that interleaves two modalities:

  • Latent Text: The model utilizes its internal hidden states from the previous step to represent textual reasoning logic. This preserves richer information than a single decoded word could provide.
  • Latent Vision: At each step, an attention-based mechanism selects the $k$ most relevant image embeddings (patches). This allows the model to dynamically "look" at different parts of the image as its internal reasoning progresses.
Overview of the IVT-LR process: The model processes a question through multiple latent steps, interleaving text and vision representations internally before producing a final answer.

To teach the model this implicit reasoning, a Progressive Multi-Stage Training strategy is used. Starting from a model that generates full, explicit text rationales (Stage 0), the training progressively replaces explicit steps with a special <latent> token. By Stage 3, the entire reasoning process happens within the latent space.

Experimental Setup

The model was evaluated on two primary benchmarks:

  • M3CoT: A large-scale benchmark for multimodal chain-of-thought reasoning across math and science.
  • ScienceQA: A dataset of multi-choice science questions requiring visual and textual synthesis.

The researchers used Qwen2-VL-7B and Chameleon-7B as the underlying backbone models.

Key Results and Insights

The IVT-LR method demonstrates significant improvements in both performance and efficiency:

  • Accuracy: On M3CoT, IVT-LR achieved 71.8% accuracy (using Qwen2-VL), outperforming the Chain-of-Focus baseline by 7.5%.
  • Speed: By moving reasoning to the latent space, the number of autoregressive steps dropped significantly (e.g., from 185 steps to ~11). This resulted in a speedup of over 5 times, with average generation time on M3CoT dropping from 2.63s to 0.65s.
  • Dynamic Visual Focus: Attention analysis revealed that the model shifts its focus over time—initially prioritizing image features and gradually shifting toward internal "latent text" as it converges on an answer.
  • Escaping Language Constraints: Latent reasoning allows the model to maintain continuous, high-dimensional representations, avoiding the information loss inherent in forcing thoughts into human language.

Conclusion and Future Work

IVT-LR represents a shift toward more efficient and powerful multimodal reasoning. While highly effective, current limitations include a fixed number of reasoning steps. Future work may explore "dynamic" reasoning where the model adjusts the number of latent steps based on task complexity. Additionally, extending this to sequential tasks like robotic planning and improving the interpretability of "latent thoughts" remain key areas for exploration.

Summary updated

Abstract

Multimodal reasoning aims to enhance the capabilities of MLLMs by incorporating intermediate reasoning steps before reaching the final answer. It has evolved from text-only reasoning to the integration of visual information, enabling the thought process to be conveyed through both images and text. Despite its effectiveness, current multimodal reasoning methods depend on explicit reasoning steps that require labor-intensive vision-text annotations and inherently introduce significant inference latency. To address these issues, we introduce multimodal latent reasoning with the advantages of multimodal representation, reduced annotation, and inference efficiency. To facilicate it, we propose Interleaved Vision-Text Latent Reasoning (IVT-LR), which injects both visual and textual information in the reasoning process within the latent space. Specifically, IVT-LR represents each reasoning step by combining two implicit parts: latent text (the hidden states from the previous step) and latent vision (a set of selected image embeddings). We further introduce a progressive multi-stage training strategy to enable MLLMs to perform the above multimodal latent reasoning steps. Experiments on M3CoT and ScienceQA demonstrate that our IVT-LR method achieves an average performance increase of 5.45% in accuracy, while simultaneously achieving a speed increase of over 5 times compared to existing approaches. Code available at https://github.com/FYYDCC/IVT-LR.
BibTeX
@misc{chen2025reasoning,
  author = {Chao Chen and Zhixin Ma and Yongqi Li and Yupeng Hu and Yinwei Wei and Wenjie Li and Liqiang Nie},
  title = {{Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space}},
  year = {2025},
  eprint = {2510.12603},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12603},
}
Figure