Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought

Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang, Alex Jinpeng Wang, Zhi Chen, Wanxiang Che, Libo Qin

Introduction

Large Vision-Language Models (LVLMs) have seen significant performance boosts through "Chain-of-Thought" (CoT) prompting, where models are tasked with explaining their reasoning step-by-step. This approach generally falls into two categories: Textual-MCoT, which generates text rationales, and Interleaved-MCoT, which produces a mixture of text and edited or generated images. Despite their success, the underlying mechanism explaining why these methods work remained unclear. This paper introduces the concept of "Visual Thoughts" (VT) as a unified framework, positing that these intermediate reasoning steps act as a distilled information cache that bridges the gap between raw pixels and high-level logic.

The Visual Thought Framework

The authors define Visual Thoughts as intermediate, logic-driven cross-modal representations. Using a computer-system analogy, raw images are treated as slow "external storage," while Visual Thoughts function as a "high-speed cache." This cache extracts instruction-relevant regions and stores them in a format that is more accessible for the model's deeper reasoning layers.

Visual Thoughts are categorized into four distinct expressions:

  1. Natural Language (N-LANG): Descriptive captions.
  2. Structured Language (S-LANG): JSON-format scene graphs.
  3. Edited Image (E-IMG): Images modified with tools like bounding boxes or segmentation.
  4. Generative Image (G-IMG): Entirely new images generated to assist reasoning.
Textual-MCoT generates linguistic rationales that serve as a text-based Visual Thought cache.

Mechanisms of Reasoning: Attention Shifting

A critical insight from the study is the phenomenon of Attention Shifting. By performing "disturbation" analysis—manually blocking information flow in transformer layers—the researchers found that in deeper layers (typically after layer 12 in models like LLaVA), the model's attention to the original raw image drops to nearly zero. Instead, the model focuses almost entirely on the generated Visual Thoughts.

This suggests that the model does not re-process raw pixels at every step but relies on the distilled logic stored in the VT. Interestingly, the effectiveness of a Visual Thought depends more on its expression clarity and conciseness than on its visual fidelity to the original source.

Interleaved-MCoT produces visual rationales, such as function plots, to facilitate complex logical reasoning.

Experimental Evaluation

The framework was tested across several benchmarks, including MMVP (coarse perception), V*Bench (fine-grained attributes), M3CoT (multi-step reasoning), CoMT (complex operations), and IsoBench (mathematical/logic tasks).

Key results include:

  • Consistent Improvement: Visual Thoughts improved performance across LLaVA-1.5, Qwen2-VL, and GPT-4o models.
  • Visual vs. Textual: Image-form Visual Thoughts generally outperformed text-only rationales, especially in complex scenarios where visual features are difficult to describe linguistically.
  • Quality Correlation: There is a strong positive correlation (Spearman’s $\rho > 0.8$) between the clarity of a Visual Thought and the model's eventual accuracy.

Conclusion and Future Directions

The "Visual Thoughts" framework provides a unified explanation for the success of Multimodal Chain-of-Thought. By shifting internal attention from raw pixels to distilled, logic-driven representations, models can perform more sophisticated reasoning. Future research points toward "Diverse-VT," a strategy that ensembles different types of visual thoughts (e.g., combining a caption with an edited image) to provide a multi-perspective reasoning advantage.

Summary updated

BibTeX
@inproceedings{cheng2025visual,
  author = {Zihui Cheng and Qiguang Chen and Xiao Xu and Jiaqi Wang and Weiyun Wang and Hao Fei and Yidong Wang and Alex Jinpeng Wang and Zhi Chen and Wanxiang Che and Libo Qin},
  title = {{Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=xPcKmKSEis},
}
Figure