Emergent Vision-Shaped Thinking in Text-Only Reasoning through Multimodal Training

Qi Zhang, Bohan Wang, Qianru Sun, Hanwang ZhangarXiv 2026 · arXiv

Does multimodal training—teaching a model to "see" and "read" simultaneously—change how it processes plain text? According to a new paper from researchers at Nanyang Technological University and Singapore Management University, the answer is a resounding yes.

The study introduces the concept of Emergent Vision-Shaped Thinking (EVT): a phenomenon where a language model’s text-only reasoning causally relies on internal representations that were either introduced or significantly strengthened during multimodal training.

TL;DR

Multimodal training doesn't just help models understand images; it fundamentally reshapes the "geometry" of how they reason about text. By surgically removing a specific "vision-language axis" from a model's internal states, researchers found they could selectively break its ability to solve spatial puzzles (like navigation) while leaving its logical reasoning (like boolean algebra) completely intact.

What Problem Does This Paper Address?

We’ve known for a while that Vision-Language Models (VLMs) often outperform their text-only counterparts on spatial reasoning tasks, even when no images are provided. However, it hasn't been clear why. Is the model simply better at language, or has the visual training created a new kind of internal "spatial scaffold"?

Previous evidence was mostly behavioral—observing what the model does. This paper moves into the realm of mechanistic interpretability, seeking to prove a causal link: showing that the model uses visual-style representations to solve text-only problems.

Figure 1: Emergent Vision-Shaped Thinking. Multimodal mod- els use vision-shaped structure even on text-only spatial reasoning. Ablating a vision–language direction selectively damages spatial- imagery tasks (Navigation) while leaving symbol-manipulation tasks (Logic) intact.

Methods

The researchers tested 10 multimodal configurations across six model families (including Chameleon, Qwen3-VL, and Janus-Pro). Their approach involved three main steps:

  1. Identifying the VL Axis: They identified a "vision-language (VL) axis" in the model's hidden-state space. This was done using a "DiffMean" estimator—essentially finding the vector that best separates textual representations from visual ones within the model's internal layers.
  2. Causal Ablation: During inference on text-only tasks, they "ablated" (removed) the model's projection onto this VL axis. If the model relies on "vision-shaped" thinking, removing this specific direction should hinder its performance.
  3. Double Dissociation: To ensure they weren't just causing generic brain damage to the model, they compared performance on two types of tasks:
  4. Spatial-Imagery Tasks: Textual inputs that invite mental scene construction (e.g., "You turn right, walk 10m... where are you?").
  5. Symbol-Manipulation Tasks: Tasks solvable by formal rules alone (e.g., evaluating boolean expressions).

Datasets

The study utilized a wide battery of benchmarks:

  • Primary: BIG-Bench Hard (BBH), specifically the navigate and boolean_expressions tasks.
  • External Spatial Reasoning: bAbI, SpartQA, and StepGame.
  • Mental Simulation: Newly constructed or adapted tasks for mental rotation, paper folding, and perspective taking.
  • General Knowledge: MMLU-Pro and GSM8K were used as controls to ensure the interventions were selective and didn't destroy the model's general intelligence.

Key Insights and Ideas

  • The "Causal Handle" in Unified Models: In models that both generate and understand images (like Janus-Pro), the researchers found that the "causal handle" for spatial reasoning lives primarily in the generation pathway. While the "understanding" pathway also encodes spatial info, it is "entangled" with general capacity and isn't the primary lever the model uses for text-only spatial reasoning.
  • Introduction vs. Amplification: Multimodal training affects models differently. In the Chameleon family, visual training introduces a new spatial scaffold that wasn't there in the text-only base. In the Qwen family, it amplifies and "purifies" a scaffold that already existed in a weak form.
  • Probing is Not Using: A major takeaway for the interpretability community is that decodability $\neq$ causal use. Just because a researcher can "probe" a model's hidden states to find spatial information doesn't mean the model actually uses that information to solve a task. The authors propose three diagnostics—concentration, alignment, and semantic location—to bridge this gap.

Key Results

The most striking result is the 33.2 percentage point (pp) double dissociation found in Chameleon-7B. Ablating the VL axis caused:

  • Accuracy on BBH navigate to drop by 27.6 pp.
  • Accuracy on boolean expressions to actually increase by 5.6 pp.

As models scale, this effect becomes even more pronounced. In Qwen3-VL, the "direction-specificity" (measured as a z-score) of the spatial effect exploded from +27.7 at the 2B scale to a massive +262.5 at the 32B scale, indicating that larger models develop much more specialized and "pure" spatial representations.

Figure 4b: In Qwen3-VL, the spatial-effect becomes strongest at 32B, while control-task effects stay near zero.

Limitations and Future Work

  • Linearity: The study focused on linear and low-rank interventions. It’s possible that more complex, non-linear "vision-shaped" mechanisms exist that this method missed.
  • Task Taxonomy: While the researchers tested a variety of tasks, they acknowledge their battery is a "diagnostic" rather than a complete taxonomy of human-like spatial reasoning.
  • The "How" vs. the "What": The study proves that these representations are necessary for spatial reasoning, but it doesn't fully map out the specific circuits that write and read these signals.

Future work could involve "circuit breaking" at a more granular level or investigating how different multimodal pre-training objectives (e.g., contrastive vs. generative) change the resulting internal geometry.

Summary updated

Abstract

Does multimodal training shape how a language model thinks, even when the input is text only? We test this with targeted causal ablations across 10 multimodal configurations from 6 model families, with matched text-only backbones and cross-family transfer controls. We contrast spatial-imagery tasks that invite mental scene construction with symbol-manipulation tasks solvable by symbolic rules alone. We identify a vision-language axis in hidden-state space and ablate its projection at inference. In Chameleon-7B, this ablation drops accuracy on BIG-Bench Hard (BBH) navigate by 27.6 pp while leaving boolean expressions intact (+5.6 pp), a 33.2 pp double dissociation. The effect is direction-specific and layer-localized, extends to mental-simulation tasks such as mental rotation, paper folding, and perspective taking, and is not explained by task difficulty, reasoning depth, or sequential update load. Together, these results reveal emergent vision-shaped thinking (EVT): text-only reasoning that causally recruits representations introduced or strengthened by vision-language training, without assuming an image-like internal format. The measured effect is spatially concentrated at 7-8B and broadens to non-spatial visual attribute binding in the tested 32B configuration. In dual-path unified models (BAGEL, Janus-Pro), the causal handle localizes to the generation pathway as a compact one-dimensional carrier. The understanding pathway also encodes spatial information in a higher-rank, probe-decodable form, but is not the causal handle. To separate decodability from causal use, we analyze concentration, alignment, and semantic location. These diagnostics explain why behavior, probing, and intervention can diverge. Multimodal training thus reshapes the geometry of text-only reasoning, not just how visual input is processed.
BibTeX
@misc{zhang2026emergent,
  author = {Qi Zhang and Bohan Wang and Qianru Sun and Hanwang Zhang},
  title = {{Emergent Vision-Shaped Thinking in Text-Only Reasoning through Multimodal Training}},
  year = {2026},
  eprint = {2611.14942},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14942},
}
Figure