Does multimodal training—teaching a model to "see" and "read" simultaneously—change how it processes plain text? According to a new paper from researchers at Nanyang Technological University and Singapore Management University, the answer is a resounding yes.
The study introduces the concept of Emergent Vision-Shaped Thinking (EVT): a phenomenon where a language model’s text-only reasoning causally relies on internal representations that were either introduced or significantly strengthened during multimodal training.
TL;DR
Multimodal training doesn't just help models understand images; it fundamentally reshapes the "geometry" of how they reason about text. By surgically removing a specific "vision-language axis" from a model's internal states, researchers found they could selectively break its ability to solve spatial puzzles (like navigation) while leaving its logical reasoning (like boolean algebra) completely intact.
What Problem Does This Paper Address?
We’ve known for a while that Vision-Language Models (VLMs) often outperform their text-only counterparts on spatial reasoning tasks, even when no images are provided. However, it hasn't been clear why. Is the model simply better at language, or has the visual training created a new kind of internal "spatial scaffold"?
Previous evidence was mostly behavioral—observing what the model does. This paper moves into the realm of mechanistic interpretability, seeking to prove a causal link: showing that the model uses visual-style representations to solve text-only problems.
Methods
The researchers tested 10 multimodal configurations across six model families (including Chameleon, Qwen3-VL, and Janus-Pro). Their approach involved three main steps:
- Identifying the VL Axis: They identified a "vision-language (VL) axis" in the model's hidden-state space. This was done using a "DiffMean" estimator—essentially finding the vector that best separates textual representations from visual ones within the model's internal layers.
- Causal Ablation: During inference on text-only tasks, they "ablated" (removed) the model's projection onto this VL axis. If the model relies on "vision-shaped" thinking, removing this specific direction should hinder its performance.
- Double Dissociation: To ensure they weren't just causing generic brain damage to the model, they compared performance on two types of tasks:
- Spatial-Imagery Tasks: Textual inputs that invite mental scene construction (e.g., "You turn right, walk 10m... where are you?").
- Symbol-Manipulation Tasks: Tasks solvable by formal rules alone (e.g., evaluating boolean expressions).
Datasets
The study utilized a wide battery of benchmarks:
- Primary: BIG-Bench Hard (BBH), specifically the navigate and boolean_expressions tasks.
- External Spatial Reasoning: bAbI, SpartQA, and StepGame.
- Mental Simulation: Newly constructed or adapted tasks for mental rotation, paper folding, and perspective taking.
- General Knowledge: MMLU-Pro and GSM8K were used as controls to ensure the interventions were selective and didn't destroy the model's general intelligence.
Key Insights and Ideas
- The "Causal Handle" in Unified Models: In models that both generate and understand images (like Janus-Pro), the researchers found that the "causal handle" for spatial reasoning lives primarily in the generation pathway. While the "understanding" pathway also encodes spatial info, it is "entangled" with general capacity and isn't the primary lever the model uses for text-only spatial reasoning.
- Introduction vs. Amplification: Multimodal training affects models differently. In the Chameleon family, visual training introduces a new spatial scaffold that wasn't there in the text-only base. In the Qwen family, it amplifies and "purifies" a scaffold that already existed in a weak form.
- Probing is Not Using: A major takeaway for the interpretability community is that decodability $\neq$ causal use. Just because a researcher can "probe" a model's hidden states to find spatial information doesn't mean the model actually uses that information to solve a task. The authors propose three diagnostics—concentration, alignment, and semantic location—to bridge this gap.
Key Results
The most striking result is the 33.2 percentage point (pp) double dissociation found in Chameleon-7B. Ablating the VL axis caused:
- Accuracy on BBH navigate to drop by 27.6 pp.
- Accuracy on boolean expressions to actually increase by 5.6 pp.
As models scale, this effect becomes even more pronounced. In Qwen3-VL, the "direction-specificity" (measured as a z-score) of the spatial effect exploded from +27.7 at the 2B scale to a massive +262.5 at the 32B scale, indicating that larger models develop much more specialized and "pure" spatial representations.
Limitations and Future Work
- Linearity: The study focused on linear and low-rank interventions. It’s possible that more complex, non-linear "vision-shaped" mechanisms exist that this method missed.
- Task Taxonomy: While the researchers tested a variety of tasks, they acknowledge their battery is a "diagnostic" rather than a complete taxonomy of human-like spatial reasoning.
- The "How" vs. the "What": The study proves that these representations are necessary for spatial reasoning, but it doesn't fully map out the specific circuits that write and read these signals.
Future work could involve "circuit breaking" at a more granular level or investigating how different multimodal pre-training objectives (e.g., contrastive vs. generative) change the resulting internal geometry.
Summary updated