Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

Yige Xu, Yongjie Wang, Zizhuo Wu, Kaisong Song, Jun Lin, Zhiqi ShenarXiv 2026 · arXiv

TL;DR

State-of-the-art Vision-Language Models (VLMs) possess formidable textual reasoning capabilities but suffer from a severe "modality gap" when facing identical visual tasks. By introducing CROSSMATH, a benchmark with strict information equivalence across text and vision, this paper demonstrates that visual inputs often act as distractors rather than aids. However, the authors show that targeted post-training via Supervised Fine-Tuning (SFT) and Reinforcement Learning (GRPO) can significantly mitigate this deficit and drive robust out-of-domain generalization.

What Problem Does This Paper Address?

Modern VLMs seamlessly blend vision encoders with large language model (LLM) backbones to excel at multimodal tasks. However, it remains unclear whether their success stems from genuine visual reasoning or if they are simply relying on the powerful textual priors of their underlying LLMs.

Evaluating this is exceptionally difficult with current benchmarks. Many existing datasets either test surface-level visual recognition or feature "modality-entangled" problems (where a question is inherently unsolvable without both the text and the image combined). To truly isolate and measure a VLM's visual reasoning capacity, we need a "vision-first" benchmark where the exact same problem is presented in text-only, image-only, and hybrid formats, ensuring absolute information symmetry.

Methods

The CROSSMATH Benchmark

The authors designed a procedural generation pipeline to create CROSSMATH, a benchmark consisting of 2D grids of intersecting arithmetic equations (akin to a crossword puzzle but with math). To solve it, a model must understand spatial layouts, identify which cells belong to which equations, and perform multi-step logical deduction.

CROSSMATH puzzle formats: Markdown table, raw image, and solution

To eliminate confounding factors, every puzzle is formulated into three strictly equivalent formats:

  1. Image-only grid
  2. Text-only Markdown table
  3. Multimodal (Image + Text) prompt

Visual Style Augmentation

To prevent models from simply memorizing fixed templates, the authors introduced four distinct visual style perturbations: removing cell borders, adding complex background textures, and stochastically altering fonts and color palettes.

Visual style augmentations used to test robustness

Post-Training Pipeline

To bridge the discovered modality gap, the authors developed a two-stage post-training strategy on Qwen3.5-9B using only image-based inputs:

  • Supervised Fine-Tuning (SFT): The model is trained on high-quality, step-by-step Chain-of-Thought (CoT) trajectories generated by Qwen3-VL-Max to internalize structured thinking patterns.
  • Reinforcement Learning with Verifiable Rewards (RLVR): Using Group Relative Policy Optimization (GRPO), the authors implemented a position-weighted reward strategy. This implicitly penalizes shortcuts by assigning higher reward weights to target cells that require deeper logical reasoning depths (more "hops").

Datasets

  • CROSSMATH: The primary dataset introduced, containing 5,000 training samples and 250 core evaluation samples (which expand to 1,000 evaluation instances when accounting for the four visual style augmentations). All evaluation samples were manually verified by human annotators to guarantee cross-modal equivalence.
  • MMMU & MathVerse (Vision-Only subset): Used as external, out-of-domain (OOD) benchmarks to evaluate how well the fine-tuned visual reasoning models generalize to other mathematical and scientific visual tasks.

Key Insights and Ideas

  • The Modality Gap is Massive: SOTA models exhibit a staggering performance drop when moving from text to vision. For example, Qwen3.5-Plus achieves near-perfect text reasoning but fails dramatically when given the exact same data as an image.
  • Images Can Act as Distractors: Counterintuitively, presenting a VLM with both text and image inputs frequently degrades performance compared to giving it the text alone. Redundant visual data appears to interfere with the model's internal language-space logic.
  • Perception is Not the Bottleneck: The failure is not due to poor eyesight. The models achieve very low OCR error rates when converting the images to text. Instead, the bottleneck is a lack of structural inductive bias—the models struggle to map accurately perceived visual symbols into a coherent spatial reasoning chain.
  • Reasoning Depth Multiplies Failure: As the number of logical steps (hops) increases, visual accuracy plummets drastically compared to textual accuracy. Propagating logic across extended visual dependency chains causes the model's reasoning to collapse.
  • Model Scaling Doesn't Fix Vision Reasoning: While increasing LLM backbone size yields a clear, monotonic scaling trend in text-only reasoning, it does not consistently improve image-only performance. This implies that visual-to-text grounding capacity, rather than language model scale, is the primary bottleneck.

Key Results

  • Zero-Shot Disparity: Qwen3.5-Plus achieved a 92.80% Macro Accuracy in the text-only setting, which plummeted to just 12.40% in the image-only setting.
  • Post-Training Impact: Applying SFT + GRPO to the lightweight Qwen3.5-9B model yielded massive gains in the image-only domain, boosting its Micro Accuracy from 23.25% to 62.33% and its Macro Accuracy from 3.20% to 50.40%.
  • Robust OOD Transfer: Training on CROSSMATH provided a "rising tide" effect on external benchmarks, yielding an incremental boost of +2.46% on MathVerse and +1.39% on MMMU, proving that the model learned transferable visual-symbolic reasoning rather than simple pattern matching.

Limitations and Future Work

Despite the significant boost from SFT + GRPO, a clear modality gap remains: the post-trained model's image-only performance (50.40% Macro Accuracy) still lags far behind its text-only performance (76.40%).

The authors conclude that simply aligning visual embeddings with textual spaces via projection layers is fundamentally insufficient for complex spatial logic. Future progress will require architectural shifts toward more powerful visual foundation models that can natively internalize the structural and physical constraints of the visual world.

Summary updated

Abstract

Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.
BibTeX
@misc{xu2026do,
  author = {Yige Xu and Yongjie Wang and Zizhuo Wu and Kaisong Song and Jun Lin and Zhiqi Shen},
  title = {{Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap}},
  year = {2026},
  eprint = {2604.16256},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.16256},
}

Loading related content...

Figure