TL;DR
State-of-the-art Vision-Language Models (VLMs) possess formidable textual reasoning capabilities but suffer from a severe "modality gap" when facing identical visual tasks. By introducing CROSSMATH, a benchmark with strict information equivalence across text and vision, this paper demonstrates that visual inputs often act as distractors rather than aids. However, the authors show that targeted post-training via Supervised Fine-Tuning (SFT) and Reinforcement Learning (GRPO) can significantly mitigate this deficit and drive robust out-of-domain generalization.
What Problem Does This Paper Address?
Modern VLMs seamlessly blend vision encoders with large language model (LLM) backbones to excel at multimodal tasks. However, it remains unclear whether their success stems from genuine visual reasoning or if they are simply relying on the powerful textual priors of their underlying LLMs.
Evaluating this is exceptionally difficult with current benchmarks. Many existing datasets either test surface-level visual recognition or feature "modality-entangled" problems (where a question is inherently unsolvable without both the text and the image combined). To truly isolate and measure a VLM's visual reasoning capacity, we need a "vision-first" benchmark where the exact same problem is presented in text-only, image-only, and hybrid formats, ensuring absolute information symmetry.
Methods
The CROSSMATH Benchmark
The authors designed a procedural generation pipeline to create CROSSMATH, a benchmark consisting of 2D grids of intersecting arithmetic equations (akin to a crossword puzzle but with math). To solve it, a model must understand spatial layouts, identify which cells belong to which equations, and perform multi-step logical deduction.
To eliminate confounding factors, every puzzle is formulated into three strictly equivalent formats:
- Image-only grid
- Text-only Markdown table
- Multimodal (Image + Text) prompt
Visual Style Augmentation
To prevent models from simply memorizing fixed templates, the authors introduced four distinct visual style perturbations: removing cell borders, adding complex background textures, and stochastically altering fonts and color palettes.
Post-Training Pipeline
To bridge the discovered modality gap, the authors developed a two-stage post-training strategy on Qwen3.5-9B using only image-based inputs:
- Supervised Fine-Tuning (SFT): The model is trained on high-quality, step-by-step Chain-of-Thought (CoT) trajectories generated by Qwen3-VL-Max to internalize structured thinking patterns.
- Reinforcement Learning with Verifiable Rewards (RLVR): Using Group Relative Policy Optimization (GRPO), the authors implemented a position-weighted reward strategy. This implicitly penalizes shortcuts by assigning higher reward weights to target cells that require deeper logical reasoning depths (more "hops").
Datasets
- CROSSMATH: The primary dataset introduced, containing 5,000 training samples and 250 core evaluation samples (which expand to 1,000 evaluation instances when accounting for the four visual style augmentations). All evaluation samples were manually verified by human annotators to guarantee cross-modal equivalence.
- MMMU & MathVerse (Vision-Only subset): Used as external, out-of-domain (OOD) benchmarks to evaluate how well the fine-tuned visual reasoning models generalize to other mathematical and scientific visual tasks.
Key Insights and Ideas
- The Modality Gap is Massive: SOTA models exhibit a staggering performance drop when moving from text to vision. For example, Qwen3.5-Plus achieves near-perfect text reasoning but fails dramatically when given the exact same data as an image.
- Images Can Act as Distractors: Counterintuitively, presenting a VLM with both text and image inputs frequently degrades performance compared to giving it the text alone. Redundant visual data appears to interfere with the model's internal language-space logic.
- Perception is Not the Bottleneck: The failure is not due to poor eyesight. The models achieve very low OCR error rates when converting the images to text. Instead, the bottleneck is a lack of structural inductive bias—the models struggle to map accurately perceived visual symbols into a coherent spatial reasoning chain.
- Reasoning Depth Multiplies Failure: As the number of logical steps (hops) increases, visual accuracy plummets drastically compared to textual accuracy. Propagating logic across extended visual dependency chains causes the model's reasoning to collapse.
- Model Scaling Doesn't Fix Vision Reasoning: While increasing LLM backbone size yields a clear, monotonic scaling trend in text-only reasoning, it does not consistently improve image-only performance. This implies that visual-to-text grounding capacity, rather than language model scale, is the primary bottleneck.
Key Results
- Zero-Shot Disparity: Qwen3.5-Plus achieved a 92.80% Macro Accuracy in the text-only setting, which plummeted to just 12.40% in the image-only setting.
- Post-Training Impact: Applying SFT + GRPO to the lightweight Qwen3.5-9B model yielded massive gains in the image-only domain, boosting its Micro Accuracy from 23.25% to 62.33% and its Macro Accuracy from 3.20% to 50.40%.
- Robust OOD Transfer: Training on CROSSMATH provided a "rising tide" effect on external benchmarks, yielding an incremental boost of +2.46% on MathVerse and +1.39% on MMMU, proving that the model learned transferable visual-symbolic reasoning rather than simple pattern matching.
Limitations and Future Work
Despite the significant boost from SFT + GRPO, a clear modality gap remains: the post-trained model's image-only performance (50.40% Macro Accuracy) still lags far behind its text-only performance (76.40%).
The authors conclude that simply aligning visual embeddings with textual spaces via projection layers is fundamentally insufficient for complex spatial logic. Future progress will require architectural shifts toward more powerful visual foundation models that can natively internalize the structural and physical constraints of the visual world.
Summary updated