When a vision-language model gets a diagram question wrong, there are two possible causes. It may not have seen what it needed, because the image was shrunk to fit the vision encoder and the key label, tick mark or small object turned into a few blurry pixels. Or it saw everything and still reasoned badly. Current benchmarks report a single accuracy number that mixes these causes. As a result, the field cannot answer a basic question about its most popular training recipes: when reasoning skills learned on text-only math are transferred into a multimodal model, does that transfer improve reasoning, perception, or both?
We propose to measure the two causes separately by changing something the experimenter controls, namely image resolution. Every benchmark item is shown to every model at several downsampling levels. We then fit an item response model. Item response theory (IRT) is the psychometric method that explains each right-or-wrong answer as an interaction between a model's hidden ability and an item's hidden difficulty. Our version adds a second item property, a legibility threshold: the resolution below which the item's needed visual information becomes unrecoverable. Each model receives two abilities, a reasoning ability and a perceptual acuity. Because resolution varies inside each item while the reasoning content stays fixed, the two factors become statistically separable. A model that does not see resolution cannot separate them. We then use this measurement to test a sharp prediction about cross-domain transfer.
Core Claim
Our hypothesis is a double dissociation. Text-to-vision reasoning transfer of the kind performed by Open Vision Reasoner raises reasoning ability and leaves perceptual acuity unchanged. Active-perception training of the kind in ACTIVE-o3 and Reason with Thumbnails, Answer with Focus raises perceptual acuity and leaves reasoning roughly unchanged. Each claim already has informal support. Open Vision Reasoner (OVR) reports that its text cold-start plus multimodal reinforcement learning scales deduction but fails to scale counting and fine OCR. ACTIVE-o3 reports the reverse direction, with perception-only training lifting RealWorldQA from 67.9 to 69.7. Neither paper can say how much of its gain falls on which axis, because neither varies the visual information available to the model.
The contribution has two parts. The first is a measurement model that makes this question answerable. The second is a controlled experiment that answers it. If the dissociation holds, it tells practitioners which item-level failures each kind of training will fix. If it fails, for example because perception training cross-loads onto reasoning ability, that result is just as informative. It would mean "perception" and "reasoning" are not separable skills in these models.
The Measurement Model
The closest prior work is M3IRT. It splits model ability and item difficulty into text-only, image-only and cross-modal parts across 24 vision-language models, and it finds that MathVista scores mostly reflect text ability. M3IRT treats the image as a fixed channel that is either present or absent. Its cross-modal difficulty therefore cannot distinguish "the needed detail is illegible at encoder resolution" from "the detail is legible but the inference is hard." That is the gap we fill. We reuse M3IRT's estimation strategy, stochastic gradient descent on a sparse response matrix, which is also the approach of Scalable Learning of Item Response Theory Models.
We use a conjunctive form, in which an answer is correct only if the model both perceives the item and reasons through it:
$$P(X_{jir}=1)=\gamma_i+(1-\gamma_i)\,\sigma\big(\alpha_i(\theta^{R}_j-\beta_i)\big)\,\sigma\big(\kappa_i(\theta^{P}_j+\log r-\lambda_i)\big)$$
The symbols are defined as follows:
- $X_{jir}$ is model $j$'s binary outcome on item $i$ shown at resolution fraction $r$.
- $\gamma_i$ is the guessing floor for multiple-choice items.
- $\theta^{R}_j$ is the model's reasoning ability.
- $\beta_i$ and $\alpha_i$ are the item's reasoning difficulty and discrimination.
- $\theta^{P}_j$ is perceptual acuity, which acts as a shift in effective log-resolution.
- $\lambda_i$ is the item's legibility threshold in log-resolution units.
- $\kappa_i$ is how sharply legibility switches on.
The $\log r$ term is fixed by the experimental design and is not learned. That fixed term is what makes $\lambda_i$ identifiable apart from $\beta_i$. Variation across resolutions within one item can only move the perceptual factor.
Two predictions follow directly from the form. As $r\to$ full resolution, the right factor saturates for most items, and the model reduces to ordinary 3PL IRT. Items whose $\lambda_i$ sits far below the lowest tested resolution are "text-shortcut" items. They are the same kind of item MathVerse and M3IRT flag, and the fit should recover them without being told.
The model scores binary outcomes, so we restrict the study to items with short verifiable answers. MathVerse and MATH-Vision both provide such answers. Free-form chain-of-thought grading remains out of scope, as it is for M3IRT.
Research Program
The first stage builds the response tensor. We evaluate about two dozen open-weight vision-language models at five resolution levels, spanning the 1/64 to 1/4 thumbnail range used by Reason with Thumbnails, Answer with Focus (RTAF), plus full native resolution. The models include:
- Qwen2.5-VL-7B and its derivatives OVR-7B and the Game-RL checkpoint from Game-RL;
- the active-perception models ACTIVE-o3 and RTAF;
- the patch-tiling models Monkey and UReader.
The items come from MathVerse (including its text-dominant and vision-only variants), MATH-Vision, MathVista, the Chart101 elementary chart probes from Plot Twist, DocVQA and TextVQA, and HR-Bench-4K.
For active-perception models, $r$ is the resolution of the image file we hand them. This means their learned cropping shows up as higher $\theta^{P}$, which is the intended reading. We do not need the full tensor. Following the evidence that a few percent of items can preserve rankings (Polo et al.), we sample cells sparsely and hold out cells for validation.
The second stage is a controlled transfer experiment. The public checkpoints differ in data, compute and recipe, so they can only provide correlational evidence. We therefore train three arms from one Qwen2.5-VL-7B base under matched GRPO compute:
- text-only verifiable-reward math training, following OVR's cold-start idea;
- crop-proposal perception training, following ACTIVE-o3;
- synthetic game training, following Game-RL.
We estimate each arm's $(\theta^{R},\theta^{P})$ against the frozen item parameters from stage one, with bootstrap intervals over items.
The third stage tests whether the item parameters are useful and not merely descriptive. Item parameters are fitted on models trained before the intervention, and they are used to predict which specific items each arm will newly solve. If the decomposition is meaningful, high-$\lambda$ items should be fixed by perception training and high-$\beta$ items by reasoning training.
We include a small follow-on as a secondary test. It routes multimodal RL training data by fitted parameters, sending perception-bound items to the perception arm and reasoning-bound items to the reasoning arm. We compare this against RTAF's heuristic "resolution sensitive" bins.
Baselines
The naive reference point is aggregate accuracy plus a per-item accuracy-versus-resolution table. Here an item's "resolution sensitivity" is simply the drop in mean accuracy between full and lowest resolution, and a model's gain from an intervention is its accuracy change on sensitive versus insensitive items. This uses exactly the same response tensor as our method, so it costs nothing extra. Our model beats it only if two things hold. It must predict held-out (model, item, resolution) outcomes with better log-likelihood and AUC. And its item parameters must predict, better than the raw accuracy drop, which items each trained arm newly solves.
The strongest existing method is M3IRT (Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory). We fit it on the same models and items, using full-resolution and text-removed conditions as its modality channels. It gets the same optimizer and tuning budget, and the same held-out cells. M3IRT reports reconstructing leaderboards from 10% of items at 13% of inference cost. We found no reported held-out response-prediction result on Lacuna for M3IRT, so the comparison number will be produced here. To beat it, our model must predict held-out low-resolution cells better. It must also give a reasoning ability whose ranking of the three training arms differs from M3IRT's cross-modal ability in the direction the MathVerse validation below supports.
The ablation that isolates the key ingredient is an otherwise identical model with the $\log r$ covariate removed. In this ablation, resolution levels are treated as separate unlinked item copies with their own difficulties. This tests whether placing resolution on a shared, known axis is what makes perception and reasoning separable. A second ablation replaces the conjunctive product with an additive two-dimensional logistic. The conjunctive product is a substantive claim that seeing and reasoning are both necessary, so it should be tested against the additive alternative and not assumed.
For the training arms, the reference numbers are the published ones. OVR-7B reports 51.8% on MATH-Vision. Game-RL reports a +2.65% average gain across seven benchmarks. ACTIVE-o3 reports 69.7 versus 67.9 on RealWorldQA. RTAF reports +5.8% on VisCoT. Our matched re-implementations should land near these figures before their ability shifts are interpreted.
Why not just…?
Why not just plot accuracy against resolution for each benchmark? This is the real threat, because it uses the same data and is far simpler. Its weakness is that benchmark-level curves average over items with very different legibility thresholds and reasoning difficulties. A flat curve can mean "nothing needs detail" or "the model fails the reasoning on everything that needs detail." Per-item curves are noisy, with one binary outcome per model per level, and they are not comparable across models of different strength. Still, we do not argue this point; we test it. The naive baseline above is precisely this method. If its sensitivity scores predict intervention-specific item gains as well as $\lambda_i$ and $\beta_i$ do, we will report that the IRT layer adds nothing for this purpose.
Why not just remove the image or transcribe it into text, as GeoSQA and MathVerse do? Transcription removes perception from the problem. GeoSQA shows that models stay near chance (26.22% against a 25% floor) even with human-written diagram descriptions. MathVerse shows some models gain 5% or more when the diagram is dropped. Transcription, however, needs human annotation per item and exists only for a few benchmarks. It also changes the task's wording along with its perceptual demand. Resolution sweeps apply to any image benchmark automatically. We treat transcription as ground truth for validation, not as a competitor. MathVerse's text-dominant variants put the essential conditions into the text, so our fit should assign them much lower $\lambda$ than their vision-only twins while assigning similar $\beta$. If it does not, the decomposition is wrong.
Why not just raise resolution or tile the image, and skip the diagnosis? Tiling clearly fixes some failures. UReader's cropping lifts DocVQA from 22.0 to 65.4, and Monkey moves DeepForm from 4.1 to 40.6. But it floods the context window, and RTAF matches or beats denser approaches with roughly half the tokens. More importantly, MATH-Vision attributes 42.2% of GPT-4V's errors to reasoning after correct recognition. More pixels cannot fix that fraction. Our model estimates how large that fraction is per benchmark. It predicts that Monkey and UReader gain on $\theta^{P}$ only, which is a checkable consequence.
Decisive Experiments
Three results would settle the proposal.
The first is predictive. On held-out cells, the resolution-indexed conjunctive model must beat the naive table, M3IRT, and both ablations in log-likelihood. The largest margin should appear at the lowest resolutions, where those models have no mechanism to extrapolate.
The second is external validity. The text-dominant versus vision-only check on MathVerse must show a $\lambda$ drop with stable $\beta$. Chart101's elementary coordinate questions, which Plot Twist shows models fail despite solving harder ChartQA items, should land at high $\lambda$ and low $\beta$.
The third is the dissociation in the matched-compute arms, judged by bootstrap intervals on $(\theta^{R},\theta^{P})$ shifts against the base model. The text-reasoning arm should move $\theta^{R}$ with a $\theta^{P}$ interval covering zero. The perception arm should move $\theta^{P}$. Where the game arm falls is an open question we expect to learn from.
If perception training also moves $\theta^{R}$, as ACTIVE-o3's reasoning gains hint it might, the model should show this as a joint shift and not hide it. We would then report that the two skills are coupled in current architectures. That is a finding about the models, not a failure of the measurement.
References
- Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning
- ACTIVE-o3 : Empowering MLLMs with Active Perception via Pure Reinforcement Learning
- Reason with Thumbnails, Answer with Focus: An Efficient and Effective Paradigm for Multimodal Grounded Visual Reasoning
- Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
- Scalable Learning of Item Response Theory Models
- Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning
- Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
- UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- tinyBenchmarks: evaluating LLMs with fewer examples
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?