TL;DR
VisAlloc is a new framework that improves visual reasoning in Multimodal Large Language Models (MLLMs) by intelligently allocating supervision during training. Instead of treating all data equally, it filters for samples where visual detail is critical and reweights response tokens based on their sensitivity to visual evidence, leading to state-of-the-art performance on fine-grained reasoning benchmarks.
What Problem Does This Paper Address?
Multimodal Large Language Models (MLLMs) often suffer from "lazy reasoning"—they might answer a question based on linguistic priors (guessing based on the text) rather than actually looking at the image. To fix this, researchers use On-Policy Self-Distillation (OPSD). In this setup, a "teacher" model gets "privileged" information (like a high-resolution zoom of a specific object) and guides a "student" model that only sees the full, lower-resolution image.
However, existing methods like Vision-OPD have two major flaws:
- Sample Inefficiency: They apply the same amount of supervision to every training example, even if the question is so easy that the model doesn't need the privileged zoom to answer it.
- Token Dilution: They supervise every token in a response equally. This "uniform" approach dilutes the signal for tokens that actually require visual grounding (like "blue ribbon") with generic linguistic filler (like "The shows a").
VisAlloc addresses these by asking: How should we allocate visual supervision across training samples and response tokens to maximize learning?
Methods
VisAlloc introduces a dual-level optimization framework that moves away from uniform supervision toward a "counterfactual" approach—comparing what the model sees versus what it could see.
1. Sample-Level Visual Filtering
Before training, VisAlloc runs a filter. It creates two views for every image: a positive crop (centered on the relevant evidence) and a negative crop (spatially displaced away from the evidence). It only keeps training samples where a teacher model answers correctly with the positive crop but incorrectly with the negative crop. This ensures the training set is concentrated on examples where the specific visual detail is the deciding factor.
2. Token-Level Counterfactual Reweighting
During training, VisAlloc uses two teachers (EMA copies of the model) to score the student's generated response.
- View Sensitivity: It calculates the difference in log-probabilities between the teacher seeing the positive crop and the teacher seeing the negative crop. Tokens with a high difference are "view-sensitive"—they clearly depend on the visual evidence.
- Gating: To avoid over-supervising common phrases (templates), it uses gates that filter out frequent bigrams and tokens where the student is already very close to the teacher.
- Inverse-Frequency Redistribution: To prevent frequent but less informative tokens from hogging the supervision "budget," VisAlloc redistributes weight toward rarer, more informative tokens within each batch.
Datasets
The authors trained their models using 6,241 paired visual-evidence examples derived from the Vision-OPD dataset, which were then filtered down to 5,795 high-quality samples.
Evaluation was performed across a comprehensive suite of eight benchmarks:
- Fine-grained tasks: V*Bench, ZoomBench, HR-Bench 4K, and HR-Bench 8K.
- Holdout/General tasks: MMVP, CV-Bench, MMStar, and POPE.
- Additional: MME-RealWorld (EN/CN) for broader context.
Key Insights and Ideas
- Contrast is More Informative Than Zoom: Simply showing a model a zoomed-in image (Vision-OPD) tells it what is there. Showing it both a zoomed-in image and a "wrong" crop (VisAlloc) tells it why that specific information matters.
- The "Budget" Perspective: Instead of just adding more supervision, VisAlloc treats supervision as a finite budget. By redistributing this budget away from "easy" samples and "generic" tokens, the model learns more efficiently.
- Geometry-Matched Crops: To ensure the difference between teachers is purely about the content of the crop and not the shape or resolution, VisAlloc ensures both positive and negative crops share the same rendering geometry.
Key Results
VisAlloc consistently outperforms previous on-policy distillation methods across different model scales (4B and 9B parameters).
- 4B Model: Achieved a macro-average of 81.44% across eight benchmarks, outperforming the Vision-OPD reproduction by 0.97 points.
- 9B Model: Achieved a macro-average of 84.92%, a 0.52 point improvement over the baseline.
- Radar Analysis: As shown in the radar chart below, VisAlloc leads on 8 out of 10 evaluated axes compared to Vision-OPD, showing particularly strong gains in fine-grained benchmarks like V*Bench and ZoomBench.
Limitations and Future Work
- Teacher Dependency: The sample filter relies on the teacher's correctness. If the teacher model itself has flaws or biases, those will be inherited by the filter.
- Negative Crop Ambiguity: In some complex images, a "displaced" negative crop might still accidentally contain relevant evidence or make the question ambiguous, potentially sending a confusing signal.
- Backbone Specificity: The current results are focused on the Qwen3.5 family. While the logic is general, the specific hyperparameters (like the "bonus strength" $\lambda$) might need tuning for other architectures.
- Future Direction: The authors suggest exploring how this "allocation" logic could be applied to other modalities, such as temporal grounding in video.
Summary updated