Multimodal Large Language Models (MLLMs) are impressive, but they often suffer from "myopia"—they struggle to see tiny details like distant street signs or small text. While one solution is to let the model "zoom in" at inference time, this is computationally expensive. A more elegant approach is On-Policy Self-Distillation (OPSD), where a model learns to internalize these fine-grained details during training by learning from a "teacher" version of itself that has access to zoomed-in crops.
However, a new paper titled "Teach with Blinks" identifies a critical flaw in this process: linguistic momentum.
TL;DR
BlinkOPD is a highly data-efficient distillation method that prevents multimodal models from being "distracted" by their own incorrect text prefixes during training. By having the teacher model compare its view with its "eyes closed," it provides a much sharper corrective signal, allowing a 4B model to outperform state-of-the-art baselines using 12x less training data.
What Problem Does This Paper Address?
The authors highlight a phenomenon called linguistic momentum. In standard OPSD, the teacher model provides supervision by predicting the next token based on the student's generated prefix. If the student starts off with a wrong answer, the teacher—despite having a better view of the image—often gets "dragged" into agreeing with the student's mistake.
As shown in the figure below, as the teacher reads more of a wrong prefix, its "Follow Rate" (endorsing the mistake) sky-rockets, and the "Survival Score" of the actual visual evidence plummets.
Existing fixes try to use "contrastive" references (like comparing the teacher's view to a black or gray image), but the authors show these methods either fail to fully cancel the linguistic drift or introduce "false beliefs"—essentially tricking the model into thinking the object it's looking for isn't there at all.
Methods
The authors propose Blink On-Policy Distillation (BlinkOPD), which introduces two main innovations:
- The "Eyes-Closed" Reference: To isolate the visual signal, the teacher processes the student's prefix twice. Once with the zoomed crop ("eyes open") and once with the image-content tokens removed entirely ("eyes closed"). Crucially, it keeps the vision markers and original positional indices. This allows the model to calculate a "Privileged Visual Evidence" score ($r_t$) that cancels out the linguistic bias shared by both passes.
- First-Glance Gain: Before seeing any text prefix, the teacher takes a "first glance" at the zoomed crop to determine the likely correct answer. During training, if the student's rollout conflicts with this first glance, the distillation gain is boosted. This ensures the model tries harder to correct itself when it's most likely to be wrong.
Datasets
- Training: The authors used a decontaminated version of the Vision-OPD training pool, consisting of approximately 6.2K images. Remarkably, their best results often required only a random subset of 500 images.
- Evaluation: The model was tested across 16 benchmarks, including fine-grained perception (e.g., V, ZoomBench, HR-Bench), general perception (MMVP, CV-Bench), hallucination mitigation (POPE, HallusionBench), and multimodal reasoning (MathVista, DynaMath*).
Key Insights and Ideas
- Linguistic bias is stronger than visual evidence: In vanilla distillation, the text prefix acts like a magnet, pulling the teacher away from the truth.
- Structural integrity matters: Simply replacing an image with a black box (Figure 4a) doesn't cancel linguistic drift as effectively as removing the tokens while keeping the positional "frame" (Figure 4c).
- Quality over Quantity: By providing "denser" corrective supervision, BlinkOPD makes every training sample count. This explains why it can beat models trained on 12x more data.
Key Results
- Superior Performance: Using Qwen3.5-4B as a base, BlinkOPD achieved an average accuracy of 79.1 on fine-grained benchmarks with just 500 images. This outperformed all reproduced OPSD baselines trained on 6.2K images.
- Scaling: When scaled to 9B parameters and the full 6.2K dataset, it reached an average of 80.4, surpassing massive proprietary models like Gemini 3.1 Pro (79.3) and open-weight giants like Qwen3.5-397B (77.4).
- Zero-Shot Generalization: Despite being trained only on fine-grained perception data, the model showed significant improvements in multimodal reasoning (+8.4 points at 4B) and hallucination mitigation.
Limitations and Future Work
The authors acknowledge that while BlinkOPD significantly reduces linguistic momentum, it doesn't eliminate it entirely; the teacher still occasionally follows a wrong prefix. Additionally, the current study is limited to the Qwen3.5 model family. Future work could involve validating the "blink" mechanism on other architectures and finding even more robust ways to completely decouple linguistic bias from visual reasoning.
Summary updated