A growing way to make vision-language models reason better is to split the job in two. A vision-language model looks at the image and writes a description, and a separate text-only language model reasons over that description. This split works well. A 72B vision model paired with a 32B text reasoner reaches 52.6 on MATH-Vision (Decoupling Reasoning and Perception), where GPT-4o alone scores 30.39% (MATH-Vision). The description, however, is written in the vision model's own habits of phrasing and is then read by a different model with different habits. Separate work on text-only reasoning shows that such mismatches are costly. Small models trained on a large teacher's reasoning traces get worse, because the traces contain "surprisal spikes": tokens the student itself would assign under 1% probability. Filtering those tokens restores performance (In Their Own Words).
We propose to carry that finding over to the handoff between the "eyes" and the "brain." At decoding time, the vision model proposes each next word of its description. The text reasoner vetoes words it finds very unlikely, and the vision model then picks its next-best candidate that the reasoner accepts. What the description says stays under the control of the model that sees the image. How it is said moves toward the reader's own distribution. The central claim is that part of the remaining error in decoupled systems is a prompt-sensitivity failure at the interface, not a perception or reasoning failure. That part should be recoverable without training, and only by tailoring the interface to the specific reasoner.
Where this sits
The proposal draws on two literatures. The first, on model-specific prompt sensitivity, motivates it. TP-Eval optimizes prompts per multimodal model and finds that prompts tuned for Mini-InternVL hurt LLaVA and DeepSeek-VL. PromISe finds almost no overlap among the best prompts across model families. UniAPO contradicts both: its GPT-4o-optimized prompts transfer to Qwen2.5-VL-72B, but it tested only classification and keyword tasks. Work on retrieval makes the same point about generated context rather than instructions. Bridging the Preference Gap between Retrievers and LLMs shows that the way upstream context is arranged has to be fitted to the particular generator. All of these tune the prompt or the context as a whole and judge it by accuracy on a small labeled set. We act token by token on the machine-written context itself and use no labels.
The second literature, on the multimodal reasoning gap, supplies the setting. Jia et al. separate a text Reasoner from a vision Observer. They report that scaling the Reasoner helps more than scaling the Observer, and they name the prompt sensitivity of the Reasoner–Observer interface as unaddressed. Their interface uses fixed hand-written templates. RAPID instead trains the perceiver with reinforcement learning to write captions aimed at a frozen reasoner. It reports 53.2% average accuracy with Qwen2.5-VL-7B plus Qwen3-8B.
RAPID is the most direct competitor. It adapts the interface through weight updates and claims its captions generalize across reasoners. Our method adapts per reasoner at decoding time. If RAPID's cross-reasoner claim fully holds, our specificity hypothesis is wrong, so we test against it directly. The MATH-Vision error taxonomy constrains what we can expect. Of GPT-4V's errors, 31.9% are vision errors and 42.2% are reasoning errors. A method that only rephrases the interface should reduce reasoning errors, not vision errors. Position: Reasoning After Perception Means Reasoning Without Vision argues that converting images to text discards spatial information that no text-level fix can restore. We take that as a boundary: our method should do nothing on tasks where perception itself fails.
Core Claim and Mechanism
Let the perceiver $P$ generate a description $c$ of image $x$ given a query $q$, and let the reasoner $R$ answer from $(q, c)$. At step $t$, $P$ proposes its top-$k$ candidate tokens. We accept the highest-ranked candidate $w$ that satisfies
$$p_R(w \mid q, c_{<t}) \ge \tau,$$
where $p_R$ is the reasoner's next-token probability and $\tau$ is a threshold, defaulting to 1% as in In Their Own Words. If no candidate passes, we keep $P$'s top choice. This keeps the perceiver's content decisions intact whenever the reasoner has no fluent alternative.
The token-level version needs a shared vocabulary. The Qwen2.5-VL perceivers and Qwen3 reasoners used in the prior decoupled work are the natural first pairs. For pairs with different vocabularies, a sequence-level variant samples $N$ full descriptions from $P$ and keeps the one with the fewest reasoner surprisal spikes. It must be compared to majority voting over the same $N$ descriptions.
The claim makes three falsifiable predictions. First, the number of surprisal spikes a reasoner meets in a description predicts that reasoner's errors, even after controlling for whether the description is factually correct. Second, filtering against the reasoner that will actually read the description beats filtering against a different reasoner. This is the model-specificity prediction, and it bears on the TP-Eval/UniAPO disagreement for multi-step reasoning. Third, gains concentrate in reasoning-type errors and vanish on perception-bound tasks.
Research Program
We first build the diagnostic. We run fixed-template decoupled pipelines, following Jia et al., on MATH-Vision, MathVerse and MathVista. For each description we log the reasoner's per-token surprisal. A strong vision-language judge labels each description's factual fidelity to the image, and we audit a stratified subsample by hand.
We then fit the relationship between spike count and reasoner correctness within each fidelity stratum. If the relationship disappears once fidelity is controlled, the premise fails and we stop. If it holds, we implement filtered decoding and the sequence-level variant. We then run a full transfer matrix. Descriptions filtered for reasoner A are fed to reasoner B, across Qwen3-8B and Qwen3-32B reasoners and 7B and 72B perceivers. This measures how much of the benefit is specific to the reasoner.
Baselines
The naive reference point is the single vision-language model answering directly with standard chain-of-thought. MATH-Vision reports this sometimes lowers accuracy relative to direct answering, so we report both. It is run on the same test splits with the same answer extraction.
The primary structural baseline is the fixed-template decoupled pipeline of Jia et al., which reports 52.6 on MATH-Vision and 54.4 on MathVerse with Qwen2.5-VL-72B plus Qwen3-32B. We reproduce it with identical models, templates and decoding temperature. Our method changes only the decoding of descriptions. Beating it means a statistically reliable gain on both benchmarks over our reproduction, not merely over the published number.
The strongest training-based competitor is RAPID, at 53.2% average over its seven benchmarks with Qwen2.5-VL-7B plus Qwen3-8B. The file gives no per-benchmark breakdown, so we compare against our own evaluation on the shared benchmarks. If RAPID's trained perceiver is available, we test two things. The first is whether filtering adds gains on top of RAPID when paired with the reasoner RAPID was trained for. The second, the more important case, is the same test when paired with a reasoner it was not trained for. If the checkpoint is unavailable, we reproduce reinforcement-learning training of the perceiver only at 7B scale on ViRL39K and state that limitation.
The strongest prompt-optimization competitor is accuracy-driven optimization of the perceiver's caption prompt per reasoner, in the style of TP-Eval. It gets the same number of reasoner forward passes our filtering consumes, and its labeled tuning set is drawn from the training portion of the benchmarks.
For the sequence-level variant, the compute-matched baseline is best-of-$N$ descriptions with majority voting over answers. No reported result was found on Lacuna for majority-voted decoupled pipelines on these benchmarks.
Two ablations isolate the key ingredient. The first replaces tokens at the same rate but at random positions, which tests whether any perturbation helps. The second filters against a mismatched reasoner's distribution, which tests whether the reader's own distribution matters.
Why not just…?
Why not just optimize the perceiver's prompt for accuracy, as TP-Eval does? This is a real alternative and it is in the baselines. The concern is that TP-Eval tunes on about 20 examples per task, a known overfitting risk. It also changes the whole instruction rather than the word choices that trip up a particular reader. If accuracy-tuned prompts match our filtering at equal reasoner calls, the interface problem is solved more simply and we will report that. Our label-free method would then matter only where labeled tuning data is absent.
Why not just train the perceiver with RAPID? RAPID already fits captions to a reasoner, and its claimed generalization across reasoners directly threatens our specificity prediction. The transfer arm settles it. If RAPID's perceiver loses nothing when paired with an unseen reasoner, and filtering adds nothing on top, the mismatch we target does not exist after training. Filtering would then only be a cheaper substitute requiring no training.
Why not just use a bigger reasoner, or let the reasoner paraphrase the description? Scaling the reasoner helps (Jia et al.), but it is orthogonal. We test at two reasoner scales and ask whether the gains shrink with size. Letting the reasoner paraphrase puts the reasoner, which never sees the image, in charge of content, so it can smooth away visual facts. We include paraphrasing as a baseline and compare fidelity scores to show whether this happens.
Why not just sample several descriptions and majority-vote? Voting may recover the same errors by averaging rather than by fitting to the reader. Matching compute against it is the decisive check for the sequence-level variant.
Decisive Checks
The idea stands or falls on four results. First, spike counts must predict reasoner errors within fidelity strata. Second, filtering against the matched reasoner must beat filtering against a mismatched one in the transfer matrix. Third, gains must appear mainly in MATH-Vision's reasoning-error category, not its vision-error category. Fourth, on the Turing Eye Test, where frontier models score near zero because perception fails, filtering should yield no gain; a gain there would signal an artifact. A positive result would show that part of the multimodal reasoning gap is really a reader-writer language mismatch that can be fixed at inference time. A negative result on the specificity test would suggest that prompt transfer holds up better for machine-written context than TP-Eval found for instructions.
References
- Decoupling Reasoning and Perception: An LLM-LMM Framework for Faithful Visual Reasoning
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners
- TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Prompts
- PromISe: Releasing the Capabilities of LLMs with Prompt Introspective Search
- UniAPO: Unified Multimodal Automated Prompt Optimization
- Bridging the Preference Gap between Retrievers and LLMs
- Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
- Position: Reasoning After Perception Means Reasoning Without Vision