Where Misleading Context Wins: Spending Test-Time Compute at the Commitment Window of Diffusion Multimodal Reasoners

A diffusion language model writes an answer by starting from a fully masked sequence and filling in tokens over many rounds. At every round it predicts the whole output, then keeps only its most confident guesses. Image diffusion models make their big, irreversible decisions, such as layout and object category, in the first small fraction of steps. Later steps only polish detail. This proposal tests whether the same holds when a multimodal diffusion reasoner reads an image alongside a misleading text snippet. Our hypothesis is that the contest between "what the picture shows" and "what the distracting text claims" is settled in a short early window of denoising steps. If so, extra inference compute, meaning more samples and stronger visual guidance, should be spent inside that window and almost nowhere else.

This matters because the two best-documented failures of reasoning-at-inference point the wrong way for naive compute scaling. With misleading snippets, a text reasoning model with thinking turned on falls from 96.22% to 77.03% accuracy on SciAux, and does barely better than with thinking off (74.87%) (Thinking in a Crowd). Separately, longer multimodal reasoning pulls attention away from image tokens (More Thinking, Less Seeing?). More compute, spent uniformly, makes things worse. We propose to measure when context-driven commitment happens, detect that moment cheaply at test time, and concentrate compute there. The result should be a method that is more robust to noisy context at equal or lower cost.

Core Claim

In diffusion multimodal language models such as LaViDa and MMaDa, the answer tokens' dependence on misleading auxiliary text becomes effectively irreversible after a commitment step $t^*$. That step arrives early, and it can be estimated per example from the model's own full-sequence predictions. Concentrating branching and image-anchoring guidance before $t^*$, and running the rest of the trajectory as a single cheap path, will beat uniformly allocated test-time compute at matched cost on context-corrupted multimodal questions. It should also do this without hurting accuracy when the context is helpful.

The claim has two halves. The first is a measurable fact about dynamics, the location and sharpness of a point of no return. The second is an allocation policy that only makes sense if that fact holds. The first half can fail independently, and we would report that.

Relation to Prior Work

The closest work is Thinking Diffusion. It shows that diffusion multimodal models commit to answer tokens prematurely and lean on the image only late in generation. It fixes this with a step penalty and a guidance term that amplifies visual conditioning, applied at every step, and reports up to 7.5% higher accuracy on M³COT, ScienceQA, MMBench and V* Bench. That paper motivates our hypothesis and supplies our guidance operator. It never introduces misleading context, though, and its guidance runs at a fixed strength at every step. That doubles the forward passes throughout and, the authors note, overrides the text instruction when the guidance is set too strong. We differ in two ways. The adversary is contextual text that contradicts the image. The guidance and branching are confined to an estimated commitment window rather than applied uniformly.

Two papers motivate the phase logic. Early Semantic Commitment in Diffusion Sampling detects category flips with paired-seed sensitivity analysis on the predicted clean image, far earlier than raw state divergence. It leaves open whether one can intervene after onset, and it works only in continuous pixel space. An Efficient Test-Time Scaling Approach for Image Generation shows that search methods "dump" compute into late texture steps. Gating step advancement on verifier improvement lets it match its baseline at a quarter of the compute. We carry its principle, that compute belongs where decisions are still open, into discrete multimodal reasoning. We also replace its hand-set threshold with a per-example commitment estimate.

On the reasoning side, DiffCoT treats chain-of-thought as denoising but assigns lower noise to early reasoning steps. That is a design choice our measurements could contradict, if early premises turn out to be the fragile ones. RFG applies static-strength likelihood-ratio guidance in diffusion language models. Our scheduled guidance can be read as a time-varying generalization of it. PEAR shows that reasoning entropy is concentrated in the thinking phase and that penalizing it saves tokens on clean math. Whether entropy reduction helps or hurts under noisy context is untested, and our autoregressive transfer experiment speaks to it.

Two papers constrain the design. Plot Twist shows that chart answers track titles: accuracy reaches 79% when the title contains the answer and falls to about 30% when the title is removed or misleading. Misleading context therefore lives inside the image as well as beside it, and any method that simply discards the text channel will break. DCoT finds that adding image features to retrieval hurt it. So we do not assume the image is always the cleaner signal. Our selector must handle cases where the text is helpful and the image is cluttered.

Research Program

Measuring the point of no return. The first stage is measurement. We take an image question $(v, q)$ with auxiliary text $c$, and we have a clean counterpart $c_0$, which is either helpful or no snippet at all. We run the diffusion reasoner with $c$ until step $s$. At that point we swap in $c_0$ while keeping all tokens already unmasked, finish decoding, and record whether the final answer matches the clean-context answer. This gives a recovery curve:

$$R(s) = \Pr\big[\hat a(c \rightarrow c_0 \text{ at } s) = \hat a(c_0)\big],$$

Here $\hat a(\cdot)$ is the final decoded answer and the probability is over examples and sampling seeds. We will read off three properties: how early $R(s)$ falls, how sharp the drop is, and whether the commitment step differs between visual-contradiction items and text-only distractors. This is the discrete-token, interventional analogue of the trajectory sensitivity analysis in Early Semantic Commitment. It directly answers that paper's open question of whether commitment can be undone after onset. If $R(s)$ decays slowly and roughly linearly, the phase story is wrong for this setting, and the allocation half of the project stops.

A cheap commitment detector. At test time, $c_0$ is unknown. The detector must therefore work from the model's own predictions. At every step a diffusion language model already outputs a distribution over all masked positions, including the answer slot, which is the discrete counterpart of the clean-image estimate used in An Efficient Test-Time Scaling Approach for Image Generation. We track two signals on the answer slot. The first is stability: whether the argmax answer survives $K$ cheap perturbations of the unmasking order. The second is context leverage: a divergence $D_t$ between the answer distribution under the full input and under the input with $c$ masked out, computed once every few steps. The estimated commitment step $\hat t^*$ is the first step after which the answer is stable across perturbations. The window of interest is the set of steps before $\hat t^*$ where $D_t$ is large. We calibrate the detector against the $R(s)$ curves from the measurement stage and report how well it predicts the true point of no return.

Allocating compute to the window. Inside the window, the method spends compute in two ways. It branches into $N$ trajectories. It also applies image-anchoring guidance:

$$\log \tilde p_t(x) = (1+w_t)\,\log p(x \mid v,q,c) - w_t \,\log p(x \mid \varnothing,q,c),$$

Here $\varnothing$ denotes masked visual tokens and $w_t$ is nonzero only inside the estimated window. The context $c$ appears in both terms, so the guidance amplifies what the image adds beyond the text without deleting the text. This is the property needed to keep helpful snippets useful. At $\hat t^*$, the method keeps the branch whose answer distribution is most stable when $c$ is masked. If text-masking actually changes that branch's answer, it falls back to the majority branch. It then finishes that single branch with no guidance and an aggressive step count, much as Thinking Diffusion reports that fewer steps suffice. Total forward passes are logged and matched across all methods.

Baselines

The naive reference point is the diffusion multimodal model with default decoding and no extra compute, run on clean, helpful, irrelevant and misleading context conditions. That gives each setting its own size of the drop due to noise. A second naive point is the same model with the auxiliary text simply deleted. This is an upper bound on robustness for misleading snippets and a lower bound on usefulness for helpful ones.

The strongest existing method is Thinking Diffusion's combined step penalty and visual guidance (Thinking Diffusion), applied at every step. It is the only published inference-time method that targets both early commitment and weak visual grounding in these models, so it is the competitor our phase claim must beat. We run it on the same backbones (LaViDa, MMaDa) and the same inputs. Its guidance scale is tuned on the same held-out split, with the same number of tuning trials as our window parameters. We report its accuracy at its natural cost and at a cost matched to ours. On clean benchmarks its reported number is up to 7.5% higher accuracy across M³COT, ScienceQA, MMBench and V* Bench. On context-corrupted inputs we found no reported result on Lacuna for diffusion multimodal models. Beating it means higher accuracy under misleading context at equal forward passes, with no more than a one-point loss on its clean benchmarks.

The compute-matched search baseline is uniform Best-of-N with majority voting over full trajectories. A second variant uses an external verifier that scores the final answer against the image. $N$ is set so that total forward passes equal those of our method.

Three ablations isolate the key ingredient, the placement of compute. The first gives the identical guidance and branching budget to a window of the same length placed late. The second places it at random. The third uses a fixed early window shared across examples, which tests whether per-example detection matters at all. If the fixed early window matches the adaptive one, the detector is unnecessary and we will say so.

Why not just…?

Why not just run Best-of-N or self-consistency with more samples? Under misleading context, the error is systematic rather than random. The SciAux results show that reasoning amplifies the false premise (Thinking in a Crowd), so majority voting over many samples is expected to converge on the rationalized answer. The image-generation evidence also says uniform search wastes most of its budget on steps where nothing is still undecided (An Efficient Test-Time Scaling Approach for Image Generation). This alternative is settled empirically by the matched-compute comparison. If Best-of-N closes the gap at equal forward passes, the allocation claim fails.

Why not just apply visual guidance at every step, as Thinking Diffusion does? This is the real threat. If uniform guidance is as robust as windowed guidance, the phase story adds nothing except a cost saving. Two things favor windowing. Uniform guidance pays for doubled passes at all steps. Its authors also report that high guidance strength makes the model ignore the text instruction, and windowing should avoid that. The late-window and uniform comparisons at matched cost settle the question directly. A cost-only win is still worth reporting, but we will not present it as a robustness gain.

Why not just strip or rerank the context before the model sees it? Misleading snippets are often topically relevant, which is exactly why they mislead. Superposition-style pruning showed that filtering can also discard pieces that multi-hop questions need (Superposition Prompting). Rerankers also cannot touch distractors embedded in the image itself, such as chart titles (Plot Twist). We include a reranker-filtered input as an extra baseline. The helpful-context condition will show whether it throws away useful evidence.

Why not just use an autoregressive reasoning model with thinking off? On SciAux, thinking off is only about two points worse than thinking on under misleading snippets, but it gives up the 10–13% gain that thinking provides without context (Thinking in a Crowd). We report an autoregressive multimodal reasoner, both thinking-on and thinking-off, as an external reference. We also run a transfer test in which branching is concentrated on the first reasoning tokens instead of spread across the whole chain.

Proposed Experiments

The primary testbed is a multimodal version of SciAux (billhdzhao/SciAux). SciAux is derived from ScienceQA, so we can re-attach the original images to items that had them and pair each with SciAux's helpful, irrelevant and misleading snippet combinations. The first engineering step is to verify how many items retain images. If too few do, we will write misleading snippets for image-bearing ScienceQA items following the SciAux protocol.

The second testbed is Chart101 (Plot Twist), with titles that are original, removed, or rewritten to be misleading. Here the distractor sits inside the visual input, and the reported 79% versus ~30% title effect is the reference gap. RH-Bench (More Thinking, Less Seeing?) measures whether our method trades perception for reasoning as the budget grows. We report RH-AUC against the published best of 0.63 (Ocean-R1-7B), noting that it was obtained with autoregressive models. M³COT, ScienceQA, MMBench and V* Bench guard against regressions on clean inputs.

Metrics are accuracy for each context condition, the noise-induced drop relative to clean, total forward passes, and wall-clock latency. All are reported as accuracy-versus-compute curves rather than single points.

The decisive checks follow from the claim. First, the measured recovery curve $R(s)$ must fall sharply within the first part of decoding. Second, the estimated commitment step must track the true one on held-out items. Third, at matched forward passes, the early adaptive window must beat the same budget placed late, placed at random, and spread uniformly. Fourth, accuracy in the helpful-context condition must not fall below the no-guidance reference. Each failure has a specific reading. A flat $R(s)$ means commitment is not early in this setting. A fixed window that equals the adaptive one means the detector is unnecessary. A uniform budget that equals the window means the gain, if any, comes from guidance and not from timing. A drop in helpful-context accuracy means the guidance is suppressing text rather than anchoring to the image.