Adding vision to a language model tends to damage its text reasoning. Math scores drop after multimodal training, and a common repair is to blend the original text-only weights back in. In the other direction, when a vision-language model reasons step by step, it often starts with a correct visual observation and then argues itself out of it, siding with what language makes plausible instead of what the image shows. This note argues that these are one problem seen from two ends. The blending weight that restores text reasoning is, we suspect, the same knob that strengthens the language habits behind visual hallucination. If so, no single setting of that knob can be right for a whole reasoning trace. Steps that report what is in the image want the multimodal weights. Steps that do algebra want the text weights.
We propose to set the blending weight separately for each generated token. A small probe reads the model's hidden state and predicts whether the next token should depend on the image. That prediction decides how much of the "vision update" (the weight difference between the multimodal model and its text-only base) to apply. If the vision update is stored in low-rank form, meaning it is compressed into a few thin matrices, per-token blending costs one extra small matrix product per layer rather than a second model. The claim is testable in two parts. First, global blending should trade text-math accuracy against hallucination under step-by-step reasoning. Second, token-level blending should reach points on that trade-off that no global blend and no per-input switch can reach.
Core Claim
Several findings motivate the claim. Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning shows GSM8K collapsing after multimodal instruction tuning. That paper recovers much of the loss by TIES merging, a method that combines two sets of weights while discarding small and conflicting weight changes, with a blend near $\alpha \approx 0.1$ toward the multimodal weights. Is Extending Modality The Right Path Towards Omni-Modality? reports a drop of up to 10.2% on MATH after modality extension. It finds that weighted merging only partly restores reasoning, and that head masking shows the damage is spread across all attention heads.
Neither paper measures hallucination. The hallucination literature, meanwhile, blames exactly the language priors that merging restores. FREAK shows step-by-step prompting lowering accuracy by over 5% on images that violate common sense. What Color Is It? shows reasoning models writing "I see blue text" and then answering with the printed word's meaning. That benchmark reports 0% on its Color subset for Qwen2.5-VL-7B and InternVL3.5-14B.
Our first claim is that sweeping the global blend $\alpha$ traces a real trade-off. Text reasoning rises as $\alpha$ falls toward the base model, while step-by-step hallucination on FREAK and What Color Is It? gets worse. Nobody has measured this curve.
The second claim concerns what happens inside a single trace. Don't Blink finds that attention to the relevant image region decays during reasoning in 83.5–100% of samples. It also finds this decay is harmless on symbolic tasks once the visual variables have been read off. Steer Where It Matters finds that dependence on the image is concentrated in a small fraction of output tokens. Together these suggest the ideal blend changes across a trace: near the multimodal model on the few visually grounded tokens, near the base model on the rest. We write the per-token forward pass for each adapted weight matrix as
$$h = W_{\text{base}}\,x + \alpha_t\, U V^\top x,$$
where $x$ is the layer input, $W_{\text{base}}$ is the text-only weight, $UV^\top$ is a rank-$r$ approximation of the (TIES-trimmed) vision update, and $\alpha_t \in [0,1]$ is set per token by the probe. We use the probe rather than the model's current attention to the image because the failure we target is precisely the model not looking when it should. The signal must predict whether the model ought to look, not whether it currently does.
Research Program
The work has three stages, each gated by the previous one.
Stage 1: map the global trade-off. For paired models (LLaVA-1.5 with its Vicuna base, following the merging paper; Qwen2-VL with its Qwen2 base, following the omni-modality study; and R1-OneVision-7B with its text base, since Diagnosing and Mitigating Modality Interference found it still vulnerable to distractors), we sweep a global $\alpha$ from 0 to 1. At each setting we record GSM8K and MATH, and FREAK and What Color Is It? both with and without step-by-step prompting. We also record HallusionBench question-pair accuracy and the evidence-region attention mass from Don't Blink. This tests the mediation story directly: if lowering $\alpha$ worsens hallucination, attention collapse should get faster at the same time.
Stage 2: an oracle ceiling. Before building anything deployable, we compute an upper bound. Offline, we label each token of each trace with the divergence between the model's image-conditioned and image-free predictions, following the visual-sensitivity score of TLVS. That score costs two forward passes per token, which is acceptable offline but not at deployment. We then set $\alpha_t$ from this oracle label. If the oracle schedule does not beat the best global $\alpha$ on both axes at once, the project stops here.
Stage 3: a deployable probe. We train a linear or two-layer probe on middle-layer hidden states to predict the oracle label one token ahead, then map its output to $\alpha_t$ through a single threshold-and-slope pair tuned on held-out data. Deployment then needs only one forward pass. We will report two things separately: how accurately the probe predicts the oracle label, and how much the end tasks gain. We will also check specifically whether the probe fails on traces where the model has already drifted into prior-driven text, since that is where it matters most.
Baselines
The naive reference points are the unmodified multimodal model ($\alpha = 1$), run with and without step-by-step prompting, and the text base ($\alpha = 0$) on text-only tasks. These give the two ends of every trade-off plot. Plain step-by-step prompting is not assumed to help: FREAK reports that it lowers accuracy by over 5% for large models.
The strongest weight-space competitor is global TIES merging at the best single $\alpha$ from Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning. We tune $\alpha$ on the same validation split and with the same number of trial settings used to tune our threshold and slope. That paper reports its gains qualitatively (GSM8K "plummeted" and merging recovers it); no reported result on FREAK, What Color Is It?, or HallusionBench was found on Lacuna for merged models, so we produce that number ourselves.
The most dangerous simple baseline is input-level routing. It uses the merged low-$\alpha$ model when no image is present and the full multimodal model when an image is present. It costs nothing and could match us on pure-text and pure-perception benchmarks. It can only lose on inputs that need both careful grounding and real symbolic reasoning, so MathVista and MathVerse are where our method must beat it. No reported result was found on Lacuna for input-level routing.
The strongest inference-time competitor is Functional Attention Control. It boosts attention to image tokens in shallow "perception" heads and to text evidence in deep "reasoning" heads by fixed gains of 1.1–1.4, and it reports an average 4.2 percentage-point gain across five benchmarks at about 9% latency overhead. We run it at its published gains on the unmodified model, and also stacked on our method, since the two act on different axes. For grounding sanity checks we include TLVS, which reports POPE accuracy on LLaVA-1.5 rising from 85.09% to 89.78%. We note that TLVS doubles per-token cost, so we compare at matched wall-clock time as well as at matched quality.
Three ablations isolate the key ingredient. The first replaces the probe with the model's current attention mass on image tokens, which tests our argument that attention is the wrong signal. The second replaces the probe with a random schedule that has the same average $\alpha_t$. The third sweeps the rank $r$ to confirm that compressing the vision update, not the schedule, is not what drives the results.
Beating the baselines means the following. At matched latency within 15% of the unmodified model, our method must recover at least half of the MATH gap between the multimodal model and its base. At the same time it must match or exceed the unmodified model with step-by-step prompting on FREAK and What Color Is It?, and beat input-level routing on MathVista or MathVerse by more than the seed-to-seed spread.
Why not just…?
Why not just route by input, with text weights for text and vision weights for images? This is the real threat. If Stage 3 cannot beat it on MathVista and MathVerse, the per-token idea has no practical value. It would then reduce to a finding about the global trade-off plus a recommendation to route. We accept that outcome as a legitimate result. Our reason to expect a gap is that Don't Blink shows diagram-math traces need the image early and not late, which is a within-trace structure that input routing cannot express.
Why not just skip or shorten step-by-step reasoning on visual questions? FREAK's backfire result makes this tempting. However, Thinking Before Looking shows structured reasoning improving MMVP by 31.74% for Gemini 1.5 Pro, so turning reasoning off also throws away real gains. We include no-reasoning and truncated-trace conditions, including answering straight from the first stated visual observation. If truncation matches our method on FREAK and What Color Is It?, we will say so. It would still leave the text-math loss from multimodal training untouched.
Why not just use attention rescaling or activation steering, which already exist? FAC uses fixed layer boundaries and fixed gains. TLVS was never tested on long reasoning traces and doubles decoding cost. Neither method changes the weights, so neither addresses the MATH and GSM8K damage documented by the merging papers. Our experiments run both methods as baselines and stacked with ours, so the data will show whether the weight-side schedule adds anything beyond attention control.
Why not just re-look at the image mid-trace, as AVCR does? Analyzing Reasoning Consistency under Cross-Modal Conflicts adds a multi-pass check, re-reason, and summarize loop. It raises explicit self-correction from about 5% to 29%, but at considerable latency, and it is validated on a 100-sample STAR video subset. We compare against a single-pass re-look prompt on its STAR setup at matched token budgets. If our method loses there, that suggests grounding failures in video need re-reading of the input, which reweighting cannot supply.
Decisive Checks
Three checks decide the project. First, the Stage 1 sweep must show that hallucination with step-by-step prompting worsens as $\alpha$ moves toward the base model; without that curve the premise fails. Second, the oracle schedule must dominate every global $\alpha$ on at least one of the paired models; without that, the per-token claim is false. Third, the probe schedule must beat input-level routing on MathVista or MathVerse while holding the What Color Is It? Color subset and FREAK at or above the unmodified model. Secondary checks include agreement between the probe and the oracle on traces that drift into prior-driven reasoning, and a confirmation on POPE and CHAIR that general grounding is not harmed.
References
- Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning
- Is Extending Modality The Right Path Towards Omni-Modality?
- FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs
- What Color Is It? A Text-Interference Multimodal Hallucination Benchmark
- Don’t Blink: Evidence Collapse during Multimodal Reasoning
- Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
- Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
- Mitigating Hallucination in Multimodal Reasoning via Functional Attention Control
- Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination
- Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts