When a vision-language model solves a diagram or chart problem step by step, each sentence of its reasoning does one or both of two jobs. It can pull new facts out of the image, which is a sensing job. It can also move the model's belief toward an answer, which is closer to a communication job. Radar-communication engineering faced the same pairing and resolved it by measuring both jobs in one unit, mutual information, and then plotting the achievable trade-off between them. This proposal carries that move into multimodal reasoning. For every step of a model's rationale we measure two rates: how much image-specific information the step carries, and how much it changes the model's belief about the answer. We then test whether the pair of rates explains the known failures of visual chain-of-thought better than any single score. Those failures are misreading the figure, reasoning wrongly from a correct reading, answering from text priors, and chain-of-thought that makes accuracy worse.
The practical payoff is a decoding rule that spends tokens on "looking again" only when the reasoning is committing to an answer faster than the image supports. The rule is tested against the simpler alternatives a reviewer would reach for first. One is to describe the image fully and then reason. Another is to boost visual grounding at every token. The borrowed engineering result supplies a sharp, falsifiable hypothesis. In radar-communication systems, integrated designs strictly enclose the rate region of systems that split the band between the two functions. The analogue here is that interleaving sensing inside the rationale should beat "describe first, reason later" at a matched token budget.
Core Claim
The source of the lens is Integrated Sensing and Communication (ISAC), where one waveform must both carry data and probe the environment. Integrated Sensing and Communications: A Mutual Information-Based Framework expresses both functions as mutual information. This lets it compute a joint rate region and show that the integrated design encloses frequency-division designs, the ones that give each function its own band, with a higher high-SNR slope. Related work does the same for waveform power allocation in Waveform Design for MIMO-OFDM Integrated Sensing and Communication System: An Information Theoretical Approach. A further paper, Waveform design based on mutual information upper bound for joint detection and estimation, folds detection and estimation into a single bound. We take no algorithms from this literature, since its Gaussian channel models do not transfer. We take its methodological stance: measure two competing functions in one currency, and study the frontier between them rather than a single scalar.
Concretely, let $Q$ be the question, $I$ the image, $r_{<t}$ the rationale before step $t$, and $a^*$ the gold answer. Define the step's sensing rate as a conditional pointwise mutual information between the step text and the image:
$$s_t = \log p(r_t \mid Q, I, r_{<t}) - \log p(r_t \mid Q, \tilde I, r_{<t}),$$
where $\tilde I$ is a counterfactual image, either deleted or replaced by a matched distractor figure. Define the step's reasoning rate as the change in log-belief in the gold answer, read by an answer probe appended after the prefix:
$$c_t = \log p(a^* \mid Q, I, r_{\le t}) - \log p(a^* \mid Q, I, r_{<t}).$$
We also compute a blind counterpart $c^{\text{blind}}_t$, the same quantity with $I$ replaced by $\tilde I$ but the rationale held fixed. The gap $c_t - c^{\text{blind}}_t$ measures how much of the answer progress still depends on the image rather than on what the rationale has already written down. This follows the chain rule $I(A; I, R \mid Q) = I(A; R \mid Q) + I(A; I \mid Q, R)$. A healthy rationale should move image information into text (positive $s_t$ early) and then drive the residual term $I(A; I \mid Q, R)$ toward zero while $c_t$ rises.
The claim is that the joint trajectory of $(s_t, c_t, c^{\text{blind}}_t)$ separates the failure types that single scores merge. A vision error should appear as low $s_t$. A reasoning error should appear as healthy $s_t$ with negative $c_t$. A text shortcut should appear as large $c^{\text{blind}}_t$ throughout. Chain-of-thought degradation should appear as "ungrounded commitment": late steps with $s_t \approx 0$ and rising confidence in a wrong answer.
Relation to Existing Work
Three papers already put information measures inside vision-language models, and each fixes one coordinate of what we propose.
See First, Reason Later: Mutual Information-Guided Reinforcement Learning for Vision-Language Models (MIRL) scores short visual descriptions by contrasting token probabilities with and without the image. It forks only high-scoring descriptions into full reasoning and gives a separate perception reward. It reports that description MI correlates with MathVista accuracy up to Pearson $r = 0.94$. MIRL is our strongest motivation and our closest competitor. It is, however, a two-stage design: one sensing score on a prefix, then a binary correctness reward. In ISAC terms this is the frequency-division design, and it never measures information flow inside the reasoning itself.
Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (CMI-VLD) rewards tokens with high conditional pointwise MI during decoding and prunes a fixed 20% of visual tokens. It supplies our sensing estimator almost directly. It applies grounding uniformly, though, and it is evaluated only on descriptive hallucination benchmarks, not on multi-step math.
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition splits a model's decision information into unique-vision, unique-text, redundant, and synergistic parts across layers. It uses the synergy term to reweight fine-tuning (+2.3% on MMStar over uniform fine-tuning). It is restricted to multiple-choice outputs and to layer-wise rather than step-wise analysis. Our answer probe over a candidate set drawn from sampled completions is a direct way to extend step-wise measurement to free-form numeric answers. We therefore use this paper's synergy score as a competing diagnostic, not as our method.
The phenomenon side supplies the targets. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset attributes 42.2% of GPT-4V errors to reasoning and 31.9% to vision recognition, and reports that chain-of-thought can lower accuracy. We treat this as the anomaly our trajectories must explain. Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details shows the "harder is easier" pattern: title hints push accuracy to 79%, and it falls to the 30% range without them. That is a behavioral finding our blind rate $c^{\text{blind}}_t$ should reproduce from inside the model. GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level acts as a constraint. Even perfect human transcriptions of diagrams left models near chance (best 26.22% against 25% random), so transcription alone is not a solution. This warns us not to assume that high sensing guarantees success.
Research Program
The first stage builds the measurement. We segment rationales into sentence-level steps. We compute $s_t$ with two counterfactual images, deletion and a same-type distractor figure from the same benchmark, because the PID paper flags masking-based counterfactuals as a source of out-of-distribution artifacts. Agreement between the two counterfactuals is itself reported. For the answer probe we append a fixed "so the answer is" continuation. For free-form problems we score a candidate set formed from the gold answer plus answers from eight sampled completions, which gives a normalized belief without needing multiple-choice options. Cost scales with the number of steps times a small constant number of forward passes. We report it explicitly, since MIRL's overhead from two forward passes is already a noted limitation.
The second stage is diagnosis. On open-weight models of the families used in the cited work (LLaVA-1.5 and LLaVA-NeXT, InternVL-Chat), we collect rationales on MATH-Vision, MathVista, ChartQA, and Chart101. We then hand-label a stratified sample of several hundred errors with the MATH-Vision taxonomy. The test is whether trajectory features predict the error label and correctness, and whether they explain the chain-of-thought-hurts cases.
The third stage is the intervention, a two-rate controller at decode time. When cumulative answer confidence rises while recent $s_t$ stays near zero, the controller inserts a short "re-read" step. That step is decoded with CMI-style visual boosting and is prompted to restate the figure element relevant to the current step. Otherwise decoding proceeds normally. Because $a^*$ is unknown at test time, the controller uses the model's own top-candidate belief in place of $c_t$. The gold-answer version is used only for analysis and, optionally, as a dense step reward in a small reinforcement-learning follow-up compared against MIRL on ViRL39K.
Baselines
The naive reference points are direct answering and plain chain-of-thought with the same model and a matched maximum token budget. Plain chain-of-thought matters most, because MATH-Vision reports that it can underperform direct answering, and our controller must at least recover that loss. For scale, published accuracies on MATH-Vision are 30.39% for GPT-4o and 16.97% for the open-source leader InternVL-Chat-V1-2-Plus (Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset). On MathVista, GPT-4V scores 49.9% (MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts). We will re-measure our own open models rather than compare to these numbers directly.
The strongest existing method for the diagnostic question is MIRL's description-level visual MI. We reimplement its scoring rule, ten candidate descriptions with the top ones forked into reasoning, on the same models. For its training results, MIRL reports a 70.22% six-benchmark average against 69.76% for DAPO-16 while using 25% fewer full trajectories. No reported result was found on Lacuna for MIRL on MATH-Vision or Chart101. Beating MIRL diagnostically means our trajectory features give a clearly higher correctness AUC than its single prefix score on the same problems. Beating it as a decoding procedure means higher accuracy at equal total generated tokens, counting its discarded candidates.
The strongest existing method for the intervention is CMI-VLD-style visual boosting applied at every token of the rationale, with the boosting weight tuned on the same held-out split and with the same search budget as our controller's threshold. No reported result was found on Lacuna for CMI-VLD on MATH-Vision, MathVista, or ChartQA. We run the decoding rule without its trained purifier so that no side gets extra supervision. We also run a describe-then-reason pipeline, the frequency-division analogue, at matched token budget. The PID synergy score is used as a competing diagnostic on the multiple-choice subsets where it applies.
Two ablations isolate the key ingredient. The first controller triggers on $s_t$ alone. The second triggers on answer-confidence growth alone. A third heuristic re-reads every $k$ steps with the same total number of re-read steps as our controller. If the periodic version matches ours, the information signal is not doing the work.
Why not just…?
Why not just describe the image in full, then reason over the description? This is the obvious engineering answer and MIRL's implicit design. GeoSQA already shows that perfect transcriptions do not rescue reasoning. Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs reports that derendered tables lose color and style cues that questions refer to. Its chart-to-table training nonetheless reaches 77.3% on ChartQA (81.3% with program-of-thought), so on charts this alternative is a real threat. The matched-budget comparison settles it. If describe-first equals interleaved sensing on ChartQA but loses on MATH-Vision geometry, where the relevant detail depends on the current step, that is the result the ISAC analogy predicts. If it wins everywhere, the rate-region framing is wrong for this domain and we will say so.
Why not just apply grounded decoding everywhere? Uniform visual boosting is cheaper and needs no controller. Our hypothesis is that it taxes pure-calculation steps, where the image should not matter and where the image-versus-no-image contrast rewards irrelevant visual mentions. The tuned uniform CMI baseline tests this directly. If it matches the controller, the second rate adds nothing and the paper becomes a diagnostic study.
Why not just use the model's answer confidence or self-consistency to catch errors? Confidence alone cannot distinguish a shortcut from a grounded derivation. Plot Twist shows models can be confidently right for text-prior reasons. The confidence-only ablation, and the comparison of $c_t$ against $c_t - c^{\text{blind}}_t$ on Chart101's title-stripped versus title-hinted charts, measures exactly this. We expect the blind gap, not confidence, to track the 79%-versus-30% swing.
Why not just train end to end with correctness rewards? Outcome-only reinforcement learning cannot say whether a failed trajectory misread the figure or miscalculated, which MIRL calls reward blindness. We do not claim to beat reinforcement learning at scale. The optional follow-up compares a dense two-rate step reward against MIRL and a DAPO-style outcome reward on ViRL39K at equal trajectory counts. It is framed as a secondary check.
Decisive Checks
Three results would confirm the claim, and their absence would refute it.
The first is diagnostic separation. On the hand-labeled MATH-Vision error sample, a classifier on trajectory features should separate vision, reasoning, and shortcut errors well above a classifier on MIRL's prefix score or PID synergy. Most importantly, reasoning errors should show normal sensing with falling gold-answer belief, which no single score can express.
The second is a mechanistic account of chain-of-thought degradation. On problems where chain-of-thought is wrong but direct answering is right, late steps should show near-zero sensing together with rising confidence in the wrong answer, at a higher rate than on problems where chain-of-thought helps.
The third is the rate-region test. Plotting mean cumulative sensing against final accuracy at several token budgets, the interleaved controller's frontier should lie outside the describe-then-reason frontier on MATH-Vision and MathVista. It should also beat uniform CMI decoding and the periodic re-read heuristic at matched budgets.
A null result on the third check with success on the first two would still be useful: a step-level instrument for free-form visual reasoning that extends the multiple-choice-only PID analysis.
References
- Integrated Sensing and Communications: A Mutual Information-Based Framework
- Waveform Design for MIMO-OFDM Integrated Sensing and Communication System: An Information Theoretical Approach
- Waveform design based on mutual information upper bound for joint detection and estimation
- See First, Reason Later: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
- Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs