Measuring Modality Bias Against Known Information Budgets

Multimodal language models are often accused of "ignoring the image": they answer from language habits instead of from what the picture shows. The standard evidence is behavioral. Researchers delete or noise the image and check whether the answer changes, or they count how much attention weight lands on image tokens. Both tests share a blind spot. They cannot tell a model that ignores useful visual evidence apart from a dataset in which the text already gives the answer away. A model that leans on text is behaving correctly when the text is sufficient, and is biased only when the image holds answer-relevant information the text lacks and the model fails to use it.

We propose to define and measure modality bias as an information-use deficit. The deficit is the gap between how much answer-relevant information the image contains beyond the text, and how much of that information the model's predictions actually extract. To make this measurable, we build image–text–answer datasets where these quantities are known in closed form. We then check which high-dimensional mutual information estimators recover them from real model inputs and hidden states. Once an estimator is validated, we ask whether popular hallucination fixes shrink the deficit or only move the model's answer prior. Mutual information here means the number of bits one variable reveals about another.

Core Claim

Existing modality-bias metrics mix a property of the data, namely how much the text alone predicts the answer, with a property of the model, namely how much available visual evidence it uses. Let $V$ be the image, $T$ the text context, and $Y$ the answer. The information the image uniquely offers is the conditional mutual information $I(Y;V\mid T)$. What the model uses can be measured as predictive $\mathcal{V}$-information. That quantity, from A Theory of Usable Information under Computational Constraints, is information that a restricted family of predictors can actually extract; it can run in one direction only and stays estimable in high dimensions. We define

$$\mathrm{Def}_\theta = I(Y;V\mid T) - \Big(\mathbb{E}\left[\log p_\theta(y\mid v,t)\right] - \mathbb{E}\left[\log q_\theta(y\mid t)\right]\Big).$$

Here $p_\theta$ is the multimodal model and $q_\theta$ is the same architecture fine-tuned with the image removed, so it is the best text-only member of that model family. We also report the normalized deficit $\mathrm{Def}_\theta / I(Y;V\mid T)$, the fraction of unique visual information left unused. The claim we test is this. Once available information is controlled, the standard proxies rank models and interventions differently from the true deficit. Among these proxies are consistency under image ablation (MLLMs are Deeply Affected by Modality Bias), Blind Gap and Visual Gain (From Accuracy to Visual Dependence), and the layer-wise modality attention ratio (LLaVA Steering). A validated estimator of the deficit transfers to real benchmarks and changes conclusions about which fixes add visual grounding.

The first term, $I(Y;V\mid T)$, is where high-dimensional estimation becomes necessary. On real data it must be estimated from images of thousands of dimensions. The answer $Y$ is only a few bits, so the dependency is low-rank. That is the regime in which the latent-bottleneck estimator of Approximating mutual information of high-dimensional variables using learned representations is expected to work. Its authors report stable estimates above 1,000 dimensions, while other estimators fail beyond about 30.

Relation to Prior Work

Several papers motivate the gap. MLLMs are Deeply Affected by Modality Bias reports that Qwen2.5-VL on MMMU-Pro agrees with its text-only self 56.53% of the time. It uses output consistency alone and does not ask how much the image could have contributed. Unveiling the Ignorance of MLLMs argues that models "see clearly but answer incorrectly." Our layer-wise measurements test that claim directly, by checking whether visual information is present in hidden states yet missing from outputs. Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention finds attention heads that favor text roughly 5× over vision, and concedes that attention weight is not information transfer. We measure the transfer itself.

Other work comes close to our question but does not ask it. Mutual Information Analysis in Multimodal Learning Systems estimates camera–LiDAR mutual information with a compression model and finds that lower cross-sensor information goes with higher detection mAP. Two Effects, One Trigger finds the opposite pattern in CLIP-style models, where richer shared information reduces object bias. We read this tension as evidence for our framing. Both papers measure information between modalities, whereas the quantity that governs rational reliance is information each modality holds about the answer given the other. Neither paper studies generative multimodal language models.

Exploiting Contextual Knowledge in LLMs through V-usable Information based Layer Enhancement tracks usable context information across the layers of text-only models. It finds a peak at layers 20–26 followed by decay. We carry that layer-wise analysis over to visual tokens and give it a ground-truth reference.

The method comes from two sources. Multimodal Datasets with Controllable Mutual Information supplies the generator. Gaussian latents are mixed with known dependencies, then passed through an invertible flow into realistic CIFAR-10-like images, and the invertible step preserves the exact mutual information. The estimator suite comes from MINE, InfoNCE, NWJ, SMILE, LMI, and MIST. One constraint matters from the start. MIST was trained on at most 32 dimensions and cannot exceed the range of values it saw in training, so it serves only on low-dimensional projections. InfoNCE is capped at the logarithm of the batch size, which rules it out wherever the true value is large.

Research Program

The first component is a controlled benchmark. We sample three groups of latents: image-only latents $z_v$, text-only latents $z_t$, and shared latents. The answer is a quantized linear-Gaussian readout of chosen combinations of these latents. The image latents pass through the invertible flow to become images. The text latents are quantized and written into the prompt as short numeric or categorical statements.

All the relevant quantities are finite-dimensional and nearly Gaussian before quantization, so we can compute them numerically to high precision. These quantities are $I(Y;V\mid T)$, $I(Y;T\mid V)$, and the redundant part. Varying the mixing weights produces a grid of datasets. At one end the text is sufficient; at the other only the image is informative. In between, the two modalities are partially redundant, which is the case where behavioral proxies should diverge from the deficit. A second axis controls rank: the visual evidence is spread over either 2 or 32 independent latent directions. This deliberately probes the high-rank failure mode reported for LMI.

The second component is estimator validation at two levels. At the input level, each estimator must recover $I(Y;V\mid T)$ from pixels, or from frozen vision-encoder features, together with text. We estimate the conditional quantity as $I(Y;V,T) - I(Y;T)$, using the same estimator for both terms so their errors partly cancel.

At the hidden-state level, we estimate $I(H_\ell; z_v)$ and $I(H_\ell; Y)$ at each layer $\ell$ of the multimodal model, at the answer position. Here $H_\ell$ is the hidden state, of dimension 4096 or more, and $z_v$ is the known image latent, of dimension 2 to 32. We have no exact ground truth at this level. The data-processing inequality does give a hard ceiling: no layer can hold more information about $z_v$ than the image does. Any estimate above that ceiling marks the estimator as failed. Using the known low-dimensional latent in place of pixels also keeps the dependency low-rank, which is the setting where LMI should hold up.

The third component uses the models. We fine-tune LLaVA-v1.5-7B and Qwen2.5-VL on each synthetic dataset, alongside text-only counterparts $q_\theta$ trained with identical budgets, and compute the deficit exactly. The layer-wise curves then locate where visual information is lost. The loss could occur in the connector, which would mean the information never enters, or in deep layers, which would mean it enters and then decays, the "seeing clearly, answering incorrectly" pattern. The curves also give a principled way to choose which layer to steer, a choice MoReS currently makes by trial and error.

The fourth component moves to real benchmarks. With the validated estimator we compute the estimated available information $\hat I(Y;V\mid T)$ and the deficit on POPE and MMMU-Pro. We also run them on traffic VideoQA sets: MM-AU, which has a Blind Gap near 40% and negative Visual Gain, and SUTD-TrafficQA. We then measure how the deficit changes under several existing fixes: RBD, MoReS, Vittle (code at vittle), and MFPO where checkpoints are available. We expect some reported gains, such as POPE accuracy, to come mainly from lowering the "yes" rate, which reduces text-prior error without increasing visual information use. The deficit separates these two effects. Accuracy does not.

Baselines

The naive reference point is behavioral ablation with no information estimate. It includes image-removal consistency and the Modality Imbalance Ratio from MLLMs are Deeply Affected by Modality Bias, and Blind Gap and Visual Gain from From Accuracy to Visual Dependence. We compute them on exactly the same model–dataset pairs and score each by Spearman rank correlation with the true deficit across the synthetic grid. On MM-AU, that paper's continuous shortcut score reduced Blind Gap from 25.70 to 10.38 and raised Visual Gain from −0.24 to +7.42. This continuous score is the strongest existing dataset-side competitor, since it already grades rather than filters. We reimplement it with the same model panel. Our measure beats it if it tracks the true deficit more closely in the partially redundant regime, and if deficit-based data reweighting matches or exceeds its Visual Gain on MM-AU at equal retained data.

The attention-based baseline is the layer-wise modality attention ratio from LLaVA Steering, computed per layer over the same prompts. It wins if its layer ranking matches our information curves; otherwise we win.

The strongest model-side baseline is contrastive decoding. RBD reaches 88.0% POPE accuracy on LLaVA-v1.5-7B and reduces CHAIR_S from 20.8% to 17.8% (Re-Balancing Contrastive Decoding). For comparison, MoReS reports 88.2 POPE accuracy and Vittle reports gains of up to +1.6% on corrupted POPE. We treat these as interventions to be diagnosed rather than competitors to be beaten on accuracy. The prediction that could fail is that their deficit reduction will be smaller than their accuracy gain implies.

For the estimators, NWJ and SMILE had the lowest error on the controllable-flow data, where all estimators exceeded 0.99 correlation (Multimodal Datasets with Controllable Mutual Information). We found no reported result on Lacuna for any estimator on multimodal language model hidden states. All estimators get the same samples, the same critic sizes, and the same tuning budget.

The key ablation removes the availability term. We score models by usable visual information alone, without subtracting it from $I(Y;V\mid T)$, and check whether rank agreement with the true deficit collapses in the partially redundant cells. If it does not collapse, the core claim fails.

Why not just…?

Why not just use Blind Gap or image-ablation consistency? When the text carries no answer information, Blind Gap and the deficit coincide, so those metrics are adequate there. The disagreement appears only when the text is partially informative, which is also the case on real benchmarks such as MM-AU, with its answer phrase repeated 234 times. The synthetic grid measures exactly how large the disagreement is. If it turns out small across the grid, we will report that the cheap proxies are sufficient.

Why not just read attention ratios? The authors of the attention-head attribution study concede that attention weight is not information flow. The layer-wise comparison against the ceiling set by the data-processing inequality tests directly whether the two diverge.

Why not just use the contrastive-decoding log-ratio as the measure? This is the most serious threat. The log-ratio between predictions with the original image and with a degraded image is a pointwise estimate of the usable-information term. What it lacks is the availability term. Its reference branch is also impure: MFPO reports 91% accuracy with heavily noised images, so a noised image still carries text-shortcut behavior. We therefore compare three reference distributions: the noised-image branch, the fine-tuned text-only $q_\theta$, and a blank image. If the noised-image branch matches $q_\theta$ in rank agreement, we will adopt it as the cheaper numerator. The contribution would then shift to the availability estimate.

Why not just filter shortcut questions with a text-only model? The traffic VideoQA audit already shows that binary filters discard valid samples and that continuous scores do better. Our estimator is a continuous score that also accounts for how much the image offers, and we test it against the continuous shortcut score head to head.

Decisive Checks

The proposal succeeds or fails on three results. The first is on the synthetic grid. The validated estimator's deficit must reach clearly higher rank correlation with the true deficit than Blind Gap, the imbalance ratio, and the attention ratio in the partially redundant cells, while all methods agree in the text-useless cells. The second concerns estimation accuracy. Input-level estimates of $I(Y;V\mid T)$ must stay within a small fraction of a bit of the true value in the low-rank configurations. We expect them to degrade in the 32-direction configurations, which tells us where real-data estimates can be trusted. The third is on POPE and MM-AU. At least one widely used fix must show a significantly smaller deficit reduction than its accuracy gain implies, confirmed by its effect on the "yes" rate. If none do, the deficit adds nothing to accuracy on these benchmarks. We would report that as a negative result on the value of information-based bias measurement.