How Many Bits of a Demonstration Does Multimodal In-Context Learning Use?

Multimodal models can learn a task "in context": they read a few worked examples, each an image with a question and answer, followed by a new question, and they answer it without any weight updates. Which examples you show matters a great deal. The standard recipe picks them by comparing images first and question text second. That recipe quietly assumes every stored example image is pristine. Real example banks are often not. They sit compressed on edge devices, travel over lossy links, or come from degraded sensors.

This proposal replaces the all-or-nothing tests used to date, which either keep an image or delete it, with a graded one. We push demonstration images through a channel whose capacity we control, measured in bits per pixel, and we apply it separately at the two places the images are used: when examples are retrieved and when the model reads them. The result is a pair of curves showing how much image information each stage actually consumes. Those curves answer an open mechanistic question: is the known finding that models ignore demonstration images an artifact of one architecture? They also yield a simple, testable retrieval fix for degraded example banks.

Core Claim

Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning? blanked out images in VQAv2, OK-VQA and GQA and reported a "blind demo" asymmetry. Removing demonstration images barely changes accuracy, while removing the query image is catastrophic. Even so, using images to select demonstrations helps a lot: their two-stage selector (MMICES, which filters by image similarity and then reranks by text) at 4 shots beat random selection at 32 shots on GQA. The authors attribute the blindness to masked cross-attention, in which question text cannot attend to earlier demonstration images.

We claim that this asymmetry is best described quantitatively, as two separate information budgets. For each place $\ell$ where a degradation can be applied, we define its used-bits threshold:

$$R^*_\ell = \min \{ R : \mathrm{Acc}_\ell(R) \ge \mathrm{Acc}_{\text{clean}} - \delta \}$$

Here $R$ is the bits per pixel allowed through the channel, $\mathrm{Acc}_\ell(R)$ is task accuracy when only location $\ell$ is degraded, and $\delta$ is a fixed tolerance of one accuracy point. The four locations are the bank images at retrieval, the query image at retrieval, the demonstration images at inference, and the query image at inference.

We make three predictions:

  1. Retrieval needs only a coarse, low-rate signal.
  2. Inference-time demonstration images have $R^*$ near zero for masked cross-attention models.
  3. Whether that threshold stays near zero for models with full attention over interleaved images and text decides whether "blind demo" is architectural or a deeper preference for text.

There is also a practical consequence. When the bank is degraded, the retrieval distances computed by MMICES are miscalibrated, because clean queries are being compared against corrupted candidates. Passing the query through the same channel before embedding it should restore calibrated rankings. We call this channel-matched retrieval.

Relation to Prior Work

The motivation comes from Chen et al. above. We keep their tasks and their two-stage selector. We differ in three ways: we use a continuous rate instead of blank squares, we separate the retrieval stage from the inference stage, and we test the masking explanation directly on a second architecture family. Their paper leaves robustness of selection to a degraded bank unexamined.

The information-theory work plays the role of a lens, not a source of theorems.

We do not claim capacity bounds. Bits per pixel is used as a model-agnostic unit of measurement.

The ablation literature constrains our design. From Accuracy to Visual Dependence filters out VideoQA questions that text-only models can solve. Detecting Multimodal Situations with Insufficient Context shows that VQAv2, OK-VQA, A-OKVQA and GQA contain many questions whose answers do not depend on the image. Together these imply that flat curves on raw benchmarks could simply reflect text shortcuts. We therefore report every curve on an image-dependent subset as well. Why Foundation Models Struggle with Cross-Modal Context finds that attention imbalance drives conflict detection from 90% in one modality down to as low as 3% across modalities. That result suggests text dominance may persist even without masking. It is the prior our third prediction tests against.

Research Program

Channels. We build a degradation harness with three channel families:

  • a lossy image codec swept across bits-per-pixel levels, to model rate limits;
  • random patch erasure at controlled fractions, to model packet loss;
  • additive Gaussian noise at controlled signal-to-noise ratios, to model a noisy sensor.

Codec rate gives the primary $R$ axis. The other two families test whether conclusions depend on the channel type.

Models. We apply each channel independently at each of the four locations. We do this for an OpenFlamingo-style masked cross-attention model and for an interleaved full-attention multimodal model, at matched shot counts (4, 8 and 16).

Retrieval analysis. For retrieval we also record how much the selected demonstration set overlaps with the set chosen from a clean bank (Jaccard overlap, the size of the intersection divided by the size of the union). This separates "selection changed" from "accuracy changed."

Channel-matched retrieval. The method component is deliberately small. When the bank has passed through a known or estimated channel, the query image goes through the same channel before its image embedding is computed, and MMICES then runs unchanged. When the channel is unknown, we estimate its rate from simple codec statistics of the bank images.

Supporting probe. To tie the macroscopic curves to mechanism, we measure attention mass from answer tokens onto demonstration image tokens as a function of $R$ in the full-attention model. We treat this only as a supporting signal, since the research file notes that attention proxies can be confounded.

Baselines

No selection. The reference point is random selection of demonstrations from the same degraded bank, at each shot count. This is the floor any selector must clear.

Text-only retrieval. This selects demonstrations using a dense text embedding of the question alone. It needs no image bits at retrieval, so it is the natural competitor once the bank is degraded. Beating it at a given rate is the bar for claiming that image information still helps selection at that rate.

MMICES. The strongest existing method is MMICES from Chen et al., run on the degraded bank with a clean query embedding, exactly as it would be deployed today. The file reports only the qualitative result that 4-shot MMICES beats 32-shot random selection on GQA. There is no reported result found on Lacuna for MMICES absolute accuracy on VQAv2, OK-VQA or GQA, so we re-run it ourselves. We use the same image encoder, text encoder, candidate pool size $K$ and shot count $N$ as our method, and sweep $K$ over the same grid for both. Channel-matched retrieval counts as beating MMICES if it gains at least one accuracy point, averaged over three seeds, at rates at or below the MMICES retrieval threshold $R^*$, while staying within noise of MMICES on a clean bank.

Restore, then retrieve. An off-the-shelf image restoration model denoises or deblocks bank images before MMICES runs. It gets the same evaluation budget.

Ablation of the key ingredient. We apply the same degradation to the query at inference only, not at retrieval. This separates the benefit of matched distances from any general effect of a degraded query.

Why not just…?

Why not just retrieve by text only, since models ignore demonstration images anyway? This is a real threat, and at low enough rates it should win. Chen et al. show that image-based filtering beats text-only selection on a clean bank, but nothing says it survives a corrupted one. The experiments report the crossover rate at which MMICES and channel-matched retrieval fall below text-only retrieval. If channel-matched retrieval never moves that crossover toward lower rates, the method claim fails. The measurement result would still stand.

Why not just use standard corruption suites and call it robustness testing? Named corruptions such as blur or JPEG artifacts are included as channel families. Their weakness is that they share no common axis, so "how much signal was used" cannot be compared across models, locations or corruption types. The codec rate supplies that axis. We check whether the thresholds $R^*$ are consistent across the three channel families. If they are not, the rate framing is only a convenience and we will say so.

Why not just restore degraded images before retrieval? Restoration is included as a baseline and may perform well at moderate noise. At low rates, however, restoration invents detail, which can mislead nearest-neighbor search. Matched degradation instead discards the same information from both sides of the comparison. Whichever approach produces higher accuracy and higher overlap with the clean-bank selection settles the question.

Decisive Checks

We run everything on VQAv2, OK-VQA and GQA, each also restricted to an image-dependent subset. The subset keeps questions that a text-only pass of the same model answers incorrectly, following Korkut et al..

Four outcomes decide the proposal:

  • If the inference-time demonstration threshold stays near zero for the full-attention model on the image-dependent subset, "blind demo" is not a masking artifact, and the masking explanation in Chen et al. needs revision. If that threshold rises clearly above zero only for the full-attention model, masking is confirmed as the cause.
  • If the retrieval threshold for bank images is far below the inference threshold for the query image, image banks for in-context learning can be stored at a small fraction of their size with no loss. That is a concrete deployment result.
  • If channel-matched retrieval beats MMICES on degraded banks at matched budgets, the miscalibration diagnosis is supported.
  • If it fails to beat text-only retrieval anywhere below the MMICES retrieval threshold, the method claim is dropped in favor of the simpler recommendation to fall back to text-only retrieval below a measured rate.