Modern Large Multimodal Models (LMMs) excel at describing individual photographs, answering complex queries about single scenes, and grounding text in visual tokens. However, this proficiency masks a fundamental cognitive bottleneck: when asked to reason across multiple images, their performance drops precipitously.
As demonstrated by Mehran Kazemi et al. in their work on the ReMI benchmark (ReMI: A Dataset for Reasoning with Multiple Images), a massive performance gap exists between humans and frontier AI models when tasks require synthesizing information distributed across separate visual sources, such as diagrams, maps, or sequential frames. This limitation is not merely a matter of model scale. Fei Wang et al. found that training models exclusively on massive single-image datasets does not lead to the emergence of multi-image reasoning capabilities (MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding). Instead, models perform near random chance on tasks requiring cross-image logic, proving that multi-image synthesis is a distinct cognitive skill that requires explicit architectural or data-driven interventions.
The Root Cause: Single-Image Bias and Attention Dilution
Why do models that effortlessly parse a single complex image fail when that same information is split across two or more images? The problem lies in how modern vision-language architectures handle multi-image inputs.
Most current models are "single-image natives." They project visual features into a continuous sequence of tokens and append them to the text prompt. When multiple images are introduced, the model treats them as a long, interleaved sequence. As noted in a controlled analysis of the MIMIC benchmark (More Images, More Problems? A Controlled Analysis of VLM Failure Modes), instead of actively comparing the images, the model behaves like a single-image model overwhelmed by a long context window.
Standard Single-Image Processing:
[Image Tokens] -> [Text Tokens] -> High-Fidelity Grounding
Naive Multi-Image Processing (Interleaved):
[Image 1 Tokens] -> [Image 2 Tokens] -> [Text Tokens] -> Attention Dilution & Hallucination
This architectural setup causes several distinct failure modes:
- Attention Dilution and Hallucination: In a standard self-attention layer, tokens from Image A attend to tokens from Image B uniformly. Without a structured framework to map inter-image dependencies, models fail to perform fine-grained comparisons. Jiale Li et al. identify this in MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models, noting that models frequently suffer from "multi-image hallucinations," failing to maintain object identity or track states across sequences.
- Lack of Comparative Mechanisms: Standard perception architectures process images in isolation. As argued by Rujie Wu et al. in Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World, true concept induction requires a model's perception of one image to be actively informed by the broader context of the entire set. Without explicit cross-image attention, the model cannot build a contrastive strategy to isolate shared abstract traits.
The Dimensions of Cross-Image Reasoning
To resolve this gap, researchers categorize multi-image reasoning along several distinct axes, each presenting unique structural challenges:
Comparative and Contrastive Logic
This requires identifying subtle differences or commonalities in object counts, positions, or attributes across distinct frames. Early foundations like NLVR2 (A Corpus for Reasoning about Natural Language Grounded in Photographs by Alane Suhr et al.) forced models to evaluate a single caption against a pair of distinct images, testing cardinality and spatial relationships.
Temporal and Sequential Continuity
Tracking changes over time is critical for specialized domains. Examples include observing disease progression in longitudinal medical imaging (Heejun Shin et al.'s Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays) or following a narrative sequence across a slide deck (SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images by Kyosuke Nishida et al.).
Contextual Dependency
In benchmarks like MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts, tasks are split into independent versus mutually dependent images. When the context of one image strictly depends on another, standard models experience a sharp performance drop because they lack native cross-image attention mechanisms capable of holistic spatial reasoning.
Emerging Strategies to Bridge the Gap
The research community is actively developing techniques to transition models from single-image processors to multi-image reasoners. These strategies broadly fall into data engineering and architectural modifications.
Interleaved Pre-training and Instruction Tuning
The most direct way to foster cross-image reasoning is to expose models to data where text and multiple images are natively intertwined. Anas Awadalla et al. pioneered this at scale with Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text, providing the long-form documents necessary for in-context learning.
Building on this, Andrew Tao et al. demonstrated in VILA: On Pre-training for Visual Language Models that unfreezing the LLM during pre-training on interleaved datasets is a critical requirement for unlocking emergent multi-image capabilities. To refine granular comparison skills, Muhammad Jehanzeb Mirza et al. introduced Comparison Visual Instruction Tuning, explicitly training models on image pairs to distinguish subtle variations in actions and positions.
Attention Engineering and Preference Optimization
Rather than relying solely on data scaling, researchers are modifying how models distribute attention across image sets. For instance, the authors of the MIMIC benchmark proposed a layer-wise attention-masking technique to prevent visual tokens from overwhelming the context window.
On the optimization side, Ke Li developed a hierarchical preference optimization framework called CcDPO (Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs). CcDPO trains models to first grasp the global context across an image sequence before "zooming in" on fine-grained visual cues, drastically reducing multi-image hallucinations and forcing the model to build explicit relational links between distinct visual inputs.
Key Foundational Work
A Corpus for Reasoning about Natural Language Grounded in Photographs (2019) — contributed 3 concept elements to this direction.