Abstract
Existing Multimodal Large Language Models (MLLMs) often struggle to maintain semantic consistency across multiple images, particularly when those images feature vastly different contexts. This limitation stems from "Independent Perception," where visual features are extracted for each image in isolation before being fused by the language model. We present SAM (Semantic Alignment for MLLMs), an architecture that introduces bidirectional semantic guidance to align visual tokens across images during the perception phase. Supported by a novel 69K-sample synthetic dataset, MmLINK, SAM achieves significant performance gains in multi-image tasks such as visual storytelling and group captioning.
The Challenge of Independent Perception
Current state-of-the-art MLLMs, including GPT-4V and LLaVA, excel at reasoning over single images or pairs of highly similar images (e.g., "spot the difference" tasks). However, their performance degrades sharply when asked to analyze multiple images with significantly different contexts—such as tracking a specific character across various scenes in a narrative.
The root cause is the standard MLLM pipeline's reliance on independent feature extraction. When a model processes Image A and Image B separately, the visual encoder may prioritize different elements in each (e.g., a beach in one and a car in the other), potentially discarding the "linking information" (like a recurring person) as background noise. By the time these tokens reach the Large Language Model (LLM) for integrated analysis, the semantic misalignment is already present, making complex cross-image reasoning difficult.
Methods: Bidirectional Semantic Guidance
To address this, SAM replaces independent perception with a Bidirectional Semantic Guidance mechanism. This approach allows images to inform each other's feature extraction process before the tokens are passed to the LLM. The architecture consists of two primary interactive components:
- Assisted Visual Token Extraction (Q-former): A standard Q-former extracts initial visual tokens for the target image.
- Contextual Semantic Generation (W-former): A novel W-former analyzes the other images in the prompt (the context). It uses the initial tokens from the Q-former as a guide to identify relevant features in the contextual images. An Adaptive Adjustment module assigns weights to different image patches, effectively filtering out irrelevant background noise.
- The Interaction Loop: The refined "contextual semantics" from the W-former are fed back into the later layers of the Q-former. This allows the Q-former to update and align the final visual tokens for the target image based on the broader multi-image context.
To maintain training efficiency, the vision encoder and the LLM remain frozen. Only a small fraction of the model—approximately 4.3M parameters (0.05%)—is fine-tuned.
Data Strategy: The MmLINK Dataset
Because standard datasets often focus on similar image pairs, the authors developed MmLINK, a 69K-sample synthetic dataset designed to teach models object permanence across diverse backgrounds. The synthesis pipeline uses Segment Anything to isolate objects from source images, which are then placed into new masks. Stable-Diffusion-Inpainting, guided by ChatGPT, generates entirely new, contextually different backgrounds for these objects. This forces the model to learn to identify the same entity regardless of its surroundings.
Key Results and Insights
SAM demonstrates a substantial leap in performance across several multi-image benchmarks:
- Group Captioning: SAM outperformed all baselines, including proprietary models like GPT-4V and Gemini Pro. It achieved an average CIDEr score of 19.49, representing a 37% improvement over the leading open-source model, Cheetah (12.60).
- Visual Storytelling: The model achieved an average CIDEr score of 25.47, a 22% improvement over Cheetah (20.84).
- Change Captioning: Interestingly, SAM also improved performance on highly similar images (CIDEr 3.84 vs. GPT-4V's 3.38), suggesting that bidirectional guidance enhances fine-grained reasoning even when context doesn't shift dramatically.
Conclusion
The success of SAM highlights that early-stage semantic alignment is critical for multi-image reasoning. By forcing the visual tokenizer to consider contextual images, SAM preserves vital linking concepts that independent perception typically discards. Future work may involve extending this bidirectional guidance to video processing, where context shifts are inherent and continuous.
Summary updated