Semantic Alignment for Multimodal Large Language Models

Tao Wu, Mengze Li, Jingyuan Chen, Wei Ji, Wang Lin, Jinyang Gao, Kun Kuang, Zhou Zhao, Fei Wu

Abstract

Existing Multimodal Large Language Models (MLLMs) often struggle to maintain semantic consistency across multiple images, particularly when those images feature vastly different contexts. This limitation stems from "Independent Perception," where visual features are extracted for each image in isolation before being fused by the language model. We present SAM (Semantic Alignment for MLLMs), an architecture that introduces bidirectional semantic guidance to align visual tokens across images during the perception phase. Supported by a novel 69K-sample synthetic dataset, MmLINK, SAM achieves significant performance gains in multi-image tasks such as visual storytelling and group captioning.

The Challenge of Independent Perception

Current state-of-the-art MLLMs, including GPT-4V and LLaVA, excel at reasoning over single images or pairs of highly similar images (e.g., "spot the difference" tasks). However, their performance degrades sharply when asked to analyze multiple images with significantly different contexts—such as tracking a specific character across various scenes in a narrative.

Figure 1: Current MLLMs struggle with contextually different images compared to highly similar ones.

The root cause is the standard MLLM pipeline's reliance on independent feature extraction. When a model processes Image A and Image B separately, the visual encoder may prioritize different elements in each (e.g., a beach in one and a car in the other), potentially discarding the "linking information" (like a recurring person) as background noise. By the time these tokens reach the Large Language Model (LLM) for integrated analysis, the semantic misalignment is already present, making complex cross-image reasoning difficult.

Methods: Bidirectional Semantic Guidance

To address this, SAM replaces independent perception with a Bidirectional Semantic Guidance mechanism. This approach allows images to inform each other's feature extraction process before the tokens are passed to the LLM. The architecture consists of two primary interactive components:

  1. Assisted Visual Token Extraction (Q-former): A standard Q-former extracts initial visual tokens for the target image.
  2. Contextual Semantic Generation (W-former): A novel W-former analyzes the other images in the prompt (the context). It uses the initial tokens from the Q-former as a guide to identify relevant features in the contextual images. An Adaptive Adjustment module assigns weights to different image patches, effectively filtering out irrelevant background noise.
  3. The Interaction Loop: The refined "contextual semantics" from the W-former are fed back into the later layers of the Q-former. This allows the Q-former to update and align the final visual tokens for the target image based on the broader multi-image context.

To maintain training efficiency, the vision encoder and the LLM remain frozen. Only a small fraction of the model—approximately 4.3M parameters (0.05%)—is fine-tuned.

Because standard datasets often focus on similar image pairs, the authors developed MmLINK, a 69K-sample synthetic dataset designed to teach models object permanence across diverse backgrounds. The synthesis pipeline uses Segment Anything to isolate objects from source images, which are then placed into new masks. Stable-Diffusion-Inpainting, guided by ChatGPT, generates entirely new, contextually different backgrounds for these objects. This forces the model to learn to identify the same entity regardless of its surroundings.

Key Results and Insights

SAM demonstrates a substantial leap in performance across several multi-image benchmarks:

  • Group Captioning: SAM outperformed all baselines, including proprietary models like GPT-4V and Gemini Pro. It achieved an average CIDEr score of 19.49, representing a 37% improvement over the leading open-source model, Cheetah (12.60).
  • Visual Storytelling: The model achieved an average CIDEr score of 25.47, a 22% improvement over Cheetah (20.84).
  • Change Captioning: Interestingly, SAM also improved performance on highly similar images (CIDEr 3.84 vs. GPT-4V's 3.38), suggesting that bidirectional guidance enhances fine-grained reasoning even when context doesn't shift dramatically.

Conclusion

The success of SAM highlights that early-stage semantic alignment is critical for multi-image reasoning. By forcing the visual tokenizer to consider contextual images, SAM preserves vital linking concepts that independent perception typically discards. Future work may involve extending this bidirectional guidance to video processing, where context shifts are inherent and continuous.

Summary updated

Abstract

Research on Multi-modal Large Language Models (MLLMs) towards the multi-image cross-modal instruction has received increasing attention and made significant progress, particularly in scenarios involving closely resembling images (e.g., change captioning). Existing MLLMs typically follow a two-step process in their pipelines: first, extracting visual tokens independently for each input image, and then aligning these visual tokens from different images with the Large Language Model (LLM) in its textual feature space. However, the independent extraction of visual tokens for each image may result in different semantics being prioritized for different images in the first step, leading to a lack of preservation of linking information among images for subsequent LLM analysis. This issue becomes more serious in scenarios where significant variations exist among the images (e.g., visual storytelling). To address this challenge, we introduce Semantic Alignment for Multi-modal large language models (SAM). By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis and align the semantics of different images before feeding them into LLM. As the test bed, we propose a large-scale dataset named MmLINK consisting of 69K samples. Different from most existing datasets for MLLMs fine-tuning, our MmLINK dataset comprises multi-modal instructions with significantly diverse images. Extensive experiments on the group captioning task and the storytelling task prove the effectiveness of our SAM model, surpassing the state-of-the-art methods by a large margin (+37% for group captioning and +22% for storytelling on CIDEr score). Project page: https://anonymous.4open.science/r/SAM-F596.
BibTeX
@inproceedings{wu2024semantic,
  author = {Tao Wu and Mengze Li and Jingyuan Chen and Wei Ji and Wang Lin and Jinyang Gao and Kun Kuang and Zhou Zhao and Fei Wu},
  title = {{Semantic Alignment for Multimodal Large Language Models}},
  booktitle = {ACM Multimedia},
  year = {2024},
}
Figure