Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignment

Introduction

Multimodal retrieval aims to identify relevant documents containing both text and images based on a user's query. Traditional approaches typically map these modalities into a shared latent space by maximizing their similarity. However, this methodology often overlooks a critical aspect of document composition: images are rarely mere visual mirrors of their accompanying text. Instead, they frequently provide complementary information—such as specific colors, spatial arrangements, or environmental details—that the text omits.

Expand

Source, p. 1

When a user asks a specific question like "What color is the bridge at night?", models that focus solely on semantic overlap (e.g., the word "bridge") may fail to retrieve the correct document if the textual portion does not explicitly state the color. To address this, we introduce CIEA (Complementary Information Extraction and Alignment), a framework designed to identify and preserve visual details that are absent from the text.

The CIEA Framework

The CIEA architecture consists of three primary stages designed to move beyond simple similarity matching:

1. Project-Based Encoding: The system utilizes a Transformer-based Language Model (LM) backbone alongside a frozen CLIP model for visual features. A linear projector maps CLIP image patches into the same embedding space as the LM's text tokens, allowing for initial cross-modal interaction.
2. Complementary Information Extractor: This core component calculates the distance between every image patch and every text token. It identifies patches that are most dissimilar to the textual content. An attention layer then re-weights these "different" patches, forcing the model to prioritize unique visual details that the text lacks.
3. Dual-Contrastive Optimization: The model is trained using two distinct losses:
4. Semantic Integrity ($L_c$): A standard contrastive loss ensuring the overall document (text + image) aligns with the query.
5. Complementary Alignment ($L_{comp}$): A novel "masked query" strategy. By masking parts of the query that correspond to the document's text, the model is forced to align the remaining query segments (the information missing from the text) specifically with the image representation.

Experimental Evaluation

CIEA was evaluated on two major benchmarks: WebQA, a large-scale open-domain multimodal QA dataset, and EDIS (Entity-Driven Image Search).

Expand

Source, p. 1

Key Results

CIEA consistently outperformed strong baselines, including the previous state-of-the-art model, MARVEL:

• WebQA-Multi: CIEA achieved an MRR@10 of 66.16 and NDCG@10 of 63.89, surpassing MARVEL (65.43 and 63.07, respectively).
• WebQA-Image: On the image-only subset, CIEA reached an MRR@10 of 67.40, demonstrating a significant advantage over standard project-based models.
• Ablation Studies: Removing either the Complementary Information Extractor or the Masked Query Loss led to a noticeable drop in performance, confirming that the focus on complementary data is the primary driver of the model's success.

Discussion and Insights

The success of CIEA highlights that similarity is a double-edged sword. While aligning similar features is necessary for basic retrieval, over-emphasizing it can "wash out" the unique details that make images valuable. By treating the query as a filter and modeling dissimilarity, CIEA effectively turns retrieval into a task where the image provides the missing evidence required to satisfy the user's information need.

Furthermore, the framework proved to be backbone agnostic, showing performance gains across various architectures, including T5 (encoder-decoder), BERT (encoder-only), and GPT-2 (decoder-only).

Limitations and Future Work

While CIEA represents a significant step forward, several areas remain for future exploration. The current study is limited to text queries and image-text documents; expanding this to include audio and video modalities is a promising direction. Additionally, while CIEA excels at finding the correct image, the downstream task of reasoning about that image to generate a final answer remains a separate challenge within the broader Retrieval-Augmented Generation (RAG) pipeline.

@inproceedings{zeng2025enhancing,
  author = {Delong Zeng and Yuexiang Xie and Yaliang Li and Ying Shen},
  title = {{Enhancing Multimodal Retrieval via Complementary Information Extraction and Alignment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}