Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models

Yuhao Cui, Xinxing Zu, Wenhua Zhang, Zhongzhou Zhao, Jinyang GaoCVPR 2025 · DOI

Introduction

While Large Language Models (LLMs) have revolutionized text representation, their multimodal counterparts—Multimodal LLMs (MLLMs)—have historically struggled to match the performance of specialized image-text retrieval models like the CLIP family. Despite their reasoning capabilities, MLLMs often lack the fine-grained alignment between vision and language required for high-precision retrieval. This paper introduces DeKR (Dense Knowledge Representation), a model that bridges this "alignment gap" by shifting training focus from sparse alt-text to dense, descriptive knowledge.

The Alignment Gap in Retrieval

The current landscape of multimodal retrieval is dominated by dual-encoder models trained on billions of noisy pairs. However, these models often struggle with complex reasoning or composed queries. Conversely, MLLMs typically face an alignment gap because they are trained on short, sparse captions (e.g., "a dog in a park"). This sparse knowledge fails to activate the model's understanding of specific attributes or spatial relationships, leading to "Action Errors" and "Attribute Errors" during retrieval tasks.

DeKR: Architecture and Methods

DeKR utilizes the Qwen2VL architecture as a backbone, employing a single, parameter-shared MLLM as a unified encoder. This allows images, videos, and text to be processed through the same transformer blocks, fostering a more integrated multimodal space.

Key technical features include:

  • Contrastive Learning with Hard Negatives: The model uses a modified contrastive loss that introduces difficult-to-distinguish samples to sharpen discriminative power.
  • Task-Specific Instructions: To prevent knowledge confusion, DeKR uses specific prompts for different tasks, such as "summarize the content" for representation and "Do they match?" for ranking.
  • Resolution Handling: By implementing a pooling operation in the projector, DeKR can handle input resolutions up to 560x560, improving its ability to capture fine-grained visual details.

The DeKon5M Dataset and Dense Knowledge

The core innovation of DeKR is the use of DeKon5M, a 5-million-sample dataset enriched with "Dense Knowledge" (DK). DK provides exhaustive descriptions of objects and actions, forcing the model to align fine-grained visual details with language.

Comparison of Sparse vs. Dense Knowledge and its distribution across datasets.

As illustrated in the figure above, Dense Knowledge is significantly more informative than standard captions—roughly 4x more dense than Flickr30k data. This density allows DeKR to achieve state-of-the-art results with 1,000x less training data than the CLIP series. Ablation studies suggest an optimal training mix of 80% dense knowledge and 20% sparse text to maintain generalizability to human search queries.

Experimental Results

DeKR achieves state-of-the-art performance across 14 retrieval and ranking benchmarks, including COCO-5K, Flickr-1K, and MSRVTT.

Radar chart showing DeKR performance vs. SoTA across various retrieval and ranking tasks.
  • Cross-Modal Retrieval: DeKR-7B achieved an average score of 87.1 on Image-Text retrieval, surpassing larger models like InternVL-C (13B).
  • Video and Composed Retrieval: The model outperformed InternVideo-L by 4.7 points in video tasks and demonstrated superior performance in multimodal composed retrieval (e.g., CIRR score of 33.7), where reasoning power is critical.

Conclusion and Future Work

DeKR demonstrates that unified MLLMs can become retrieval superstars by prioritizing knowledge density over data volume. While performance gains currently plateau at high resolutions due to data limitations, the model's efficiency and reasoning capabilities mark a significant step forward. Future work will explore higher resolution training and further refinement of composed modality tasks.

Summary updated

BibTeX
@inproceedings{cui2025incorporating,
  author = {Yuhao Cui and Xinxing Zu and Wenhua Zhang and Zhongzhou Zhao and Jinyang Gao},
  title = {{Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02768},
}
Figure