Introduction
While Large Language Models (LLMs) have revolutionized text representation, their multimodal counterparts—Multimodal LLMs (MLLMs)—have historically struggled to match the performance of specialized image-text retrieval models like the CLIP family. Despite their reasoning capabilities, MLLMs often lack the fine-grained alignment between vision and language required for high-precision retrieval. This paper introduces DeKR (Dense Knowledge Representation), a model that bridges this "alignment gap" by shifting training focus from sparse alt-text to dense, descriptive knowledge.
The Alignment Gap in Retrieval
The current landscape of multimodal retrieval is dominated by dual-encoder models trained on billions of noisy pairs. However, these models often struggle with complex reasoning or composed queries. Conversely, MLLMs typically face an alignment gap because they are trained on short, sparse captions (e.g., "a dog in a park"). This sparse knowledge fails to activate the model's understanding of specific attributes or spatial relationships, leading to "Action Errors" and "Attribute Errors" during retrieval tasks.
DeKR: Architecture and Methods
DeKR utilizes the Qwen2VL architecture as a backbone, employing a single, parameter-shared MLLM as a unified encoder. This allows images, videos, and text to be processed through the same transformer blocks, fostering a more integrated multimodal space.
Key technical features include:
- Contrastive Learning with Hard Negatives: The model uses a modified contrastive loss that introduces difficult-to-distinguish samples to sharpen discriminative power.
- Task-Specific Instructions: To prevent knowledge confusion, DeKR uses specific prompts for different tasks, such as "summarize the content" for representation and "Do they match?" for ranking.
- Resolution Handling: By implementing a pooling operation in the projector, DeKR can handle input resolutions up to 560x560, improving its ability to capture fine-grained visual details.
The DeKon5M Dataset and Dense Knowledge
The core innovation of DeKR is the use of DeKon5M, a 5-million-sample dataset enriched with "Dense Knowledge" (DK). DK provides exhaustive descriptions of objects and actions, forcing the model to align fine-grained visual details with language.
As illustrated in the figure above, Dense Knowledge is significantly more informative than standard captions—roughly 4x more dense than Flickr30k data. This density allows DeKR to achieve state-of-the-art results with 1,000x less training data than the CLIP series. Ablation studies suggest an optimal training mix of 80% dense knowledge and 20% sparse text to maintain generalizability to human search queries.
Experimental Results
DeKR achieves state-of-the-art performance across 14 retrieval and ranking benchmarks, including COCO-5K, Flickr-1K, and MSRVTT.
- Cross-Modal Retrieval: DeKR-7B achieved an average score of 87.1 on Image-Text retrieval, surpassing larger models like InternVL-C (13B).
- Video and Composed Retrieval: The model outperformed InternVideo-L by 4.7 points in video tasks and demonstrated superior performance in multimodal composed retrieval (e.g., CIRR score of 33.7), where reasoning power is critical.
Conclusion and Future Work
DeKR demonstrates that unified MLLMs can become retrieval superstars by prioritizing knowledge density over data volume. While performance gains currently plateau at high resolutions due to data limitations, the model's efficiency and reasoning capabilities mark a significant step forward. Future work will explore higher resolution training and further refinement of composed modality tasks.
Summary updated