Introduction
Traditional image retrieval models are typically trained using metric learning objectives, such as triplet or contrastive losses, which focus on clustering similar images in an embedding space. A significant limitation of these approaches is the decoupling of the training phase from the actual retrieval task; the model rarely interacts with the target database during training. Furthermore, self-supervised methods often suffer from representation collapse, where the encoder maps diverse inputs to a single point.
To address these challenges, we present DMCAC (Divergence Minimization with Cross-Attention Classification). This framework bridges the gap between training and inference by incorporating the database directly into the training loop. By minimizing the divergence between similarity distributions of augmented query views and employing cross-attention for classification, DMCAC achieves state-of-the-art performance across major benchmarks while preventing representation collapse.
The DMCAC Framework
The DMCAC architecture consists of three primary components designed to create a robust and semantically meaningful embedding space.
1. Joint Query-Database Learning
Unlike standard methods, DMCAC utilizes both a query set and a database set simultaneously during training. For a given query, multiple augmented views (e.g., different crops or color jitters) are passed through an encoder. The model then computes a similarity distribution for these views against the database. This allows the model to learn the query's relationship to the entire database distribution rather than just a few random negatives in a minibatch.
2. Divergence Minimization via Frobenius Loss
Instead of simple point-wise distance minimization, DMCAC minimizes the Jensen-Shannon (JS) Divergence between the similarity distributions of different views of the same query. This ensures that various augmentations of an image "perceive" the database in a consistent manner. The divergence is optimized using a Frobenius loss, providing a stronger training signal for retrieval than traditional metric learning.
3. Cross-Attention Classification (CAC)
To prevent representation collapse, the model is tasked with predicting the query's class label by attending to retrieved database embeddings. This Cross-Attention Classification (CAC) mechanism forces the retrieved items to carry semantic relevance to the query. By grounding the embedding space in a classification task, the model ensures that the learned representations remain distinct and informative.
Experimental Results
The DMCAC framework was evaluated on four standard fine-grained image retrieval benchmarks: CUB-200, In-Shop Clothes Retrieval, Cars-196, and Stanford Online Products (SOP). We utilized Vision Transformer (ViT-S) and DeiT-S backbones for our experiments.
As shown in the results table, DMCAC achieves state-of-the-art (SOTA) or highly competitive performance across all datasets. Notably, on the In-Shop dataset, DMCAC reached a Recall@1 of 92.7%, surpassing previous SOTA methods like Hyp-DINO. On CUB-200, the model achieved 86.2% Recall@1, representing a significant improvement over other ViT-based approaches.
Furthermore, the framework demonstrates excellent scalability. Experiments with "Approximate Retrieval"—using only the top-k neighbors during training via FAISS—showed performance nearly identical to full exhaustive retrieval. This confirms that DMCAC is viable for large-scale industrial applications where memory and computational efficiency are paramount.
Conclusion and Future Work
DMCAC introduces a paradigm shift in image retrieval by treating the database as an active participant in the training process. The combination of distribution matching and cross-attention grounding effectively mitigates representation collapse and improves retrieval accuracy.
While the method is highly effective, it does require maintaining database embeddings, which are updated periodically to manage computational overhead. Future research will explore applying this joint learning approach to even larger industrial databases and extending the framework to cross-modal retrieval tasks, such as text-to-image search.
Summary updated