GMM-Based Comprehensive Feature Extraction and Relative Distance Preservation For Few-Shot Cross-Modal Retrieval

Chengsong Sun, Weiping Li, Xiang Li, Yuankun Liu, Lianlei ShanarXiv 2025 · DOI · arXiv

Introduction

Cross-modal retrieval, such as finding relevant images from text queries, typically demands extensive labeled datasets. In few-shot scenarios—where only a few examples are available for novel categories—standard models often fail due to two primary biases: intra-modal and inter-modal. Intra-modal bias arises when models assume a "single-peak" distribution (one average vector per category), failing to capture the inherent diversity of a class. Inter-modal bias occurs when the visual and textual manifolds diverge in sparse data regimes, making alignment difficult.

The GCRDP Framework

To address these challenges, researchers introduced GCRDP (Gaussian-based Comprehensive feature extraction and Relative Distance Preservation). This framework employs a two-stage optimization strategy to refine feature representation and cross-modal alignment.

The GCRDP model overview illustrates the training pipeline using base image-text pairs and the two core modules: (a) Comprehensive Feature Extraction and (b) Relative Distance Preservation.

Comprehensive Feature Extraction (GC)

Unlike traditional methods that use Global Average Pooling (GAP) to produce a single prototype, GCRDP utilizes Gaussian Mixture Models (GMMs). This approach represents each category as a mixture of multiple "semantic units," capturing various aspects or poses of an object. By using the Expectation-Maximization (EM) algorithm and a multi-positive contrastive learning loss, the model ensures these semantic "peaks" remain distinct yet related.

A comparison between standard Global Average Pooling (top), which collapses features into a single peak, and the proposed GMM approach (bottom), which identifies multiple semantic regions and produces more granular attention maps.

Relative Distance Preservation (RDP)

The "secret sauce" of GCRDP is the RDP loss. Instead of forcing point-to-point identity between image and text vectors—which is often noisy in few-shot settings—RDP preserves the relative distances between samples across modalities. This treats alignment as a manifold matching problem: if two images are visually similar, their corresponding text descriptions should maintain a proportional proximity in the linguistic space, ensuring the structural integrity of the semantic space is preserved.

Experimental Evaluation

The model was benchmarked on four datasets: Wikipedia, Pascal Sentence, NUS-WIDE-10k, and a subset of the large-scale NUS-WIDE. Experiments spanned zero-shot and few-shot (1, 3, and 5-shot) configurations.

GCRDP consistently outperformed state-of-the-art methods like CLIP and FLEX-CLIP. In zero-shot tasks, it improved over FLEX-CLIP by 2.1 to 2.8 percentage points. In 3-shot scenarios—identified as a "sweet spot" for balancing distribution learning and overfitting—GCRDP surpassed CLIP by approximately 10 percentage points and FLEX-CLIP by an average of 3 percentage points. Qualitative results further showed the model's superior ability to handle fine-grained retrieval, such as distinguishing between specific aircraft types.

Conclusion

GCRDP demonstrates that moving beyond single-prototype representations and focusing on manifold structure significantly enhances few-shot cross-modal retrieval. While the iterative EM algorithm adds computational overhead, the gains in accuracy and robustness to distribution shifts make it a compelling approach for data-scarce environments. Future work may explore integrating these GMM-based techniques into larger foundation models.

Summary updated

BibTeX
@misc{sun2025gmm,
  author = {Chengsong Sun and Weiping Li and Xiang Li and Yuankun Liu and Lianlei Shan},
  title = {{GMM-Based Comprehensive Feature Extraction and Relative Distance Preservation For Few-Shot Cross-Modal Retrieval}},
  year = {2025},
  eprint = {2505.13306},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.13306},
}
Figure