Introduction
Cross-modal retrieval, such as finding relevant images from text queries, typically demands extensive labeled datasets. In few-shot scenarios—where only a few examples are available for novel categories—standard models often fail due to two primary biases: intra-modal and inter-modal. Intra-modal bias arises when models assume a "single-peak" distribution (one average vector per category), failing to capture the inherent diversity of a class. Inter-modal bias occurs when the visual and textual manifolds diverge in sparse data regimes, making alignment difficult.
The GCRDP Framework
To address these challenges, researchers introduced GCRDP (Gaussian-based Comprehensive feature extraction and Relative Distance Preservation). This framework employs a two-stage optimization strategy to refine feature representation and cross-modal alignment.
Comprehensive Feature Extraction (GC)
Unlike traditional methods that use Global Average Pooling (GAP) to produce a single prototype, GCRDP utilizes Gaussian Mixture Models (GMMs). This approach represents each category as a mixture of multiple "semantic units," capturing various aspects or poses of an object. By using the Expectation-Maximization (EM) algorithm and a multi-positive contrastive learning loss, the model ensures these semantic "peaks" remain distinct yet related.
Relative Distance Preservation (RDP)
The "secret sauce" of GCRDP is the RDP loss. Instead of forcing point-to-point identity between image and text vectors—which is often noisy in few-shot settings—RDP preserves the relative distances between samples across modalities. This treats alignment as a manifold matching problem: if two images are visually similar, their corresponding text descriptions should maintain a proportional proximity in the linguistic space, ensuring the structural integrity of the semantic space is preserved.
Experimental Evaluation
The model was benchmarked on four datasets: Wikipedia, Pascal Sentence, NUS-WIDE-10k, and a subset of the large-scale NUS-WIDE. Experiments spanned zero-shot and few-shot (1, 3, and 5-shot) configurations.
GCRDP consistently outperformed state-of-the-art methods like CLIP and FLEX-CLIP. In zero-shot tasks, it improved over FLEX-CLIP by 2.1 to 2.8 percentage points. In 3-shot scenarios—identified as a "sweet spot" for balancing distribution learning and overfitting—GCRDP surpassed CLIP by approximately 10 percentage points and FLEX-CLIP by an average of 3 percentage points. Qualitative results further showed the model's superior ability to handle fine-grained retrieval, such as distinguishing between specific aircraft types.
Conclusion
GCRDP demonstrates that moving beyond single-prototype representations and focusing on manifold structure significantly enhances few-shot cross-modal retrieval. While the iterative EM algorithm adds computational overhead, the gains in accuracy and robustness to distribution shifts make it a compelling approach for data-scarce environments. Future work may explore integrating these GMM-based techniques into larger foundation models.
Summary updated