Abstract
Contrastive learning has become the standard for Vision-Language Pre-training (VLP), effectively aligning paired image-text embeddings. However, this paradigm often overlooks the global geometric structure of the embedding space, resulting in the "Affinity Inconsistency Problem" where inter-sample relationships are not preserved across modalities. We propose SaCo Loss (Sample-wise affinity Consistency), a regularizer that enforces consistent similarity structures between the visual and textual domains. Our method includes a pseudo-affinity mimicking strategy to stabilize early training. Experiments demonstrate that SaCo Loss significantly enhances zero-shot classification and cross-modal retrieval performance across multiple benchmarks, including ImageNet-1K and Flickr30K.
Introduction
The success of models like CLIP is largely attributed to contrastive learning, which pulls matched image-text pairs together while pushing unmatched ones apart. Despite its effectiveness, contrastive loss is inherently "local" — it focuses on individual pairs rather than the broader manifold structure. This leads to Affinity Inconsistency: two visually similar images (e.g., different instances of a "school bus") may have highly dissimilar text embeddings in the language space.
As illustrated in Figure 1(a), while standard contrastive loss aligns pairs, SaCo Loss ensures that the "social circle" or affinity of a sample is preserved across modalities. This geometric harmony leads to a more robust embedding space, as evidenced by the performance gains shown in Figure 1(b).
Methodology
SaCo Loss
The core of our approach is the SaCo Loss, an auxiliary objective designed to minimize the discrepancy between image-wise and text-wise similarity matrices. For a batch of $N$ pairs, we compute the image similarity matrix $S^I$ and the text similarity matrix $S^T$. The SaCo loss is defined as the L1 distance between these matrices: $$\mathcal{L}_{SaCo} = \frac{1}{N^2} \sum_{i,j} |S^I_{i,j} - S^T_{i,j}|$$ By minimizing this distance, the model is forced to align the inter-sample affinities of both modalities.
Pseudo-affinity Mimicking
Directly applying SaCo loss from scratch can be unstable due to the noisy nature of initial embeddings. To provide a stable geometric anchor, we introduce Pseudo-affinity Mimicking. During the initial training phase, the model is encouraged to mimic the affinities of a pre-trained "teacher" model (e.g., DINOv2 or a frozen CLIP). This provides a "warm-up" for the geometric structure before the model transitions to self-consistency.
Experimental Results
We evaluated SaCo Loss on standard VLP benchmarks using CC3M and YFCC15M for pre-training.
Affinity Consistency Analysis
A key metric for our method is the Pearson correlation between image and text affinities. As shown in Figure 3(a), baseline models often exhibit negative or near-zero affinity consistency. The addition of SaCo Loss dramatically improves this metric across various architectures (CLIP-R50, ViT, BLIP). Furthermore, Figure 3(b) shows that our proposed components—incremental loss weighting and pseudo-affinity mimicking—effectively reduce affinity disparity over the course of training.
Downstream Performance
The geometric improvements translate directly to better downstream performance:
- Zero-Shot Classification: On ImageNet-1K, a ViT-B/32 model saw a +6.4% improvement in Top-1 accuracy.
- Cross-Modal Retrieval: On Flickr30K, Recall@1 improved by 9.3% for image-to-text and 6.1% for text-to-image retrieval.
- Continuous Pre-training: Applying SaCo Loss to a converged OpenAI CLIP model for just a few epochs yielded a 1.0% gain on ImageNet and up to 8% on retrieval tasks.
Conclusion
SaCo Loss addresses a fundamental limitation of contrastive VLP by enforcing sample-wise affinity consistency. By acting as a global regularizer, it creates a more geometrically consistent embedding space that generalizes better to zero-shot and retrieval tasks. While it introduces a minor $O(N^2)$ computational overhead for similarity matrix calculation, the significant performance gains and the ability to enhance existing pre-trained models make it a highly effective tool for vision-language research.
Summary updated