SaCo Loss: Sample-Wise Affinity Consistency for Vision-Language Pre-Training

Sitong Wu, Haoru Tan, Zhuotao Tian, Yukang Chen, Xiaojuan Qi, Jiaya Jia

Abstract

Contrastive learning has become the standard for Vision-Language Pre-training (VLP), effectively aligning paired image-text embeddings. However, this paradigm often overlooks the global geometric structure of the embedding space, resulting in the "Affinity Inconsistency Problem" where inter-sample relationships are not preserved across modalities. We propose SaCo Loss (Sample-wise affinity Consistency), a regularizer that enforces consistent similarity structures between the visual and textual domains. Our method includes a pseudo-affinity mimicking strategy to stabilize early training. Experiments demonstrate that SaCo Loss significantly enhances zero-shot classification and cross-modal retrieval performance across multiple benchmarks, including ImageNet-1K and Flickr30K.

Introduction

The success of models like CLIP is largely attributed to contrastive learning, which pulls matched image-text pairs together while pushing unmatched ones apart. Despite its effectiveness, contrastive loss is inherently "local" — it focuses on individual pairs rather than the broader manifold structure. This leads to Affinity Inconsistency: two visually similar images (e.g., different instances of a "school bus") may have highly dissimilar text embeddings in the language space.

Conceptual overview of SaCo Loss and its performance impact across various tasks.

As illustrated in Figure 1(a), while standard contrastive loss aligns pairs, SaCo Loss ensures that the "social circle" or affinity of a sample is preserved across modalities. This geometric harmony leads to a more robust embedding space, as evidenced by the performance gains shown in Figure 1(b).

Methodology

SaCo Loss

The core of our approach is the SaCo Loss, an auxiliary objective designed to minimize the discrepancy between image-wise and text-wise similarity matrices. For a batch of $N$ pairs, we compute the image similarity matrix $S^I$ and the text similarity matrix $S^T$. The SaCo loss is defined as the L1 distance between these matrices: $$\mathcal{L}_{SaCo} = \frac{1}{N^2} \sum_{i,j} |S^I_{i,j} - S^T_{i,j}|$$ By minimizing this distance, the model is forced to align the inter-sample affinities of both modalities.

Pseudo-affinity Mimicking

Directly applying SaCo loss from scratch can be unstable due to the noisy nature of initial embeddings. To provide a stable geometric anchor, we introduce Pseudo-affinity Mimicking. During the initial training phase, the model is encouraged to mimic the affinities of a pre-trained "teacher" model (e.g., DINOv2 or a frozen CLIP). This provides a "warm-up" for the geometric structure before the model transitions to self-consistency.

Experimental Results

We evaluated SaCo Loss on standard VLP benchmarks using CC3M and YFCC15M for pre-training.

Affinity Consistency Analysis

A key metric for our method is the Pearson correlation between image and text affinities. As shown in Figure 3(a), baseline models often exhibit negative or near-zero affinity consistency. The addition of SaCo Loss dramatically improves this metric across various architectures (CLIP-R50, ViT, BLIP). Furthermore, Figure 3(b) shows that our proposed components—incremental loss weighting and pseudo-affinity mimicking—effectively reduce affinity disparity over the course of training.

Quantitative analysis of affinity consistency (Pearson correlation) and disparity across different models and training epochs.

Downstream Performance

The geometric improvements translate directly to better downstream performance:

  • Zero-Shot Classification: On ImageNet-1K, a ViT-B/32 model saw a +6.4% improvement in Top-1 accuracy.
  • Cross-Modal Retrieval: On Flickr30K, Recall@1 improved by 9.3% for image-to-text and 6.1% for text-to-image retrieval.
  • Continuous Pre-training: Applying SaCo Loss to a converged OpenAI CLIP model for just a few epochs yielded a 1.0% gain on ImageNet and up to 8% on retrieval tasks.

Conclusion

SaCo Loss addresses a fundamental limitation of contrastive VLP by enforcing sample-wise affinity consistency. By acting as a global regularizer, it creates a more geometrically consistent embedding space that generalizes better to zero-shot and retrieval tasks. While it introduces a minor $O(N^2)$ computational overhead for similarity matrix calculation, the significant performance gains and the ability to enhance existing pre-trained models make it a highly effective tool for vision-language research.

Summary updated

Abstract

Vision-language pre-training (VLP) aims to learn joint representations of vision and language modalities. The contrastive paradigm is currently dominant in this field. However, we observe a notable misalignment phenomenon, that is, the affinity between samples has an obvious disparity across different modalities, namely “Affinity Inconsistency Problem”. Our intuition is that, for a well-aligned model, two images that look similar to each other should have the same level of similarity as their corresponding texts that describe them. In this paper, we first investigate the reason of this inconsistency problem. We discover that the lack of consideration for sample-wise affinity consistency across modalities in existing training objectives is the central cause. To address this problem, we propose a novel loss function, named Sample-wise affinity Consistency (SaCo) loss, which is designed to enhance such consistency by minimizing the distance between image embedding similarity and text embedding similarity for any two samples. Our SaCo loss can be easily incorporated into existing vision-language models as an additional loss due to its complementarity for most training objectives. In addition, considering that pre-training from scratch is computationally expensive, we also provide a more efficient way to continuously pre-train on a converged model by integrating our loss. Experimentally, the model trained with our SaCo loss significantly outperforms the baseline on a variety of vision and language tasks.
BibTeX
@inproceedings{wu2024saco,
  author = {Sitong Wu and Haoru Tan and Zhuotao Tian and Yukang Chen and Xiaojuan Qi and Jiaya Jia},
  title = {{SaCo Loss: Sample-Wise Affinity Consistency for Vision-Language Pre-Training}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02583},
}
Figure