Improving Foundation Model Group Robustness with Auxiliary Sentence Embeddings

Sisuo Lyu, Hong Liu, Jie Li, Yan Teng, Yingchun WangTMLR 2025 · OpenReview

TL;DR

This paper introduces DoubleCCA, a novel framework to improve the robustness of vision-language foundation models (like CLIP) against group-based biases. By leveraging auxiliary sentence embeddings and a two-stage Canonical Correlation Analysis, DoubleCCA enriches text representations to create more invariant and fair models, outperforming existing methods.

What Problem Does This Paper Address?

Vision-language foundation models (VLMs), such as CLIP, are powerful tools for zero-shot image classification. However, they suffer from "group-based biases," meaning their performance can be significantly worse for certain demographic groups or underrepresented backgrounds. These biases arise from spurious correlations in the training data, where attributes correlated with the ground truth labels (like background or gender) are not truly relevant to the classification task. Existing debiasing methods often require extensive labeled data, complex architectures, or rely heavily on external LLMs for prompt engineering, limiting their generalization and efficiency. This paper aims to address this by developing a method that enhances group robustness without needing explicit group labels during training and is computationally efficient and easy to integrate.

Methods

The core of the proposed method, DoubleCCA, involves two main innovations:

  1. Auxiliary Sentence Embeddings: The authors hypothesize that CLIP's text encoder, optimized for vision-language alignment, has limitations in capturing rich semantic nuances. To compensate, they use an auxiliary sentence embedding model (e.g., Sentence-BERT, BART, HiT) to generate richer text representations. These auxiliary embeddings are designed to capture more diverse semantic perspectives, aiming to counteract biases.
  2. Two-Stage Canonical Correlation Analysis (CCA):
    • First CCA: This stage aligns the embeddings from CLIP's text encoder and the auxiliary sentence embedding model into a common, semantically correlated space. This helps to merge the complementary information from both sources.
  3. Second CCA: This stage reconstructs and merges these aligned representations, projecting them back into CLIP's original embedding space. This ensures compatibility with the visual features and further enhances group robustness by creating invariant features.

Additionally, they propose a simple sentence augmentation approach to generate more diverse sentence embeddings, which is crucial for stable CCA estimation, especially when dealing with datasets with few class descriptions. This augmentation involves combining original prompts with random sequences and LLM-generated contextual descriptions.

Datasets

The authors evaluate their method on two widely used datasets for group robustness evaluation:

  • Waterbirds: This dataset contains images of birds with associated background attributes (e.g., land, water).
  • CelebA: This dataset contains images of celebrities with associated attributes like gender, age, and hair color.

They also conduct experiments on five additional benchmark datasets to demonstrate generalization: CMNIST, FairFace, Counter Animal, ImageNet-A, and COCO-FP.

Key Insights and Ideas

  • CLIP's Text Encoder Limitations: The paper provides strong evidence (through t-SNE visualizations and quantitative bias scores) that CLIP's text encoder, while good for vision-language alignment, can produce representations that are sensitive to spurious correlations, leading to group biases.
  • Auxiliary Encoders as a Solution: Using dedicated sentence embedding models (like BERT, BART, HiT) that are optimized for semantic similarity tasks can generate more invariant representations, significantly reducing attribute bias compared to CLIP's own text encoder.
  • The Power of Two-Stage CCA: The two-stage CCA approach is key. The first stage aligns different semantic spaces, while the second stage merges them and projects them back into a vision-compatible space, effectively balancing semantic richness with visual discriminability.
  • Data Augmentation for Stability: For CCA to learn stable transformations, sufficient diverse data is needed. The proposed sentence augmentation strategy, combining random and LLM-generated text, effectively addresses this, especially for datasets with limited class descriptions.
  • Trade-offs are Expected: The authors acknowledge that improving group robustness can sometimes lead to a slight decrease in average accuracy, a common trade-off in fairness-aware machine learning. However, their method aims to achieve a good balance.

Key Results

  • Improved Group Robustness: DoubleCCA consistently outperforms the baseline CLIP model and other state-of-the-art methods (like PerceptionCLIP, WaffleCLIP, Oth-Cal) on Waterbirds and CelebA datasets in terms of worst-group accuracy and attribute bias reduction. For example, on Waterbirds with ViT-L/14, their method achieves a worst-group accuracy of 51.78% compared to CLIP's 44.64%.
  • Comparable Average Accuracy: While worst-group accuracy improves, average accuracy is maintained or only slightly decreased, indicating a good trade-off.
  • Generalization: The method shows significant improvements on robustness metrics across various challenging datasets like FairFace and Counter Animal, demonstrating its generalizability beyond Waterbirds/CelebA.
  • Efficiency: DoubleCCA introduces negligible additional computational overhead (time, memory, FLOPs) compared to standard CLIP inference, making it a practical solution.
  • Effectiveness of Auxiliary Models: Experiments show that using auxiliary sentence embedding models like HiT, Sentence-BERT, and GTE significantly improves performance and robustness compared to using CLIP alone. HiT, in particular, yields the best results.

Limitations and Future Work

  • Small Datasets and CCA Stability: The paper notes that with very few class descriptions (e.g., two classes), CCA can struggle to learn stable transformation matrices. Their data augmentation strategy helps mitigate this, but it's an area of ongoing research.
  • Trade-off Management: While DoubleCCA achieves a good balance, the inherent trade-off between average accuracy and worst-group robustness is acknowledged. Future work could explore theoretical foundations and black-box optimization schemes to further enhance this balance.
  • Exploring More Complex Scenarios: The authors suggest future work could investigate their approach in more complex settings like scene understanding, multi-label classification, and context-dependent recognition tasks.
  • Optimal Dimension Selection: The optimal dimension for the CCA subspace is discussed, with a recommendation of 64, but the authors suggest automatic dimension selection methods could be explored for future work.
  • Further Analysis of Sentence Embeddings: While they show different sentence embedding models impact performance, a deeper exploration of why certain models work better and their specific effects on group robustness is left for future work.

Summary updated

Abstract

This paper addresses the critical challenge of mitigating group-based biases in vision-language foundation models, a pressing issue for ensuring trustworthy AI deployment. We introduce DoubleCCA, a novel and computationally efficient framework that systematically enriches textual representations to enhance group robustness. Our key innovation is to leverage an auxiliary large sentence embedding model to capture diverse semantic perspectives, counteracting biased representations induced by limited training data. To this end, we propose a two-stage Canonical Correlation Analysis (DoubleCCA) technique: first, aligning augmented and original embeddings in a shared space; second, reconstructing invariant features to align with visual representations, thus enhancing the model's group robustness. We further propose a simple sentence augmentation approach, which aims to improve the robustness of CCA-induced subspaces. Our method is simple to implement and can be easily integrated into existing models, making it a practical solution for improving the robustness of vision-language foundation models to group-based biases. The experiments on a variety of datasets demonstrate that our method outperforms existing methods in terms of both performance and robustness. Our code is available at https://github.com/sisuolv/doublecca.
BibTeX
@article{lyu2025improving,
  author = {Sisuo Lyu and Hong Liu and Jie Li and Yan Teng and Yingchun Wang},
  title = {{Improving Foundation Model Group Robustness with Auxiliary Sentence Embeddings}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=5rMtiB96cg},
}
Figure