Abstract
Deep learning models frequently rely on spurious correlations—non-essential associations between labels and contexts—which leads to poor generalization on out-of-distribution data. Mitigating these biases typically requires expensive, manually-curated group labels. We introduce SPUME (SPUriousness-aware MEta-learning), a novel framework that automates the detection and mitigation of spurious attributes. SPUME utilizes pre-trained Vision-Language Models (VLMs) to identify potential shortcuts and employs a meta-learning strategy to simulate correlation shifts during training. Our experiments demonstrate that SPUME significantly improves worst-group accuracy on benchmarks such as Waterbirds and CelebA, achieving state-of-the-art performance without requiring human-annotated group labels.
Introduction: The Challenge of Spurious Correlations
The tendency of deep neural networks to exploit "shortcuts" or spurious correlations is a well-documented obstacle to robust machine learning. For example, a model trained to identify "landbirds" might rely on a "forest" background rather than the bird's morphology. When tested on a "landbird" in a "beach" setting, the model fails. While methods like Group Distributionally Robust Optimization (Group DRO) can mitigate this, they depend on "group labels" that specify the spurious attribute for every training sample. Such labels are costly to obtain and often unavailable in real-world scenarios. SPUME addresses this limitation by automating the entire pipeline of attribute discovery and robust training.
The SPUME Framework
SPUME operates in three integrated stages to build robustness without manual intervention.
1. Automatic Attribute Detection
SPUME leverages the extensive world knowledge of pre-trained VLMs (e.g., BLIP) to generate textual descriptions for training images. From these captions, the system extracts informative nouns and adjectives to serve as a candidate set of spurious attributes $\mathcal{A}$. This removes the need for human domain experts to pre-define potential biases.
2. Measuring Spuriousness
To distinguish between benign and harmful attributes, SPUME defines a spuriousness metric. This metric measures the performance discrepancy of a baseline model on images of a specific class with and without a given attribute. A high discrepancy indicates a strong, brittle reliance on that attribute.
3. Spuriousness-Aware Meta-Learning
The core innovation is a meta-learning strategy that turns training into a "robustness gym." SPUME constructs tasks where the support set and query set are intentionally mismatched regarding their spurious correlations. By forcing the model to minimize loss on a query set with different backgrounds than the support set, the model learns to prioritize invariant, intrinsic features over transient shortcuts.
Experimental Evaluation
SPUME was evaluated on five major benchmarks, including Waterbirds, CelebA, and ImageNet-A. On the Waterbirds dataset, SPUME achieved a worst-group accuracy of 85.9%, a substantial improvement over the 66.4% achieved by standard Empirical Risk Minimization (ERM). Similarly, on CelebA, SPUME reached 86.0% worst-group accuracy, nearly doubling the ERM baseline of 45.7%. Furthermore, SPUME outperformed existing "no-group-label" methods like SIFER and DecAug on real-world OOD datasets like NICO, demonstrating its ability to generalize to unseen contexts.
Conclusion
SPUME provides a scalable, interpretable path toward robust image classification. By repurposing VLMs as automated annotators and meta-learning as a robustness simulator, it eliminates the dependency on expensive manual group labels. While the method's efficacy is linked to the descriptive quality of the underlying VLM, it represents a significant step toward models that learn what objects are rather than where they usually appear.
Summary updated