Abstract
Vision-Language Models (VLMs) like CLIP are known for their impressive zero-shot capabilities, yet they frequently rely on "shortcuts"—spurious correlations between objects and their typical backgrounds—for classification. This reliance significantly impairs their performance in Out-of-Distribution (OOD) detection, where environmental noise can lead to high-confidence false positives. We present OSPCoOp (Overcoming Shortcut Problem via Co-Op), a method that decouples foreground objects from their backgrounds using external segmentation and inpainting. By utilizing these "pure" backgrounds as pseudo-OOD samples and implementing mask-guided regularization, OSPCoOp forces the model to ignore environmental shortcuts. Our approach achieves state-of-the-art performance on traditional OOD benchmarks and the newly introduced ImageNet-Bg dataset, even in extreme few-shot scenarios.
Introduction: The Shortcut Problem
The core challenge in robust Out-of-Distribution (OOD) detection for VLMs is the tendency of these models to "cheat" by using background textures as classification shortcuts. Because models like CLIP are trained on massive web-scale image-text pairs, they learn strong correlations between an object and its typical environment (e.g., a "whale" is almost always depicted in "sea water").
When deployed for OOD detection—the task of identifying images that do not belong to any known class—this correlation becomes a liability. If a model encounters an image of just sea water (an OOD sample), it may confidently label it as a "whale" simply because of the background context.
Existing few-shot OOD methods attempt to mitigate this by identifying "ID-irrelevant" regions, but they often rely on the model's own biased attention to find those regions, creating a circular problem where the model cannot fix the bias it uses to perceive the world.
Methodology: OSPCoOp
The authors propose OSPCoOp to break this cycle through three main components:
1. Background Decoupling
Instead of trusting the VLM's internal attention, OSPCoOp uses external segmentation models (such as SAM) to create precise masks of foreground objects. Inpainting models are then used to remove the object and fill the gap, creating "pure" background images that isolate the environmental shortcuts.
2. Strategic Data Augmentation
- Pseudo-OOD Supervision: The "pure" background images are used as synthetic OOD samples during training, teaching the model to associate these textures with low confidence for ID classes.
- ID Enhancement: The framework creates new In-Distribution (ID) samples by swapping the backgrounds of known objects, forcing the model to realize the object itself defines the class, not the environment.
3. Mask-Guided Region Regularization
The authors introduce a loss function that uses segmentation masks to explicitly penalize the model for being too "certain" about regions identified as background. By maximizing the entropy of predictions in these areas, the model learns to ignore them during the decision-making process.
Experimental Results
OSPCoOp was evaluated on traditional OOD benchmarks (iNaturalist, SUN, Places, and Texture) and a new benchmark introduced by the authors: ImageNet-Bg. This dataset specifically tests background robustness by using ~48,000 images where all ID-relevant objects have been removed, leaving only background interference.
Key Findings
- State-of-the-Art Performance: In a 16-shot setting, OSPCoOp achieved a mean AUROC of 94.75% and reduced the False Positive Rate (FPR95) to 25.13%, outperforming previous leaders like LoCoOp and NegPrompt.
- Superior Robustness: On the ImageNet-Bg dataset, OSPCoOp significantly outperformed existing methods (91.37% AUROC), proving it is much less likely to be fooled by background shortcuts.
- Few-Shot Efficiency: Even with only a single training example per class, the model achieved a mean AUROC of 93.81%, demonstrating extreme practical utility.
Discussion and Limitations
A key insight of this work is that we cannot rely on a biased model to fix its own bias; independent segmentation masks provide a necessary "ground truth" for what the model should ignore. However, OSPCoOp's effectiveness is dependent on the quality of the external segmentation and inpainting models. If segmentation fails to capture the foreground accurately, the regularization might inadvertently suppress important semantic features. Future research is needed to explore how these shortcuts form during the initial massive pre-training phase of VLMs.
Summary updated