Introduction
Vision-Language Models (VLMs) like CLIP have revolutionized zero-shot image classification by enabling models to recognize objects without explicit training on specific classes. However, these models often behave as "black boxes," performing simple text retrieval that is susceptible to spurious correlations. For example, a model might classify a bird as a "waterbird" primarily because of a water background rather than the bird's physical characteristics. This gap between VLM inference and human perception—which naturally identifies context before objects—limits generalization and model interpretability.
The PerceptionCLIP Framework
PerceptionCLIP is a training-free, two-step inference method designed to mimic the hierarchical nature of human vision. Unlike standard CLIP, which uses generic templates like "a photo of a {class}", PerceptionCLIP explicitly models the relationship between an object and its environment.
Step 1: Inferring Context
In the first stage, the model uses CLIP to estimate the probability distribution of various "contextual attributes" ($z$). These attributes include scene layout (e.g., "on the grass," "in water"), image quality (e.g., "blurry"), and orientation (e.g., "upside-down"). Interestingly, the researchers found that CLIP is remarkably effective at identifying its own context, achieving an average accuracy of ~74% on these tasks without any fine-tuning.
Step 2: Conditioning on Context
In the second stage, the model predicts the object class ($y$) by marginalizing over these inferred attributes. It employs an attribute-aware CLIP score, calculating similarity based on descriptive, concatenated prompts such as "a photo of a dog, on the grass, upright." This process essentially "explains away" the background features, forcing the model to focus on the core features of the object itself.
Key Insights and Results
PerceptionCLIP was evaluated across 11 benchmarks, including standard datasets like ImageNet and robustness-focused datasets like Waterbirds and CelebA.
- Improved Generalization: The method consistently outperformed standard CLIP and traditional prompt ensembling. Notably, it achieved a 5% accuracy gain on EuroSAT (satellite imagery) and significant improvements on ImageNet-Sketch.
- De-biasing and Robustness: On the Waterbirds dataset, PerceptionCLIP reduced the accuracy gap between the easiest and hardest subgroups by 19%. By explicitly identifying the background, the model avoids being misled by common co-occurrences.
- Interpretability: Grad-CAM visualizations (as seen in Figure 1) demonstrate that PerceptionCLIP shifts the model's attention from spurious background elements (like grass) to the core object (the dog). Furthermore, the model provides a clear "rationale" by revealing the contextual attributes it identified.
- Human-in-the-Loop Potential: Because the process is decoupled, humans can intervene to correct misidentified contexts (e.g., correcting an orientation error), which further enhances final classification accuracy.
Limitations and Future Work
While PerceptionCLIP offers significant advantages, it remains sensitive to the specific phrasing of attribute descriptions. Additionally, considering multiple attribute combinations increases computational overhead compared to single-prompt methods. Future research aims to automate the attribute discovery process using Large Language Models (LLMs) and explore integrating this perception-aware logic directly into the pre-training phase of future VLMs.
Summary updated