More Context, Less Distraction: Visual Classification by Inferring and Conditioning on Contextual Attributes

Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, Furong Huang

Introduction

Vision-Language Models (VLMs) like CLIP have revolutionized zero-shot image classification by enabling models to recognize objects without explicit training on specific classes. However, these models often behave as "black boxes," performing simple text retrieval that is susceptible to spurious correlations. For example, a model might classify a bird as a "waterbird" primarily because of a water background rather than the bird's physical characteristics. This gap between VLM inference and human perception—which naturally identifies context before objects—limits generalization and model interpretability.

The PerceptionCLIP Framework

PerceptionCLIP is a training-free, two-step inference method designed to mimic the hierarchical nature of human vision. Unlike standard CLIP, which uses generic templates like "a photo of a {class}", PerceptionCLIP explicitly models the relationship between an object and its environment.

Overview of PerceptionCLIP: The model first infers contextual attributes (like orientation and background) to create a conditioned prompt, shifting focus from spurious to core features.

Step 1: Inferring Context

In the first stage, the model uses CLIP to estimate the probability distribution of various "contextual attributes" ($z$). These attributes include scene layout (e.g., "on the grass," "in water"), image quality (e.g., "blurry"), and orientation (e.g., "upside-down"). Interestingly, the researchers found that CLIP is remarkably effective at identifying its own context, achieving an average accuracy of ~74% on these tasks without any fine-tuning.

Step 2: Conditioning on Context

In the second stage, the model predicts the object class ($y$) by marginalizing over these inferred attributes. It employs an attribute-aware CLIP score, calculating similarity based on descriptive, concatenated prompts such as "a photo of a dog, on the grass, upright." This process essentially "explains away" the background features, forcing the model to focus on the core features of the object itself.

Key Insights and Results

PerceptionCLIP was evaluated across 11 benchmarks, including standard datasets like ImageNet and robustness-focused datasets like Waterbirds and CelebA.

  • Improved Generalization: The method consistently outperformed standard CLIP and traditional prompt ensembling. Notably, it achieved a 5% accuracy gain on EuroSAT (satellite imagery) and significant improvements on ImageNet-Sketch.
  • De-biasing and Robustness: On the Waterbirds dataset, PerceptionCLIP reduced the accuracy gap between the easiest and hardest subgroups by 19%. By explicitly identifying the background, the model avoids being misled by common co-occurrences.
  • Interpretability: Grad-CAM visualizations (as seen in Figure 1) demonstrate that PerceptionCLIP shifts the model's attention from spurious background elements (like grass) to the core object (the dog). Furthermore, the model provides a clear "rationale" by revealing the contextual attributes it identified.
  • Human-in-the-Loop Potential: Because the process is decoupled, humans can intervene to correct misidentified contexts (e.g., correcting an orientation error), which further enhances final classification accuracy.

Limitations and Future Work

While PerceptionCLIP offers significant advantages, it remains sensitive to the specific phrasing of attribute descriptions. Additionally, considering multiple attribute combinations increases computational overhead compared to single-prompt methods. Future research aims to automate the attribute discovery process using Large Language Models (LLMs) and explore integrating this perception-aware logic directly into the pre-training phase of future VLMs.

Summary updated

BibTeX
@inproceedings{an2024more,
  author = {Bang An and Sicheng Zhu and Michael-Andrei Panaitescu-Liess and Chaithanya Kumar Mummadi and Furong Huang},
  title = {{More Context, Less Distraction: Visual Classification by Inferring and Conditioning on Contextual Attributes}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=2Oiee202rd},
}
Figure