TL;DR
ViG-Bias (Visually Grounded Bias Discovery and Mitigation) is a simple yet powerful framework that uses visual explanations—like heatmaps—to pinpoint exactly where a model is looking when it makes a mistake. By masking images to focus only on these "suspicious" regions before passing them to Vision-Language Models (VLMs), the researchers significantly improved the ability to automatically identify and fix hidden spurious correlations in image classifiers.
What Problem Does This Paper Address?
Machine learning models are notorious for taking "shortcuts." A model might learn to identify a "waterbird" not by the bird's features, but simply because there is water in the background. These spurious correlations lead to biased systems that fail on "bias-conflicting" samples (e.g., a waterbird on land).
The challenge is that we often don't know what these biases are ahead of time. While recent methods use Vision-Language Models (like CLIP) to describe why a model is failing, these tools often look at the entire image. If the model is failing because of a small, localized feature, the VLM might get "distracted" by the rest of the image, leading to vague or incorrect descriptions of the bias.
Methods
The core idea of ViG-Bias is visual grounding. Instead of giving a VLM the original image, the authors propose a mapping function $h(x)$ that masks the image, keeping only the areas where the biased model is focusing its attention.
- Generate Visual Explanations: The authors use techniques like Grad-CAM to produce a heatmap showing which pixels most influenced the model's (incorrect) prediction.
- Apply a Mask: They threshold this heatmap to create a binary mask and apply it to the original image. This effectively "crops" the image to show only what the biased model was "thinking" about.
- Direct the VLM: This masked image is then fed into existing bias-discovery frameworks:
- ViG-DOMINO/FACTS: The masked images are converted into CLIP embeddings. Because the background is removed, the resulting clusters are much more likely to represent the actual spurious feature.
- ViG-B2T (Bias-to-Text): The masked image is sent to an image captioning model. This forces the model to generate keywords specifically about the suspicious region (e.g., "water" or "face") rather than the main object (e.g., "bird" or "hair").
Datasets
The authors evaluated their method on three standard benchmarks for spurious correlations:
- CelebA: A dataset of celebrity faces where "blond hair" is spuriously correlated with "female."
- Waterbirds: A dataset where "waterbirds" and "landbirds" are placed on "water" or "land" backgrounds.
- NICO++: A large-scale benchmark designed specifically for domain generalization and identifying spurious attributes across various contexts.
Key Insights and Ideas
- The "Aha!" Moment: The authors quantitatively proved that when a biased model fails, its visual explanations (heatmaps) overlap significantly more with the spurious features than with the core features. As shown in Figure 2, in the CelebA dataset, the model's attention shifts from the hair (core feature) to the face (spurious feature) when it relies on bias.
- Focusing the VLM: By removing the "correct" parts of the image, they prevent the VLM from being "polite." Instead of describing a "bird on the water," the VLM is forced to just see "water," which is exactly the keyword needed to identify the bias.
- Plug-and-Play: ViG-Bias isn't a new standalone model; it's a preprocessing step that can be integrated into almost any existing discovery or mitigation pipeline.
Key Results
The researchers found that adding visual grounding consistently boosted performance across the board:
- Better Discovery: In the CelebA dataset, ViG-B2T improved the precision of discovered bias slices from 64.0% to 70.0%.
- Effective Mitigation: When using these discovered keywords to retrain models (using GroupDRO), they saw massive jumps in Worst-Group Accuracy (WGA). On the Waterbirds dataset, WGA increased from 55.0% to 63.1% compared to the standard Bias-to-Text method.
- Reduced Performance Gap: The method consistently narrowed the gap between average accuracy and the accuracy of the most disadvantaged subgroup, leading to fairer models.
Limitations and Future Work
- Localized Biases Only: ViG-Bias relies on the assumption that the bias is something that can be "pointed to" in an image (like a background or a specific object). It may struggle with global biases, such as changes in overall lighting, image resolution, or digital noise.
- Dependence on the Explainer: The quality of the discovery depends on the quality of the visual explanation method (e.g., Grad-CAM). If the explainer fails to capture the model's logic, the mask will be useless.
- Future Directions: The authors suggest exploring how to integrate ViG-Bias with even more diverse discovery frameworks and investigating its utility in other modalities beyond static images.
Summary updated