UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models

Arvindkumar Krishnakumar, Viraj Prabhu, Sruthi Sudhakar, Judy Hoffman

Introduction

Deep learning models often achieve high accuracy by latching onto spurious correlations—patterns in training data that correlate with a label but are not causally related to the concept. For instance, if a dataset primarily features male doctors, a model might learn to identify "maleness" as a feature of being a doctor rather than medical expertise.

Traditionally, identifying these biases requires researchers to pre-define "protected attributes" (like race or gender) and manually annotate thousands of images to check for performance disparities. This process is not only expensive and slow but also limited to biases that researchers already suspect exist. UDIS (Unsupervised Discovery of Bias) addresses this gap by providing an automated pipeline to discover "unknown unknowns" in model behavior without requiring manual attribute labels.

The UDIS Framework

UDIS audits a trained model by analyzing its internal representations to find systematic failure modes. The framework follows a multi-step pipeline:

  1. Feature Extraction: The model being audited is used as a feature extractor. Embeddings are taken from the penultimate layer, representing how the model "sees" the input.
  2. Hierarchical Clustering: Bottom-up agglomerative clustering is performed on these embeddings to create a "cluster tree," grouping images that the model treats as mathematically similar.
  3. Optimal Selection: The system uses the silhouette score to determine the most meaningful level of the tree to analyze, ensuring the clusters represent distinct visual concepts.
  4. Performance Filtering: UDIS specifically targets clusters where the model’s accuracy is significantly lower (specifically, less than two-thirds of the overall test accuracy).
  5. Visual Explanations: Finally, Grad-CAM (Gradient-weighted Class Activation Mapping) produces heatmaps to show which parts of the image the model focused on when making its decision.
The UDIS pipeline for unsupervised bias discovery, from feature extraction to failure mode identification.

Experimental Evaluation

The authors evaluated UDIS on two major computer vision benchmarks: CelebA (face attributes) and MS COCO (multi-label object recognition). In one experiment, they created a "Biased CelebA" variant where they intentionally manipulated the data so that "Black Hair" was a shortcut for "Smiling." UDIS successfully surfaced this specific subpopulation as a high-failure group, proving it can catch intentional or systemic data imbalances.

Discovering "Unknown Unknowns"

One of the most striking findings was that UDIS uncovered biases that humans would likely never think to label.

  • Non-Human Biases: In the CelebA "Smiling" classifier, UDIS found clusters with 0% accuracy. Grad-CAM revealed the model was ignoring the mouth entirely and focusing on background artifacts above people's heads or specific clothing items like hats and collarbones to predict smiles.
  • Contextual Traps: In MS COCO, UDIS revealed that the model often predicted "cup" simply because it saw a computer screen. Because cups and screens frequently co-occur in office settings, the model learned a contextual shortcut rather than the visual essence of a cup. It also identified a spurious correlation between "beds" and "teddy bears," frequently mislabeling the latter as the former.
  • The "Nearest Neighbor" Debugger: UDIS allows developers to compare a failure cluster with its "nearest neighbor" cluster where the model succeeds. This comparison highlights exactly where the model's logic diverges, serving as a powerful debugging tool.
Visualizing failure modes in the CelebA dataset. The heatmaps show the model focusing on spurious features like hair or background rather than the target attribute.

Key Results and Efficiency

By ranking hundreds of clusters and presenting only the most "suspicious" ones, UDIS drastically reduces the manual effort required for error analysis. On a model with 92% overall accuracy, UDIS was able to isolate specific visual subgroups where the model failed completely, providing actionable insights for developers to retrain or balance their datasets.

Limitations and Future Work

While powerful, UDIS has specific boundaries. It defines bias as a correlation that leads to a drop in accuracy; it cannot currently find "positive" biases—spurious correlations that happen to help the model's performance on the test set but might fail in the real world. Additionally, not every failure cluster is a bias; some may simply be "hard" images due to poor optimization. Future research aims to make the interpretation of these clusters more automatic, reducing the need for humans to manually inspect the Grad-CAM heatmaps to name the discovered bias.

Summary updated

BibTeX
@inproceedings{krishnakumar2021udis,
  author = {Arvindkumar Krishnakumar and Viraj Prabhu and Sruthi Sudhakar and Judy Hoffman},
  title = {{UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=0c1wg5A9Df},
}

Loading related content...

Figure