UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models

Arvindkumar Krishnakumar, Viraj Prabhu, Sruthi Sudhakar, Judy Hoffman

TL;DR

This paper introduces UDIS, an unsupervised method to automatically discover subpopulations in visual datasets where deep learning models systematically underperform. By clustering dataset embeddings and visualizing failure modes with GradCAM, UDIS helps identify potential biases without requiring manual annotations of protected attributes, making bias detection more scalable and accessible.

What Problem Does This Paper Address?

Deep learning models are increasingly used in high-stakes applications, but they can learn spurious correlations from training data, leading to systematic performance drops for certain subpopulations. Traditional methods for identifying these biases involve expensive and time-consuming manual annotation of protected attributes (like race or gender) and performance measurement across these groups. This approach is difficult to scale to large datasets and may miss biases related to attributes not explicitly considered. The paper aims to address this by providing an unsupervised method to discover these failure modes automatically.

Methods

The core of UDIS involves several key steps:

  1. Feature Extraction: The penultimate layer embeddings (features) of a trained model are extracted for each image in the test dataset.
  2. Hierarchical Clustering: These embeddings are then clustered using an agglomerative hierarchical clustering approach. This process recursively groups similar image embeddings, forming a binary tree of clusters. The method observes that this clustering naturally tends to separate higher-accuracy clusters from lower-accuracy ones.
  3. Cluster Selection and Thresholding: To identify potentially biased clusters, UDIS uses the silhouette score to find the optimal clustering level. It then filters these clusters by retaining only those where the model's accuracy is significantly lower than the overall test set accuracy (specifically, less than two-thirds of the overall accuracy). This filtering aims to isolate clusters indicative of failure modes rather than simple misclassifications.
  4. Visual Explanations: For the identified underperforming clusters, UDIS uses Gradient-weighted Class Activation Mapping (GradCAM) to visualize which regions of the input images the model is focusing on for its classification decisions. This helps in understanding why the model is failing for these specific subpopulations.

Datasets

The authors evaluate UDIS on the following datasets:

  • CelebA: Used for single-attribute prediction tasks (e.g., smiling). They also create a "biased CelebA" dataset by intentionally subsampling to induce bias related to the "Black Hair" attribute.
  • MS COCO: Used for a multi-label 80-way classification task.

Key Insights and Ideas

  • Unsupervised Bias Discovery: The most significant insight is that bias can be discovered without explicit knowledge of protected attributes. By clustering model embeddings, UDIS can identify subpopulations that the model struggles with, which may correlate with unannotated biases.
  • Spurious Correlations as Artifacts: UDIS effectively surfaces "spurious correlations" by highlighting cases where models rely on image artifacts (e.g., something above a person's head, or a collarbone) rather than the actual semantic content relevant to the task (e.g., facial expression for smiling).
  • Scalability: The unsupervised nature of UDIS makes it a scalable solution for bias detection, overcoming the limitations of manual annotation.
  • Visual Explanations are Crucial: GradCAM visualizations are key to understanding why a cluster is underperforming, revealing the specific image features (or artifacts) the model is attending to.

Key Results

  • UDIS successfully identified failure modes in models trained on CelebA and MS COCO datasets.
  • On the CelebA dataset, UDIS discovered that models predicting "smiling" were sometimes focusing on artifacts above the head or around the collarbone, rather than the mouth.
  • In the "biased CelebA" experiment, UDIS identified clusters where the model's decision was influenced by hair color (specifically, "Black Hair"), demonstrating its ability to uncover induced biases.
  • For MS COCO, UDIS found biases related to the "cup" category, where the model overpredicted "cup" when a screen was present, and for the "bed" category, where the model seemed to associate "teddy bear" with "bed."

Limitations and Future Work

  • Focus on Failure Modes: UDIS exclusively focuses on discovering bias in the form of performance drops (failure modes). It does not detect biases that might improve model performance or biases arising from optimization/generalization errors that aren't true model biases.
  • Correlation with Cultural Biases: It remains an open question how frequently the biases discovered by UDIS correlate with known cultural biases.
  • Interpretable Bias: The method does not guarantee that all discovered failure modes correspond to interpretable biases against protected attributes like race or gender.
  • Future Work: The authors suggest extending the method to discover other kinds of bias, including those that lead to performance improvements, and to better understand the relationship between discovered biases and known cultural biases.

Summary updated

Abstract

Deep learning models have been shown to learn spurious correlations from data that sometimes lead to systematic failures for certain subpopulations. Prior work has typically diagnosed this by crowdsourcing annotations for various protected attributes and measuring performance, which is both expensive to acquire and difficult to scale. In this work, we propose UDIS, an unsupervised algorithm for surfacing and analyzing such failure modes. UDIS identifies subpopulations via hierarchical clustering of dataset embeddings and surfaces systematic failure modes by visualizing low performing clusters along with their gradient-weighted class-activation maps. We show the effectiveness of UDIS in identifying failure modes in models trained for image classification on the CelebA and MSCOCO datasets.
BibTeX
@inproceedings{krishnakumar2021udis,
  author = {Arvindkumar Krishnakumar and Viraj Prabhu and Sruthi Sudhakar and Judy Hoffman},
  title = {{UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=0c1wg5A9Df},
}
Figure