Examining CNN representations with respect to Dataset Bias

Quanshi Zhang, Wenguan Wang, Song-Chun Zhu

Abstract

Given a pre-trained CNN without any testing samples, this paper proposes a simple yet effective method to diagnose feature representations of the CNN. We aim to discover representation flaws caused by potential dataset bias. More specifically, when the CNN is trained to estimate image attributes, we mine latent relationships between representations of different attributes inside the CNN. Then, we compare the mined attribute relationships with ground-truth attribute relationships to discover the CNN's blind spots and failure modes due to dataset bias. Representation flaws caused by dataset bias cannot be examined by conventional evaluation strategies based on testing images, because testing images may also have a similar bias. Experiments have demonstrated the effectiveness of our method.

Introduction: The Problem of Dataset Bias

Deep learning models are often treated as "black boxes," typically evaluated solely on test set accuracy. However, high accuracy can be deceptive if training and testing datasets share the same biases. For example, if a dataset frequently pairs "black hair" with "smiling," a CNN might learn to use the presence of a smile as a shortcut for predicting hair color. Standard evaluation fails to catch these problematic representations because the test set is often as biased as the training set. This paper addresses the challenge of identifying these hidden flaws—where the model relies on unreliable co-appearing contexts—without requiring a perfectly balanced, unbiased test set.

Methodology

The core of the proposed approach is to audit the "internal logic" of the CNN through a three-step process:

  1. Attribute Representation Mining: For a given attribute, such as "wearing lipstick," the CNN's prediction is approximated as a linear combination of neural activations in its convolutional layers. A sparse "mask" of neurons most influential for that attribute is identified.
  2. Relationship Extraction: The cosine similarity between these internal representations for pairs of attributes is computed across the training set, creating a "mined" distribution of attribute relationships.
  3. Diagnosis via KL-Divergence: This mined distribution is compared against human-defined "ground-truth" relationships (e.g., "smiling" and "black hair" should be semantically unrelated). By calculating the Kullback-Leibler (KL) divergence between the network's internal belief and human common sense, biased representations are pinpointed.

Datasets

The method was evaluated on two widely used benchmarks:

  • CelebA (Large-scale CelebFaces Attributes): Contains over 200,000 celebrity images with 40 attribute annotations.
  • SUN Attribute Database: A scene-centric dataset with 14,000 images and 102 attributes.
Sample image from the CelebA dataset used for attribute estimation tasks.

Key Results and Insights

The experiments revealed that a model can be highly accurate while being fundamentally "wrong" in its reasoning. For instance, a CNN might predict "wearing lipstick" based on patterns in the eyes or nose rather than the mouth due to contextual co-occurrences in the training data.

Diagnosis of biased features: The CNN's prediction for 'wearing lipstick' (middle) persists even when the mouth is masked (bottom), demonstrating reliance on non-mouth features like eyes or nose.

Key findings include:

  • Zero-Shot Diagnosis: The method discovers biased representations in a batch without requiring new labeled testing images.
  • Blind Spots vs. Failure Modes: The method distinguishes between "blind spots" (failing to learn real relationships) and "failure modes" (learning false relationships).
  • Predicting Performance Drops: Identified failure modes were highly predictive of real-world failures. On images triggering these modes, model accuracy dropped significantly (e.g., from ~72% to ~46% on CelebA), outperforming standard entropy-based methods for finding errors.

Conclusion

This research demonstrates that internal representation mining can effectively diagnose dataset bias in CNNs without additional testing data. By identifying where a model's internal logic deviates from human common sense, we can better understand and eventually mitigate the risks of biased AI. Future work involves using these mined relationships as priors during training to enforce more robust feature learning.

Summary updated

Abstract

Given a pre-trained CNN without any testing samples, this paper proposes a simple yet effective method to diagnose feature representations of the CNN. We aim to discover representation flaws caused by potential dataset bias. More specifically, when the CNN is trained to estimate image attributes, we mine latent relationships between representations of different attributes inside the CNN. Then, we compare the mined attribute relationships with ground-truth attribute relationships to discover the CNN's blind spots and failure modes due to dataset bias. In fact, representation flaws caused by dataset bias cannot be examined by conventional evaluation strategies based on testing images, because testing images may also have a similar bias. Experiments have demonstrated the effectiveness of our method.
BibTeX
@inproceedings{zhang2018examining,
  author = {Quanshi Zhang and Wenguan Wang and Song-Chun Zhu},
  title = {{Examining CNN representations with respect to Dataset Bias}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11833},
}
Figure