Abstract
Given a pre-trained CNN without any testing samples, this paper proposes a simple yet effective method to diagnose feature representations of the CNN. We aim to discover representation flaws caused by potential dataset bias. More specifically, when the CNN is trained to estimate image attributes, we mine latent relationships between representations of different attributes inside the CNN. Then, we compare the mined attribute relationships with ground-truth attribute relationships to discover the CNN's blind spots and failure modes due to dataset bias. Representation flaws caused by dataset bias cannot be examined by conventional evaluation strategies based on testing images, because testing images may also have a similar bias. Experiments have demonstrated the effectiveness of our method.
Introduction: The Problem of Dataset Bias
Deep learning models are often treated as "black boxes," typically evaluated solely on test set accuracy. However, high accuracy can be deceptive if training and testing datasets share the same biases. For example, if a dataset frequently pairs "black hair" with "smiling," a CNN might learn to use the presence of a smile as a shortcut for predicting hair color. Standard evaluation fails to catch these problematic representations because the test set is often as biased as the training set. This paper addresses the challenge of identifying these hidden flaws—where the model relies on unreliable co-appearing contexts—without requiring a perfectly balanced, unbiased test set.
Methodology
The core of the proposed approach is to audit the "internal logic" of the CNN through a three-step process:
- Attribute Representation Mining: For a given attribute, such as "wearing lipstick," the CNN's prediction is approximated as a linear combination of neural activations in its convolutional layers. A sparse "mask" of neurons most influential for that attribute is identified.
- Relationship Extraction: The cosine similarity between these internal representations for pairs of attributes is computed across the training set, creating a "mined" distribution of attribute relationships.
- Diagnosis via KL-Divergence: This mined distribution is compared against human-defined "ground-truth" relationships (e.g., "smiling" and "black hair" should be semantically unrelated). By calculating the Kullback-Leibler (KL) divergence between the network's internal belief and human common sense, biased representations are pinpointed.
Datasets
The method was evaluated on two widely used benchmarks:
- CelebA (Large-scale CelebFaces Attributes): Contains over 200,000 celebrity images with 40 attribute annotations.
- SUN Attribute Database: A scene-centric dataset with 14,000 images and 102 attributes.
Key Results and Insights
The experiments revealed that a model can be highly accurate while being fundamentally "wrong" in its reasoning. For instance, a CNN might predict "wearing lipstick" based on patterns in the eyes or nose rather than the mouth due to contextual co-occurrences in the training data.
Key findings include:
- Zero-Shot Diagnosis: The method discovers biased representations in a batch without requiring new labeled testing images.
- Blind Spots vs. Failure Modes: The method distinguishes between "blind spots" (failing to learn real relationships) and "failure modes" (learning false relationships).
- Predicting Performance Drops: Identified failure modes were highly predictive of real-world failures. On images triggering these modes, model accuracy dropped significantly (e.g., from ~72% to ~46% on CelebA), outperforming standard entropy-based methods for finding errors.
Conclusion
This research demonstrates that internal representation mining can effectively diagnose dataset bias in CNNs without additional testing data. By identifying where a model's internal logic deviates from human common sense, we can better understand and eventually mitigate the risks of biased AI. Future work involves using these mined relationships as priors during training to enforce more robust feature learning.
Summary updated