Abstract
Contemporary learning models for computer vision are typically trained on very large benchmark datasets with millions of samples. These may, however, contain biases, artifacts, or errors that have gone unnoticed and are exploitable by the model. In the worst case, the trained model becomes a "Clever-Hans" (CH) predictor that bases its decisions on spurious correlations in the training data. We contribute a comprehensive analysis framework based on a scalable statistical analysis of attributions from explanation methods. Based on Spectral Relevance Analysis (SpRAy), we propose: (a) a scalable quantification of artifactual classes where models exhibit CH behavior, and (b) Class Artifact Compensation (ClArC) techniques to effectively reduce this behavior. We demonstrate that ClArC can be implemented during training, fine-tuning, or post-hoc, allowing for improved, more representative, and fairer models.
Introduction: The "Clever Hans" Problem
The "Clever Hans" (CH) effect is a notorious pitfall in machine learning, named after a 1900s horse that appeared to perform arithmetic but was actually responding to subtle cues from its trainer. In deep learning, this describes models that solve tasks for the "wrong reasons"—for instance, classifying an image as a "horse" based on a photographer's watermark rather than the animal's features.
As datasets like ImageNet scale to millions of samples, manual inspection for such biases becomes impossible. While eXplainable AI (XAI) methods provide "local" explanations for individual predictions, they do not naturally scale to provide "global" dataset-level insights. Furthermore, once an artifact is identified, practitioners often lack a principled way to fix the model without discarding data or starting from scratch.
Methodology: Identification and Compensation
The proposed framework consists of a two-stage pipeline: Identification and Compensation.
1. Scalable Identification (SpRAy + FDA)
The authors extend Spectral Relevance Analysis (SpRAy), which utilizes Layer-wise Relevance Propagation (LRP) to generate attribution heatmaps.
- Clustering: SpRAy applies spectral clustering to these heatmaps to identify groups of images where the model employs similar strategies.
- Automation: They introduce Fisher Discriminant Analysis (FDA) to calculate a "Clever Hans-ness" score ($\tau$). This automates the ranking of classes, allowing researchers to efficiently pinpoint which of the 1,000 ImageNet classes are most reliant on artifacts.
2. Class Artifact Compensation (ClArC)
Once an artifact is identified and modeled, two primary techniques are used to neutralize it:
- A-ClArC (Augmentative): A training-time fix where the identified artifact (e.g., a watermark) is strategically augmented across the entire training set. By making the artifact ubiquitous across all classes, it becomes non-informative, forcing the model to focus on actual object features.
- P-ClArC (Projective): A post-hoc architectural fix using Concept Activation Vectors (CAVs) to identify the "direction" of the artifact in the model's latent space. A new layer is injected to project activations onto a subspace orthogonal to that direction, filtering out the artifact signal during inference without retraining.
Experimental Results and Insights
The framework was validated across several diverse datasets:
- ImageNet: Automatically flagged classes like "garbage truck" (watermarks) and "stole" (mannequin heads).
- Medical Imaging (ISIC 2019): Successfully suppressed "band-aid" artifacts in skin lesion analysis. Heatmaps confirmed that after compensation, the model's attention shifted from the band-aid back to the lesion.
- Fairness (Adience): Reduced gender bias by removing the "shirt collar" concept, leading to predictions less dependent on stereotypical clothing.
- Malicious Attacks: In CIFAR-10, the authors compared CH artifacts to backdoor attacks. While backdoors are often statistical outliers, CH artifacts are "naturally occurring" and often correlate better with labels than true features.
- Layer-wise Complexity: Simple artifacts (like borders) are best handled in early layers, while complex semantic artifacts (like mannequin heads) are better addressed in deeper, more invariant layers.
Limitations and Future Work
While the ranking is automated, human-in-the-loop verification is still required to confirm if a cluster represents a true artifact. Additionally, the effectiveness of P-ClArC depends on the precision of the CAV; highly variable artifacts may require non-linear modeling. Finally, while P-ClArC masks bad features, it does not "teach" the model new ones; future work could combine these methods with active learning to fill knowledge gaps.
Conclusion
This work demonstrates that "un-Hansing" a model does not necessarily require perfectly clean data. By combining automated XAI analysis with ClArC techniques, researchers can implicitly clean data corpora during or after the learning process, paving the way for more trustworthy and representative deep learning models.
Summary updated