Introduction
Contemporary computer vision models are typically trained on massive benchmark datasets like ImageNet. However, these datasets often contain hidden biases, artifacts, or spurious correlations. When a model exploits these shortcuts instead of learning generalizable features, it exhibits "Clever Hans" (CH) behavior—named after a horse that appeared to solve math problems but was actually reacting to subtle cues from its trainer. For example, a model might identify a "horse" primarily by detecting a specific photographer's watermark common in the training set. This paper introduces a framework to automatically identify these artifacts and surgically remove their influence.
Scalable Artifact Discovery with SpRAy
Local explanation methods (XAI) like Layer-wise Relevance Propagation (LRP) can reveal why a model made a specific decision, but they do not scale to millions of samples. Manually inspecting every heatmap is impossible, and simple data inspection often fails to reveal the model's internal logic.
The authors propose using Spectral Relevance Analysis (SpRAy) to automate this discovery. SpRAy works by clustering LRP heatmaps to group images that the model "looks at" in similar ways. By identifying outlier clusters, researchers can find systematic artifacts (like watermarks or specific backgrounds) that the model has adopted as shortcuts. An automated ranking system using Fisher Discriminant Analysis (FDA) further highlights the most suspicious clusters, enabling discovery across massive corpora like ImageNet.
Class Artifact Compensation (ClArC)
Once an artifact is identified and modeled—either explicitly in input space or as a Concept Activation Vector (CAV) in latent space—the paper introduces Class Artifact Compensation (ClArC) to "un-Hans" the model.
The framework offers two primary strategies for model correction:
- A-ClArC (Augmentative): A training-time intervention where the identified artifact is strategically added to other classes during fine-tuning. By making the artifact appear across all classes, the model learns it is no longer a useful discriminative signal and is forced to focus on the actual object features.
- P-ClArC (Projective): A post-hoc architectural fix. By identifying the "direction" of the artifact in the model's latent space, a projection layer is injected to "flatten" this signal. This effectively makes the model blind to the artifact without requiring full retraining.
Key Results and Insights
The authors uncovered a "zoo" of artifacts in ImageNet, including source watermarks (e.g., "Pferdefotoarchiv") and background cues. In medical imaging (ISIC 2019), they found models using colorful band-aids as a shortcut for melanoma diagnosis. Applying P-ClArC successfully shifted the model's focus away from these artifacts and back onto the actual clinical features, restoring the model's intended predictive logic.
Quantitative results show that the automated SpRAy extension achieved an AUC of 1.0 in detecting backdoor attacks, significantly outperforming standard statistical outlier detection. Furthermore, the ClArC methods were able to reduce reliance on stereotypical features in face analysis datasets (Adience), leading to fairer and more robust classifiers.
Conclusion
This framework provides a scalable path toward "functional cleaning" of datasets. By moving beyond simple data removal to surgical model correction, we can ensure that deep learning models learn the right things for the right reasons, moving us closer to truly "un-Hans'ed" AI.
Summary updated