Analyzing ImageNet with Spectral Relevance Analysis: Towards ImageNet un-Hans'ed

Christopher J. Anders, Talmaj Marinc, David Neumann, Wojciech Samek, Klaus-Robert Müller, Sebastian Lapuschkin

Introduction

Contemporary computer vision models are typically trained on massive benchmark datasets like ImageNet. However, these datasets often contain hidden biases, artifacts, or spurious correlations. When a model exploits these shortcuts instead of learning generalizable features, it exhibits "Clever Hans" (CH) behavior—named after a horse that appeared to solve math problems but was actually reacting to subtle cues from its trainer. For example, a model might identify a "horse" primarily by detecting a specific photographer's watermark common in the training set. This paper introduces a framework to automatically identify these artifacts and surgically remove their influence.

Scalable Artifact Discovery with SpRAy

Local explanation methods (XAI) like Layer-wise Relevance Propagation (LRP) can reveal why a model made a specific decision, but they do not scale to millions of samples. Manually inspecting every heatmap is impossible, and simple data inspection often fails to reveal the model's internal logic.

Motivation for SpRAy: manual inspection of data or individual XAI heatmaps is infeasible for large datasets.

The authors propose using Spectral Relevance Analysis (SpRAy) to automate this discovery. SpRAy works by clustering LRP heatmaps to group images that the model "looks at" in similar ways. By identifying outlier clusters, researchers can find systematic artifacts (like watermarks or specific backgrounds) that the model has adopted as shortcuts. An automated ranking system using Fisher Discriminant Analysis (FDA) further highlights the most suspicious clusters, enabling discovery across massive corpora like ImageNet.

Class Artifact Compensation (ClArC)

Once an artifact is identified and modeled—either explicitly in input space or as a Concept Activation Vector (CAV) in latent space—the paper introduces Class Artifact Compensation (ClArC) to "un-Hans" the model.

The complete ClArC workflow: from identifying artifacts with SpRAy to updating the model via projection or unlearning.

The framework offers two primary strategies for model correction:

  • A-ClArC (Augmentative): A training-time intervention where the identified artifact is strategically added to other classes during fine-tuning. By making the artifact appear across all classes, the model learns it is no longer a useful discriminative signal and is forced to focus on the actual object features.
  • P-ClArC (Projective): A post-hoc architectural fix. By identifying the "direction" of the artifact in the model's latent space, a projection layer is injected to "flatten" this signal. This effectively makes the model blind to the artifact without requiring full retraining.

Key Results and Insights

The authors uncovered a "zoo" of artifacts in ImageNet, including source watermarks (e.g., "Pferdefotoarchiv") and background cues. In medical imaging (ISIC 2019), they found models using colorful band-aids as a shortcut for melanoma diagnosis. Applying P-ClArC successfully shifted the model's focus away from these artifacts and back onto the actual clinical features, restoring the model's intended predictive logic.

Quantitative results show that the automated SpRAy extension achieved an AUC of 1.0 in detecting backdoor attacks, significantly outperforming standard statistical outlier detection. Furthermore, the ClArC methods were able to reduce reliance on stereotypical features in face analysis datasets (Adience), leading to fairer and more robust classifiers.

Conclusion

This framework provides a scalable path toward "functional cleaning" of datasets. By moving beyond simple data removal to surgical model correction, we can ensure that deep learning models learn the right things for the right reasons, moving us closer to truly "un-Hans'ed" AI.

Summary updated

Abstract

Contemporary learning models for computer vision are typically trained on very large (benchmark) datasets with millions of samples. These may, however, contain biases, artifacts, or errors that have gone unnoticed and are exploitable by the model. In the worst case, the trained model does not learn a valid and generalizable strategy to solve the problem it was trained for, and becomes a 'Clever-Hans' (CH) predictor that bases its decisions on spurious correlations in the training data, potentially yielding an unrepresentative or unfair, and possibly even hazardous predictor. In this paper, we contribute by providing a comprehensive analysis framework based on a scalable statistical analysis of attributions from explanation methods for large data corpora. Based on a recent technique - Spectral Relevance Analysis - we propose the following technical contributions and resulting findings: (a) a scalable quantification of artifactual and poisoned classes where the machine learning models under study exhibit CH behavior, (b) several approaches denoted as Class Artifact Compensation (ClArC), which are able to effectively and significantly reduce a model's CH behavior. I.e., we are able to un-Hans models trained on (poisoned) datasets, such as the popular ImageNet data corpus. We demonstrate that ClArC, defined in a simple theoretical framework, may be implemented as part of a Neural Network's training or fine-tuning process, or in a post-hoc manner by injecting additional layers, preventing any further propagation of undesired CH features, into the network architecture. Using our proposed methods, we provide qualitative and quantitative analyses of the biases and artifacts in various datasets. We demonstrate that these insights can give rise to improved, more representative and fairer models operating on implicitly cleaned data corpora.
BibTeX
@misc{anders2019analyzing,
  author = {Christopher J. Anders and Talmaj Marinc and David Neumann and Wojciech Samek and Klaus-Robert Müller and Sebastian Lapuschkin},
  title = {{Analyzing ImageNet with Spectral Relevance Analysis: Towards ImageNet un-Hans'ed}},
  year = {2019},
  eprint = {1912.11425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.11425},
}
Figure