Finding and Removing Clever Hans: Using Explanation Methods to Debug and\n Improve Deep Models

Christopher J. Anders, Leander Weber, David Neumann, Wojciech Samek, Klaus‐Robert Müller, Sebastian Lapuschkin

Abstract

Contemporary learning models for computer vision are typically trained on very large benchmark datasets with millions of samples. These may, however, contain biases, artifacts, or errors that have gone unnoticed and are exploitable by the model. In the worst case, the trained model becomes a "Clever-Hans" (CH) predictor that bases its decisions on spurious correlations in the training data. We contribute a comprehensive analysis framework based on a scalable statistical analysis of attributions from explanation methods. Based on Spectral Relevance Analysis (SpRAy), we propose: (a) a scalable quantification of artifactual classes where models exhibit CH behavior, and (b) Class Artifact Compensation (ClArC) techniques to effectively reduce this behavior. We demonstrate that ClArC can be implemented during training, fine-tuning, or post-hoc, allowing for improved, more representative, and fairer models.

Introduction: The "Clever Hans" Problem

The "Clever Hans" (CH) effect is a notorious pitfall in machine learning, named after a 1900s horse that appeared to perform arithmetic but was actually responding to subtle cues from its trainer. In deep learning, this describes models that solve tasks for the "wrong reasons"—for instance, classifying an image as a "horse" based on a photographer's watermark rather than the animal's features.

As datasets like ImageNet scale to millions of samples, manual inspection for such biases becomes impossible. While eXplainable AI (XAI) methods provide "local" explanations for individual predictions, they do not naturally scale to provide "global" dataset-level insights. Furthermore, once an artifact is identified, practitioners often lack a principled way to fix the model without discarding data or starting from scratch.

Methodology: Identification and Compensation

The proposed framework consists of a two-stage pipeline: Identification and Compensation.

Comprehensive workflow for identifying, estimating, and compensating for class artifacts.

1. Scalable Identification (SpRAy + FDA)

The authors extend Spectral Relevance Analysis (SpRAy), which utilizes Layer-wise Relevance Propagation (LRP) to generate attribution heatmaps.

  • Clustering: SpRAy applies spectral clustering to these heatmaps to identify groups of images where the model employs similar strategies.
  • Automation: They introduce Fisher Discriminant Analysis (FDA) to calculate a "Clever Hans-ness" score ($\tau$). This automates the ranking of classes, allowing researchers to efficiently pinpoint which of the 1,000 ImageNet classes are most reliant on artifacts.

2. Class Artifact Compensation (ClArC)

Once an artifact is identified and modeled, two primary techniques are used to neutralize it:

Detailed view of the ClArC methods: P-ClArC (Projective) and A-ClArC (Augmentative) compared to the original model.
  • A-ClArC (Augmentative): A training-time fix where the identified artifact (e.g., a watermark) is strategically augmented across the entire training set. By making the artifact ubiquitous across all classes, it becomes non-informative, forcing the model to focus on actual object features.
  • P-ClArC (Projective): A post-hoc architectural fix using Concept Activation Vectors (CAVs) to identify the "direction" of the artifact in the model's latent space. A new layer is injected to project activations onto a subspace orthogonal to that direction, filtering out the artifact signal during inference without retraining.

Experimental Results and Insights

The framework was validated across several diverse datasets:

  • ImageNet: Automatically flagged classes like "garbage truck" (watermarks) and "stole" (mannequin heads).
  • Medical Imaging (ISIC 2019): Successfully suppressed "band-aid" artifacts in skin lesion analysis. Heatmaps confirmed that after compensation, the model's attention shifted from the band-aid back to the lesion.
  • Fairness (Adience): Reduced gender bias by removing the "shirt collar" concept, leading to predictions less dependent on stereotypical clothing.
  • Malicious Attacks: In CIFAR-10, the authors compared CH artifacts to backdoor attacks. While backdoors are often statistical outliers, CH artifacts are "naturally occurring" and often correlate better with labels than true features.
  • Layer-wise Complexity: Simple artifacts (like borders) are best handled in early layers, while complex semantic artifacts (like mannequin heads) are better addressed in deeper, more invariant layers.

Limitations and Future Work

While the ranking is automated, human-in-the-loop verification is still required to confirm if a cluster represents a true artifact. Additionally, the effectiveness of P-ClArC depends on the precision of the CAV; highly variable artifacts may require non-linear modeling. Finally, while P-ClArC masks bad features, it does not "teach" the model new ones; future work could combine these methods with active learning to fill knowledge gaps.

Conclusion

This work demonstrates that "un-Hansing" a model does not necessarily require perfectly clean data. By combining automated XAI analysis with ClArC techniques, researchers can implicitly clean data corpora during or after the learning process, paving the way for more trustworthy and representative deep learning models.

Summary updated

Abstract

Contemporary learning models for computer vision are typically trained on\nvery large (benchmark) datasets with millions of samples. These may, however,\ncontain biases, artifacts, or errors that have gone unnoticed and are\nexploitable by the model. In the worst case, the trained model does not learn a\nvalid and generalizable strategy to solve the problem it was trained for, and\nbecomes a 'Clever-Hans' (CH) predictor that bases its decisions on spurious\ncorrelations in the training data, potentially yielding an unrepresentative or\nunfair, and possibly even hazardous predictor. In this paper, we contribute by\nproviding a comprehensive analysis framework based on a scalable statistical\nanalysis of attributions from explanation methods for large data corpora. Based\non a recent technique - Spectral Relevance Analysis - we propose the following\ntechnical contributions and resulting findings: (a) a scalable quantification\nof artifactual and poisoned classes where the machine learning models under\nstudy exhibit CH behavior, (b) several approaches denoted as Class Artifact\nCompensation (ClArC), which are able to effectively and significantly reduce a\nmodel's CH behavior. I.e., we are able to un-Hans models trained on (poisoned)\ndatasets, such as the popular ImageNet data corpus. We demonstrate that ClArC,\ndefined in a simple theoretical framework, may be implemented as part of a\nNeural Network's training or fine-tuning process, or in a post-hoc manner by\ninjecting additional layers, preventing any further propagation of undesired CH\nfeatures, into the network architecture. Using our proposed methods, we provide\nqualitative and quantitative analyses of the biases and artifacts in various\ndatasets. We demonstrate that these insights can give rise to improved, more\nrepresentative and fairer models operating on implicitly cleaned data corpora.\n
BibTeX
@misc{anders2019finding,
  author = {Christopher J. Anders and Leander Weber and David Neumann and Wojciech Samek and Klaus‐Robert Müller and Sebastian Lapuschkin},
  title = {{Finding and Removing Clever Hans: Using Explanation Methods to Debug and\textbackslash\{\}n Improve Deep Models}},
  year = {2019},
  eprint = {1912.11425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.11425},
}
Figure