Towards Fixing Clever-Hans Predictors with Counterfactual Knowledge Distillation

Sidney Bender, Christopher J. Anders, Pattarawatt Chormai, Heike Marxfeld, Jan Herrmann, Grégoire Montavon

TL;DR

This paper introduces Counterfactual Knowledge Distillation (CFKD), a method to fix "Clever Hans" predictors—models that achieve high accuracy by relying on spurious correlations (confounders) rather than causal features. By generating counterfactual explanations and incorporating human-expert feedback into an iterative retraining loop, CFKD successfully steers models toward biologically or logically sound decision strategies, significantly improving performance on unbiased test data.

What Problem Does This Paper Address?

In machine learning, a "Clever Hans" predictor is a model that appears to perform well but is actually "cheating." Just as the famous horse Hans seemed to solve math problems by reading his trainer's body language, these models might identify a "cancerous" biopsy not by the cells themselves, but by a specific hospital's watermark or a staining artifact.

This is a massive liability in safety-critical fields like medicine or autonomous driving. Standard validation accuracy fails to detect this because the confounders are often present in both the training and validation sets. The authors aim to bridge this gap by creating a way to detect these spurious dependencies and "unlearn" them using expert knowledge.

Methods

The core of the paper is the Counterfactual Knowledge Distillation (CFKD) algorithm. It consists of three main components:

  1. Diffeomorphic Counterfactuals: To understand a model's strategy, the authors generate counterfactuals—minimal changes to an image that flip the model's prediction. Unlike standard adversarial attacks that produce grainy noise, they use a generative model (Glow) to ensure changes are semantically meaningful and stay "on the data manifold" (see Figure 5). Generative models help keep counterfactuals realistic and semantically sound.

  2. The Feedback Loop: An expert (human or an "oracle" model) reviews these counterfactuals. If the model changes a causal feature (e.g., adding cell layers to a follicle), the expert marks it as a "True Counterfactual." If it changes a spurious feature (e.g., just making the follicle bigger), it's a "False Counterfactual."

  3. Iterative Retraining: These labeled counterfactuals are added back into the training set. This forces the model to learn that the spurious features do not reliably indicate the class, effectively distilling the expert's causal knowledge into the student model.
Figure 5. Diffeomorphic Counterfactual Explanations [5]. As can be seen one has to use a generative model as a regularizer instead of directly using gradient ascend in input space to avoid creating adversarial examples.

The authors also introduce Feedback Accuracy, a new metric that measures the proportion of counterfactuals the teacher accepts. They show this metric is a much better proxy for true generalization than standard validation accuracy.

Datasets

The researchers evaluated CFKD across three distinct scenarios:

  • Rat Ovarian Follicle Dataset: A real-world histopathology dataset (1,600 images) where the model initially relied on follicle size rather than the number of granulosa cell layers.
  • Poisoned CelebA: A subset of the CelebA face dataset (40k images) manually "poisoned" with three types of confounders: copyright tags, intensity shifts, and color shifts.
  • Colorectal Cancer Tissue 100K: A large-scale histopathology dataset used to test the method's ability to handle staining-based confounders common in medical imaging.

Key Insights and Ideas

  • Counterfactuals > Heatmaps: Traditional XAI methods like LRP (Layer-wise Relevance Propagation) produce heatmaps that can be ambiguous. Counterfactuals are more intuitive because they show what needs to change to flip a decision, making it easier for experts to spot when a model is looking at the wrong thing.
  • The Validation Trap: The paper highlights that high validation accuracy is often a false signal. In their CelebA experiments, models achieved near-perfect validation accuracy while failing miserably on unpoisoned test data.
  • Feedback as a Regularizer: CFKD acts as a form of human-guided regularization. By labeling "False Counterfactuals" with their original class, the model is explicitly taught to ignore the confounder.

Key Results

  • Follicle Detection: Before CFKD, the model's counterfactuals for "growing follicles" simply increased the size of the object. After CFKD, the counterfactuals correctly showed the addition of granulosa cell layers, aligning with biological reality (see Figure 4). Counterfactuals before and after teaching show a shift from size-based to feature-based logic.

  • CelebA Performance: In cases of 100% poisoning (where the confounder and class are perfectly correlated in training), the uncorrected model's test accuracy was no better than random (50%). CFKD restored performance significantly, approaching the accuracy of an oracle model.

  • Metric Reliability: The authors demonstrated that "Feedback Accuracy" peaks exactly when the model's true test performance is highest, whereas validation accuracy remains misleadingly flat or even decreases.
Figure 4. Left: produced counterfactuals transforming a primor- dial follicle into a growing follicle. Right: counterfactuals for transforming a growing follicle into a primordial one.

Limitations and Future Work

  • Human Bottleneck: The human-in-the-loop approach (Figure 6) is precise but time-consuming and expensive, especially for niche domains requiring highly trained experts. The web interface used for human experts to provide feedback on counterfactual pairs.

  • Generative Model Quality: The success of CFKD depends heavily on the quality of the underlying generative model. If the generator cannot accurately represent the data manifold, the counterfactuals may be nonsensical.

  • Scalability: While the authors suggest using SpRAy (clustering explanations) to reduce human effort, further work is needed to make this process fully scalable for massive datasets.
The web interface used for human experts to provide feedback on counterfactual pairs.

Summary updated

Abstract

This paper introduces a novel technique called counterfactual knowledge distillation (CFKD) to detect and remove reliance on confounders in deep learning models with the help of human expert feedback. Confounders are spurious features that models tend to rely on, which can result in unexpected errors in regulated or safety-critical domains. The paper highlights the benefit of CFKD in such domains and shows some advantages of counterfactual explanations over other types of explanations. We propose an experiment scheme to quantitatively evaluate the success of CFKD and different teachers that can give feedback to the model. We also introduce a new metric that is better correlated with true test performance than validation accuracy. The paper demonstrates the effectiveness of CFKD on synthetically augmented datasets and on real-world histopathological datasets.
BibTeX
@inproceedings{bender2023towards,
  author = {Sidney Bender and Christopher J. Anders and Pattarawatt Chormai and Heike Marxfeld and Jan Herrmann and Grégoire Montavon},
  title = {{Towards Fixing Clever-Hans Predictors with Counterfactual Knowledge Distillation}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
}
Figure