Comprehensive Knowledge Distillation with Causal Intervention

Xiang Deng, Zhongfei ZhangNeurIPS 2021 · OpenReview

TL;DR

Standard knowledge distillation (KD) often forces a student model to blindly mimic a teacher, including its flaws. This paper introduces CID (Comprehensive, Interventional Distillation), which not only transfers class-level structural knowledge but also uses causal inference to "filter out" the teacher's dataset-induced biases. The result is a student model that is more robust and generalizes better than those trained with traditional KD.

The Problem with Standard Knowledge Distillation

Deep Neural Networks (DNNs) are powerful but often too bulky for mobile or edge devices. Knowledge Distillation (KD) solves this by training a small "student" model to mimic a large "teacher" model. However, the authors identify two major gaps in current KD research:

  1. Ignoring Class Structure: Most methods focus on aligning individual sample features (e.g., ensuring a student's representation of a specific dog matches the teacher's). They ignore the class representation—how different samples within a class relate to one another.
  2. The "Blind Mimicry" Trap: Teachers aren't perfect. They often pick up "bias knowledge" from the training data context. For example, if most dogs in the training set are on grass, the teacher might learn that "grass = dog." Standard KD forces the student to inherit this bias, which hurts performance on new datasets where dogs might be indoors or cats might be outdoors.
Illustration of dataset bias where context (like grass or indoor furniture) misleads the model's predictions.

The CID Framework

To address these issues, the authors propose a two-pronged framework called CID:

1. Comprehensive Distillation (The "What" to learn)

  • Sample-Level: Instead of standard Mean Square Error (MSE), which can be biased toward samples with large feature norms, CID uses Normalized MSE (NM_MSE). This ensures every sample contributes equally to the student's learning.
  • Class-Level: The framework represents a class as a graph where nodes are samples and edges are similarities. The student is tasked with matching the "shape" of the teacher’s class graphs, preserving the complex relationships between images of the same category.

2. Interventional Distillation (The "How" to filter)

To solve the bias problem, the authors turn to Causal Inference. They model the distillation process as a causal graph where the teacher's "softened logits" (the probability outputs) are treated as a confounder—a variable that influences both the image features and the final prediction.

  • Backdoor Adjustment: Using "do-calculus," they perform a causal intervention. Instead of just learning $P(Y|X)$ (the probability of a label given an image), the student learns $P(Y|do(X))$, which effectively "blocks" the influence of the teacher's biased context.
  • Implementation: The teacher's logits are used as context information to remove bias. Essentially, the student is told: "Learn the object features the teacher sees, but ignore the background noise the teacher is obsessed with."

Key Insights and Results

The authors evaluated CID across standard computer vision benchmarks, including CIFAR-10/100, Tiny ImageNet, ImageNet, and STL-10 (for transferability).

  • Classes have "Shapes": By forcing the student to match the teacher's sample-to-sample similarity matrix, the student learns a much richer internal representation than just matching single points in feature space.
  • The Teacher as a Confounder: Viewing the teacher as a source of both signal and noise (bias) is a brilliant conceptual shift. Causal intervention allows the student to keep the signal and drop the noise.
  • Robustness in Scarcity: CID's advantage is even more pronounced when training data is limited. In "data-starved" scenarios, models are more prone to overfit to background biases, making CID’s causal filtering highly valuable.

Performance Highlights: CID consistently outperformed state-of-the-art (SOTA) methods like CRD and SRRL across all benchmarks.

  • CIFAR-100: Distilling a WideResNet-40-2 into a WRN-16-2, CID achieved 76.40% accuracy, significantly higher than the teacher's 75.61% and standard KD's 74.52%.
  • ImageNet: CID achieved a Top-1 accuracy of 71.9%, outperforming the previous best method (SRRL at 71.7%).
  • Transferability: In a CIFAR-100 to STL-10 transfer task, CID reached 72.5% accuracy, while the next best competitor trailed at 71.0%. This proves the student learned more generalizable features rather than just memorizing the training set's quirks.

Limitations and Future Work

While highly effective, the current causal model assumes the teacher and student are trained on data from the same distribution. If the student's training data is significantly different, new biases might be introduced that CID isn't designed to handle. Additionally, the class-level graph matching adds computational overhead compared to vanilla KD. Future work may explore how to dynamically adjust the teacher's supervision based on the quality of the student's training data.

Summary updated

Abstract

Knowledge distillation (KD) addresses model compression by distilling knowledge from a large model (teacher) to a smaller one (student). The existing distillation approaches mainly focus on using different criteria to align the sample representations learned by the student and the teacher, while they fail to transfer the class representations. Good class representations can benefit the sample representation learning by shaping the sample representation distribution. On the other hand, the existing approaches enforce the student to fully imitate the teacher while ignoring the fact that the teacher is typically not perfect. Although the teacher has learned rich and powerful representations, it also contains unignorable bias knowledge which is usually induced by the context prior (e.g., background) in the training data. To address these two issues, in this paper, we propose comprehensive, interventional distillation (CID) that captures both sample and class representations from the teacher while removing the bias with causal intervention. Different from the existing literature that uses the softened logits of the teacher as the training targets, CID considers the softened logits as the context information of an image, which is further used to remove the biased knowledge based on causal inference. Keeping the good representations while removing the bad bias enables CID to have a better generalization ability on test data and a better transferability across different datasets against the existing state-of-the-art approaches, which is demonstrated by extensive experiments on several benchmark datasets.
BibTeX
@inproceedings{deng2021comprehensive,
  author = {Xiang Deng and Zhongfei Zhang},
  title = {{Comprehensive Knowledge Distillation with Causal Intervention}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=ch9qlCdrHD7},
}
Figure