What Mechanisms Does Knowledge Distillation Distill?

Cindy Wu, Ekdeep Singh Lubana, Bruno Kacper Mlodozeniec, Robert Kirk, David KruegerUniReps @ NeurIPS 2023 · OpenReview

Abstract

Knowledge distillation is a commonly-used compression method in ML due to the popularity of increasingly large-scale models, but it is unclear if all the information a teacher model contains is distilled into the smaller student model. We aim to formalize the concept of 'knowledge' to investigate how knowledge is transferred during distillation, focusing on shared invariant outputs to counterfactual changes of dataset latent variables (we call these latents mechanisms). We define a student model to be a good stand-in model for a teacher if it shares the teacher's learned mechanisms, and find that Jacobian matching and contrastive representation learning are viable methods by which to train such models. While these methods do not result in perfect transfer of mechanisms, we show they often improve student fidelity or mitigate simplicity bias (as measured by the teacher-to-student KL divergence and accuracy on various out-of-distribution test datasets), especially on datasets with spurious statistical correlations.

Introduction

In the current era of large-scale machine learning, Knowledge Distillation (KD) has become the standard technique for model compression. The goal is to train a compact "student" model to mimic the outputs of a larger, more complex "teacher" model, with the expectation that the student will inherit the teacher's superior reasoning and generalization capabilities. However, empirical evidence often contradicts this assumption: distilled students frequently exhibit different error patterns than their teachers or, in some cases, even outperform them. This discrepancy suggests that the student may not be learning the teacher's underlying logic, but rather finding "shortcuts" or simpler statistical correlations to match the teacher's output.

This research addresses the fundamental question: What exactly is being transferred during distillation? The authors formalize "knowledge" not merely as predictive accuracy, but as mechanistic similarity—whether the student and teacher rely on the same underlying features (latent variables) to arrive at their predictions.

Formalizing Knowledge as Mechanisms

To study knowledge transfer rigorously, the authors define a Mechanism as a specific set of latent variables within a dataset that a model utilizes for prediction. A model is said to "use" a mechanism if its prediction remains invariant when other latent variables are changed counterfactually, while the target mechanism remains fixed.

Formalizing mechanisms through counterfactual changes in latent variables.

Under this framework, a student model is considered a high-fidelity "stand-in" for the teacher if it shares the teacher's learned mechanisms. This allows for a more nuanced evaluation of distillation beyond simple accuracy metrics.

Distillation Strategies

The study compares three primary distillation methods to evaluate their effectiveness in transferring mechanisms:

  1. Standard (Base) Distillation: Minimizing the KL-divergence between the teacher’s and student’s output probability distributions (logits).
  2. Jacobian Matching: Forcing the student to match the gradients of the teacher's logits with respect to the input. This method encourages the student to match the teacher's sensitivity to input changes, effectively guiding the student to "reason" similarly across the input space.
  3. Contrastive Representation Distillation (CRD): Maximizing the mutual information between the internal representations (typically the penultimate layer) of the teacher and student models.

Experimental Setup

The researchers utilize two synthetic datasets designed to isolate and track specific predictive features:

  • Dominoes: A hybrid image dataset combining CIFAR-10 images (representing a "hard" mechanism), Fashion-MNIST images (a "simple" mechanism), and a "box" of randomized pixels (a spurious cue). This setup tests whether a student learns the complex CIFAR-10 features or succumbs to simplicity bias by relying on the "box" or Fashion-MNIST components.
  • Spurious Parity: A synthetic text dataset where a simple parity task is statistically correlated with a more complex parity task, testing the student's ability to prioritize robust mechanisms over easy shortcuts.
The Dominoes dataset used to test simplicity bias.

Key Results and Discussion

The investigation yields several critical insights into the nature of distillation:

  • The Prevalence of Simplicity Bias: Standard logit-matching is highly susceptible to simplicity bias. If a distillation dataset contains an easy shortcut (like the "box" in the Dominoes dataset), the student often ignores the teacher's complex reasoning in favor of the easier path.
  • Effectiveness of Jacobian Matching: Jacobian matching acts as a powerful regularizer. By requiring the student to match local behavior across the entire input space, it makes shortcut learning more difficult and improves the transfer of the teacher's core mechanisms, especially when the teacher was trained on cleaner data than the student.
  • Fidelity vs. Mechanism Transfer: High accuracy on the distillation set does not guarantee mechanistic similarity. Models with near-identical performance on training data often behave drastically differently when tested on out-of-distribution (OOD) data, highlighting the importance of evaluating mechanism transfer.
  • CRD Trade-offs: While Contrastive Representation Distillation often achieves the lowest KL divergence (highest behavioral fidelity), it is computationally expensive and can occasionally hinder the student from learning new, beneficial features.

Conclusion

This work demonstrates that standard knowledge distillation often fails to transfer the underlying "logic" of a teacher model, primarily due to simplicity bias. While advanced methods like Jacobian matching and CRD significantly improve mechanism transfer and mitigate bias, achieving perfect fidelity remains a challenge. Future work should focus on circuit discovery and modular representations to enable more precise "copy-pasting" of specific reasoning modules from teachers to students, particularly as we scale these insights to large language models.

Summary updated

Abstract

Knowledge distillation is a commonly-used compression method in ML due to the popularity of increasingly large-scale models, but it is unclear if all the information a teacher model contains is distilled into the smaller student model. We aim to formalize the concept of `knowledge' to investigate how knowledge is transferred during distillation, focusing on shared invariant outputs to counterfactual changes of dataset latent variables (we call these latents mechanisms). We define a student model to be a good stand-in model for a teacher if it shares the teacher's learned mechanisms, and find that Jacobian matching and contrastive representation learning are viable methods by which to train such models. While these methods do not result in perfect transfer of mechanisms, we show they often improve student fidelity or mitigate simplicity bias (as measured by the teacher-to-student KL divergence and accuracy on various out-of-distribution test datasets), especially on datasets with spurious statistical correlations.
BibTeX
@inproceedings{wu2023what,
  author = {Cindy Wu and Ekdeep Singh Lubana and Bruno Kacper Mlodozeniec and Robert Kirk and David Krueger},
  title = {{What Mechanisms Does Knowledge Distillation Distill?}},
  booktitle = {UniReps @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=uVZB637Dgx},
}

Loading related content...

Figure