Abstract
Knowledge distillation is a commonly-used compression method in ML due to the popularity of increasingly large-scale models, but it is unclear if all the information a teacher model contains is distilled into the smaller student model. We aim to formalize the concept of 'knowledge' to investigate how knowledge is transferred during distillation, focusing on shared invariant outputs to counterfactual changes of dataset latent variables (we call these latents mechanisms). We define a student model to be a good stand-in model for a teacher if it shares the teacher's learned mechanisms, and find that Jacobian matching and contrastive representation learning are viable methods by which to train such models. While these methods do not result in perfect transfer of mechanisms, we show they often improve student fidelity or mitigate simplicity bias (as measured by the teacher-to-student KL divergence and accuracy on various out-of-distribution test datasets), especially on datasets with spurious statistical correlations.
Introduction
In the current era of large-scale machine learning, Knowledge Distillation (KD) has become the standard technique for model compression. The goal is to train a compact "student" model to mimic the outputs of a larger, more complex "teacher" model, with the expectation that the student will inherit the teacher's superior reasoning and generalization capabilities. However, empirical evidence often contradicts this assumption: distilled students frequently exhibit different error patterns than their teachers or, in some cases, even outperform them. This discrepancy suggests that the student may not be learning the teacher's underlying logic, but rather finding "shortcuts" or simpler statistical correlations to match the teacher's output.
This research addresses the fundamental question: What exactly is being transferred during distillation? The authors formalize "knowledge" not merely as predictive accuracy, but as mechanistic similarity—whether the student and teacher rely on the same underlying features (latent variables) to arrive at their predictions.
Formalizing Knowledge as Mechanisms
To study knowledge transfer rigorously, the authors define a Mechanism as a specific set of latent variables within a dataset that a model utilizes for prediction. A model is said to "use" a mechanism if its prediction remains invariant when other latent variables are changed counterfactually, while the target mechanism remains fixed.
Under this framework, a student model is considered a high-fidelity "stand-in" for the teacher if it shares the teacher's learned mechanisms. This allows for a more nuanced evaluation of distillation beyond simple accuracy metrics.
Distillation Strategies
The study compares three primary distillation methods to evaluate their effectiveness in transferring mechanisms:
- Standard (Base) Distillation: Minimizing the KL-divergence between the teacher’s and student’s output probability distributions (logits).
- Jacobian Matching: Forcing the student to match the gradients of the teacher's logits with respect to the input. This method encourages the student to match the teacher's sensitivity to input changes, effectively guiding the student to "reason" similarly across the input space.
- Contrastive Representation Distillation (CRD): Maximizing the mutual information between the internal representations (typically the penultimate layer) of the teacher and student models.
Experimental Setup
The researchers utilize two synthetic datasets designed to isolate and track specific predictive features:
- Dominoes: A hybrid image dataset combining CIFAR-10 images (representing a "hard" mechanism), Fashion-MNIST images (a "simple" mechanism), and a "box" of randomized pixels (a spurious cue). This setup tests whether a student learns the complex CIFAR-10 features or succumbs to simplicity bias by relying on the "box" or Fashion-MNIST components.
- Spurious Parity: A synthetic text dataset where a simple parity task is statistically correlated with a more complex parity task, testing the student's ability to prioritize robust mechanisms over easy shortcuts.
Key Results and Discussion
The investigation yields several critical insights into the nature of distillation:
- The Prevalence of Simplicity Bias: Standard logit-matching is highly susceptible to simplicity bias. If a distillation dataset contains an easy shortcut (like the "box" in the Dominoes dataset), the student often ignores the teacher's complex reasoning in favor of the easier path.
- Effectiveness of Jacobian Matching: Jacobian matching acts as a powerful regularizer. By requiring the student to match local behavior across the entire input space, it makes shortcut learning more difficult and improves the transfer of the teacher's core mechanisms, especially when the teacher was trained on cleaner data than the student.
- Fidelity vs. Mechanism Transfer: High accuracy on the distillation set does not guarantee mechanistic similarity. Models with near-identical performance on training data often behave drastically differently when tested on out-of-distribution (OOD) data, highlighting the importance of evaluating mechanism transfer.
- CRD Trade-offs: While Contrastive Representation Distillation often achieves the lowest KL divergence (highest behavioral fidelity), it is computationally expensive and can occasionally hinder the student from learning new, beneficial features.
Conclusion
This work demonstrates that standard knowledge distillation often fails to transfer the underlying "logic" of a teacher model, primarily due to simplicity bias. While advanced methods like Jacobian matching and CRD significantly improve mechanism transfer and mitigate bias, achieving perfect fidelity remains a challenge. Future work should focus on circuit discovery and modular representations to enable more precise "copy-pasting" of specific reasoning modules from teachers to students, particularly as we scale these insights to large language models.
Summary updated