Abstract
Knowledge distillation (KD) is an effective method for model compression and transferring knowledge between models. However, its effect on model's robustness against spurious correlations that degrade performance on out-of-distribution data remains underexplored. This study investigates the effect of knowledge distillation on the transferability of "debiasing" capabilities from teacher models to student models on natural language inference (NLI) and image classification tasks. Through extensive experiments, we illustrate several key findings: (i) overall the debiasing capability of a model is undermined post-KD; (ii) training a debiased model does not benefit from injecting teacher knowledge; (iii) although the overall robustness of a model may remain stable post-distillation, significant variations can occur across different types of biases; and (iv) we pin-point the internal attention pattern and circuit that causes the distinct behavior post-KD. Given the above findings, we propose three effective solutions to improve the distillability of debiasing methods: developing high quality data for augmentation, implementing iterative knowledge distillation, and initializing student models with weights obtained from teacher models. Our findings provide understandings on how KD works and how to design better debiasing methods.
Introduction
Machine learning models are notorious for taking "shortcuts"—relying on spurious correlations, such as associating a specific background with a label, rather than learning robust, underlying features. While various "debiasing" methods have been developed to mitigate this, the resulting robust models are often too large for edge deployment. Knowledge Distillation (KD), where a small "student" model mimics a large "teacher," is the standard solution for compression. However, this paper identifies a critical gap: it is unknown if the student inherits the teacher's hard-won robustness. This study investigates the "distillability" of bias mitigation across different model scales and tasks.
Methodology
The researchers established a rigorous framework to compare models across three dimensions: Teacher vs. Student (C1), Non-KD vs. KD (C2), and internal mechanisms. The study evaluates both Natural Language Understanding (NLU) and Computer Vision tasks using datasets like MNLI, QQP, CelebA, and Waterbird. Specialized "hard" sets (e.g., HANS, PAWS) were utilized to test for out-of-distribution (OOD) robustness.
To analyze internal logic, the study employed Centralized Kernel Alignment (CKA) to compare activation patterns and Circuit Discovery via Edge-level Attribution Patching to see if the "logic gates" of the models align.
Key Insights and Results
The Distillability Gap
Debiasing is fragile. Even when a teacher is robust, the student often fails to capture the nuance of why the teacher made a decision. Students frequently focus on matching final logits, which can inadvertently re-inject bias. Surprisingly, students often amplify the residual biases of the teacher; if a teacher is 90% debiased, the student might only capture the 10% of bias that remains because it is "easier" to learn.
Scale and Performance
KD effectiveness is highly sensitive to model scale. When the scale gap is too large (e.g., a 24-layer teacher vs. a 2-layer student), the student lacks the capacity to mirror the teacher’s complex internal circuits. In vision tasks, the "Spurious Gap" (the performance difference between easy and hard data) was often 12.7 points higher in students compared to teachers.
Furthermore, models trained from scratch without KD (Non-KD) were often less biased than those trained with KD, although they had lower overall accuracy. This suggests the distillation objective itself might distract the model from learning robust features.
Proposed Solutions
To bridge the distillability gap, the authors propose three strategies:
- Data Augmentation (DA): Using high-quality, balanced datasets to ensure the student sees fewer shortcuts during distillation. This reduced the spurious gap by 4.5 absolute points.
- Iterative KD (IKD): Distilling through intermediate sizes (e.g., Large → Base → Small) helps preserve robustness, providing a 2.8 point improvement.
- Weight Initialization (Init): Initializing the student with a subset of teacher weights offered a modest 1.2 point gain by giving the student a head start in representation space.
Conclusion
This study reveals that standard logit-based distillation often fails to transfer debiasing capabilities. While students can mimic teachers on easy data, their internal representations diverge on hard, OOD samples. Future work should investigate feature-based distillation and self-distillation to further improve the robustness of compressed models.
Summary updated