Knowledge Distillation (KD) has long been a staple for model compression, typically involving a large "teacher" network guiding a smaller "student." However, most KD methods are designed for homogeneous pairs—like a big ResNet teaching a small ResNet. When you try to have a Vision Transformer (ViT) teach a compact CNN, traditional methods often stumble.
A new paper, "HACD: Heterogeneous Asymmetric Contrastive Distillation," introduces a novel framework to bridge this architectural gap by focusing on how a teacher relates different images to one another, rather than just what specific values it predicts for a single image.
TL;DR
HACD is a logit-only distillation method designed for heterogeneous teacher-student pairs (e.g., Transformer to CNN). By mapping predictions to a shared hypersphere and using a bidirectional contrastive objective, it transfers the teacher's relational knowledge without requiring additional learnable parameters or complex feature alignment.
What Problem Does This Paper Address?
The core challenge is the representational gap. Different architectural families (CNNs, Transformers, MLPs) process information differently. As shown in the authors' CKA similarity analysis (Figure 1), internal layers of a CNN-to-CNN pair align beautifully, but CNN-to-Transformer pairs show massive discrepancies.
Even at the output level (logits), architectures differ in "logit offset, scale, and sharpness." For instance, Transformers and MLPs tend to produce much sharper probability distributions than CNNs (Figure 2). Traditional KD tries to force the student to mimic these exact values, which can be counterproductive if the student's architecture simply can't represent them naturally.
Methods
HACD moves away from value-matching and toward relational structure transfer. The framework consists of three main pillars:
- Shared Unit Hypersphere Mapping: To make outputs comparable, HACD maps both teacher and student logits to "centered, temperature-softened probability directions" on a unit hypersphere. Centering removes architecture-dependent offsets, while temperature ($T$) controls how much of the teacher's sharpness is retained.
- Asymmetric View Construction: Contrastive learning thrives on data augmentation. However, teacher predictions on heavily distorted images can be unreliable. HACD uses an asymmetric approach: the teacher only sees a "moderate" view of an image, while the student processes both a "moderate" and a "strong" view. This ensures the supervision remains anchored to reliable targets.
- Bidirectional Instance Contrast: Instead of a standard KL-divergence loss, HACD uses a contrastive objective (similar to CLIP). For a given batch, the student must identify its own teacher target among all other teacher targets in the batch. This forces the student to arrange its outputs in the same relational structure as the teacher.
Datasets
The authors evaluated HACD across 27 heterogeneous teacher-student pairs using:
- CIFAR-100: 50,000 training images across 100 classes.
- ImageNet-1K: The standard large-scale benchmark with ~1.28 million training images.
Key Insights and Ideas
- Relations over Values: The most critical insight is that a teacher's "dark knowledge" is better captured by how it arranges samples relative to one another than by the raw probability values it assigns.
- The Necessity of Centering: The authors mathematically prove that without centering the probability distributions, representations tend to collapse toward a common direction, especially when predictions are close to uniform. Centering is the "secret sauce" that preserves discriminative structure.
- Architecture-Specific Sharpness: Transformers and MLPs are naturally "sharper" than CNNs. Figure 2 shows that their non-target classes are also less semantically organized (e.g., the top-5 non-target classes are less likely to belong to the same superclass as the ground truth). HACD's temperature scaling helps mitigate the negative impact of this "unorganized tail."
Key Results
HACD consistently outperformed both vanilla KD and state-of-the-art heterogeneous distillation methods:
- CIFAR-100: Achieved the best or second-best accuracy on 11 of 12 pairs. Remarkably, it achieved these results without using ground-truth labels during distillation, relying solely on the teacher's relational structure. It improved over vanilla KD by an average of 8.82 percentage points.
- ImageNet-1K: Achieved the best accuracy on 14 of 15 pairs, with an average gain of 1.33 percentage points over vanilla KD.
- Efficiency: Unlike many feature-based methods that require extra "projector" layers (adding millions of parameters), HACD adds zero learnable parameters and requires only a single teacher forward pass per sample.
Limitations and Future Work
While HACD is highly effective, the authors note a few areas for expansion:
- Label Space: The method currently assumes teacher and student share the same label space. Extending this to cross-label space distillation would be a significant next step.
- Dense Prediction: The current work focuses on image-level classification. Adapting the relational contrastive objective for dense tasks like object detection or segmentation (per-pixel or per-region logits) is a promising future direction.
- Temperature Selection: While $T=4$ is a robust default, the authors suggest that dynamically selecting $T$ based on the teacher's output distribution could further optimize results.
Summary updated