Heterogeneous Asymmetric Contrastive Distillation

Qi Wang, Jinjia ZhouarXiv 2026 · arXiv

Knowledge Distillation (KD) has long been a staple for model compression, typically involving a large "teacher" network guiding a smaller "student." However, most KD methods are designed for homogeneous pairs—like a big ResNet teaching a small ResNet. When you try to have a Vision Transformer (ViT) teach a compact CNN, traditional methods often stumble.

A new paper, "HACD: Heterogeneous Asymmetric Contrastive Distillation," introduces a novel framework to bridge this architectural gap by focusing on how a teacher relates different images to one another, rather than just what specific values it predicts for a single image.

TL;DR

HACD is a logit-only distillation method designed for heterogeneous teacher-student pairs (e.g., Transformer to CNN). By mapping predictions to a shared hypersphere and using a bidirectional contrastive objective, it transfers the teacher's relational knowledge without requiring additional learnable parameters or complex feature alignment.

What Problem Does This Paper Address?

The core challenge is the representational gap. Different architectural families (CNNs, Transformers, MLPs) process information differently. As shown in the authors' CKA similarity analysis (Figure 1), internal layers of a CNN-to-CNN pair align beautifully, but CNN-to-Transformer pairs show massive discrepancies.

Figure 1: CKA similarity analysis of representations learned by homogeneous and heterogeneous architectures. Each entry shows the CKA similarity between the representations of a layer of the first network (horizontal axis) and a layer of the second network (vertical axis), with darker colors indicating higher similarity. The CNN-to-CNN pair exhibits more consistent representation patterns, whereas heterogeneous pairs involving CNNs, Transformers, and MLPs show substantial discrepancies.

Even at the output level (logits), architectures differ in "logit offset, scale, and sharpness." For instance, Transformers and MLPs tend to produce much sharper probability distributions than CNNs (Figure 2). Traditional KD tries to force the student to mimic these exact values, which can be counterproductive if the student's architecture simply can't represent them naturally.

Methods

HACD moves away from value-matching and toward relational structure transfer. The framework consists of three main pillars:

  1. Shared Unit Hypersphere Mapping: To make outputs comparable, HACD maps both teacher and student logits to "centered, temperature-softened probability directions" on a unit hypersphere. Centering removes architecture-dependent offsets, while temperature ($T$) controls how much of the teacher's sharpness is retained.
  2. Asymmetric View Construction: Contrastive learning thrives on data augmentation. However, teacher predictions on heavily distorted images can be unreliable. HACD uses an asymmetric approach: the teacher only sees a "moderate" view of an image, while the student processes both a "moderate" and a "strong" view. This ensures the supervision remains anchored to reliable targets.
  3. Bidirectional Instance Contrast: Instead of a standard KL-divergence loss, HACD uses a contrastive objective (similar to CLIP). For a given batch, the student must identify its own teacher target among all other teacher targets in the batch. This forces the student to arrange its outputs in the same relational structure as the teacher.
Overview of the HACD framework.

Datasets

The authors evaluated HACD across 27 heterogeneous teacher-student pairs using:

  • CIFAR-100: 50,000 training images across 100 classes.
  • ImageNet-1K: The standard large-scale benchmark with ~1.28 million training images.

Key Insights and Ideas

  • Relations over Values: The most critical insight is that a teacher's "dark knowledge" is better captured by how it arranges samples relative to one another than by the raw probability values it assigns.
  • The Necessity of Centering: The authors mathematically prove that without centering the probability distributions, representations tend to collapse toward a common direction, especially when predictions are close to uniform. Centering is the "secret sauce" that preserves discriminative structure.
  • Architecture-Specific Sharpness: Transformers and MLPs are naturally "sharper" than CNNs. Figure 2 shows that their non-target classes are also less semantically organized (e.g., the top-5 non-target classes are less likely to belong to the same superclass as the ground truth). HACD's temperature scaling helps mitigate the negative impact of this "unorganized tail."

Key Results

HACD consistently outperformed both vanilla KD and state-of-the-art heterogeneous distillation methods:

  • CIFAR-100: Achieved the best or second-best accuracy on 11 of 12 pairs. Remarkably, it achieved these results without using ground-truth labels during distillation, relying solely on the teacher's relational structure. It improved over vanilla KD by an average of 8.82 percentage points.
  • ImageNet-1K: Achieved the best accuracy on 14 of 15 pairs, with an average gain of 1.33 percentage points over vanilla KD.
  • Efficiency: Unlike many feature-based methods that require extra "projector" layers (adding millions of parameters), HACD adds zero learnable parameters and requires only a single teacher forward pass per sample.

Limitations and Future Work

While HACD is highly effective, the authors note a few areas for expansion:

  • Label Space: The method currently assumes teacher and student share the same label space. Extending this to cross-label space distillation would be a significant next step.
  • Dense Prediction: The current work focuses on image-level classification. Adapting the relational contrastive objective for dense tasks like object detection or segmentation (per-pixel or per-region logits) is a promising future direction.
  • Temperature Selection: While $T=4$ is a robust default, the authors suggest that dynamically selecting $T$ based on the teacher's output distribution could further optimize results.

Summary updated

Abstract

Knowledge distillation (KD) between architectural families, e.g., from a Transformer teacher to a CNN student, is difficult because the two networks learn different representations. Logits share a label space, but heterogeneous teachers differ in logit offset, scale, and sharpness, and their non-target classes are less semantically organized. Conventional logit distillation matches these values sample by sample, asking the student to imitate how another architecture expresses its predictions. We argue that a heterogeneous teacher should pass on the relational structure of its predictions, i.e., how it arranges samples relative to one another, and propose Heterogeneous Asymmetric Contrastive Distillation (HACD) to transfer this structure. HACD maps teacher and student logits to centered, temperature-softened probability directions on a shared unit hypersphere, which removes architecture-dependent offsets and lets the temperature control how sharply peaked outputs enter the representation while keeping informative non-target classes. Each student output must then identify its own teacher target within the batch. Because every teacher target also acts as a negative for the others, HACD computes the targets on a moderately augmented view and contrasts both a moderate and a strong student view against them. HACD adds no learnable parameters and needs one teacher pass per sample. It achieves the best or second-best accuracy on 11 of 12 heterogeneous pairs on CIFAR-100, without ground-truth labels, and on 14 of 15 pairs on ImageNet-1K, improving over the reported vanilla KD results by 8.82 and 1.33 percentage points on average. Checkpoints are available at https://github.com/BTAEYANG/HACD-Heterogeneous-Asymmetric-Contrastive-Distillation; the code will be released upon publication.
BibTeX
@misc{wang2026heterogeneous,
  author = {Qi Wang and Jinjia Zhou},
  title = {{Heterogeneous Asymmetric Contrastive Distillation}},
  year = {2026},
  eprint = {2611.10672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10672},
}
Figure