CORSD: Class-Oriented Relational Self Distillation

Muzhou Yu, Sia Huat Tan, Kailu Wu, Runpei Dong, Linfeng Zhang, Kaisheng Ma

Abstract

Knowledge Distillation (KD) is a powerful technique for model compression, yet it often struggles with feature divergence between different architectures and the limitations of handcrafted relational functions. We introduce Class-Oriented Relational Self Distillation (CORSD), a novel framework that enables a model to act as its own teacher. By employing trainable relation networks and auxiliary classifiers, CORSD distills sophisticated inter-class and intra-class relationships from deep layers to shallow ones. Our approach demonstrates significant accuracy gains across CIFAR100, ImageNet, and CUB-200-2011 benchmarks with zero additional inference cost.

Introduction

Traditional Knowledge Distillation (KD) faces two primary challenges: feature divergence and simplistic relational modeling. When teacher and student architectures differ, their internal feature representations often lack alignment. Furthermore, many KD methods rely on fixed mathematical functions, such as Euclidean distance, to measure data relationships. These "handcrafted" metrics frequently fail to capture the nuanced semantic structures that deep networks utilize for classification.

To address these issues, we propose Self-Distillation (SD)—distilling knowledge from a model's deep layers to its own shallow layers—combined with a learnable, class-oriented relational framework.

Methodology: The CORSD Framework

The CORSD framework moves beyond rigid formulas by treating data relationships as learnable features.

Overview of the CORSD training framework and the internal structure of the Relation Network.

Core Components

  1. Structured Input Triplets: Training utilizes triplets consisting of an anchor image ($x_o$), a positive sample ($x_p$) from the same class, and a negative sample ($x_n$) from a different class.
  2. Trainable Relation Networks: These modules replace fixed distance metrics. They consist of a Feature Projection Layer (FPL) for specialized feature mapping and a Relational Interaction Block (RIB) that uses fully connected layers to compute relational distances.
  3. Relational Self-Distillation: The model enforces consistency between the relational patterns of shallow layers and the final, most mature layer using an $L_2$ loss.
  4. Auxiliary Classifiers: To ensure that distilled relational features remain task-relevant, small classifiers are attached to intermediate stages, guiding the network toward class-oriented representations.

Experimental Results

We evaluated CORSD on three major benchmarks: CIFAR100, ImageNet, and the fine-grained CUB-200-2011 dataset.

Performance Gains

CORSD consistently outperformed baseline models and existing state-of-the-art distillation methods:

  • CIFAR100: Average accuracy increased by 3.8% (e.g., ResNet18 improved from 77.09% to 81.10%).
  • ImageNet: Achieved a 1.5% average improvement.
  • CUB-200-2011: Observed a significant 4.5% boost, highlighting the framework's strength in fine-grained classification where class distinctions are subtle.
t-SNE visualizations comparing the baseline (BL) with CORSD components across ResNet blocks.

Visual evidence from t-SNE plots confirms that CORSD leads to more distinct and compact class clusters much earlier in the network's architecture compared to the baseline.

Discussion and Conclusion

The primary insight of CORSD is that data relationships are themselves features that benefit from neural learning. By forcing shallow layers to mirror the relational perception of deep layers, the model achieves higher efficiency. Crucially, because the auxiliary components are only used during training, the final deployed model maintains its original size and speed.

While CORSD increases training complexity—with a reported ~152% increase in training time for ResNet18—the performance benefits and zero inference overhead make it a compelling strategy for high-performance model deployment. Future work may extend this class-oriented relational approach to tasks beyond classification, such as object detection and segmentation.

Summary updated

Abstract

Knowledge distillation conducts an effective model compression method while holding some limitations:(1) the feature based distillation methods only focus on distilling the feature map but are lack of transferring the relation of data examples; (2) the relational distillation methods are either limited to the handcrafted functions for relation extraction, such as L2 norm, or weak in inter- and intra- class relation modeling. Besides, the feature divergence of heterogeneous teacher-student architectures may lead to inaccurate relational knowledge transferring. In this work, we propose a novel training framework named Class-Oriented Relational Self Distillation (CORSD) to address the limitations. The trainable relation networks are designed to extract relation of structured data input, and they enable the whole model to better classify samples by transferring the relational knowledge from the deepest layer of the model to shallow layers. Besides, auxiliary classifiers are proposed to make relation networks capture class-oriented relation that benefits classification task. Experiments demonstrate that CORSD achieves remarkable improvements. Compared to baseline, 3.8%, 1.5% and 4.5% averaged accuracy boost can be observed on CIFAR100, ImageNet and CUB-200-2011, respectively.
BibTeX
@inproceedings{yu2023corsd,
  author = {Muzhou Yu and Sia Huat Tan and Kailu Wu and Runpei Dong and Linfeng Zhang and Kaisheng Ma},
  title = {{CORSD: Class-Oriented Relational Self Distillation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=4KZ-vZTJtn},
}
Figure