Abstract
Knowledge Distillation (KD) is a powerful technique for model compression, yet it often struggles with feature divergence between different architectures and the limitations of handcrafted relational functions. We introduce Class-Oriented Relational Self Distillation (CORSD), a novel framework that enables a model to act as its own teacher. By employing trainable relation networks and auxiliary classifiers, CORSD distills sophisticated inter-class and intra-class relationships from deep layers to shallow ones. Our approach demonstrates significant accuracy gains across CIFAR100, ImageNet, and CUB-200-2011 benchmarks with zero additional inference cost.
Introduction
Traditional Knowledge Distillation (KD) faces two primary challenges: feature divergence and simplistic relational modeling. When teacher and student architectures differ, their internal feature representations often lack alignment. Furthermore, many KD methods rely on fixed mathematical functions, such as Euclidean distance, to measure data relationships. These "handcrafted" metrics frequently fail to capture the nuanced semantic structures that deep networks utilize for classification.
To address these issues, we propose Self-Distillation (SD)—distilling knowledge from a model's deep layers to its own shallow layers—combined with a learnable, class-oriented relational framework.
Methodology: The CORSD Framework
The CORSD framework moves beyond rigid formulas by treating data relationships as learnable features.
Core Components
- Structured Input Triplets: Training utilizes triplets consisting of an anchor image ($x_o$), a positive sample ($x_p$) from the same class, and a negative sample ($x_n$) from a different class.
- Trainable Relation Networks: These modules replace fixed distance metrics. They consist of a Feature Projection Layer (FPL) for specialized feature mapping and a Relational Interaction Block (RIB) that uses fully connected layers to compute relational distances.
- Relational Self-Distillation: The model enforces consistency between the relational patterns of shallow layers and the final, most mature layer using an $L_2$ loss.
- Auxiliary Classifiers: To ensure that distilled relational features remain task-relevant, small classifiers are attached to intermediate stages, guiding the network toward class-oriented representations.
Experimental Results
We evaluated CORSD on three major benchmarks: CIFAR100, ImageNet, and the fine-grained CUB-200-2011 dataset.
Performance Gains
CORSD consistently outperformed baseline models and existing state-of-the-art distillation methods:
- CIFAR100: Average accuracy increased by 3.8% (e.g., ResNet18 improved from 77.09% to 81.10%).
- ImageNet: Achieved a 1.5% average improvement.
- CUB-200-2011: Observed a significant 4.5% boost, highlighting the framework's strength in fine-grained classification where class distinctions are subtle.
Visual evidence from t-SNE plots confirms that CORSD leads to more distinct and compact class clusters much earlier in the network's architecture compared to the baseline.
Discussion and Conclusion
The primary insight of CORSD is that data relationships are themselves features that benefit from neural learning. By forcing shallow layers to mirror the relational perception of deep layers, the model achieves higher efficiency. Crucially, because the auxiliary components are only used during training, the final deployed model maintains its original size and speed.
While CORSD increases training complexity—with a reported ~152% increase in training time for ResNet18—the performance benefits and zero inference overhead make it a compelling strategy for high-performance model deployment. Future work may extend this class-oriented relational approach to tasks beyond classification, such as object detection and segmentation.
Summary updated