Abstract
Standard knowledge distillation (KD) often suffers from "spatial amnesia" because it relies on global average pooling, which discards the rich local structures of an image. We introduce GLD (Global and Local logit Distillation), a method that transfers not just the final class scores, but also localized "local logits" and the dense web of relationships between them. By preserving these spatial and relational cues, GLD significantly boosts student performance across image classification, object detection, and semantic segmentation.
Introduction: The Problem of Spatial Information Loss
Knowledge distillation is the process of training a small "student" network to mimic a large, high-performing "teacher." Most logit-based distillation methods focus on the final output of the network. However, modern classifiers typically use Global Average Pooling (GAP) right before the final layer.
While GAP is effective for making models invariant to object position, it is a "destructive" process for distillation: it collapses all spatial information into a single vector. The student learns what the teacher thinks is in the image, but loses the teacher's nuanced understanding of where specific features are located or how different parts of a scene relate to one another. GLD addresses this by capturing what GAP typically throws away.
The GLD Framework
The authors propose a three-pronged approach to capture and transfer localized knowledge:
1. Local Logits
Instead of pooling the entire penultimate feature map into one vector, GLD divides it into a grid (e.g., $2 \times 2$ or $4 \times 4$). Each grid cell is treated as a "local feature" and passed through the same classifier used for the global feature. This produces "local logits" that represent the model's class-wise confidence for specific regions of the image.
2. Densely Connected Relations
The core innovation is transferring the relationships between these logits. GLD constructs a distance matrix that captures how every logit (global and local) in a mini-batch relates to every other logit. This includes intra-relationships (how a local part relates to the global whole) and inter-relationships (how features in one image relate to those in another).
3. Adaptive Distillation Loss ($L_{ND}$)
Standard KD uses a fixed "temperature" ($\tau$) to soften logit distributions. Because global and local logits have different statistical properties, GLD uses the standard deviation of the logits as an adaptive softening factor. This ensures the student can effectively learn from both high-confidence global scores and more varied local scores.
Key Results and Performance
GLD was evaluated across a wide range of benchmarks, including CIFAR-100, ImageNet, and downstream tasks like Object Detection and Semantic Segmentation.
As shown in the experimental results, GLD consistently outperforms nine other state-of-the-art distillation methods (such as CRD, RKD, and AT) across various architectures. For a ResNet110 teacher and ResNet20 student (Setup a), GLD achieved 71.37% accuracy, compared to 70.27% for standard KD. On ImageNet, GLD improved a ResNet18 student's Top-1 accuracy by 1.88% over the baseline.
Beyond classification, students trained with GLD proved to be better "backbones" for other tasks. In Object Detection on VOC2007, the GLD-distilled student reached 71.83 mAP, significantly higher than the baseline (70.97) and other KD methods.
Conclusion
GLD demonstrates that logits can act as powerful, low-dimensional proxies for spatial features when applied locally. By focusing on co-activation patterns and dense relationships, GLD allows student networks to inherit a much richer representation of the teacher's logic. While grid size selection adds a tuning step for fine-grained tasks, the performance gains across diverse benchmarks establish GLD as a robust new standard for logit-based distillation.
Summary updated