Distilling Global and Local Logits with Densely Connected Relations

Youmin Kim, Jinbae Park, Younho Jang, Muhammad Salman Ali, Tae-Hyun Oh, Sung-Ho Bae

Abstract

Standard knowledge distillation (KD) often suffers from "spatial amnesia" because it relies on global average pooling, which discards the rich local structures of an image. We introduce GLD (Global and Local logit Distillation), a method that transfers not just the final class scores, but also localized "local logits" and the dense web of relationships between them. By preserving these spatial and relational cues, GLD significantly boosts student performance across image classification, object detection, and semantic segmentation.

Introduction: The Problem of Spatial Information Loss

Knowledge distillation is the process of training a small "student" network to mimic a large, high-performing "teacher." Most logit-based distillation methods focus on the final output of the network. However, modern classifiers typically use Global Average Pooling (GAP) right before the final layer.

While GAP is effective for making models invariant to object position, it is a "destructive" process for distillation: it collapses all spatial information into a single vector. The student learns what the teacher thinks is in the image, but loses the teacher's nuanced understanding of where specific features are located or how different parts of a scene relate to one another. GLD addresses this by capturing what GAP typically throws away.

The GLD Framework

The authors propose a three-pronged approach to capture and transfer localized knowledge:

Overview of the GLD framework showing the distillation of local/global logits and their relationships.

1. Local Logits

Instead of pooling the entire penultimate feature map into one vector, GLD divides it into a grid (e.g., $2 \times 2$ or $4 \times 4$). Each grid cell is treated as a "local feature" and passed through the same classifier used for the global feature. This produces "local logits" that represent the model's class-wise confidence for specific regions of the image.

2. Densely Connected Relations

The core innovation is transferring the relationships between these logits. GLD constructs a distance matrix that captures how every logit (global and local) in a mini-batch relates to every other logit. This includes intra-relationships (how a local part relates to the global whole) and inter-relationships (how features in one image relate to those in another).

3. Adaptive Distillation Loss ($L_{ND}$)

Standard KD uses a fixed "temperature" ($\tau$) to soften logit distributions. Because global and local logits have different statistical properties, GLD uses the standard deviation of the logits as an adaptive softening factor. This ensures the student can effectively learn from both high-confidence global scores and more varied local scores.

Key Results and Performance

GLD was evaluated across a wide range of benchmarks, including CIFAR-100, ImageNet, and downstream tasks like Object Detection and Semantic Segmentation.

Comparison of GLD against state-of-the-art distillation methods across different architectural setups.

As shown in the experimental results, GLD consistently outperforms nine other state-of-the-art distillation methods (such as CRD, RKD, and AT) across various architectures. For a ResNet110 teacher and ResNet20 student (Setup a), GLD achieved 71.37% accuracy, compared to 70.27% for standard KD. On ImageNet, GLD improved a ResNet18 student's Top-1 accuracy by 1.88% over the baseline.

Beyond classification, students trained with GLD proved to be better "backbones" for other tasks. In Object Detection on VOC2007, the GLD-distilled student reached 71.83 mAP, significantly higher than the baseline (70.97) and other KD methods.

Conclusion

GLD demonstrates that logits can act as powerful, low-dimensional proxies for spatial features when applied locally. By focusing on co-activation patterns and dense relationships, GLD allows student networks to inherit a much richer representation of the teacher's logic. While grid size selection adds a tuning step for fine-grained tasks, the performance gains across diverse benchmarks establish GLD as a robust new standard for logit-based distillation.

Summary updated

Abstract

In prevalent knowledge distillation, logits in most image recognition models are computed by global average pooling, then used to learn to encode the high-level and task-relevant knowledge. In this work, we solve the limitation of this global logit transfer in this distillation context. We point out that it prevents the transfer of informative spatial information, which provides localized knowledge as well as rich relational information across contexts of an input scene. To exploit the rich spatial information, we propose a simple yet effective logit distillation approach. We add a local spatial pooling layer branch to the penultimate layer, thereby our method extends the standard logit distillation and enables learning of both finely-localized knowledge and holistic representation. Our proposed method shows favorable accuracy improvement against the state-of-the-art methods on several image classification datasets. We show that our distilled students trained on the image classification task can be successfully leveraged for object detection and semantic segmentation tasks; this result demonstrates our method’s high transferability.
BibTeX
@inproceedings{kim2021distilling,
  author = {Youmin Kim and Jinbae Park and Younho Jang and Muhammad Salman Ali and Tae-Hyun Oh and Sung-Ho Bae},
  title = {{Distilling Global and Local Logits with Densely Connected Relations}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00623},
}
Figure