Natural Statistics Of Network Activations And Implications For Knowledge Distillation

Michael Rotman, Lior Wolf

Abstract

Recent research reveals that deep neural network activations exhibit statistical regularities akin to natural images, specifically following a power law in the frequency domain. This paper explores these spectral properties and demonstrates that the spectral exponent increases linearly with network depth. By incorporating Fourier-domain loss terms—specifically a spectral $L_1$ loss and a cross-power spectrum loss—into the Knowledge Distillation (KD) framework, we enable student networks to better capture the non-local structural characteristics of teacher models. Experimental results on CIFAR-100 and ImageNet show that our method consistently achieves state-of-the-art performance across various teacher-student architectures.

Introduction

Knowledge Distillation (KD) aims to transfer knowledge from a large, high-performance "teacher" network to a compact "student" network. While traditional KD focused on matching final logit outputs, modern approaches emphasize matching intermediate feature maps. However, standard spatial metrics like $L_1$ or $L_2$ pixel-wise differences often fail to account for the intrinsic statistical structure of these activations.

Just as natural images follow a "1/f" power law, network activations exhibit universal patterns that evolve as data progresses through the model. This paper identifies these patterns and leverages them to develop a more effective similarity metric for distillation, moving beyond local pixel matching to capture global structural information.

Spectral Properties of Activations

The core of this work is the discovery that activation maps are not random but follow a power law $P(|k|) \propto |k|^\alpha$ in their power spectrum. As shown in Figure 1, the spectral exponent $\alpha$ becomes increasingly negative with depth, indicating a steeper decay and a shift toward low-frequency dominance.

Spectral analysis showing the power law behavior of input images and activation maps at different depths.

Analytically, we provide a mathematical derivation showing that the application of multiple convolutional layers naturally leads to this linear increase in the power law exponent with depth. This suggests that deeper layers focus on higher-level, global semantic information by concentrating energy in the low-frequency range.

Proposed Distillation Framework

Building on these insights, we introduce two Fourier-based loss terms to enhance the distillation process:

  1. Spectral $L_1$ Loss ($L_{L1}$): This loss is calculated on the Fourier transform of the feature maps, improving the robustness of the student's frequency-domain representation.
  2. Cross-Power Spectrum Loss ($L_{CPS}$): This term matches the rotational invariant power spectra between the teacher and student. It ensures that the student captures the same non-local structural characteristics as the teacher, which is often missed by purely local pixel-wise losses.

These terms are integrated into the "Overhaul" distillation framework, providing a comprehensive approach that matches both local details and global statistics.

Experimental Results

We evaluated our method on CIFAR-100 and ImageNet benchmarks using various teacher-student pairs.

CIFAR-100 Performance

On CIFAR-100, our spectral method outperformed previous state-of-the-art techniques in nearly all tested configurations. As detailed in the table below, our approach achieved the lowest error rates for 5 out of 6 setups, including WideResNet and PyramidNet architectures.

Comparison of distillation methods on CIFAR-100 across various teacher-student setups.

ImageNet Performance

On the large-scale ImageNet dataset, distilling a ResNet-50 teacher into a MobileNet student resulted in a Top-1 error of 27.49%. This represents a significant improvement over the baseline (31.13%) and the previous state-of-the-art "Overhaul" method (28.75%).

Conclusion

This research demonstrates that network activations are "natural" in their statistical distribution. By aligning the spectral properties of student and teacher networks, we can significantly improve the efficiency of knowledge transfer. While currently focused on CNNs for image classification, the universal nature of these power laws suggests potential applications in other domains and architectures, such as Vision Transformers, which remain avenues for future work.

Summary updated

Abstract

In a matter that is analogous to the study of natural image statistics, we study the natural statistics of the deep neural network activations at various layers. As we show, these statistics, similar to image statistics, follow a power law. We also show, both analytically and empirically, that with depth the exponent of this power law increases at a linear rate.As a direct implication of our discoveries, we present a method for performing Knowledge Distillation (KD). While classical KD methods consider the logits of the teacher network, more recent methods obtain a leap in performance by considering the activation maps. This, however, uses metrics that are suitable for comparing images. We propose to employ two additional loss terms that are based on the spectral properties of the intermediate activation maps. The proposed method obtains state of the art results on multiple image recognition KD benchmarks.
BibTeX
@inproceedings{rotman2021natural,
  author = {Michael Rotman and Lior Wolf},
  title = {{Natural Statistics Of Network Activations And Implications For Knowledge Distillation}},
  booktitle = {ICIP},
  year = {2021},
  url = {https://openreview.net/forum?id=OntLfIKYYP4X},
}

Loading related content...

Figure