Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data

Zhijian Li, Biao Yang, Penghang Yin, Yingyong Qi, Jack Xin

TL;DR

This paper introduces Feature Affinity Assisted Knowledge Distillation (FAQD), a novel method for training quantized neural networks using unlabeled data. By guiding the student network through intermediate feature maps of a teacher network, FAQD significantly improves accuracy and enables effective quantization even without ground-truth labels. They also propose a faster approximation, FFA loss, to make this approach practical for high-resolution images.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are powerful but computationally expensive, requiring significant memory and processing power. Quantization, which reduces the precision of network weights and activations, is a key technique for making DNNs more efficient. However, quantization often leads to a drop in accuracy. Knowledge Distillation (KD) is a method to mitigate this by training a smaller "student" network to mimic a larger, pre-trained "teacher" network. Conventional KD typically focuses on matching the final output (logits) of the networks. This paper addresses the limitation that traditional KD requires labeled data, which is expensive and time-consuming to obtain. They propose a method that leverages KD for quantization but operates on unlabeled data by focusing on intermediate feature maps, effectively teaching the student the "steps" of the solution rather than just the final answer.

Methods

The core of the paper is the Feature Affinity (FA) Assisted Knowledge Distillation (FAQD) framework. Unlike standard KD that only matches output logits, FAQD also enforces similarity between intermediate feature maps of the teacher and student networks. This is achieved through a Feature Affinity (FA) loss, which measures the angular distance between feature map pixels. This loss acts as a "teaching signal" for the student's intermediate layers.

The authors propose two main objectives:

  • Supervised FAQD: Combines a quantized KD loss (using either MSE or KL divergence, with MSE found to be better for end-to-end training) with the FA loss and a standard ground-truth label loss (NLL).
  • Label-Free FAQD: Uses only the quantized KD loss and the FA loss, making it applicable when labels are unavailable.

To address the computational cost of the FA loss, especially for high-resolution images, they introduce Fast Feature Affinity (FFA) loss. This is a randomized approximation of the FA loss using low-rank random matrices, significantly reducing computational complexity from O(W²H²) to O(kWH), where k is the number of random vectors used.

Datasets

The paper evaluates their methods on the following datasets:

  • CIFAR-10
  • CIFAR-100
  • Tiny ImageNet

Key Insights and Ideas

  • Intermediate Feature Matching is Crucial for Quantization: Teaching the student network to mimic intermediate feature maps (feature affinity) provides a stronger supervisory signal than just matching final outputs, leading to better accuracy for quantized models.
  • Unlabeled Data is Sufficient for Effective Quantization: The proposed label-free FAQD demonstrates that high-quality quantization can be achieved without relying on expensive ground-truth labels, making it highly practical.
  • MSE Outperforms KL for End-to-End Quantization: For training quantized models from scratch, Mean Squared Error (MSE) as a distillation loss yields better results than Kullback-Leibler (KL) divergence.
  • FFA Loss Enables Scalability: The Fast Feature Affinity (FFA) loss is a critical innovation that makes the FA-assisted distillation practical for high-resolution images by drastically reducing computational overhead.
  • Theoretical Justification for FA Loss: The paper provides theoretical grounding for the FA loss using a Johnson-Lindenstrauss lemma for angular distances, explaining why it can preserve pairwise distances in lower-dimensional embeddings.

Key Results

  • FAQD, both supervised and label-free, consistently outperforms existing quantization-aware training (QAT) methods on CIFAR-10, CIFAR-100, and Tiny ImageNet.
  • On CIFAR-10, their 4-bit quantized student network achieves accuracy comparable to or exceeding its full-precision counterpart.
  • The label-free FAQD method achieves competitive results, even surpassing some supervised methods.
  • The FFA loss accelerates training significantly while maintaining performance close to the exact FA loss, especially with a proper ensemble size (k=5 or 15). For example, on Tiny ImageNet, FFA loss with k=15 achieved 61.12% accuracy, approaching the exact FA loss performance.

Limitations and Future Work

  • FFA Loss Variance: While FFA loss is faster, it introduces variance, and a small ensemble size (k=1) can lead to inconsistent results. The paper suggests choosing an appropriate ensemble size (k=5 or 15) to stabilize performance.
  • Computational Cost of Exact FA Loss: The exact FA loss remains computationally expensive for very high-resolution images, necessitating the use of approximations like FFA.
  • Future Work: The paper suggests that FFA loss can accelerate pairwise distance computations in other areas, hinting at broader applications beyond quantization. The theoretical analysis of FFA loss convergence and error bounds provides a foundation for further research in this direction.

Summary updated

Abstract

In this paper, we propose a feature affinity (FA) assisted knowledge distillation (KD) method to improve quantization-aware training of deep neural networks (DNN). The FA loss on intermediate feature maps of DNNs plays the role of teaching middle steps of a solution to a student instead of only giving final answers in the conventional KD where the loss acts on the network logits at the output level. Combining logit loss and FA loss, we found that the quantized student network receives stronger supervision than from the labeled ground-truth data. The resulting FAQD is capable of compressing model on label-free data, which brings immediate practical benefits as pre-trained teacher models are readily available and unlabeled data are abundant. In contrast, data labeling is often laborious and expensive. Finally, we propose a fast feature affinity (FFA) loss that accurately approximates FA loss with a lower order of computational complexity, which helps speed up training for high resolution image input.
BibTeX
@article{li2023feature,
  author = {Zhijian Li and Biao Yang and Penghang Yin and Yingyong Qi and Jack Xin},
  title = {{Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3297890},
}
Figure