Pacemaker: Intermediate Teacher Knowledge Distillation For On-The-Fly Convolutional Neural Network

Wonchul Son, Youngbin Kim, Wonseok Song, Youngsu Moon, Wonjun HwangarXiv 2020 · OpenReview

Introduction

Deploying high-performance Convolutional Neural Networks (CNNs) on edge devices, such as System-on-Chips (SoCs) or embedded systems, is often hindered by severe memory constraints. While standard CNNs utilize square NxN filters, these require significant line buffering to process spatial data. "On-the-fly" CNNs address this by employing 1xN filters, which process data linearly with minimal memory overhead. However, training these constrained models via traditional Knowledge Distillation (KD) is challenging due to the massive capacity gap between the NxN teacher and the 1xN student. This paper proposes Pacemaker Knowledge Distillation (PMKD) to bridge this gap using an intermediate ensemble model.

The Memory Bottleneck in Edge AI

Standard NxN convolutions (e.g., 3x3) require the system to buffer multiple rows of image pixels to compute a single output. In memory-constrained environments, this buffer can exceed available resources. On-the-fly CNNs replace these with 1xN filters, effectively eliminating the need for multi-line buffering.

Comparison of memory requirements for standard NxN convolution vs. on-the-fly 1xN convolution.

While 1xN filters solve the hardware constraint, they introduce a learning bottleneck. A 1xN filter has approximately 66% fewer parameters than a 3x3 filter and is restricted to perceiving horizontal features. When a standard NxN teacher attempts to transfer its complex 2D spatial knowledge to a 1xN student, the student often fails to converge effectively, sometimes performing worse than if trained from scratch.

Pacemaker Knowledge Distillation (PMKD)

To mitigate the capacity gap, the authors introduce a "Pacemaker"—an intermediate model that facilitates a gradual transfer of knowledge. The PMKD framework operates in three distinct stages, as illustrated below.

The PMKD framework featuring the Teacher, the intermediate Ensemble Pacemaker, and the Student model.

The Three-Stage Process

  1. Teacher to Pacemaker: A pre-trained NxN Teacher model trains the Pacemaker. The Pacemaker is designed as an ensemble of two models: one using 1xN (horizontal) filters and another using Nx1 (vertical) filters. This ensemble can capture 2D spatial information while maintaining a structural similarity to the final student.
  2. Pacemaker to Student: The 1xN Student model learns from the Pacemaker ensemble. Because the Pacemaker's components share the same filter geometry as the student, the knowledge transfer is more "digestible."
  3. Student to Teacher: In the final stage, the Student is fine-tuned directly against the original NxN Teacher to capture any remaining high-level nuances.

The loss function incorporates Cross-Entropy, KL-divergence for logit matching, and L2 distance for hidden feature map alignment. A key advantage of this approach is that the student and teacher share the same depth, allowing for direct feature distillation without the need for additional regressor layers that might distort information.

Experimental Results

The PMKD method was evaluated on CIFAR-10, CIFAR-100, and SVHN datasets across various architectures (VGG, ResNet, WideResNet).

Accuracy and Stability

The results demonstrate that PMKD consistently outperforms both baseline training and standard KD methods. For instance, on CIFAR-10 with VGG16, standard KD actually reduced accuracy by 2.73% relative to the baseline, whereas PMKD achieved a 1.46% improvement.

Furthermore, PMKD significantly enhances training stability. In experiments with WideResNet 28-6 on CIFAR-100, standard KD exhibited high variance in final accuracy (ranging from 44.69% to 49.51%). PMKD narrowed this variance to a much tighter and higher range (50.72% to 51.63%), ensuring more reliable deployment.

Conclusion

Pacemaker Knowledge Distillation addresses the fundamental difficulty of training highly constrained on-the-fly CNNs. By introducing an intermediate ensemble pacemaker, the method successfully bridges the capacity gap between standard teachers and single-line students. This approach not only improves accuracy but also provides the training stability necessary for real-world edge AI applications.

Summary updated

Abstract

There is a need for an on-the-fly computational process with very low performance system such as system-on-chip (SoC) and embedded device etc. This paper presents pacemaker knowledge distillation as intermediate ensemble teacher to use convolutional neural network in these systems. For on-the-fly system, we consider student model using 1xN shape on-the-fly filter and teacher model using normal NxN shape filter. We note three points about training student model, caused by applying on-the-fly filter. First, same depth but unavoidable thin model compression. Second, the large capacity gap and parameter size gap due to only the horizontal field must be selected not the vertical receptive. Third, the performance instability and degradation of direct distilling. To solve these problems, we propose intermediate teacher, named pacemaker, for an on-the-fly student. So, student can be trained from pacemaker and original teacher step by step. Experiments prove our proposed method make significant performance (accuracy) improvements: on CIFAR100, 5.39% increased in WRN-40-4 than conventional knowledge distillation which shows even low performance than baseline. And we solve train instability, occurred when conventional knowledge distillation was applied without proposed method, by reducing deviation range by applying proposed method pacemaker knowledge distillation.
BibTeX
@misc{son2020pacemaker,
  author = {Wonchul Son and Youngbin Kim and Wonseok Song and Youngsu Moon and Wonjun Hwang},
  title = {{Pacemaker: Intermediate Teacher Knowledge Distillation For On-The-Fly Convolutional Neural Network}},
  year = {2020},
  eprint = {2003.03944},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.03944},
}
Figure