Introduction
Deploying high-performance Convolutional Neural Networks (CNNs) on edge devices, such as System-on-Chips (SoCs) or embedded systems, is often hindered by severe memory constraints. While standard CNNs utilize square NxN filters, these require significant line buffering to process spatial data. "On-the-fly" CNNs address this by employing 1xN filters, which process data linearly with minimal memory overhead. However, training these constrained models via traditional Knowledge Distillation (KD) is challenging due to the massive capacity gap between the NxN teacher and the 1xN student. This paper proposes Pacemaker Knowledge Distillation (PMKD) to bridge this gap using an intermediate ensemble model.
The Memory Bottleneck in Edge AI
Standard NxN convolutions (e.g., 3x3) require the system to buffer multiple rows of image pixels to compute a single output. In memory-constrained environments, this buffer can exceed available resources. On-the-fly CNNs replace these with 1xN filters, effectively eliminating the need for multi-line buffering.
While 1xN filters solve the hardware constraint, they introduce a learning bottleneck. A 1xN filter has approximately 66% fewer parameters than a 3x3 filter and is restricted to perceiving horizontal features. When a standard NxN teacher attempts to transfer its complex 2D spatial knowledge to a 1xN student, the student often fails to converge effectively, sometimes performing worse than if trained from scratch.
Pacemaker Knowledge Distillation (PMKD)
To mitigate the capacity gap, the authors introduce a "Pacemaker"—an intermediate model that facilitates a gradual transfer of knowledge. The PMKD framework operates in three distinct stages, as illustrated below.
The Three-Stage Process
- Teacher to Pacemaker: A pre-trained NxN Teacher model trains the Pacemaker. The Pacemaker is designed as an ensemble of two models: one using 1xN (horizontal) filters and another using Nx1 (vertical) filters. This ensemble can capture 2D spatial information while maintaining a structural similarity to the final student.
- Pacemaker to Student: The 1xN Student model learns from the Pacemaker ensemble. Because the Pacemaker's components share the same filter geometry as the student, the knowledge transfer is more "digestible."
- Student to Teacher: In the final stage, the Student is fine-tuned directly against the original NxN Teacher to capture any remaining high-level nuances.
The loss function incorporates Cross-Entropy, KL-divergence for logit matching, and L2 distance for hidden feature map alignment. A key advantage of this approach is that the student and teacher share the same depth, allowing for direct feature distillation without the need for additional regressor layers that might distort information.
Experimental Results
The PMKD method was evaluated on CIFAR-10, CIFAR-100, and SVHN datasets across various architectures (VGG, ResNet, WideResNet).
Accuracy and Stability
The results demonstrate that PMKD consistently outperforms both baseline training and standard KD methods. For instance, on CIFAR-10 with VGG16, standard KD actually reduced accuracy by 2.73% relative to the baseline, whereas PMKD achieved a 1.46% improvement.
Furthermore, PMKD significantly enhances training stability. In experiments with WideResNet 28-6 on CIFAR-100, standard KD exhibited high variance in final accuracy (ranging from 44.69% to 49.51%). PMKD narrowed this variance to a much tighter and higher range (50.72% to 51.63%), ensuring more reliable deployment.
Conclusion
Pacemaker Knowledge Distillation addresses the fundamental difficulty of training highly constrained on-the-fly CNNs. By introducing an intermediate ensemble pacemaker, the method successfully bridges the capacity gap between standard teachers and single-line students. This approach not only improves accuracy but also provides the training stability necessary for real-world edge AI applications.
Summary updated