Abstract
Deep Neural Networks (DNNs) have revolutionized audio classification, yet their deployment on resource-constrained edge devices remains challenging due to high computational demands. This paper introduces a novel adversarial feature distillation framework designed to compress large teacher models into efficient student networks without compromising accuracy. By forcing the student to mimic the internal feature representations of the teacher through a Generative Adversarial Network (GAN) style game, the proposed method achieves a 76:1 reduction in computational complexity (FLOPS) while improving classification performance on key benchmarks.
Introduction
The rise of "machine listening" has enabled applications ranging from acoustic scene classification to keyword spotting. However, the high-performance models required for these tasks often carry a heavy "computational tax," making them unsuitable for smartphones or IoT sensors. Knowledge Distillation (KD) has emerged as a popular solution, where a smaller "student" network learns from a larger "teacher."
Traditional KD typically focuses on "soft labels"—the final probability outputs of the teacher. However, for complex and noisy audio data, these labels often discard critical structural information. This paper addresses the challenge of transferring the "internal logic" or feature maps of a complex model to a simpler one, even when their architectures differ significantly.
The Shift from Labels to Features
Standard Knowledge Distillation (KD) relies on the student matching the teacher's final output. In contrast, Feature Distillation (FD) focuses on intermediate layers, which contain richer information about the data's underlying structure.
As shown in the figure above, while KD (a) minimizes similarity loss at the output level, FD (b) attempts to align internal feature maps. The authors argue that for audio tasks, these intermediate representations are more robust than final labels alone.
Adversarial Feature Distillation Framework
The proposed framework introduces several key innovations to bridge the gap between teacher and student architectures:
- Multi-level Feature Distillation: The student mimics the teacher’s intermediate feature maps across multiple layers, capturing a hierarchy of acoustic features.
- Alignment Units: To handle architectural mismatches (e.g., a student with fewer channels or smaller layers), adaptive pooling is used to "resize" feature maps for direct comparison.
- Adversarial Learning Strategy: This is the core "secret sauce." A discriminator (Adversarial Unit) is trained to distinguish between feature maps from the teacher and the student. This forces the student to produce features so high-quality that they are indistinguishable from the teacher's, effectively pulling the student's internal representations toward the teacher's distribution.
- Mixup Data Augmentation: Blending audio samples during training improves the model's ability to generalize to unseen sounds.
Experimental Setup and Datasets
The method was evaluated on three distinct audio challenges:
- DCASE18 Task 1A: Acoustic Scene Classification (e.g., "airport," "park").
- DCASE18 Task 2: General Audio Tagging (41 sound categories).
- Google Speech Commands: Keyword recognition (30 spoken words).
The researchers explored "cross-structure" distillation, such as using a ResNet teacher to train a VGGNet student. They found that different architectures often capture complementary knowledge, providing a superior learning signal for the student compared to same-structure distillation.
Key Results and Insights
The adversarial approach yielded significant improvements in both efficiency and accuracy:
- Massive Compression: The "SmallCNN" student achieved a 76:1 reduction in FLOPS and a 3:1 reduction in parameters compared to the ensemble teacher models.
- Performance Gains: On the General Audio Tagging task, a distilled VGGNet reached 92.54% mAP@3, outperforming the baseline (91.5%) and rivaling complex ensemble models.
- Efficiency: The distilled SmallCNN achieved 91.28% mAP@3 on DCASE18 Task 2, proving that a compact model can match the performance of a much larger ensemble when trained via adversarial distillation.
The study also highlighted that audio-level scoring (averaging across an entire file) is more accurate than segment-level scoring (looking at 1.5-second snippets), as acoustic events are often transient.
Conclusion
This research demonstrates that adversarial feature distillation is a powerful tool for deploying high-performance audio classification on edge devices. By focusing on internal feature distributions rather than just final labels, the method allows for extreme model compression without sacrificing accuracy. Future work will likely explore multi-representation distillation and hardware-specific optimizations to further reduce latency on embedded systems.
Summary updated