An Adversarial Feature Distillation Method for Audio Classification

Liang Gao, Haibo Mi, Boqing Zhu, Dawei Feng, Yicong Li, Yuxing Peng

Abstract

Deep Neural Networks (DNNs) have revolutionized audio classification, yet their deployment on resource-constrained edge devices remains challenging due to high computational demands. This paper introduces a novel adversarial feature distillation framework designed to compress large teacher models into efficient student networks without compromising accuracy. By forcing the student to mimic the internal feature representations of the teacher through a Generative Adversarial Network (GAN) style game, the proposed method achieves a 76:1 reduction in computational complexity (FLOPS) while improving classification performance on key benchmarks.

Introduction

The rise of "machine listening" has enabled applications ranging from acoustic scene classification to keyword spotting. However, the high-performance models required for these tasks often carry a heavy "computational tax," making them unsuitable for smartphones or IoT sensors. Knowledge Distillation (KD) has emerged as a popular solution, where a smaller "student" network learns from a larger "teacher."

Traditional KD typically focuses on "soft labels"—the final probability outputs of the teacher. However, for complex and noisy audio data, these labels often discard critical structural information. This paper addresses the challenge of transferring the "internal logic" or feature maps of a complex model to a simpler one, even when their architectures differ significantly.

The Shift from Labels to Features

Standard Knowledge Distillation (KD) relies on the student matching the teacher's final output. In contrast, Feature Distillation (FD) focuses on intermediate layers, which contain richer information about the data's underlying structure.

Comparison of Knowledge Distillation (KD) and Feature Distillation (FD) paradigms.

As shown in the figure above, while KD (a) minimizes similarity loss at the output level, FD (b) attempts to align internal feature maps. The authors argue that for audio tasks, these intermediate representations are more robust than final labels alone.

Adversarial Feature Distillation Framework

The proposed framework introduces several key innovations to bridge the gap between teacher and student architectures:

  1. Multi-level Feature Distillation: The student mimics the teacher’s intermediate feature maps across multiple layers, capturing a hierarchy of acoustic features.
  2. Alignment Units: To handle architectural mismatches (e.g., a student with fewer channels or smaller layers), adaptive pooling is used to "resize" feature maps for direct comparison.
  3. Adversarial Learning Strategy: This is the core "secret sauce." A discriminator (Adversarial Unit) is trained to distinguish between feature maps from the teacher and the student. This forces the student to produce features so high-quality that they are indistinguishable from the teacher's, effectively pulling the student's internal representations toward the teacher's distribution.
  4. Mixup Data Augmentation: Blending audio samples during training improves the model's ability to generalize to unseen sounds.
The proposed adversarial feature distillation architecture, highlighting the alignment, distillation, and adversarial units.

Experimental Setup and Datasets

The method was evaluated on three distinct audio challenges:

  • DCASE18 Task 1A: Acoustic Scene Classification (e.g., "airport," "park").
  • DCASE18 Task 2: General Audio Tagging (41 sound categories).
  • Google Speech Commands: Keyword recognition (30 spoken words).

The researchers explored "cross-structure" distillation, such as using a ResNet teacher to train a VGGNet student. They found that different architectures often capture complementary knowledge, providing a superior learning signal for the student compared to same-structure distillation.

Key Results and Insights

The adversarial approach yielded significant improvements in both efficiency and accuracy:

  • Massive Compression: The "SmallCNN" student achieved a 76:1 reduction in FLOPS and a 3:1 reduction in parameters compared to the ensemble teacher models.
  • Performance Gains: On the General Audio Tagging task, a distilled VGGNet reached 92.54% mAP@3, outperforming the baseline (91.5%) and rivaling complex ensemble models.
  • Efficiency: The distilled SmallCNN achieved 91.28% mAP@3 on DCASE18 Task 2, proving that a compact model can match the performance of a much larger ensemble when trained via adversarial distillation.

The study also highlighted that audio-level scoring (averaging across an entire file) is more accurate than segment-level scoring (looking at 1.5-second snippets), as acoustic events are often transient.

Conclusion

This research demonstrates that adversarial feature distillation is a powerful tool for deploying high-performance audio classification on edge devices. By focusing on internal feature distributions rather than just final labels, the method allows for extreme model compression without sacrificing accuracy. Future work will likely explore multi-representation distillation and hardware-specific optimizations to further reduce latency on embedded systems.

Summary updated

Abstract

The audio classification task aims to discriminate between different audio signal types. In this task, deep neural networks have achieved better performance than the traditional shallow architecture-based machine-learning method. However, deep neural networks often require huge computational and storage requirements that hinder the deployment in embedded devices. In this paper, we proposed a distillation method which transfers knowledge from well-trained networks to a small network, and the method can compress model size while improving audio classification precision. The contributions of the proposed method are two folds: a multi-level feature distillation method was proposed and an adversarial learning strategy was employed to improve the knowledge transfer. The extensive experiments are conducted on three audio classification tasks, audio scene classification, general audio tagging, and speech command recognition. The experimental results demonstrate that: the small network can provide better performance while achieves the calculated amount of floating-point operations per second (FLOPS) compression ratio of 76:1 and parameters compression ratio of 3:1.
BibTeX
@article{gao2019adversarial,
  author = {Liang Gao and Haibo Mi and Boqing Zhu and Dawei Feng and Yicong Li and Yuxing Peng},
  title = {{An Adversarial Feature Distillation Method for Audio Classification}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2931656},
}
Figure