Neural Network Distillation on IoT Platforms for Sound Event Detection

Gianmarco Cerutti, Rahul Prasad, Alessio Brutti, Elisabetta FarellaInterspeech 2019

TL;DR

This paper tackles the challenge of deploying sophisticated sound event detection (SED) models on resource-constrained Internet of Things (IoT) devices. They achieve this by using knowledge distillation to significantly compress a powerful VGGish-based neural network, enabling near state-of-the-art performance with drastically reduced computational and memory footprints, making it suitable for embedded platforms.

What Problem Does This Paper Address?

The increasing prevalence of smart cities and the Internet of Things (IoT) necessitates intelligent devices capable of understanding their environment. Sound Event Detection (SED) is a key technology for applications like traffic monitoring, crowd analysis, and building occupancy detection. However, state-of-the-art SED models, often based on deep neural networks, are computationally intensive and require significant memory, making them unsuitable for low-power, low-complexity IoT devices. This creates a gap between the potential of advanced AI and the practical limitations of embedded hardware, hindering the widespread deployment of such intelligent capabilities at the edge.

Methods

The core of this paper's approach is Knowledge Distillation (KD), a technique where a smaller "student" network learns to mimic the behavior of a larger, pre-trained "teacher" network. Specifically, they leverage the VGGish feature extractor, a powerful model trained on a massive audio dataset, as the teacher. The student network is designed to be significantly smaller, with reduced layers and filters.

They explore several KD strategies by combining different loss functions:

  • Hard Labels (Th): Standard training using ground-truth labels (cross-entropy loss).
  • Soft Labels (Ts): Training the student to match the softened outputs (logits) of the teacher network.
  • Embedding Loss (Te): Training the student to replicate the intermediate feature representations generated by the teacher.
  • Compound Loss (Th,s,e): A combination of all three loss functions, which proved to be the most effective.

The authors also investigate different degrees of compression by creating student models with varying numbers of parameters (from ~72.1M down to ~30.6k).

Datasets

  • UrbanSound8K: This dataset is used for training and evaluating the SED models. It contains 8732 audio clips from urban environments, labeled with ten distinct sound classes (e.g., car horn, dog bark, siren). The paper uses a 10-fold cross-validation strategy.
  • Youtube-8M Dataset: VGGish, the feature extractor used as the teacher, was pre-trained on this large-scale dataset.

Key Insights and Ideas

  • Extreme Compression is Achievable: The paper demonstrates that it's possible to achieve very aggressive compression factors (up to 4.2 x 10⁻⁴ for parameters and 1.2 x 10⁻³ for operations compared to VGGish) while retaining a significant portion of the performance.
  • Compound Loss is Key: The combination of hard labels, soft labels, and embedding loss (Th,s,e) consistently yielded the best results across different model sizes, highlighting the benefit of leveraging multiple sources of knowledge from the teacher.
  • Data Augmentation Boosts Performance: Augmenting the training data (e.g., through pitch shifting) significantly improved the accuracy of all models, even allowing some distilled models to outperform the teacher. This suggests that the UrbanSound8K dataset, while useful, might not be rich enough to fully train a complex model from scratch to mimic the teacher's performance without augmentation.
  • Teacher's Performance as a Baseline: The teacher's performance on UrbanSound8K was surprisingly not always superior to the distilled models, especially without data augmentation. This points to the challenges of directly applying a model trained on a different, larger dataset to a specific, potentially less diverse, target dataset.

Key Results

  • The best performing training strategy (Th,s,e) achieved an accuracy of 70% for the smallest model (M20k) without data augmentation, a significant improvement over standard training (57.5%).
  • With data augmentation, the M2M model achieved an accuracy of 74.7%, slightly outperforming the teacher baseline.
  • The smallest model, M20k, with only ~30.6k parameters and ~2.11M operations, achieved 69.7% accuracy with data augmentation, representing a remarkable compression of over 99.95% in parameters and 99.88% in operations compared to the VGGish feature extractor.
  • Hardware analysis showed that only the M20k and M200k models were suitable for the most constrained embedded platforms, with M200k (approx. 202k parameters) being a strong candidate, achieving 72.5% accuracy and requiring minimal resources.

Limitations and Future Work

  • Dataset Specificity: The performance of the distilled models might be sensitive to the specific characteristics of the UrbanSound8K dataset.
  • Hardware Assumptions: The hardware comparison relies on approximations for instruction counts and memory usage, and doesn't account for all processing stages.
  • Future Work: The authors suggest further exploring different KD strategies, investigating other network architectures, and performing real-world deployment tests on various IoT platforms to validate their findings. They also mention the potential for further optimization through techniques like 8-bit quantization for inference.

Summary updated

BibTeX
@inproceedings{cerutti2019neural,
  author = {Gianmarco Cerutti and Rahul Prasad and Alessio Brutti and Elisabetta Farella},
  title = {{Neural Network Distillation on IoT Platforms for Sound Event Detection}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure