HiSSNet: Sound Event Detection and Speaker Identification via Hierarchical Prototypical Networks for Low-Resource Headphones

N. Shashaank, Berker Banar, Mohammad Rasool Izadi, Jeremy Kemmerer, Shuo Zhang, Chuan-Che Jeff Huang

TL;DR

HiSSNet is a unified, hierarchical machine learning model designed to run on low-power headphones to detect both general sound events (like alarms) and specific speakers (like a spouse). By using a hierarchical prototypical network, it achieves state-of-the-art performance while fitting within the strict 1–4 MB memory constraints of embedded microcontrollers.

What Problem Does This Paper Address?

Modern noise-canceling headphones are excellent at blocking out the world, but they often work too well, silencing important sounds like a doorbell, a passing truck, or a family member trying to get your attention.

To solve this, headphones need Sound Event Detection (SED) and Speaker Identification (SID). However, implementing these features on-device faces three major hurdles:

  1. User Friction: Most users won't spend time "training" their headphones; the system needs to work well out of the box.
  2. Personalization: The model must eventually learn which specific sounds (e.g., your baby crying vs. a baby on TV) matter to the individual user.
  3. Resource Constraints: Headphones run on ultra-low-power chips with tiny amounts of memory (often just 1–4 MB), making it nearly impossible to run multiple large, specialized models simultaneously.

Methods

The authors introduce HiSSNet (Hierarchical SED and SID Network). Its core architecture is a Hierarchical Prototypical Network, a few-shot learning approach that organizes sounds into a tree structure rather than a flat list.

  • The Hierarchy: The model uses a three-level taxonomy. Level 1 contains general categories (e.g., "Speech" or "Alarms"), while Level 3 contains specific instances (e.g., "Speaker M1" or "Emergency Siren").
  • The Encoder: They used MobileNetV2, a convolutional neural network optimized for mobile and embedded devices, to transform audio into latent embeddings.
  • Weighted Loss Function: A key innovation is a loss function that uses a hyperparameter ($\alpha$) to balance the importance of general vs. specific classes during training.
  • Distance Metric: After testing several options, they found that squared Euclidean distance provided the best accuracy for classifying query samples against "prototypes" (mean embeddings) in the hierarchy.

Datasets

The researchers compiled a massive "SEID" (SED + SID) dataset by combining seven existing datasets: ESC50, TUT Sound Events 2016, TAU MREAL, FSD50K, BBC Sound Effects, VCTK, and LibriSpeech.

The resulting dataset contains approximately 673.4 hours of audio across 211.6k files. They mapped these into a custom taxonomy consisting of:

  • 7 top-level classes.
  • 19 middle-level classes.
  • 1,319 lower-level classes (56 for SED and 1,263 for SID).

Key Insights and Ideas

  • Hierarchy Improves "General" Accuracy: One of the most interesting findings was that forcing the model to learn a hierarchy actually makes it better at general classification. By understanding that a specific siren belongs to the "Alarm" category, the model clusters its embeddings more effectively.
  • The Power of $\alpha=1$: The authors found that giving more weight to the specific, lower-level classes during training ($\alpha=1$) resulted in the best performance across all levels of the hierarchy.
  • Unified Efficiency: Instead of running two separate models for SED and SID, HiSSNet handles both in one pass. This significantly reduces the memory footprint and computational overhead (MAC operations), which is critical for battery-powered wearables.

Key Results

HiSSNet demonstrated significant improvements over both standard baselines and specialized state-of-the-art (SOTA) models:

  • Vs. Standard ProtoNet: HiSSNet outperformed a non-hierarchical version by 8.6% on general sound detection and 8.3% on specific speaker identification.
  • Vs. SOTA SED Models: It achieved 93.5% accuracy on general sound events, beating specialized SED-only models by about 5.2%.
  • Vs. SOTA SID Models: While slightly trailing specialized speaker ID models in raw accuracy (a 6.7% drop in specific SID), it achieved an Equal Error Rate (EER) of 1.47%, which the authors argue is an acceptable trade-off for the massive reduction in memory and power usage.

Limitations and Future Work

While HiSSNet is a leap forward for embedded audio AI, the authors note a few areas for growth:

  • Open-Set Recognition: The paper mentions treating novel sounds as "unknown" via an open-set approach, but more work is needed to ensure the model doesn't confidently misclassify a completely new sound into an existing category.
  • Implicit Interaction: The model's ability to learn from "implicit" user cues (like a user turning their head toward a sound) is a proposed feature that requires further real-world validation.
  • Hardware Deployment: While the model is designed for low-resource chips, the paper focuses on software benchmarks; future work will likely involve testing the real-world latency and battery impact on specific headphone hardware.

Summary updated

Abstract

Modern noise-cancelling headphones have significantly improved users’ auditory experiences by removing unwanted background noise, but they can also block out sounds that matter to users. Machine learning (ML) models for sound event detection (SED) and speaker identification (SID) can enable headphones to selectively pass through important sounds; however, implementing these models for a user-centric experience presents several unique challenges. First, most people spend limited time customizing their headphones, so the sound detection should work reasonably well out of the box. Second, the models should be able to learn over time the specific sounds that are important to users based on their implicit and explicit interactions. Finally, such models should have a small memory footprint to run on low-power headphones with limited on-chip memory. In this paper, we propose addressing these challenges using HiSSNet (Hierarchical SED and SID Network). HiSSNet is an SEID (SED and SID) model that uses a hierarchical prototypical network to detect both general and specific sounds of interest and characterize both alarm-like and speech sounds. We show that HiSSNet outperforms an SEID model trained using non-hierarchical prototypical networks by 6.9 – 8.6%. When compared to state-of-the-art (SOTA) models trained specifically for SED or SID alone, HiSSNet achieves similar or better performance while reducing the memory footprint required to support multiple capabilities on-device.
BibTeX
@inproceedings{shashaank2023hissnet,
  author = {N. Shashaank and Berker Banar and Mohammad Rasool Izadi and Jeremy Kemmerer and Shuo Zhang and Chuan-Che Jeff Huang},
  title = {{HiSSNet: Sound Event Detection and Speaker Identification via Hierarchical Prototypical Networks for Low-Resource Headphones}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=U8XSTHOSVD},
}

Loading related content...

Figure