Abstract
Modern noise-cancelling headphones have significantly improved users' auditory experiences by removing unwanted background noise, but they can also block out sounds that matter to users. Machine learning (ML) models for sound event detection (SED) and speaker identification (SID) can enable headphones to selectively pass through important sounds; however, implementing these models for a user-centric experience presents several unique challenges. First, most people spend limited time customizing their headphones, so the sound detection should work reasonably well out of the box. Second, the models should be able to learn over time the specific sounds that are important to users based on their implicit and explicit interactions. Finally, such models should have a small memory footprint to run on low-power headphones with limited on-chip memory. In this paper, we propose addressing these challenges using HiSSNet (Hierarchical SED and SID Network). HiSSNet is an SEID (SED and SID) model that uses a hierarchical prototypical network to detect both general and specific sounds of interest and characterize both alarm-like and speech sounds. We show that HiSSNet outperforms an SEID model trained using non-hierarchical prototypical networks by 6.9 - 8.6 percent. When compared to state-of-the-art (SOTA) models trained specifically for SED or SID alone, HiSSNet achieves similar or better performance while reducing the memory footprint required to support multiple capabilities on-device.
Introduction
Modern Active Noise Cancellation (ANC) technology has reached a point where it can effectively silence the outside world. However, this isolation creates a safety and social barrier, often blocking critical auditory cues such as doorbells, emergency sirens, or a family member's voice. To bridge this gap, headphones require integrated Sound Event Detection (SED) and Speaker Identification (SID) capabilities.
Implementing these features on wearable hardware introduces three primary constraints. First, the system must minimize user friction, functioning effectively "out of the box" without extensive training. Second, it must support personalization, eventually distinguishing between a generic sound and a user-specific sound (e.g., a specific baby's cry). Finally, these models must operate within the extreme resource constraints of embedded microcontrollers, which typically offer only 1–4 MB of on-chip memory.
Methodology: The HiSSNet Framework
To address these challenges, we introduce HiSSNet (Hierarchical SED and SID Network). Unlike traditional models that treat classification as a flat list of categories, HiSSNet utilizes a Hierarchical Prototypical Network. This few-shot learning approach organizes sounds into a structured taxonomy, allowing the model to share features across related classes.
Hierarchical Taxonomy
The model architecture is built upon a three-level taxonomy:
- Level 1 (General): Broad categories such as "Speech" or "Alarms."
- Level 2 (Intermediate): Sub-categories like "Male Speech" or "Emergency Vehicles."
- Level 3 (Specific): Granular instances, such as a specific user's voice ("Speaker M1") or a specific "Emergency Siren."
Architecture and Training
HiSSNet employs MobileNetV2 as its backbone encoder, a choice driven by the need for high performance within a small memory footprint. The encoder transforms raw audio into latent embeddings. During training, we utilize a weighted loss function governed by a hyperparameter ($\alpha$). This allows the model to balance the importance of general versus specific classification. Our research found that setting $\alpha=1$—placing maximum weight on the most specific, lower-level classes—actually yielded the best performance across all levels of the hierarchy. This suggests that learning fine-grained distinctions forces the model to develop more robust representations for broader categories.
Experimental Setup
To train and evaluate HiSSNet, we curated a comprehensive "SEID" (Sound Event and Identification) dataset by aggregating seven major audio datasets: ESC50, TUT Sound Events 2016, TAU MREAL, FSD50K, BBC Sound Effects, VCTK, and LibriSpeech. The resulting corpus includes approximately 673.4 hours of audio across 211,600 files. This data was mapped into a custom taxonomy featuring 7 top-level classes, 19 middle-level classes, and 1,319 lower-level classes (comprising 56 sound events and 1,263 unique speakers).
Results and Discussion
HiSSNet demonstrates that a unified, hierarchical approach is superior to maintaining separate, specialized models for SED and SID.
- Performance Gains: HiSSNet outperformed a standard, non-hierarchical Prototypical Network (ProtoNet) by 8.6% on general sound detection and 8.3% on specific speaker identification.
- Comparison to SOTA: In general sound event detection, HiSSNet achieved 93.5% accuracy, surpassing specialized SED-only models by 5.2%. While specialized SID models still hold a slight edge in raw accuracy (a 6.7% difference), HiSSNet achieved a competitive Equal Error Rate (EER) of 1.47%.
- Efficiency: By unifying SED and SID into a single pass, HiSSNet significantly reduces the total number of Multiply-Accumulate (MAC) operations and the memory footprint, making it viable for 1–4 MB embedded systems.
Conclusion
HiSSNet provides a scalable and efficient solution for bringing intelligent sound awareness to low-power wearables. By leveraging a hierarchical structure, the model improves classification accuracy while remaining small enough for on-device deployment. Future work will focus on refining open-set recognition to better handle completely novel sounds and validating the model's performance in real-world environments using implicit user interactions, such as head-tracking, to further personalize the auditory experience.
Summary updated