Abstract
There is an increasing interest in smart environment and a growing adoption of smart devices. Smart assistants such as Google Home and Amazon Alexa, although focus on speech, could be extended to identify domestic events in real-time to provide more and better smart functions. Sound event detection aims to detect multiple target sound events that may happen simultaneously. The task is challenging due to the overlapping of sound events, the highly imbalanced nature of target and non-target data, and the complicated real-world background noise. In this paper, we proposed a unified approach that takes advantages of both the deep learning and data augmentation. A convolutional neural network (CNN) was combined with a feed-forward neural network (FNN) to improve the detection performance, and a dynamic time warping based data augmentation (DA) method was proposed to address the data imbalance problem. Experiments on several datasets showed a more than 7% increase in accuracy compared to the state-of-the-art approaches.
Introduction
Identifying rare but critical sound events—such as a window breaking or a smoke alarm—within the noisy, overlapping soundscape of a modern smart home remains a significant challenge. While smart assistants excel at speech processing, they often struggle with real-time domestic sound event detection (SED) due to overlapping sounds, severe data imbalance, and unpredictable background noise. Existing data augmentation (DA) methods often fail to capture natural temporal variations, leading to unrealistic "artifact" sounds that hinder model learning.
Proposed Methodology
DTW-Based Data Augmentation
To address data scarcity, we developed a "rescaling and merging" strategy. Traditional linear mixing often distorts the acoustic signature of rare events. Instead, we utilize Dynamic Time Warping (DTW) to find the optimal alignment between sequences before averaging them. This ensures the prototypical "shape" of the sound is preserved in the synthetic samples.
We further modified the standard DTW Barycenter Averaging (DBA) algorithm to include weights, allowing for better generalization across varied acoustic environments.
Dual-Model Ensemble Architecture
Our framework employs a multi-branch neural network architecture to capture diverse audio features:
- FNN Branch: Processes Mel-Frequency Cepstral Coefficients (MFCCs), a standard compressed audio representation.
- CNN Branch: Processes raw Short-Time Fourier Transform (STFT) coefficients to extract spatial-like features from spectrograms.
- Model Fusion: A Support Vector Machine (SVM) with an RBF kernel serves as the final decision-maker, effectively resolving conflicts between the FNN and CNN outputs.
Experimental Results
The approach was evaluated on three DCASE datasets (2016, 2017, and 2019). Key findings include:
- Performance Gains: The ensemble method outperformed baselines by over 6% across all datasets, reaching a 7%+ improvement on DCASE 2016 and 2017.
- Augmentation Efficacy: On the DCASE 2017 rare event task, the Equal Error Rate (EER) dropped significantly from 19.67% to 13.46% when using DTW-based augmentation.
- Alignment Advantage: DTW-based merging consistently outperformed standard "mix-up" techniques, proving that temporal alignment is crucial for generating realistic acoustic training data.
Conclusion
This research demonstrates that effective sound event detection requires both robust feature extraction and high-quality data augmentation. By aligning samples via DTW, we can generate synthetic data that preserves the temporal integrity of rare sounds. Future work will explore multi-channel audio integration and automated weight optimization for the DBA process.
This paper was originally presented at INTERSPEECH 2019 by researchers from Samsung Research America.
Summary updated