Abstract
This paper presents a novel hybrid architecture for polyphonic sound event detection (SED) that combines Bidirectional Long Short-Term Memory (BLSTM) networks with Hidden Markov Models (HMMs). By integrating HMMs to explicitly model sound event durations, the system eliminates the need for heuristic post-processing and manual thresholding. Evaluated on the DCASE 2016 dataset, the proposed model achieves state-of-the-art performance, demonstrating the effectiveness of adapting speech recognition paradigms to multi-label audio classification tasks.
Introduction
Sound Event Detection (SED) is the process of identifying the onset, offset, and category of various sounds within an audio stream. Polyphonic SED presents a significant challenge as multiple sound events—such as a door slam occurring simultaneously with a telephone ring—often overlap. While deep learning models like Recurrent Neural Networks (RNNs) have shown promise in frame-by-frame classification, they often produce noisy, fragmented outputs that require extensive manual post-processing to define coherent event boundaries.
This research addresses these limitations by introducing a BLSTM-HMM hybrid system. By borrowing and extending the DNN-HMM framework from automatic speech recognition, the authors provide a principled way to model temporal continuity and event duration directly within the network architecture.
The Challenge of Polyphonic Modeling
Standard BLSTM-RNNs are effective at capturing temporal dependencies but lack an inherent mechanism to enforce duration constraints. Consequently, raw frame-level predictions often "flicker," leading to high error rates in event-based metrics. Researchers typically mitigate this using median filtering or ad-hoc thresholding. The proposed hybrid approach replaces these manual steps with a Hidden Markov Model (HMM) layer that treats overlapping sounds as parallel sequences, smoothing the output through probabilistic state transitions.
Methodology
BLSTM with Projection Layers
To generate acoustic scores, the system utilizes a deep BLSTM network. To manage the high computational complexity associated with deep bidirectional structures, the authors employ LSTM layers with projection (LSTMP), which reduce the dimensionality of the hidden state before it is passed to the next layer or the output.
HMM-based Duration Modeling
The core innovation lies in assigning a 3-state left-to-right HMM to every sound event class. Each class-specific HMM includes three active states and one "non-active" state. This structure allows the model to learn the typical duration of an event through transition probabilities. During inference, the Viterbi algorithm is used to decode the most likely sequence of states across time, inherently providing clean event boundaries.
Data Synthesis and Training
Deep learning models require vast amounts of data, which is often scarce in SED tasks. To overcome this, the authors synthesized 111 hours of training data by mixing clean sound samples with background noise at various Signal-to-Noise Ratios (SNRs) ranging from -9 dB to 9 dB. This synthetic dataset allowed the BLSTM to learn robust features for overlapping sounds in diverse acoustic environments.
Key Results
The BLSTM-HMM system significantly outperformed both the Non-Negative Matrix Factorization (NMF) baseline and standard BLSTM-RNNs. On the DCASE 2016 Task 2 test set, the model achieved an event-based F1 score of 67.1% and a segment-based F1 score of 76.0%.
On the development set, the hybrid model reached a 76.6% event-based F1 score, surpassing a standard BLSTM with median filtering (75.8%) and vastly outperforming the NMF baseline (14.6%). The model demonstrated strong noise robustness, maintaining a 79.6% F1 score at 6 dB SNR, though performance dipped to 73.7% at -6 dB due to difficulties in detecting quieter sounds like page turns.
Conclusion and Future Work
The transition of the hybrid DNN-HMM architecture from speech recognition to polyphonic SED proves highly effective. By replacing manual post-processing with learned duration modeling, the system achieves more reliable event detection. Future research will focus on investigating the model's performance on real-world (non-synthetic) datasets and exploring sequence discriminative training to further refine the BLSTM-HMM's predictive accuracy.
Summary updated