Bidirectional LSTM-HMM Hybrid System for Polyphonic Sound Event Detection

Tomoki Hayashi, Shinji Watanabe, Tomoki Toda, Takaaki Hori, Jonathan Le Roux, Kazuya TakedaDCASE 2016 · OpenReview

Abstract

This paper presents a novel hybrid architecture for polyphonic sound event detection (SED) that combines Bidirectional Long Short-Term Memory (BLSTM) networks with Hidden Markov Models (HMMs). By integrating HMMs to explicitly model sound event durations, the system eliminates the need for heuristic post-processing and manual thresholding. Evaluated on the DCASE 2016 dataset, the proposed model achieves state-of-the-art performance, demonstrating the effectiveness of adapting speech recognition paradigms to multi-label audio classification tasks.

Introduction

Sound Event Detection (SED) is the process of identifying the onset, offset, and category of various sounds within an audio stream. Polyphonic SED presents a significant challenge as multiple sound events—such as a door slam occurring simultaneously with a telephone ring—often overlap. While deep learning models like Recurrent Neural Networks (RNNs) have shown promise in frame-by-frame classification, they often produce noisy, fragmented outputs that require extensive manual post-processing to define coherent event boundaries.

This research addresses these limitations by introducing a BLSTM-HMM hybrid system. By borrowing and extending the DNN-HMM framework from automatic speech recognition, the authors provide a principled way to model temporal continuity and event duration directly within the network architecture.

The Challenge of Polyphonic Modeling

Standard BLSTM-RNNs are effective at capturing temporal dependencies but lack an inherent mechanism to enforce duration constraints. Consequently, raw frame-level predictions often "flicker," leading to high error rates in event-based metrics. Researchers typically mitigate this using median filtering or ad-hoc thresholding. The proposed hybrid approach replaces these manual steps with a Hidden Markov Model (HMM) layer that treats overlapping sounds as parallel sequences, smoothing the output through probabilistic state transitions.

Methodology

BLSTM with Projection Layers

To generate acoustic scores, the system utilizes a deep BLSTM network. To manage the high computational complexity associated with deep bidirectional structures, the authors employ LSTM layers with projection (LSTMP), which reduce the dimensionality of the hidden state before it is passed to the next layer or the output.

The proposed deep BLSTM architecture featuring projection layers to manage computational complexity and output posterior probabilities for HMM states.

HMM-based Duration Modeling

The core innovation lies in assigning a 3-state left-to-right HMM to every sound event class. Each class-specific HMM includes three active states and one "non-active" state. This structure allows the model to learn the typical duration of an event through transition probabilities. During inference, the Viterbi algorithm is used to decode the most likely sequence of states across time, inherently providing clean event boundaries.

The HMM state transition topology used for each sound event class, incorporating three active states (n=1,2,3) and a non-active state (n=4).

Data Synthesis and Training

Deep learning models require vast amounts of data, which is often scarce in SED tasks. To overcome this, the authors synthesized 111 hours of training data by mixing clean sound samples with background noise at various Signal-to-Noise Ratios (SNRs) ranging from -9 dB to 9 dB. This synthetic dataset allowed the BLSTM to learn robust features for overlapping sounds in diverse acoustic environments.

Key Results

The BLSTM-HMM system significantly outperformed both the Non-Negative Matrix Factorization (NMF) baseline and standard BLSTM-RNNs. On the DCASE 2016 Task 2 test set, the model achieved an event-based F1 score of 67.1% and a segment-based F1 score of 76.0%.

On the development set, the hybrid model reached a 76.6% event-based F1 score, surpassing a standard BLSTM with median filtering (75.8%) and vastly outperforming the NMF baseline (14.6%). The model demonstrated strong noise robustness, maintaining a 79.6% F1 score at 6 dB SNR, though performance dipped to 73.7% at -6 dB due to difficulties in detecting quieter sounds like page turns.

Conclusion and Future Work

The transition of the hybrid DNN-HMM architecture from speech recognition to polyphonic SED proves highly effective. By replacing manual post-processing with learned duration modeling, the system achieves more reliable event detection. Future research will focus on investigating the model's performance on real-world (non-synthetic) datasets and exploring sequence discriminative training to further refine the BLSTM-HMM's predictive accuracy.

Summary updated

BibTeX
@inproceedings{hayashi2016bidirectional,
  author = {Tomoki Hayashi and Shinji Watanabe and Tomoki Toda and Takaaki Hori and Jonathan Le Roux and Kazuya Takeda},
  title = {{Bidirectional LSTM-HMM Hybrid System for Polyphonic Sound Event Detection}},
  booktitle = {DCASE},
  year = {2016},
  url = {https://openreview.net/forum?id=Eanv3DArcrn},
}
Figure