A Harmonic Excitation State-Space Approach to Blind Separation of Speech

Rasmus Kongsgaard Olsson, Lars Kai HansenNeurIPS 2004 · OpenReview

Introduction

The "Cocktail Party Problem"—isolating a single voice from a noisy mixture—remains a classic challenge in signal processing. While many algorithms exist for Blind Source Separation (BSS), they often struggle with convolutive mixtures, where signals reach microphones at different times due to room acoustics and echoes.

Mathematically, if sources are treated as stationary white noise, the problem suffers from a "rotational invariance" flaw: one cannot uniquely identify the original sources using only second-order statistics. To break this ambiguity, researchers must exploit the unique properties of speech. This paper addresses the gap by leveraging two specific traits: non-stationarity (speech statistics change over time) and harmonicity (voiced speech has a periodic structure). By explicitly modeling speech as a combination of harmonic tones and colored noise, the authors demonstrate that it is possible to perform blind source separation and pitch tracking simultaneously.

The Harmonic plus Noise (H+N) Model

The authors formulate a block-based generative model where each source is represented by a Harmonic plus Noise (H+N) model:

  1. Harmonic Part: A sum of sinusoids representing the "voiced" components of speech.
  2. Noise Part: An Auto-Regressive (AR) process representing "unvoiced" sounds or colored noise.

This representation is embedded into a linear state-space framework. To learn the parameters—such as the mixing filters, noise statistics, and pitch—the authors employ the Expectation-Maximization (EM) algorithm:

  • E-step: Uses a Kalman smoother to estimate the most likely hidden source signals given the observed mixture.
  • M-step: Updates the model parameters (like the mixing matrix and fundamental frequency) to maximize the likelihood of the data.

Because pitch estimation is non-linear and prone to local maxima (e.g., getting stuck on sub-harmonics), the authors use a constrained grid search within the typical human vocal range (50–400 Hz) to track the fundamental frequency.

Experimental Setup

The approach was evaluated using two primary setups:

  • Artificial Mixtures: Speech signals from the TIMIT corpus were mixed using known FIR filters and degraded with additive white noise (20dB SNR). This allowed the authors to compare their pitch estimates against "ground truth" Electro-laryngograph (EGG) recordings, which measure vocal fold vibration directly.
  • Real Room Recordings: A standard benchmark dataset featuring two male speakers synchronously counting in English and Spanish, recorded in a real acoustic environment.

Key Results and Insights

Simultaneous Estimation and Identifiability

Instead of treating pitch tracking and source separation as two separate steps, the state-space model allows them to inform one another. Better separation leads to better pitch tracking, and vice versa. Furthermore, because speech is non-stationary, the "rotational" ambiguity of BSS is resolved. The model uses the changing statistics across different time frames to lock onto the correct mixing filters.

Pitch Accuracy

In the artificial mixture experiment, the estimated pitches closely followed the EGG ground truth. The estimates were "frame-local" yet remarkably smooth, indicating the model's robustness.

Pitch tracking comparison

Separation Quality and Re-synthesis

Spectrogram analysis showed that the algorithm successfully recovered the distinct harmonic structures of two overlapping male voices.

Spectrogram analysis of separated sources

Beyond just separating the signals, the model allows for the re-synthesis of speech from the estimated H+N parameters. This produces intelligible speech that is often cleaner than the raw Maximum A Posteriori (MAP) estimates. In the real-world English/Spanish counting experiment, listening tests confirmed successful separation and high-quality synthesis, even with 20dB of noise.

Limitations and Future Work

While highly effective, the authors acknowledge that the approach is computationally intensive, making it currently unsuitable for real-time applications without further optimization or approximate inference techniques. Additionally, while the model handles convolutive mixtures well, more realistic models of long-term reverberation are needed for complex environments like large halls. Future work could involve incorporating more sophisticated linguistic or phonetic priors to further improve separation in extremely noisy conditions.

Summary updated

Abstract

We discuss an identification framework for noisy speech mixtures. A block-based generative model is formulated that explicitly incorporates the time-varying harmonic plus noise (H+N) model for a number of latent sources observed through noisy convolutive mixtures. All parameters including the pitches of the source signals, the amplitudes and phases of the sources, the mixing filters and the noise statistics are estimated by maximum likelihood, using an EM-algorithm. Exact averaging over the hidden sources is obtained using the Kalman smoother. We show that pitch estimation and source separation can be performed simultaneously. The pitch estimates are compared to laryngograph (EGG) measurements. Artificial and real room mixtures are used to demonstrate the viability of the approach. Intelligible speech signals are re-synthesized from the estimated H+N models.
BibTeX
@inproceedings{olsson2004harmonic,
  author = {Rasmus Kongsgaard Olsson and Lars Kai Hansen},
  title = {{A Harmonic Excitation State-Space Approach to Blind Separation of Speech}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2004},
  url = {https://openreview.net/forum?id=S1WbytbO-r},
}
Figure