Adaptive weighting of microphone arrays for distant-talking F0 and voiced/unvoiced estimation

Federico Flego, Christian Zieger, Maurizio OmologoInterspeech 2007

TL;DR

This paper introduces a novel method for robustly estimating fundamental frequency (F0) and classifying voiced/unvoiced speech from distant microphones in noisy, reverberant environments. By adaptively weighting microphone signals based on their reliability, the system significantly improves performance over traditional single-microphone approaches, paving the way for more natural human-computer interaction.

What Problem Does This Paper Address?

In real-world scenarios, speech is often captured by multiple microphones placed at a distance from the speaker (e.g., in conference rooms or smart homes). This "distant-talking" setup introduces significant challenges: the speech signal is degraded by noise and reverberation, making it difficult to accurately extract fundamental frequency (F0) – a key characteristic for speech analysis and synthesis – and to distinguish between voiced (e.g., vowels) and unvoiced (e.g., 's' sounds) speech segments. Existing methods often struggle with this degradation, and while multi-microphone systems offer redundancy, effectively combining their signals without prior knowledge of microphone or speaker positions has been a persistent challenge.

Methods

The core innovation lies in an Adaptive Multi-microphone Periodicity Function (MPF). This function is derived by adaptively weighting the magnitude spectra from each microphone. The weighting scheme is crucial: it dynamically assesses the "reliability" of each microphone's signal for F0 estimation. This reliability is determined by comparing a smoothed average spectrum across all microphones with each individual microphone's spectrum. Microphones that provide consistent spectral information are given higher weights, effectively downplaying noisy or heavily reverberated signals.

For F0 estimation, the MPF is computed and then interpolated to find the fundamental period. For voiced/unvoiced (V/UV) classification, a Hidden Markov Model (HMM) is employed. The HMM uses a three-dimensional feature vector: two features are derived from the MPF (representing the ratio of the MPF peak to its value at zero lag, and the ratio of the MPF at lag 1 to its value at zero lag), and the third is an energy-based feature. These features are designed to be discriminative even in challenging acoustic conditions.

Datasets

The paper evaluates its methods on two datasets:

  1. Real spontaneous speech corpus: Collected from seminar sessions at Karlsruhe University, featuring 13 recordings with a reverberation time (T60) of approximately 0.45 seconds.
  2. Simulated speech and noise: A simulated environment mimicking the real room layout, allowing for controlled experiments with varying Signal-to-Noise Ratios (SNR) and noise positions.

Key Insights and Ideas

  • Adaptive weighting is key: Simply averaging or combining microphone signals isn't enough. Dynamically assessing and weighting microphone reliability based on spectral consistency is crucial for robust performance in adverse conditions.
  • MPF captures shared harmonic structure: The MPF, by combining weighted spectra, effectively highlights the underlying periodic structure of speech, even when individual microphone signals are heavily distorted.
  • HMMs are well-suited for V/UV classification: The proposed HMM framework, using MPF-derived features and energy, demonstrates effective discrimination between voiced and unvoiced segments in distant-talking scenarios.
  • Distant-talking challenges are significant but surmountable: The paper shows that while distant-talking speech is inherently difficult to process, multi-microphone techniques with adaptive weighting can achieve performance close to that of close-talking microphones.

Key Results

  • F0 Estimation: The adaptive MPF method achieved Gross Error Rates (GER) of 1.7% (GER20) and 7% (GER5) on the real database, showing only a marginal improvement when channel reliability was considered due to diffuse noise. However, on simulated data with coherent noise, the adaptive weighting consistently improved performance, especially at lower SNRs.
  • V/UV Classification: The DMN-based V/UV classifier outperformed single-microphone approaches. While still lagging behind the close-talking reference by about 6% in VUVerror, it demonstrated significant gains in noisy and reverberant conditions.

Limitations and Future Work

  • Limitations: The paper notes that the performance improvement from channel reliability is most evident at very low SNRs. For higher SNRs, the curves tend to converge. The V/UV classifier, while improved, still has a gap compared to close-talking microphones.
  • Future Work: The authors plan to extend these techniques to other distant-talking tasks such as speaker identification and acoustic event classification within the context of interactive TV scenarios.

Summary updated

BibTeX
@inproceedings{flego2007adaptive,
  author = {Federico Flego and Christian Zieger and Maurizio Omologo},
  title = {{Adaptive weighting of microphone arrays for distant-talking F0 and voiced/unvoiced estimation}},
  booktitle = {Interspeech},
  year = {2007},
}
Figure