Quality conversion of non-acoustic signals for facilitating human-to-human speech communication under harsh acoustic conditions

Seyed Omid Sadjadi, Sanjay A. Patil, John H. L. HansenInterspeech 2010

TL;DR

This paper introduces a novel probabilistic transformation technique to significantly improve the perceptual quality and intelligibility of speech captured by non-acoustic sensors (like the PMIC). By mapping these signals into the acoustic space of traditional microphones, the method aims to make communication clearer and less effortful, especially in noisy environments where standard microphones struggle.

What Problem Does This Paper Address?

Effective human-to-human speech communication is crucial, but harsh acoustic environments pose a significant challenge. Traditional speech enhancement methods often improve perceived speech quality but fail to boost intelligibility, and can even introduce undesirable artifacts in very noisy conditions. While non-acoustic contact sensors (like the Physiological Microphone, or PMIC) offer robustness against ambient noise, the speech signals they capture often suffer from reduced quality and intelligibility due to their different sensing principles and pickup locations. This paper tackles the problem of converting these lower-quality non-acoustic signals into a form that is perceptually closer to what a traditional microphone would capture in clean conditions, thereby facilitating clearer communication.

Methods

The core of this paper is a probabilistic transformation algorithm. The authors propose a method to map the spectral envelopes of PMIC signals to those of conventional close-talk microphones (CTM). This is achieved by modeling the acoustic spaces of both PMIC and CTM signals using codebooks derived from vector quantization (VQ) of their spectral envelopes.

The transformation itself is a piecewise linear mapping. Crucially, instead of a simple one-to-one mapping, the authors employ a Gaussian Mixture Model (GMM) fitted on the VQ codebooks. This allows for a "soft decision making" approach, where the transformation for a given PMIC signal vector is a weighted sum of transformations from multiple CTM codebook entries, with weights determined by the posterior probability of the PMIC vector belonging to each class. This probabilistic approach is termed probabilistic VQ (VQProb). The transformation matrix is learned using a least squares estimation on parallel training data. The authors also consider extending the feature vector to include neighboring frames for better interpolation.

Datasets

The paper uses parallel recordings of speech from an adult male speaker captured simultaneously by both a PMIC and a CTM. For training and evaluation, they utilize 60 sentences from the IEEE database. This database is described as phonetically balanced and having low word-context predictability, making it suitable for subjective quality and intelligibility tests. A leave-one-out (LOO) cross-validation scheme is employed for evaluation to ensure unbiased results.

Key Insights and Ideas

  • Non-acoustic sensors are promising but imperfect: While PMICs offer excellent noise immunity, their inherent signal characteristics lead to a noticeable drop in perceptual speech quality and intelligibility compared to traditional microphones.
  • Bridging the acoustic gap is key: The paper highlights that the mismatch in spectral characteristics between PMIC and CTM signals is the primary cause of the quality degradation. Their method directly addresses this by transforming the PMIC's spectral space to match the CTM's.
  • Probabilistic mapping outperforms deterministic approaches: The proposed VQProb method, which uses soft decisions based on GMM probabilities, shows superior performance over simpler VQ-based mapping strategies. This suggests that accounting for uncertainty in signal classification leads to better transformation results.
  • Source excitation is well-captured by PMIC: The authors note that the PMIC reliably captures the speech excitation signal, meaning their post-processing focuses solely on transforming the spectral envelope, not the excitation itself.

Key Results

  • Objective Improvement: The log-spectral distortion between the original PMIC and CTM spectra was reduced from 15.99 dB to 0.51 dB after applying the proposed transformation. Spectrogram analysis also visually demonstrates the reconstruction of attenuated spectral contents and improved temporal structure.
  • Performance Comparison: The VQProb method outperformed other transformation strategies (plain VQ and a weighted VQ) in terms of both Perceptual Evaluation of Speech Quality (PESQ) and log-likelihood ratio (LLR) metrics.
  • Subjective Improvement: Listening experiments showed a "dramatic elevation" in the overall quality of the enhanced PMIC signals. Listening effort was reduced, and intelligibility, as measured by Word Error Rate (WER), significantly improved compared to the original PMIC signals, approaching CTM levels (p < 0.001).
  • Codebook Size Impact: Increasing the codebook size improved performance up to a certain point (7 bits), after which performance degraded. This is attributed to data scarcity in some VQ clusters making the autocorrelation matrix ill-conditioned.

Limitations and Future Work

  • Data Scarcity: The paper acknowledges that data scarcity in some VQ clusters can lead to ill-conditioned matrices and performance degradation, particularly with larger codebook sizes. This suggests that more extensive training data might be beneficial.
  • Focus on Spectral Envelope: The current approach does not address transforming the source excitation signal, as it's assumed to be well-captured by the PMIC. However, future work could explore if any improvements can be made in this area.
  • Generalizability: While the method is evaluated on an adult male speaker, further research could investigate its effectiveness across different speakers, accents, and a wider range of acoustic conditions.
  • Real-time Implementation: The paper focuses on offline processing. Adapting this probabilistic transformation for real-time applications would be a valuable future direction.

Summary updated

BibTeX
@inproceedings{sadjadi2010quality,
  author = {Seyed Omid Sadjadi and Sanjay A. Patil and John H. L. Hansen},
  title = {{Quality conversion of non-acoustic signals for facilitating human-to-human speech communication under harsh acoustic conditions}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure