TL;DR
This paper tackles the challenge of improving the audio quality of throat microphone (TM) speech, which is often perceptually poorer than standard close-speaking microphone (CSM) speech. The authors propose a novel approach using multilayer feed-forward neural networks (MLFFNNs) to map both the "source" (excitation) and "system" (vocal tract) features of TM speech to their CSM counterparts, leading to significantly enhanced speech quality.
What Problem Does This Paper Address?
Throat microphones capture speech directly from the larynx, bypassing the natural filtering and shaping effects of the vocal tract. This results in speech that, while clear in terms of intelligibility, often lacks the natural richness and perceived quality of speech recorded by a close-speaking microphone (CSM). Previous enhancement methods primarily focused on mapping the spectral characteristics of the vocal tract system. However, these approaches were often limited by how they handled the "source" of the speech – the excitation signal generated by the vocal cords. This excitation signal is highly non-linear and varies rapidly, making it difficult to model and map effectively. The paper aims to bridge this gap by developing a method that simultaneously maps both the vocal tract system features and the more complex excitation source features.
Methods
The core of the proposed method relies on Multilayer Feed-Forward Neural Networks (MLFFNNs) for feature mapping.
- Source Feature Mapping: The key innovation here is anchoring the analysis around Glottal Closure Instants (GCIs). GCIs are critical moments of excitation in voiced speech and are considered characteristic of the source. For voiced segments, the authors extract 4ms of residual signal data (representing the source) around each GCI from both TM and CSM speech. These TM and CSM residual segments are then used as input-output pairs to train an MLFFNN. This network learns the non-linear relationship between the TM and CSM source features.
- System Feature Mapping: The paper also refines existing methods for mapping vocal tract system features. Linear Prediction Coefficients (LPCs), derived from 20-30ms frames, are converted to Linear Prediction Cepstral Coefficients (LPCCs). To incorporate temporal dynamics, LPCCs from three consecutive frames (i-1, i, i+1) are concatenated to form a feature vector. This vector is then used to train an MLFFNN to map TM system features to CSM system features.
- GCI Estimation: A zero-frequency (ZF) filter-based method is used to reliably estimate GCIs from the TM speech signal, which is crucial for anchoring the source feature extraction.
- Residual Signal Alignment: A critical step for accurate source mapping is aligning the residual signals. The authors propose aligning the peaks of the Hilbert envelope of the TM and CSM residual signals around the estimated GCIs. This ensures that the MLFFNN learns the correspondence between similar excitation events.
- Synthesis: The enhanced speech is synthesized by using the mapped excitation signal (generated by the source mapping MLFFNN) to excite the mapped vocal tract system (represented by the mapped LPCs).
Datasets
The paper describes using simultaneously recorded TM and CSM speech data from speakers for training and testing. Specific details on the size or origin of these datasets are not provided beyond "2-3 minutes of speech data per speaker" for training and "2 mins of speech data (consisting of 5 to 8 utterances per speaker)" for testing. No new benchmark datasets are introduced.
Key Insights and Ideas
- GCI as the Key to Source Mapping: The paper highlights that focusing on the excitation signal around Glottal Closure Instants (GCIs) is crucial for effectively mapping the non-linear and rapidly varying source features of speech.
- Joint Source and System Mapping is Superior: The research demonstrates that enhancing TM speech requires mapping both the vocal tract system features and the excitation source features. Simply mapping system features is insufficient for optimal quality.
- MLFFNNs Excel at Non-Linear Mapping: The use of MLFFNNs proves effective in capturing the complex, non-linear relationships between TM and CSM speech features, which are difficult to model with linear techniques.
- Temporal Information Improves System Mapping: Incorporating temporal context (using features from adjacent frames) in system feature mapping further improves the quality of the enhanced speech.
Key Results
- Objective Improvement: The Itakura distance, a measure of spectral similarity, was significantly reduced. The distance between CSM and mapped LPCs (with temporal information) was reduced by approximately 70% compared to the distance between CSM and TM LPCs.
- Subjective Improvement: Informal perceptual listening tests indicated significant improvements in speech quality compared to earlier methods. The synthesized speech using the proposed techniques was found to be "significantly better than TM speech."
- Quality of Synthesized Speech: When the mapped residual was used as the source, the difference in quality between synthesized speech using CSM LPCs versus mapped LPCs became negligible, suggesting the mapped source is highly effective.
Limitations and Future Work
- Relative Gains of Residuals: The proposed mapping technique does not explicitly consider the relative gains of TM and CSM residuals, which could limit the overall quality.
- Unvoiced Segments: The current method is primarily suitable for voiced regions. Initial attempts at mapping unvoiced regions showed little to no improvement.
- Speaker-Dependence: The approach is speaker-dependent. The authors suggest that a speaker-independent mapping technique for both source and system features is essential for wider applications.
- Future Work: Addressing the limitations regarding relative gains and unvoiced segments, and developing speaker-independent models are identified as future research directions. The technique is also noted as being extendable to voice conversion applications.
Summary updated