TL;DR
This paper compares traditional methods of estimating speech features (fundamental frequency and formants) with a novel approach that predicts these features directly from Mel-Frequency Cepstral Coefficients (MFCCs). The findings show that MFCC-based prediction is more robust to noise, particularly for voicing classification, and offers competitive or superior accuracy for formant prediction, paving the way for more efficient speech reconstruction in distributed systems.
What Problem Does This Paper Address?
In modern speech processing, especially in Distributed Speech Recognition (DSR) systems, the raw audio waveform is often not transmitted to the processing backend. Instead, compact representations like MFCCs are sent. However, MFCCs are designed to discard fundamental frequency (f0) information, which is crucial for reconstructing the original speech waveform. Traditional methods for estimating f0 and formants rely on the full waveform, making them unusable in DSR. This paper addresses the challenge of obtaining accurate f0 and formant information from MFCCs alone, comparing prediction-based methods against traditional estimation techniques using hand-corrected reference data. The ultimate goal is to enable high-quality speech reconstruction from MFCCs, reducing the need to transmit additional f0 and voicing information.
Methods
The paper evaluates several established methods for estimating acoustic speech features directly from the audio waveform. For fundamental frequency (f0), these include:
- AMDF (Average Magnitude Difference Function): A simple autocorrelation-based method.
- ANALYZE (Speech Filing System): Uses normalized cross-correlation on linear prediction coefficient residuals.
- RAPT (Robust Algorithm for Pitch Tracking): Another method based on normalized cross-correlation.
- YIN: Combines autocorrelation and AMDF with further processing to improve accuracy.
- ETSI: A heuristic decision tree approach using frequency-domain spectral peaks and time-domain correlation scores.
For formant frequencies, the methods include:
- ESPS (Speech Filing System): Solves for roots of a linear predictor polynomial, using dynamic programming for tracking.
- LPC (Linear Predictive Coding): Uses LPC analysis followed by Kalman filtering.
- MFCC: A method that estimates formants from a spectral magnitude representation derived from MFCCs.
The core novel contribution lies in the MAP (Maximum A Posteriori) prediction of acoustic speech features from MFCC vectors. This method models the joint density of acoustic features (f0, formants, voicing) and MFCCs using Gaussian Mixture Models (GMMs). A voicing decision is made for each MFCC vector, and then acoustic features are predicted from the appropriate GMM based on the predicted voicing class. This prediction is then smoothed.
Datasets
The paper utilizes two primary databases for evaluation:
- VTR (Vocal Tract Resonance) database: Contains hand-corrected vocal tract resonance information for a subset of the TIMIT database. This was used for evaluating formant frequency prediction. The database was split into training and testing sets.
- Hand-corrected f0 tracks: Reference data from a female US English speaker, obtained using laryngograph measurements. This was used for evaluating f0 estimation and prediction. This data was also split into training and testing sets.
To assess robustness to noise, "exhibition hall" noise from the ETSI Aurora database was added to the clean speech data at various Signal-to-Noise Ratios (SNRs) ranging from 20 dB to -5 dB.
Key Insights and Ideas
- MFCC Prediction Excels in Voicing Classification: The MFCC-based prediction method demonstrates superior voicing classification accuracy compared to estimation methods, especially in noisy conditions. This is attributed to the fact that voicing is largely dependent on energy, which is well-represented in MFCCs.
- Noise Robustness is Key: As noise levels increase, traditional estimation methods struggle, often underestimating the number of voiced frames. In contrast, MFCC-based prediction shows significantly better robustness, making it more suitable for real-world, noisy environments.
- Formant Prediction Outperforms Estimation: For formant frequencies, prediction from MFCCs is found to be more accurate than estimation in both clean and noisy speech. This suggests that the spectral envelope information implicitly present in MFCCs is effectively leveraged by the prediction model.
- f0 Prediction Challenges: While MFCC-based prediction is robust, accurately predicting f0 from MFCCs is inherently more difficult because MFCCs are designed to discard f0 harmonics. However, even with higher f0 errors, the GMM prediction method can still produce intelligible speech.
- Reconstruction Quality: Speech reconstruction experiments show that while accurate formants are important, the GMM prediction method, despite its accuracy, didn't always yield the best reconstruction when formant bandwidths weren't predicted as accurately as with estimation methods. This highlights the interplay of different acoustic features in reconstruction.
Key Results
- Voicing Classification: MFCC-based prediction achieved the lowest voicing class error, particularly as noise increased.
- f0 Accuracy: In clean speech, estimation methods generally outperformed prediction. However, as noise increased, prediction became more competitive, and matched condition testing (training on noisy data) showed significant potential for f0 prediction accuracy.
- Formant Accuracy: GMM-based prediction consistently yielded formant frequencies closest to the hand-corrected references across various noise levels.
- Speech Reconstruction (f0): The GMM prediction method resulted in intelligible speech even at low SNRs (10 dB and 0 dB), demonstrating its practical utility despite not always having the lowest percentage f0 error.
- Speech Reconstruction (Formants): While GMM prediction produced accurate formants, estimation methods sometimes led to better reconstruction quality, suggesting that accurate formant bandwidth prediction is also critical.
Limitations and Future Work
- f0 Prediction Difficulty: The paper acknowledges that MFCCs are "too coarse to contain f0 harmonics," making precise f0 prediction challenging.
- Formant Bandwidth Prediction: The study notes that while formants were predicted well, the prediction of formant bandwidths was not as accurate as estimation, impacting reconstruction quality in some cases.
- Future Work: The authors suggest that further improvements in predicting formant bandwidths alongside frequencies would enhance speech reconstruction. The potential for predicting other acoustic features from MFCCs for DSR applications is also implied.
Summary updated