TL;DR
This paper tackles the issue of lost spectral detail in voice conversion (VC) systems, which degrades the naturalness of synthesized speech. They propose a novel "exemplar-based spectral detail compensation" method that uses a locally linear embedding (LLE) approach to predict and reintroduce these missing details, significantly improving speech quality.
What Problem Does This Paper Address?
Voice conversion (VC) aims to transform speech from one speaker to another while preserving the linguistic content. A common approach involves using vocoders, like STRAIGHT, to handle the conversion of spectral features. While methods like Gaussian Mixture Models (GMMs) are effective for converting low-dimensional spectral features (e.g., mel-cepstral coefficients or MCCs), they often result in a loss of fine-grained spectral details when reconstructing the high-dimensional spectral envelopes. This loss of detail can lead to unnatural-sounding speech and reduced similarity to the target speaker. The paper addresses this specific problem: how to recover and reintroduce these lost spectral details to improve the overall quality of converted speech.
Methods
The proposed method, "Exemplar-Based Spectral Detail Compensation" (GMM-SDC), works in two stages:
- Offline Stage:
- Paired Dictionary Construction: This involves preparing parallel speech corpora from source and target speakers. Spectral envelopes and MCCs are extracted. Crucially, "target spectral details" are computed by subtracting reconstructed target spectral envelopes from the original target spectral envelopes. Dynamic features (delta and delta-delta) are appended to both source spectral envelopes and target spectral details. Dynamic Time Warping (DTW) is used to align source and target frames, and then the aligned source spectral envelopes and target spectral details are used to construct paired dictionaries.
- Online Stage (Runtime Conversion):
- Spectral Detail Prediction: For each source speech frame, its spectral envelope vector is used to find its K nearest neighbors in the source dictionary. A reconstruction weight vector is calculated using Locally Linear Embedding (LLE) to represent the source spectral envelope as a weighted combination of its neighbors.
- Compensation: This same reconstruction weight vector is then applied to the corresponding target spectral detail dictionary to predict the target spectral details.
- Post-processing: The predicted spectral details are then refined using a Maximum Likelihood Parameter Generation (MLPG) and Global Variance (GV) module.
- Final Synthesis: Finally, these compensated spectral details are added to the spectral envelopes obtained from a baseline GMM-based VC system. The STRAIGHT vocoder is then used to reconstruct the waveform using these compensated spectral envelopes along with other acoustic features (pitch, aperiodicity).
The core novelty lies in using LLE to learn a mapping from source spectral envelopes to target spectral details, leveraging the idea that similar local geometric structures in the source and target feature spaces can be exploited.
Datasets
The experiments were conducted using the Sinica COSPRO speech corpus [21]. This corpus contains Mandarin parallel speech utterances. For their experiments, they used the "intonation-balanced dataset" (COSPRO 03), which includes 3 female and 2 male speakers. They set up 20 conversion pairs: 8 intra-gender and 12 inter-gender. For training, development, and testing, they used 10, 40, and 43 utterance pairs per conversion pair, respectively. No new benchmarks were introduced; standard datasets were used for evaluation.
Key Insights and Ideas
- Spectral Detail is Crucial: The paper strongly emphasizes that while low-dimensional features like MCCs are computationally efficient, they inherently lose high-frequency spectral detail. This loss, even if small, significantly impacts the naturalness and perceived quality of the converted speech.
- LLE for Detail Prediction: The application of Locally Linear Embedding (LLE) is a key insight. LLE is known for its ability to preserve local geometric structures. The authors hypothesize that similar local structures exist between source spectral envelopes and target spectral details, allowing LLE to effectively predict these details.
- Exemplar-Based Compensation: The method leverages "exemplars" (neighboring data points in the dictionaries) to reconstruct the missing spectral information. This exemplar-based approach allows for a more data-driven and potentially more accurate reconstruction than purely parametric methods.
- Complementary to Existing Methods: The proposed compensation method is designed to be a post-processing step that can be applied to any existing spectral conversion system that operates on low-dimensional features, making it a versatile enhancement.
Key Results
- Objective Evaluation: Modulation Spectrum (MS) analysis showed that the proposed GMM-SDC method produced converted spectral envelopes with MS closer to natural target speech, particularly in higher frequency bins (250Hz and 450Hz). Spectrogram comparisons also visually confirmed the reintroduction of more spectral details.
- Subjective Evaluation:
- Naturalness: A preference test revealed that GMM-SDC significantly outperformed the baseline GMM system, with 79.22% of listeners preferring the GMM-SDC output compared to 20.78% for the baseline (p < 0.000).
- Speaker Similarity: An ABX test showed no significant difference in speaker similarity between GMM and GMM-SDC (86.88% vs. 87.81%, p = 0.975). This indicates that the spectral detail compensation enhances naturalness without negatively impacting speaker identity.
Limitations and Future Work
- Limitations:
- The paper focuses on the STRAIGHT vocoder. While the compensation method is presented as generalizable, its performance with other vocoders is not explicitly discussed.
- The performance of inter-gender VC pairs was not detailed in the subjective evaluation, though it was mentioned that they were identified correctly in preliminary results.
- The computational complexity of the LLE-based SDP method, especially with large dictionaries, might be a consideration.
- Future Work:
- The authors suggest that the proposed method can improve any SC methods operating on low-dimensional spectral features using the STRAIGHT vocoder, implying potential for broader application.
- Further exploration of the method's effectiveness with different vocoders and across various types of voice conversion (e.g., emotional VC) could be valuable.
Summary updated