TL;DR
This paper tackles the problem of unnatural-sounding voice conversion by introducing a post-processing step that optimizes the sequence of converted speech features. By balancing frame-level conversion accuracy with temporal smoothness, their novel approach significantly improves subjective and objective quality, outperforming existing methods.
What Problem Does This Paper Address?
Voice conversion aims to make speech from one speaker sound like it was spoken by another. Traditional methods often process speech frame-by-frame, mapping source features to target features independently for each time slice. This approach, however, ignores the inherent temporal continuity of natural speech, leading to discontinuities and an unnatural, robotic sound. While some methods attempt to smooth these transitions, they often struggle to balance the accuracy of the frame-level conversion with the desired temporal smoothness, especially when dealing with different speech sounds like vowels and fricatives. This paper addresses this gap by proposing a method to optimize the entire sequence of converted features, ensuring both accurate conversion and natural temporal flow.
Methods
The core of this paper is a novel post-processing technique that refines the output of a standard frame-by-frame voice conversion algorithm. After an initial conversion (using methods like Cluster Partial Least Squares regression, CPLS, which builds local mapping models for different speech clusters), the paper proposes to optimize the resulting feature sequence. This optimization is framed as a Maximum a Posteriori (MAP) estimation problem within a dynamical system.
The key innovation lies in defining a cost function that balances two competing objectives:
- Conversion Error: How closely the converted feature matches the target feature predicted by the initial conversion.
- Smoothness: How well the converted feature sequence adheres to a temporal continuity model, predicting the current feature based on the previous one.
This cost function minimization is then transformed into a MAP estimation problem, which is solved using Sequential Monte Carlo (SMC) methods, specifically a particle filter combined with the Viterbi algorithm. This allows for efficient approximation of the optimal feature sequence. The paper also details how they build local regression matrices using Cluster Partial Least Squares (CPLS) and how they model the temporal dynamics using a simple linear prediction based on source speech dynamics.
Datasets
The paper does not explicitly name specific public datasets. Instead, they describe their experimental setup:
- Training Data: 30 sentences per speaker pair.
- Testing Data: 10 sentences per speaker pair.
- Speaker Pairs: Male-to-Female (M-F) and Female-to-Female (F-F).
- Speech Analysis: The STRAIGHT analysis-synthesis system was used to extract Mel-cepstrum coefficients (MCCs) and other features at 5 ms intervals.
Key Insights and Ideas
- Temporal Continuity is Crucial: The paper strongly emphasizes that ignoring temporal continuity is a major bottleneck in achieving natural-sounding voice conversion.
- MAP Estimation for Sequence Optimization: Framing the problem as MAP estimation within a dynamical system provides a powerful theoretical basis for optimizing the entire feature sequence.
- SMC Methods are Effective: Sequential Monte Carlo methods, particularly particle filters, are shown to be a viable and effective approach for approximating the optimal sequences in this complex optimization problem.
- Balancing Act: The proposed cost function elegantly balances frame-level accuracy with temporal smoothness, allowing for tunable trade-offs.
- CPLS as a Foundation: While the post-processing method is general, using Cluster Partial Least Squares (CPLS) for initial frame-wise mapping is shown to be effective, especially when compared to global regression matrices.
Key Results
- Objective Improvement: The proposed MAPCPLS method achieved lower mean spectral distortion compared to CPLS (without post-processing) and other baseline methods (GRM, CRM). For instance, with 20 clusters, MAPCPLS achieved 5.01 dB distortion, while CPLS achieved 5.16 dB.
- Subjective Preference: Listening tests showed a strong preference for the MAPCPLS method. It was preferred over CPLS in 86.3% of cases and over GRM in 73.8% of cases.
- Particle Filter Efficiency: The number of particles in the SMC method plateaued in terms of improvement after around 100 particles, suggesting a practical limit for computational efficiency.
- Cluster Impact: The number of clusters in the CPLS method was found to be important, with a lower number (around 5-10) generally yielding better results. MAPCPLS showed a notable ability to compensate for errors when the number of clusters was too high.
Limitations and Future Work
- Generalizability of Prediction Model: The paper uses a simple linear model for temporal continuity, predicting the next state based on the previous one. More sophisticated models for temporal dynamics could potentially yield further improvements.
- Parameter Tuning: The scaling factors ($\alpha$ and $\gamma$) in the cost function, which balance conversion error and smoothness, require careful tuning. The paper mentions determining these jointly, but further investigation into their impact and optimal settings could be beneficial.
- Multi-Dimensional Features: While the paper focuses on individual feature sequences, extending the approach to jointly optimize multi-dimensional feature vectors is suggested as a future direction.
- Computational Complexity: While SMC methods provide convergence guarantees, their computational cost can be significant. Exploring more efficient approximations or parallelization strategies could be valuable.
- Comparison Basis: The authors suggest that comparing against GRM (which preserves continuity) is more informative than comparing against conventional GMM methods that do not. This highlights a potential area for future comparative studies.
Summary updated