TL;DR
This paper introduces a new method for speaker adaptive training (SAT) applied to tensor-based arbitrary speaker conversion (TASC). By integrating SAT with their novel tensor-based speaker representation, the authors achieve more flexible and precise voice conversion, outperforming previous methods even with limited adaptation data.
What Problem Does This Paper Address?
Voice conversion (VC) aims to transform speech from one speaker to another while preserving the linguistic content. A major challenge in VC is achieving arbitrary speaker conversion, meaning the ability to convert to any target speaker without needing extensive parallel training data (utterances with the same content from both source and target speakers). Traditional GMM-based methods like Eigenvoice Conversion (EVC) use speaker spaces to address this, but their representation of speaker characteristics can be limiting, hindering performance improvements with more data. The authors previously proposed a tensor-based approach (TASC) to represent speaker spaces more flexibly, and this paper investigates how Speaker Adaptive Training (SAT), a technique known to improve speaker-independent models, can further enhance TASC.
Methods
The core of this paper is the integration of Speaker Adaptive Training (SAT) with their previously proposed Tensor-based Arbitrary Speaker Conversion (TASC).
- Tensor-based Speaker Space (TASC): Instead of representing a speaker as a high-dimensional GMM supervector (as in EVC), TASC represents each speaker as a matrix. This matrix's rows correspond to GMM components, and columns to mean vector dimensions. This allows for a more flexible representation of speaker characteristics. The "speaker space" is then derived using Tucker decomposition on a tensor formed by these speaker matrices.
- Speaker Adaptive Training (SAT): SAT is applied to TASC to construct a canonical model that is shared across speakers, aiming for improved precision and flexibility. This involves maximizing the likelihood of all individual speaker models simultaneously. The paper details an iterative update scheme for the shared parameters (GMM weights, bases matrices, covariance matrices) and individual speaker weight matrices.
Datasets
- Training Data: ATR Japanese speech database B-set was used. This included one male reference speaker and 273 pre-stored speakers (137 male, 136 female). Each speaker provided 50 sentences.
- Adaptation/Evaluation Data: Six new speakers (3 male, 3 female) were selected for evaluation. Adaptation was performed using 1 to 16 utterances, and evaluation used the remaining 21 utterances.
Key Insights and Ideas
- SAT Enhances Tensor Representation: Applying SAT to TASC significantly improves voice conversion performance. This suggests that SAT is effective not only for compacting variances in traditional GMMs but also for refining the more flexible tensor-based speaker representations.
- TASC's Flexibility Pays Off: TASC, even without SAT, outperforms EVC. This highlights the advantage of the tensor-based representation in capturing speaker characteristics more effectively than supervector-based methods.
- SAT's Impact on Variance: Figure 1 shows that SAT, for both EVC and TASC, effectively reduces the variance of target features. This indicates that SAT helps in creating more compact and consistent speaker models.
- Optimal Parameters Vary with Adaptation: Unlike EVC, where optimal representative parameters (J) remain constant, TASC's optimal representative parameters (K) vary with the number of adaptation utterances. This suggests TASC is more adaptive and can better utilize varying amounts of data.
Key Results
- Objective Evaluation (MCD): Figure 2 shows that TASC with SAT consistently achieves the lowest mel-cepstral distortion (MCD) across different numbers of adaptation sentences, outperforming EVC (with and without SAT) and even a "Traditional" VC method (which requires parallel data).
- Performance Improvement: TASC with SAT outperformed "Traditional" VC even with only 16 adaptation utterances, demonstrating its ability to capture essential speaker information efficiently.
- Parameter Optimization: Table 1 shows that for TASC, the optimal size of the representative matrix (K) can be reduced after SAT, suggesting SAT's effect in compacting variations.
Limitations and Future Work
- Subjective Evaluation: The paper focuses on objective evaluations (MCD). The authors suggest that future work should include large-scale subjective evaluations to assess the perceptual quality of the converted speech.
- Parameter Optimization: Further investigation into optimizing the representative parameters for TASC is recommended.
- Bayesian Treatment: The authors propose exploring a Bayesian approach for arbitrary speaker conversion, encompassing speaker space representation, structure optimization, and adaptation using tensor representations.
Summary updated