TL;DR
This paper introduces an "Extended Multiple Regression Gaussian Mixture Model" (EMR-GMM) for adaptive voice quality control in eigenvoice conversion. It significantly improves the ability to adapt a source speaker's voice to a target speaker's quality, even with limited target speech, while maintaining intuitive control over the converted voice characteristics.
What Problem Does This Paper Address?
Voice conversion (VC) aims to change a speaker's voice characteristics while preserving linguistic content. While existing VC methods can alter voice quality, they often struggle with two key challenges:
- Intuitive Control: Manipulating a few parameters to achieve specific voice qualities (like age or gender) is difficult. Changes often affect multiple quality aspects simultaneously.
- Adaptation to New Speakers: Training VC models typically requires large amounts of parallel data (source and target speech pairs). This is impractical when adapting to a new speaker with only a small amount of their speech available, or no parallel data at all.
Previous work introduced "Multiple Regression GMM" (MR-GMM) for more intuitive control by mapping voice quality factors to specific parameters. However, this approach limited adaptation performance because the subspace representing voice qualities was too small when using only a few control parameters. This paper addresses the need for a method that offers both intuitive control and robust adaptation to new speakers with limited data.
Methods
The core contribution is the Extended Multiple Regression GMM (EMR-GMM). Here's how it builds upon previous work:
- Eigenvoice Conversion (EVC): The foundation is a "one-to-many" EVC system. This uses a pre-trained model (EV-GMM) that can convert a source speaker's voice into many different target speaker voices. The eigenvoice technique allows for rapid adaptation by manipulating a small set of global weighting parameters.
- Multiple Regression GMM (MR-GMM): To enable intuitive control, the MR-GMM was proposed. It models the target mean vector as a linear combination of a bias vector and "representative vectors." These representative vectors are optimized to capture specific voice quality factors (e.g., "elder/younger," "heavy/light"). A "voice quality (VQ) control vector" (
wc) then weights these representative vectors, allowing for direct manipulation of these qualities. - Extended MR-GMM (EMR-GMM): This is the novel contribution. The EMR-GMM extends the MR-GMM by introducing additional representative vectors (
H) and corresponding adaptive weight vectors (wh). This expands the subspace used to model target voice qualities. The idea is that while the original representative vectors capture well-defined qualities, the additional ones can model residual voice characteristics that are not easily described by the primitive word pairs. The target mean vector is now modeled as:μ = Bwc + Hwh + b.
The paper details training and adaptation procedures for the EMR-GMM, including both supervised training with prestored speakers and unsupervised adaptation to new speakers.
Datasets
- Training Data: The MR-GMM and EMR-GMM were trained using parallel data from 40 speakers: 10 male and 10 female speakers from the Japanese Newspaper Article Sentences (JNAS) database, and 10 senior male and 10 senior female speakers from the Senior-Japanese Newspaper Article Sentences (S-JNAS) database. Each speaker uttered 50 phonetically balanced sentences. A female speaker was used as the source speaker.
- Evaluation Data: For adaptation evaluation, 10 target speakers (5 male, 5 female) from JNAS, not used in training, were selected. Two sentences were used for adaptation, and 51 sentences for evaluation.
Key Insights and Ideas
- Expanding the Voice Quality Subspace is Crucial for Adaptation: The core insight is that while MR-GMM provides good control, its limited subspace hinders adaptation performance when target speech is available. EMR-GMM's extension of this subspace with additional representative vectors directly addresses this limitation.
- Separation of Controlled vs. Residual Characteristics: The EMR-GMM implicitly separates voice characteristics into those directly controllable via primitive word pairs (modeled by
Bwc) and those that are more nuanced or residual (modeled byHwh). This allows for both fine-grained control and robust adaptation. - Unsupervised Adaptation Benefits from Extended Subspace: The paper demonstrates that the EMR-GMM significantly improves unsupervised adaptation performance. This means the system can learn to mimic a new speaker's voice quality more effectively with less data, a critical step towards practical, personalized voice conversion.
Key Results
- Manual Voice Quality Control: Perceptual evaluations showed that both MR-GMM and EMR-GMM allow for intuitive control of voice quality (e.g., elder/younger, heavy/light). The perceptual scores correlated well with the VQ control vector settings.
- Unsupervised Adaptation Performance:
- Objective Metric (Mel-cepstral distortion): EMR-GMM achieved a lower mel-cepstral distortion (5.20 dB) compared to MR-GMM (5.54 dB) after adaptation. This indicates better spectral conversion accuracy.
- Subjective Metrics (Speech Quality & Speaker Individuality): * Preference Test: EMR-GMM was preferred significantly more often (around 70% preference score) for speech quality compared to MR-GMM (around 30%).
- XAB Test (Speaker Individuality): EMR-GMM also showed a higher preference score (around 70%) for speaker individuality, meaning the converted voices sounded more like the target speaker.
Limitations and Future Work
- Limitations:
- The paper notes that a decrease in the number of representative vectors in MR-GMM reduces adaptation performance. This implies a trade-off between the number of control parameters and adaptation capability.
- While EMR-GMM improves adaptation, the paper doesn't explicitly quantify the minimum amount of target speech needed for effective adaptation.
- Future Work:
- The paper's conclusion suggests that the EMR-GMM is effective for improving adaptation while maintaining controllability. Future work could explore further optimizing the selection of primitive word pairs and the number of additional representative vectors to strike an even better balance between control and adaptation.
- Investigating the interpretability of the additional representative vectors (
H) could provide deeper insights into what aspects of voice quality they capture.
Summary updated