TL;DR
This paper tackles the challenge of generating natural-sounding singing voices with HMM-based systems, particularly when dealing with pitches that are rare in the training data. They propose a novel method of augmenting training data by "pitch-shifting" existing recordings, which significantly improves the naturalness of synthesized singing voices.
What Problem Does This Paper Address?
Singing voice synthesis (SVS) aims to create artificial singing voices, with HMM-based approaches being a popular choice due to their efficiency and flexibility. A core challenge in HMM-based SVS is the "corpus-based" nature of the models: their quality heavily depends on the training data. Specifically, pitches that rarely appear in the training set are difficult for the HMM to model accurately, leading to unnatural synthesized singing. This is a significant problem because singing involves a wide range of pitches, and covering all possibilities with a finite dataset is practically impossible. The authors aim to improve the robustness of HMM-based SVS by addressing this data sparsity issue for rare pitches.
Methods
The paper introduces a technique called pitch-shifted pseudo training data. The core idea is to artificially expand the training dataset by taking existing singing voice recordings and shifting their fundamental frequency (Fo) up or down. This process generates new "pseudo" data that effectively increases the coverage of different pitches without requiring new recordings.
Here's a breakdown of their approach:
- Data Augmentation: Existing singing voice data is pitch-shifted. Mel-Cepstral coefficients and vibrato parameters are copied from the original data, assuming they are less affected by small pitch shifts. New labels are created to reflect the shifted pitch.
- HMM Training: The augmented dataset (original + pitch-shifted pseudo data) is used to train context-dependent HMMs.
- Contextual Modeling: The system uses decision trees for context-clustering to manage the vast number of contextual factors in singing voices. The Minimum Description Length (MDL) criterion is employed to determine the optimal size of these decision trees. The authors note that the increased training data due to pitch-shifting leads to larger decision trees, and controlling their size with the MDL heuristic weight ($\alpha$) is crucial.
- Synthesis: The trained HMMs are used in a standard HMM-based synthesis pipeline, generating spectral, excitation, and vibrato parameters, which are then converted into an audio waveform using an MLSA filter.
Datasets
- Training Data: A singing voice database consisting of 70 Japanese children's songs sung by one female singer. This database also includes corresponding musical scores in MusicXML format.
- Evaluation Data: Ten songs from the training set were used for evaluation, divided into musical phrases.
Key Insights and Ideas
- Pitch-Shifted Data as a Solution to Sparsity: The most significant insight is that augmenting training data with pitch-shifted versions of existing recordings is an effective way to address the problem of rare pitches in HMM-based SVS. This bypasses the need for extensive new recordings.
- Impact of Decision Tree Size: The effectiveness of the pitch-shifting technique is strongly linked to how the decision trees for context-clustering are managed. Simply increasing data doesn't automatically guarantee better results; the size of the decision trees (controlled by the MDL heuristic weight $\alpha$) needs to be adjusted appropriately.
- Naturalness Improvement: The proposed method demonstrably improves the naturalness of synthesized singing voices, particularly for pitches that were previously underrepresented.
Key Results
Subjective listening tests were conducted using Mean Opinion Scores (MOS) on a scale of 1 to 5.
- Improved Naturalness: The proposed pitch-shifted pseudo training data method consistently improved the MOS scores compared to the baseline system.
- Optimal Pitch Shift Parameter: The best results were achieved with a pitch-shift parameter ($\alpha$) of 5.0.
- Decision Tree Size Reduction: As the pitch-shift parameter increased (leading to more data), the size of the decision trees (number of leaf nodes) decreased significantly for spectrum, Fo, and duration models. For example, with 60 songs, the number of leaf nodes for spectrum dropped from 1762 (baseline) to 539 for $\alpha=5.0$.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the optimal value for the MDL heuristic weight ($\alpha$) remains a subject of research.
- While mel-Cepstral coefficients and vibrato parameters were copied, their behavior with larger pitch shifts might warrant more sophisticated handling.
- Future Work:
- The authors suggest evaluating the impact of different amounts of pitch-shifting.
- Further research into optimizing the MDL criterion and its heuristic weight for this specific application is implied.
Summary updated