A comparative study of spectral transformation techniques for singing voice synthesis

Siu Wa Lee, Zhizheng Wu, Minghui Dong, Xiaohai Tian, Haizhou Li

Abstract

This study presents a comparative evaluation of four spectral transformation techniques for speech-to-singing (S2S) synthesis, aimed at capturing the unique acoustic characteristics of professional singing. The research compares statistical voice conversion (ML-GMM), weighted frequency warping (WFW), HMM-based model adaptation, and a rule-based baseline. Results from objective and subjective evaluations on a Mandarin Chinese dataset indicate that while ML-GMM is more mathematically accurate in spectral mapping, HMM-based model adaptation yields higher subjective quality due to its integration of musical context.

Introduction

Singing voice synthesis involves more than simply applying a melody to speech. Professional singing is characterized by specific acoustic phenomena, such as resonance tuning and the "singing formant"—a spectral peak around 3 kHz that allows a singer to be heard over an instrumental accompaniment. A critical challenge in Speech-to-Singing (S2S) systems is the spectral transformation required to map the acoustic "fingerprint" of a person's speech to that of their singing. This study addresses the lack of comparative research on which machine learning approaches most effectively handle this complex mapping.

Methodology

The researchers evaluated four distinct approaches within the Tandem-STRAIGHT framework for high-quality audio analysis and reconstruction:

  1. ML-GMM (Voice Conversion): A statistical method using a Gaussian Mixture Model to map speech features directly to singing features. This approach is purely acoustic and lacks linguistic or musical context.
  2. WFW (Weighted Frequency Warping): A variant of voice conversion that warps the frequency axis to better preserve audio quality compared to standard GMMs.
  3. Model Adaptation (HSMM-based): This method adapts a standard Hidden Semi-Markov Model (HSMM) trained on speech to singing data. It utilizes full-context labels, incorporating musical information such as pitch, duration, and tempo.
  4. Rule-based S2S: A baseline method that manually adjusts consonant/vowel duration and applies a 3 kHz bandpass filter to simulate the singing formant.

Experimental Setup

The study utilized a specialized dataset featuring 50 Mandarin Chinese pop songs performed by a professional male singer (approximately 194 minutes). Parallel recordings of the singer reading the lyrics and corresponding MIDI files provided the necessary ground-truth context. The data was partitioned into 1,848 segments for training and 54 segments for testing.

Results and Discussion

The evaluation revealed a notable discrepancy between objective metrics and human perception.

Subjective Quality and Naturalness

In Mean Opinion Score (MOS) tests, the Model Adaptation approach significantly outperformed other methods in terms of subjective quality when large datasets were available. This success is attributed to the inclusion of musical context (notes, tempo), which is vital because singing spectra vary significantly with pitch.

Mean Opinion Score (MOS) for naturalness/quality across large and little data cases.

Speaker Similarity

For large datasets, both ML-GMM and Model Adaptation effectively preserved the singer's identity. However, when training data was scarce (the "little data" case), ML-GMM proved superior at maintaining speaker similarity, even if the overall audio quality was lower.

Mean Opinion Score (MOS) for speaker similarity across large and little data cases.

Objective Accuracy vs. Perception

Interestingly, ML-GMM achieved the highest objective accuracy, with a cepstral distance of 4.74, compared to 5.15 for Model Adaptation. Despite this mathematical edge, listeners preferred the Adaptation results, highlighting that objective signal-processing metrics do not always align with human perception of "pleasantness" or naturalness. The poor performance of the rule-based method (cepstral distance of 7.37) further suggests that singing transformation is a highly non-linear process that cannot be captured by simple filters.

Conclusion

The study demonstrates that incorporating musical context is essential for high-quality singing synthesis. While statistical voice conversion is robust for identity preservation with limited data, model adaptation provides more natural results for larger datasets. Future work will explore Speaker Adaptive Training (SAT) to develop singer-independent models, potentially reducing the data requirements for synthesizing new voices.

Summary updated

BibTeX
@inproceedings{lee2014comparative,
  author = {Siu Wa Lee and Zhizheng Wu and Minghui Dong and Xiaohai Tian and Haizhou Li},
  title = {{A comparative study of spectral transformation techniques for singing voice synthesis}},
  booktitle = {Interspeech},
  year = {2014},
  doi = {10.21437/interspeech.2014-536},
}
Figure