Introduction
Singing Voice Synthesis (SVS) presents unique challenges compared to standard Text-to-Speech (TTS) systems. Singing involves significantly wider pitch ranges, longer sustained vowels, and more intense emotional expressions. Current research in SVS faces three primary hurdles: data scarcity, high computational costs for high-fidelity generation, and poor generalization to unseen singers. Researchers from Zhejiang University have addressed these issues by releasing OpenSinger, a large-scale dataset, and Multi-Singer, a high-fidelity GAN-based vocoder.
The OpenSinger Dataset
To address the data scarcity issue, the authors introduced OpenSinger, a massive new benchmark for the SVS community. The dataset comprises 50 hours of high-quality singing audio from 93 professional singers (41 female, 25 male, and others). Recorded at 24kHz in professional studios, the dataset includes manual annotations for lyrics and phoneme alignment. Statistical analysis of this data confirms that singing voices have significantly higher variance in pitch and phoneme duration compared to standard speech, justifying the need for specialized SVS models.
Multi-Singer Architecture
The proposed Multi-Singer system is a Generative Adversarial Network (GAN) designed for high-fidelity and high-speed synthesis. Its architecture incorporates three key innovations:
- Frequency-Adaptive Multi-Band Generator: The model decomposes the audio into four frequency bands (two high, two low). This allows the model to use specialized parameters for different frequency characteristics, significantly boosting both training and inference speed.
- Singer Conditional Discriminator: To ensure the output matches the intended singer's identity, the discriminator evaluates both the realism of the audio and its alignment with a specific singer's characteristics using an LSTM-based module.
- Singer Perceptual Loss: This auxiliary loss function utilizes a pre-trained speaker encoder to minimize the distance between synthetic and ground-truth voices in a high-level feature space, forcing the generator to prioritize the unique texture and timbre of the voice.
Key Results and Performance
Multi-Singer demonstrates superior performance in both quality and efficiency. In Mean Opinion Score (MOS) evaluations, it achieved a score of 3.98, significantly outperforming baseline models such as Multi-band MelGAN and Multi-band WaveRNN.
Beyond quality, the model is exceptionally fast. It operates at a Real-Time Factor (RTF) of 0.008, which is approximately 125 times faster than real-time on a single NVIDIA 2080Ti GPU. Furthermore, the system exhibits strong generalization; even for "unseen" singers not present in the training set, it maintains a high cosine similarity (0.967) to the original identity, proving it effectively learns the underlying mechanics of singing rather than just memorizing voices.
Conclusion
Multi-Singer and the OpenSinger dataset represent a significant step forward for SVS. By combining a multi-band GAN architecture with singer-specific conditioning and perceptual losses, the system achieves state-of-the-art results with remarkable efficiency. While the model excels at technical reconstruction, future work will focus on modeling emotional expressiveness—such as breathy or raspy styles—to make synthetic voices sound even more human.
Summary updated