Abstract
Singing Voice Synthesis (SVS) has long been hampered by a lack of large-scale, high-quality datasets. Unlike Text-to-Speech (TTS), SVS requires precise alignment of lyrics, pitch, and duration, often necessitating professional recording environments. This paper introduces a novel "Curation via Synthesis" approach, leveraging the ACE Studio synthesizer alongside expert manual tuning to "upcycle" existing datasets. We present two new multi-singer corpora: ACE-Opencpop and ACE-KiSing, totaling over 160 hours of high-quality data. Our experiments demonstrate that these datasets significantly enhance SVS performance through transfer learning and joint training, outperforming existing multi-singer benchmarks.
1. Introduction
While Text-to-Speech (TTS) technology has advanced rapidly due to the availability of massive speech corpora, Singing Voice Synthesis (SVS) remains constrained by data scarcity. High-quality SVS data is notoriously difficult to produce; it requires professional vocalists, studio-grade recording environments, and meticulous annotation of musical scores, including phoneme-level alignment and pitch contours. Consequently, most open-source SVS datasets are limited to single-singer recordings or very short durations, hindering the development of robust, multi-singer models capable of generalizing across diverse vocal styles.
To address this "data-starved" environment, we propose a pipeline to scale up SVS data by synthesizing new audio from existing musical scores. By combining high-fidelity synthesis with human-in-the-loop refinement, we provide the community with two substantial new resources: ACE-Opencpop and ACE-KiSing.
2. Methodology: Curation via Synthesis
The core innovation of this work is the "Curation via Synthesis" pipeline. Rather than recording new human sessions, we utilize ACE Studio, a professional-grade singing synthesizer, to generate multi-singer variations of existing musical scores. To ensure the synthetic data meets the rigorous standards required for training state-of-the-art models, we employed a four-stage refinement process:
- Data Preparation: Musical scores (MIDI) were aligned with lyrics and phonemes to create a structured foundation.
- Information Verification: Musical experts manually reviewed and refined the MIDI scores to ensure perfect correspondence with the intended vocal delivery.
- Tuning for Voice Match: To avoid the "uncanny valley" of robotic synthesis, human editors iteratively adjusted the synthesized output. This included modifying fundamental frequency (F0) contours, adding realistic breath sounds, and fine-tuning vibrato to replicate human-like nuances.
- Singer Adaptation: The data was filtered to ensure stylistic consistency. Phrases that sounded unnatural for a specific virtual singer—such as notes outside their comfortable vocal range—were removed.
3. The ACE Corpora
This methodology resulted in two major benchmarks that significantly expand the volume of available SVS data:
- ACE-Opencpop: Built upon the original 5.2-hour Opencpop dataset, this corpus expands the repertoire across 30 different singers, resulting in 128.9 hours of audio. This represents a 25-fold increase in data volume.
- ACE-KiSing: An extension of the KiSing database, featuring 34 singers and 32.5 hours of audio. A key feature of ACE-KiSing is its focus on melisma (the singing of a single syllable over multiple notes). While the original Opencpop contains only 1.5% melisma phrases, ACE-KiSing increases this to 27%, providing a critical resource for modeling complex vocal flourishes.
4. Experimental Results
We evaluated the utility of these datasets using two prominent SVS architectures: Xiaoice (a two-stage model) and VISinger2 (an end-to-end model).
4.1 Performance Gains
The inclusion of synthetic data provided measurable improvements in synthesis quality. In "Joint Training" scenarios, adding ACE-KiSing to the original KiSing dataset raised the Mean Opinion Score (MOS) from 3.58 to 3.83.
4.2 Transfer Learning and Pre-training
One of the most significant findings is the effectiveness of ACE-Opencpop as a pre-training resource. Models pre-trained on ACE-Opencpop and subsequently fine-tuned on the original human-recorded Opencpop showed a marked improvement in MOS (from 2.78 to 3.08 for the Xiaoice model). Furthermore, objective metrics indicate that models pre-trained on ACE-Opencpop achieve lower pitch error (F0 RMSE) and lower Mel-Cepstral Distortion (MCD) compared to those pre-trained on other multi-singer datasets like M4Singer.
5. Discussion and Limitations
While "Curation via Synthesis" offers a scalable path forward for SVS, several challenges remain. There is still a perceptible quality gap between synthesized voices (MOS ~3.8) and human ground truth (MOS ~4.5+). Additionally, the use of commercial software like ACE Studio for research data generation introduces complex legal and ethical considerations regarding End User License Agreements (EULAs) and the implications of training AI models on AI-generated content.
Future work will focus on closing the quality gap and expanding this curation strategy to a broader range of musical genres and languages beyond the current focus on Mandarin, Japanese, and English.
6. Conclusion
The introduction of ACE-Opencpop and ACE-KiSing marks a significant step toward solving the data scarcity problem in Singing Voice Synthesis. By demonstrating that high-quality synthetic data can serve as a powerful booster for model performance, this work provides a blueprint for the rapid scaling of vocal datasets across the research community.
Summary updated