Introduction
Singing Voice Synthesis (SVS) typically employs a modular pipeline where an acoustic model generates Mel-spectrograms from musical scores, and a vocoder synthesizes audio from these features. However, a "feature gap" often degrades performance: vocoders trained on ground-truth (GT) Mel-spectrograms struggle with the imperfect, "noisy" features produced by acoustic models during inference. This paper explores joint-training strategies to bridge this disparity.
Methodology: Bridging the Feature Gap
To address the feature gap, the authors propose a joint-training framework combining a Transformer-based acoustic model with a HiFi-GAN vocoder. The core idea is to expose the vocoder to the acoustic model's predictions during training, allowing it to learn robustness to synthetic artifacts.
The framework utilizes a scheduled sampling strategy governed by a weighting function $p(t)$. This function determines the probability of feeding predicted features versus GT features to the vocoder at epoch $t$.
Scheduled Sampling Strategies
The researchers explored several transition patterns for $p(t)$:
- JT-scratch: Training from scratch with $p(t)=1$ (always using predicted features).
- Two-Stage: The traditional approach where $p(t)=0$ (always using GT features).
- JT-Linear/Logistic: Gradual transitions from GT to predicted features.
- JT-FT (Joint Fine-Tuning): An abrupt "vertical" switch from GT to predicted features after a pre-training phase.
Key Insights
The study revealed that the JT-FT strategy—switching abruptly to predicted features after pre-training—outperformed gradual transitions. This suggests that starting from stable, pre-trained checkpoints is more critical for SVS stability than the smoothness of the feature transition. Furthermore, joint training acts as a form of regularization, providing better gradient feedback to the acoustic model and resulting in more realistic high-frequency spectral textures.
Unlike "black-box" end-to-end models (like VITS) that use latent representations, this approach keeps the Mel-spectrogram as an intermediate step. This maintains interpretability, making the system easier to debug and ensuring the output remains grounded in musical constraints like pitch and timing.
Experimental Results
Evaluations on the Ofuton-P and Opencpop datasets demonstrated the superiority of the JT-FT approach:
- Subjective Quality: On the Ofuton dataset, JT-FT achieved a Mean Opinion Score (MOS) of 3.87, significantly higher than the two-stage baseline (3.54) and the latent-based VISinger (3.48).
- Objective Metrics: Mel Cepstral Distortion (MCD) improved from 6.60 to 6.39.
- Visual Fidelity: Spectrogram analysis confirmed that JT-FT produces clearer high-frequency details and avoids the "checkerboard" noise often seen in models trained from scratch.
Conclusion
By systematically exploring joint-training schedules, this work demonstrates that a fine-tuning approach (JT-FT) effectively bridges the feature gap in SVS. This method maintains the interpretability of Mel-spectrograms while achieving audio quality superior to traditional pipelines and modern latent models. Future research will focus on scaling these strategies to larger, multi-singer datasets to ensure broader applicability.
Summary updated