A Systematic Exploration of Joint-Training for Singing Voice Synthesis

Yuning Wu, Yifeng Yu, Jiatong Shi, Tao Qian, Qin Jin

Introduction

Singing Voice Synthesis (SVS) typically employs a modular pipeline where an acoustic model generates Mel-spectrograms from musical scores, and a vocoder synthesizes audio from these features. However, a "feature gap" often degrades performance: vocoders trained on ground-truth (GT) Mel-spectrograms struggle with the imperfect, "noisy" features produced by acoustic models during inference. This paper explores joint-training strategies to bridge this disparity.

Methodology: Bridging the Feature Gap

To address the feature gap, the authors propose a joint-training framework combining a Transformer-based acoustic model with a HiFi-GAN vocoder. The core idea is to expose the vocoder to the acoustic model's predictions during training, allowing it to learn robustness to synthetic artifacts.

Comparison of (a) JT-FT, (b) Two-Stage, and (c) JT-scratch architectures.

The framework utilizes a scheduled sampling strategy governed by a weighting function $p(t)$. This function determines the probability of feeding predicted features versus GT features to the vocoder at epoch $t$.

Scheduled Sampling Strategies

The researchers explored several transition patterns for $p(t)$:

  • JT-scratch: Training from scratch with $p(t)=1$ (always using predicted features).
  • Two-Stage: The traditional approach where $p(t)=0$ (always using GT features).
  • JT-Linear/Logistic: Gradual transitions from GT to predicted features.
  • JT-FT (Joint Fine-Tuning): An abrupt "vertical" switch from GT to predicted features after a pre-training phase.
Scheduled sampling weighting functions p(t) for various training strategies.

Key Insights

The study revealed that the JT-FT strategy—switching abruptly to predicted features after pre-training—outperformed gradual transitions. This suggests that starting from stable, pre-trained checkpoints is more critical for SVS stability than the smoothness of the feature transition. Furthermore, joint training acts as a form of regularization, providing better gradient feedback to the acoustic model and resulting in more realistic high-frequency spectral textures.

Unlike "black-box" end-to-end models (like VITS) that use latent representations, this approach keeps the Mel-spectrogram as an intermediate step. This maintains interpretability, making the system easier to debug and ensuring the output remains grounded in musical constraints like pitch and timing.

Experimental Results

Evaluations on the Ofuton-P and Opencpop datasets demonstrated the superiority of the JT-FT approach:

  • Subjective Quality: On the Ofuton dataset, JT-FT achieved a Mean Opinion Score (MOS) of 3.87, significantly higher than the two-stage baseline (3.54) and the latent-based VISinger (3.48).
  • Objective Metrics: Mel Cepstral Distortion (MCD) improved from 6.60 to 6.39.
  • Visual Fidelity: Spectrogram analysis confirmed that JT-FT produces clearer high-frequency details and avoids the "checkerboard" noise often seen in models trained from scratch.

Conclusion

By systematically exploring joint-training schedules, this work demonstrates that a fine-tuning approach (JT-FT) effectively bridges the feature gap in SVS. This method maintains the interpretability of Mel-spectrograms while achieving audio quality superior to traditional pipelines and modern latent models. Future research will focus on scaling these strategies to larger, multi-singer datasets to ensure broader applicability.

Summary updated

Abstract

There has been a growing interest in using end-to-end acoustic models for singing voice synthesis (SVS). Typically, these models require an additional vocoder to transform the generated acoustic features into the final waveform. However, since the acoustic model and the vocoder are not jointly optimized, a gap can exist between the two models, leading to suboptimal performance. Although a similar problem has been addressed in the TTS systems by joint-training or by replacing acoustic features with a latent representation, adopting corresponding approaches to SVS is not an easy task. How to improve the joint-training of SVS systems has not been well explored. In this paper, we conduct a systematic investigation of how to better perform a joint-training of an acoustic model and a vocoder for SVS. We carry out extensive experiments and demonstrate that our joint-training strategy outperforms baselines, achieving more stable performance across different datasets while also increasing the interpretability of the entire framework.
BibTeX
@inproceedings{wu2023systematic,
  author = {Yuning Wu and Yifeng Yu and Jiatong Shi and Tao Qian and Qin Jin},
  title = {{A Systematic Exploration of Joint-Training for Singing Voice Synthesis}},
  booktitle = {ISCSLP},
  year = {2023},
}
Figure