Abstract
This paper proposes a generative moment matching network (GMMN)-based post-filter that provides inter-utterance pitch variation for deep neural network (DNN)-based singing voice synthesis. The natural pitch variation of a human singing voice leads to a richer musical experience and is used in double-tracking, a recording method in which two performances of the same phrase are recorded and mixed to create a richer, layered sound. However, singing voices synthesized using conventional DNN-based methods never vary because the synthesis process is deterministic and only one waveform is synthesized from one musical score. To address this problem, we use a GMMN to model the variation of the modulation spectrum of the pitch contour of natural singing voices and add a randomized inter-utterance variation to the pitch contour generated by conventional DNN-based singing voice synthesis. Experimental evaluations suggest that 1) our approach can provide perceptible inter-utterance pitch variation while preserving speech quality. We extend our approach to double-tracking, and the evaluation demonstrates that 2) GMMN-based neural double-tracking is perceptually closer to natural double-tracking than conventional signal processing-based artificial double-tracking is.
Introduction
In professional music production, double-tracking (DT) is a standard technique where a singer records the same phrase twice. Mixing these two takes creates a thick, lush, and "layered" sound due to the natural, subtle variations in pitch and timing between performances.
However, current Deep Neural Network (DNN) singing voice synthesis (SVS) systems are deterministic. Given a specific musical score, they produce the exact same waveform every time. This lack of variation leads to two main issues: the "uncanny valley" effect where perfect repetition sounds robotic, and the inability to perform effective double-tracking, as layering identical AI-generated waveforms simply increases volume without adding texture.
Existing solutions like Artificial Double-Tracking (ADT) use signal processing (e.g., delays or chorus effects) to simulate variation, but these often introduce metallic artifacts and unnatural timbre coloration.
Methodology: Neural Double-Tracking
The proposed solution is a GMMN-based random modulation post-filter designed to inject human-like stochasticity into AI vocals. Instead of redesigning the core synthesis engine, this approach adds a smart filter that specifically targets the pitch ($F_0$) contour.
Key Technical Components
- Modulation Spectrum (MS): The system analyzes the modulation spectrum to capture long-term temporal fluctuations in pitch, such as vibrato and subtle drifts, rather than just frame-by-frame values.
- Generative Moment Matching Network (GMMN): A deep generative model trained using Maximum Mean Discrepancy (MMD). GMMNs were selected for their training stability compared to Generative Adversarial Networks (GANs).
- Conditional Generation: The GMMN takes a deterministic pitch contour and a random noise vector as inputs, outputting a "naturalized" pitch contour with human-like variance.
- Neural Double-Tracking (NDT): By mixing the original AI vocal with a version processed by the GMMN post-filter, the system creates a "Neural Double-Tracked" effect that mimics natural layering.
Experimental Evaluation
The model was trained using Japanese singing voice corpora, including the HTS Singing Voice Synthesis Demo and the JSUT-song corpus. To improve robustness, the dataset was augmented by transposing songs by a semitone. Subjective listening tests were conducted to evaluate the perceptible variation and quality of the proposed method.
Key Results
- Perceptible Variation: Listeners successfully distinguished between different "takes" generated by the GMMN, confirming that the model successfully breaks the determinism of standard DNN outputs.
- Quality Preservation: The post-filter maintained the naturalness of the voice. Preference tests showed that the GMMN-filtered voice was rated as natural as the unfiltered version.
- Superior Layering: In double-tracking comparisons, NDT was overwhelmingly preferred over traditional ADT. NDT achieved a preference score of approximately 0.73, compared to 0.26 for ADT, indicating it is perceptually much closer to a naturally layered human performance.
Conclusion
This work demonstrates that modeling the modulation spectrum of pitch using a GMMN can effectively bridge the gap between deterministic AI synthesis and natural human performance. By focusing on stylistic variation rather than just the notes, the system enables high-quality Neural Double-Tracking. Future research will aim to extend this modeling to duration and spectral parameters to capture additional dimensions of human vocal variation.
Summary updated