Generative Moment Matching Network-based Random Modulation Post-filter for DNN-based Singing Voice Synthesis and Neural Double-tracking

Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, Hiroshi Saruwatari

Abstract

This paper proposes a generative moment matching network (GMMN)-based post-filter that provides inter-utterance pitch variation for deep neural network (DNN)-based singing voice synthesis. The natural pitch variation of a human singing voice leads to a richer musical experience and is used in double-tracking, a recording method in which two performances of the same phrase are recorded and mixed to create a richer, layered sound. However, singing voices synthesized using conventional DNN-based methods never vary because the synthesis process is deterministic and only one waveform is synthesized from one musical score. To address this problem, we use a GMMN to model the variation of the modulation spectrum of the pitch contour of natural singing voices and add a randomized inter-utterance variation to the pitch contour generated by conventional DNN-based singing voice synthesis. Experimental evaluations suggest that 1) our approach can provide perceptible inter-utterance pitch variation while preserving speech quality. We extend our approach to double-tracking, and the evaluation demonstrates that 2) GMMN-based neural double-tracking is perceptually closer to natural double-tracking than conventional signal processing-based artificial double-tracking is.

Introduction

In professional music production, double-tracking (DT) is a standard technique where a singer records the same phrase twice. Mixing these two takes creates a thick, lush, and "layered" sound due to the natural, subtle variations in pitch and timing between performances.

However, current Deep Neural Network (DNN) singing voice synthesis (SVS) systems are deterministic. Given a specific musical score, they produce the exact same waveform every time. This lack of variation leads to two main issues: the "uncanny valley" effect where perfect repetition sounds robotic, and the inability to perform effective double-tracking, as layering identical AI-generated waveforms simply increases volume without adding texture.

Comparison of human and synthesized singing voices. Human singers produce various waveforms for the same score, while synthesized voices are deterministic.

Existing solutions like Artificial Double-Tracking (ADT) use signal processing (e.g., delays or chorus effects) to simulate variation, but these often introduce metallic artifacts and unnatural timbre coloration.

Methodology: Neural Double-Tracking

The proposed solution is a GMMN-based random modulation post-filter designed to inject human-like stochasticity into AI vocals. Instead of redesigning the core synthesis engine, this approach adds a smart filter that specifically targets the pitch ($F_0$) contour.

Key Technical Components

  • Modulation Spectrum (MS): The system analyzes the modulation spectrum to capture long-term temporal fluctuations in pitch, such as vibrato and subtle drifts, rather than just frame-by-frame values.
  • Generative Moment Matching Network (GMMN): A deep generative model trained using Maximum Mean Discrepancy (MMD). GMMNs were selected for their training stability compared to Generative Adversarial Networks (GANs).
  • Conditional Generation: The GMMN takes a deterministic pitch contour and a random noise vector as inputs, outputting a "naturalized" pitch contour with human-like variance.
  • Neural Double-Tracking (NDT): By mixing the original AI vocal with a version processed by the GMMN post-filter, the system creates a "Neural Double-Tracked" effect that mimics natural layering.
Comparison of double-tracking methods: Natural Double-tracking (DT), Artificial Double-tracking (ADT), and the proposed Neural Double-tracking (NDT).

Experimental Evaluation

The model was trained using Japanese singing voice corpora, including the HTS Singing Voice Synthesis Demo and the JSUT-song corpus. To improve robustness, the dataset was augmented by transposing songs by a semitone. Subjective listening tests were conducted to evaluate the perceptible variation and quality of the proposed method.

Key Results

  • Perceptible Variation: Listeners successfully distinguished between different "takes" generated by the GMMN, confirming that the model successfully breaks the determinism of standard DNN outputs.
  • Quality Preservation: The post-filter maintained the naturalness of the voice. Preference tests showed that the GMMN-filtered voice was rated as natural as the unfiltered version.
  • Superior Layering: In double-tracking comparisons, NDT was overwhelmingly preferred over traditional ADT. NDT achieved a preference score of approximately 0.73, compared to 0.26 for ADT, indicating it is perceptually much closer to a naturally layered human performance.

Conclusion

This work demonstrates that modeling the modulation spectrum of pitch using a GMMN can effectively bridge the gap between deterministic AI synthesis and natural human performance. By focusing on stylistic variation rather than just the notes, the system enables high-quality Neural Double-Tracking. Future research will aim to extend this modeling to duration and spectral parameters to capture additional dimensions of human vocal variation.

Summary updated

Abstract

This paper proposes a generative moment matching network (GMMN)-based post-filter that provides inter-utterance pitch variation for deep neural network (DNN)-based singing voice synthesis. The natural pitch variation of a human singing voice leads to a richer musical experience and is used in double-tracking, a recording method in which two performances of the same phrase are recorded and mixed to create a richer, layered sound. However, singing voices synthesized using conventional DNN-based methods never vary because the synthesis process is deterministic and only one waveform is synthesized from one musical score. To address this problem, we use a GMMN to model the variation of the modulation spectrum of the pitch contour of natural singing voices and add a randomized inter-utterance variation to the pitch contour generated by conventional DNN-based singing voice synthesis. Experimental evaluations suggest that 1) our approach can provide perceptible inter-utterance pitch variation while preserving speech quality. We extend our approach to double-tracking, and the evaluation demonstrates that 2) GMMN-based neural double-tracking is perceptually closer to natural double-tracking than conventional signal processing-based artificial double-tracking is.
BibTeX
@inproceedings{tamaru2019generative,
  author = {Hiroki Tamaru and Yuki Saito and Shinnosuke Takamichi and Tomoki Koriyama and Hiroshi Saruwatari},
  title = {{Generative Moment Matching Network-based Random Modulation Post-filter for DNN-based Singing Voice Synthesis and Neural Double-tracking}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=II3ZB0a8bq},
}
Figure