Generative Moment Matching Network-based Random Modulation Post-filter for DNN-based Singing Voice Synthesis and Neural Double-tracking

Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, Hiroshi Saruwatari

TL;DR

Researchers have developed a generative post-filter that adds natural, randomized pitch variations to AI-synthesized singing. By using a Generative Moment Matching Network (GMMN) to model the "modulation spectrum" of human voices, they can create "Neural Double-Tracking"—layering multiple unique-sounding takes from a single AI model to create a richer, more professional musical sound.

What Problem Does This Paper Address?

If you ask a human to sing the same line twice, they will never produce the exact same waveform. These tiny, "inter-utterance" variations in pitch and timing are what make a voice sound alive. In professional music production, this is exploited through double-tracking (DT): recording a singer twice and mixing the takes to create a thick, lush, and "layered" sound.

Current Deep Neural Network (DNN) singing voice synthesis (SVS) systems are deterministic. Given a musical score, they produce the exact same output every single time. This creates two problems:

  1. The "Uncanny Valley" of Perfection: The lack of natural variation can make the voice sound robotic or sterile.
  2. The Double-Tracking Dilemma: You can't double-track a deterministic AI. If you layer two identical waveforms, you just get a louder version of the same sound, not the rich texture of two different performances.

Existing workarounds like Artificial Double-Tracking (ADT) use signal processing tricks (like delays or chorus effects), but these often introduce metallic artifacts and "timbre coloration."

Methods

The authors propose a GMMN-based random modulation post-filter. Instead of trying to rebuild the entire synthesis engine, they add a smart filter at the end of the process that specifically targets the pitch ($F_0$) contour.

Key Technical Components:

  • Modulation Spectrum (MS): Rather than looking at the pitch frame-by-frame, the system analyzes the modulation spectrum. This captures the long-term temporal structure of how a singer’s pitch fluctuates over time (e.g., vibrato and subtle drifts).
  • Generative Moment Matching Network (GMMN): This is a deep generative model similar to a GAN but trained using Maximum Mean Discrepancy (MMD). The authors chose GMMN because it is more stable and easier to train than GANs, which often suffer from "minimax" optimization headaches.
  • Conditional Generation: The GMMN takes the "perfect" deterministic pitch contour from a standard AI singer and a random noise vector as inputs. It then outputs a "naturalized" version of that pitch contour that retains the melody but adds human-like variance.
  • Neural Double-Tracking (NDT): To create a double-tracked effect, they take the original AI vocal and mix it with a version processed by their GMMN post-filter.

Datasets

The researchers used several Japanese singing voice corpora:

  • HTS Singing Voice Synthesis Demo: 31 songs.
  • JSUT-song corpus: 26 songs.
  • In-house data: 9 songs from the same singer as the JSUT corpus.
  • Data Augmentation: They tripled the dataset size by transposing the songs up and down a semitone to improve the model's robustness.

Key Insights and Ideas

  • Focus on the "Vibe," not just the Notes: By modeling the modulation spectrum rather than the raw pitch values, the GMMN captures the style of human variation without breaking the underlying melody.
  • Post-Filtering is Efficient: This approach doesn't require retraining the massive underlying SVS models. It acts as a "plugin" that can be applied to existing high-quality synthesis systems.
  • Stochasticity is a Feature: In most AI tasks, we want the same input to give the same output. Here, the authors argue that for creative arts, non-determinism is essential for realism. By injecting a random noise vector, they allow a producer to "roll the dice" until they get a vocal take they like.

Key Results

The authors conducted subjective listening tests using a crowdsourcing platform to compare their "Neural Double-Tracking" (NDT) against traditional "Artificial Double-Tracking" (ADT).

  • Perceptible Variation: Listeners could clearly distinguish between different "takes" generated by the GMMN, whereas they couldn't tell the difference between standard DNN outputs.
  • Quality Preservation: The post-filter added variation without degrading the naturalness of the voice. In preference tests, the GMMN-filtered voice was rated as being just as natural as the unfiltered version.
  • Superior Layering: In the double-tracking test, the proposed NDT was overwhelmingly preferred over traditional signal-processing ADT. The NDT scores were roughly 0.73 compared to 0.26 for ADT, suggesting the neural approach felt significantly more like a "naturally layered" human performance.

Limitations and Future Work

While the results for pitch variation are impressive, the paper notes a few areas for improvement:

  • Beyond Pitch: This model currently only varies the pitch ($F_0$). Real singers also vary their duration (timing) and spectral parameters (the "texture" or "color" of the voice) between takes. The authors plan to model these in the future.
  • Language and Style: The current study focused on Japanese singing. It remains to be seen how well these modulation patterns translate across different languages or extreme vocal styles (like opera vs. rock).
  • End-to-End Integration: While the post-filter is convenient, integrating this stochastic behavior directly into the synthesis engine might yield even more cohesive results.

Summary updated

Abstract

This paper proposes a generative moment matching network (GMMN)-based post-filter that provides inter-utterance pitch variation for deep neural network (DNN)-based singing voice synthesis. The natural pitch variation of a human singing voice leads to a richer musical experience and is used in double-tracking, a recording method in which two performances of the same phrase are recorded and mixed to create a richer, layered sound. However, singing voices synthesized using conventional DNN-based methods never vary because the synthesis process is deterministic and only one waveform is synthesized from one musical score. To address this problem, we use a GMMN to model the variation of the modulation spectrum of the pitch contour of natural singing voices and add a randomized inter-utterance variation to the pitch contour generated by conventional DNN-based singing voice synthesis. Experimental evaluations suggest that 1) our approach can provide perceptible inter-utterance pitch variation while preserving speech quality. We extend our approach to double-tracking, and the evaluation demonstrates that 2) GMMN-based neural double-tracking is perceptually closer to natural double-tracking than conventional signal processing-based artificial double-tracking is.
BibTeX
@inproceedings{tamaru2019generative,
  author = {Hiroki Tamaru and Yuki Saito and Shinnosuke Takamichi and Tomoki Koriyama and Hiroshi Saruwatari},
  title = {{Generative Moment Matching Network-based Random Modulation Post-filter for DNN-based Singing Voice Synthesis and Neural Double-tracking}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=II3ZB0a8bq},
}

Loading related content...

Figure