TL;DR
This paper introduces a novel method to enhance singing voice conversion (SVC) by injecting harmonic signals directly into adversarial waveform generation models. By feeding both raw and filtered sine excitations, the authors significantly improve the quality and timbre similarity of generated singing, addressing issues like pitch jitters and U/V errors common in current GAN-based SVC.
What Problem Does This Paper Address?
Singing Voice Conversion (SVC) aims to change the singer's voice while preserving the original melody and lyrics. While Generative Adversarial Networks (GANs) have become a popular choice for generating high-quality singing audio due to their speed and efficiency, they often suffer from instability. This instability can manifest as undesirable artifacts like pitch jitters and incorrect voicing (U/V errors), which degrade the smoothness and continuity of the generated harmonics, a crucial component for natural-sounding singing. Existing methods struggle to consistently produce high-fidelity singing audio that accurately captures the target singer's timbre without introducing these artifacts.
Methods
The core of this paper lies in augmenting existing adversarial waveform generation models with harmonic signals. The authors propose two types of harmonic signals:
- Raw Sine Excitation: This is a basic harmonic signal directly derived from the fundamental frequency (F0) of the singing voice. It's generated by synthesizing sine waves at harmonic frequencies, with their amplitudes determined by the F0.
- Filtered Sine Excitation: To better match the target audio, the raw sine excitation is further processed by a Linear Time-Varying (LTV) filter. The coefficients of this LTV filter are dynamically estimated by a neural network, allowing it to adapt to the input features and better capture the nuances of the target timbre and energy distribution across harmonics.
These harmonic signals are then integrated as additional inputs into two mainstream adversarial waveform generation frameworks: MelGAN and ParallelWaveGAN. The integration strategy differs slightly between the two models, involving upsampling or downsampling to match the feature scales before concatenation.
Datasets
- Training and Evaluation Dataset: A clean, multi-singer singing corpus consisting of 200 Mandarin pop songs (10 hours) sung by 10 singers (5 male, 5 female).
- Feature Extraction: F0, loudness, and 80-dimensional Mel-spectrograms were extracted at a 16kHz sample rate with a 10ms frameshift.
- Content Representation: Phonetic Posteriorgrams (PPGs) were extracted using a Transformer-CTC ASR model trained on a large-scale Mandarin speech corpus.
Key Insights and Ideas
- Harmonics are Crucial for Singing Quality: The results strongly suggest that explicitly providing harmonic information significantly improves SVC performance, particularly in sound quality. This highlights the importance of these spectral components for realistic singing synthesis.
- Filtered Excitation Offers Superior Matching: The LTV filtered sine excitation proved more effective than raw sine excitation, leading to better timbre similarity and a more natural auditory experience. This indicates that dynamic filtering can better adapt harmonic content to the target singer.
- Universal Applicability: The proposed method shows consistent improvements across two different adversarial vocoder architectures (MelGAN and ParallelWaveGAN), suggesting its generalizability to various GAN-based SVC systems.
- Addressing GAN Instability: By providing explicit harmonic cues, the method helps mitigate the pitch jitter and U/V errors often associated with unstable GAN training, leading to smoother and more continuous harmonic structures.
Key Results
- MOS Scores: The proposed method consistently achieved higher Mean Opinion Scores (MOS) for both sound quality and singer similarity compared to baseline models, across both clean and noisy test sets.
- Quality Improvement: For the clean test set, models with harmonic signals saw an improvement of over 0.3 in sound quality MOS.
- Filtered Excitation Benefits: The filtered sine excitation further improved performance over raw sine excitation, demonstrating its added value.
- Qualitative Analysis: Spectrogram analysis confirmed that the generated audio exhibited smoother and more continuous harmonics, with the filtered excitation better matching the target audio's spectral characteristics.
Limitations and Future Work
- Dataset Specificity: While the corpus is multi-singer, it is limited to Mandarin pop songs. Future work could explore generalization to other genres and languages.
- Computational Cost: While GANs are generally efficient, the additional processing for harmonic signal generation and integration might introduce some computational overhead, though the paper doesn't quantify this.
- Future Directions: The authors suggest further exploring different types of harmonic signals and their integration strategies, as well as investigating the impact of these signals on other aspects of singing synthesis beyond timbre and quality.
Summary updated