TL;DR
This paper investigates how well different voice conversion (VC) methods handle background noise, particularly in singing voice conversion. They found that a "vocoder-free" direct waveform modification approach is significantly more robust to noise than traditional vocoder-based methods, suggesting it's better suited for real-world applications where clean audio is rare.
What Problem Does This Paper Address?
Voice conversion (VC) aims to change a speaker's voice characteristics while preserving the linguistic content. This technology has potential applications in areas like augmenting vocal production and speech synthesis. However, a major hurdle for practical VC is its performance degradation in the presence of background noise (e.g., environmental sounds, music accompaniment). Existing methods often rely on noise suppression before conversion, but this can introduce its own artifacts. This paper tackles the challenge of developing VC systems that are inherently robust to background sounds, rather than relying on pre-processing steps. Specifically, they focus on singing voice conversion (SVC) and compare two main VC frameworks: traditional vocoder-based VC and a newer vocoder-free approach using direct waveform modification.
Methods
The paper compares two primary statistical voice conversion frameworks:
- Vocoder-based VC: This traditional approach involves analyzing input speech to extract acoustic features like fundamental frequency (F0), spectral envelope, and aperiodic components. These features are then converted using a trained model. Finally, a vocoder synthesizes the converted speech waveform from these features. The authors note that errors in F0 estimation, spectral modeling, and source excitation modeling within the vocoder can degrade sound quality.
- Vocoder-free VC based on DIFFVC: This method, specifically DIFFVC (Direct Waveform Modification using Log-Spectral Differential), aims to avoid the vocoder stage. Instead of converting spectral envelopes, it converts the differentials of log spectral features. This converted sequence is then used to directly filter the input speech waveform. The key advantage is that it can preserve the original source excitation signal, potentially leading to better robustness.
Both methods utilize a Gaussian Mixture Model (GMM) for the conversion model, which is advantageous as the same GMM can be shared between both frameworks, minimizing differences due to the modeling technique itself.
To objectively evaluate robustness, the authors propose using the kurtosis ratio of power spectrum components before and after conversion. Kurtosis is a statistical measure of the "tailedness" of a probability distribution. The kurtosis ratio, when applied to power spectrum components, is hypothesized to capture changes in spectral structure caused by noise and conversion. This metric is appealing because it can be calculated without needing a reference target signal.
Datasets
- Training Data: A singing voice dataset of Japanese songs was used. 50 phrases were randomly selected for training.
- Evaluation Data: Clean singing voices were used for training. For evaluation, noisy singing voices were created by superimposing environmental noise or accompaniment sound onto clean singing voices at various Signal-to-Noise Ratios (SNRs).
Key Insights and Ideas
- Vocoder-free is More Robust: The most significant finding is that the vocoder-free DIFFVC approach demonstrates substantially better robustness to background sounds compared to the vocoder-based approach. This suggests that avoiding the vocoder stage is crucial for handling noisy inputs in SVC.
- Direct Waveform Modification Preserves Structure: The vocoder-free method's ability to retain the original source excitation signal seems to be key to its robustness. This allows it to maintain a more natural spectral structure even under noisy conditions.
- Kurtosis Ratio as an Objective Metric: The kurtosis ratio of power spectrum components shows promise as an objective metric for evaluating VC robustness. It can differentiate between the spectral changes introduced by different VC methods under noise, and importantly, it doesn't require a clean target reference signal.
- Ideal Conversion vs. Noisy Conversion: The paper distinguishes between "ideal" conversion (clean input converted to clean output, but with added noise for evaluation) and "noisy" conversion (noisy input converted to noisy output). This helps isolate the effect of the VC method itself versus the effect of the initial noise.
Key Results
- Subjective Evaluation:
- Under both environmental noise and accompaniment sound, the naturalness of converted singing voices degraded significantly with decreasing SNR for the vocoder-based VC.
- The vocoder-free DIFFVC showed much less degradation. For example, at 5 dB SNR under environmental noise, DIFFVC's performance was comparable to the vocoder-based VC at 25 dB SNR.
- Objective Metric (Kurtosis Ratio):
- For clean DIFFVC + noise (ideal case), the kurtosis ratio remained close to 1, indicating minimal spectral change.
- Noisy VC (vocoder-based) showed kurtosis ratios significantly greater than 1, suggesting unnatural harmonic structures were introduced.
- Noisy DIFFVC (vocoder-free) maintained kurtosis ratios close to 1, especially at higher SNRs, indicating it preserved the spectral structure better. However, at lower SNRs under accompaniment sound, the kurtosis ratio tended to decrease, suggesting some spectral changes occurred.
Limitations and Future Work
- Limitations:
- While the kurtosis ratio is promising, it's noted that for accompaniment sounds at low SNRs, the DIFFVC's kurtosis ratio decreased, indicating some spectral changes still occurred.
- The paper focuses on same-gender singing voice conversion without F0 transformation.
- Future Work:
- The authors plan to incorporate the kurtosis ratio as an additional constraint within the conversion process to further improve performance under noisy conditions, particularly for accompaniment sounds.
- Further investigation into how to improve DIFFVC's robustness at very low SNRs is implied.
Summary updated