Abstract
Disentangling speaker identity from pitch ($f_0$) is a core challenge in voice conversion. While bottleneck auto-encoders are commonly used, fixed-size bottlenecks often struggle to balance information loss across different audio types like speech and singing. We introduce VaSAB (Variable Size Adaptive information Bottleneck), a novel approach that replaces fixed architectural constraints with tunable dropout-based "valves." By utilizing Random (RaBo) and Global (GloBo) dropout strategies, VaSAB allows a single model to adaptively manage information density. Our evaluations show that VaSAB improves pitch accuracy and subjective naturalness, enabling high-quality voice transformation even at extreme pitch ranges.
1. Introduction
In AI-driven voice transformation, the primary goal is disentanglement: isolating speaker identity from prosodic features like pitch ($f_0$), rhythm, and emotion. Bottleneck auto-encoders achieve this by forcing audio through a restricted latent space, encouraging the model to discard redundant information. However, speech and singing possess distinct information densities, leading to a "Goldilocks" problem: a bottleneck optimized for speech may be too restrictive for singing, while one optimized for singing may allow pitch leakage in speech, making transformation impossible.
2. The VaSAB Framework
VaSAB addresses this by treating the bottleneck as a tunable valve rather than a fixed structural limit. Instead of reducing the number of latent dimensions, VaSAB employs dropout on the latent code to restrict information flow.
Three primary dropout strategies were explored:
- Random dropout Bottleneck (RaBo): Applies independent dropout to each feature, allowing for continuous control over the effective bottleneck size.
- Hierarchical dropout Bottleneck (HiBo): Sets features to zero in a fixed sequence, testing if structured information reduction improves performance.
- Global dropout Bottleneck (GloBo): Occasionally zeros the entire latent code during training. This acts as a "stress test," forcing the decoder to rely exclusively on external control inputs like target $f_0$.
3. Experimental Setup
Models were trained on three distinct dataset configurations:
- Speech (sp): Approximately 70 hours of multi-speaker speech data.
- Singing (si): A curated collection of singing databases, oversampled for high-pitched recordings to improve performance in extreme ranges.
- Voice (vo): A balanced 50/50 combination of speech and singing data.
4. Results and Analysis
The effectiveness of the adaptive bottleneck was evaluated through objective pitch accuracy and subjective Mean Opinion Scores (MOS).
Objective analysis (Figure 1) reveals that the RaBo bottleneck significantly reduces $f_0$ error compared to the "NoBo" (No Dropout) baseline, particularly by eliminating pitch discretization artifacts. Furthermore, training on the combined "vo" dataset improved performance in extreme pitch ranges for both speech and singing, suggesting that the diverse pitch distribution in singing data enhances the model's fundamental understanding of $f_0$.
Subjective results for singing voice (Figure 4) demonstrate that the universal Ravo model (RaBo trained on the combined dataset) consistently outperformed state-of-the-art fixed-bottleneck baselines (Nosp/Nosi). The Ravo model achieved the highest naturalness scores across nearly all transposition ranges, proving that a single model can effectively handle both speech and singing.
5. Conclusion
VaSAB demonstrates that dropout is an effective mechanism for controlling information capacity in voice conversion models. By decoupling bottleneck size from architecture, VaSAB enables a single, universal model to handle the varying demands of speech and singing. While Hierarchical dropout (HiBo) underperformed compared to Random dropout, the overall success of the RaBo and GloBo strategies provides a robust path forward for flexible voice transformation systems.
Summary updated