VaSAB: The variable size adaptive information bottleneck for disentanglement on speech and singing voice

Frederik Bous, Axel RoebelarXiv 2023 · DOI · arXiv

Abstract

Disentangling speaker identity from pitch ($f_0$) is a core challenge in voice conversion. While bottleneck auto-encoders are commonly used, fixed-size bottlenecks often struggle to balance information loss across different audio types like speech and singing. We introduce VaSAB (Variable Size Adaptive information Bottleneck), a novel approach that replaces fixed architectural constraints with tunable dropout-based "valves." By utilizing Random (RaBo) and Global (GloBo) dropout strategies, VaSAB allows a single model to adaptively manage information density. Our evaluations show that VaSAB improves pitch accuracy and subjective naturalness, enabling high-quality voice transformation even at extreme pitch ranges.

1. Introduction

In AI-driven voice transformation, the primary goal is disentanglement: isolating speaker identity from prosodic features like pitch ($f_0$), rhythm, and emotion. Bottleneck auto-encoders achieve this by forcing audio through a restricted latent space, encouraging the model to discard redundant information. However, speech and singing possess distinct information densities, leading to a "Goldilocks" problem: a bottleneck optimized for speech may be too restrictive for singing, while one optimized for singing may allow pitch leakage in speech, making transformation impossible.

2. The VaSAB Framework

VaSAB addresses this by treating the bottleneck as a tunable valve rather than a fixed structural limit. Instead of reducing the number of latent dimensions, VaSAB employs dropout on the latent code to restrict information flow.

Three primary dropout strategies were explored:

  • Random dropout Bottleneck (RaBo): Applies independent dropout to each feature, allowing for continuous control over the effective bottleneck size.
  • Hierarchical dropout Bottleneck (HiBo): Sets features to zero in a fixed sequence, testing if structured information reduction improves performance.
  • Global dropout Bottleneck (GloBo): Occasionally zeros the entire latent code during training. This acts as a "stress test," forcing the decoder to rely exclusively on external control inputs like target $f_0$.

3. Experimental Setup

Models were trained on three distinct dataset configurations:

  • Speech (sp): Approximately 70 hours of multi-speaker speech data.
  • Singing (si): A curated collection of singing databases, oversampled for high-pitched recordings to improve performance in extreme ranges.
  • Voice (vo): A balanced 50/50 combination of speech and singing data.

4. Results and Analysis

The effectiveness of the adaptive bottleneck was evaluated through objective pitch accuracy and subjective Mean Opinion Scores (MOS).

Mean f0 error across target pitch ranges for different bottleneck types and training configurations.

Objective analysis (Figure 1) reveals that the RaBo bottleneck significantly reduces $f_0$ error compared to the "NoBo" (No Dropout) baseline, particularly by eliminating pitch discretization artifacts. Furthermore, training on the combined "vo" dataset improved performance in extreme pitch ranges for both speech and singing, suggesting that the diverse pitch distribution in singing data enhances the model's fundamental understanding of $f_0$.

Mean Opinion Scores (MOS) for singing voice transposition across different models.

Subjective results for singing voice (Figure 4) demonstrate that the universal Ravo model (RaBo trained on the combined dataset) consistently outperformed state-of-the-art fixed-bottleneck baselines (Nosp/Nosi). The Ravo model achieved the highest naturalness scores across nearly all transposition ranges, proving that a single model can effectively handle both speech and singing.

5. Conclusion

VaSAB demonstrates that dropout is an effective mechanism for controlling information capacity in voice conversion models. By decoupling bottleneck size from architecture, VaSAB enables a single, universal model to handle the varying demands of speech and singing. While Hierarchical dropout (HiBo) underperformed compared to Random dropout, the overall success of the RaBo and GloBo strategies provides a robust path forward for flexible voice transformation systems.

Summary updated

Abstract

The information bottleneck auto-encoder is a tool for disentanglement commonly used for voice transformation. The successful disentanglement relies on the right choice of bottleneck size. Previous bottleneck auto-encoders created the bottleneck by the dimension of the latent space or through vector quantization and had no means to change the bottleneck size of a specific model. As the bottleneck removes information from the disentangled representation, the choice of bottleneck size is a trade-off between disentanglement and synthesis quality. We propose to build the information bottleneck using dropout which allows us to change the bottleneck through the dropout rate and investigate adapting the bottleneck size depending on the context. We experimentally explore into using the adaptive bottleneck for pitch transformation and demonstrate that the adaptive bottleneck leads to improved disentanglement of the F0 parameter for both, speech and singing voice leading to improved synthesis quality. Using the variable bottleneck size, we were able to achieve disentanglement for singing voice including extremely high pitches and create a universal voice model, that works on both speech and singing voice with improved synthesis quality.
BibTeX
@misc{bous2023vasab,
  author = {Frederik Bous and Axel Roebel},
  title = {{VaSAB: The variable size adaptive information bottleneck for disentanglement on speech and singing voice}},
  year = {2023},
  eprint = {2310.03444},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.03444},
}
Figure