MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement

Xinyue Yu, Youqing Fang, Pingyu Wu, Guoyang Ye, Wenbo Zhou, Weiming Zhang, Song Xiao

Abstract

Generating expressive and controllable human speech is one of the core goals of generative artificial intelligence, but its progress has long been constrained by two fundamental challenges: the deep entanglement of speech factors and the coarse granularity of existing control mechanisms. To overcome these challenges, we have proposed a novel framework called MF-Speech, which consists of two core components: MF-SpeechEncoder and MF-SpeechGenerator. MF-SpeechEncoder acts as a factor purifier, adopting a multi-objective optimization strategy to decompose the original speech signal into highly pure and independent representations of content, timbre, and emotion. Subsequently, MF-SpeechGenerator functions as a conductor, achieving precise, composable and fine-grained control over these factors through dynamic fusion and Hierarchical Style Adaptive Normalization (HSAN). Experiments demonstrate that in the highly challenging multi-factor compositional speech generation task, MF-Speech significantly outperforms current state-of-the-art methods, achieving a lower word error rate (WER=4.67%), superior style control (SECS=0.5685, Corr=0.68), and the highest subjective evaluation scores(nMOS=3.96, sMOS_emotion=3.86, sMOS_style=3.78). Furthermore, the learned discrete factors exhibit strong transferability, demonstrating their significant potential as a general-purpose speech representation.

Introduction

Generating human speech that is both expressive and controllable remains a primary objective in generative AI. However, two significant hurdles persist: factor entanglement and coarse control mechanisms.

Entanglement refers to the deep intertwining of content (what is said), timbre (speaker identity), and emotion (how it is said) in natural speech. Current models often fail to isolate these, leading to "timbre leakage" where emotional reference clips inadvertently alter the speaker's identity.

Coarse Control stems from blunt integration methods like static concatenation or global modulation. These lack the precision required to balance content fidelity with stylistic nuance at a granular, frame-by-frame level.

Conceptual overview of MF-Speech achieving compositional control by combining content, timbre, and emotion from different sources.

Methodology: The MF-Speech Framework

The MF-Speech framework addresses these issues through a two-part architecture designed for factor "purification" and "conduction."

1. MF-SpeechEncoder (The Purifier)

The MF-SpeechEncoder decomposes raw speech into independent representations using a three-stream architecture:

  • Content Stream: Refines a pre-trained Wav2Vec2 model to suppress residual timbre and emotion.
  • Emotion Stream: Focuses on prosody (pitch and energy) to capture emotional cues.
  • Timbre Stream: Utilizes a SeaNet encoder with multi-head attention to extract stable speaker identity.

To prevent information leakage between these streams, the authors employ Mutual Information (MI) minimization constraints. This mathematically penalizes the model if one stream contains information redundant to another, ensuring high purity in the disentangled factors.

2. MF-SpeechGenerator (The Conductor)

The generator recombines purified factors using two key techniques:

  • Dynamic Fusion Module: Generates time-varying weights for each factor, allowing the model to prioritize content or style dynamically throughout a sentence.
  • Hierarchical Style Adaptive Normalization (HSAN): This module injects timbre and emotion at multiple network layers. By using affine parameters and residual modulation, it ensures style is deeply embedded without compromising the underlying message.
The three-stage training and inference architecture of MF-Speech, highlighting factor disentanglement (Stage 2) and conditional generation (Stage 3).

Experimental Evaluation

The framework was evaluated using the Emotional Speech Dataset (ESD), comparing it against state-of-the-art models like StyleVC, NS2VC, and DDDM-VC.

Multi-factor Compositional Generation

In tasks requiring content from one speaker, timbre from another, and emotion from a third, MF-Speech achieved:

  • Highest Naturalness: nMOS of 3.96.
  • Superior Speaker Similarity: SECS of 0.5685.
  • High Content Integrity: A Word Error Rate (WER) of 4.67%, significantly outperforming DDDM-VC (11.67%) and StyleVC (24.83%).

Ablation Studies

Ablation tests confirmed that removing MI constraints or the HSAN module resulted in "blurred" factor clusters in t-SNE visualizations, validating the necessity of these components for effective disentanglement and control.

Conclusion and Future Work

MF-Speech demonstrates that explicit prosody modeling and hierarchical style injection are critical for natural-sounding, controllable speech synthesis. While the model shows strong generalization on the ESD benchmark, future work will focus on "in-the-wild" audio diversity, improving raw waveform fidelity, and optimizing the architecture for real-time deployment on edge devices.

Summary updated

Abstract

Generating expressive and controllable human speech is one of the core goals of generative artificial intelligence, but its progress has long been constrained by two fundamental challenges: the deep entanglement of speech factors and the coarse granularity of existing control mechanisms. To overcome these challenges, we have proposed a novel framework called MF-Speech, which consists of two core components: MF-SpeechEncoder and MF-SpeechGenerator. MF-SpeechEncoder acts as a factor purifier, adopting a multi-objective optimization strategy to decompose the original speech signal into highly pure and independent representations of content, timbre, and emotion. Subsequently, MF-SpeechGenerator functions as a conductor, achieving precise, composable and fine-grained control over these factors through dynamic fusion and Hierarchical Style Adaptive Normalization (HSAN). Experiments demonstrate that in the highly challenging multi-factor compositional speech generation task, MF-Speech significantly outperforms current state-of-the-art methods, achieving a lower word error rate (WER=4.67%), superior style control (SECS=0.5685, Corr=0.68), and the highest subjective evaluation scores(nMOS=3.96, sMOS_emotion=3.86, sMOS_style=3.78). Furthermore, the learned discrete factors exhibit strong transferability, demonstrating their significant potential as a general-purpose speech representation.
BibTeX
@inproceedings{yu2026mf,
  author = {Xinyue Yu and Youqing Fang and Pingyu Wu and Guoyang Ye and Wenbo Zhou and Weiming Zhang and Song Xiao},
  title = {{MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=EX7eLapCzZ},
}
Figure