MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement

Xinyue Yu, Youqing Fang, Pingyu Wu, Guoyang Ye, Wenbo Zhou, Weiming Zhang, Song Xiao

TL;DR

MF-Speech tackles a longstanding hurdle in expressive voice conversion: cleanly separating speech into content, timbre, and emotion, and recombining them arbitrarily. By pairing a three-stream, contrastive factor purifier (MF-SpeechEncoder) with a multi-level conditioning generator (MF-SpeechGenerator) powered by Hierarchical Style Adaptive Normalization (HSAN), the system enables fine-grained, compositional control without factor leakage.

What Problem Does This Paper Address?

Generating controllable speech requires manipulating three core attributes: what is said (content), who is saying it (timbre), and how it is said (emotion). Current voice conversion and speech synthesis models face two major roadblocks:

  1. Factor Entanglement ("Gene Hybridization"): Content, timbre, and emotion are naturally intertwined. Existing disentanglement methods act as coarse filters, causing attribute leakage (e.g., source speaker timbre leaking into target speech or pitch altering identity).
  2. Coarse Control Mechanisms ("Command Failure"): Even when factors are partially separated, generators typically inject them via static concatenation or global conditioning. This sledgehammer approach fails to dynamically weight factors over time, causing models to struggle to balance content intelligibility and stylistic accuracy.

Methods

MF-Speech introduces a modular, three-stage pipeline to achieve clean factor separation and granular synthesis:

Raw Speech ──> [MF-SpeechEncoder] ──> Discrete RVQ Tokens (Content, Emotion, Timbre)
                                              │
                                              ▼
                    [MF-SpeechGenerator] ──> Dynamic Fusion Module (Time-varying Gating)
                                              │
                                              ▼
                                     [HSAN Layers + Decoder] ──> Synthesized Waveform
  • Stage 1 (Feature Foundation): Trains a baseline convolutional autoencoder (SeaNet encoder/decoder) to ensure high-fidelity conversion between raw waveforms and latent features.
  • Stage 2 (MF-SpeechEncoder): Decomposes speech into three discrete streams via Residual Vector Quantization (RVQ):
  • Content Module: Pre-trained Wav2Vec 2.0 refined via sentence-level content contrastive learning.
  • Emotion Module: Explicitly guided by F0 and energy predictors before extracting emotion representations via contrastive loss.
  • Timbre Module: Employs multi-head self-attention over SeaNet representations with timbre-specific contrastive learning.
  • Information-Theoretic Constraints: Applies mutual information (MI) minimization across factor pairs using neural estimators (CLUB and MINE) with a warm-up schedule to suppress residual cross-talk.
  • Stage 3 (MF-SpeechGenerator): Synthesizes audio using two core mechanisms:
  • Dynamic Fusion: Gating layers compute time-varying weights, dynamically balancing factor influences across speech frames.
  • Hierarchical Style Adaptive Normalization (HSAN): Injects cross-attended timbre and emotion style parameters throughout the residual generator layers using instance normalization, affine scaling/shifting, and an expressive residual modulation term.

Datasets

The authors train and evaluate MF-Speech on the Emotional Speech Dataset (ESD), which provides explicit speaker and emotion labels. The dataset is split into training, seen test, and unseen test sets. Performance is evaluated across 200 speech reconstruction samples ($c = t = e$) and 200 multi-factor compositional samples ($c \neq t \neq e$).

Key Insights and Ideas

  • Mutual information minimization alone is insufficient: Ablation experiments reveal that without branch-specific contrastive losses, factors still overlap severely in t-SNE space. Contrastive objectives are essential to enforce category identity while MI penalizes shared redundancy.
  • Emotion extraction needs explicit prosodic grounding: Removing intermediate F0 and energy supervision destabilizes emotion clustering and indirectly harms timbre purity, proving that prosody cues are the bedrock of emotional expression.
  • Style injection requires multi-level conditioning: Removing HSAN dramatically drops speaker similarity (SECS falls from 0.5685 to 0.1576). Without HSAN, the model defaults to prioritizing intelligibility over style nuances, confirming that dynamic hierarchical modulation is essential for expressive control.

Key Results

MF-Speech was benchmarked against leading voice conversion systems (StyleVC, NS2VC, DDDM-VC, and FACodec):

  • Compositional Speech Generation: When combining content, timbre, and emotion from three distinct sources, MF-Speech outperforms all baselines across intelligibility and similarity metrics:
  • WER: Achieves 4.67% (vs. 11.67% for DDDM-VC and 23.33% for NS2VC).
  • Speaker Similarity (SECS): Reaches 0.5685 (substantially higher than DDDM-VC's 0.3723).
  • F0 Correlation (Corr): Reaches 0.68 with a Log RMSE of 0.34.
  • Subjective Naturalness & Similarity: Scores highest in perceived naturalness (nMOS: 3.96), timbre similarity (sMOS$_t$: 3.86), and emotion similarity (sMOS$_e$: 3.78).
  • Factor Disentanglement: MF-SpeechEncoder exhibits minimal cross-attribute leakage in non-target probe tasks (e.g., speaker identification from the content stream drops to 14.21%, and content classification from the timbre stream drops to 0.54%).

Limitations and Future Work

  • Perceptual Score Anomaly: While MF-Speech scored highest in human subjective evaluations, its automated UTMOS score (2.67) trailed StyleVC (3.22), likely due to pacing and speaking-rate sensitivity in objective MOS estimators.
  • Dataset Scope: Experiments were concentrated on the ESD emotional dataset; further testing across large-scale, in-the-wild, and multi-lingual corpora is necessary to test generalizability.
  • Future Directions: The authors plan to optimize synthesis fidelity, scale the representations as general-purpose discrete speech tokens, and enhance zero-shot generalization to unseen acoustic environments.

Summary updated

Abstract

Generating expressive and controllable human speech is one of the core goals of generative artificial intelligence, but its progress has long been constrained by two fundamental challenges: the deep entanglement of speech factors and the coarse granularity of existing control mechanisms. To overcome these challenges, we have proposed a novel framework called MF-Speech, which consists of two core components: MF-SpeechEncoder and MF-SpeechGenerator. MF-SpeechEncoder acts as a factor purifier, adopting a multi-objective optimization strategy to decompose the original speech signal into highly pure and independent representations of content, timbre, and emotion. Subsequently, MF-SpeechGenerator functions as a conductor, achieving precise, composable and fine-grained control over these factors through dynamic fusion and Hierarchical Style Adaptive Normalization (HSAN). Experiments demonstrate that in the highly challenging multi-factor compositional speech generation task, MF-Speech significantly outperforms current state-of-the-art methods, achieving a lower word error rate (WER=4.67%), superior style control (SECS=0.5685, Corr=0.68), and the highest subjective evaluation scores (nMOS=3.96, sMOS_t=3.86, sMOS_e=3.78). Furthermore, the learned discrete factors exhibit strong transferability, demonstrating their significant potential as a general-purpose speech representation.
BibTeX
@inproceedings{yu2026mf,
  author = {Xinyue Yu and Youqing Fang and Pingyu Wu and Guoyang Ye and Wenbo Zhou and Weiming Zhang and Song Xiao},
  title = {{MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=EX7eLapCzZ},
}

Loading related content...

Figure