This paper from researchers at Xiaomi Corporation introduces a more robust way to handle emotional expression in Text-to-Speech (TTS) systems. By reimagining how Variational Autoencoders (VAEs) are constrained and how emotion data is "injected" into the text encoding process, the authors achieve more expressive and acoustically accurate emotional speech.
TL;DR
The authors propose a "Surface of the Unit Sphere" (SUS) constraint for VAEs to prevent posterior collapse in emotion embeddings and a Multi-Query Attention mechanism to aggregate text encoder layers. This approach ensures that emotion isn't just an afterthought at the decoder stage but actively shapes the syntactic and semantic representation of the text.
What Problem Does This Paper Address?
Emotional speech synthesis faces two primary hurdles:
- Embedding Quality: Learning a distinct "fingerprint" for an emotion from reference audio is often done via VAEs. However, VAEs are prone to "posterior collapse," where the model ignores the latent variables, resulting in embeddings that lack distinct clusters or meaningful variation.
- Effective Injection: Most TTS models simply concatenate an emotion vector to the decoder input. The authors argue this is insufficient because emotion doesn't just change the sound; it changes the prosody and rhythm dictated by the text itself. Current methods often ignore how emotion should influence the textual encoding.
Methods
The paper introduces two major architectural innovations:
1. SUS-constrained VAE
To solve posterior collapse, the authors replace the standard KL-divergence regularizer. Instead of pushing the latent distribution toward a standard normal distribution (which often leads to means collapsing to zero), they use a Surface of the Unit Sphere (SUS) constraint.
- The Logic: They force the means ($\mu$) of the emotion embeddings to lie on the surface of a unit sphere while keeping the standard deviation ($\sigma$) constant (set to 1).
- The Result: This ensures that different emotion clusters remain separated by a predictable distance, preventing them from overlapping or collapsing into a single point.
2. Multi-Query Text Encoder Aggregation
In standard Transformer-based TTS, only the final layer of the text encoder is typically used. However, different layers capture different levels of information (e.g., phonetics vs. syntax).
- Weighted Aggregation: The authors use Multi-Query Attention to learn how to combine representations from all encoder layers.
- Combined Multi-Query: Crucially, they use the emotion embedding from the VAE as one of the "queries" in this attention mechanism. This allows the model to extract emotion-specific sentential information, effectively letting the emotion "decide" which parts of the text's syntactic structure are most important for the final output.
Datasets
The researchers used an internal Mandarin corpus featuring:
- Speaker: One professional male voice talent.
- Emotions: 7 categories (Neutral, Happy, Sad, Angry, Shy, Concerned, Surprised).
- Size: ~4.4 hours total (4,371 utterances), with roughly 500 utterances per emotional category.
- Features: 80-band mel-spectrograms extracted from 16kHz waveforms.
Key Insights and Ideas
- Distance Matters: The authors identify that posterior collapse happens when the distance between means is much larger than their standard deviations. By forcing means to the unit sphere's surface, they maintain a "Goldilocks" distance that preserves the VAE's ability to scale and interpolate emotions.
- Emotion is Syntactic: By using emotion as a query for text encoder aggregation, the paper treats emotion as a lens through which text is interpreted. This suggests that "angry" text and "sad" text should be encoded differently at the structural level, not just rendered differently by the vocoder.
- Cluster Cohesion: t-SNE visualizations (Fig. 3) show that the SUS constraint leads to much tighter, more distinct clusters for different emotions compared to standard VAE approaches.
Key Results
The proposed system (SA-WAC: SAN-based encoder with Weighted Aggregation and Combined multi-query) outperformed the baseline across all metrics:
- Objective Accuracy: Mel Cepstral Distortion (MCD) dropped from 5.5 (Base) to 4.22 (SA-WAC), indicating the generated audio is much closer to the ground truth.
- Subjective Preference: In A/B tests, human listeners preferred the SA-WAC system over the baseline 74% of the time for emotional expressiveness.
- Prosody Correlation: The model showed significantly higher Pearson correlation coefficients for acoustic features like F0 (pitch), Duration, and Energy, proving it better captures the "musicality" of emotional speech.
Limitations and Future Work
- Single Speaker: The study was limited to a single male speaker. The authors suggest that scaling this to multi-speaker or multi-style synthesis is a logical next step.
- Fixed Variance: While setting the standard deviation to a constant (1.0) solved the collapse issue, it might limit the model's ability to represent the "intensity" or "certainty" of an emotion through variance.
- Language Specificity: The results are demonstrated on Mandarin; the impact of these syntactic aggregations might vary in languages with different prosodic structures (like English or pitch-accent languages like Japanese).
Summary updated