Abstract
Learning emotion embedding from reference audio is a straightforward approach for multi-emotion speech synthesis in encoder-decoder systems. But how to get better emotion embedding and how to inject it into TTS acoustic model more effectively are still under investigation. In this paper, we propose an innovative constraint to help VAE extract emotion embedding with better cluster cohesion. Besides, the obtained emotion embedding is used as query to aggregate latent representations of all encoder layers via attention. Moreover, the queries from encoder layers themselves are also helpful. Experiments prove the proposed methods can enhance the encoding of comprehensive syntactic and semantic information and produce more expressive emotional speech.
Introduction
Emotional speech synthesis faces two primary hurdles: embedding quality and effective injection. Learning a distinct "fingerprint" for an emotion from reference audio is often done via Variational Autoencoders (VAEs). However, VAEs are prone to "posterior collapse," where the model ignores latent variables, resulting in embeddings that lack distinct clusters. Furthermore, most TTS models simply concatenate an emotion vector to the decoder input, which fails to account for how emotion influences the prosody and rhythm dictated by the text itself.
Proposed Methodology
The researchers introduce two major architectural innovations to address these issues. First, they implement a Surface of the Unit Sphere (SUS) constraint for the VAE. Instead of the standard KL-divergence regularizer, they force the means ($\mu$) of the emotion embeddings to lie on the surface of a unit sphere while keeping the standard deviation ($\sigma$) constant. This ensures that different emotion clusters remain separated by a predictable distance, preventing them from overlapping or collapsing into a single point.
Second, the model employs Multi-Query Text Encoder Aggregation. Rather than using only the final layer of the text encoder, the system uses Multi-Query Attention to learn a weighted combination of representations from all encoder layers. Crucially, the emotion embedding from the VAE is used as one of the queries, allowing the model to extract emotion-specific sentential information, effectively letting the emotion "decide" which parts of the text's syntactic structure are most important for the final output.
Experimental Results
The system was evaluated on a Mandarin corpus featuring seven emotional categories (Neutral, Happy, Sad, Angry, Shy, Concerned, Surprised). t-SNE visualizations demonstrate that the SUS constraint leads to significantly tighter and more distinct clusters for different emotions compared to standard VAE approaches, effectively mitigating posterior collapse.
The proposed system, SA-WAC (SAN-based encoder with Weighted Aggregation and Combined multi-query), outperformed the baseline across all metrics. Mel Cepstral Distortion (MCD) dropped from 5.5 to 4.22, indicating higher acoustic accuracy. In subjective A/B tests, human listeners preferred the SA-WAC system over the baseline 74% of the time for emotional expressiveness. Additionally, the model showed higher Pearson correlation coefficients for acoustic features like pitch (F0), duration, and energy, proving it better captures the prosodic nuances of emotional speech.
Conclusion
By treating emotion as a lens through which text is interpreted and enforcing geometric constraints on the latent space, the proposed methods significantly improve the expressiveness of emotional TTS. Future work includes scaling these techniques to multi-speaker environments and exploring the impact of variable variance in the SUS-VAE to represent emotional intensity.
Summary updated