Latent Temperature Scaling for Expressive Speech Synthesis

This direction is merged into another direction: output randomness control. Its last version stays readable and citable here.

Version 1 of 114 papers21 conceptsfrozen 2026-10-04

Traditional text-to-speech (TTS) systems often suffer from "over-smoothing," producing speech that sounds robotic or lacks the natural prosodic variations of human emotion. To solve this, researchers have moved toward generative architectures like Variational Autoencoders (VAEs) and Normalizing Flows. These models don't just map text to audio; they map text and a stochastic latent variable to audio.

The core breakthrough across this research is the realization that the latent space acts as a "style controller." By manipulating the variance (or temperature) of this latent space during inference, we can precisely navigate the trade-off between the stability of the speech and its expressive diversity.

The Latent Space as a Prosody Map

In generative models, speech characteristics like pitch, rhythm, and emotional inflection are captured in a latent space, typically modeled as a standard Gaussian distribution $\mathcal{N}(0, I)$.

As Rafael Valle and colleagues demonstrated with Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis, an invertible mapping between mel-spectrograms and this latent space allows for unsupervised control. Because the model learns to organize style in this space, we can "nudge" the latent vector to change an accent or an emotion without needing manual labels for those traits.

However, sampling from this space is risky. If we sample from the high-variance "tails" of the distribution, the model might produce highly expressive but unintelligible or unstable audio.

The Temperature Knob

To manage this risk, researchers use temperature scaling. By multiplying the variance of the latent prior by a factor $T$ (where $0 < T \leq 1$), we shrink the distribution toward its mean.

  • Low Temperature ($T \to 0$): The model samples only the most "probable" or "safe" latent vectors. This results in highly stable, consistent, and clear speech, but it often sounds monotone. Kei Akuzawa et al. found in Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder that reducing variance is essential for autoregressive models to maintain intelligibility.
  • High Temperature ($T \to 1$): The model explores the full range of the learned distribution. This introduces rich prosodic variation—animated pitch shifts and varied durations—but increases the chance of "glitches" or unnatural artifacts.

Jaehyeon Kim and the team behind Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search utilized this mechanism to allow post-training control. Users can decide at the moment of synthesis whether they need a "reliable narrator" (low temperature) or a "dramatic performer" (high temperature) without retraining the model.

From Randomness to Context

While global temperature scaling is powerful, it is still somewhat blunt—it applies the same "stability" constraint to the entire utterance. Recent work has sought to make this latent sampling more intelligent by conditioning the prior on the text itself.

In Cross-Utterance Conditioned VAE for Speech Generation, Yang Li and collaborators replaced the standard Gaussian prior with a conditional prior derived from surrounding sentences using BERT. By using linguistic context to constrain the latent space, the model reduces "stochastic instability." Instead of guessing the prosody randomly, the model uses the semantic meaning of the text to pick a latent region that is both expressive and contextually appropriate.

Why Normalizing Flows?

A recurring theme in this research is the preference for Normalizing Flows over other architectures. Guangyan Zhang and Thomas Merritt compared flows and diffusion models in Comparing normalizing flows and diffusion models for prosody and acoustic modelling in text-to-speech, noting that flows are particularly adept at generating high-quality acoustic mel-spectrograms.

The mathematical invertibility of flows ensures that every point in the latent space corresponds to a valid (if not always perfect) acoustic output, making the "temperature knob" a highly predictable tool for controlling the nuance of synthetic voices.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_temperature_scaling_for_expressiv_158a836f,
  title = {Latent Temperature Scaling for Expressive Speech Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-temperature-scaling-for-expressive-speech-synthesis/rdr_158a836fdfd347b78c6990e0081b99c2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_temperature_scaling_for_expressiv_158a836f_v1,
  title = {Latent Temperature Scaling for Expressive Speech Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-temperature-scaling-for-expressive-speech-synthesis/rdr_158a836fdfd347b78c6990e0081b99c2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}